{"thread":{"id":"53809","subject":"[PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","startedAt":"2020-07-07T14:21:40Z","lastAt":"2020-08-06T17:58:32Z","messageCount":164,"participants":["Derrick Stolee via GitGitGadget","Emily Shaffer","Jonathan Tan","Derrick Stolee","Junio C Hamano","Jeff King","Son Luong Ngoc","Eric Sunshine","Taylor Blau","Đoàn Trần Công Danh","Phillip Wood","Jonathan Nieder","Chris Torek","Junio C Hamano via GitGitGadget"],"isPatch":true,"patchVersion":1,"patchTotal":21},"messages":[{"id":"401098","messageId":"pull.671.git.1594131695.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":null,"subject":"[PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:14Z","receivedAt":"2020-07-07T14:21:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a second attempt at redesigning Git's repository maintenance\npatterns. The first attempt [1] included a way to run jobs in the background\nusing a long-lived process; that idea was rejected and is not included in\nthis series. A future series will use the OS to handle scheduling tasks.\n\n[1] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nAs mentioned before, git gc already plays the role of maintaining Git\nrepositories. It has accumulated several smaller pieces in its long history,\nincluding:\n\n 1. Repacking all reachable objects into one pack-file (and deleting\n unreachable objects).\n 2. Packing refs.\n 3. Expiring reflogs.\n 4. Clearing rerere logs.\n 5. Updating the commit-graph file.\n\nWhile expiring reflogs, clearing rererelogs, and deleting unreachable\nobjects are suitable under the guise of \"garbage collection\", packing refs\nand updating the commit-graph file are not as obviously fitting. Further,\nthese operations are \"all or nothing\" in that they rewrite almost all\nrepository data, which does not perform well at extremely large scales.\nThese operations can also be disruptive to foreground Git commands when git\ngc --auto triggers during routine use.\n\nThis series does not intend to change what git gc does, but instead create\nnew choices for automatic maintenance activities, of which git gc remains\nthe only one enabled by default.\n\nThe new maintenance tasks are:\n\n * 'commit-graph' : write and verify a single layer of an incremental\n commit-graph.\n * 'loose-objects' : prune packed loose objects, then create a new pack from\n a batch of loose objects.\n * 'pack-files' : expire redundant packs from the multi-pack-index, then\n repack using the multi-pack-index's incremental repack strategy.\n * 'fetch' : fetch from each remote, storing the refs in 'refs/hidden//'.\n\nThese tasks are all disabled by default, but can be enabled with config\noptions or run explicitly using \"git maintenance run --task=\". There are\nadditional config options to allow customizing the conditions for which the\ntasks run during the '--auto' option. ('fetch' will never run with the\n'--auto' option.)\n\n Because 'gc' is implemented as a maintenance task, the most dramatic change\nof this series is to convert the 'git gc --auto' calls into 'git maintenance\nrun --auto' calls at the end of some Git commands. By default, the only\nchange is that 'git gc --auto' will be run below an additional 'git\nmaintenance' process.\n\nThe 'git maintenance' builtin has a 'run' subcommand so it can be extended\nlater with subcommands that manage background maintenance, such as 'start',\n'stop', 'pause', or 'schedule'. These are not the subject of this series, as\nit is important to focus on the maintenance activities themselves.\n\nAn expert user could set up scheduled background maintenance themselves with\nthe current series. I have the following crontab data set up to run\nmaintenance on an hourly basis:\n\n0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n\nMy config includes all tasks except the 'gc' task. The hourly run is\nover-aggressive, but is sufficient for testing. I'll replace it with daily\nwhen I feel satisfied.\n\nHopefully this direction is seen as a positive one. My goal was to add more\noptions for expert users, along with the flexibility to create background\nmaintenance via the OS in a later series.\n\nOUTLINE\n=======\n\nPatches 1-4 remove some references to the_repository in builtin/gc.c before\nwe start depending on code in that builtin.\n\nPatches 5-7 create the 'git maintenance run' builtin and subcommand as a\nsimple shim over 'git gc' and replaces calls to 'git gc --auto' from other\ncommands.\n\nPatches 8-15 create new maintenance tasks. These are the same tasks sent in\nthe previous RFC.\n\nPatches 16-21 create more customization through config and perform other\npolish items.\n\nFUTURE WORK\n===========\n\n * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n commands run in the background.\n \n \n * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n default, but might have different '--auto' conditions and more config\n options.\n \n \n * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n with use of the 'commit-graph' task.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (21):\n gc: use the_repository less often\n gc: use repository in too_many_loose_objects()\n gc: use repo config\n gc: drop the_repository in log location\n maintenance: create basic maintenance runner\n maintenance: add --quiet option\n maintenance: replace run_auto_gc()\n maintenance: initialize task array and hashmap\n maintenance: add commit-graph task\n maintenance: add --task option\n maintenance: take a lock on the objects directory\n maintenance: add fetch task\n maintenance: add loose-objects task\n maintenance: add pack-files task\n maintenance: auto-size pack-files batch\n maintenance: create maintenance..enabled config\n maintenance: use pointers to check --auto\n maintenance: add auto condition for commit-graph task\n maintenance: create auto condition for loose-objects\n maintenance: add pack-files auto condition\n midx: use start_delayed_progress()\n\n .gitignore | 1 +\n Documentation/config.txt | 2 +\n Documentation/config/maintenance.txt | 32 +\n Documentation/fetch-options.txt | 5 +-\n Documentation/git-clone.txt | 7 +-\n Documentation/git-maintenance.txt | 124 ++++\n builtin.h | 1 +\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 6 +-\n builtin/gc.c | 881 +++++++++++++++++++++++++--\n builtin/merge.c | 2 +-\n builtin/rebase.c | 4 +-\n commit-graph.c | 8 +-\n commit-graph.h | 1 +\n config.c | 24 +-\n config.h | 2 +\n git.c | 1 +\n midx.c | 12 +-\n midx.h | 1 +\n object.h | 1 +\n run-command.c | 7 +-\n run-command.h | 2 +-\n t/t5319-multi-pack-index.sh | 14 +-\n t/t5510-fetch.sh | 2 +-\n t/t5514-fetch-multiple.sh | 2 +-\n t/t7900-maintenance.sh | 211 +++++++\n 27 files changed, 1265 insertions(+), 92 deletions(-)\n create mode 100644 Documentation/config/maintenance.txt\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\n\nbase-commit: 4a0fcf9f760c9774be77f51e1e88a7499b53d2e2\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-671%2Fderrickstolee%2Fmaintenance%2Fgc-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-671/derrickstolee/maintenance/gc-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/671\n-- \ngitgitgadget\n"},{"id":"401099","messageId":"85dda7db28ddf2099ccd35924eb168964e55cf1f.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 01/21] gc: use the_repository less often","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:15Z","receivedAt":"2020-07-07T14:21:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn a later change, we will consume several static methods in\nbuiltin/gc.c for another builtin. Before doing so, let's clean up some\nuses of the_repository. These specifically are centered around accesses\nto the packed_git list.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 38 ++++++++++++++++++++------------------\n 1 file changed, 20 insertions(+), 18 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 8e0b9cf41b..5c5e0df5bf 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -192,12 +192,13 @@ static int too_many_loose_objects(void)\n \treturn needed;\n }\n \n-static struct packed_git *find_base_packs(struct string_list *packs,\n+static struct packed_git *find_base_packs(struct repository *r,\n+\t\t\t\t\t struct string_list *packs,\n \t\t\t\t\t unsigned long limit)\n {\n \tstruct packed_git *p, *base = NULL;\n \n-\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (p = get_all_packs(r); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (limit) {\n@@ -214,7 +215,7 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \treturn base;\n }\n \n-static int too_many_packs(void)\n+static int too_many_packs(struct repository *r)\n {\n \tstruct packed_git *p;\n \tint cnt;\n@@ -222,7 +223,7 @@ static int too_many_packs(void)\n \tif (gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n-\tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n+\tfor (cnt = 0, p = get_all_packs(r); p; p = p->next) {\n \t\tif (!p->pack_local)\n \t\t\tcontinue;\n \t\tif (p->pack_keep)\n@@ -334,7 +335,7 @@ static void add_repack_incremental_option(void)\n \targv_array_push(&repack, \"--no-write-bitmap-index\");\n }\n \n-static int need_to_gc(void)\n+static int need_to_gc(struct repository *r)\n {\n \t/*\n \t * Setting gc.auto to 0 or negative can disable the\n@@ -349,18 +350,18 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs()) {\n+\tif (too_many_packs(r)) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n \t\tif (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tfind_base_packs(r, &keep_pack, big_pack_threshold);\n \t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n \t\t\t\tbig_pack_threshold = 0;\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n-\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(r, &keep_pack, 0);\n \t\t\t}\n \t\t} else {\n-\t\t\tstruct packed_git *p = find_base_packs(&keep_pack, 0);\n+\t\t\tstruct packed_git *p = find_base_packs(r, &keep_pack, 0);\n \t\t\tuint64_t mem_have, mem_want;\n \n \t\t\tmem_have = total_ram();\n@@ -523,6 +524,7 @@ static void gc_before_repack(void)\n \n int cmd_gc(int argc, const char **argv, const char *prefix)\n {\n+\tstruct repository *r = the_repository;\n \tint aggressive = 0;\n \tint auto_gc = 0;\n \tint quiet = 0;\n@@ -589,7 +591,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc())\n+\t\tif (!need_to_gc(r))\n \t\t\treturn 0;\n \t\tif (!quiet) {\n \t\t\tif (detach_auto)\n@@ -623,9 +625,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\tif (keep_base_pack != -1) {\n \t\t\tif (keep_base_pack)\n-\t\t\t\tfind_base_packs(&keep_pack, 0);\n+\t\t\t\tfind_base_packs(r, &keep_pack, 0);\n \t\t} else if (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t\tfind_base_packs(r, &keep_pack, big_pack_threshold);\n \t\t}\n \n \t\tadd_repack_all_option(&keep_pack);\n@@ -652,7 +654,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tgc_before_repack();\n \n \tif (!repository_format_precious_objects) {\n-\t\tclose_object_store(the_repository->objects);\n+\t\tclose_object_store(r->objects);\n \t\tif (run_command_v_opt(repack.argv, RUN_GIT_CMD))\n \t\t\tdie(FAILED_RUN, repack.argv[0]);\n \n@@ -678,15 +680,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tdie(FAILED_RUN, rerere.argv[0]);\n \n \treport_garbage = report_pack_garbage;\n-\treprepare_packed_git(the_repository);\n+\treprepare_packed_git(r);\n \tif (pack_garbage.nr > 0) {\n-\t\tclose_object_store(the_repository->objects);\n+\t\tclose_object_store(r->objects);\n \t\tclean_pack_garbage();\n \t}\n \n-\tprepare_repo_settings(the_repository);\n-\tif (the_repository->settings.gc_write_commit_graph == 1)\n-\t\twrite_commit_graph_reachable(the_repository->objects->odb,\n+\tprepare_repo_settings(r);\n+\tif (r->settings.gc_write_commit_graph == 1)\n+\t\twrite_commit_graph_reachable(r->objects->odb,\n \t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,\n \t\t\t\t\t NULL);\n \n-- \ngitgitgadget\n\n"},{"id":"401100","messageId":"303ad4bdc7286522e83439b3c6bfdb408ed69407.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 02/21] gc: use repository in too_many_loose_objects()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:16Z","receivedAt":"2020-07-07T14:21:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change performed a mechanical change to swap the_repository\nfor a struct repository pointer when the use of the_repository was\nobvious. However, the too_many_loose_objects() method uses git_path()\ninstead of repo_git_path(), which implies a hidden dependence on\nthe_repository.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 5c5e0df5bf..6d8267cecb 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -159,7 +159,7 @@ static void gc_config(void)\n \tgit_config(git_default_config, NULL);\n }\n \n-static int too_many_loose_objects(void)\n+static int too_many_loose_objects(struct repository *r)\n {\n \t/*\n \t * Quickly check if a \"gc\" is needed, by estimating how\n@@ -174,7 +174,7 @@ static int too_many_loose_objects(void)\n \tint needed = 0;\n \tconst unsigned hexsz_loose = the_hash_algo->hexsz - 2;\n \n-\tdir = opendir(git_path(\"objects/17\"));\n+\tdir = opendir(repo_git_path(r, \"objects/17\"));\n \tif (!dir)\n \t\treturn 0;\n \n@@ -378,7 +378,7 @@ static int need_to_gc(struct repository *r)\n \n \t\tadd_repack_all_option(&keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-\t} else if (too_many_loose_objects())\n+\t} else if (too_many_loose_objects(r))\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -692,7 +692,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,\n \t\t\t\t\t NULL);\n \n-\tif (auto_gc && too_many_loose_objects())\n+\tif (auto_gc && too_many_loose_objects(r))\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n \n-- \ngitgitgadget\n\n"},{"id":"401101","messageId":"9dfa9e8f6f3df73880464afe120ce912436416b2.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 03/21] gc: use repo config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:17Z","receivedAt":"2020-07-07T14:21:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe gc builtin is working towards using a repository pointer instead of\nrelying on the_repository everywhere. Some of the instances of\nthe_repository are hidden underneath git_config...() methods. Expose\nthem by using repo_config...() instead.\n\nOne method did not previously have a repo equivalent:\ngit_config_get_expiry().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 37 +++++++++++++++++++------------------\n config.c | 24 +++++++++++++++---------\n config.h | 2 ++\n 3 files changed, 36 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 6d8267cecb..888b6444d6 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -116,12 +116,13 @@ static void process_log_file_on_signal(int signo)\n \traise(signo);\n }\n \n-static int gc_config_is_timestamp_never(const char *var)\n+static int gc_config_is_timestamp_never(struct repository *r,\n+\t\t\t\t\tconst char *var)\n {\n \tconst char *value;\n \ttimestamp_t expire;\n \n-\tif (!git_config_get_value(var, &value) && value) {\n+\tif (!repo_config_get_value(r, var, &value) && value) {\n \t\tif (parse_expiry_date(value, &expire))\n \t\t\tdie(_(\"failed to parse '%s' value '%s'\"), var, value);\n \t\treturn expire == 0;\n@@ -129,34 +130,34 @@ static int gc_config_is_timestamp_never(const char *var)\n \treturn 0;\n }\n \n-static void gc_config(void)\n+static void gc_config(struct repository *r)\n {\n \tconst char *value;\n \n-\tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n+\tif (!repo_config_get_value(r, \"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n \t\t\tpack_refs = -1;\n \t\telse\n \t\t\tpack_refs = git_config_bool(\"gc.packrefs\", value);\n \t}\n \n-\tif (gc_config_is_timestamp_never(\"gc.reflogexpire\") &&\n-\t gc_config_is_timestamp_never(\"gc.reflogexpireunreachable\"))\n+\tif (gc_config_is_timestamp_never(r, \"gc.reflogexpire\") &&\n+\t gc_config_is_timestamp_never(r, \"gc.reflogexpireunreachable\"))\n \t\tprune_reflogs = 0;\n \n-\tgit_config_get_int(\"gc.aggressivewindow\", &aggressive_window);\n-\tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n-\tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n-\tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n-\tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n+\trepo_config_get_int(r, \"gc.aggressivewindow\", &aggressive_window);\n+\trepo_config_get_int(r, \"gc.aggressivedepth\", &aggressive_depth);\n+\trepo_config_get_int(r, \"gc.auto\", &gc_auto_threshold);\n+\trepo_config_get_int(r, \"gc.autopacklimit\", &gc_auto_pack_limit);\n+\trepo_config_get_bool(r, \"gc.autodetach\", &detach_auto);\n+\trepo_config_get_expiry(r, \"gc.pruneexpire\", &prune_expire);\n+\trepo_config_get_expiry(r, \"gc.worktreepruneexpire\", &prune_worktrees_expire);\n+\trepo_config_get_expiry(r, \"gc.logexpiry\", &gc_log_expire);\n \n-\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n-\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\trepo_config_get_ulong(r, \"gc.bigpackthreshold\", &big_pack_threshold);\n+\trepo_config_get_ulong(r, \"pack.deltacachesize\", &max_delta_cache_size);\n \n-\tgit_config(git_default_config, NULL);\n+\trepo_config(r, git_default_config, NULL);\n }\n \n static int too_many_loose_objects(struct repository *r)\n@@ -562,7 +563,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \targv_array_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n-\tgc_config();\n+\tgc_config(r);\n \tif (parse_expiry_date(gc_log_expire, &gc_log_expire_time))\n \t\tdie(_(\"failed to parse gc.logexpiry value %s\"), gc_log_expire);\n \ndiff --git a/config.c b/config.c\nindex 8db9c77098..fe83f8b67c 100644\n--- a/config.c\n+++ b/config.c\n@@ -2211,6 +2211,20 @@ int repo_config_get_pathname(struct repository *repo,\n \treturn ret;\n }\n \n+int repo_config_get_expiry(struct repository *repo,\n+\t\t\t const char *key, const char **output)\n+{\n+\tint ret = repo_config_get_string_const(repo, key, output);\n+\tif (ret)\n+\t\treturn ret;\n+\tif (strcmp(*output, \"now\")) {\n+\t\ttimestamp_t now = approxidate(\"now\");\n+\t\tif (approxidate(*output) >= now)\n+\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n+\t}\n+\treturn ret;\n+}\n+\n /* Functions used historically to read configuration from 'the_repository' */\n void git_config(config_fn_t fn, void *data)\n {\n@@ -2274,15 +2288,7 @@ int git_config_get_pathname(const char *key, const char **dest)\n \n int git_config_get_expiry(const char *key, const char **output)\n {\n-\tint ret = git_config_get_string_const(key, output);\n-\tif (ret)\n-\t\treturn ret;\n-\tif (strcmp(*output, \"now\")) {\n-\t\ttimestamp_t now = approxidate(\"now\");\n-\t\tif (approxidate(*output) >= now)\n-\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n-\t}\n-\treturn ret;\n+\treturn repo_config_get_expiry(the_repository, key, output);\n }\n \n int git_config_get_expiry_in_days(const char *key, timestamp_t *expiry, timestamp_t now)\ndiff --git a/config.h b/config.h\nindex 060874488f..085bcf6917 100644\n--- a/config.h\n+++ b/config.h\n@@ -490,6 +490,8 @@ int repo_config_get_maybe_bool(struct repository *repo,\n \t\t\t const char *key, int *dest);\n int repo_config_get_pathname(struct repository *repo,\n \t\t\t const char *key, const char **dest);\n+int repo_config_get_expiry(struct repository *repo,\n+\t\t\t const char *key, const char **output);\n \n /**\n * Querying For Specific Variables\n-- \ngitgitgadget\n\n"},{"id":"401102","messageId":"5f89e0ec9b9c9a02deed5b79b35253e0462dc87c.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 05/21] maintenance: create basic maintenance runner","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:19Z","receivedAt":"2020-07-07T14:21:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'gc' builtin is our current entrypoint for automatically maintaining\na repository. This one tool does many operations, such as repacking the\nrepository, packing refs, and rewriting the commit-graph file. The name\nimplies it performs \"garbage collection\" which means several different\nthings, and some users may not want to use this operation that rewrites\nthe entire object database.\n\nCreate a new 'maintenance' builtin that will become a more general-\npurpose command. To start, it will only support the 'run' subcommand,\nbut will later expand to add subcommands for scheduling maintenance in\nthe background.\n\nFor now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.\nIn fact, the only option is the '--auto' toggle, which is handed\ndirectly to the 'gc' builtin. The current change is isolated to this\nsimple operation to prevent more interesting logic from being lost in\nall of the boilerplate of adding a new builtin.\n\nUse existing builtin/gc.c file because we want to share code between the\ntwo builtins. It is possible that we will have 'maintenance' replace the\n'gc' builtin entirely at some point, leaving 'git gc' as an alias for\nsome specific arguments to 'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-maintenance.txt | 57 +++++++++++++++++++++++++++++\n builtin.h | 1 +\n builtin/gc.c | 61 +++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t7900-maintenance.sh | 22 +++++++++++\n 6 files changed, 143 insertions(+)\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex ee509a2ad2..a5808fa30d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -90,6 +90,7 @@\n /git-ls-tree\n /git-mailinfo\n /git-mailsplit\n+/git-maintenance\n /git-merge\n /git-merge-base\n /git-merge-index\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nnew file mode 100644\nindex 0000000000..34cd2b4417\n--- /dev/null\n+++ b/Documentation/git-maintenance.txt\n@@ -0,0 +1,57 @@\n+git-maintenance(1)\n+==================\n+\n+NAME\n+----\n+git-maintenance - Run tasks to optimize Git repository data\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git maintenance' run []\n+\n+\n+DESCRIPTION\n+-----------\n+Run tasks to optimize Git repository data, speeding up other Git commands\n+and reducing storage requirements for the repository.\n++\n+Git commands that add repository data, such as `git add` or `git fetch`,\n+are optimized for a responsive user experience. These commands do not take\n+time to optimize the Git data, since such optimizations scale with the full\n+size of the repository while these user commands each perform a relatively\n+small action.\n++\n+The `git maintenance` command provides flexibility for how to optimize the\n+Git repository.\n+\n+SUBCOMMANDS\n+-----------\n+\n+run::\n+\tRun one or more maintenance tasks.\n+\n+TASKS\n+-----\n+\n+gc::\n+\tCleanup unnecessary files and optimize the local repository. \"GC\"\n+\tstands for \"garbage collection,\" but this task performs many\n+\tsmaller tasks. This task can be rather expensive for large\n+\trepositories, as it repacks all Git objects into a single pack-file.\n+\tIt can also be disruptive in some situations, as it deletes stale\n+\tdata.\n+\n+OPTIONS\n+-------\n+--auto::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain thresholds are met. For example, the `gc` task\n+\truns when the number of loose objects exceeds the number stored\n+\tin the `gc.auto` config setting, or when the number of pack-files\n+\texceeds the `gc.autoPackLimit` config setting.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/builtin.h b/builtin.h\nindex a5ae15bfe5..17c1c0ce49 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix);\n int cmd_ls_remote(int argc, const char **argv, const char *prefix);\n int cmd_mailinfo(int argc, const char **argv, const char *prefix);\n int cmd_mailsplit(int argc, const char **argv, const char *prefix);\n+int cmd_maintenance(int argc, const char **argv, const char *prefix);\n int cmd_merge(int argc, const char **argv, const char *prefix);\n int cmd_merge_base(int argc, const char **argv, const char *prefix);\n int cmd_merge_index(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 58a00be5af..07cc5f46ae 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -704,3 +704,64 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \treturn 0;\n }\n+\n+static const char * const builtin_maintenance_usage[] = {\n+\tN_(\"git maintenance run []\"),\n+\tNULL\n+};\n+\n+struct maintenance_opts {\n+\tint auto_flag;\n+} opts;\n+\n+static int maintenance_task_gc(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"gc\", NULL);\n+\n+\tif (opts.auto_flag)\n+\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n+\n+\tclose_object_store(r->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int maintenance_run(struct repository *r)\n+{\n+\treturn maintenance_task_gc(r);\n+}\n+\n+int cmd_maintenance(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct repository *r = the_repository;\n+\n+\tstatic struct option builtin_maintenance_options[] = {\n+\t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n+\t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tmemset(&opts, 0, sizeof(opts));\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t\t builtin_maintenance_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_maintenance_options,\n+\t\t\t builtin_maintenance_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (argc == 1) {\n+\t\tif (!strcmp(argv[0], \"run\"))\n+\t\t\treturn maintenance_run(r);\n+\t}\n+\n+\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t builtin_maintenance_options);\n+}\ndiff --git a/git.c b/git.c\nindex 2f021b97f3..ff56d1df24 100644\n--- a/git.c\n+++ b/git.c\n@@ -527,6 +527,7 @@ static struct cmd_struct commands[] = {\n \t{ \"ls-tree\", cmd_ls_tree, RUN_SETUP },\n \t{ \"mailinfo\", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"mailsplit\", cmd_mailsplit, NO_PARSEOPT },\n+\t{ \"maintenance\", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"merge\", cmd_merge, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"merge-base\", cmd_merge_base, RUN_SETUP },\n \t{ \"merge-file\", cmd_merge_file, RUN_SETUP_GENTLY },\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nnew file mode 100755\nindex 0000000000..d00641c4dd\n--- /dev/null\n+++ b/t/t7900-maintenance.sh\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+\n+test_description='git maintenance builtin'\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+GIT_TEST_MULTI_PACK_INDEX=0\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'help text' '\n+\ttest_must_fail git maintenance -h 2>err &&\n+\ttest_i18ngrep \"usage: git maintenance run\" err\n+'\n+\n+test_expect_success 'gc [--auto]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"401103","messageId":"018a9331e204fd6b1b92f5b53f3f373fa9e23bc2.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 06/21] maintenance: add --quiet option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:20Z","receivedAt":"2020-07-07T14:21:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance activities are commonly used as steps in larger scripts.\nProviding a '--quiet' option allows those scripts to be less noisy when\nrun on a terminal window. Turn this mode on by default when stderr is\nnot a terminal.\n\nPipe the option to the 'git gc' child process.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 3 +++\n builtin/gc.c | 7 +++++++\n t/t7900-maintenance.sh | 8 +++++---\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 34cd2b4417..089fa4cedc 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -52,6 +52,9 @@ OPTIONS\n \tin the `gc.auto` config setting, or when the number of pack-files\n \texceeds the `gc.autoPackLimit` config setting.\n \n+--quiet::\n+\tDo not report progress or other information over `stderr`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 07cc5f46ae..3881a99e9d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -712,6 +712,7 @@ static const char * const builtin_maintenance_usage[] = {\n \n struct maintenance_opts {\n \tint auto_flag;\n+\tint quiet;\n } opts;\n \n static int maintenance_task_gc(struct repository *r)\n@@ -723,6 +724,8 @@ static int maintenance_task_gc(struct repository *r)\n \n \tif (opts.auto_flag)\n \t\targv_array_pushl(&cmd, \"--auto\", NULL);\n+\tif (opts.quiet)\n+\t\targv_array_pushl(&cmd, \"--quiet\", NULL);\n \n \tclose_object_store(r->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n@@ -743,6 +746,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstatic struct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n+\t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_END()\n \t};\n \n@@ -752,6 +757,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_maintenance_usage,\n \t\t\t\t builtin_maintenance_options);\n \n+\topts.quiet = !isatty(2);\n+\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\n \t\t\t builtin_maintenance_usage,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex d00641c4dd..e4e4036e50 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -12,11 +12,13 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n-test_expect_success 'gc [--auto]' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+test_expect_success 'gc [--auto|--quiet]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"401106","messageId":"5cdd38afa60cdf768dd194f90ae0b2190123fdea.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 08/21] maintenance: initialize task array and hashmap","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:22Z","receivedAt":"2020-07-07T14:21:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of implementing multiple maintenance tasks inside the\n'maintenance' builtin, use a list and hashmap of structs to describe the\nwork to be done.\n\nThe struct maintenance_task stores the name of the task (as given by a\nfuture command-line argument) along with a function pointer to its\nimplementation and a boolean for whether the step is enabled.\n\nA list of pointers to these structs are initialized with the full list\nof implemented tasks along with a default order. For now, this list only\ncontains the \"gc\" task. This task is also the only task enabled by\ndefault.\n\nThis list is also inserted into a hashmap. This allows command-line\narguments to quickly find the tasks by name, not sensitive to case. To\nensure this list and hashmap work well together, the list only contains\npointers to the struct information. This will allow a sort on the list\nwhile preserving the hashmap data.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 62 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3881a99e9d..c143bf50df 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -705,6 +705,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+#define MAX_NUM_TASKS 1\n+\n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n \tNULL\n@@ -734,9 +736,67 @@ static int maintenance_task_gc(struct repository *r)\n \treturn result;\n }\n \n+typedef int maintenance_task_fn(struct repository *r);\n+\n+struct maintenance_task {\n+\tstruct hashmap_entry ent;\n+\tconst char *name;\n+\tmaintenance_task_fn *fn;\n+\tunsigned enabled:1;\n+};\n+\n+static int task_entry_cmp(const void *unused_cmp_data,\n+\t\t\t const struct hashmap_entry *eptr,\n+\t\t\t const struct hashmap_entry *entry_or_key,\n+\t\t\t const void *keydata)\n+{\n+\tconst struct maintenance_task *e1, *e2;\n+\tconst char *name = keydata;\n+\n+\te1 = container_of(eptr, const struct maintenance_task, ent);\n+\te2 = container_of(entry_or_key, const struct maintenance_task, ent);\n+\n+\treturn strcasecmp(e1->name, name ? name : e2->name);\n+}\n+\n+struct maintenance_task *tasks[MAX_NUM_TASKS];\n+int num_tasks;\n+struct hashmap task_map;\n+\n static int maintenance_run(struct repository *r)\n {\n-\treturn maintenance_task_gc(r);\n+\tint i;\n+\tint result = 0;\n+\n+\tfor (i = 0; !result && i < num_tasks; i++) {\n+\t\tif (!tasks[i]->enabled)\n+\t\t\tcontinue;\n+\t\tresult = tasks[i]->fn(r);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static void initialize_tasks(void)\n+{\n+\tint i;\n+\tnum_tasks = 0;\n+\n+\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n+\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n+\n+\ttasks[num_tasks]->name = \"gc\";\n+\ttasks[num_tasks]->fn = maintenance_task_gc;\n+\ttasks[num_tasks]->enabled = 1;\n+\tnum_tasks++;\n+\n+\thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n+\n+\tfor (i = 0; i < num_tasks; i++) {\n+\t\thashmap_entry_init(&tasks[i]->ent,\n+\t\t\t\t strihash(tasks[i]->name));\n+\t\thashmap_add(&task_map, &tasks[i]->ent);\n+\t}\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -758,6 +818,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n+\tinitialize_tasks();\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\n-- \ngitgitgadget\n\n"},{"id":"401104","messageId":"335a8938c697293bf70f553dcd4b8c53b5097b3c.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 07/21] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:21Z","receivedAt":"2020-07-07T14:21:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe run_auto_gc() method is used in several places to trigger a check\nfor repo maintenance after some Git commands, such as 'git commit' or\n'git fetch'.\n\nTo allow for extra customization of this maintenance activity, replace\nthe 'git gc --auto [--quiet]' call with one to 'git maintenance run\n--auto [--quiet]'. As we extend the maintenance builtin with other\nsteps, users will be able to select different maintenance activities.\n\nRename run_auto_gc() to run_auto_maintenance() to be clearer what is\nhappening on this call, and to expose all callers in the current diff.\n\nSince 'git fetch' already allows disabling the 'git gc --auto'\nsubprocess, add an equivalent option with a different name to be more\ndescriptive of the new behavior: '--[no-]maintenance'. Update the\ndocumentation to include these options at the same time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 5 +++--\n Documentation/git-clone.txt | 7 ++++---\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 6 ++++--\n builtin/merge.c | 2 +-\n builtin/rebase.c | 4 ++--\n run-command.c | 7 +++++--\n run-command.h | 2 +-\n t/t5510-fetch.sh | 2 +-\n 10 files changed, 23 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 6e2a160a47..d73224844e 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -86,9 +86,10 @@ ifndef::git-pull[]\n \tAllow several and arguments to be\n \tspecified. No s may be specified.\n \n+--[no-]maintenance::\n --[no-]auto-gc::\n-\tRun `git gc --auto` at the end to perform garbage collection\n-\tif needed. This is enabled by default.\n+\tRun `git maintenance run --auto` at the end to perform garbage\n+\tcollection if needed. This is enabled by default.\n \n --[no-]write-commit-graph::\n \tWrite a commit-graph after fetching. This overrides the config\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex c898310099..aa25aba7d9 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -78,9 +78,10 @@ repository using this option and then delete branches (or use any\n other Git command that makes any existing commit unreferenced) in the\n source repository, some objects may become unreferenced (or dangling).\n These objects may be removed by normal Git operations (such as `git commit`)\n-which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n-If these objects are removed and were referenced by the cloned repository,\n-then the cloned repository will become corrupt.\n+which automatically call `git maintenance run --auto` and `git gc --auto`.\n+(See linkgit:git-maintenance[1] and linkgit:git-gc[1].) If these objects\n+are removed and were referenced by the cloned repository, then the cloned\n+repository will become corrupt.\n +\n Note that running `git repack` without the `--local` option in a repository\n cloned with `--shared` will copy objects from the source repository into a pack\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 69e50de018..ff895125f6 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume)\n \tif (!state->rebasing) {\n \t\tam_destroy(state);\n \t\tclose_object_store(the_repository->objects);\n-\t\trun_auto_gc(state->quiet);\n+\t\trun_auto_maintenance(state->quiet);\n \t}\n }\n \ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex d1b7396052..658b158659 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tgit_test_write_commit_graph_or_die();\n \n \trepo_rerere(the_repository, 0);\n-\trun_auto_gc(quiet);\n+\trun_auto_maintenance(quiet);\n \trun_commit_hook(use_editor, get_index_file(), \"post-commit\", NULL);\n \tif (amend && !no_post_rewrite) {\n \t\tcommit_post_rewrite(the_repository, current_head, &oid);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82ac4be8a5..49a4d727d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n \t\t\tN_(\"report that we have only objects reachable from this object\")),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n-\t\t N_(\"run 'gc --auto' after fetching\")),\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"show-forced-updates\", &fetch_show_forced_updates,\n \t\t N_(\"check for forced-updates on all updated branches\")),\n \tOPT_BOOL(0, \"write-commit-graph\", &fetch_write_commit_graph,\n@@ -1882,7 +1884,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tclose_object_store(the_repository->objects);\n \n \tif (enable_auto_gc)\n-\t\trun_auto_gc(verbosity < 0);\n+\t\trun_auto_maintenance(verbosity < 0);\n \n \treturn result;\n }\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 7da707bf55..c068e73037 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -457,7 +457,7 @@ static void finish(struct commit *head_commit,\n \t\t\t * user should see them.\n \t\t\t */\n \t\t\tclose_object_store(the_repository->objects);\n-\t\t\trun_auto_gc(verbosity < 0);\n+\t\t\trun_auto_maintenance(verbosity < 0);\n \t\t}\n \t}\n \tif (new_head && show_diffstat) {\ndiff --git a/builtin/rebase.c b/builtin/rebase.c\nindex 37ba76ac3d..0c4ee98f08 100644\n--- a/builtin/rebase.c\n+++ b/builtin/rebase.c\n@@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts)\n \tapply_autostash(state_dir_path(\"autostash\", opts));\n \tclose_object_store(the_repository->objects);\n \t/*\n-\t * We ignore errors in 'gc --auto', since the\n+\t * We ignore errors in 'git maintenance run --auto', since the\n \t * user should see them.\n \t */\n-\trun_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n+\trun_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n \tif (opts->type == REBASE_MERGE) {\n \t\tstruct replay_opts replay = REPLAY_OPTS_INIT;\n \ndiff --git a/run-command.c b/run-command.c\nindex 9b3a57d1e3..82ad241638 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1865,14 +1865,17 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \treturn result;\n }\n \n-int run_auto_gc(int quiet)\n+int run_auto_maintenance(int quiet)\n {\n \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n \tint status;\n \n-\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n+\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n \tif (quiet)\n \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n+\telse\n+\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n+\n \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n \targv_array_clear(&argv_gc_auto);\n \treturn status;\ndiff --git a/run-command.h b/run-command.h\nindex 191dfcdafe..d9a800e700 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args);\n /*\n * Trigger an auto-gc\n */\n-int run_auto_gc(int quiet);\n+int run_auto_maintenance(int quiet);\n \n #define RUN_COMMAND_NO_STDIN 1\n #define RUN_GIT_CMD\t 2\t/*If this is to be git sub-command */\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex a66dbe0bde..9850ecde5d 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' '\n \t\tgit config fetch.unpackLimit 1 &&\n \t\tgit config gc.autoPackLimit 1 &&\n \t\tgit config gc.autoDetach false &&\n-\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch >fetch.out 2>&1 &&\n+\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch --verbose >fetch.out 2>&1 &&\n \t\ttest_i18ngrep \"Auto packing the repository\" fetch.out &&\n \t\t! grep \"Should I try again\" fetch.out\n \t)\n-- \ngitgitgadget\n\n"},{"id":"401105","messageId":"c8fbd14d41093e12905027ff628b98652cd931d0.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 09/21] maintenance: add commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:23Z","receivedAt":"2020-07-07T14:21:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe first new task in the 'git maintenance' builtin is the\n'commit-graph' job. It is based on the sequence of events in the\n'commit-graph' job in Scalar [1]. This sequence is as follows:\n\n1. git commit-graph write --reachable --split\n2. git commit-graph verify --shallow\n3. If the verify succeeds, stop.\n4. Delete the commit-graph-chain file.\n5. git commit-graph write --reachable --split\n\nBy writing an incremental commit-graph file using the \"--split\"\noption we minimize the disruption from this operation. The default\nbehavior is to merge layers until the new \"top\" layer is less than\nhalf the size of the layer below. This provides quick writes most\nof the time, with the longer writes following a power law\ndistribution.\n\nMost importantly, concurrent Git processes only look at the\ncommit-graph-chain file for a very short amount of time, so they\nwill verly likely not be holding a handle to the file when we try\nto replace it. (This only matters on Windows.)\n\nIf a concurrent process reads the old commit-graph-chain file, but\nour job expires some of the .graph files before they can be read,\nthen those processes will see a warning message (but not fail).\nThis could be avoided by a future update to use the --expire-time\nargument when writing the commit-graph.\n\nBy using 'git commit-graph verify --shallow' we can ensure that\nthe file we just wrote is valid. This is an extra safety precaution\nthat is faster than our 'write' subcommand. In the rare situation\nthat the newest layer of the commit-graph is corrupt, we can \"fix\"\nthe corruption by deleting the commit-graph-chain file and rewrite\nthe full commit-graph as a new one-layer commit graph. This does\nnot completely prevent _that_ file from being corrupt, but it does\nrecompute the commit-graph by parsing commits from the object\ndatabase. In our use of this step in Scalar and VFS for Git, we\nhave only seen this issue arise because our microsoft/git fork\nreverted 43d3561 (\"commit-graph write: don't die if the existing\ngraph is corrupt\" 2019-03-25) for a while to keep commit-graph\nwrites very fast. We dropped the revert when updating to v2.23.0.\nThe verify still has potential for catching corrupt data across\nthe layer boundary: if the new file has commit X with parent Y\nin an old file but the commit ID for Y in the old file had a\nbitswap, then we will notice that in the 'verify' command.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 18 ++++++++\n builtin/gc.c | 76 ++++++++++++++++++++++++++++++-\n commit-graph.c | 8 ++--\n commit-graph.h | 1 +\n t/t7900-maintenance.sh | 2 +-\n 5 files changed, 99 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 089fa4cedc..35b0be7d40 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -35,6 +35,24 @@ run::\n TASKS\n -----\n \n+commit-graph::\n+\tThe `commit-graph` job updates the `commit-graph` files incrementally,\n+\tthen verifies that the written data is correct. If the new layer has an\n+\tissue, then the chain file is removed and the `commit-graph` is\n+\trewritten from scratch.\n++\n+The verification only checks the top layer of the `commit-graph` chain.\n+If the incremental write merged the new commits with at least one\n+existing layer, then there is potential for on-disk corruption being\n+carried forward into the new file. This will be noticed and the new\n+commit-graph file will be clean as Git reparses the commit data from\n+the object database.\n++\n+The incremental write is safe to run alongside concurrent Git processes\n+since it will not expire `.graph` files that were in the previous\n+`commit-graph-chain` file. They will be deleted by a later run based on\n+the expiration delay.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c143bf50df..a6b080627f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -705,7 +705,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 1\n+#define MAX_NUM_TASKS 2\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -717,6 +717,76 @@ struct maintenance_opts {\n \tint quiet;\n } opts;\n \n+static int run_write_commit_graph(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"commit-graph\", \"write\",\n+\t\t\t \"--split\", \"--reachable\",\n+\t\t\t NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_pushl(&cmd, \"--no-progress\", NULL);\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int run_verify_commit_graph(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"commit-graph\", \"verify\",\n+\t\t\t \"--shallow\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_pushl(&cmd, \"--no-progress\", NULL);\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int maintenance_task_commit_graph(struct repository *r)\n+{\n+\tchar *chain_path;\n+\n+\t/* Skip commit-graph when --auto is specified. */\n+\tif (opts.auto_flag)\n+\t\treturn 0;\n+\n+\tclose_object_store(r->objects);\n+\tif (run_write_commit_graph(r)) {\n+\t\terror(_(\"failed to write commit-graph\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (!run_verify_commit_graph(r))\n+\t\treturn 0;\n+\n+\twarning(_(\"commit-graph verify caught error, rewriting\"));\n+\n+\tchain_path = get_commit_graph_chain_filename(r->objects->odb);\n+\tif (unlink(chain_path)) {\n+\t\tUNLEAK(chain_path);\n+\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n+\t}\n+\tfree(chain_path);\n+\n+\tif (!run_write_commit_graph(r))\n+\t\treturn 0;\n+\n+\terror(_(\"failed to rewrite commit-graph\"));\n+\treturn 1;\n+}\n+\n static int maintenance_task_gc(struct repository *r)\n {\n \tint result;\n@@ -790,6 +860,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->enabled = 1;\n \tnum_tasks++;\n \n+\ttasks[num_tasks]->name = \"commit-graph\";\n+\ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n+\tnum_tasks++;\n+\n \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n \n \tfor (i = 0; i < num_tasks; i++) {\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fdd1c4fa7c..57278a9ab5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb,\n \t\t oid_hex);\n }\n \n-static char *get_chain_filename(struct object_directory *odb)\n+char *get_commit_graph_chain_filename(struct object_directory *odb)\n {\n \treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", odb->path);\n }\n@@ -520,7 +520,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \tstruct stat st;\n \tstruct object_id *oids;\n \tint i = 0, valid = 1, count;\n-\tchar *chain_name = get_chain_filename(odb);\n+\tchar *chain_name = get_commit_graph_chain_filename(odb);\n \tFILE *fp;\n \tint stat_res;\n \n@@ -1635,7 +1635,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t}\n \n \tif (ctx->split) {\n-\t\tchar *lock_name = get_chain_filename(ctx->odb);\n+\t\tchar *lock_name = get_commit_graph_chain_filename(ctx->odb);\n \n \t\thold_lock_file_for_update_mode(&lk, lock_name,\n \t\t\t\t\t LOCK_DIE_ON_ERROR, 0444);\n@@ -2012,7 +2012,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time = ctx->split_opts->expire_time;\n \tif (!ctx->split) {\n-\t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n+\t\tchar *chain_file_name = get_commit_graph_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n \t\tfree(chain_file_name);\n \t\tctx->num_commit_graphs_after = 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 28f89cdf3e..3c202748c3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -25,6 +25,7 @@ struct commit;\n struct bloom_filter_settings;\n \n char *get_commit_graph_filename(struct object_directory *odb);\n+char *get_commit_graph_chain_filename(struct object_directory *odb);\n int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n \n /*\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e4e4036e50..216ac0b19e 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -12,7 +12,7 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n-test_expect_success 'gc [--auto|--quiet]' '\n+test_expect_success 'run [--auto|--quiet]' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n-- \ngitgitgadget\n\n"},{"id":"401108","messageId":"c081a3bd293383da92041f0b8991e151303d9e56.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 10/21] maintenance: add --task option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:24Z","receivedAt":"2020-07-07T14:21:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to only run certain maintenance tasks in a certain\norder. Add the --task= option, which allows a user to specify an\nordered list of tasks to run. These cannot be run multiple times,\nhowever.\n\nHere is where our array of maintenance_task pointers becomes critical.\nWe can sort the array of pointers based on the task order, but we do not\nwant to move the struct data itself in order to preserve the hashmap\nreferences. We use the hashmap to match the --task= arguments into\nthe task struct data.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 4 ++\n builtin/gc.c | 62 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 23 ++++++++++++\n 3 files changed, 87 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 35b0be7d40..9204762e21 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -73,6 +73,10 @@ OPTIONS\n --quiet::\n \tDo not report progress or other information over `stderr`.\n \n+--task=::\n+\tIf this option is specified one or more times, then only run the\n+\tspecified tasks in the specified order.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a6b080627f..8f2143862c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -715,6 +715,7 @@ static const char * const builtin_maintenance_usage[] = {\n struct maintenance_opts {\n \tint auto_flag;\n \tint quiet;\n+\tint tasks_selected;\n } opts;\n \n static int run_write_commit_graph(struct repository *r)\n@@ -812,7 +813,9 @@ struct maintenance_task {\n \tstruct hashmap_entry ent;\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n-\tunsigned enabled:1;\n+\tint task_order;\n+\tunsigned enabled:1,\n+\t\t selected:1;\n };\n \n static int task_entry_cmp(const void *unused_cmp_data,\n@@ -833,14 +836,30 @@ struct maintenance_task *tasks[MAX_NUM_TASKS];\n int num_tasks;\n struct hashmap task_map;\n \n+static int compare_tasks_by_selection(const void *a_, const void *b_)\n+{\n+\tconst struct maintenance_task *a, *b;\n+\ta = (const struct maintenance_task *)a_;\n+\tb = (const struct maintenance_task *)b_;\n+\n+\treturn b->task_order - a->task_order;\n+}\n+\n static int maintenance_run(struct repository *r)\n {\n \tint i;\n \tint result = 0;\n \n+\tif (opts.tasks_selected)\n+\t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n+\n \tfor (i = 0; !result && i < num_tasks; i++) {\n-\t\tif (!tasks[i]->enabled)\n+\t\tif (opts.tasks_selected && !tasks[i]->selected)\n+\t\t\tcontinue;\n+\n+\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n+\n \t\tresult = tasks[i]->fn(r);\n \t}\n \n@@ -873,6 +892,42 @@ static void initialize_tasks(void)\n \t}\n }\n \n+static int task_option_parse(const struct option *opt,\n+\t\t\t const char *arg, int unset)\n+{\n+\tstruct maintenance_task *task;\n+\tstruct maintenance_task key;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\n+\tif (!arg || !strlen(arg)) {\n+\t\terror(_(\"--task requires a value\"));\n+\t\treturn 1;\n+\t}\n+\n+\topts.tasks_selected++;\n+\n+\tkey.name = arg;\n+\thashmap_entry_init(&key.ent, strihash(key.name));\n+\n+\ttask = hashmap_get_entry(&task_map, &key, ent, NULL);\n+\n+\tif (!task) {\n+\t\terror(_(\"'%s' is not a valid task\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (task->selected) {\n+\t\terror(_(\"task '%s' cannot be selected multiple times\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\ttask->selected = 1;\n+\ttask->task_order = opts.tasks_selected;\n+\n+\treturn 0;\n+}\n+\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n \tstruct repository *r = the_repository;\n@@ -882,6 +937,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n+\t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n+\t\t\tN_(\"run a specific task\"),\n+\t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 216ac0b19e..c09a9eb90b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -21,4 +21,27 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'run --task=' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" git maintenance run --task=commit-graph --task=gc &&\n+\t! grep \",\\\"gc\\\"\" run-commit-graph.txt &&\n+\tgrep \",\\\"gc\\\"\" run-gc.txt &&\n+\tgrep \",\\\"gc\\\"\" run-both.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-commit-graph.txt &&\n+\t! grep \",\\\"commit-graph\\\",\\\"write\\\"\" run-gc.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-both.txt\n+'\n+\n+test_expect_success 'run --task=bogus' '\n+\ttest_must_fail git maintenance run --task=bogus 2>err &&\n+\ttest_i18ngrep \"is not a valid task\" err\n+'\n+\n+test_expect_success 'run --task duplicate' '\n+\ttest_must_fail git maintenance run --task=gc --task=gc 2>err &&\n+\ttest_i18ngrep \"cannot be selected multiple times\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401107","messageId":"fc1fb5f3cceef894e44b3c6d4010e1bac1e7e284.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 11/21] maintenance: take a lock on the objects directory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:25Z","receivedAt":"2020-07-07T14:21:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPerforming maintenance on a Git repository involves writing data to the\n.git directory, which is not safe to do with multiple writers attempting\nthe same operation. Ensure that only one 'git maintenance' process is\nrunning at a time by holding a file-based lock. Simply the presence of\nthe .git/maintenance.lock file will prevent future maintenance. This\nlock is never committed, since it does not represent meaningful data.\nInstead, it is only a placeholder.\n\nIf the lock file already exists, then fail silently. This will become\nvery important later when we implement the 'fetch' task, as this is our\nstop-gap from creating a recursive process loop between 'git fetch' and\n'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 8f2143862c..e3c634fc3b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -849,6 +849,24 @@ static int maintenance_run(struct repository *r)\n {\n \tint i;\n \tint result = 0;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/maintenance\", r->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) {\n+\t\t/*\n+\t\t * Another maintenance command is running.\n+\t\t *\n+\t\t * If --auto was provided, then it is likely due to a\n+\t\t * recursive process stack. Do not report an error in\n+\t\t * that case.\n+\t\t */\n+\t\tif (!opts.auto_flag && !opts.quiet)\n+\t\t\terror(_(\"lock file '%s' exists, skipping maintenance\"),\n+\t\t\t lock_path);\n+\t\tfree(lock_path);\n+\t\treturn 0;\n+\t}\n+\tfree(lock_path);\n \n \tif (opts.tasks_selected)\n \t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n@@ -863,6 +881,7 @@ static int maintenance_run(struct repository *r)\n \t\tresult = tasks[i]->fn(r);\n \t}\n \n+\trollback_lock_file(&lk);\n \treturn result;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"401112","messageId":"130130b66293584f9f7aacae7d485b1acb6ed7e9.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 17/21] maintenance: use pointers to check --auto","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:31Z","receivedAt":"2020-07-07T14:21:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' command has an '--auto' option. This is used\nby other Git commands such as 'git commit' or 'git fetch' to check if\nmaintenance should be run after adding data to the repository.\n\nPreviously, this --auto option was only used to add the argument to the\n'git gc' command as part of the 'gc' task. We will be expanding the\nother tasks to perform a check to see if they should do work as part of\nthe --auto flag, when they are enabled by config.\n\nFirst, update the 'gc' task to perform the auto check inside the\nmaintenance process. This prevents running an extra 'git gc --auto'\ncommand when not needed. It also shows a model for other tasks.\n\nSecond, use the 'auto_condition' function pointer as a signal for\nwhether we enable the maintenance task under '--auto'. For instance, we\ndo not want to enable the 'fetch' task in '--auto' mode, so that\nfunction pointer will remain NULL.\n\nNow that we are not automatically calling 'git gc', a test in\nt5514-fetch-multiple.sh must be changed to watch for 'git maintenance'\ninstead.\n\nWe continue to pass the '--auto' option to the 'git gc' command when\nnecessary, because of the gc.autoDetach config option changes behavior.\nLikely, we will want to absorb the daemonizing behavior implied by\ngc.autoDetach as a maintenance.autoDetach config option.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 15 +++++++++++++++\n t/t5514-fetch-multiple.sh | 2 +-\n t/t7900-maintenance.sh | 2 +-\n 3 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 6ffe2213b4..dd24026b41 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1132,10 +1132,18 @@ static int maintenance_task_pack_files(struct repository *r)\n \n typedef int maintenance_task_fn(struct repository *r);\n \n+/*\n+ * An auto condition function returns 1 if the task should run\n+ * and 0 if the task should NOT run. See needs_to_gc() for an\n+ * example.\n+ */\n+typedef int maintenance_auto_fn(struct repository *r);\n+\n struct maintenance_task {\n \tstruct hashmap_entry ent;\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n+\tmaintenance_auto_fn *auto_condition;\n \tint task_order;\n \tunsigned enabled:1,\n \t\t selected:1;\n@@ -1201,6 +1209,11 @@ static int maintenance_run(struct repository *r)\n \t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n \n+\t\tif (opts.auto_flag &&\n+\t\t (!tasks[i]->auto_condition ||\n+\t\t !tasks[i]->auto_condition(r)))\n+\t\t\tcontinue;\n+\n \t\tresult = tasks[i]->fn(r);\n \t}\n \n@@ -1231,6 +1244,7 @@ static void initialize_tasks(struct repository *r)\n \n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n+\ttasks[num_tasks]->auto_condition = need_to_gc;\n \ttasks[num_tasks]->enabled = 1;\n \tnum_tasks++;\n \n@@ -1315,6 +1329,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n+\tgc_config(r);\n \tinitialize_tasks(r);\n \n \targc = parse_options(argc, argv, prefix,\ndiff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh\nindex de8e2f1531..bd202ec6f3 100755\n--- a/t/t5514-fetch-multiple.sh\n+++ b/t/t5514-fetch-multiple.sh\n@@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' '\n \t GIT_TRACE=1 git fetch --multiple one two 2>trace &&\n \t git branch -r > output &&\n \t test_cmp ../expect output &&\n-\t grep \"built-in: git gc\" trace >gc &&\n+\t grep \"built-in: git maintenance\" trace >gc &&\n \t test_line_count = 1 gc\n \t)\n '\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 08aa50a724..315bba2447 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -17,7 +17,7 @@ test_expect_success 'run [--auto|--quiet]' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\t! grep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"401109","messageId":"4744fdaae9771ef0c0a79edc09941e299b6a1682.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 19/21] maintenance: create auto condition for loose-objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:33Z","receivedAt":"2020-07-07T14:21:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe loose-objects task deletes loose objects that already exist in a\npack-file, then place the remaining loose objects into a new pack-file.\nIf this step runs all the time, then we risk creating pack-files with\nvery few objects with every 'git commit' process. To prevent\noverwhelming the packs directory with small pack-files, place a minimum\nnumber of objects to justify the task.\n\nThe 'maintenance.loose-objects.auto' config option specifies a minimum\nnumber of loose objects to justify the task to run under the '--auto'\noption. This defaults to 100 loose objects. Setting the value to zero\nwill prevent the step from running under '--auto' while a negative value\nwill force it to run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 +++++++++\n builtin/gc.c | 30 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 25 +++++++++++++++++++++++\n 3 files changed, 64 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 9bd69b9df3..a9442dd260 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -12,3 +12,12 @@ maintenance.commit-graph.auto::\n \treachable commits that are not in the commit-graph file is at least\n \tthe value of `maintenance.commit-graph.auto`. The default value is\n \t100.\n+\n+maintenance.loose-objects.auto::\n+\tThis integer config option controls how often the `loose-objects` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `loose-objects` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\tloose objects is at least the value of `maintenance.loose-objects.auto`.\n+\tThe default value is 100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 81b076b012..391e1e2121 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -960,6 +960,35 @@ struct write_loose_object_data {\n \tint batch_size;\n };\n \n+static int loose_object_auto_limit = 100;\n+\n+static int loose_object_count(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\tint *count = (int*)data;\n+\tif (++(*count) >= loose_object_auto_limit)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int loose_object_auto_condition(struct repository *r)\n+{\n+\tint count = 0;\n+\n+\trepo_config_get_int(r, \"maintenance.loose-objects.auto\",\n+\t\t\t &loose_object_auto_limit);\n+\n+\tif (!loose_object_auto_limit)\n+\t\treturn 0;\n+\tif (loose_object_auto_limit < 0)\n+\t\treturn 1;\n+\n+\treturn for_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t\t loose_object_count,\n+\t\t\t\t\t NULL, NULL, &count);\n+}\n+\n static int loose_object_exists(const struct object_id *oid,\n \t\t\t const char *path,\n \t\t\t void *data)\n@@ -1311,6 +1340,7 @@ static void initialize_tasks(struct repository *r)\n \n \ttasks[num_tasks]->name = \"loose-objects\";\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n+\ttasks[num_tasks]->auto_condition = loose_object_auto_condition;\n \tnum_tasks++;\n \n \ttasks[num_tasks]->name = \"pack-files\";\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 315bba2447..a55c36d249 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -115,6 +115,31 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'maintenance.loose-objects.auto' '\n+\tgit repack -adk &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-lo1.txt\" \\\n+\t\tgit -c maintenance.loose-objects.auto=1 maintenance \\\n+\t\trun --auto --task=loose-objects &&\n+\t! grep \"\\\"prune-packed\\\"\" trace-lo1.txt &&\n+\tfor i in 1 2\n+\tdo\n+\t\tprintf data-A-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loA-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\t! grep \"\\\"prune-packed\\\"\" trace-loA-$i &&\n+\t\tprintf data-B-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loB-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loB-$i &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loC-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loC-$i || return 1\n+\tdone\n+'\n+\n test_expect_success 'pack-files task' '\n \tpackDir=.git/objects/pack &&\n \tfor i in $(test_seq 1 5)\n-- \ngitgitgadget\n\n"},{"id":"401110","messageId":"fbe03b9af9a9a1db8d893a09ee3a95be05bd9e46.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 20/21] maintenance: add pack-files auto condition","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:34Z","receivedAt":"2020-07-07T14:21:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe pack-files task updates the multi-pack-index by deleting pack-files\nthat have been replaced with new packs, then repacking a batch of small\npack-files into a larger pack-file. This incremental repack is faster\nthan rewriting all object data, but is slower than some other\nmaintenance activities.\n\nThe 'maintenance.pack-files.auto' config option specifies how many\npack-files should exist outside of the multi-pack-index before running\nthe step. These pack-files could be created by 'git fetch' commands or\nby the loose-objects task. The default value is 10.\n\nSetting the option to zero disables the task with the '--auto' option,\nand a negative value makes the task run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 ++++++++\n builtin/gc.c | 31 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 30 +++++++++++++++++++++++++++\n 3 files changed, 70 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a9442dd260..77b255318c 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -21,3 +21,12 @@ maintenance.loose-objects.auto::\n \tpositive value implies the command should run when the number of\n \tloose objects is at least the value of `maintenance.loose-objects.auto`.\n \tThe default value is 100.\n+\n+maintenance.pack-files.auto::\n+\tThis integer config option controls how often the `pack-files` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `pack-files` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\tpack-files not in the multi-pack-index is at least the value of\n+\t`maintenance.pack-files.auto`. The default value is 10.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 391e1e2121..c3531561c2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"refs.h\"\n+#include \"object-store.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1072,6 +1073,35 @@ static int maintenance_task_loose_objects(struct repository *r)\n \treturn prune_packed(r) || pack_loose(r);\n }\n \n+static int pack_files_auto_condition(struct repository *r)\n+{\n+\tstruct packed_git *p;\n+\tint enabled;\n+\tint pack_files_auto_limit = 10;\n+\tint count = 0;\n+\n+\tif (repo_config_get_bool(r, \"core.multiPackIndex\", &enabled) ||\n+\t !enabled)\n+\t\treturn 0;\n+\n+\trepo_config_get_int(r, \"maintenance.pack-files.auto\",\n+\t\t\t &pack_files_auto_limit);\n+\n+\tif (!pack_files_auto_limit)\n+\t\treturn 0;\n+\tif (pack_files_auto_limit < 0)\n+\t\treturn 1;\n+\n+\tfor (p = get_packed_git(r);\n+\t count < pack_files_auto_limit && p;\n+\t p = p->next) {\n+\t\tif (!p->multi_pack_index)\n+\t\t\tcount++;\n+\t}\n+\n+\treturn count >= pack_files_auto_limit;\n+}\n+\n static int multi_pack_index_write(struct repository *r)\n {\n \tint result;\n@@ -1345,6 +1375,7 @@ static void initialize_tasks(struct repository *r)\n \n \ttasks[num_tasks]->name = \"pack-files\";\n \ttasks[num_tasks]->fn = maintenance_task_pack_files;\n+\ttasks[num_tasks]->auto_condition = pack_files_auto_condition;\n \tnum_tasks++;\n \n \ttasks[num_tasks]->name = \"gc\";\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex a55c36d249..1714d11bd9 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -178,4 +178,34 @@ test_expect_success 'pack-files task' '\n \ttest_line_count = 2 packs-after\n '\n \n+test_expect_success 'maintenance.pack-files.auto' '\n+\tgit repack -adk &&\n+\tgit config core.multiPackIndex true &&\n+\tgit multi-pack-index write &&\n+\tGIT_TRACE2_EVENT=1 git -c maintenance.pack-files.auto=1 maintenance \\\n+\t\trun --auto --task=pack-files >out &&\n+\t! grep \"\\\"multi-pack-index\\\"\" out &&\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit A-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-A-$i git \\\n+\t\t\t-c maintenance.pack-files.auto=2 \\\n+\t\t\tmaintenance run --auto --task=pack-files &&\n+\t\t! grep \"\\\"multi-pack-index\\\"\" trace-A-$i &&\n+\t\ttest_commit B-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-B-$i git \\\n+\t\t\t-c maintenance.pack-files.auto=2 \\\n+\t\t\tmaintenance run --auto --task=pack-files >out &&\n+\t\tgrep \"\\\"multi-pack-index\\\"\" trace-B-$i >/dev/null || return 1\n+\tdone\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401111","messageId":"f7fdf72e9dc6a470900408fe6c5d2d020e992807.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 21/21] midx: use start_delayed_progress()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:35Z","receivedAt":"2020-07-07T14:22:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that the multi-pack-index may be written as part of auto maintenance\nat the end of a command, reduce the progress output when the operations\nare quick. Use start_delayed_progress() instead of start_progress().\n\nUpdate t5319-multi-pack-index.sh to use GIT_PROGRESS_DELAY=0 now that\nthe progress indicators are conditional.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 10 +++++-----\n t/t5319-multi-pack-index.sh | 14 +++++++-------\n 2 files changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 57a8a00082..d4022e4aef 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -837,7 +837,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tpacks.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tpacks.progress = NULL;\n \n@@ -974,7 +974,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Writing chunks to multi-pack-index\"),\n+\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (written != chunk_offsets[i])\n@@ -1109,7 +1109,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t\treturn 0;\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Looking for referenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n@@ -1230,7 +1230,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Counting referenced objects\"),\n+\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n \t\t\t\t\t m->num_objects);\n \tfor (i = 0; i < m->num_objects; i++) {\n \t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n@@ -1240,7 +1240,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tstop_progress(&progress);\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Finding and deleting unreferenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tchar *pack_name;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 7214cab36c..12f41dfc18 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -172,12 +172,12 @@ test_expect_success 'write progress off for redirected stderr' '\n '\n \n test_expect_success 'write force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress write 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'write with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -334,17 +334,17 @@ test_expect_success 'git-fsck incorrect offset' '\n '\n \n test_expect_success 'repack progress off for redirected stderr' '\n-\tgit multi-pack-index --object-dir=$objdir repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n test_expect_success 'repack force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'repack with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -488,7 +488,7 @@ test_expect_success 'expire progress off for redirected stderr' '\n test_expect_success 'expire force progress on for stderr' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --progress expire 2>err &&\n \t\ttest_file_not_empty err\n \t)\n '\n@@ -496,7 +496,7 @@ test_expect_success 'expire force progress on for stderr' '\n test_expect_success 'expire with the --no-progress option' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --no-progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --no-progress expire 2>err &&\n \t\ttest_line_count = 0 err\n \t)\n '\n-- \ngitgitgadget\n"},{"id":"401113","messageId":"1551aec4fd55b26601aa8673bee946c7ce7438a6.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 16/21] maintenance: create maintenance..enabled config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:30Z","receivedAt":"2020-07-07T14:22:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCurrently, a normal run of \"git maintenance run\" will only run the 'gc'\ntask, as it is the only one enabled. This is mostly for backwards-\ncompatible reasons since \"git maintenance run --auto\" commands replaced\nprevious \"git gc --auto\" commands after some Git processes. Users could\nmanually run specific maintenance tasks by calling \"git maintenance run\n--task=\" directly.\n\nAllow users to customize which steps are run automatically using config.\nThe 'maintenance..enabled' option then can turn on these other\ntasks (or turn off the 'gc' task).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 ++\n Documentation/config/maintenance.txt | 4 ++++\n Documentation/git-maintenance.txt | 6 +++++-\n builtin/gc.c | 15 +++++++++++++--\n t/t7900-maintenance.sh | 12 ++++++++++++\n 5 files changed, 36 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/maintenance.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..2783b825f9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -396,6 +396,8 @@ include::config/mailinfo.txt[]\n \n include::config/mailmap.txt[]\n \n+include::config/maintenance.txt[]\n+\n include::config/man.txt[]\n \n include::config/merge.txt[]\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nnew file mode 100644\nindex 0000000000..370cbfb42f\n--- /dev/null\n+++ b/Documentation/config/maintenance.txt\n@@ -0,0 +1,4 @@\n+maintenance..enabled::\n+\tThis boolean config option controls whether the maintenance task\n+\twith name `` is run when no `--task` option is specified.\n+\tBy default, only `maintenance.gc.enabled` is true.\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 945fda368b..261d2e0ee1 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -30,7 +30,11 @@ SUBCOMMANDS\n -----------\n \n run::\n-\tRun one or more maintenance tasks.\n+\tRun one or more maintenance tasks. If one or more `--task` options\n+\tare specified, then those tasks are run in that order. Otherwise,\n+\tthe tasks are determined by which `maintenance..enabled`\n+\tconfig options are true. By default, only `maintenance.gc.enabled`\n+\tis true.\n \n TASKS\n -----\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 582219156a..6ffe2213b4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1208,9 +1208,10 @@ static int maintenance_run(struct repository *r)\n \treturn result;\n }\n \n-static void initialize_tasks(void)\n+static void initialize_tasks(struct repository *r)\n {\n \tint i;\n+\tstruct strbuf config_name = STRBUF_INIT;\n \tnum_tasks = 0;\n \n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n@@ -1240,10 +1241,20 @@ static void initialize_tasks(void)\n \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n \n \tfor (i = 0; i < num_tasks; i++) {\n+\t\tint config_value;\n+\n \t\thashmap_entry_init(&tasks[i]->ent,\n \t\t\t\t strihash(tasks[i]->name));\n \t\thashmap_add(&task_map, &tasks[i]->ent);\n+\n+\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\", tasks[i]->name);\n+\n+\t\tif (!repo_config_get_bool(r, config_name.buf, &config_value))\n+\t\t\ttasks[i]->enabled = config_value;\n \t}\n+\n+\tstrbuf_release(&config_name);\n }\n \n static int task_option_parse(const struct option *opt,\n@@ -1304,7 +1315,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n-\tinitialize_tasks();\n+\tinitialize_tasks(r);\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 43d32c131b..08aa50a724 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -21,6 +21,18 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'maintenance..enabled' '\n+\tgit config maintenance.gc.enabled false &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-config.txt\" git maintenance run &&\n+\t! grep \",\\\"fetch\\\"\" run-config.txt &&\n+\t! grep \",\\\"gc\\\"\" run-config.txt &&\n+\t! grep \",\\\"multi-pack-index\\\"\" run-config.txt &&\n+\tgrep \",\\\"commit-graph\\\"\" run-config.txt &&\n+\tgrep \",\\\"prune-packed\\\"\" run-config.txt\n+'\n+\n test_expect_success 'run --task=' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n-- \ngitgitgadget\n\n"},{"id":"401114","messageId":"37fa3f6157ab7799f29331e8cf11f55e7f123e49.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 18/21] maintenance: add auto condition for commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:32Z","receivedAt":"2020-07-07T14:22:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of writing a new commit-graph in every 'git maintenance run\n--auto' process (when maintenance.commit-graph.enalbed is configured to\nbe true), only write when there are \"enough\" commits not in a\ncommit-graph file.\n\nThis count is controlled by the maintenance.commit-graph.auto config\noption.\n\nTo compute the count, use a depth-first search starting at each ref, and\nleaving markers using the PARENT1 flag. If this count reaches the limit,\nthen terminate early and start the task. Otherwise, this operation will\npeel every ref and parse the commit it points to. If these are all in\nthe commit-graph, then this is typically a very fast operation. Users\nwith many refs might feel a slow-down, and hence could consider updating\ntheir limit to be very small. A negative value will force the step to\nrun every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 10 ++++\n builtin/gc.c | 76 ++++++++++++++++++++++++++++\n object.h | 1 +\n 3 files changed, 87 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 370cbfb42f..9bd69b9df3 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -2,3 +2,13 @@ maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified.\n \tBy default, only `maintenance.gc.enabled` is true.\n+\n+maintenance.commit-graph.auto::\n+\tThis integer config option controls how often the `commit-graph` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `commit-graph` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\treachable commits that are not in the commit-graph file is at least\n+\tthe value of `maintenance.commit-graph.auto`. The default value is\n+\t100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex dd24026b41..81b076b012 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -30,6 +30,7 @@\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n #include \"midx.h\"\n+#include \"refs.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -720,6 +721,80 @@ struct maintenance_opts {\n \tint tasks_selected;\n } opts;\n \n+/* Remember to update object flag allocation in object.h */\n+#define PARENT1\t\t(1u<<16)\n+\n+static int num_commits_not_in_graph = 0;\n+static int limit_commits_not_in_graph = 100;\n+\n+static int dfs_on_ref(const char *refname,\n+\t\t const struct object_id *oid, int flags,\n+\t\t void *cb_data)\n+{\n+\tint result = 0;\n+\tstruct object_id peeled;\n+\tstruct commit_list *stack = NULL;\n+\tstruct commit *commit;\n+\n+\tif (!peel_ref(refname, &peeled))\n+\t\toid = &peeled;\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tcommit = lookup_commit(the_repository, oid);\n+\tif (!commit)\n+\t\treturn 0;\n+\tif (parse_commit(commit))\n+\t\treturn 0;\n+\n+\tcommit_list_append(commit, &stack);\n+\n+\twhile (!result && stack) {\n+\t\tstruct commit_list *parent;\n+\n+\t\tcommit = pop_commit(&stack);\n+\n+\t\tfor (parent = commit->parents; parent; parent = parent->next) {\n+\t\t\tif (parse_commit(parent->item) ||\n+\t\t\t commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH ||\n+\t\t\t parent->item->object.flags & PARENT1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tparent->item->object.flags |= PARENT1;\n+\t\t\tnum_commits_not_in_graph++;\n+\n+\t\t\tif (num_commits_not_in_graph >= limit_commits_not_in_graph) {\n+\t\t\t\tresult = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tcommit_list_append(parent->item, &stack);\n+\t\t}\n+\t}\n+\n+\tfree_commit_list(stack);\n+\treturn result;\n+}\n+\n+static int should_write_commit_graph(struct repository *r)\n+{\n+\tint result;\n+\n+\trepo_config_get_int(r, \"maintenance.commit-graph.auto\",\n+\t\t\t &limit_commits_not_in_graph);\n+\n+\tif (!limit_commits_not_in_graph)\n+\t\treturn 0;\n+\tif (limit_commits_not_in_graph < 0)\n+\t\treturn 1;\n+\n+\tresult = for_each_ref(dfs_on_ref, NULL);\n+\n+\tclear_commit_marks_all(PARENT1);\n+\n+\treturn result;\n+}\n+\n static int run_write_commit_graph(struct repository *r)\n {\n \tint result;\n@@ -1250,6 +1325,7 @@ static void initialize_tasks(struct repository *r)\n \n \ttasks[num_tasks]->name = \"commit-graph\";\n \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n+\ttasks[num_tasks]->auto_condition = should_write_commit_graph;\n \tnum_tasks++;\n \n \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\ndiff --git a/object.h b/object.h\nindex 38dc2d5a6c..4f886495d7 100644\n--- a/object.h\n+++ b/object.h\n@@ -73,6 +73,7 @@ struct object_array {\n * list-objects-filter.c: 21\n * builtin/fsck.c: 0--3\n * builtin/index-pack.c: 2021\n+ * builtin/maintenance.c: 16\n * builtin/pack-objects.c: 20\n * builtin/reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n-- \ngitgitgadget\n\n"},{"id":"401116","messageId":"66a1f662cee7a1ae18df6e93c70b481157eb2494.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 13/21] maintenance: add loose-objects task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:27Z","receivedAt":"2020-07-07T14:22:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOne goal of background maintenance jobs is to allow a user to\ndisable auto-gc (gc.auto=0) but keep their repository in a clean\nstate. Without any cleanup, loose objects will clutter the object\ndatabase and slow operations. In addition, the loose objects will\ntake up extra space because they are not stored with deltas against\nsimilar objects.\n\nCreate a 'loose-objects' task for the 'git maintenance run' command.\nThis helps clean up loose objects without disrupting concurrent Git\ncommands using the following sequence of events:\n\n1. Run 'git prune-packed' to delete any loose objects that exist\n in a pack-file. Concurrent commands will prefer the packed\n version of the object to the loose version. (Of course, there\n are exceptions for commands that specifically care about the\n location of an object. These are rare for a user to run on\n purpose, and we hope a user that has selected background\n maintenance will not be trying to do foreground maintenance.)\n\n2. Run 'git pack-objects' on a batch of loose objects. These\n objects are grouped by scanning the loose object directories in\n lexicographic order until listing all loose objects -or-\n reaching 50,000 objects. This is more than enough if the loose\n objects are created only by a user doing normal development.\n We noticed users with _millions_ of loose objects because VFS\n for Git downloads blobs on-demand when a file read operation\n requires populating a virtual file. This has potential of\n happening in partial clones if someone runs 'git grep' or\n otherwise evades the batch-download feature for requesting\n promisor objects.\n\nThis step is based on a similar step in Scalar [1] and VFS for Git.\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/LooseObjectsStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n builtin/gc.c | 107 +++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 35 ++++++++++\n 3 files changed, 152 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex e0be3f520f..bf792d446f 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -73,6 +73,17 @@ gc::\n \tIt can also be disruptive in some situations, as it deletes stale\n \tdata.\n \n+loose-objects::\n+\tThe `loose-objects` job cleans up loose objects and places them into\n+\tpack-files. In order to prevent race conditions with concurrent Git\n+\tcommands, it follows a two-step process. First, it deletes any loose\n+\tobjects that already exist in a pack-file; concurrent Git processes\n+\twill examine the pack-file for the object data instead of the loose\n+\tobject. Second, it creates a new pack-file (starting with \"loose-\")\n+\tcontaining a batch of loose objects. The batch size is limited to 50\n+\tthousand objects to prevent the job from taking too long on a\n+\trepository with many loose objects.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 2d30ae758c..dda71fe39c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -706,7 +706,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 3\n+#define MAX_NUM_TASKS 4\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -866,6 +866,107 @@ static int maintenance_task_gc(struct repository *r)\n \treturn result;\n }\n \n+\n+static int prune_packed(struct repository *r)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", r->worktree, \"prune-packed\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--quiet\");\n+\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+struct write_loose_object_data {\n+\tFILE *in;\n+\tint count;\n+\tint batch_size;\n+};\n+\n+static int loose_object_exists(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\treturn 1;\n+}\n+\n+static int write_loose_object_to_stdin(const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_loose_object_data *d = (struct write_loose_object_data *)data;\n+\n+\tfprintf(d->in, \"%s\\n\", oid_to_hex(oid));\n+\n+\treturn ++(d->count) > d->batch_size;\n+}\n+\n+static int pack_loose(struct repository *r)\n+{\n+\tint result = 0;\n+\tstruct write_loose_object_data data;\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tstruct child_process *pack_proc;\n+\n+\t/*\n+\t * Do not start pack-objects process\n+\t * if there are no loose objects.\n+\t */\n+\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t\t loose_object_exists,\n+\t\t\t\t\t NULL, NULL, NULL))\n+\t\treturn 0;\n+\n+\tpack_proc = xmalloc(sizeof(*pack_proc));\n+\n+\tchild_process_init(pack_proc);\n+\n+\tstrbuf_addstr(&prefix, r->objects->odb->path);\n+\tstrbuf_addstr(&prefix, \"/pack/loose\");\n+\n+\targv_array_pushl(&pack_proc->args, \"git\", \"-C\", r->worktree,\n+\t\t\t \"pack-objects\", NULL);\n+\tif (opts.quiet)\n+\t\targv_array_push(&pack_proc->args, \"--quiet\");\n+\targv_array_push(&pack_proc->args, prefix.buf);\n+\n+\tpack_proc->in = -1;\n+\n+\tif (start_command(pack_proc)) {\n+\t\terror(_(\"failed to start 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tdata.in = xfdopen(pack_proc->in, \"w\");\n+\tdata.count = 0;\n+\tdata.batch_size = 50000;\n+\n+\tfor_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t write_loose_object_to_stdin,\n+\t\t\t\t NULL,\n+\t\t\t\t NULL,\n+\t\t\t\t &data);\n+\n+\tfclose(data.in);\n+\n+\tif (finish_command(pack_proc)) {\n+\t\terror(_(\"failed to finish 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&prefix);\n+\tfree(pack_proc);\n+\treturn result;\n+}\n+\n+static int maintenance_task_loose_objects(struct repository *r)\n+{\n+\treturn prune_packed(r) || pack_loose(r);\n+}\n+\n typedef int maintenance_task_fn(struct repository *r);\n \n struct maintenance_task {\n@@ -956,6 +1057,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_fetch;\n \tnum_tasks++;\n \n+\ttasks[num_tasks]->name = \"loose-objects\";\n+\ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 0abfc4a9da..8cb33624c6 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -68,4 +68,39 @@ test_expect_success 'fetch multiple remotes' '\n \tgit log hidden/remote2/two\n '\n \n+test_expect_success 'loose-objects task' '\n+\t# Repack everything so we know the state of the object dir\n+\tgit repack -adk &&\n+\n+\t# Hack to stop maintenance from running during \"git commit\"\n+\techo in use >.git/objects/maintenance.lock &&\n+\ttest_commit create-loose-object &&\n+\trm .git/objects/maintenance.lock &&\n+\n+\tls .git/objects >obj-dir-before &&\n+\ttest_file_not_empty obj-dir-before &&\n+\tls .git/objects/pack/*.pack >packs-before &&\n+\ttest_line_count = 1 packs-before &&\n+\n+\t# The first run creates a pack-file\n+\t# but does not delete loose objects.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-between &&\n+\ttest_cmp obj-dir-before obj-dir-between &&\n+\tls .git/objects/pack/*.pack >packs-between &&\n+\ttest_line_count = 2 packs-between &&\n+\n+\t# The second run deletes loose objects\n+\t# but does not create a pack-file.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-after &&\n+\tcat >expect <<-\\EOF &&\n+\tinfo\n+\tpack\n+\tEOF\n+\ttest_cmp expect obj-dir-after &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_cmp packs-between packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401117","messageId":"8be89707d2d5a8b527e104a2aeadb5ea7f65d476.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 15/21] maintenance: auto-size pack-files batch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:29Z","receivedAt":"2020-07-07T14:22:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen repacking during the 'pack-files' job, we use the --batch-size\noption in 'git multi-pack-index repack'. The initial setting used\n--batch-size=0 to repack everything into a single pack-file. This is not\nsustaintable for a large repository. The amount of work required is also\nlikely to use too many system resources for a background job.\n\nUpdate the 'pack-files' maintenance task by dynamically computing a\n--batch-size option based on the current pack-file structure.\n\nThe dynamic default size is computed with this idea in mind for a client\nrepository that was cloned from a very large remote: there is likely one\n\"big\" pack-file that was created at clone time. Thus, do not try\nrepacking it as it is likely packed efficiently by the server.\n\nInstead, we select the second-largest pack-file, and create a batch size\nthat is one larger than that pack-file. If there are three or more\npack-files, then this guarantees that at least two will be combined into\na new pack-file.\n\nOf course, this means that the second-largest pack-file size is likely\nto grow over time and may eventually surpass the initially-cloned\npack-file. Recall that the pack-file batch is selected in a greedy\nmanner: the packs are considered from oldest to newest and are selected\nif they have size smaller than the batch size until the total selected\nsize is larger than the batch size. Thus, that oldest \"clone\" pack will\nbe first to repack after the new data creates a pack larger than that.\n\nWe also want to place some limits on how large these pack-files become,\nin order to bound the amount of time spent repacking. A maximum\nbatch-size of two gigabytes means that large repositories will never be\npacked into a single pack-file using this job, but also that repack is\nrather expensive. This is a trade-off that is valuable to have if the\nmaintenance is being run automatically or in the background. Users who\ntruly want to optimize for space and performance (and are willing to pay\nthe upfront cost of a full repack) can use the 'gc' task to do so.\n\nReported-by: Son Luong Ngoc \nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 47 +++++++++++++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 5 +++--\n 2 files changed, 49 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 259b0475c0..582219156a 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1032,20 +1032,65 @@ static int multi_pack_index_expire(struct repository *r)\n \treturn result;\n }\n \n+#define TWO_GIGABYTES (2147483647)\n+#define UNSET_BATCH_SIZE ((unsigned long)-1)\n+\n+static off_t get_auto_pack_size(struct repository *r)\n+{\n+\t/*\n+\t * The \"auto\" value is special: we optimize for\n+\t * one large pack-file (i.e. from a clone) and\n+\t * expect the rest to be small and they can be\n+\t * repacked quickly.\n+\t *\n+\t * The strategy we select here is to select a\n+\t * size that is one more than the second largest\n+\t * pack-file. This ensures that we will repack\n+\t * at least two packs if there are three or more\n+\t * packs.\n+\t */\n+\toff_t max_size = 0;\n+\toff_t second_largest_size = 0;\n+\toff_t result_size;\n+\tstruct packed_git *p;\n+\n+\treprepare_packed_git(r);\n+\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tif (p->pack_size > max_size) {\n+\t\t\tsecond_largest_size = max_size;\n+\t\t\tmax_size = p->pack_size;\n+\t\t} else if (p->pack_size > second_largest_size)\n+\t\t\tsecond_largest_size = p->pack_size;\n+\t}\n+\n+\tresult_size = second_largest_size + 1;\n+\n+\t/* But limit ourselves to a batch size of 2g */\n+\tif (result_size > TWO_GIGABYTES)\n+\t\tresult_size = TWO_GIGABYTES;\n+\n+\treturn result_size;\n+}\n+\n static int multi_pack_index_repack(struct repository *r)\n {\n \tint result;\n \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf batch_arg = STRBUF_INIT;\n+\n \targv_array_pushl(&cmd, \"-C\", r->worktree,\n \t\t\t \"multi-pack-index\", \"repack\", NULL);\n \n \tif (opts.quiet)\n \t\targv_array_push(&cmd, \"--no-progress\");\n \n-\targv_array_push(&cmd, \"--batch-size=0\");\n+\tstrbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n+\t\t\t (uintmax_t)get_auto_pack_size(r));\n+\targv_array_push(&cmd, batch_arg.buf);\n \n \tclose_object_store(r->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\tstrbuf_release(&batch_arg);\n \n \tif (result && multi_pack_index_verify(r)) {\n \t\twarning(_(\"multi-pack-index verify failed after repack\"));\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex a6be8456aa..43d32c131b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -134,10 +134,11 @@ test_expect_success 'pack-files task' '\n \ttest_line_count = 4 packs-between &&\n \n \t# the job deletes the two old packs, and does not write\n-\t# a new one because only one pack remains.\n+\t# a new one because the batch size is not high enough to\n+\t# pack the largest pack-file.\n \tgit maintenance run --task=pack-files &&\n \tls .git/objects/pack/*.pack >packs-after &&\n-\ttest_line_count = 1 packs-after\n+\ttest_line_count = 2 packs-after\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"401115","messageId":"f98790024f10b70cbd2d88cb09126c5a83c9c400.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 14/21] maintenance: add pack-files task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:28Z","receivedAt":"2020-07-07T14:22:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change cleaned up loose objects using the\n'loose-objects' that can be run safely in the background. Add a\nsimilar job that performs similar cleanups for pack-files.\n\nOne issue with running 'git repack' is that it is designed to\nrepack all pack-files into a single pack-file. While this is the\nmost space-efficient way to store object data, it is not time or\nmemory efficient. This becomes extremely important if the repo is\nso large that a user struggles to store two copies of the pack on\ntheir disk.\n\nInstead, perform an \"incremental\" repack by collecting a few small\npack-files into a new pack-file. The multi-pack-index facilitates\nthis process ever since 'git multi-pack-index expire' was added in\n19575c7 (multi-pack-index: implement 'expire' subcommand,\n2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1\n(midx: implement midx_repack(), 2019-06-10).\n\nThe 'pack-files' job runs the following steps:\n\n1. 'git multi-pack-index write' creates a multi-pack-index file if\n one did not exist, and otherwise will update the multi-pack-index\n with any new pack-files that appeared since the last write. This\n is particularly relevant with the background fetch job.\n\n When the multi-pack-index sees two copies of the same object, it\n stores the offset data into the newer pack-file. This means that\n some old pack-files could become \"unreferenced\" which I will use\n to mean \"a pack-file that is in the pack-file list of the\n multi-pack-index but none of the objects in the multi-pack-index\n reference a location inside that pack-file.\"\n\n2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n and updaes the multi-pack-index to drop those pack-files from the\n list. This is safe to do as concurrent Git processes will see the\n multi-pack-index and not open those packs when looking for object\n contents. (Similar to the 'loose-objects' job, there are some Git\n commands that open pack-files regardless of the multi-pack-index,\n but they are rarely used. Further, a user that self-selects to\n use background operations would likely refrain from using those\n commands.)\n\n3. 'git multi-pack-index repack --bacth-size=' collects a set\n of pack-files that are listed in the multi-pack-index and creates\n a new pack-file containing the objects whose offsets are listed\n by the multi-pack-index to be in those objects. The set of pack-\n files is selected greedily by sorting the pack-files by modified\n time and adding a pack-file to the set if its \"expected size\" is\n smaller than the batch size until the total expected size of the\n selected pack-files is at least the batch size. The \"expected\n size\" is calculated by taking the size of the pack-file divided\n by the number of objects in the pack-file and multiplied by the\n number of objects from the multi-pack-index with offset in that\n pack-file. The expected size approximats how much data from that\n pack-file will contribute to the resulting pack-file size. The\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n\n The next run of the pack-files job will delete these repacked\n pack-files during the 'expire' step.\n\n In this version, the batch size is set to \"0\" which ignores the\n size restrictions when selecting the pack-files. It instead\n selects all pack-files and repacks all packed objects into a\n single pack-file. This will be updated in the next change, but\n it requires doing some calculations that are better isolated to\n a separate change.\n\nEach of the above steps update the multi-pack-index file. After\neach step, we verify the new multi-pack-index. If the new\nmulti-pack-index is corrupt, then delete the multi-pack-index,\nrewrite it from scratch, and stop doing the later steps of the\njob. This is intended to be an extra-safe check without leaving\na repo with many pack-files without a multi-pack-index.\n\nThese steps are based on a similar background maintenance step in\nScalar (and VFS for Git) [1]. This was incredibly effective for\nusers of the Windows OS repository. After using the same VFS for Git\nrepository for over a year, some users had _thousands_ of pack-files\nthat combined to up to 250 GB of data. We noticed a few users were\nrunning into the open file descriptor limits (due in part to a bug\nin the multi-pack-index fixed by af96fe3 (midx: add packs to\npacked_git linked list, 2019-04-29).\n\nThese pack-files were mostly small since they contained the commits\nand trees that were pushed to the origin in a given hour. The GVFS\nprotocol includes a \"prefetch\" step that asks for pre-computed pack-\nfiles containing commits and trees by timestamp. These pack-files\nwere grouped into \"daily\" pack-files once a day for up to 30 days.\nIf a user did not request prefetch packs for over 30 days, then they\nwould get the entire history of commits and trees in a new, large\npack-file. This led to a large number of pack-files that had poor\ndelta compression.\n\nBy running this pack-file maintenance step once per day, these repos\nwith thousands of packs spanning 200+ GB dropped to dozens of pack-\nfiles spanning 30-50 GB. This was done all without removing objects\nfrom the system and using a constant batch size of two gigabytes.\nOnce the work was done to reduce the pack-files to small sizes, the\nbatch size of two gigabytes means that not every run triggers a\nrepack operation, so the following run will not expire a pack-file.\nThis has kept these repos in a \"clean\" state.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 ++++\n builtin/gc.c | 124 +++++++++++++++++++++++++++++-\n midx.c | 2 +-\n midx.h | 1 +\n t/t7900-maintenance.sh | 37 +++++++++\n 5 files changed, 177 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex bf792d446f..945fda368b 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -84,6 +84,21 @@ loose-objects::\n \tthousand objects to prevent the job from taking too long on a\n \trepository with many loose objects.\n \n+pack-files::\n+\tThe `pack-files` job incrementally repacks the object directory\n+\tusing the `multi-pack-index` feature. In order to prevent race\n+\tconditions with concurrent Git commands, it follows a two-step\n+\tprocess. First, it deletes any pack-files included in the\n+\t`multi-pack-index` where none of the objects in the\n+\t`multi-pack-index` reference those pack-files; this only happens\n+\tif all objects in the pack-file are also stored in a newer\n+\tpack-file. Second, it selects a group of pack-files whose \"expected\n+\tsize\" is below the batch size until the group has total expected\n+\tsize at least the batch size; see the `--batch-size` option for\n+\tthe `repack` subcommand in linkgit:git-multi-pack-index[1]. The\n+\tdefault batch-size is zero, which is a special case that attempts\n+\tto repack all pack-files into a single pack-file.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex dda71fe39c..259b0475c0 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -29,6 +29,7 @@\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n+#include \"midx.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -706,7 +707,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 4\n+#define MAX_NUM_TASKS 5\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -967,6 +968,123 @@ static int maintenance_task_loose_objects(struct repository *r)\n \treturn prune_packed(r) || pack_loose(r);\n }\n \n+static int multi_pack_index_write(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"multi-pack-index\", \"write\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int rewrite_multi_pack_index(struct repository *r)\n+{\n+\tchar *midx_name = get_midx_filename(r->objects->odb->path);\n+\n+\tunlink(midx_name);\n+\tfree(midx_name);\n+\n+\tif (multi_pack_index_write(r)) {\n+\t\terror(_(\"failed to rewrite multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int multi_pack_index_verify(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"multi-pack-index\", \"verify\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int multi_pack_index_expire(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"multi-pack-index\", \"expire\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tclose_object_store(r->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int multi_pack_index_repack(struct repository *r)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"multi-pack-index\", \"repack\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\targv_array_push(&cmd, \"--batch-size=0\");\n+\n+\tclose_object_store(r->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tif (result && multi_pack_index_verify(r)) {\n+\t\twarning(_(\"multi-pack-index verify failed after repack\"));\n+\t\tresult = rewrite_multi_pack_index(r);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int maintenance_task_pack_files(struct repository *r)\n+{\n+\tif (multi_pack_index_write(r)) {\n+\t\terror(_(\"failed to write multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify(r)) {\n+\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n+\t\treturn rewrite_multi_pack_index(r);\n+\t}\n+\n+\tif (multi_pack_index_expire(r)) {\n+\t\terror(_(\"multi-pack-index expire failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify(r)) {\n+\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n+\t\treturn rewrite_multi_pack_index(r);\n+\t}\n+\n+\tif (multi_pack_index_repack(r)) {\n+\t\terror(_(\"multi-pack-index repack failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n typedef int maintenance_task_fn(struct repository *r);\n \n struct maintenance_task {\n@@ -1061,6 +1179,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n \tnum_tasks++;\n \n+\ttasks[num_tasks]->name = \"pack-files\";\n+\ttasks[num_tasks]->fn = maintenance_task_pack_files;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/midx.c b/midx.c\nindex 6d1584ca51..57a8a00082 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -36,7 +36,7 @@\n \n #define PACK_EXPIRED UINT_MAX\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..baeecc70c9 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -37,6 +37,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_filename(const char *object_dir);\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8cb33624c6..a6be8456aa 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -103,4 +103,41 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'pack-files task' '\n+\tpackDir=.git/objects/pack &&\n+\tfor i in $(test_seq 1 5)\n+\tdo\n+\t\ttest_commit $i || return 1\n+\tdone &&\n+\n+\t# Create three disjoint pack-files with size BIG, small, small.\n+\techo HEAD~2 | git pack-objects --revs $packDir/test-1 &&\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-2 <<-\\EOF &&\n+\tHEAD~1\n+\t^HEAD~2\n+\tEOF\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-3 <<-\\EOF &&\n+\tHEAD\n+\t^HEAD~1\n+\tEOF\n+\trm -f $packDir/pack-* &&\n+\trm -f $packDir/loose-* &&\n+\tls $packDir/*.pack >packs-before &&\n+\ttest_line_count = 3 packs-before &&\n+\n+\t# the job repacks the two into a new pack, but does not\n+\t# delete the old ones.\n+\tgit maintenance run --task=pack-files &&\n+\tls $packDir/*.pack >packs-between &&\n+\ttest_line_count = 4 packs-between &&\n+\n+\t# the job deletes the two old packs, and does not write\n+\t# a new one because only one pack remains.\n+\tgit maintenance run --task=pack-files &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_line_count = 1 packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401118","messageId":"e13d351e9ff19695b2ff638af01fdb4527fb7a6d.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 04/21] gc: drop the_repository in log location","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:18Z","receivedAt":"2020-07-07T14:22:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe report_last_gc_error() method use git_pathdup() which implicitly\nuses the_repository. Replace this with strbuf_repo_path() to get a\npath buffer we control that uses a given repository pointer.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 20 +++++++++++---------\n 1 file changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 888b6444d6..58a00be5af 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -462,28 +462,30 @@ static const char *lock_repo_for_gc(int force, pid_t* ret_pid)\n * gc should not proceed due to an error in the last run. Prints a\n * message and returns -1 if an error occurred while reading gc.log\n */\n-static int report_last_gc_error(void)\n+static int report_last_gc_error(struct repository *r)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint ret = 0;\n \tssize_t len;\n \tstruct stat st;\n-\tchar *gc_log_path = git_pathdup(\"gc.log\");\n+\tstruct strbuf gc_log_path = STRBUF_INIT;\n \n-\tif (stat(gc_log_path, &st)) {\n+\tstrbuf_repo_git_path(&gc_log_path, r, \"gc.log\");\n+\n+\tif (stat(gc_log_path.buf, &st)) {\n \t\tif (errno == ENOENT)\n \t\t\tgoto done;\n \n-\t\tret = error_errno(_(\"cannot stat '%s'\"), gc_log_path);\n+\t\tret = error_errno(_(\"cannot stat '%s'\"), gc_log_path.buf);\n \t\tgoto done;\n \t}\n \n \tif (st.st_mtime < gc_log_expire_time)\n \t\tgoto done;\n \n-\tlen = strbuf_read_file(&sb, gc_log_path, 0);\n+\tlen = strbuf_read_file(&sb, gc_log_path.buf, 0);\n \tif (len < 0)\n-\t\tret = error_errno(_(\"cannot read '%s'\"), gc_log_path);\n+\t\tret = error_errno(_(\"cannot read '%s'\"), gc_log_path.buf);\n \telse if (len > 0) {\n \t\t/*\n \t\t * A previous gc failed. Report the error, and don't\n@@ -496,12 +498,12 @@ static int report_last_gc_error(void)\n \t\t\t \"Automatic cleanup will not be performed \"\n \t\t\t \"until the file is removed.\\n\\n\"\n \t\t\t \"%s\"),\n-\t\t\t gc_log_path, sb.buf);\n+\t\t\t gc_log_path.buf, sb.buf);\n \t\tret = 1;\n \t}\n \tstrbuf_release(&sb);\n done:\n-\tfree(gc_log_path);\n+\tstrbuf_release(&gc_log_path);\n \treturn ret;\n }\n \n@@ -602,7 +604,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n \t\tif (detach_auto) {\n-\t\t\tint ret = report_last_gc_error();\n+\t\t\tint ret = report_last_gc_error(r);\n \t\t\tif (ret < 0)\n \t\t\t\t/* an I/O error occurred, already reported */\n \t\t\t\texit(128);\n-- \ngitgitgadget\n\n"},{"id":"401119","messageId":"cbaa5ecc4f23eed0823fbbb53ffef28c9f7d6913.1594131695.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH 12/21] maintenance: add fetch task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-07T14:21:26Z","receivedAt":"2020-07-07T14:22:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen working with very large repositories, an incremental 'git fetch'\ncommand can download a large amount of data. If there are many other\nusers pushing to a common repo, then this data can rival the initial\npack-file size of a 'git clone' of a medium-size repo.\n\nUsers may want to keep the data on their local repos as close as\npossible to the data on the remote repos by fetching periodically in\nthe background. This can break up a large daily fetch into several\nsmaller hourly fetches.\n\nHowever, if we simply ran 'git fetch ' in the background,\nthen the user running a foregroudn 'git fetch ' would lose\nsome important feedback when a new branch appears or an existing\nbranch updates. This is especially true if a remote branch is\nforce-updated and this isn't noticed by the user because it occurred\nin the background. Further, the functionality of 'git push\n--force-with-lease' becomes suspect.\n\nWhen running 'git fetch ' in the background, use\nthe following options for careful updating:\n\n1. --no-tags prevents getting a new tag when a user wants to see\n the new tags appear in their foreground fetches.\n\n2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n\n3. By adding a new refspec \"+refs/heads/*:refs/hidden//*\"\n we can ensure that we actually load the new values somewhere in\n our refspace while not updating refs/heads or refs/remotes. By\n storing these refs here, the commit-graph job will update the\n commit-graph with the commits from these hidden refs.\n\n4. --prune will delete the refs/hidden/ refs that no\n longer appear on the remote.\n\nWe've been using this step as a critical background job in Scalar\n[1] (and VFS for Git). This solved a pain point that was showing up\nin user reports: fetching was a pain! Users do not like waiting to\ndownload the data that was created while they were away from their\nmachines. After implementing background fetch, the foreground fetch\ncommands sped up significantly because they mostly just update refs\nand download a small amount of new data. The effect is especially\ndramatic when paried with --no-show-forced-udpates (through\nfetch.showForcedUpdates=false).\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 12 ++++++\n builtin/gc.c | 65 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 24 ++++++++++++\n 3 files changed, 100 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 9204762e21..e0be3f520f 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -53,6 +53,18 @@ since it will not expire `.graph` files that were in the previous\n `commit-graph-chain` file. They will be deleted by a later run based on\n the expiration delay.\n \n+fetch::\n+\tThe `fetch` job updates the object directory with the latest objects\n+\tfrom all registered remotes. For each remote, a `git fetch` command\n+\tis run. The refmap is custom to avoid updating local or remote\n+\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n+\tremote refs are stored in `refs/hidden//`. Also, no tags are\n+\tupdated.\n++\n+This means that foreground fetches are still required to update the\n+remote refs, but the users is notified when the branches and tags are\n+updated on the remote.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e3c634fc3b..2d30ae758c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -28,6 +28,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n+#include \"remote.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -705,7 +706,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 2\n+#define MAX_NUM_TASKS 3\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -788,6 +789,64 @@ static int maintenance_task_commit_graph(struct repository *r)\n \treturn 1;\n }\n \n+static int fetch_remote(struct repository *r, const char *remote)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf refmap = STRBUF_INIT;\n+\n+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n+\t\t\t \"fetch\", remote, \"--prune\",\n+\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n+\n+\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/hidden/%s/*\", remote);\n+\targv_array_push(&cmd, refmap.buf);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--quiet\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tstrbuf_release(&refmap);\n+\treturn result;\n+}\n+\n+static int fill_each_remote(struct remote *remote, void *cbdata)\n+{\n+\tstruct string_list *remotes = (struct string_list *)cbdata;\n+\n+\tstring_list_append(remotes, remote->name);\n+\treturn 0;\n+}\n+\n+static int maintenance_task_fetch(struct repository *r)\n+{\n+\tint result = 0;\n+\tstruct string_list_item *item;\n+\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n+\n+\tif (for_each_remote(fill_each_remote, &remotes)) {\n+\t\terror(_(\"failed to fill remotes\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * Do not modify the result based on the success of the 'fetch'\n+\t * operation, as a loss of network could cause 'fetch' to fail\n+\t * quickly. We do not want that to stop the rest of our\n+\t * background operations.\n+\t */\n+\tfor (item = remotes.items;\n+\t item && item < remotes.items + remotes.nr;\n+\t item++)\n+\t\tfetch_remote(r, item->string);\n+\n+cleanup:\n+\tstring_list_clear(&remotes, 0);\n+\treturn result;\n+}\n+\n static int maintenance_task_gc(struct repository *r)\n {\n \tint result;\n@@ -893,6 +952,10 @@ static void initialize_tasks(void)\n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n \n+\ttasks[num_tasks]->name = \"fetch\";\n+\ttasks[num_tasks]->fn = maintenance_task_fetch;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex c09a9eb90b..0abfc4a9da 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -44,4 +44,28 @@ test_expect_success 'run --task duplicate' '\n \ttest_i18ngrep \"cannot be selected multiple times\" err\n '\n \n+test_expect_success 'run --task=fetch with no remotes' '\n+\tgit maintenance run --task=fetch 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_expect_success 'fetch multiple remotes' '\n+\tgit clone . clone1 &&\n+\tgit clone . clone2 &&\n+\tgit remote add remote1 \"file://$(pwd)/clone1\" &&\n+\tgit remote add remote2 \"file://$(pwd)/clone2\" &&\n+\tgit -C clone1 switch -c one &&\n+\tgit -C clone2 switch -c two &&\n+\ttest_commit -C clone1 one &&\n+\ttest_commit -C clone2 two &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-fetch.txt\" git maintenance run --task=fetch &&\n+\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-fetch.txt &&\n+\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-fetch.txt &&\n+\ttest_path_is_missing .git/refs/remotes &&\n+\ttest_cmp clone1/.git/refs/heads/one .git/refs/hidden/remote1/one &&\n+\ttest_cmp clone2/.git/refs/heads/two .git/refs/hidden/remote2/two &&\n+\tgit log hidden/remote1/one &&\n+\tgit log hidden/remote2/two\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401206","messageId":"20200708235719.GA3189386@google.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-08T23:57:19Z","receivedAt":"2020-07-08T23:57:28Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Tue, Jul 07, 2020 at 02:21:14PM +0000, Derrick Stolee via GitGitGadget wrote:\n> \n> This is a second attempt at redesigning Git's repository maintenance\n> patterns. The first attempt [1] included a way to run jobs in the background\n> using a long-lived process; that idea was rejected and is not included in\n> this series. A future series will use the OS to handle scheduling tasks.\n> \n> [1] \n> https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n> \n> As mentioned before, git gc already plays the role of maintaining Git\n> repositories. It has accumulated several smaller pieces in its long history,\n> including:\n> \n> 1. Repacking all reachable objects into one pack-file (and deleting\n> unreachable objects).\n> 2. Packing refs.\n> 3. Expiring reflogs.\n> 4. Clearing rerere logs.\n> 5. Updating the commit-graph file.\n> \n> While expiring reflogs, clearing rererelogs, and deleting unreachable\n> objects are suitable under the guise of \"garbage collection\", packing refs\n> and updating the commit-graph file are not as obviously fitting. Further,\n> these operations are \"all or nothing\" in that they rewrite almost all\n> repository data, which does not perform well at extremely large scales.\n> These operations can also be disruptive to foreground Git commands when git\n> gc --auto triggers during routine use.\n> \n> This series does not intend to change what git gc does, but instead create\n> new choices for automatic maintenance activities, of which git gc remains\n> the only one enabled by default.\n> \n> The new maintenance tasks are:\n> \n> * 'commit-graph' : write and verify a single layer of an incremental\n> commit-graph.\n> * 'loose-objects' : prune packed loose objects, then create a new pack from\n> a batch of loose objects.\n> * 'pack-files' : expire redundant packs from the multi-pack-index, then\n> repack using the multi-pack-index's incremental repack strategy.\n> * 'fetch' : fetch from each remote, storing the refs in 'refs/hidden//'.\n> \n> These tasks are all disabled by default, but can be enabled with config\n> options or run explicitly using \"git maintenance run --task=\". There are\n> additional config options to allow customizing the conditions for which the\n> tasks run during the '--auto' option. ('fetch' will never run with the\n> '--auto' option.)\n> \n> Because 'gc' is implemented as a maintenance task, the most dramatic change\n> of this series is to convert the 'git gc --auto' calls into 'git maintenance\n> run --auto' calls at the end of some Git commands. By default, the only\n> change is that 'git gc --auto' will be run below an additional 'git\n> maintenance' process.\n> \n> The 'git maintenance' builtin has a 'run' subcommand so it can be extended\n> later with subcommands that manage background maintenance, such as 'start',\n> 'stop', 'pause', or 'schedule'. These are not the subject of this series, as\n> it is important to focus on the maintenance activities themselves.\n> \n> An expert user could set up scheduled background maintenance themselves with\n> the current series. I have the following crontab data set up to run\n> maintenance on an hourly basis:\n> \n> 0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n\nOne thing I wonder about - now I have to go and make a new crontab\n(which is easy) or Task Scheduler task (which is a pain) for every repo,\nright?\n\nIs it infeasible to ask for 'git maintenance' to learn something like\n'--on / --on /'? Or better yet, some\nconfig like \"maintenance.targetRepo = /\"?\n\n> \n> My config includes all tasks except the 'gc' task. The hourly run is\n> over-aggressive, but is sufficient for testing. I'll replace it with daily\n> when I feel satisfied.\n> \n> Hopefully this direction is seen as a positive one. My goal was to add more\n> options for expert users, along with the flexibility to create background\n> maintenance via the OS in a later series.\n> \n> OUTLINE\n> =======\n> \n> Patches 1-4 remove some references to the_repository in builtin/gc.c before\n> we start depending on code in that builtin.\n> \n> Patches 5-7 create the 'git maintenance run' builtin and subcommand as a\n> simple shim over 'git gc' and replaces calls to 'git gc --auto' from other\n> commands.\n\nFor me, I'd prefer to see 'git maintenance run' get bigger and 'git gc\n--auto' get smaller or disappear. Is there a plan towards that\ndirection, or is that out of scope for 'git maintenance run'? Similar\nexamples I can think of include 'git annotate' and 'git pickaxe'.\n\n> \n> Patches 8-15 create new maintenance tasks. These are the same tasks sent in\n> the previous RFC.\n> \n> Patches 16-21 create more customization through config and perform other\n> polish items.\n> \n> FUTURE WORK\n> ===========\n> \n> * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n> commands run in the background.\n> \n> \n> * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n> default, but might have different '--auto' conditions and more config\n> options.\n\nLike I mentioned above, for me, I'd rather just see the 'gc' builtin go\naway :)\n\n> \n> \n> * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n> with use of the 'commit-graph' task.\n> \n> \n> \n> Thanks, -Stolee\n> \n> Derrick Stolee (21):\n> gc: use the_repository less often\n> gc: use repository in too_many_loose_objects()\n> gc: use repo config\n> gc: drop the_repository in log location\n> maintenance: create basic maintenance runner\n> maintenance: add --quiet option\n> maintenance: replace run_auto_gc()\n> maintenance: initialize task array and hashmap\n> maintenance: add commit-graph task\n> maintenance: add --task option\n> maintenance: take a lock on the objects directory\n> maintenance: add fetch task\n> maintenance: add loose-objects task\n> maintenance: add pack-files task\n> maintenance: auto-size pack-files batch\n> maintenance: create maintenance..enabled config\n> maintenance: use pointers to check --auto\n> maintenance: add auto condition for commit-graph task\n> maintenance: create auto condition for loose-objects\n> maintenance: add pack-files auto condition\n> midx: use start_delayed_progress()\n> \n> .gitignore | 1 +\n> Documentation/config.txt | 2 +\n> Documentation/config/maintenance.txt | 32 +\n> Documentation/fetch-options.txt | 5 +-\n> Documentation/git-clone.txt | 7 +-\n> Documentation/git-maintenance.txt | 124 ++++\n> builtin.h | 1 +\n> builtin/am.c | 2 +-\n> builtin/commit.c | 2 +-\n> builtin/fetch.c | 6 +-\n> builtin/gc.c | 881 +++++++++++++++++++++++++--\n> builtin/merge.c | 2 +-\n> builtin/rebase.c | 4 +-\n> commit-graph.c | 8 +-\n> commit-graph.h | 1 +\n> config.c | 24 +-\n> config.h | 2 +\n> git.c | 1 +\n> midx.c | 12 +-\n> midx.h | 1 +\n> object.h | 1 +\n> run-command.c | 7 +-\n> run-command.h | 2 +-\n> t/t5319-multi-pack-index.sh | 14 +-\n> t/t5510-fetch.sh | 2 +-\n> t/t5514-fetch-multiple.sh | 2 +-\n> t/t7900-maintenance.sh | 211 +++++++\n> 27 files changed, 1265 insertions(+), 92 deletions(-)\n> create mode 100644 Documentation/config/maintenance.txt\n> create mode 100644 Documentation/git-maintenance.txt\n> create mode 100755 t/t7900-maintenance.sh\n> \n> \n> base-commit: 4a0fcf9f760c9774be77f51e1e88a7499b53d2e2\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-671%2Fderrickstolee%2Fmaintenance%2Fgc-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-671/derrickstolee/maintenance/gc-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/671\n> -- \n> gitgitgadget\n"},{"id":"401221","messageId":"20200709022257.36743-1-jonathantanmy@google.com","threadId":"53809","inReplyTo":"e13d351e9ff19695b2ff638af01fdb4527fb7a6d.1594131695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/21] gc: drop the_repository in log location","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-07-09T02:22:57Z","receivedAt":"2020-07-09T02:23:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> The report_last_gc_error() method use git_pathdup() which implicitly\n> uses the_repository. Replace this with strbuf_repo_path() to get a\n> path buffer we control that uses a given repository pointer.\n> \n> Signed-off-by: Derrick Stolee \n\nRegarding the first 4 patches, it would be great if there was a test\nlike the one in test_repository.c (together with a code coverage report,\nperhaps) that verifies that all code paths do not use the_repository.\n\nBut set aside that test for now - I don't think gc.c fully supports\narbitrary repositories. In particular, when running a subprocess, it\ninherits the environment from the current process. I see that future\npatches try to resolve this by passing \"-C\", but that does not work if\nenvironment variables like GIT_DIR are set (because the environment\nvariables override the \"-C\"). Perhaps we need a function that runs a\nsubprocess in a specific repository. I ran into the same problem when\nattempting to make fetch-pack (which runs index-pack as a subprocess)\nsupport arbitrary repositories, but I haven't looked deeply into\nresolving this yet (and I haven't looked at that problem in a while).\n\nHaving said that, I'm fine with these patches being in the set - the\nonly negative is that perhaps a reader would be misled into thinking\nthat GC supports arbitrary repositories.\n"},{"id":"401222","messageId":"20200709022519.38407-1-jonathantanmy@google.com","threadId":"53809","inReplyTo":"5cdd38afa60cdf768dd194f90ae0b2190123fdea.1594131695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/21] maintenance: initialize task array and hashmap","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-07-09T02:25:19Z","receivedAt":"2020-07-09T02:25:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> This list is also inserted into a hashmap. This allows command-line\n> arguments to quickly find the tasks by name, not sensitive to case. To\n> ensure this list and hashmap work well together, the list only contains\n> pointers to the struct information. This will allow a sort on the list\n> while preserving the hashmap data.\n\nI think having the hashmap is unnecessarily complicated in this case -\nwith the small number of tasks, a list would be fine. But I don't feel\nstrongly about this.\n"},{"id":"401223","messageId":"20200709022942.39848-1-jonathantanmy@google.com","threadId":"53809","inReplyTo":"c8fbd14d41093e12905027ff628b98652cd931d0.1594131695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/21] maintenance: add commit-graph task","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-07-09T02:29:42Z","receivedAt":"2020-07-09T02:29:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +static int run_write_commit_graph(struct repository *r)\n> +{\n> +\tint result;\n> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> +\n> +\targv_array_pushl(&cmd, \"-C\", r->worktree,\n> +\t\t\t \"commit-graph\", \"write\",\n> +\t\t\t \"--split\", \"--reachable\",\n> +\t\t\t NULL);\n\nAs mentioned in my reply to an earlier patch (sent a few minutes ago),\nthis won't work if there are environment variables like GIT_DIR present.\n\nBesides that, the overall design looks reasonable.\n"},{"id":"401224","messageId":"20200709023531.41660-1-jonathantanmy@google.com","threadId":"53809","inReplyTo":"cbaa5ecc4f23eed0823fbbb53ffef28c9f7d6913.1594131695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 12/21] maintenance: add fetch task","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2020-07-09T02:35:31Z","receivedAt":"2020-07-09T02:35:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> 3. By adding a new refspec \"+refs/heads/*:refs/hidden//*\"\n> we can ensure that we actually load the new values somewhere in\n> our refspace while not updating refs/heads or refs/remotes. By\n> storing these refs here, the commit-graph job will update the\n> commit-graph with the commits from these hidden refs.\n> \n> 4. --prune will delete the refs/hidden/ refs that no\n> longer appear on the remote.\n\nHaving a ref path where Git can place commit IDs that it needs persisted\nis useful, not only in this case but in other cases (e.g. when fetching\na submodule commit by hash, we might not have a ref name for that commit\nbut want to persist it anyway), so I look forward to having something\nlike this.\n\nThe name of this special ref path and its specific nature could be\ndiscussed further, but maybe it is sufficient for now to just say that\nthe refs under this special ref path are controlled by Git, and their\nlayout is experimental and subject to change (e.g. future versions of\nGit could just erase the entire path and rewrite the refs its own way).\n"},{"id":"401236","messageId":"4170de4d-c77e-e8a9-1df4-e258e5785360@gmail.com","threadId":"53809","inReplyTo":"20200709022257.36743-1-jonathantanmy@google.com","subject":"Re: [PATCH 04/21] gc: drop the_repository in log location","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T11:13:18Z","receivedAt":"2020-07-09T11:13:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2020 10:22 PM, Jonathan Tan wrote:\n>> From: Derrick Stolee \n>>\n>> The report_last_gc_error() method use git_pathdup() which implicitly\n>> uses the_repository. Replace this with strbuf_repo_path() to get a\n>> path buffer we control that uses a given repository pointer.\n>>\n>> Signed-off-by: Derrick Stolee \n> \n> Regarding the first 4 patches, it would be great if there was a test\n> like the one in test_repository.c (together with a code coverage report,\n> perhaps) that verifies that all code paths do not use the_repository.\n> \n> But set aside that test for now - I don't think gc.c fully supports\n> arbitrary repositories. In particular, when running a subprocess, it\n> inherits the environment from the current process. I see that future\n> patches try to resolve this by passing \"-C\", but that does not work if\n> environment variables like GIT_DIR are set (because the environment\n> variables override the \"-C\"). Perhaps we need a function that runs a\n> subprocess in a specific repository. I ran into the same problem when\n> attempting to make fetch-pack (which runs index-pack as a subprocess)\n> support arbitrary repositories, but I haven't looked deeply into\n> resolving this yet (and I haven't looked at that problem in a while).\n> \n> Having said that, I'm fine with these patches being in the set - the\n> only negative is that perhaps a reader would be misled into thinking\n> that GC supports arbitrary repositories.\n\nI agree. I hope that I do not give the impression that GC is now\nsafe for arbitrary repositories. I only thought that this was prudent\nto do before I start taking new dependencies on the code.\n\nIt' probably time for someone to do a round of the_repository cleanups\nagain, and perhaps the RC window is a good time to think about that\n(for submission after the release).\n\nThanks,\n-Stolee\n\n\n"},{"id":"401237","messageId":"8dacf017-c445-578c-c82c-f6d6f93469c9@gmail.com","threadId":"53809","inReplyTo":"20200709022942.39848-1-jonathantanmy@google.com","subject":"Re: [PATCH 09/21] maintenance: add commit-graph task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T11:14:41Z","receivedAt":"2020-07-09T11:14:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2020 10:29 PM, Jonathan Tan wrote:\n>> +static int run_write_commit_graph(struct repository *r)\n>> +{\n>> +\tint result;\n>> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n>> +\n>> +\targv_array_pushl(&cmd, \"-C\", r->worktree,\n>> +\t\t\t \"commit-graph\", \"write\",\n>> +\t\t\t \"--split\", \"--reachable\",\n>> +\t\t\t NULL);\n> \n> As mentioned in my reply to an earlier patch (sent a few minutes ago),\n> this won't work if there are environment variables like GIT_DIR present.\n\nDo we not pass GIT_DIR to the subcommand? Or does using \"-C\" override\nthe GIT_DIR?\n\nThanks,\n-Stolee\n\n"},{"id":"401239","messageId":"ef6e2704-b122-f620-a952-75f3f9efab78@gmail.com","threadId":"53809","inReplyTo":"20200708235719.GA3189386@google.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T11:21:00Z","receivedAt":"2020-07-09T11:21:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2020 7:57 PM, Emily Shaffer wrote:\n> On Tue, Jul 07, 2020 at 02:21:14PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>\n>> This is a second attempt at redesigning Git's repository maintenance\n>> patterns. The first attempt [1] included a way to run jobs in the background\n>> using a long-lived process; that idea was rejected and is not included in\n>> this series. A future series will use the OS to handle scheduling tasks.\n>>\n>> [1] \n>> https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n>>\n>> As mentioned before, git gc already plays the role of maintaining Git\n>> repositories. It has accumulated several smaller pieces in its long history,\n>> including:\n>>\n>> 1. Repacking all reachable objects into one pack-file (and deleting\n>> unreachable objects).\n>> 2. Packing refs.\n>> 3. Expiring reflogs.\n>> 4. Clearing rerere logs.\n>> 5. Updating the commit-graph file.\n>>\n>> While expiring reflogs, clearing rererelogs, and deleting unreachable\n>> objects are suitable under the guise of \"garbage collection\", packing refs\n>> and updating the commit-graph file are not as obviously fitting. Further,\n>> these operations are \"all or nothing\" in that they rewrite almost all\n>> repository data, which does not perform well at extremely large scales.\n>> These operations can also be disruptive to foreground Git commands when git\n>> gc --auto triggers during routine use.\n>>\n>> This series does not intend to change what git gc does, but instead create\n>> new choices for automatic maintenance activities, of which git gc remains\n>> the only one enabled by default.\n>>\n>> The new maintenance tasks are:\n>>\n>> * 'commit-graph' : write and verify a single layer of an incremental\n>> commit-graph.\n>> * 'loose-objects' : prune packed loose objects, then create a new pack from\n>> a batch of loose objects.\n>> * 'pack-files' : expire redundant packs from the multi-pack-index, then\n>> repack using the multi-pack-index's incremental repack strategy.\n>> * 'fetch' : fetch from each remote, storing the refs in 'refs/hidden//'.\n>>\n>> These tasks are all disabled by default, but can be enabled with config\n>> options or run explicitly using \"git maintenance run --task=\". There are\n>> additional config options to allow customizing the conditions for which the\n>> tasks run during the '--auto' option. ('fetch' will never run with the\n>> '--auto' option.)\n>>\n>> Because 'gc' is implemented as a maintenance task, the most dramatic change\n>> of this series is to convert the 'git gc --auto' calls into 'git maintenance\n>> run --auto' calls at the end of some Git commands. By default, the only\n>> change is that 'git gc --auto' will be run below an additional 'git\n>> maintenance' process.\n>>\n>> The 'git maintenance' builtin has a 'run' subcommand so it can be extended\n>> later with subcommands that manage background maintenance, such as 'start',\n>> 'stop', 'pause', or 'schedule'. These are not the subject of this series, as\n>> it is important to focus on the maintenance activities themselves.\n>>\n>> An expert user could set up scheduled background maintenance themselves with\n>> the current series. I have the following crontab data set up to run\n>> maintenance on an hourly basis:\n>>\n>> 0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n> \n> One thing I wonder about - now I have to go and make a new crontab\n> (which is easy) or Task Scheduler task (which is a pain) for every repo,\n> right?\n> \n> Is it infeasible to ask for 'git maintenance' to learn something like\n> '--on / --on /'? Or better yet, some\n> config like \"maintenance.targetRepo = /\"?\n> \n>>\n>> My config includes all tasks except the 'gc' task. The hourly run is\n>> over-aggressive, but is sufficient for testing. I'll replace it with daily\n>> when I feel satisfied.\n>>\n>> Hopefully this direction is seen as a positive one. My goal was to add more\n>> options for expert users, along with the flexibility to create background\n>> maintenance via the OS in a later series.\n>>\n>> OUTLINE\n>> =======\n>>\n>> Patches 1-4 remove some references to the_repository in builtin/gc.c before\n>> we start depending on code in that builtin.\n>>\n>> Patches 5-7 create the 'git maintenance run' builtin and subcommand as a\n>> simple shim over 'git gc' and replaces calls to 'git gc --auto' from other\n>> commands.\n> \n> For me, I'd prefer to see 'git maintenance run' get bigger and 'git gc\n> --auto' get smaller or disappear. Is there a plan towards that\n> direction, or is that out of scope for 'git maintenance run'? Similar\n> examples I can think of include 'git annotate' and 'git pickaxe'.\n\nThanks for these examples of prior work. I'll keep them in mind.\n\n>> * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n>> default, but might have different '--auto' conditions and more config\n>> options.\n> \n> Like I mentioned above, for me, I'd rather just see the 'gc' builtin go\n> away :)\n\nMy hope is that we can absolutely do that. I didn't want to start that\nexercise yet, as I don't want to disrupt existing workflows more than\nI already am.\n\nIt is important to recognize that there are already several \"tasks\" that\nrun inside 'gc' including:\n\n1. Expiring reflogs.\n2. Repacking all reachable objects.\n3. Deleting unreachable objects.\n4. Packing refs.\n\nBefore trying to \"remove\" the gc builtin, we would want these to be\nrepresented in the 'git maintenance run' as tasks.\n\nIn that direction, I realized after submitting that I should rename\nthe 'pack-files' task in this submission to 'incremental-repack'\ninstead, allowing a later 'full-repack' task to represent the role\nof that step in the 'gc' task. Some users will prefer one over the\nother. Perhaps this incremental/full distinction makes it clear that\nthere are trade-offs in both directions.\n\nThanks,\n-Stolee\n"},{"id":"401242","messageId":"d0d30dab-7883-199a-3beb-48c87cde3550@gmail.com","threadId":"53809","inReplyTo":"ef6e2704-b122-f620-a952-75f3f9efab78@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T12:43:48Z","receivedAt":"2020-07-09T12:43:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2020 7:21 AM, Derrick Stolee wrote:\n> On 7/8/2020 7:57 PM, Emily Shaffer wrote:\n>> On Tue, Jul 07, 2020 at 02:21:14PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>> An expert user could set up scheduled background maintenance themselves with\n>>> the current series. I have the following crontab data set up to run\n>>> maintenance on an hourly basis:\n>>>\n>>> 0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n>>\n>> One thing I wonder about - now I have to go and make a new crontab\n>> (which is easy) or Task Scheduler task (which is a pain) for every repo,\n>> right?\n>>\n>> Is it infeasible to ask for 'git maintenance' to learn something like\n>> '--on / --on /'? Or better yet, some\n>> config like \"maintenance.targetRepo = /\"?\n\nSorry that I missed this comment on my first reply.\n\nThe intention is that this cron entry will be simpler after I follow up\nwith the \"background\" part of maintenance. The idea is to use global\nor system config to register a list of repositories that want background\nmaintenance and have cron execute something like \"git maintenance run --all-repos\"\nto span \"git -C maintenance run --scheduled\" for all repos in\nthe config.\n\nFor now, this manual setup does end up a bit cluttered if you have a\nlot of repos to maintain.\n\nThanks,\n-Stolee\n\n"},{"id":"401243","messageId":"329a6be3-df8f-971d-14ac-5d4f9377a25f@gmail.com","threadId":"53809","inReplyTo":"20200709022519.38407-1-jonathantanmy@google.com","subject":"Re: [PATCH 08/21] maintenance: initialize task array and hashmap","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T13:15:35Z","receivedAt":"2020-07-09T13:15:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/8/2020 10:25 PM, Jonathan Tan wrote:\n>> This list is also inserted into a hashmap. This allows command-line\n>> arguments to quickly find the tasks by name, not sensitive to case. To\n>> ensure this list and hashmap work well together, the list only contains\n>> pointers to the struct information. This will allow a sort on the list\n>> while preserving the hashmap data.\n> \n> I think having the hashmap is unnecessarily complicated in this case -\n> with the small number of tasks, a list would be fine. But I don't feel\n> strongly about this.\n\nYou're probably right that iterating through a list with (hopefully)\nat most a dozen entries is fast enough that a hashmap is overkill here.\n\nNow is the real test: can I change this patch in v2 without needing\nto mess with any of the others? The intention here was to make adding\ntasks as simple as possible, so we shall see. :D\n\nThanks,\n-Stolee\n\n\n"},{"id":"401245","messageId":"xmqqzh88yexk.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"329a6be3-df8f-971d-14ac-5d4f9377a25f@gmail.com","subject":"Re: [PATCH 08/21] maintenance: initialize task array and hashmap","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-09T13:51:35Z","receivedAt":"2020-07-09T13:51:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 7/8/2020 10:25 PM, Jonathan Tan wrote:\n>>> This list is also inserted into a hashmap. This allows command-line\n>>> arguments to quickly find the tasks by name, not sensitive to case. To\n>>> ensure this list and hashmap work well together, the list only contains\n>>> pointers to the struct information. This will allow a sort on the list\n>>> while preserving the hashmap data.\n>> \n>> I think having the hashmap is unnecessarily complicated in this case -\n>> with the small number of tasks, a list would be fine. But I don't feel\n>> strongly about this.\n>\n> You're probably right that iterating through a list with (hopefully)\n> at most a dozen entries is fast enough that a hashmap is overkill here.\n>\n> Now is the real test: can I change this patch in v2 without needing\n> to mess with any of the others? The intention here was to make adding\n> tasks as simple as possible, so we shall see. :D\n\nAdding a new element to a list would be simple no matter how the\nlist is represented. But I think the real question is what access\npattern we expect. Do we need to look up by name a single one or\nselected few? Do we need the iteration/enumeration be stable? etc.\n\n"},{"id":"401246","messageId":"xmqqv9iwyeac.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"ef6e2704-b122-f620-a952-75f3f9efab78@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-09T14:05:31Z","receivedAt":"2020-07-09T14:05:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> It is important to recognize that there are already several \"tasks\" that\n> run inside 'gc' including:\n>\n> 1. Expiring reflogs.\n> 2. Repacking all reachable objects.\n> 3. Deleting unreachable objects.\n> 4. Packing refs.\n>\n> Before trying to \"remove\" the gc builtin, we would want these to be\n> represented in the 'git maintenance run' as tasks.\n\nYup. I like the overall direction of this approach to (1) have a\nsingle subcommand that helps all the housekeeping tasks, and to (2)\nmake sure existing housekeeping tasks are supported by the new one.\n\nI can understand why it is tempting to start with a new 'main()'\nunder a new subcommand name because we expect to add a lot more\ntasks, but the name of that subcommand is much less important.\n\nAs can be seen in the list you have above, \"gc\" already does a lot\nmore than garbage collection (just #3 is the \"garbage collection\"\nproper), as it has grown by following the same approach.\n\nWhat's more important is (2) above. While the tool has grown under\nthe same \"gc\" name, it was easier to arrange---it fell out naturally\nas a consequence of the development being an enhancement on top of\nthe prior work. Now that we are reimplementing, we need to actively\ncare. As long as we recognize that, I am perfectly happy with the\ncurrent effort.\n\nFor existing callers, \"git gc --auto\" may want to be left alive,\nmerely as a thin wrapper around \"git maintenance --auto\", and as\nlong as the latter is done in the same spirit of the former, i.e.\nperform a lightweight check to see if the repository is so out of\nshape and then do a minimum cleaning, it would be welcomed by users\nif it does a lot more than the current \"git gc --auto\".\n\nThanks.\n"},{"id":"401248","messageId":"52715fe8-a545-5c42-2f3a-e59bcbc43497@gmail.com","threadId":"53809","inReplyTo":"xmqqv9iwyeac.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T15:54:20Z","receivedAt":"2020-07-09T15:54:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2020 10:05 AM, Junio C Hamano wrote:\n> For existing callers, \"git gc --auto\" may want to be left alive,\n> merely as a thin wrapper around \"git maintenance --auto\", and as\n> long as the latter is done in the same spirit of the former, i.e.\n> perform a lightweight check to see if the repository is so out of\n> shape and then do a minimum cleaning, it would be welcomed by users\n> if it does a lot more than the current \"git gc --auto\".\n\nIt's entirely possible that (after the 'maintenance' builtin\nstabilizes) that we make 'git gc --auto' become an alias of something\nlike 'git maintenance run --task=gc --auto' (or itemize all of the\nsub-tasks) so that 'git gc --auto' doesn't change behavior.\n\nThat's a big motivation for adding all code into builtin/gc.c so\nwe can access these tasks inside GC without needing to move or\ncopy the code. I'm trying to preserve history as much as possible.\n\nThanks,\n-Stolee\n\n"},{"id":"401249","messageId":"xmqqmu48y7rw.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"52715fe8-a545-5c42-2f3a-e59bcbc43497@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-09T16:26:11Z","receivedAt":"2020-07-09T16:26:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 7/9/2020 10:05 AM, Junio C Hamano wrote:\n>> For existing callers, \"git gc --auto\" may want to be left alive,\n>> merely as a thin wrapper around \"git maintenance --auto\", and as\n>> long as the latter is done in the same spirit of the former, i.e.\n>> perform a lightweight check to see if the repository is so out of\n>> shape and then do a minimum cleaning, it would be welcomed by users\n>> if it does a lot more than the current \"git gc --auto\".\n>\n> It's entirely possible that (after the 'maintenance' builtin\n> stabilizes) that we make 'git gc --auto' become an alias of something\n> like 'git maintenance run --task=gc --auto' (or itemize all of the\n> sub-tasks) so that 'git gc --auto' doesn't change behavior.\n\nYes, it is possible, but I doubt it is desirable.\n\nThe current users of \"gc --auto\" do not (and should not) care the\ndetails of what tasks are performed. We surely have added more\nstuff that need maintenance since \"gc --auto\" was originally\nwritten, and after people have started using \"gc --auto\" in their\nworkflows. For example, I think \"gc --auto\" predates \"rerere gc\"\nand those who had \"gc --auto\" in their script had a moment when\nsuddenly it started to clean stale entries in the rerere database.\n\nWere they got upset when it happened? Will they get upset when it\nstarts cleaning up stale commit-graph leftover files?\n\nAs long as \"gc --auto\" kept the same spirit of doing a lightweight\ncheck to see if the repository is so out of shape to require\ncleaning and performing a minimum maintenance when it started\ncalling \"rerere gc\", and as long as \"maintenance --auto\" does the\nsame, I would think the users would be delighted without complaints.\n\nSo, I wouldn't worry too much about what exactly happens with the\nfuture versions of \"gc --auto\". The world has changed, and we have\nmore items in the repository that needs maintenance/cruft removal.\nThe command in the new world should deal with these new stuff, too.\n\nThanks.\n"},{"id":"401251","messageId":"ecadda92-2290-4e96-0e3c-27fbe7e0a6e6@gmail.com","threadId":"53809","inReplyTo":"xmqqmu48y7rw.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T16:56:14Z","receivedAt":"2020-07-09T16:56:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2020 12:26 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 7/9/2020 10:05 AM, Junio C Hamano wrote:\n>>> For existing callers, \"git gc --auto\" may want to be left alive,\n>>> merely as a thin wrapper around \"git maintenance --auto\", and as\n>>> long as the latter is done in the same spirit of the former, i.e.\n>>> perform a lightweight check to see if the repository is so out of\n>>> shape and then do a minimum cleaning, it would be welcomed by users\n>>> if it does a lot more than the current \"git gc --auto\".\n>>\n>> It's entirely possible that (after the 'maintenance' builtin\n>> stabilizes) that we make 'git gc --auto' become an alias of something\n>> like 'git maintenance run --task=gc --auto' (or itemize all of the\n>> sub-tasks) so that 'git gc --auto' doesn't change behavior.\n> \n> Yes, it is possible, but I doubt it is desirable.\n> \n> The current users of \"gc --auto\" do not (and should not) care the\n> details of what tasks are performed. We surely have added more\n> stuff that need maintenance since \"gc --auto\" was originally\n> written, and after people have started using \"gc --auto\" in their\n> workflows. For example, I think \"gc --auto\" predates \"rerere gc\"\n> and those who had \"gc --auto\" in their script had a moment when\n> suddenly it started to clean stale entries in the rerere database.\n> \n> Were they got upset when it happened? Will they get upset when it\n> starts cleaning up stale commit-graph leftover files?\n> \n> As long as \"gc --auto\" kept the same spirit of doing a lightweight\n> check to see if the repository is so out of shape to require\n> cleaning and performing a minimum maintenance when it started\n> calling \"rerere gc\", and as long as \"maintenance --auto\" does the\n> same, I would think the users would be delighted without complaints.\n> \n> So, I wouldn't worry too much about what exactly happens with the\n> future versions of \"gc --auto\". The world has changed, and we have\n> more items in the repository that needs maintenance/cruft removal.\n> The command in the new world should deal with these new stuff, too.\n\nSounds good to me. The extra context around this helps a lot!\n\n-Stolee\n"},{"id":"401273","messageId":"20200709225246.GA664420@coredump.intra.peff.net","threadId":"53809","inReplyTo":"8dacf017-c445-578c-c82c-f6d6f93469c9@gmail.com","subject":"Re: [PATCH 09/21] maintenance: add commit-graph task","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-07-09T22:52:46Z","receivedAt":"2020-07-09T22:52:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 09, 2020 at 07:14:41AM -0400, Derrick Stolee wrote:\n\n> On 7/8/2020 10:29 PM, Jonathan Tan wrote:\n> >> +static int run_write_commit_graph(struct repository *r)\n> >> +{\n> >> +\tint result;\n> >> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> >> +\n> >> +\targv_array_pushl(&cmd, \"-C\", r->worktree,\n> >> +\t\t\t \"commit-graph\", \"write\",\n> >> +\t\t\t \"--split\", \"--reachable\",\n> >> +\t\t\t NULL);\n> > \n> > As mentioned in my reply to an earlier patch (sent a few minutes ago),\n> > this won't work if there are environment variables like GIT_DIR present.\n> \n> Do we not pass GIT_DIR to the subcommand? Or does using \"-C\" override\n> the GIT_DIR?\n\nWe do pass GIT_DIR to the subcommand, and \"-C\" does not override it. I\nthink this code would work as long as \"r\" is the_repository, which it\nwould be in the current code. But then the \"-C\" would be doing nothing\nuseful (it might change to the top of the worktree if we weren't there\nfor some reason, but I don't think \"commit-graph write\" would care\neither way).\n\nBut if \"r\" is some other repository, \"commit-graph\" would continue to\noperate in the parent process repository because of the inherited\nGIT_DIR. Using \"--git-dir\" would solve that, but as a general practice,\nif you're spawning a sub-process that might be in another repository,\nyou should clear any repo-specific environment variables. The list is in\nlocal_repo_env, which you can feed to the \"env\" or \"env_array\" parameter\nof a child_process (see the use in connect.c for an example).\n\nEven in the current scheme where \"r\" is always the_repository, I suspect\nthis might still be buggy. If we're in a bare repository, presumably\nr->worktree would be NULL.\n\n-Peff\n"},{"id":"401277","messageId":"20200709231624.GE664420@coredump.intra.peff.net","threadId":"53809","inReplyTo":"d0d30dab-7883-199a-3beb-48c87cde3550@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-07-09T23:16:24Z","receivedAt":"2020-07-09T23:16:26Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 09, 2020 at 08:43:48AM -0400, Derrick Stolee wrote:\n\n> >> Is it infeasible to ask for 'git maintenance' to learn something like\n> >> '--on / --on /'? Or better yet, some\n> >> config like \"maintenance.targetRepo = /\"?\n> \n> Sorry that I missed this comment on my first reply.\n> \n> The intention is that this cron entry will be simpler after I follow up\n> with the \"background\" part of maintenance. The idea is to use global\n> or system config to register a list of repositories that want background\n> maintenance and have cron execute something like \"git maintenance run --all-repos\"\n> to span \"git -C maintenance run --scheduled\" for all repos in\n> the config.\n> \n> For now, this manual setup does end up a bit cluttered if you have a\n> lot of repos to maintain.\n\nI think it might be useful to have a general command to run a subcommand\nin a bunch of repositories. Something like:\n\n git for-each-repo --recurse /path/to/repos git maintenance ...\n\nwhich would root around in /path/to/repos for any git-dirs and run \"git\n--git-dir=$GIT_DIR maintenance ...\" on each of them.\n\nAnd/or:\n\n git for-each-repo --config maintenance.repos git maintenance ...\n\nwhich would pull the set of repos from the named config variable instead\nof looking around the filesystem.\n\nYou could use either as a one-liner in the crontab (depending on which\nis easier with your repo layout).\n\n-Peff\n"},{"id":"401278","messageId":"59d11581-c182-8e00-2698-8b632683808b@gmail.com","threadId":"53809","inReplyTo":"20200709225246.GA664420@coredump.intra.peff.net","subject":"Re: [PATCH 09/21] maintenance: add commit-graph task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T23:41:16Z","receivedAt":"2020-07-09T23:41:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2020 6:52 PM, Jeff King wrote:\n> On Thu, Jul 09, 2020 at 07:14:41AM -0400, Derrick Stolee wrote:\n> \n>> On 7/8/2020 10:29 PM, Jonathan Tan wrote:\n>>>> +static int run_write_commit_graph(struct repository *r)\n>>>> +{\n>>>> +\tint result;\n>>>> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n>>>> +\n>>>> +\targv_array_pushl(&cmd, \"-C\", r->worktree,\n>>>> +\t\t\t \"commit-graph\", \"write\",\n>>>> +\t\t\t \"--split\", \"--reachable\",\n>>>> +\t\t\t NULL);\n>>>\n>>> As mentioned in my reply to an earlier patch (sent a few minutes ago),\n>>> this won't work if there are environment variables like GIT_DIR present.\n>>\n>> Do we not pass GIT_DIR to the subcommand? Or does using \"-C\" override\n>> the GIT_DIR?\n> \n> We do pass GIT_DIR to the subcommand, and \"-C\" does not override it. I\n> think this code would work as long as \"r\" is the_repository, which it\n> would be in the current code. But then the \"-C\" would be doing nothing\n> useful (it might change to the top of the worktree if we weren't there\n> for some reason, but I don't think \"commit-graph write\" would care\n> either way).\n> \n> But if \"r\" is some other repository, \"commit-graph\" would continue to\n> operate in the parent process repository because of the inherited\n> GIT_DIR. Using \"--git-dir\" would solve that, but as a general practice,\n> if you're spawning a sub-process that might be in another repository,\n> you should clear any repo-specific environment variables. The list is in\n> local_repo_env, which you can feed to the \"env\" or \"env_array\" parameter\n> of a child_process (see the use in connect.c for an example).\n> \n> Even in the current scheme where \"r\" is always the_repository, I suspect\n> this might still be buggy. If we're in a bare repository, presumably\n> r->worktree would be NULL.\n\nAh. I'll investigate this more and work to create a way to\nrun a subcommand in a given repository. Your pointers will\nhelp a lot.\n\nThanks,\n-Stolee\n\n"},{"id":"401279","messageId":"28c72d69-f524-7c4f-4cc3-847ca91cad33@gmail.com","threadId":"53809","inReplyTo":"20200709231624.GE664420@coredump.intra.peff.net","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-09T23:45:47Z","receivedAt":"2020-07-09T23:45:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/9/2020 7:16 PM, Jeff King wrote:\n> On Thu, Jul 09, 2020 at 08:43:48AM -0400, Derrick Stolee wrote:\n> \n>>>> Is it infeasible to ask for 'git maintenance' to learn something like\n>>>> '--on / --on /'? Or better yet, some\n>>>> config like \"maintenance.targetRepo = /\"?\n>>\n>> Sorry that I missed this comment on my first reply.\n>>\n>> The intention is that this cron entry will be simpler after I follow up\n>> with the \"background\" part of maintenance. The idea is to use global\n>> or system config to register a list of repositories that want background\n>> maintenance and have cron execute something like \"git maintenance run --all-repos\"\n>> to span \"git -C maintenance run --scheduled\" for all repos in\n>> the config.\n>>\n>> For now, this manual setup does end up a bit cluttered if you have a\n>> lot of repos to maintain.\n> \n> I think it might be useful to have a general command to run a subcommand\n> in a bunch of repositories. Something like:\n> \n> git for-each-repo --recurse /path/to/repos git maintenance ...\n> \n> which would root around in /path/to/repos for any git-dirs and run \"git\n> --git-dir=$GIT_DIR maintenance ...\" on each of them.\n> \n> And/or:\n> \n> git for-each-repo --config maintenance.repos git maintenance ...\n> \n> which would pull the set of repos from the named config variable instead\n> of looking around the filesystem.\n\nYes! This! That's a good way to make something generic that solves\nthe problem at hand, but might also have other applications! Most\nexcellent.\n\n> You could use either as a one-liner in the crontab (depending on which\n> is easier with your repo layout).\n\nThe hope is that we can have such a clean layout. I'm particularly\nfond of the config option because users may want to opt-in to\nbackground maintenance only on some repos, even if they put them\nin a consistent location.\n\nIn the _far_ future, we might even want to add a repo to this\n\"maintenance.repos\" list during 'git init' and 'git clone' so\nthis is automatic. It then becomes opt-out at that point, which\nis why I saw the _far, far_ future.\n\nThanks,\n-Stolee\n"},{"id":"401362","messageId":"20200710184651.GB3189386@google.com","threadId":"53809","inReplyTo":"28c72d69-f524-7c4f-4cc3-847ca91cad33@gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-10T18:46:51Z","receivedAt":"2020-07-10T18:47:01Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 09, 2020 at 07:45:47PM -0400, Derrick Stolee wrote:\n> \n> On 7/9/2020 7:16 PM, Jeff King wrote:\n> > On Thu, Jul 09, 2020 at 08:43:48AM -0400, Derrick Stolee wrote:\n> > \n> >>>> Is it infeasible to ask for 'git maintenance' to learn something like\n> >>>> '--on / --on /'? Or better yet, some\n> >>>> config like \"maintenance.targetRepo = /\"?\n> >>\n> >> Sorry that I missed this comment on my first reply.\n> >>\n> >> The intention is that this cron entry will be simpler after I follow up\n> >> with the \"background\" part of maintenance. The idea is to use global\n> >> or system config to register a list of repositories that want background\n> >> maintenance and have cron execute something like \"git maintenance run --all-repos\"\n> >> to span \"git -C maintenance run --scheduled\" for all repos in\n> >> the config.\n> >>\n> >> For now, this manual setup does end up a bit cluttered if you have a\n> >> lot of repos to maintain.\n> > \n> > I think it might be useful to have a general command to run a subcommand\n> > in a bunch of repositories. Something like:\n> > \n> > git for-each-repo --recurse /path/to/repos git maintenance ...\n> > \n> > which would root around in /path/to/repos for any git-dirs and run \"git\n> > --git-dir=$GIT_DIR maintenance ...\" on each of them.\n> > \n> > And/or:\n> > \n> > git for-each-repo --config maintenance.repos git maintenance ...\n> > \n> > which would pull the set of repos from the named config variable instead\n> > of looking around the filesystem.\n> \n> Yes! This! That's a good way to make something generic that solves\n> the problem at hand, but might also have other applications! Most\n> excellent.\n\nI'm glad I wasn't the only one super geeked when I read this idea. I'd\nuse the heck out of this in my .bashrc too. Sounds awesome. I actually\nhad a short-lived fling last year with a script to summarize my\nuncommitted changes in all repos at the beginning of every session\n(dropped because it became one more thing to gloss over) and could have\nreally used this command.\n\n> \n> > You could use either as a one-liner in the crontab (depending on which\n> > is easier with your repo layout).\n> \n> The hope is that we can have such a clean layout. I'm particularly\n> fond of the config option because users may want to opt-in to\n> background maintenance only on some repos, even if they put them\n> in a consistent location.\n> \n> In the _far_ future, we might even want to add a repo to this\n> \"maintenance.repos\" list during 'git init' and 'git clone' so\n> this is automatic. It then becomes opt-out at that point, which\n> is why I saw the _far, far_ future.\n\nOh, I like this idea a lot. Then I can do something silly like\n\n alias reproclone=\"git clone --no-maintainenance\"\n\nand get the benefits on everything else that I plan to be using\nfrequently.\n\n - Emily\n"},{"id":"401370","messageId":"23F67854-B596-4728-86CD-A9927546DD43@gmail.com","threadId":"53809","inReplyTo":"20200710184651.GB3189386@google.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Son Luong Ngoc","fromEmail":"sluongng@gmail.com","sentAt":"2020-07-10T19:30:37Z","receivedAt":"2020-07-10T19:30:42Z","isPatch":true,"sender":{"key":"sluongng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/26684313?v=4"},"body":"\n\n> On Jul 10, 2020, at 20:46, Emily Shaffer wrote:\n> \n> On Thu, Jul 09, 2020 at 07:45:47PM -0400, Derrick Stolee wrote:\n>> \n>> On 7/9/2020 7:16 PM, Jeff King wrote:\n>>> On Thu, Jul 09, 2020 at 08:43:48AM -0400, Derrick Stolee wrote:\n>>> \n>>>>>> Is it infeasible to ask for 'git maintenance' to learn something like\n>>>>>> '--on / --on /'? Or better yet, some\n>>>>>> config like \"maintenance.targetRepo = /\"?\n>>>> \n>>>> Sorry that I missed this comment on my first reply.\n>>>> \n>>>> The intention is that this cron entry will be simpler after I follow up\n>>>> with the \"background\" part of maintenance. The idea is to use global\n>>>> or system config to register a list of repositories that want background\n>>>> maintenance and have cron execute something like \"git maintenance run --all-repos\"\n>>>> to span \"git -C maintenance run --scheduled\" for all repos in\n>>>> the config.\n>>>> \n>>>> For now, this manual setup does end up a bit cluttered if you have a\n>>>> lot of repos to maintain.\n>>> \n>>> I think it might be useful to have a general command to run a subcommand\n>>> in a bunch of repositories. Something like:\n>>> \n>>> git for-each-repo --recurse /path/to/repos git maintenance ...\n>>> \n>>> which would root around in /path/to/repos for any git-dirs and run \"git\n>>> --git-dir=$GIT_DIR maintenance ...\" on each of them.\n>>> \n>>> And/or:\n>>> \n>>> git for-each-repo --config maintenance.repos git maintenance ...\n>>> \n>>> which would pull the set of repos from the named config variable instead\n>>> of looking around the filesystem.\n>> \n>> Yes! This! That's a good way to make something generic that solves\n>> the problem at hand, but might also have other applications! Most\n>> excellent.\n> \n> I'm glad I wasn't the only one super geeked when I read this idea. I'd\n> use the heck out of this in my .bashrc too. Sounds awesome. I actually\n> had a short-lived fling last year with a script to summarize my\n> uncommitted changes in all repos at the beginning of every session\n> (dropped because it became one more thing to gloss over) and could have\n> really used this command.\n\nI was planning to build a CLI tool that help manage multiple repos maintenance\nlike what was just described here.\nMy experience using my poor-man-scalar [1] bash script is: For multiple repositories,\nthe process count could get out of control quite quickly and there are probably other\nissues that I have not thought of / encountered...\n\nThere is definitely a need to keep all the repos updated with pre-fetch \nand updated commit-graph, while staying compact / garbage free.\nHaving this in Git does simplify a lot of daily operations for end users.\n\n> \n>> \n>>> You could use either as a one-liner in the crontab (depending on which\n>>> is easier with your repo layout).\n>> \n>> The hope is that we can have such a clean layout. I'm particularly\n>> fond of the config option because users may want to opt-in to\n>> background maintenance only on some repos, even if they put them\n>> in a consistent location.\n>> \n>> In the _far_ future, we might even want to add a repo to this\n>> \"maintenance.repos\" list during 'git init' and 'git clone' so\n>> this is automatic. It then becomes opt-out at that point, which\n>> is why I saw the _far, far_ future.\n> \n> Oh, I like this idea a lot. Then I can do something silly like\n> \n> alias reproclone=\"git clone --no-maintainenance\"\n> \n> and get the benefits on everything else that I plan to be using\n> frequently.\n\nThis started to remind me of automatic updates in some of the popular OS.\nWhere download/install/cleanup update of multiple software components are\nmanaged under a single tool.\n\nI wonder if this is the path git should take in the 'new world' that Junio mentioned. [2]\n\nBut I am also super geeked reading this. :)\n\n> \n> - Emily\n\nRegards,\nSon Luong.\n\n[1]: https://github.com/sluongng/git-care\n[2]: https://lore.kernel.org/git/xmqqmu48y7rw.fsf@gitster.c.googlers.com/"},{"id":"401931","messageId":"63ec602a07756a41f8ccddd745562c567a4b3ed7.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:23Z","receivedAt":"2020-07-23T17:56:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'gc' builtin is our current entrypoint for automatically maintaining\na repository. This one tool does many operations, such as repacking the\nrepository, packing refs, and rewriting the commit-graph file. The name\nimplies it performs \"garbage collection\" which means several different\nthings, and some users may not want to use this operation that rewrites\nthe entire object database.\n\nCreate a new 'maintenance' builtin that will become a more general-\npurpose command. To start, it will only support the 'run' subcommand,\nbut will later expand to add subcommands for scheduling maintenance in\nthe background.\n\nFor now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.\nIn fact, the only option is the '--auto' toggle, which is handed\ndirectly to the 'gc' builtin. The current change is isolated to this\nsimple operation to prevent more interesting logic from being lost in\nall of the boilerplate of adding a new builtin.\n\nUse existing builtin/gc.c file because we want to share code between the\ntwo builtins. It is possible that we will have 'maintenance' replace the\n'gc' builtin entirely at some point, leaving 'git gc' as an alias for\nsome specific arguments to 'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-maintenance.txt | 57 +++++++++++++++++++++++++++++\n builtin.h | 1 +\n builtin/gc.c | 59 +++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t7900-maintenance.sh | 22 ++++++++++++\n 6 files changed, 141 insertions(+)\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex ee509a2ad2..a5808fa30d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -90,6 +90,7 @@\n /git-ls-tree\n /git-mailinfo\n /git-mailsplit\n+/git-maintenance\n /git-merge\n /git-merge-base\n /git-merge-index\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nnew file mode 100644\nindex 0000000000..34cd2b4417\n--- /dev/null\n+++ b/Documentation/git-maintenance.txt\n@@ -0,0 +1,57 @@\n+git-maintenance(1)\n+==================\n+\n+NAME\n+----\n+git-maintenance - Run tasks to optimize Git repository data\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git maintenance' run []\n+\n+\n+DESCRIPTION\n+-----------\n+Run tasks to optimize Git repository data, speeding up other Git commands\n+and reducing storage requirements for the repository.\n++\n+Git commands that add repository data, such as `git add` or `git fetch`,\n+are optimized for a responsive user experience. These commands do not take\n+time to optimize the Git data, since such optimizations scale with the full\n+size of the repository while these user commands each perform a relatively\n+small action.\n++\n+The `git maintenance` command provides flexibility for how to optimize the\n+Git repository.\n+\n+SUBCOMMANDS\n+-----------\n+\n+run::\n+\tRun one or more maintenance tasks.\n+\n+TASKS\n+-----\n+\n+gc::\n+\tCleanup unnecessary files and optimize the local repository. \"GC\"\n+\tstands for \"garbage collection,\" but this task performs many\n+\tsmaller tasks. This task can be rather expensive for large\n+\trepositories, as it repacks all Git objects into a single pack-file.\n+\tIt can also be disruptive in some situations, as it deletes stale\n+\tdata.\n+\n+OPTIONS\n+-------\n+--auto::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain thresholds are met. For example, the `gc` task\n+\truns when the number of loose objects exceeds the number stored\n+\tin the `gc.auto` config setting, or when the number of pack-files\n+\texceeds the `gc.autoPackLimit` config setting.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/builtin.h b/builtin.h\nindex a5ae15bfe5..17c1c0ce49 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix);\n int cmd_ls_remote(int argc, const char **argv, const char *prefix);\n int cmd_mailinfo(int argc, const char **argv, const char *prefix);\n int cmd_mailsplit(int argc, const char **argv, const char *prefix);\n+int cmd_maintenance(int argc, const char **argv, const char *prefix);\n int cmd_merge(int argc, const char **argv, const char *prefix);\n int cmd_merge_base(int argc, const char **argv, const char *prefix);\n int cmd_merge_index(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 8e0b9cf41b..8d73c77f3a 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -699,3 +699,62 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \treturn 0;\n }\n+\n+static const char * const builtin_maintenance_usage[] = {\n+\tN_(\"git maintenance run []\"),\n+\tNULL\n+};\n+\n+static struct maintenance_opts {\n+\tint auto_flag;\n+} opts;\n+\n+static int maintenance_task_gc(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"gc\", NULL);\n+\n+\tif (opts.auto_flag)\n+\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n+\n+\tclose_object_store(the_repository->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int maintenance_run(void)\n+{\n+\treturn maintenance_task_gc();\n+}\n+\n+int cmd_maintenance(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_maintenance_options[] = {\n+\t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n+\t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tmemset(&opts, 0, sizeof(opts));\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t\t builtin_maintenance_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_maintenance_options,\n+\t\t\t builtin_maintenance_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (argc == 1) {\n+\t\tif (!strcmp(argv[0], \"run\"))\n+\t\t\treturn maintenance_run();\n+\t}\n+\n+\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t builtin_maintenance_options);\n+}\ndiff --git a/git.c b/git.c\nindex 2f021b97f3..ff56d1df24 100644\n--- a/git.c\n+++ b/git.c\n@@ -527,6 +527,7 @@ static struct cmd_struct commands[] = {\n \t{ \"ls-tree\", cmd_ls_tree, RUN_SETUP },\n \t{ \"mailinfo\", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"mailsplit\", cmd_mailsplit, NO_PARSEOPT },\n+\t{ \"maintenance\", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"merge\", cmd_merge, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"merge-base\", cmd_merge_base, RUN_SETUP },\n \t{ \"merge-file\", cmd_merge_file, RUN_SETUP_GENTLY },\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nnew file mode 100755\nindex 0000000000..d00641c4dd\n--- /dev/null\n+++ b/t/t7900-maintenance.sh\n@@ -0,0 +1,22 @@\n+#!/bin/sh\n+\n+test_description='git maintenance builtin'\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+GIT_TEST_MULTI_PACK_INDEX=0\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'help text' '\n+\ttest_must_fail git maintenance -h 2>err &&\n+\ttest_i18ngrep \"usage: git maintenance run\" err\n+'\n+\n+test_expect_success 'gc [--auto]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"401932","messageId":"1d37e55cb714ea579744d28d1aeb332a63815342.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 02/18] maintenance: add --quiet option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:24Z","receivedAt":"2020-07-23T17:56:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance activities are commonly used as steps in larger scripts.\nProviding a '--quiet' option allows those scripts to be less noisy when\nrun on a terminal window. Turn this mode on by default when stderr is\nnot a terminal.\n\nPipe the option to the 'git gc' child process.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 3 +++\n builtin/gc.c | 7 +++++++\n t/t7900-maintenance.sh | 8 +++++---\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 34cd2b4417..089fa4cedc 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -52,6 +52,9 @@ OPTIONS\n \tin the `gc.auto` config setting, or when the number of pack-files\n \texceeds the `gc.autoPackLimit` config setting.\n \n+--quiet::\n+\tDo not report progress or other information over `stderr`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 8d73c77f3a..c8cde28436 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -707,6 +707,7 @@ static const char * const builtin_maintenance_usage[] = {\n \n static struct maintenance_opts {\n \tint auto_flag;\n+\tint quiet;\n } opts;\n \n static int maintenance_task_gc(void)\n@@ -718,6 +719,8 @@ static int maintenance_task_gc(void)\n \n \tif (opts.auto_flag)\n \t\targv_array_pushl(&cmd, \"--auto\", NULL);\n+\tif (opts.quiet)\n+\t\targv_array_pushl(&cmd, \"--quiet\", NULL);\n \n \tclose_object_store(the_repository->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n@@ -736,6 +739,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstatic struct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n+\t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_END()\n \t};\n \n@@ -745,6 +750,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_maintenance_usage,\n \t\t\t\t builtin_maintenance_options);\n \n+\topts.quiet = !isatty(2);\n+\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\n \t\t\t builtin_maintenance_usage,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex d00641c4dd..e4e4036e50 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -12,11 +12,13 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n-test_expect_success 'gc [--auto]' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+test_expect_success 'gc [--auto|--quiet]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"401933","messageId":"f164d1a0b4363eb6eb174e90e88211829f8a1d55.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:25Z","receivedAt":"2020-07-23T17:56:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe run_auto_gc() method is used in several places to trigger a check\nfor repo maintenance after some Git commands, such as 'git commit' or\n'git fetch'.\n\nTo allow for extra customization of this maintenance activity, replace\nthe 'git gc --auto [--quiet]' call with one to 'git maintenance run\n--auto [--quiet]'. As we extend the maintenance builtin with other\nsteps, users will be able to select different maintenance activities.\n\nRename run_auto_gc() to run_auto_maintenance() to be clearer what is\nhappening on this call, and to expose all callers in the current diff.\n\nSince 'git fetch' already allows disabling the 'git gc --auto'\nsubprocess, add an equivalent option with a different name to be more\ndescriptive of the new behavior: '--[no-]maintenance'. Update the\ndocumentation to include these options at the same time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 5 +++--\n Documentation/git-clone.txt | 7 ++++---\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 6 ++++--\n builtin/merge.c | 2 +-\n builtin/rebase.c | 4 ++--\n run-command.c | 7 +++++--\n run-command.h | 2 +-\n t/t5510-fetch.sh | 2 +-\n 10 files changed, 23 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 6e2a160a47..d73224844e 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -86,9 +86,10 @@ ifndef::git-pull[]\n \tAllow several and arguments to be\n \tspecified. No s may be specified.\n \n+--[no-]maintenance::\n --[no-]auto-gc::\n-\tRun `git gc --auto` at the end to perform garbage collection\n-\tif needed. This is enabled by default.\n+\tRun `git maintenance run --auto` at the end to perform garbage\n+\tcollection if needed. This is enabled by default.\n \n --[no-]write-commit-graph::\n \tWrite a commit-graph after fetching. This overrides the config\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex c898310099..aa25aba7d9 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -78,9 +78,10 @@ repository using this option and then delete branches (or use any\n other Git command that makes any existing commit unreferenced) in the\n source repository, some objects may become unreferenced (or dangling).\n These objects may be removed by normal Git operations (such as `git commit`)\n-which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n-If these objects are removed and were referenced by the cloned repository,\n-then the cloned repository will become corrupt.\n+which automatically call `git maintenance run --auto` and `git gc --auto`.\n+(See linkgit:git-maintenance[1] and linkgit:git-gc[1].) If these objects\n+are removed and were referenced by the cloned repository, then the cloned\n+repository will become corrupt.\n +\n Note that running `git repack` without the `--local` option in a repository\n cloned with `--shared` will copy objects from the source repository into a pack\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 69e50de018..ff895125f6 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume)\n \tif (!state->rebasing) {\n \t\tam_destroy(state);\n \t\tclose_object_store(the_repository->objects);\n-\t\trun_auto_gc(state->quiet);\n+\t\trun_auto_maintenance(state->quiet);\n \t}\n }\n \ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex d1b7396052..658b158659 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tgit_test_write_commit_graph_or_die();\n \n \trepo_rerere(the_repository, 0);\n-\trun_auto_gc(quiet);\n+\trun_auto_maintenance(quiet);\n \trun_commit_hook(use_editor, get_index_file(), \"post-commit\", NULL);\n \tif (amend && !no_post_rewrite) {\n \t\tcommit_post_rewrite(the_repository, current_head, &oid);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82ac4be8a5..49a4d727d4 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n \t\t\tN_(\"report that we have only objects reachable from this object\")),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n-\t\t N_(\"run 'gc --auto' after fetching\")),\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"show-forced-updates\", &fetch_show_forced_updates,\n \t\t N_(\"check for forced-updates on all updated branches\")),\n \tOPT_BOOL(0, \"write-commit-graph\", &fetch_write_commit_graph,\n@@ -1882,7 +1884,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tclose_object_store(the_repository->objects);\n \n \tif (enable_auto_gc)\n-\t\trun_auto_gc(verbosity < 0);\n+\t\trun_auto_maintenance(verbosity < 0);\n \n \treturn result;\n }\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 7da707bf55..c068e73037 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -457,7 +457,7 @@ static void finish(struct commit *head_commit,\n \t\t\t * user should see them.\n \t\t\t */\n \t\t\tclose_object_store(the_repository->objects);\n-\t\t\trun_auto_gc(verbosity < 0);\n+\t\t\trun_auto_maintenance(verbosity < 0);\n \t\t}\n \t}\n \tif (new_head && show_diffstat) {\ndiff --git a/builtin/rebase.c b/builtin/rebase.c\nindex 37ba76ac3d..0c4ee98f08 100644\n--- a/builtin/rebase.c\n+++ b/builtin/rebase.c\n@@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts)\n \tapply_autostash(state_dir_path(\"autostash\", opts));\n \tclose_object_store(the_repository->objects);\n \t/*\n-\t * We ignore errors in 'gc --auto', since the\n+\t * We ignore errors in 'git maintenance run --auto', since the\n \t * user should see them.\n \t */\n-\trun_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n+\trun_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n \tif (opts->type == REBASE_MERGE) {\n \t\tstruct replay_opts replay = REPLAY_OPTS_INIT;\n \ndiff --git a/run-command.c b/run-command.c\nindex 9b3a57d1e3..82ad241638 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1865,14 +1865,17 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \treturn result;\n }\n \n-int run_auto_gc(int quiet)\n+int run_auto_maintenance(int quiet)\n {\n \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n \tint status;\n \n-\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n+\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n \tif (quiet)\n \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n+\telse\n+\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n+\n \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n \targv_array_clear(&argv_gc_auto);\n \treturn status;\ndiff --git a/run-command.h b/run-command.h\nindex 191dfcdafe..d9a800e700 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args);\n /*\n * Trigger an auto-gc\n */\n-int run_auto_gc(int quiet);\n+int run_auto_maintenance(int quiet);\n \n #define RUN_COMMAND_NO_STDIN 1\n #define RUN_GIT_CMD\t 2\t/*If this is to be git sub-command */\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex a66dbe0bde..9850ecde5d 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' '\n \t\tgit config fetch.unpackLimit 1 &&\n \t\tgit config gc.autoPackLimit 1 &&\n \t\tgit config gc.autoDetach false &&\n-\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch >fetch.out 2>&1 &&\n+\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch --verbose >fetch.out 2>&1 &&\n \t\ttest_i18ngrep \"Auto packing the repository\" fetch.out &&\n \t\t! grep \"Should I try again\" fetch.out\n \t)\n-- \ngitgitgadget\n\n"},{"id":"401934","messageId":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.git.1594131695.gitgitgadget@gmail.com","subject":"[PATCH v2 00/18] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:22Z","receivedAt":"2020-07-23T17:56:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a second attempt at redesigning Git's repository maintenance\npatterns. The first attempt [1] included a way to run jobs in the background\nusing a long-lived process; that idea was rejected and is not included in\nthis series. A future series will use the OS to handle scheduling tasks.\n\n[1] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nAs mentioned before, git gc already plays the role of maintaining Git\nrepositories. It has accumulated several smaller pieces in its long history,\nincluding:\n\n 1. Repacking all reachable objects into one pack-file (and deleting\n unreachable objects).\n 2. Packing refs.\n 3. Expiring reflogs.\n 4. Clearing rerere logs.\n 5. Updating the commit-graph file.\n 6. Pruning worktrees.\n\nWhile expiring reflogs, clearing rererelogs, and deleting unreachable\nobjects are suitable under the guise of \"garbage collection\", packing refs\nand updating the commit-graph file are not as obviously fitting. Further,\nthese operations are \"all or nothing\" in that they rewrite almost all\nrepository data, which does not perform well at extremely large scales.\nThese operations can also be disruptive to foreground Git commands when git\ngc --auto triggers during routine use.\n\nThis series does not intend to change what git gc does, but instead create\nnew choices for automatic maintenance activities, of which git gc remains\nthe only one enabled by default.\n\nThe new maintenance tasks are:\n\n * 'commit-graph' : write and verify a single layer of an incremental\n commit-graph.\n * 'loose-objects' : prune packed loose objects, then create a new pack from\n a batch of loose objects.\n * 'pack-files' : expire redundant packs from the multi-pack-index, then\n repack using the multi-pack-index's incremental repack strategy.\n * 'fetch' : fetch from each remote, storing the refs in 'refs/prefetch//'.\n\nThese tasks are all disabled by default, but can be enabled with config\noptions or run explicitly using \"git maintenance run --task=\". There are\nadditional config options to allow customizing the conditions for which the\ntasks run during the '--auto' option. ('fetch' will never run with the\n'--auto' option.)\n\n Because 'gc' is implemented as a maintenance task, the most dramatic change\nof this series is to convert the 'git gc --auto' calls into 'git maintenance\nrun --auto' calls at the end of some Git commands. By default, the only\nchange is that 'git gc --auto' will be run below an additional 'git\nmaintenance' process.\n\nThe 'git maintenance' builtin has a 'run' subcommand so it can be extended\nlater with subcommands that manage background maintenance, such as 'start',\n'stop', 'pause', or 'schedule'. These are not the subject of this series, as\nit is important to focus on the maintenance activities themselves.\n\nAn expert user could set up scheduled background maintenance themselves with\nthe current series. I have the following crontab data set up to run\nmaintenance on an hourly basis:\n\n0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n\nMy config includes all tasks except the 'gc' task. The hourly run is\nover-aggressive, but is sufficient for testing. I'll replace it with daily\nwhen I feel satisfied.\n\nHopefully this direction is seen as a positive one. My goal was to add more\noptions for expert users, along with the flexibility to create background\nmaintenance via the OS in a later series.\n\nOUTLINE\n=======\n\nPatches 1-4 remove some references to the_repository in builtin/gc.c before\nwe start depending on code in that builtin.\n\nPatches 5-7 create the 'git maintenance run' builtin and subcommand as a\nsimple shim over 'git gc' and replaces calls to 'git gc --auto' from other\ncommands.\n\nPatches 8-15 create new maintenance tasks. These are the same tasks sent in\nthe previous RFC.\n\nPatches 16-21 create more customization through config and perform other\npolish items.\n\nFUTURE WORK\n===========\n\n * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n commands run in the background. You can see my progress towards this goal\n here: https://github.com/gitgitgadget/git/pull/680\n \n \n * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n default, but might have different '--auto' conditions and more config\n options.\n \n \n * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n with use of the 'commit-graph' task.\n \n \n * Update the builtin to use struct repository *r properly, especially when\n calling subcommands.\n \n \n\nUPDATES in V2\n=============\n\nI'm sending this between v2.28.0-rc2 adn v2.28.0 as the release things have\nbecome a bit quiet.\n\n * The biggest disruption to the range-diff is that I removed the premature\n use of struct repository *r and instead continue to rely on \n the_repository. This means several patches were dropped that did prep\n work in builtin/gc.c.\n \n \n * I dropped the task hashmap and opted for a linear scan. This task list\n will always be too small to justify the extra complication of the\n hashmap.\n \n \n * struct maintenance_opts is properly static now.\n \n \n * Some tasks are renamed: fetch -> prefetch, pack-files ->\n incremental-repack.\n \n \n * With the rename, the prefetch task uses refs/prefetch/ instead of \n refs/hidden/.\n \n \n * A trace2 region around the task executions are added.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (18):\n maintenance: create basic maintenance runner\n maintenance: add --quiet option\n maintenance: replace run_auto_gc()\n maintenance: initialize task array\n maintenance: add commit-graph task\n maintenance: add --task option\n maintenance: take a lock on the objects directory\n maintenance: add prefetch task\n maintenance: add loose-objects task\n maintenance: add incremental-repack task\n maintenance: auto-size incremental-repack batch\n maintenance: create maintenance..enabled config\n maintenance: use pointers to check --auto\n maintenance: add auto condition for commit-graph task\n maintenance: create auto condition for loose-objects\n maintenance: add incremental-repack auto condition\n midx: use start_delayed_progress()\n maintenance: add trace2 regions for task execution\n\n .gitignore | 1 +\n Documentation/config.txt | 2 +\n Documentation/config/maintenance.txt | 32 ++\n Documentation/fetch-options.txt | 5 +-\n Documentation/git-clone.txt | 7 +-\n Documentation/git-maintenance.txt | 124 +++++\n builtin.h | 1 +\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 6 +-\n builtin/gc.c | 753 +++++++++++++++++++++++++++\n builtin/merge.c | 2 +-\n builtin/rebase.c | 4 +-\n commit-graph.c | 8 +-\n commit-graph.h | 1 +\n git.c | 1 +\n midx.c | 12 +-\n midx.h | 1 +\n object.h | 1 +\n run-command.c | 7 +-\n run-command.h | 2 +-\n t/t5319-multi-pack-index.sh | 14 +-\n t/t5510-fetch.sh | 2 +-\n t/t5514-fetch-multiple.sh | 2 +-\n t/t7900-maintenance.sh | 211 ++++++++\n 25 files changed, 1169 insertions(+), 34 deletions(-)\n create mode 100644 Documentation/config/maintenance.txt\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\n\nbase-commit: 4a0fcf9f760c9774be77f51e1e88a7499b53d2e2\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-671%2Fderrickstolee%2Fmaintenance%2Fgc-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-671/derrickstolee/maintenance/gc-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/671\n\nRange-diff vs v1:\n\n 1: 85dda7db28 < -: ---------- gc: use the_repository less often\n 2: 303ad4bdc7 < -: ---------- gc: use repository in too_many_loose_objects()\n 3: 9dfa9e8f6f < -: ---------- gc: use repo config\n 4: e13d351e9f < -: ---------- gc: drop the_repository in log location\n 5: 5f89e0ec9b ! 1: 63ec602a07 maintenance: create basic maintenance runner\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n +\tNULL\n +};\n +\n -+struct maintenance_opts {\n ++static struct maintenance_opts {\n +\tint auto_flag;\n +} opts;\n +\n -+static int maintenance_task_gc(struct repository *r)\n ++static int maintenance_task_gc(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n +\tif (opts.auto_flag)\n +\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n +\n -+\tclose_object_store(r->objects);\n ++\tclose_object_store(the_repository->objects);\n +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\targv_array_clear(&cmd);\n +\n +\treturn result;\n +}\n +\n -+static int maintenance_run(struct repository *r)\n ++static int maintenance_run(void)\n +{\n -+\treturn maintenance_task_gc(r);\n ++\treturn maintenance_task_gc();\n +}\n +\n +int cmd_maintenance(int argc, const char **argv, const char *prefix)\n +{\n -+\tstruct repository *r = the_repository;\n -+\n +\tstatic struct option builtin_maintenance_options[] = {\n +\t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n +\t\t\t N_(\"run tasks based on the state of the repository\")),\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n +\n +\tif (argc == 1) {\n +\t\tif (!strcmp(argv[0], \"run\"))\n -+\t\t\treturn maintenance_run(r);\n ++\t\t\treturn maintenance_run();\n +\t}\n +\n +\tusage_with_options(builtin_maintenance_usage,\n 6: 018a9331e2 ! 2: 1d37e55cb7 maintenance: add --quiet option\n @@ Documentation/git-maintenance.txt: OPTIONS\n ## builtin/gc.c ##\n @@ builtin/gc.c: static const char * const builtin_maintenance_usage[] = {\n \n - struct maintenance_opts {\n + static struct maintenance_opts {\n \tint auto_flag;\n +\tint quiet;\n } opts;\n \n - static int maintenance_task_gc(struct repository *r)\n -@@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n + static int maintenance_task_gc(void)\n +@@ builtin/gc.c: static int maintenance_task_gc(void)\n \n \tif (opts.auto_flag)\n \t\targv_array_pushl(&cmd, \"--auto\", NULL);\n +\tif (opts.quiet)\n +\t\targv_array_pushl(&cmd, \"--quiet\", NULL);\n \n - \tclose_object_store(r->objects);\n + \tclose_object_store(the_repository->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstatic struct option builtin_maintenance_options[] = {\n 7: 335a8938c6 = 3: f164d1a0b4 maintenance: replace run_auto_gc()\n 8: 5cdd38afa6 ! 4: 8e260bccf1 maintenance: initialize task array and hashmap\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: initialize task array and hashmap\n + maintenance: initialize task array\n \n In anticipation of implementing multiple maintenance tasks inside the\n - 'maintenance' builtin, use a list and hashmap of structs to describe the\n - work to be done.\n + 'maintenance' builtin, use a list of structs to describe the work to be\n + done.\n \n The struct maintenance_task stores the name of the task (as given by a\n future command-line argument) along with a function pointer to its\n @@ Commit message\n contains the \"gc\" task. This task is also the only task enabled by\n default.\n \n - This list is also inserted into a hashmap. This allows command-line\n - arguments to quickly find the tasks by name, not sensitive to case. To\n - ensure this list and hashmap work well together, the list only contains\n - pointers to the struct information. This will allow a sort on the list\n - while preserving the hashmap data.\n -\n Signed-off-by: Derrick Stolee \n \n ## builtin/gc.c ##\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n \tNULL\n -@@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_task_gc(void)\n \treturn result;\n }\n \n -+typedef int maintenance_task_fn(struct repository *r);\n ++typedef int maintenance_task_fn(void);\n +\n +struct maintenance_task {\n -+\tstruct hashmap_entry ent;\n +\tconst char *name;\n +\tmaintenance_task_fn *fn;\n +\tunsigned enabled:1;\n +};\n +\n -+static int task_entry_cmp(const void *unused_cmp_data,\n -+\t\t\t const struct hashmap_entry *eptr,\n -+\t\t\t const struct hashmap_entry *entry_or_key,\n -+\t\t\t const void *keydata)\n -+{\n -+\tconst struct maintenance_task *e1, *e2;\n -+\tconst char *name = keydata;\n -+\n -+\te1 = container_of(eptr, const struct maintenance_task, ent);\n -+\te2 = container_of(entry_or_key, const struct maintenance_task, ent);\n -+\n -+\treturn strcasecmp(e1->name, name ? name : e2->name);\n -+}\n ++static struct maintenance_task *tasks[MAX_NUM_TASKS];\n ++static int num_tasks;\n +\n -+struct maintenance_task *tasks[MAX_NUM_TASKS];\n -+int num_tasks;\n -+struct hashmap task_map;\n -+\n - static int maintenance_run(struct repository *r)\n + static int maintenance_run(void)\n {\n --\treturn maintenance_task_gc(r);\n +-\treturn maintenance_task_gc();\n +\tint i;\n +\tint result = 0;\n +\n +\tfor (i = 0; !result && i < num_tasks; i++) {\n +\t\tif (!tasks[i]->enabled)\n +\t\t\tcontinue;\n -+\t\tresult = tasks[i]->fn(r);\n ++\t\tresult = tasks[i]->fn();\n +\t}\n +\n +\treturn result;\n @@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +\ttasks[num_tasks]->fn = maintenance_task_gc;\n +\ttasks[num_tasks]->enabled = 1;\n +\tnum_tasks++;\n -+\n -+\thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n -+\n -+\tfor (i = 0; i < num_tasks; i++) {\n -+\t\thashmap_entry_init(&tasks[i]->ent,\n -+\t\t\t\t strihash(tasks[i]->name));\n -+\t\thashmap_add(&task_map, &tasks[i]->ent);\n -+\t}\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n 9: c8fbd14d41 ! 5: 04552b1d2e maintenance: add commit-graph task\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n -@@ builtin/gc.c: struct maintenance_opts {\n +@@ builtin/gc.c: static struct maintenance_opts {\n \tint quiet;\n } opts;\n \n -+static int run_write_commit_graph(struct repository *r)\n ++static int run_write_commit_graph(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n +\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"commit-graph\", \"write\",\n -+\t\t\t \"--split\", \"--reachable\",\n -+\t\t\t NULL);\n ++\targv_array_pushl(&cmd, \"commit-graph\", \"write\",\n ++\t\t\t \"--split\", \"--reachable\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_pushl(&cmd, \"--no-progress\", NULL);\n ++\t\targv_array_push(&cmd, \"--no-progress\");\n +\n +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\targv_array_clear(&cmd);\n @@ builtin/gc.c: struct maintenance_opts {\n +\treturn result;\n +}\n +\n -+static int run_verify_commit_graph(struct repository *r)\n ++static int run_verify_commit_graph(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n +\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"commit-graph\", \"verify\",\n ++\targv_array_pushl(&cmd, \"commit-graph\", \"verify\",\n +\t\t\t \"--shallow\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_pushl(&cmd, \"--no-progress\", NULL);\n ++\t\targv_array_push(&cmd, \"--no-progress\");\n +\n +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\targv_array_clear(&cmd);\n @@ builtin/gc.c: struct maintenance_opts {\n +\treturn result;\n +}\n +\n -+static int maintenance_task_commit_graph(struct repository *r)\n ++static int maintenance_task_commit_graph(void)\n +{\n ++\tstruct repository *r = the_repository;\n +\tchar *chain_path;\n +\n +\t/* Skip commit-graph when --auto is specified. */\n @@ builtin/gc.c: struct maintenance_opts {\n +\t\treturn 0;\n +\n +\tclose_object_store(r->objects);\n -+\tif (run_write_commit_graph(r)) {\n ++\tif (run_write_commit_graph()) {\n +\t\terror(_(\"failed to write commit-graph\"));\n +\t\treturn 1;\n +\t}\n +\n -+\tif (!run_verify_commit_graph(r))\n ++\tif (!run_verify_commit_graph())\n +\t\treturn 0;\n +\n +\twarning(_(\"commit-graph verify caught error, rewriting\"));\n @@ builtin/gc.c: struct maintenance_opts {\n +\t}\n +\tfree(chain_path);\n +\n -+\tif (!run_write_commit_graph(r))\n ++\tif (!run_write_commit_graph())\n +\t\treturn 0;\n +\n +\terror(_(\"failed to rewrite commit-graph\"));\n +\treturn 1;\n +}\n +\n - static int maintenance_task_gc(struct repository *r)\n + static int maintenance_task_gc(void)\n {\n \tint result;\n @@ builtin/gc.c: static void initialize_tasks(void)\n + \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\n \tnum_tasks++;\n - \n ++\n +\ttasks[num_tasks]->name = \"commit-graph\";\n +\ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n +\tnum_tasks++;\n -+\n - \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n + }\n \n - \tfor (i = 0; i < num_tasks; i++) {\n + int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n ## commit-graph.c ##\n @@ commit-graph.c: static char *get_split_graph_filename(struct object_directory *odb,\n 10: c081a3bd29 ! 6: a09b1c1687 maintenance: add --task option\n @@ Documentation/git-maintenance.txt: OPTIONS\n \n ## builtin/gc.c ##\n @@ builtin/gc.c: static const char * const builtin_maintenance_usage[] = {\n - struct maintenance_opts {\n + static struct maintenance_opts {\n \tint auto_flag;\n \tint quiet;\n +\tint tasks_selected;\n } opts;\n \n - static int run_write_commit_graph(struct repository *r)\n -@@ builtin/gc.c: struct maintenance_task {\n - \tstruct hashmap_entry ent;\n + static int run_write_commit_graph(void)\n +@@ builtin/gc.c: typedef int maintenance_task_fn(void);\n + struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n -\tunsigned enabled:1;\n @@ builtin/gc.c: struct maintenance_task {\n +\t\t selected:1;\n };\n \n - static int task_entry_cmp(const void *unused_cmp_data,\n -@@ builtin/gc.c: struct maintenance_task *tasks[MAX_NUM_TASKS];\n - int num_tasks;\n - struct hashmap task_map;\n + static struct maintenance_task *tasks[MAX_NUM_TASKS];\n + static int num_tasks;\n \n +static int compare_tasks_by_selection(const void *a_, const void *b_)\n +{\n @@ builtin/gc.c: struct maintenance_task *tasks[MAX_NUM_TASKS];\n +\treturn b->task_order - a->task_order;\n +}\n +\n - static int maintenance_run(struct repository *r)\n + static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n @@ builtin/gc.c: struct maintenance_task *tasks[MAX_NUM_TASKS];\n +\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n +\n - \t\tresult = tasks[i]->fn(r);\n + \t\tresult = tasks[i]->fn();\n \t}\n \n @@ builtin/gc.c: static void initialize_tasks(void)\n - \t}\n + \tnum_tasks++;\n }\n \n +static int task_option_parse(const struct option *opt,\n +\t\t\t const char *arg, int unset)\n +{\n -+\tstruct maintenance_task *task;\n -+\tstruct maintenance_task key;\n ++\tint i;\n ++\tstruct maintenance_task *task = NULL;\n +\n +\tBUG_ON_OPT_NEG(unset);\n +\n @@ builtin/gc.c: static void initialize_tasks(void)\n +\n +\topts.tasks_selected++;\n +\n -+\tkey.name = arg;\n -+\thashmap_entry_init(&key.ent, strihash(key.name));\n -+\n -+\ttask = hashmap_get_entry(&task_map, &key, ent, NULL);\n ++\tfor (i = 0; i < MAX_NUM_TASKS; i++) {\n ++\t\tif (tasks[i] && !strcasecmp(tasks[i]->name, arg)) {\n ++\t\t\ttask = tasks[i];\n ++\t\t\tbreak;\n ++\t\t}\n ++\t}\n +\n +\tif (!task) {\n +\t\terror(_(\"'%s' is not a valid task\"), arg);\n @@ builtin/gc.c: static void initialize_tasks(void)\n +\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n - \tstruct repository *r = the_repository;\n + \tstatic struct option builtin_maintenance_options[] = {\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n 11: fc1fb5f3cc ! 7: e9260a9c3f maintenance: take a lock on the objects directory\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int maintenance_run(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n +\tstruct lock_file lk;\n ++\tstruct repository *r = the_repository;\n +\tchar *lock_path = xstrfmt(\"%s/maintenance\", r->objects->odb->path);\n +\n +\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) {\n @@ builtin/gc.c: static int maintenance_run(struct repository *r)\n \n \tif (opts.tasks_selected)\n \t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n -@@ builtin/gc.c: static int maintenance_run(struct repository *r)\n - \t\tresult = tasks[i]->fn(r);\n +@@ builtin/gc.c: static int maintenance_run(void)\n + \t\tresult = tasks[i]->fn();\n \t}\n \n +\trollback_lock_file(&lk);\n 12: cbaa5ecc4f ! 8: 3165b8916d maintenance: add fetch task\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: add fetch task\n + maintenance: add prefetch task\n \n When working with very large repositories, an incremental 'git fetch'\n command can download a large amount of data. If there are many other\n @@ Commit message\n the background. This can break up a large daily fetch into several\n smaller hourly fetches.\n \n + The task is called \"prefetch\" because it is work done in advance\n + of a foreground fetch to make that 'git fetch' command much faster.\n +\n However, if we simply ran 'git fetch ' in the background,\n then the user running a foregroudn 'git fetch ' would lose\n some important feedback when a new branch appears or an existing\n @@ Commit message\n 2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n \n - 3. By adding a new refspec \"+refs/heads/*:refs/hidden//*\"\n + 3. By adding a new refspec \"+refs/heads/*:refs/prefetch//*\"\n we can ensure that we actually load the new values somewhere in\n our refspace while not updating refs/heads or refs/remotes. By\n storing these refs here, the commit-graph job will update the\n commit-graph with the commits from these hidden refs.\n \n - 4. --prune will delete the refs/hidden/ refs that no\n + 4. --prune will delete the refs/prefetch/ refs that no\n longer appear on the remote.\n \n We've been using this step as a critical background job in Scalar\n @@ Documentation/git-maintenance.txt: since it will not expire `.graph` files that\n `commit-graph-chain` file. They will be deleted by a later run based on\n the expiration delay.\n \n -+fetch::\n -+\tThe `fetch` job updates the object directory with the latest objects\n ++prefetch::\n ++\tThe `fetch` task updates the object directory with the latest objects\n +\tfrom all registered remotes. For each remote, a `git fetch` command\n +\tis run. The refmap is custom to avoid updating local or remote\n +\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n -+\tremote refs are stored in `refs/hidden//`. Also, no tags are\n -+\tupdated.\n ++\tremote refs are stored in `refs/prefetch//`. Also, tags are\n ++\tnot updated.\n ++\n +This means that foreground fetches are still required to update the\n +remote refs, but the users is notified when the branches and tags are\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n -@@ builtin/gc.c: static int maintenance_task_commit_graph(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_task_commit_graph(void)\n \treturn 1;\n }\n \n -+static int fetch_remote(struct repository *r, const char *remote)\n ++static int fetch_remote(const char *remote)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n +\tstruct strbuf refmap = STRBUF_INIT;\n +\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"fetch\", remote, \"--prune\",\n ++\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n +\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n +\n -+\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/hidden/%s/*\", remote);\n ++\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n +\targv_array_push(&cmd, refmap.buf);\n +\n +\tif (opts.quiet)\n @@ builtin/gc.c: static int maintenance_task_commit_graph(struct repository *r)\n +\treturn 0;\n +}\n +\n -+static int maintenance_task_fetch(struct repository *r)\n ++static int maintenance_task_prefetch(void)\n +{\n +\tint result = 0;\n +\tstruct string_list_item *item;\n @@ builtin/gc.c: static int maintenance_task_commit_graph(struct repository *r)\n +\tfor (item = remotes.items;\n +\t item && item < remotes.items + remotes.nr;\n +\t item++)\n -+\t\tfetch_remote(r, item->string);\n ++\t\tfetch_remote(item->string);\n +\n +cleanup:\n +\tstring_list_clear(&remotes, 0);\n +\treturn result;\n +}\n +\n - static int maintenance_task_gc(struct repository *r)\n + static int maintenance_task_gc(void)\n {\n \tint result;\n @@ builtin/gc.c: static void initialize_tasks(void)\n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n \n -+\ttasks[num_tasks]->name = \"fetch\";\n -+\ttasks[num_tasks]->fn = maintenance_task_fetch;\n ++\ttasks[num_tasks]->name = \"prefetch\";\n ++\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n +\tnum_tasks++;\n +\n \ttasks[num_tasks]->name = \"gc\";\n @@ t/t7900-maintenance.sh: test_expect_success 'run --task duplicate' '\n \ttest_i18ngrep \"cannot be selected multiple times\" err\n '\n \n -+test_expect_success 'run --task=fetch with no remotes' '\n -+\tgit maintenance run --task=fetch 2>err &&\n ++test_expect_success 'run --task=prefetch with no remotes' '\n ++\tgit maintenance run --task=prefetch 2>err &&\n +\ttest_must_be_empty err\n +'\n +\n -+test_expect_success 'fetch multiple remotes' '\n ++test_expect_success 'prefetch multiple remotes' '\n +\tgit clone . clone1 &&\n +\tgit clone . clone2 &&\n +\tgit remote add remote1 \"file://$(pwd)/clone1\" &&\n @@ t/t7900-maintenance.sh: test_expect_success 'run --task duplicate' '\n +\tgit -C clone2 switch -c two &&\n +\ttest_commit -C clone1 one &&\n +\ttest_commit -C clone2 two &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/run-fetch.txt\" git maintenance run --task=fetch &&\n -+\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-fetch.txt &&\n -+\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-fetch.txt &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch &&\n ++\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-prefetch.txt &&\n ++\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-prefetch.txt &&\n +\ttest_path_is_missing .git/refs/remotes &&\n -+\ttest_cmp clone1/.git/refs/heads/one .git/refs/hidden/remote1/one &&\n -+\ttest_cmp clone2/.git/refs/heads/two .git/refs/hidden/remote2/two &&\n -+\tgit log hidden/remote1/one &&\n -+\tgit log hidden/remote2/two\n ++\ttest_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&\n ++\ttest_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&\n ++\tgit log prefetch/remote1/one &&\n ++\tgit log prefetch/remote2/two\n +'\n +\n test_done\n 13: 66a1f662ce ! 9: 83648f4865 maintenance: add loose-objects task\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n -@@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_task_gc(void)\n \treturn result;\n }\n \n -+\n -+static int prune_packed(struct repository *r)\n ++static int prune_packed(void)\n +{\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree, \"prune-packed\", NULL);\n ++\targv_array_pushl(&cmd, \"prune-packed\", NULL);\n +\n +\tif (opts.quiet)\n +\t\targv_array_push(&cmd, \"--quiet\");\n @@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +\treturn ++(d->count) > d->batch_size;\n +}\n +\n -+static int pack_loose(struct repository *r)\n ++static int pack_loose(void)\n +{\n ++\tstruct repository *r = the_repository;\n +\tint result = 0;\n +\tstruct write_loose_object_data data;\n +\tstruct strbuf prefix = STRBUF_INIT;\n @@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +\tstrbuf_addstr(&prefix, r->objects->odb->path);\n +\tstrbuf_addstr(&prefix, \"/pack/loose\");\n +\n -+\targv_array_pushl(&pack_proc->args, \"git\", \"-C\", r->worktree,\n -+\t\t\t \"pack-objects\", NULL);\n ++\targv_array_pushl(&pack_proc->args, \"git\", \"pack-objects\", NULL);\n +\tif (opts.quiet)\n +\t\targv_array_push(&pack_proc->args, \"--quiet\");\n +\targv_array_push(&pack_proc->args, prefix.buf);\n @@ builtin/gc.c: static int maintenance_task_gc(struct repository *r)\n +\treturn result;\n +}\n +\n -+static int maintenance_task_loose_objects(struct repository *r)\n ++static int maintenance_task_loose_objects(void)\n +{\n -+\treturn prune_packed(r) || pack_loose(r);\n ++\treturn prune_packed() || pack_loose();\n +}\n +\n - typedef int maintenance_task_fn(struct repository *r);\n + typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n @@ builtin/gc.c: static void initialize_tasks(void)\n - \ttasks[num_tasks]->fn = maintenance_task_fetch;\n + \ttasks[num_tasks]->fn = maintenance_task_prefetch;\n \tnum_tasks++;\n \n +\ttasks[num_tasks]->name = \"loose-objects\";\n @@ builtin/gc.c: static void initialize_tasks(void)\n \ttasks[num_tasks]->enabled = 1;\n \n ## t/t7900-maintenance.sh ##\n -@@ t/t7900-maintenance.sh: test_expect_success 'fetch multiple remotes' '\n - \tgit log hidden/remote2/two\n +@@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n + \tgit log prefetch/remote2/two\n '\n \n +test_expect_success 'loose-objects task' '\n 14: f98790024f ! 10: b6328c2106 maintenance: add pack-files task\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: add pack-files task\n + maintenance: add incremental-repack task\n \n The previous change cleaned up loose objects using the\n 'loose-objects' that can be run safely in the background. Add a\n @@ Commit message\n 2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1\n (midx: implement midx_repack(), 2019-06-10).\n \n - The 'pack-files' job runs the following steps:\n + The 'incremental-repack' task runs the following steps:\n \n 1. 'git multi-pack-index write' creates a multi-pack-index file if\n one did not exist, and otherwise will update the multi-pack-index\n @@ Commit message\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n \n - The next run of the pack-files job will delete these repacked\n - pack-files during the 'expire' step.\n + The next run of the incremental-repack task will delete these\n + repacked pack-files during the 'expire' step.\n \n In this version, the batch size is set to \"0\" which ignores the\n size restrictions when selecting the pack-files. It instead\n @@ Documentation/git-maintenance.txt: loose-objects::\n \tthousand objects to prevent the job from taking too long on a\n \trepository with many loose objects.\n \n -+pack-files::\n -+\tThe `pack-files` job incrementally repacks the object directory\n ++incremental-repack::\n ++\tThe `incremental-repack` job repacks the object directory\n +\tusing the `multi-pack-index` feature. In order to prevent race\n +\tconditions with concurrent Git commands, it follows a two-step\n +\tprocess. First, it deletes any pack-files included in the\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n -@@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n - \treturn prune_packed(r) || pack_loose(r);\n +@@ builtin/gc.c: static int maintenance_task_loose_objects(void)\n + \treturn prune_packed() || pack_loose();\n }\n \n -+static int multi_pack_index_write(struct repository *r)\n ++static int multi_pack_index_write(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"multi-pack-index\", \"write\", NULL);\n ++\targv_array_pushl(&cmd, \"multi-pack-index\", \"write\", NULL);\n +\n +\tif (opts.quiet)\n +\t\targv_array_push(&cmd, \"--no-progress\");\n @@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n +\treturn result;\n +}\n +\n -+static int rewrite_multi_pack_index(struct repository *r)\n ++static int rewrite_multi_pack_index(void)\n +{\n ++\tstruct repository *r = the_repository;\n +\tchar *midx_name = get_midx_filename(r->objects->odb->path);\n +\n +\tunlink(midx_name);\n +\tfree(midx_name);\n +\n -+\tif (multi_pack_index_write(r)) {\n ++\tif (multi_pack_index_write()) {\n +\t\terror(_(\"failed to rewrite multi-pack-index\"));\n +\t\treturn 1;\n +\t}\n @@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n +\treturn 0;\n +}\n +\n -+static int multi_pack_index_verify(struct repository *r)\n ++static int multi_pack_index_verify(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"multi-pack-index\", \"verify\", NULL);\n ++\targv_array_pushl(&cmd, \"multi-pack-index\", \"verify\", NULL);\n +\n +\tif (opts.quiet)\n +\t\targv_array_push(&cmd, \"--no-progress\");\n @@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n +\treturn result;\n +}\n +\n -+static int multi_pack_index_expire(struct repository *r)\n ++static int multi_pack_index_expire(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"multi-pack-index\", \"expire\", NULL);\n ++\targv_array_pushl(&cmd, \"multi-pack-index\", \"expire\", NULL);\n +\n +\tif (opts.quiet)\n +\t\targv_array_push(&cmd, \"--no-progress\");\n +\n -+\tclose_object_store(r->objects);\n ++\tclose_object_store(the_repository->objects);\n +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\targv_array_clear(&cmd);\n +\n +\treturn result;\n +}\n +\n -+static int multi_pack_index_repack(struct repository *r)\n ++static int multi_pack_index_repack(void)\n +{\n +\tint result;\n +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"-C\", r->worktree,\n -+\t\t\t \"multi-pack-index\", \"repack\", NULL);\n ++\targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n +\n +\tif (opts.quiet)\n +\t\targv_array_push(&cmd, \"--no-progress\");\n +\n +\targv_array_push(&cmd, \"--batch-size=0\");\n +\n -+\tclose_object_store(r->objects);\n ++\tclose_object_store(the_repository->objects);\n +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\n -+\tif (result && multi_pack_index_verify(r)) {\n ++\tif (result && multi_pack_index_verify()) {\n +\t\twarning(_(\"multi-pack-index verify failed after repack\"));\n -+\t\tresult = rewrite_multi_pack_index(r);\n ++\t\tresult = rewrite_multi_pack_index();\n +\t}\n +\n +\treturn result;\n +}\n +\n -+static int maintenance_task_pack_files(struct repository *r)\n ++static int maintenance_task_incremental_repack(void)\n +{\n -+\tif (multi_pack_index_write(r)) {\n ++\tif (multi_pack_index_write()) {\n +\t\terror(_(\"failed to write multi-pack-index\"));\n +\t\treturn 1;\n +\t}\n +\n -+\tif (multi_pack_index_verify(r)) {\n ++\tif (multi_pack_index_verify()) {\n +\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n -+\t\treturn rewrite_multi_pack_index(r);\n ++\t\treturn rewrite_multi_pack_index();\n +\t}\n +\n -+\tif (multi_pack_index_expire(r)) {\n ++\tif (multi_pack_index_expire()) {\n +\t\terror(_(\"multi-pack-index expire failed\"));\n +\t\treturn 1;\n +\t}\n +\n -+\tif (multi_pack_index_verify(r)) {\n ++\tif (multi_pack_index_verify()) {\n +\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n -+\t\treturn rewrite_multi_pack_index(r);\n ++\t\treturn rewrite_multi_pack_index();\n +\t}\n +\n -+\tif (multi_pack_index_repack(r)) {\n ++\tif (multi_pack_index_repack()) {\n +\t\terror(_(\"multi-pack-index repack failed\"));\n +\t\treturn 1;\n +\t}\n @@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n +\treturn 0;\n +}\n +\n - typedef int maintenance_task_fn(struct repository *r);\n + typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n @@ builtin/gc.c: static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n \tnum_tasks++;\n \n -+\ttasks[num_tasks]->name = \"pack-files\";\n -+\ttasks[num_tasks]->fn = maintenance_task_pack_files;\n ++\ttasks[num_tasks]->name = \"incremental-repack\";\n ++\ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n +\tnum_tasks++;\n +\n \ttasks[num_tasks]->name = \"gc\";\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n -+test_expect_success 'pack-files task' '\n ++test_expect_success 'incremental-repack task' '\n +\tpackDir=.git/objects/pack &&\n +\tfor i in $(test_seq 1 5)\n +\tdo\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n +\n +\t# the job repacks the two into a new pack, but does not\n +\t# delete the old ones.\n -+\tgit maintenance run --task=pack-files &&\n ++\tgit maintenance run --task=incremental-repack &&\n +\tls $packDir/*.pack >packs-between &&\n +\ttest_line_count = 4 packs-between &&\n +\n +\t# the job deletes the two old packs, and does not write\n +\t# a new one because only one pack remains.\n -+\tgit maintenance run --task=pack-files &&\n ++\tgit maintenance run --task=incremental-repack &&\n +\tls .git/objects/pack/*.pack >packs-after &&\n +\ttest_line_count = 1 packs-after\n +'\n 15: 8be89707d2 ! 11: 478c7f1d0b maintenance: auto-size pack-files batch\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: auto-size pack-files batch\n + maintenance: auto-size incremental-repack batch\n \n - When repacking during the 'pack-files' job, we use the --batch-size\n - option in 'git multi-pack-index repack'. The initial setting used\n - --batch-size=0 to repack everything into a single pack-file. This is not\n - sustaintable for a large repository. The amount of work required is also\n - likely to use too many system resources for a background job.\n + When repacking during the 'incremental-repack' task, we use the\n + --batch-size option in 'git multi-pack-index repack'. The initial setting\n + used --batch-size=0 to repack everything into a single pack-file. This is\n + not sustaintable for a large repository. The amount of work required is\n + also likely to use too many system resources for a background job.\n \n - Update the 'pack-files' maintenance task by dynamically computing a\n + Update the 'incremental-repack' task by dynamically computing a\n --batch-size option based on the current pack-file structure.\n \n The dynamic default size is computed with this idea in mind for a client\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int multi_pack_index_expire(struct repository *r)\n +@@ builtin/gc.c: static int multi_pack_index_expire(void)\n \treturn result;\n }\n \n +#define TWO_GIGABYTES (2147483647)\n +#define UNSET_BATCH_SIZE ((unsigned long)-1)\n +\n -+static off_t get_auto_pack_size(struct repository *r)\n ++static off_t get_auto_pack_size(void)\n +{\n +\t/*\n +\t * The \"auto\" value is special: we optimize for\n @@ builtin/gc.c: static int multi_pack_index_expire(struct repository *r)\n +\toff_t second_largest_size = 0;\n +\toff_t result_size;\n +\tstruct packed_git *p;\n ++\tstruct repository *r = the_repository;\n +\n +\treprepare_packed_git(r);\n +\tfor (p = get_all_packs(r); p; p = p->next) {\n @@ builtin/gc.c: static int multi_pack_index_expire(struct repository *r)\n +\treturn result_size;\n +}\n +\n - static int multi_pack_index_repack(struct repository *r)\n + static int multi_pack_index_repack(void)\n {\n \tint result;\n \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n +\tstruct strbuf batch_arg = STRBUF_INIT;\n +\n - \targv_array_pushl(&cmd, \"-C\", r->worktree,\n - \t\t\t \"multi-pack-index\", \"repack\", NULL);\n + \targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n \n \tif (opts.quiet)\n \t\targv_array_push(&cmd, \"--no-progress\");\n \n -\targv_array_push(&cmd, \"--batch-size=0\");\n +\tstrbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n -+\t\t\t (uintmax_t)get_auto_pack_size(r));\n ++\t\t (uintmax_t)get_auto_pack_size());\n +\targv_array_push(&cmd, batch_arg.buf);\n \n - \tclose_object_store(r->objects);\n + \tclose_object_store(the_repository->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\tstrbuf_release(&batch_arg);\n \n - \tif (result && multi_pack_index_verify(r)) {\n + \tif (result && multi_pack_index_verify()) {\n \t\twarning(_(\"multi-pack-index verify failed after repack\"));\n \n ## t/t7900-maintenance.sh ##\n -@@ t/t7900-maintenance.sh: test_expect_success 'pack-files task' '\n +@@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '\n \ttest_line_count = 4 packs-between &&\n \n \t# the job deletes the two old packs, and does not write\n -\t# a new one because only one pack remains.\n +\t# a new one because the batch size is not high enough to\n +\t# pack the largest pack-file.\n - \tgit maintenance run --task=pack-files &&\n + \tgit maintenance run --task=incremental-repack &&\n \tls .git/objects/pack/*.pack >packs-after &&\n -\ttest_line_count = 1 packs-after\n +\ttest_line_count = 2 packs-after\n 16: 1551aec4fd ! 12: a3c64930a0 maintenance: create maintenance..enabled config\n @@ Documentation/git-maintenance.txt: SUBCOMMANDS\n -----\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int maintenance_run(struct repository *r)\n - \treturn result;\n - }\n - \n --static void initialize_tasks(void)\n -+static void initialize_tasks(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_run(void)\n + static void initialize_tasks(void)\n {\n \tint i;\n +\tstruct strbuf config_name = STRBUF_INIT;\n @@ builtin/gc.c: static int maintenance_run(struct repository *r)\n \n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n @@ builtin/gc.c: static void initialize_tasks(void)\n - \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n - \n - \tfor (i = 0; i < num_tasks; i++) {\n -+\t\tint config_value;\n + \ttasks[num_tasks]->name = \"commit-graph\";\n + \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n + \tnum_tasks++;\n +\n - \t\thashmap_entry_init(&tasks[i]->ent,\n - \t\t\t\t strihash(tasks[i]->name));\n - \t\thashmap_add(&task_map, &tasks[i]->ent);\n ++\tfor (i = 0; i < num_tasks; i++) {\n ++\t\tint config_value;\n +\n +\t\tstrbuf_setlen(&config_name, 0);\n +\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\", tasks[i]->name);\n +\n -+\t\tif (!repo_config_get_bool(r, config_name.buf, &config_value))\n ++\t\tif (!git_config_get_bool(config_name.buf, &config_value))\n +\t\t\ttasks[i]->enabled = config_value;\n - \t}\n ++\t}\n +\n +\tstrbuf_release(&config_name);\n }\n \n static int task_option_parse(const struct option *opt,\n -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\t\t\t builtin_maintenance_options);\n - \n - \topts.quiet = !isatty(2);\n --\tinitialize_tasks();\n -+\tinitialize_tasks(r);\n - \n - \targc = parse_options(argc, argv, prefix,\n - \t\t\t builtin_maintenance_options,\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'run [--auto|--quiet]' '\n 17: 130130b662 ! 13: dbacc2b76c maintenance: use pointers to check --auto\n @@ Commit message\n Signed-off-by: Derrick Stolee \n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int maintenance_task_pack_files(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_task_incremental_repack(void)\n \n - typedef int maintenance_task_fn(struct repository *r);\n + typedef int maintenance_task_fn(void);\n \n +/*\n + * An auto condition function returns 1 if the task should run\n + * and 0 if the task should NOT run. See needs_to_gc() for an\n + * example.\n + */\n -+typedef int maintenance_auto_fn(struct repository *r);\n ++typedef int maintenance_auto_fn(void);\n +\n struct maintenance_task {\n - \tstruct hashmap_entry ent;\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n +\tmaintenance_auto_fn *auto_condition;\n \tint task_order;\n \tunsigned enabled:1,\n \t\t selected:1;\n -@@ builtin/gc.c: static int maintenance_run(struct repository *r)\n +@@ builtin/gc.c: static int maintenance_run(void)\n \t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n \n +\t\tif (opts.auto_flag &&\n +\t\t (!tasks[i]->auto_condition ||\n -+\t\t !tasks[i]->auto_condition(r)))\n ++\t\t !tasks[i]->auto_condition()))\n +\t\t\tcontinue;\n +\n - \t\tresult = tasks[i]->fn(r);\n + \t\tresult = tasks[i]->fn();\n \t}\n \n -@@ builtin/gc.c: static void initialize_tasks(struct repository *r)\n +@@ builtin/gc.c: static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefi\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n -+\tgc_config(r);\n - \tinitialize_tasks(r);\n ++\tgc_config();\n + \tinitialize_tasks();\n \n \targc = parse_options(argc, argv, prefix,\n \n 18: 37fa3f6157 ! 14: 9af2309f08 maintenance: add auto condition for commit-graph task\n @@ builtin/gc.c\n \n #define FAILED_RUN \"failed to run %s\"\n \n -@@ builtin/gc.c: struct maintenance_opts {\n +@@ builtin/gc.c: static struct maintenance_opts {\n \tint tasks_selected;\n } opts;\n \n @@ builtin/gc.c: struct maintenance_opts {\n +\treturn result;\n +}\n +\n -+static int should_write_commit_graph(struct repository *r)\n ++static int should_write_commit_graph(void)\n +{\n +\tint result;\n +\n -+\trepo_config_get_int(r, \"maintenance.commit-graph.auto\",\n -+\t\t\t &limit_commits_not_in_graph);\n ++\tgit_config_get_int(\"maintenance.commit-graph.auto\",\n ++\t\t\t &limit_commits_not_in_graph);\n +\n +\tif (!limit_commits_not_in_graph)\n +\t\treturn 0;\n @@ builtin/gc.c: struct maintenance_opts {\n +\treturn result;\n +}\n +\n - static int run_write_commit_graph(struct repository *r)\n + static int run_write_commit_graph(void)\n {\n \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(struct repository *r)\n +@@ builtin/gc.c: static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"commit-graph\";\n \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n +\ttasks[num_tasks]->auto_condition = should_write_commit_graph;\n \tnum_tasks++;\n \n - \thashmap_init(&task_map, task_entry_cmp, NULL, MAX_NUM_TASKS);\n + \tfor (i = 0; i < num_tasks; i++) {\n \n ## object.h ##\n @@ object.h: struct object_array {\n 19: 4744fdaae9 ! 15: 42e316ca58 maintenance: create auto condition for loose-objects\n @@ builtin/gc.c: struct write_loose_object_data {\n +\treturn 0;\n +}\n +\n -+static int loose_object_auto_condition(struct repository *r)\n ++static int loose_object_auto_condition(void)\n +{\n +\tint count = 0;\n +\n -+\trepo_config_get_int(r, \"maintenance.loose-objects.auto\",\n -+\t\t\t &loose_object_auto_limit);\n ++\tgit_config_get_int(\"maintenance.loose-objects.auto\",\n ++\t\t\t &loose_object_auto_limit);\n +\n +\tif (!loose_object_auto_limit)\n +\t\treturn 0;\n +\tif (loose_object_auto_limit < 0)\n +\t\treturn 1;\n +\n -+\treturn for_each_loose_file_in_objdir(r->objects->odb->path,\n ++\treturn for_each_loose_file_in_objdir(the_repository->objects->odb->path,\n +\t\t\t\t\t loose_object_count,\n +\t\t\t\t\t NULL, NULL, &count);\n +}\n @@ builtin/gc.c: struct write_loose_object_data {\n static int loose_object_exists(const struct object_id *oid,\n \t\t\t const char *path,\n \t\t\t void *data)\n -@@ builtin/gc.c: static void initialize_tasks(struct repository *r)\n +@@ builtin/gc.c: static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"loose-objects\";\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n +\ttasks[num_tasks]->auto_condition = loose_object_auto_condition;\n \tnum_tasks++;\n \n - \ttasks[num_tasks]->name = \"pack-files\";\n + \ttasks[num_tasks]->name = \"incremental-repack\";\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n +\tdone\n +'\n +\n - test_expect_success 'pack-files task' '\n + test_expect_success 'incremental-repack task' '\n \tpackDir=.git/objects/pack &&\n \tfor i in $(test_seq 1 5)\n 20: fbe03b9af9 ! 16: 3d527cb0dd maintenance: add pack-files auto condition\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: add pack-files auto condition\n + maintenance: add incremental-repack auto condition\n \n - The pack-files task updates the multi-pack-index by deleting pack-files\n - that have been replaced with new packs, then repacking a batch of small\n - pack-files into a larger pack-file. This incremental repack is faster\n + The incremental-repack task updates the multi-pack-index by deleting pack-\n + files that have been replaced with new packs, then repacking a batch of\n + small pack-files into a larger pack-file. This incremental repack is faster\n than rewriting all object data, but is slower than some other\n maintenance activities.\n \n - The 'maintenance.pack-files.auto' config option specifies how many\n + The 'maintenance.incremental-repack.auto' config option specifies how many\n pack-files should exist outside of the multi-pack-index before running\n the step. These pack-files could be created by 'git fetch' commands or\n by the loose-objects task. The default value is 10.\n @@ Documentation/config/maintenance.txt: maintenance.loose-objects.auto::\n \tloose objects is at least the value of `maintenance.loose-objects.auto`.\n \tThe default value is 100.\n +\n -+maintenance.pack-files.auto::\n -+\tThis integer config option controls how often the `pack-files` task\n -+\tshould be run as part of `git maintenance run --auto`. If zero, then\n -+\tthe `pack-files` task will not run with the `--auto` option. A\n -+\tnegative value will force the task to run every time. Otherwise, a\n -+\tpositive value implies the command should run when the number of\n -+\tpack-files not in the multi-pack-index is at least the value of\n -+\t`maintenance.pack-files.auto`. The default value is 10.\n ++maintenance.incremental-repack.auto::\n ++\tThis integer config option controls how often the `incremental-repack`\n ++\ttask should be run as part of `git maintenance run --auto`. If zero,\n ++\tthen the `incremental-repack` task will not run with the `--auto`\n ++\toption. A negative value will force the task to run every time.\n ++\tOtherwise, a positive value implies the command should run when the\n ++\tnumber of pack-files not in the multi-pack-index is at least the value\n ++\tof `maintenance.incremental-repack.auto`. The default value is 10.\n \n ## builtin/gc.c ##\n @@\n @@ builtin/gc.c\n \n #define FAILED_RUN \"failed to run %s\"\n \n -@@ builtin/gc.c: static int maintenance_task_loose_objects(struct repository *r)\n - \treturn prune_packed(r) || pack_loose(r);\n +@@ builtin/gc.c: static int maintenance_task_loose_objects(void)\n + \treturn prune_packed() || pack_loose();\n }\n \n -+static int pack_files_auto_condition(struct repository *r)\n ++static int incremental_repack_auto_condition(void)\n +{\n +\tstruct packed_git *p;\n +\tint enabled;\n -+\tint pack_files_auto_limit = 10;\n ++\tint incremental_repack_auto_limit = 10;\n +\tint count = 0;\n +\n -+\tif (repo_config_get_bool(r, \"core.multiPackIndex\", &enabled) ||\n ++\tif (git_config_get_bool(\"core.multiPackIndex\", &enabled) ||\n +\t !enabled)\n +\t\treturn 0;\n +\n -+\trepo_config_get_int(r, \"maintenance.pack-files.auto\",\n -+\t\t\t &pack_files_auto_limit);\n ++\tgit_config_get_int(\"maintenance.incremental-repack.auto\",\n ++\t\t\t &incremental_repack_auto_limit);\n +\n -+\tif (!pack_files_auto_limit)\n ++\tif (!incremental_repack_auto_limit)\n +\t\treturn 0;\n -+\tif (pack_files_auto_limit < 0)\n ++\tif (incremental_repack_auto_limit < 0)\n +\t\treturn 1;\n +\n -+\tfor (p = get_packed_git(r);\n -+\t count < pack_files_auto_limit && p;\n ++\tfor (p = get_packed_git(the_repository);\n ++\t count < incremental_repack_auto_limit && p;\n +\t p = p->next) {\n +\t\tif (!p->multi_pack_index)\n +\t\t\tcount++;\n +\t}\n +\n -+\treturn count >= pack_files_auto_limit;\n ++\treturn count >= incremental_repack_auto_limit;\n +}\n +\n - static int multi_pack_index_write(struct repository *r)\n + static int multi_pack_index_write(void)\n {\n \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(struct repository *r)\n +@@ builtin/gc.c: static void initialize_tasks(void)\n \n - \ttasks[num_tasks]->name = \"pack-files\";\n - \ttasks[num_tasks]->fn = maintenance_task_pack_files;\n -+\ttasks[num_tasks]->auto_condition = pack_files_auto_condition;\n + \ttasks[num_tasks]->name = \"incremental-repack\";\n + \ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n ++\ttasks[num_tasks]->auto_condition = incremental_repack_auto_condition;\n \tnum_tasks++;\n \n \ttasks[num_tasks]->name = \"gc\";\n \n ## t/t7900-maintenance.sh ##\n -@@ t/t7900-maintenance.sh: test_expect_success 'pack-files task' '\n +@@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '\n \ttest_line_count = 2 packs-after\n '\n \n -+test_expect_success 'maintenance.pack-files.auto' '\n ++test_expect_success 'maintenance.incremental-repack.auto' '\n +\tgit repack -adk &&\n +\tgit config core.multiPackIndex true &&\n +\tgit multi-pack-index write &&\n -+\tGIT_TRACE2_EVENT=1 git -c maintenance.pack-files.auto=1 maintenance \\\n -+\t\trun --auto --task=pack-files >out &&\n ++\tGIT_TRACE2_EVENT=1 git -c maintenance.incremental-repack.auto=1 \\\n ++\t\tmaintenance run --auto --task=incremental-repack >out &&\n +\t! grep \"\\\"multi-pack-index\\\"\" out &&\n +\tfor i in 1 2\n +\tdo\n @@ t/t7900-maintenance.sh: test_expect_success 'pack-files task' '\n +\t\t^HEAD~1\n +\t\tEOF\n +\t\tGIT_TRACE2_EVENT=$(pwd)/trace-A-$i git \\\n -+\t\t\t-c maintenance.pack-files.auto=2 \\\n -+\t\t\tmaintenance run --auto --task=pack-files &&\n ++\t\t\t-c maintenance.incremental-repack.auto=2 \\\n ++\t\t\tmaintenance run --auto --task=incremental-repack &&\n +\t\t! grep \"\\\"multi-pack-index\\\"\" trace-A-$i &&\n +\t\ttest_commit B-$i &&\n +\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n @@ t/t7900-maintenance.sh: test_expect_success 'pack-files task' '\n +\t\t^HEAD~1\n +\t\tEOF\n +\t\tGIT_TRACE2_EVENT=$(pwd)/trace-B-$i git \\\n -+\t\t\t-c maintenance.pack-files.auto=2 \\\n -+\t\t\tmaintenance run --auto --task=pack-files >out &&\n ++\t\t\t-c maintenance.incremental-repack.auto=2 \\\n ++\t\t\tmaintenance run --auto --task=incremental-repack >out &&\n +\t\tgrep \"\\\"multi-pack-index\\\"\" trace-B-$i >/dev/null || return 1\n +\tdone\n +'\n 21: f7fdf72e9d = 17: a0f00f8ab8 midx: use start_delayed_progress()\n -: ---------- > 18: f24db7739f maintenance: add trace2 regions for task execution\n\n-- \ngitgitgadget\n"},{"id":"401936","messageId":"8e260bccf1a0b6cd799a6bc78798b31ebed8ad7e.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 04/18] maintenance: initialize task array","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:26Z","receivedAt":"2020-07-23T17:56:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of implementing multiple maintenance tasks inside the\n'maintenance' builtin, use a list of structs to describe the work to be\ndone.\n\nThe struct maintenance_task stores the name of the task (as given by a\nfuture command-line argument) along with a function pointer to its\nimplementation and a boolean for whether the step is enabled.\n\nA list of pointers to these structs are initialized with the full list\nof implemented tasks along with a default order. For now, this list only\ncontains the \"gc\" task. This task is also the only task enabled by\ndefault.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 39 ++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c8cde28436..c28fb0b16d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -700,6 +700,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n+#define MAX_NUM_TASKS 1\n+\n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n \tNULL\n@@ -729,9 +731,43 @@ static int maintenance_task_gc(void)\n \treturn result;\n }\n \n+typedef int maintenance_task_fn(void);\n+\n+struct maintenance_task {\n+\tconst char *name;\n+\tmaintenance_task_fn *fn;\n+\tunsigned enabled:1;\n+};\n+\n+static struct maintenance_task *tasks[MAX_NUM_TASKS];\n+static int num_tasks;\n+\n static int maintenance_run(void)\n {\n-\treturn maintenance_task_gc();\n+\tint i;\n+\tint result = 0;\n+\n+\tfor (i = 0; !result && i < num_tasks; i++) {\n+\t\tif (!tasks[i]->enabled)\n+\t\t\tcontinue;\n+\t\tresult = tasks[i]->fn();\n+\t}\n+\n+\treturn result;\n+}\n+\n+static void initialize_tasks(void)\n+{\n+\tint i;\n+\tnum_tasks = 0;\n+\n+\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n+\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n+\n+\ttasks[num_tasks]->name = \"gc\";\n+\ttasks[num_tasks]->fn = maintenance_task_gc;\n+\ttasks[num_tasks]->enabled = 1;\n+\tnum_tasks++;\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -751,6 +787,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n+\tinitialize_tasks();\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\n-- \ngitgitgadget\n\n"},{"id":"401938","messageId":"04552b1d2ed751a11eb7c50f6898cbc078b552b4.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:27Z","receivedAt":"2020-07-23T17:56:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe first new task in the 'git maintenance' builtin is the\n'commit-graph' job. It is based on the sequence of events in the\n'commit-graph' job in Scalar [1]. This sequence is as follows:\n\n1. git commit-graph write --reachable --split\n2. git commit-graph verify --shallow\n3. If the verify succeeds, stop.\n4. Delete the commit-graph-chain file.\n5. git commit-graph write --reachable --split\n\nBy writing an incremental commit-graph file using the \"--split\"\noption we minimize the disruption from this operation. The default\nbehavior is to merge layers until the new \"top\" layer is less than\nhalf the size of the layer below. This provides quick writes most\nof the time, with the longer writes following a power law\ndistribution.\n\nMost importantly, concurrent Git processes only look at the\ncommit-graph-chain file for a very short amount of time, so they\nwill verly likely not be holding a handle to the file when we try\nto replace it. (This only matters on Windows.)\n\nIf a concurrent process reads the old commit-graph-chain file, but\nour job expires some of the .graph files before they can be read,\nthen those processes will see a warning message (but not fail).\nThis could be avoided by a future update to use the --expire-time\nargument when writing the commit-graph.\n\nBy using 'git commit-graph verify --shallow' we can ensure that\nthe file we just wrote is valid. This is an extra safety precaution\nthat is faster than our 'write' subcommand. In the rare situation\nthat the newest layer of the commit-graph is corrupt, we can \"fix\"\nthe corruption by deleting the commit-graph-chain file and rewrite\nthe full commit-graph as a new one-layer commit graph. This does\nnot completely prevent _that_ file from being corrupt, but it does\nrecompute the commit-graph by parsing commits from the object\ndatabase. In our use of this step in Scalar and VFS for Git, we\nhave only seen this issue arise because our microsoft/git fork\nreverted 43d3561 (\"commit-graph write: don't die if the existing\ngraph is corrupt\" 2019-03-25) for a while to keep commit-graph\nwrites very fast. We dropped the revert when updating to v2.23.0.\nThe verify still has potential for catching corrupt data across\nthe layer boundary: if the new file has commit X with parent Y\nin an old file but the commit ID for Y in the old file had a\nbitswap, then we will notice that in the 'verify' command.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 18 ++++++++\n builtin/gc.c | 74 ++++++++++++++++++++++++++++++-\n commit-graph.c | 8 ++--\n commit-graph.h | 1 +\n t/t7900-maintenance.sh | 2 +-\n 5 files changed, 97 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 089fa4cedc..35b0be7d40 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -35,6 +35,24 @@ run::\n TASKS\n -----\n \n+commit-graph::\n+\tThe `commit-graph` job updates the `commit-graph` files incrementally,\n+\tthen verifies that the written data is correct. If the new layer has an\n+\tissue, then the chain file is removed and the `commit-graph` is\n+\trewritten from scratch.\n++\n+The verification only checks the top layer of the `commit-graph` chain.\n+If the incremental write merged the new commits with at least one\n+existing layer, then there is potential for on-disk corruption being\n+carried forward into the new file. This will be noticed and the new\n+commit-graph file will be clean as Git reparses the commit data from\n+the object database.\n++\n+The incremental write is safe to run alongside concurrent Git processes\n+since it will not expire `.graph` files that were in the previous\n+`commit-graph-chain` file. They will be deleted by a later run based on\n+the expiration delay.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c28fb0b16d..2cd17398ec 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -700,7 +700,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 1\n+#define MAX_NUM_TASKS 2\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -712,6 +712,74 @@ static struct maintenance_opts {\n \tint quiet;\n } opts;\n \n+static int run_write_commit_graph(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"commit-graph\", \"write\",\n+\t\t\t \"--split\", \"--reachable\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int run_verify_commit_graph(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"commit-graph\", \"verify\",\n+\t\t\t \"--shallow\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int maintenance_task_commit_graph(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tchar *chain_path;\n+\n+\t/* Skip commit-graph when --auto is specified. */\n+\tif (opts.auto_flag)\n+\t\treturn 0;\n+\n+\tclose_object_store(r->objects);\n+\tif (run_write_commit_graph()) {\n+\t\terror(_(\"failed to write commit-graph\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (!run_verify_commit_graph())\n+\t\treturn 0;\n+\n+\twarning(_(\"commit-graph verify caught error, rewriting\"));\n+\n+\tchain_path = get_commit_graph_chain_filename(r->objects->odb);\n+\tif (unlink(chain_path)) {\n+\t\tUNLEAK(chain_path);\n+\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n+\t}\n+\tfree(chain_path);\n+\n+\tif (!run_write_commit_graph())\n+\t\treturn 0;\n+\n+\terror(_(\"failed to rewrite commit-graph\"));\n+\treturn 1;\n+}\n+\n static int maintenance_task_gc(void)\n {\n \tint result;\n@@ -768,6 +836,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\n \tnum_tasks++;\n+\n+\ttasks[num_tasks]->name = \"commit-graph\";\n+\ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n+\tnum_tasks++;\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fdd1c4fa7c..57278a9ab5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb,\n \t\t oid_hex);\n }\n \n-static char *get_chain_filename(struct object_directory *odb)\n+char *get_commit_graph_chain_filename(struct object_directory *odb)\n {\n \treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", odb->path);\n }\n@@ -520,7 +520,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \tstruct stat st;\n \tstruct object_id *oids;\n \tint i = 0, valid = 1, count;\n-\tchar *chain_name = get_chain_filename(odb);\n+\tchar *chain_name = get_commit_graph_chain_filename(odb);\n \tFILE *fp;\n \tint stat_res;\n \n@@ -1635,7 +1635,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t}\n \n \tif (ctx->split) {\n-\t\tchar *lock_name = get_chain_filename(ctx->odb);\n+\t\tchar *lock_name = get_commit_graph_chain_filename(ctx->odb);\n \n \t\thold_lock_file_for_update_mode(&lk, lock_name,\n \t\t\t\t\t LOCK_DIE_ON_ERROR, 0444);\n@@ -2012,7 +2012,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time = ctx->split_opts->expire_time;\n \tif (!ctx->split) {\n-\t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n+\t\tchar *chain_file_name = get_commit_graph_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n \t\tfree(chain_file_name);\n \t\tctx->num_commit_graphs_after = 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 28f89cdf3e..3c202748c3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -25,6 +25,7 @@ struct commit;\n struct bloom_filter_settings;\n \n char *get_commit_graph_filename(struct object_directory *odb);\n+char *get_commit_graph_chain_filename(struct object_directory *odb);\n int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n \n /*\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e4e4036e50..216ac0b19e 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -12,7 +12,7 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n-test_expect_success 'gc [--auto|--quiet]' '\n+test_expect_success 'run [--auto|--quiet]' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n-- \ngitgitgadget\n\n"},{"id":"401935","messageId":"e9260a9c3ffa4791f65a6690112788f681c54a16.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 07/18] maintenance: take a lock on the objects directory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:29Z","receivedAt":"2020-07-23T17:56:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPerforming maintenance on a Git repository involves writing data to the\n.git directory, which is not safe to do with multiple writers attempting\nthe same operation. Ensure that only one 'git maintenance' process is\nrunning at a time by holding a file-based lock. Simply the presence of\nthe .git/maintenance.lock file will prevent future maintenance. This\nlock is never committed, since it does not represent meaningful data.\nInstead, it is only a placeholder.\n\nIf the lock file already exists, then fail silently. This will become\nvery important later when we implement the 'fetch' task, as this is our\nstop-gap from creating a recursive process loop between 'git fetch' and\n'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c58dea6fa5..5d99b4b805 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -826,6 +826,25 @@ static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n+\tstruct lock_file lk;\n+\tstruct repository *r = the_repository;\n+\tchar *lock_path = xstrfmt(\"%s/maintenance\", r->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) {\n+\t\t/*\n+\t\t * Another maintenance command is running.\n+\t\t *\n+\t\t * If --auto was provided, then it is likely due to a\n+\t\t * recursive process stack. Do not report an error in\n+\t\t * that case.\n+\t\t */\n+\t\tif (!opts.auto_flag && !opts.quiet)\n+\t\t\terror(_(\"lock file '%s' exists, skipping maintenance\"),\n+\t\t\t lock_path);\n+\t\tfree(lock_path);\n+\t\treturn 0;\n+\t}\n+\tfree(lock_path);\n \n \tif (opts.tasks_selected)\n \t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n@@ -840,6 +859,7 @@ static int maintenance_run(void)\n \t\tresult = tasks[i]->fn();\n \t}\n \n+\trollback_lock_file(&lk);\n \treturn result;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"401937","messageId":"a09b1c16874c0fad6dd0df3ace573a2ffe34519e.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 06/18] maintenance: add --task option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:28Z","receivedAt":"2020-07-23T17:56:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to only run certain maintenance tasks in a certain\norder. Add the --task= option, which allows a user to specify an\nordered list of tasks to run. These cannot be run multiple times,\nhowever.\n\nHere is where our array of maintenance_task pointers becomes critical.\nWe can sort the array of pointers based on the task order, but we do not\nwant to move the struct data itself in order to preserve the hashmap\nreferences. We use the hashmap to match the --task= arguments into\nthe task struct data.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 4 ++\n builtin/gc.c | 64 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 23 +++++++++++\n 3 files changed, 89 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 35b0be7d40..9204762e21 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -73,6 +73,10 @@ OPTIONS\n --quiet::\n \tDo not report progress or other information over `stderr`.\n \n+--task=::\n+\tIf this option is specified one or more times, then only run the\n+\tspecified tasks in the specified order.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 2cd17398ec..c58dea6fa5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -710,6 +710,7 @@ static const char * const builtin_maintenance_usage[] = {\n static struct maintenance_opts {\n \tint auto_flag;\n \tint quiet;\n+\tint tasks_selected;\n } opts;\n \n static int run_write_commit_graph(void)\n@@ -804,20 +805,38 @@ typedef int maintenance_task_fn(void);\n struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n-\tunsigned enabled:1;\n+\tint task_order;\n+\tunsigned enabled:1,\n+\t\t selected:1;\n };\n \n static struct maintenance_task *tasks[MAX_NUM_TASKS];\n static int num_tasks;\n \n+static int compare_tasks_by_selection(const void *a_, const void *b_)\n+{\n+\tconst struct maintenance_task *a, *b;\n+\ta = (const struct maintenance_task *)a_;\n+\tb = (const struct maintenance_task *)b_;\n+\n+\treturn b->task_order - a->task_order;\n+}\n+\n static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n \n+\tif (opts.tasks_selected)\n+\t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n+\n \tfor (i = 0; !result && i < num_tasks; i++) {\n-\t\tif (!tasks[i]->enabled)\n+\t\tif (opts.tasks_selected && !tasks[i]->selected)\n+\t\t\tcontinue;\n+\n+\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n+\n \t\tresult = tasks[i]->fn();\n \t}\n \n@@ -842,6 +861,44 @@ static void initialize_tasks(void)\n \tnum_tasks++;\n }\n \n+static int task_option_parse(const struct option *opt,\n+\t\t\t const char *arg, int unset)\n+{\n+\tint i;\n+\tstruct maintenance_task *task = NULL;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\n+\tif (!arg || !strlen(arg)) {\n+\t\terror(_(\"--task requires a value\"));\n+\t\treturn 1;\n+\t}\n+\n+\topts.tasks_selected++;\n+\n+\tfor (i = 0; i < MAX_NUM_TASKS; i++) {\n+\t\tif (tasks[i] && !strcasecmp(tasks[i]->name, arg)) {\n+\t\t\ttask = tasks[i];\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!task) {\n+\t\terror(_(\"'%s' is not a valid task\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (task->selected) {\n+\t\terror(_(\"task '%s' cannot be selected multiple times\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\ttask->selected = 1;\n+\ttask->task_order = opts.tasks_selected;\n+\n+\treturn 0;\n+}\n+\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_maintenance_options[] = {\n@@ -849,6 +906,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n+\t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n+\t\t\tN_(\"run a specific task\"),\n+\t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 216ac0b19e..c09a9eb90b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -21,4 +21,27 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'run --task=' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" git maintenance run --task=commit-graph --task=gc &&\n+\t! grep \",\\\"gc\\\"\" run-commit-graph.txt &&\n+\tgrep \",\\\"gc\\\"\" run-gc.txt &&\n+\tgrep \",\\\"gc\\\"\" run-both.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-commit-graph.txt &&\n+\t! grep \",\\\"commit-graph\\\",\\\"write\\\"\" run-gc.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-both.txt\n+'\n+\n+test_expect_success 'run --task=bogus' '\n+\ttest_must_fail git maintenance run --task=bogus 2>err &&\n+\ttest_i18ngrep \"is not a valid task\" err\n+'\n+\n+test_expect_success 'run --task duplicate' '\n+\ttest_must_fail git maintenance run --task=gc --task=gc 2>err &&\n+\ttest_i18ngrep \"cannot be selected multiple times\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401939","messageId":"3165b8916d2d80bf72dac6596a42c871ccd4cbe6.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 08/18] maintenance: add prefetch task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:30Z","receivedAt":"2020-07-23T17:56:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen working with very large repositories, an incremental 'git fetch'\ncommand can download a large amount of data. If there are many other\nusers pushing to a common repo, then this data can rival the initial\npack-file size of a 'git clone' of a medium-size repo.\n\nUsers may want to keep the data on their local repos as close as\npossible to the data on the remote repos by fetching periodically in\nthe background. This can break up a large daily fetch into several\nsmaller hourly fetches.\n\nThe task is called \"prefetch\" because it is work done in advance\nof a foreground fetch to make that 'git fetch' command much faster.\n\nHowever, if we simply ran 'git fetch ' in the background,\nthen the user running a foregroudn 'git fetch ' would lose\nsome important feedback when a new branch appears or an existing\nbranch updates. This is especially true if a remote branch is\nforce-updated and this isn't noticed by the user because it occurred\nin the background. Further, the functionality of 'git push\n--force-with-lease' becomes suspect.\n\nWhen running 'git fetch ' in the background, use\nthe following options for careful updating:\n\n1. --no-tags prevents getting a new tag when a user wants to see\n the new tags appear in their foreground fetches.\n\n2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n\n3. By adding a new refspec \"+refs/heads/*:refs/prefetch//*\"\n we can ensure that we actually load the new values somewhere in\n our refspace while not updating refs/heads or refs/remotes. By\n storing these refs here, the commit-graph job will update the\n commit-graph with the commits from these hidden refs.\n\n4. --prune will delete the refs/prefetch/ refs that no\n longer appear on the remote.\n\nWe've been using this step as a critical background job in Scalar\n[1] (and VFS for Git). This solved a pain point that was showing up\nin user reports: fetching was a pain! Users do not like waiting to\ndownload the data that was created while they were away from their\nmachines. After implementing background fetch, the foreground fetch\ncommands sped up significantly because they mostly just update refs\nand download a small amount of new data. The effect is especially\ndramatic when paried with --no-show-forced-udpates (through\nfetch.showForcedUpdates=false).\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 12 ++++++\n builtin/gc.c | 64 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 24 ++++++++++++\n 3 files changed, 99 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 9204762e21..0927643247 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -53,6 +53,18 @@ since it will not expire `.graph` files that were in the previous\n `commit-graph-chain` file. They will be deleted by a later run based on\n the expiration delay.\n \n+prefetch::\n+\tThe `fetch` task updates the object directory with the latest objects\n+\tfrom all registered remotes. For each remote, a `git fetch` command\n+\tis run. The refmap is custom to avoid updating local or remote\n+\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n+\tremote refs are stored in `refs/prefetch//`. Also, tags are\n+\tnot updated.\n++\n+This means that foreground fetches are still required to update the\n+remote refs, but the users is notified when the branches and tags are\n+updated on the remote.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 5d99b4b805..969c127877 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -28,6 +28,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n+#include \"remote.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -700,7 +701,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 2\n+#define MAX_NUM_TASKS 3\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -781,6 +782,63 @@ static int maintenance_task_commit_graph(void)\n \treturn 1;\n }\n \n+static int fetch_remote(const char *remote)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf refmap = STRBUF_INIT;\n+\n+\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n+\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n+\n+\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\targv_array_push(&cmd, refmap.buf);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--quiet\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tstrbuf_release(&refmap);\n+\treturn result;\n+}\n+\n+static int fill_each_remote(struct remote *remote, void *cbdata)\n+{\n+\tstruct string_list *remotes = (struct string_list *)cbdata;\n+\n+\tstring_list_append(remotes, remote->name);\n+\treturn 0;\n+}\n+\n+static int maintenance_task_prefetch(void)\n+{\n+\tint result = 0;\n+\tstruct string_list_item *item;\n+\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n+\n+\tif (for_each_remote(fill_each_remote, &remotes)) {\n+\t\terror(_(\"failed to fill remotes\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * Do not modify the result based on the success of the 'fetch'\n+\t * operation, as a loss of network could cause 'fetch' to fail\n+\t * quickly. We do not want that to stop the rest of our\n+\t * background operations.\n+\t */\n+\tfor (item = remotes.items;\n+\t item && item < remotes.items + remotes.nr;\n+\t item++)\n+\t\tfetch_remote(item->string);\n+\n+cleanup:\n+\tstring_list_clear(&remotes, 0);\n+\treturn result;\n+}\n+\n static int maintenance_task_gc(void)\n {\n \tint result;\n@@ -871,6 +929,10 @@ static void initialize_tasks(void)\n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n \n+\ttasks[num_tasks]->name = \"prefetch\";\n+\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex c09a9eb90b..8b04a04c79 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -44,4 +44,28 @@ test_expect_success 'run --task duplicate' '\n \ttest_i18ngrep \"cannot be selected multiple times\" err\n '\n \n+test_expect_success 'run --task=prefetch with no remotes' '\n+\tgit maintenance run --task=prefetch 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_expect_success 'prefetch multiple remotes' '\n+\tgit clone . clone1 &&\n+\tgit clone . clone2 &&\n+\tgit remote add remote1 \"file://$(pwd)/clone1\" &&\n+\tgit remote add remote2 \"file://$(pwd)/clone2\" &&\n+\tgit -C clone1 switch -c one &&\n+\tgit -C clone2 switch -c two &&\n+\ttest_commit -C clone1 one &&\n+\ttest_commit -C clone2 two &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch &&\n+\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-prefetch.txt &&\n+\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-prefetch.txt &&\n+\ttest_path_is_missing .git/refs/remotes &&\n+\ttest_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&\n+\ttest_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&\n+\tgit log prefetch/remote1/one &&\n+\tgit log prefetch/remote2/two\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401940","messageId":"83648f48655ba68126110018d81c1d2e2bcc7a6f.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:31Z","receivedAt":"2020-07-23T17:56:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOne goal of background maintenance jobs is to allow a user to\ndisable auto-gc (gc.auto=0) but keep their repository in a clean\nstate. Without any cleanup, loose objects will clutter the object\ndatabase and slow operations. In addition, the loose objects will\ntake up extra space because they are not stored with deltas against\nsimilar objects.\n\nCreate a 'loose-objects' task for the 'git maintenance run' command.\nThis helps clean up loose objects without disrupting concurrent Git\ncommands using the following sequence of events:\n\n1. Run 'git prune-packed' to delete any loose objects that exist\n in a pack-file. Concurrent commands will prefer the packed\n version of the object to the loose version. (Of course, there\n are exceptions for commands that specifically care about the\n location of an object. These are rare for a user to run on\n purpose, and we hope a user that has selected background\n maintenance will not be trying to do foreground maintenance.)\n\n2. Run 'git pack-objects' on a batch of loose objects. These\n objects are grouped by scanning the loose object directories in\n lexicographic order until listing all loose objects -or-\n reaching 50,000 objects. This is more than enough if the loose\n objects are created only by a user doing normal development.\n We noticed users with _millions_ of loose objects because VFS\n for Git downloads blobs on-demand when a file read operation\n requires populating a virtual file. This has potential of\n happening in partial clones if someone runs 'git grep' or\n otherwise evades the batch-download feature for requesting\n promisor objects.\n\nThis step is based on a similar step in Scalar [1] and VFS for Git.\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/LooseObjectsStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 ++++\n builtin/gc.c | 106 +++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 35 ++++++++++\n 3 files changed, 151 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 0927643247..557915a653 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -73,6 +73,17 @@ gc::\n \tIt can also be disruptive in some situations, as it deletes stale\n \tdata.\n \n+loose-objects::\n+\tThe `loose-objects` job cleans up loose objects and places them into\n+\tpack-files. In order to prevent race conditions with concurrent Git\n+\tcommands, it follows a two-step process. First, it deletes any loose\n+\tobjects that already exist in a pack-file; concurrent Git processes\n+\twill examine the pack-file for the object data instead of the loose\n+\tobject. Second, it creates a new pack-file (starting with \"loose-\")\n+\tcontaining a batch of loose objects. The batch size is limited to 50\n+\tthousand objects to prevent the job from taking too long on a\n+\trepository with many loose objects.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 969c127877..fa65304580 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -701,7 +701,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 3\n+#define MAX_NUM_TASKS 4\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -858,6 +858,106 @@ static int maintenance_task_gc(void)\n \treturn result;\n }\n \n+static int prune_packed(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"prune-packed\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--quiet\");\n+\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+struct write_loose_object_data {\n+\tFILE *in;\n+\tint count;\n+\tint batch_size;\n+};\n+\n+static int loose_object_exists(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\treturn 1;\n+}\n+\n+static int write_loose_object_to_stdin(const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_loose_object_data *d = (struct write_loose_object_data *)data;\n+\n+\tfprintf(d->in, \"%s\\n\", oid_to_hex(oid));\n+\n+\treturn ++(d->count) > d->batch_size;\n+}\n+\n+static int pack_loose(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tint result = 0;\n+\tstruct write_loose_object_data data;\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tstruct child_process *pack_proc;\n+\n+\t/*\n+\t * Do not start pack-objects process\n+\t * if there are no loose objects.\n+\t */\n+\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t\t loose_object_exists,\n+\t\t\t\t\t NULL, NULL, NULL))\n+\t\treturn 0;\n+\n+\tpack_proc = xmalloc(sizeof(*pack_proc));\n+\n+\tchild_process_init(pack_proc);\n+\n+\tstrbuf_addstr(&prefix, r->objects->odb->path);\n+\tstrbuf_addstr(&prefix, \"/pack/loose\");\n+\n+\targv_array_pushl(&pack_proc->args, \"git\", \"pack-objects\", NULL);\n+\tif (opts.quiet)\n+\t\targv_array_push(&pack_proc->args, \"--quiet\");\n+\targv_array_push(&pack_proc->args, prefix.buf);\n+\n+\tpack_proc->in = -1;\n+\n+\tif (start_command(pack_proc)) {\n+\t\terror(_(\"failed to start 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tdata.in = xfdopen(pack_proc->in, \"w\");\n+\tdata.count = 0;\n+\tdata.batch_size = 50000;\n+\n+\tfor_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t write_loose_object_to_stdin,\n+\t\t\t\t NULL,\n+\t\t\t\t NULL,\n+\t\t\t\t &data);\n+\n+\tfclose(data.in);\n+\n+\tif (finish_command(pack_proc)) {\n+\t\terror(_(\"failed to finish 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&prefix);\n+\tfree(pack_proc);\n+\treturn result;\n+}\n+\n+static int maintenance_task_loose_objects(void)\n+{\n+\treturn prune_packed() || pack_loose();\n+}\n+\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n@@ -933,6 +1033,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_prefetch;\n \tnum_tasks++;\n \n+\ttasks[num_tasks]->name = \"loose-objects\";\n+\ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8b04a04c79..94bb493733 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -68,4 +68,39 @@ test_expect_success 'prefetch multiple remotes' '\n \tgit log prefetch/remote2/two\n '\n \n+test_expect_success 'loose-objects task' '\n+\t# Repack everything so we know the state of the object dir\n+\tgit repack -adk &&\n+\n+\t# Hack to stop maintenance from running during \"git commit\"\n+\techo in use >.git/objects/maintenance.lock &&\n+\ttest_commit create-loose-object &&\n+\trm .git/objects/maintenance.lock &&\n+\n+\tls .git/objects >obj-dir-before &&\n+\ttest_file_not_empty obj-dir-before &&\n+\tls .git/objects/pack/*.pack >packs-before &&\n+\ttest_line_count = 1 packs-before &&\n+\n+\t# The first run creates a pack-file\n+\t# but does not delete loose objects.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-between &&\n+\ttest_cmp obj-dir-before obj-dir-between &&\n+\tls .git/objects/pack/*.pack >packs-between &&\n+\ttest_line_count = 2 packs-between &&\n+\n+\t# The second run deletes loose objects\n+\t# but does not create a pack-file.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-after &&\n+\tcat >expect <<-\\EOF &&\n+\tinfo\n+\tpack\n+\tEOF\n+\ttest_cmp expect obj-dir-after &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_cmp packs-between packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401941","messageId":"b6328c210625e1ba98e2065208a2a478c2c64f94.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 10/18] maintenance: add incremental-repack task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:32Z","receivedAt":"2020-07-23T17:56:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change cleaned up loose objects using the\n'loose-objects' that can be run safely in the background. Add a\nsimilar job that performs similar cleanups for pack-files.\n\nOne issue with running 'git repack' is that it is designed to\nrepack all pack-files into a single pack-file. While this is the\nmost space-efficient way to store object data, it is not time or\nmemory efficient. This becomes extremely important if the repo is\nso large that a user struggles to store two copies of the pack on\ntheir disk.\n\nInstead, perform an \"incremental\" repack by collecting a few small\npack-files into a new pack-file. The multi-pack-index facilitates\nthis process ever since 'git multi-pack-index expire' was added in\n19575c7 (multi-pack-index: implement 'expire' subcommand,\n2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1\n(midx: implement midx_repack(), 2019-06-10).\n\nThe 'incremental-repack' task runs the following steps:\n\n1. 'git multi-pack-index write' creates a multi-pack-index file if\n one did not exist, and otherwise will update the multi-pack-index\n with any new pack-files that appeared since the last write. This\n is particularly relevant with the background fetch job.\n\n When the multi-pack-index sees two copies of the same object, it\n stores the offset data into the newer pack-file. This means that\n some old pack-files could become \"unreferenced\" which I will use\n to mean \"a pack-file that is in the pack-file list of the\n multi-pack-index but none of the objects in the multi-pack-index\n reference a location inside that pack-file.\"\n\n2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n and updaes the multi-pack-index to drop those pack-files from the\n list. This is safe to do as concurrent Git processes will see the\n multi-pack-index and not open those packs when looking for object\n contents. (Similar to the 'loose-objects' job, there are some Git\n commands that open pack-files regardless of the multi-pack-index,\n but they are rarely used. Further, a user that self-selects to\n use background operations would likely refrain from using those\n commands.)\n\n3. 'git multi-pack-index repack --bacth-size=' collects a set\n of pack-files that are listed in the multi-pack-index and creates\n a new pack-file containing the objects whose offsets are listed\n by the multi-pack-index to be in those objects. The set of pack-\n files is selected greedily by sorting the pack-files by modified\n time and adding a pack-file to the set if its \"expected size\" is\n smaller than the batch size until the total expected size of the\n selected pack-files is at least the batch size. The \"expected\n size\" is calculated by taking the size of the pack-file divided\n by the number of objects in the pack-file and multiplied by the\n number of objects from the multi-pack-index with offset in that\n pack-file. The expected size approximats how much data from that\n pack-file will contribute to the resulting pack-file size. The\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n\n The next run of the incremental-repack task will delete these\n repacked pack-files during the 'expire' step.\n\n In this version, the batch size is set to \"0\" which ignores the\n size restrictions when selecting the pack-files. It instead\n selects all pack-files and repacks all packed objects into a\n single pack-file. This will be updated in the next change, but\n it requires doing some calculations that are better isolated to\n a separate change.\n\nEach of the above steps update the multi-pack-index file. After\neach step, we verify the new multi-pack-index. If the new\nmulti-pack-index is corrupt, then delete the multi-pack-index,\nrewrite it from scratch, and stop doing the later steps of the\njob. This is intended to be an extra-safe check without leaving\na repo with many pack-files without a multi-pack-index.\n\nThese steps are based on a similar background maintenance step in\nScalar (and VFS for Git) [1]. This was incredibly effective for\nusers of the Windows OS repository. After using the same VFS for Git\nrepository for over a year, some users had _thousands_ of pack-files\nthat combined to up to 250 GB of data. We noticed a few users were\nrunning into the open file descriptor limits (due in part to a bug\nin the multi-pack-index fixed by af96fe3 (midx: add packs to\npacked_git linked list, 2019-04-29).\n\nThese pack-files were mostly small since they contained the commits\nand trees that were pushed to the origin in a given hour. The GVFS\nprotocol includes a \"prefetch\" step that asks for pre-computed pack-\nfiles containing commits and trees by timestamp. These pack-files\nwere grouped into \"daily\" pack-files once a day for up to 30 days.\nIf a user did not request prefetch packs for over 30 days, then they\nwould get the entire history of commits and trees in a new, large\npack-file. This led to a large number of pack-files that had poor\ndelta compression.\n\nBy running this pack-file maintenance step once per day, these repos\nwith thousands of packs spanning 200+ GB dropped to dozens of pack-\nfiles spanning 30-50 GB. This was done all without removing objects\nfrom the system and using a constant batch size of two gigabytes.\nOnce the work was done to reduce the pack-files to small sizes, the\nbatch size of two gigabytes means that not every run triggers a\nrepack operation, so the following run will not expire a pack-file.\nThis has kept these repos in a \"clean\" state.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 ++++\n builtin/gc.c | 121 +++++++++++++++++++++++++++++-\n midx.c | 2 +-\n midx.h | 1 +\n t/t7900-maintenance.sh | 37 +++++++++\n 5 files changed, 174 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 557915a653..bda8df4aaa 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -84,6 +84,21 @@ loose-objects::\n \tthousand objects to prevent the job from taking too long on a\n \trepository with many loose objects.\n \n+incremental-repack::\n+\tThe `incremental-repack` job repacks the object directory\n+\tusing the `multi-pack-index` feature. In order to prevent race\n+\tconditions with concurrent Git commands, it follows a two-step\n+\tprocess. First, it deletes any pack-files included in the\n+\t`multi-pack-index` where none of the objects in the\n+\t`multi-pack-index` reference those pack-files; this only happens\n+\tif all objects in the pack-file are also stored in a newer\n+\tpack-file. Second, it selects a group of pack-files whose \"expected\n+\tsize\" is below the batch size until the group has total expected\n+\tsize at least the batch size; see the `--batch-size` option for\n+\tthe `repack` subcommand in linkgit:git-multi-pack-index[1]. The\n+\tdefault batch-size is zero, which is a special case that attempts\n+\tto repack all pack-files into a single pack-file.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fa65304580..eb4b01c104 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -29,6 +29,7 @@\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n+#include \"midx.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -701,7 +702,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-#define MAX_NUM_TASKS 4\n+#define MAX_NUM_TASKS 5\n \n static const char * const builtin_maintenance_usage[] = {\n \tN_(\"git maintenance run []\"),\n@@ -958,6 +959,120 @@ static int maintenance_task_loose_objects(void)\n \treturn prune_packed() || pack_loose();\n }\n \n+static int multi_pack_index_write(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"write\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int rewrite_multi_pack_index(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tchar *midx_name = get_midx_filename(r->objects->odb->path);\n+\n+\tunlink(midx_name);\n+\tfree(midx_name);\n+\n+\tif (multi_pack_index_write()) {\n+\t\terror(_(\"failed to rewrite multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int multi_pack_index_verify(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"verify\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int multi_pack_index_expire(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"expire\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\tclose_object_store(the_repository->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\targv_array_clear(&cmd);\n+\n+\treturn result;\n+}\n+\n+static int multi_pack_index_repack(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n+\n+\tif (opts.quiet)\n+\t\targv_array_push(&cmd, \"--no-progress\");\n+\n+\targv_array_push(&cmd, \"--batch-size=0\");\n+\n+\tclose_object_store(the_repository->objects);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tif (result && multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after repack\"));\n+\t\tresult = rewrite_multi_pack_index();\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int maintenance_task_incremental_repack(void)\n+{\n+\tif (multi_pack_index_write()) {\n+\t\terror(_(\"failed to write multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n+\t\treturn rewrite_multi_pack_index();\n+\t}\n+\n+\tif (multi_pack_index_expire()) {\n+\t\terror(_(\"multi-pack-index expire failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n+\t\treturn rewrite_multi_pack_index();\n+\t}\n+\n+\tif (multi_pack_index_repack()) {\n+\t\terror(_(\"multi-pack-index repack failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n@@ -1037,6 +1152,10 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n \tnum_tasks++;\n \n+\ttasks[num_tasks]->name = \"incremental-repack\";\n+\ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n+\tnum_tasks++;\n+\n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n \ttasks[num_tasks]->enabled = 1;\ndiff --git a/midx.c b/midx.c\nindex 6d1584ca51..57a8a00082 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -36,7 +36,7 @@\n \n #define PACK_EXPIRED UINT_MAX\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..baeecc70c9 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -37,6 +37,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_filename(const char *object_dir);\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 94bb493733..3ec813979a 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -103,4 +103,41 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'incremental-repack task' '\n+\tpackDir=.git/objects/pack &&\n+\tfor i in $(test_seq 1 5)\n+\tdo\n+\t\ttest_commit $i || return 1\n+\tdone &&\n+\n+\t# Create three disjoint pack-files with size BIG, small, small.\n+\techo HEAD~2 | git pack-objects --revs $packDir/test-1 &&\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-2 <<-\\EOF &&\n+\tHEAD~1\n+\t^HEAD~2\n+\tEOF\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-3 <<-\\EOF &&\n+\tHEAD\n+\t^HEAD~1\n+\tEOF\n+\trm -f $packDir/pack-* &&\n+\trm -f $packDir/loose-* &&\n+\tls $packDir/*.pack >packs-before &&\n+\ttest_line_count = 3 packs-before &&\n+\n+\t# the job repacks the two into a new pack, but does not\n+\t# delete the old ones.\n+\tgit maintenance run --task=incremental-repack &&\n+\tls $packDir/*.pack >packs-between &&\n+\ttest_line_count = 4 packs-between &&\n+\n+\t# the job deletes the two old packs, and does not write\n+\t# a new one because only one pack remains.\n+\tgit maintenance run --task=incremental-repack &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_line_count = 1 packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401942","messageId":"478c7f1d0b858755c2c4b98605405214910b6f4c.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:33Z","receivedAt":"2020-07-23T17:57:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen repacking during the 'incremental-repack' task, we use the\n--batch-size option in 'git multi-pack-index repack'. The initial setting\nused --batch-size=0 to repack everything into a single pack-file. This is\nnot sustaintable for a large repository. The amount of work required is\nalso likely to use too many system resources for a background job.\n\nUpdate the 'incremental-repack' task by dynamically computing a\n--batch-size option based on the current pack-file structure.\n\nThe dynamic default size is computed with this idea in mind for a client\nrepository that was cloned from a very large remote: there is likely one\n\"big\" pack-file that was created at clone time. Thus, do not try\nrepacking it as it is likely packed efficiently by the server.\n\nInstead, we select the second-largest pack-file, and create a batch size\nthat is one larger than that pack-file. If there are three or more\npack-files, then this guarantees that at least two will be combined into\na new pack-file.\n\nOf course, this means that the second-largest pack-file size is likely\nto grow over time and may eventually surpass the initially-cloned\npack-file. Recall that the pack-file batch is selected in a greedy\nmanner: the packs are considered from oldest to newest and are selected\nif they have size smaller than the batch size until the total selected\nsize is larger than the batch size. Thus, that oldest \"clone\" pack will\nbe first to repack after the new data creates a pack larger than that.\n\nWe also want to place some limits on how large these pack-files become,\nin order to bound the amount of time spent repacking. A maximum\nbatch-size of two gigabytes means that large repositories will never be\npacked into a single pack-file using this job, but also that repack is\nrather expensive. This is a trade-off that is valuable to have if the\nmaintenance is being run automatically or in the background. Users who\ntruly want to optimize for space and performance (and are willing to pay\nthe upfront cost of a full repack) can use the 'gc' task to do so.\n\nReported-by: Son Luong Ngoc \nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 48 +++++++++++++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 5 +++--\n 2 files changed, 50 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex eb4b01c104..889d97afe7 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n \treturn result;\n }\n \n+#define TWO_GIGABYTES (2147483647)\n+#define UNSET_BATCH_SIZE ((unsigned long)-1)\n+\n+static off_t get_auto_pack_size(void)\n+{\n+\t/*\n+\t * The \"auto\" value is special: we optimize for\n+\t * one large pack-file (i.e. from a clone) and\n+\t * expect the rest to be small and they can be\n+\t * repacked quickly.\n+\t *\n+\t * The strategy we select here is to select a\n+\t * size that is one more than the second largest\n+\t * pack-file. This ensures that we will repack\n+\t * at least two packs if there are three or more\n+\t * packs.\n+\t */\n+\toff_t max_size = 0;\n+\toff_t second_largest_size = 0;\n+\toff_t result_size;\n+\tstruct packed_git *p;\n+\tstruct repository *r = the_repository;\n+\n+\treprepare_packed_git(r);\n+\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tif (p->pack_size > max_size) {\n+\t\t\tsecond_largest_size = max_size;\n+\t\t\tmax_size = p->pack_size;\n+\t\t} else if (p->pack_size > second_largest_size)\n+\t\t\tsecond_largest_size = p->pack_size;\n+\t}\n+\n+\tresult_size = second_largest_size + 1;\n+\n+\t/* But limit ourselves to a batch size of 2g */\n+\tif (result_size > TWO_GIGABYTES)\n+\t\tresult_size = TWO_GIGABYTES;\n+\n+\treturn result_size;\n+}\n+\n static int multi_pack_index_repack(void)\n {\n \tint result;\n \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf batch_arg = STRBUF_INIT;\n+\n \targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n \n \tif (opts.quiet)\n \t\targv_array_push(&cmd, \"--no-progress\");\n \n-\targv_array_push(&cmd, \"--batch-size=0\");\n+\tstrbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n+\t\t (uintmax_t)get_auto_pack_size());\n+\targv_array_push(&cmd, batch_arg.buf);\n \n \tclose_object_store(the_repository->objects);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\tstrbuf_release(&batch_arg);\n \n \tif (result && multi_pack_index_verify()) {\n \t\twarning(_(\"multi-pack-index verify failed after repack\"));\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3ec813979a..ab5c961eb9 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -134,10 +134,11 @@ test_expect_success 'incremental-repack task' '\n \ttest_line_count = 4 packs-between &&\n \n \t# the job deletes the two old packs, and does not write\n-\t# a new one because only one pack remains.\n+\t# a new one because the batch size is not high enough to\n+\t# pack the largest pack-file.\n \tgit maintenance run --task=incremental-repack &&\n \tls .git/objects/pack/*.pack >packs-after &&\n-\ttest_line_count = 1 packs-after\n+\ttest_line_count = 2 packs-after\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"401944","messageId":"dbacc2b76c79bfc874484bc2d1c11084569fd872.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 13/18] maintenance: use pointers to check --auto","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:35Z","receivedAt":"2020-07-23T17:57:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' command has an '--auto' option. This is used\nby other Git commands such as 'git commit' or 'git fetch' to check if\nmaintenance should be run after adding data to the repository.\n\nPreviously, this --auto option was only used to add the argument to the\n'git gc' command as part of the 'gc' task. We will be expanding the\nother tasks to perform a check to see if they should do work as part of\nthe --auto flag, when they are enabled by config.\n\nFirst, update the 'gc' task to perform the auto check inside the\nmaintenance process. This prevents running an extra 'git gc --auto'\ncommand when not needed. It also shows a model for other tasks.\n\nSecond, use the 'auto_condition' function pointer as a signal for\nwhether we enable the maintenance task under '--auto'. For instance, we\ndo not want to enable the 'fetch' task in '--auto' mode, so that\nfunction pointer will remain NULL.\n\nNow that we are not automatically calling 'git gc', a test in\nt5514-fetch-multiple.sh must be changed to watch for 'git maintenance'\ninstead.\n\nWe continue to pass the '--auto' option to the 'git gc' command when\nnecessary, because of the gc.autoDetach config option changes behavior.\nLikely, we will want to absorb the daemonizing behavior implied by\ngc.autoDetach as a maintenance.autoDetach config option.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 15 +++++++++++++++\n t/t5514-fetch-multiple.sh | 2 +-\n t/t7900-maintenance.sh | 2 +-\n 3 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b6dc4b1832..31696a2595 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1121,9 +1121,17 @@ static int maintenance_task_incremental_repack(void)\n \n typedef int maintenance_task_fn(void);\n \n+/*\n+ * An auto condition function returns 1 if the task should run\n+ * and 0 if the task should NOT run. See needs_to_gc() for an\n+ * example.\n+ */\n+typedef int maintenance_auto_fn(void);\n+\n struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n+\tmaintenance_auto_fn *auto_condition;\n \tint task_order;\n \tunsigned enabled:1,\n \t\t selected:1;\n@@ -1175,6 +1183,11 @@ static int maintenance_run(void)\n \t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n \t\t\tcontinue;\n \n+\t\tif (opts.auto_flag &&\n+\t\t (!tasks[i]->auto_condition ||\n+\t\t !tasks[i]->auto_condition()))\n+\t\t\tcontinue;\n+\n \t\tresult = tasks[i]->fn();\n \t}\n \n@@ -1205,6 +1218,7 @@ static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"gc\";\n \ttasks[num_tasks]->fn = maintenance_task_gc;\n+\ttasks[num_tasks]->auto_condition = need_to_gc;\n \ttasks[num_tasks]->enabled = 1;\n \tnum_tasks++;\n \n@@ -1283,6 +1297,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n+\tgc_config();\n \tinitialize_tasks();\n \n \targc = parse_options(argc, argv, prefix,\ndiff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh\nindex de8e2f1531..bd202ec6f3 100755\n--- a/t/t5514-fetch-multiple.sh\n+++ b/t/t5514-fetch-multiple.sh\n@@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' '\n \t GIT_TRACE=1 git fetch --multiple one two 2>trace &&\n \t git branch -r > output &&\n \t test_cmp ../expect output &&\n-\t grep \"built-in: git gc\" trace >gc &&\n+\t grep \"built-in: git maintenance\" trace >gc &&\n \t test_line_count = 1 gc\n \t)\n '\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3ee51723e0..373b8dbe04 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -17,7 +17,7 @@ test_expect_success 'run [--auto|--quiet]' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\t! grep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"401943","messageId":"9af2309f0804cc3a2b26c3cc5c5526d51e86aac2.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 14/18] maintenance: add auto condition for commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:36Z","receivedAt":"2020-07-23T17:57:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of writing a new commit-graph in every 'git maintenance run\n--auto' process (when maintenance.commit-graph.enalbed is configured to\nbe true), only write when there are \"enough\" commits not in a\ncommit-graph file.\n\nThis count is controlled by the maintenance.commit-graph.auto config\noption.\n\nTo compute the count, use a depth-first search starting at each ref, and\nleaving markers using the PARENT1 flag. If this count reaches the limit,\nthen terminate early and start the task. Otherwise, this operation will\npeel every ref and parse the commit it points to. If these are all in\nthe commit-graph, then this is typically a very fast operation. Users\nwith many refs might feel a slow-down, and hence could consider updating\ntheir limit to be very small. A negative value will force the step to\nrun every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 10 ++++\n builtin/gc.c | 76 ++++++++++++++++++++++++++++\n object.h | 1 +\n 3 files changed, 87 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 370cbfb42f..9bd69b9df3 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -2,3 +2,13 @@ maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified.\n \tBy default, only `maintenance.gc.enabled` is true.\n+\n+maintenance.commit-graph.auto::\n+\tThis integer config option controls how often the `commit-graph` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `commit-graph` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\treachable commits that are not in the commit-graph file is at least\n+\tthe value of `maintenance.commit-graph.auto`. The default value is\n+\t100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 31696a2595..84ad360d17 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -30,6 +30,7 @@\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n #include \"midx.h\"\n+#include \"refs.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -715,6 +716,80 @@ static struct maintenance_opts {\n \tint tasks_selected;\n } opts;\n \n+/* Remember to update object flag allocation in object.h */\n+#define PARENT1\t\t(1u<<16)\n+\n+static int num_commits_not_in_graph = 0;\n+static int limit_commits_not_in_graph = 100;\n+\n+static int dfs_on_ref(const char *refname,\n+\t\t const struct object_id *oid, int flags,\n+\t\t void *cb_data)\n+{\n+\tint result = 0;\n+\tstruct object_id peeled;\n+\tstruct commit_list *stack = NULL;\n+\tstruct commit *commit;\n+\n+\tif (!peel_ref(refname, &peeled))\n+\t\toid = &peeled;\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tcommit = lookup_commit(the_repository, oid);\n+\tif (!commit)\n+\t\treturn 0;\n+\tif (parse_commit(commit))\n+\t\treturn 0;\n+\n+\tcommit_list_append(commit, &stack);\n+\n+\twhile (!result && stack) {\n+\t\tstruct commit_list *parent;\n+\n+\t\tcommit = pop_commit(&stack);\n+\n+\t\tfor (parent = commit->parents; parent; parent = parent->next) {\n+\t\t\tif (parse_commit(parent->item) ||\n+\t\t\t commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH ||\n+\t\t\t parent->item->object.flags & PARENT1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tparent->item->object.flags |= PARENT1;\n+\t\t\tnum_commits_not_in_graph++;\n+\n+\t\t\tif (num_commits_not_in_graph >= limit_commits_not_in_graph) {\n+\t\t\t\tresult = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tcommit_list_append(parent->item, &stack);\n+\t\t}\n+\t}\n+\n+\tfree_commit_list(stack);\n+\treturn result;\n+}\n+\n+static int should_write_commit_graph(void)\n+{\n+\tint result;\n+\n+\tgit_config_get_int(\"maintenance.commit-graph.auto\",\n+\t\t\t &limit_commits_not_in_graph);\n+\n+\tif (!limit_commits_not_in_graph)\n+\t\treturn 0;\n+\tif (limit_commits_not_in_graph < 0)\n+\t\treturn 1;\n+\n+\tresult = for_each_ref(dfs_on_ref, NULL);\n+\n+\tclear_commit_marks_all(PARENT1);\n+\n+\treturn result;\n+}\n+\n static int run_write_commit_graph(void)\n {\n \tint result;\n@@ -1224,6 +1299,7 @@ static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"commit-graph\";\n \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n+\ttasks[num_tasks]->auto_condition = should_write_commit_graph;\n \tnum_tasks++;\n \n \tfor (i = 0; i < num_tasks; i++) {\ndiff --git a/object.h b/object.h\nindex 38dc2d5a6c..4f886495d7 100644\n--- a/object.h\n+++ b/object.h\n@@ -73,6 +73,7 @@ struct object_array {\n * list-objects-filter.c: 21\n * builtin/fsck.c: 0--3\n * builtin/index-pack.c: 2021\n+ * builtin/maintenance.c: 16\n * builtin/pack-objects.c: 20\n * builtin/reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n-- \ngitgitgadget\n\n"},{"id":"401945","messageId":"a3c64930a0947315f2ee512f9b1a11b806f00186.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 12/18] maintenance: create maintenance..enabled config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:34Z","receivedAt":"2020-07-23T17:57:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCurrently, a normal run of \"git maintenance run\" will only run the 'gc'\ntask, as it is the only one enabled. This is mostly for backwards-\ncompatible reasons since \"git maintenance run --auto\" commands replaced\nprevious \"git gc --auto\" commands after some Git processes. Users could\nmanually run specific maintenance tasks by calling \"git maintenance run\n--task=\" directly.\n\nAllow users to customize which steps are run automatically using config.\nThe 'maintenance..enabled' option then can turn on these other\ntasks (or turn off the 'gc' task).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 ++\n Documentation/config/maintenance.txt | 4 ++++\n Documentation/git-maintenance.txt | 6 +++++-\n builtin/gc.c | 13 +++++++++++++\n t/t7900-maintenance.sh | 12 ++++++++++++\n 5 files changed, 36 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/maintenance.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..2783b825f9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -396,6 +396,8 @@ include::config/mailinfo.txt[]\n \n include::config/mailmap.txt[]\n \n+include::config/maintenance.txt[]\n+\n include::config/man.txt[]\n \n include::config/merge.txt[]\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nnew file mode 100644\nindex 0000000000..370cbfb42f\n--- /dev/null\n+++ b/Documentation/config/maintenance.txt\n@@ -0,0 +1,4 @@\n+maintenance..enabled::\n+\tThis boolean config option controls whether the maintenance task\n+\twith name `` is run when no `--task` option is specified.\n+\tBy default, only `maintenance.gc.enabled` is true.\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex bda8df4aaa..4a61441bbc 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -30,7 +30,11 @@ SUBCOMMANDS\n -----------\n \n run::\n-\tRun one or more maintenance tasks.\n+\tRun one or more maintenance tasks. If one or more `--task` options\n+\tare specified, then those tasks are run in that order. Otherwise,\n+\tthe tasks are determined by which `maintenance..enabled`\n+\tconfig options are true. By default, only `maintenance.gc.enabled`\n+\tis true.\n \n TASKS\n -----\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 889d97afe7..b6dc4b1832 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1185,6 +1185,7 @@ static int maintenance_run(void)\n static void initialize_tasks(void)\n {\n \tint i;\n+\tstruct strbuf config_name = STRBUF_INIT;\n \tnum_tasks = 0;\n \n \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n@@ -1210,6 +1211,18 @@ static void initialize_tasks(void)\n \ttasks[num_tasks]->name = \"commit-graph\";\n \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n \tnum_tasks++;\n+\n+\tfor (i = 0; i < num_tasks; i++) {\n+\t\tint config_value;\n+\n+\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\", tasks[i]->name);\n+\n+\t\tif (!git_config_get_bool(config_name.buf, &config_value))\n+\t\t\ttasks[i]->enabled = config_value;\n+\t}\n+\n+\tstrbuf_release(&config_name);\n }\n \n static int task_option_parse(const struct option *opt,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex ab5c961eb9..3ee51723e0 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -21,6 +21,18 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'maintenance..enabled' '\n+\tgit config maintenance.gc.enabled false &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-config.txt\" git maintenance run &&\n+\t! grep \",\\\"fetch\\\"\" run-config.txt &&\n+\t! grep \",\\\"gc\\\"\" run-config.txt &&\n+\t! grep \",\\\"multi-pack-index\\\"\" run-config.txt &&\n+\tgrep \",\\\"commit-graph\\\"\" run-config.txt &&\n+\tgrep \",\\\"prune-packed\\\"\" run-config.txt\n+'\n+\n test_expect_success 'run --task=' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n-- \ngitgitgadget\n\n"},{"id":"401947","messageId":"f24db7739fc4bf3f947cb7bfd3fe19908a1b4715.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 18/18] maintenance: add trace2 regions for task execution","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:40Z","receivedAt":"2020-07-23T17:57:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b040c7d31d..7d9e6c34b7 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1322,7 +1322,9 @@ static int maintenance_run(void)\n \t\t !tasks[i]->auto_condition()))\n \t\t\tcontinue;\n \n+\t\ttrace2_region_enter(\"maintenance\", tasks[i]->name, r);\n \t\tresult = tasks[i]->fn();\n+\t\ttrace2_region_leave(\"maintenance\", tasks[i]->name, r);\n \t}\n \n \trollback_lock_file(&lk);\n-- \ngitgitgadget\n"},{"id":"401946","messageId":"a0f00f8ab89469907b1304e09bd298a0236286bd.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 17/18] midx: use start_delayed_progress()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:39Z","receivedAt":"2020-07-23T17:57:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that the multi-pack-index may be written as part of auto maintenance\nat the end of a command, reduce the progress output when the operations\nare quick. Use start_delayed_progress() instead of start_progress().\n\nUpdate t5319-multi-pack-index.sh to use GIT_PROGRESS_DELAY=0 now that\nthe progress indicators are conditional.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 10 +++++-----\n t/t5319-multi-pack-index.sh | 14 +++++++-------\n 2 files changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex 57a8a00082..d4022e4aef 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -837,7 +837,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tpacks.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tpacks.progress = NULL;\n \n@@ -974,7 +974,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Writing chunks to multi-pack-index\"),\n+\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (written != chunk_offsets[i])\n@@ -1109,7 +1109,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t\treturn 0;\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Looking for referenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n@@ -1230,7 +1230,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Counting referenced objects\"),\n+\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n \t\t\t\t\t m->num_objects);\n \tfor (i = 0; i < m->num_objects; i++) {\n \t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n@@ -1240,7 +1240,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tstop_progress(&progress);\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Finding and deleting unreferenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tchar *pack_name;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 7214cab36c..12f41dfc18 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -172,12 +172,12 @@ test_expect_success 'write progress off for redirected stderr' '\n '\n \n test_expect_success 'write force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress write 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'write with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -334,17 +334,17 @@ test_expect_success 'git-fsck incorrect offset' '\n '\n \n test_expect_success 'repack progress off for redirected stderr' '\n-\tgit multi-pack-index --object-dir=$objdir repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n test_expect_success 'repack force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'repack with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -488,7 +488,7 @@ test_expect_success 'expire progress off for redirected stderr' '\n test_expect_success 'expire force progress on for stderr' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --progress expire 2>err &&\n \t\ttest_file_not_empty err\n \t)\n '\n@@ -496,7 +496,7 @@ test_expect_success 'expire force progress on for stderr' '\n test_expect_success 'expire with the --no-progress option' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --no-progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --no-progress expire 2>err &&\n \t\ttest_line_count = 0 err\n \t)\n '\n-- \ngitgitgadget\n\n"},{"id":"401948","messageId":"3d527cb0dda20a5b89a9a213fbdd4a28586a4e4f.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 16/18] maintenance: add incremental-repack auto condition","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:38Z","receivedAt":"2020-07-23T17:57:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe incremental-repack task updates the multi-pack-index by deleting pack-\nfiles that have been replaced with new packs, then repacking a batch of\nsmall pack-files into a larger pack-file. This incremental repack is faster\nthan rewriting all object data, but is slower than some other\nmaintenance activities.\n\nThe 'maintenance.incremental-repack.auto' config option specifies how many\npack-files should exist outside of the multi-pack-index before running\nthe step. These pack-files could be created by 'git fetch' commands or\nby the loose-objects task. The default value is 10.\n\nSetting the option to zero disables the task with the '--auto' option,\nand a negative value makes the task run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 ++++++++\n builtin/gc.c | 31 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 30 +++++++++++++++++++++++++++\n 3 files changed, 70 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a9442dd260..22229e7174 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -21,3 +21,12 @@ maintenance.loose-objects.auto::\n \tpositive value implies the command should run when the number of\n \tloose objects is at least the value of `maintenance.loose-objects.auto`.\n \tThe default value is 100.\n+\n+maintenance.incremental-repack.auto::\n+\tThis integer config option controls how often the `incremental-repack`\n+\ttask should be run as part of `git maintenance run --auto`. If zero,\n+\tthen the `incremental-repack` task will not run with the `--auto`\n+\toption. A negative value will force the task to run every time.\n+\tOtherwise, a positive value implies the command should run when the\n+\tnumber of pack-files not in the multi-pack-index is at least the value\n+\tof `maintenance.incremental-repack.auto`. The default value is 10.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ae59a28203..b040c7d31d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"refs.h\"\n+#include \"object-store.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1063,6 +1064,35 @@ static int maintenance_task_loose_objects(void)\n \treturn prune_packed() || pack_loose();\n }\n \n+static int incremental_repack_auto_condition(void)\n+{\n+\tstruct packed_git *p;\n+\tint enabled;\n+\tint incremental_repack_auto_limit = 10;\n+\tint count = 0;\n+\n+\tif (git_config_get_bool(\"core.multiPackIndex\", &enabled) ||\n+\t !enabled)\n+\t\treturn 0;\n+\n+\tgit_config_get_int(\"maintenance.incremental-repack.auto\",\n+\t\t\t &incremental_repack_auto_limit);\n+\n+\tif (!incremental_repack_auto_limit)\n+\t\treturn 0;\n+\tif (incremental_repack_auto_limit < 0)\n+\t\treturn 1;\n+\n+\tfor (p = get_packed_git(the_repository);\n+\t count < incremental_repack_auto_limit && p;\n+\t p = p->next) {\n+\t\tif (!p->multi_pack_index)\n+\t\t\tcount++;\n+\t}\n+\n+\treturn count >= incremental_repack_auto_limit;\n+}\n+\n static int multi_pack_index_write(void)\n {\n \tint result;\n@@ -1319,6 +1349,7 @@ static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"incremental-repack\";\n \ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n+\ttasks[num_tasks]->auto_condition = incremental_repack_auto_condition;\n \tnum_tasks++;\n \n \ttasks[num_tasks]->name = \"gc\";\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e4244d7c3c..0b29674805 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -178,4 +178,34 @@ test_expect_success 'incremental-repack task' '\n \ttest_line_count = 2 packs-after\n '\n \n+test_expect_success 'maintenance.incremental-repack.auto' '\n+\tgit repack -adk &&\n+\tgit config core.multiPackIndex true &&\n+\tgit multi-pack-index write &&\n+\tGIT_TRACE2_EVENT=1 git -c maintenance.incremental-repack.auto=1 \\\n+\t\tmaintenance run --auto --task=incremental-repack >out &&\n+\t! grep \"\\\"multi-pack-index\\\"\" out &&\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit A-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-A-$i git \\\n+\t\t\t-c maintenance.incremental-repack.auto=2 \\\n+\t\t\tmaintenance run --auto --task=incremental-repack &&\n+\t\t! grep \"\\\"multi-pack-index\\\"\" trace-A-$i &&\n+\t\ttest_commit B-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-B-$i git \\\n+\t\t\t-c maintenance.incremental-repack.auto=2 \\\n+\t\t\tmaintenance run --auto --task=incremental-repack >out &&\n+\t\tgrep \"\\\"multi-pack-index\\\"\" trace-B-$i >/dev/null || return 1\n+\tdone\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"401949","messageId":"42e316ca5851992f29fa2658e38a08ebb7dd3e31.1595527000.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v2 15/18] maintenance: create auto condition for loose-objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-23T17:56:37Z","receivedAt":"2020-07-23T17:57:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe loose-objects task deletes loose objects that already exist in a\npack-file, then place the remaining loose objects into a new pack-file.\nIf this step runs all the time, then we risk creating pack-files with\nvery few objects with every 'git commit' process. To prevent\noverwhelming the packs directory with small pack-files, place a minimum\nnumber of objects to justify the task.\n\nThe 'maintenance.loose-objects.auto' config option specifies a minimum\nnumber of loose objects to justify the task to run under the '--auto'\noption. This defaults to 100 loose objects. Setting the value to zero\nwill prevent the step from running under '--auto' while a negative value\nwill force it to run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 +++++++++\n builtin/gc.c | 30 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 25 +++++++++++++++++++++++\n 3 files changed, 64 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 9bd69b9df3..a9442dd260 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -12,3 +12,12 @@ maintenance.commit-graph.auto::\n \treachable commits that are not in the commit-graph file is at least\n \tthe value of `maintenance.commit-graph.auto`. The default value is\n \t100.\n+\n+maintenance.loose-objects.auto::\n+\tThis integer config option controls how often the `loose-objects` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `loose-objects` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\tloose objects is at least the value of `maintenance.loose-objects.auto`.\n+\tThe default value is 100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 84ad360d17..ae59a28203 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -951,6 +951,35 @@ struct write_loose_object_data {\n \tint batch_size;\n };\n \n+static int loose_object_auto_limit = 100;\n+\n+static int loose_object_count(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\tint *count = (int*)data;\n+\tif (++(*count) >= loose_object_auto_limit)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int loose_object_auto_condition(void)\n+{\n+\tint count = 0;\n+\n+\tgit_config_get_int(\"maintenance.loose-objects.auto\",\n+\t\t\t &loose_object_auto_limit);\n+\n+\tif (!loose_object_auto_limit)\n+\t\treturn 0;\n+\tif (loose_object_auto_limit < 0)\n+\t\treturn 1;\n+\n+\treturn for_each_loose_file_in_objdir(the_repository->objects->odb->path,\n+\t\t\t\t\t loose_object_count,\n+\t\t\t\t\t NULL, NULL, &count);\n+}\n+\n static int loose_object_exists(const struct object_id *oid,\n \t\t\t const char *path,\n \t\t\t void *data)\n@@ -1285,6 +1314,7 @@ static void initialize_tasks(void)\n \n \ttasks[num_tasks]->name = \"loose-objects\";\n \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n+\ttasks[num_tasks]->auto_condition = loose_object_auto_condition;\n \tnum_tasks++;\n \n \ttasks[num_tasks]->name = \"incremental-repack\";\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 373b8dbe04..e4244d7c3c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -115,6 +115,31 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'maintenance.loose-objects.auto' '\n+\tgit repack -adk &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-lo1.txt\" \\\n+\t\tgit -c maintenance.loose-objects.auto=1 maintenance \\\n+\t\trun --auto --task=loose-objects &&\n+\t! grep \"\\\"prune-packed\\\"\" trace-lo1.txt &&\n+\tfor i in 1 2\n+\tdo\n+\t\tprintf data-A-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loA-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\t! grep \"\\\"prune-packed\\\"\" trace-loA-$i &&\n+\t\tprintf data-B-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loB-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loB-$i &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loC-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loC-$i || return 1\n+\tdone\n+'\n+\n test_expect_success 'incremental-repack task' '\n \tpackDir=.git/objects/pack &&\n \tfor i in $(test_seq 1 5)\n-- \ngitgitgadget\n\n"},{"id":"401956","messageId":"xmqqr1t2qa1n.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"8e260bccf1a0b6cd799a6bc78798b31ebed8ad7e.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T19:57:24Z","receivedAt":"2020-07-23T19:57:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static void initialize_tasks(void)\n> +{\n> +\tint i;\n> +\tnum_tasks = 0;\n> +\n> +\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n> +\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n> +\n> +\ttasks[num_tasks]->name = \"gc\";\n> +\ttasks[num_tasks]->fn = maintenance_task_gc;\n> +\ttasks[num_tasks]->enabled = 1;\n> +\tnum_tasks++;\n\nAre we going to have 47 different tasks initialized by code like\nthis in the future? I would have expected that you'd have a table\nof tasks that serves as the blueprint copy and copy it to the table\nto be used if there is some need to mutate the table-to-be-used.\n\n> }\n> \n> int cmd_maintenance(int argc, const char **argv, const char *prefix)\n> @@ -751,6 +787,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n> \t\t\t\t builtin_maintenance_options);\n> \n> \topts.quiet = !isatty(2);\n> +\tinitialize_tasks();\n> \n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_maintenance_options,\n"},{"id":"401958","messageId":"xmqqimeeq8x4.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"a09b1c16874c0fad6dd0df3ace573a2ffe34519e.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/18] maintenance: add --task option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T20:21:43Z","receivedAt":"2020-07-23T20:21:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 35b0be7d40..9204762e21 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -73,6 +73,10 @@ OPTIONS\n> --quiet::\n> \tDo not report progress or other information over `stderr`.\n> \n> +--task=::\n> +\tIf this option is specified one or more times, then only run the\n> +\tspecified tasks in the specified order.\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 2cd17398ec..c58dea6fa5 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -710,6 +710,7 @@ static const char * const builtin_maintenance_usage[] = {\n> static struct maintenance_opts {\n> \tint auto_flag;\n> \tint quiet;\n> +\tint tasks_selected;\n> } opts;\n> \n> static int run_write_commit_graph(void)\n> @@ -804,20 +805,38 @@ typedef int maintenance_task_fn(void);\n> struct maintenance_task {\n> \tconst char *name;\n> \tmaintenance_task_fn *fn;\n> -\tunsigned enabled:1;\n> +\tint task_order;\n> +\tunsigned enabled:1,\n> +\t\t selected:1;\n> };\n> \n> static struct maintenance_task *tasks[MAX_NUM_TASKS];\n> static int num_tasks;\n> \n> +static int compare_tasks_by_selection(const void *a_, const void *b_)\n> +{\n> +\tconst struct maintenance_task *a, *b;\n> +\ta = (const struct maintenance_task *)a_;\n> +\tb = (const struct maintenance_task *)b_;\n> +\n> +\treturn b->task_order - a->task_order;\n> +}\n\nIt forces the reader to know intimately that task_order *is*\nselection order in order to understand why this is \"tasks by\nselection\". Perhaps rename the field to match what it is\n(i.e. \"selection_order\")?\n\n> static int maintenance_run(void)\n> {\n> \tint i;\n> \tint result = 0;\n> \n> +\tif (opts.tasks_selected)\n> +\t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n> +\n> \tfor (i = 0; !result && i < num_tasks; i++) {\n> -\t\tif (!tasks[i]->enabled)\n> +\t\tif (opts.tasks_selected && !tasks[i]->selected)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n> \t\t\tcontinue;\n\nI am not sure about this. Even if the task is disabled, if the\nuser says --task=, it is run anyway? Doesn't make an immediate\nsense to me.\n\nAs I am bad at deciphering de Morgan, I'd find it easier to read if\nthe first were written more like so:\n\n\t\tif (!(!opts.tasks_selected || tasks[i]->selected))\n\t\t\tcontinue;\n\nThat is, \"do not skip any when not limited, and do not skip the ones\nthat are selected when limited\". And that would easily extend to\n\n\t\tif (!tasks[i]->enabled ||\n\t\t !(!opts.tasks_selected || tasks[i]->selected))\n\t\t\tcontinue;\n> +\n> \t\tresult = tasks[i]->fn();\n> \t}\n\n> @@ -842,6 +861,44 @@ static void initialize_tasks(void)\n> \tnum_tasks++;\n> }\n> \n> +static int task_option_parse(const struct option *opt,\n> +\t\t\t const char *arg, int unset)\n> +{\n> +\tint i;\n> +\tstruct maintenance_task *task = NULL;\n> +\n> +\tBUG_ON_OPT_NEG(unset);\n> +\n> +\tif (!arg || !strlen(arg)) {\n> +\t\terror(_(\"--task requires a value\"));\n> +\t\treturn 1;\n\nThere is no need to special case an empty string that was explicitly\ngiven as the value---it will be caught as \"'' is not a valid task\".\n\n> +\t}\n> +\n> +\topts.tasks_selected++;\n> +\n> +\tfor (i = 0; i < MAX_NUM_TASKS; i++) {\n> +\t\tif (tasks[i] && !strcasecmp(tasks[i]->name, arg)) {\n> +\t\t\ttask = tasks[i];\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\tif (!task) {\n> +\t\terror(_(\"'%s' is not a valid task\"), arg);\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (task->selected) {\n> +\t\terror(_(\"task '%s' cannot be selected multiple times\"), arg);\n> +\t\treturn 1;\n> +\t}\n> +\n> +\ttask->selected = 1;\n> +\ttask->task_order = opts.tasks_selected;\n> +\n> +\treturn 0;\n> +}\n"},{"id":"401959","messageId":"xmqqblk6q8ws.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"f164d1a0b4363eb6eb174e90e88211829f8a1d55.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T20:21:55Z","receivedAt":"2020-07-23T20:22:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +--[no-]maintenance::\n> --[no-]auto-gc::\n> -\tRun `git gc --auto` at the end to perform garbage collection\n> -\tif needed. This is enabled by default.\n> +\tRun `git maintenance run --auto` at the end to perform garbage\n> +\tcollection if needed. This is enabled by default.\n\nShouldn't the new synonym be called --auto-maintenance or an\nabbreviation thereof? It is not like we will run the full\nmaintenance suite when \"--no-maintenance\" is omitted, which\ncertainly is not the impression we want to give our readers.\n\n> These objects may be removed by normal Git operations (such as `git commit`)\n> -which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n> -If these objects are removed and were referenced by the cloned repository,\n> -then the cloned repository will become corrupt.\n> +which automatically call `git maintenance run --auto` and `git gc --auto`.\n\nHmph. Perhaps the picture may change in the end of the series but I\ngot an impression that \"gc --auto\" would eventually become just part\nof \"maintenance --auto\" and the users won't have to be even aware of\nits existence? Wouldn't we rather want to say something like\n\n\t--[no-]auto-maintenance::\n\t--[no-]auto-gc::\n Run `git maintenance run --auto` at the end to perform\n garbage collection if needed (`--[no-]auto-gc` is a\n synonym). This is enabled by default.\n\t\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 82ac4be8a5..49a4d727d4 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n> \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n> \t\t\tN_(\"report that we have only objects reachable from this object\")),\n> \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n\n> +\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n> +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n> \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n> +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n\nOK, so this is truly a backward-compatible synonym at this point.\n\n> diff --git a/run-command.c b/run-command.c\n> index 9b3a57d1e3..82ad241638 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1865,14 +1865,17 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> \treturn result;\n> }\n> \n> -int run_auto_gc(int quiet)\n> +int run_auto_maintenance(int quiet)\n> {\n> \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n> \tint status;\n> \n> -\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n> +\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n> \tif (quiet)\n> \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n> +\telse\n> +\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n> +\n> \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n> \targv_array_clear(&argv_gc_auto);\n> \treturn status;\n\nDon't we want to replace all _gc_ with _maintenance_ in this\nfunction? I think the first business before we can do so would be\nto rethink if spelling out \"maintenance\" fully in code is a good\nidea in the first space. It would make names for variables,\nstructures and fields unnecessarily long without contributing to\nease of understanding an iota, and a easy-to-remember short-form or\nan abbreviation may be needed. Using a short-form/abbreviation\nwouldn't worsen the end-user experience, and not the developer\nexperience for that matter.\n\nIf we choose \"gc\" as the short-hand, most of the change in this step\nwould become unnecessary. I also do not mind if we some other words\nor word-fragment (perhaps \"maint\"???) is chosen.\n\n> diff --git a/run-command.h b/run-command.h\n> index 191dfcdafe..d9a800e700 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args);\n> /*\n> * Trigger an auto-gc\n> */\n> -int run_auto_gc(int quiet);\n> +int run_auto_maintenance(int quiet);\n> \n> #define RUN_COMMAND_NO_STDIN 1\n> #define RUN_GIT_CMD\t 2\t/*If this is to be git sub-command */\n> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n> index a66dbe0bde..9850ecde5d 100755\n> --- a/t/t5510-fetch.sh\n> +++ b/t/t5510-fetch.sh\n> @@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' '\n> \t\tgit config fetch.unpackLimit 1 &&\n> \t\tgit config gc.autoPackLimit 1 &&\n> \t\tgit config gc.autoDetach false &&\n> -\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch >fetch.out 2>&1 &&\n> +\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch --verbose >fetch.out 2>&1 &&\n> \t\ttest_i18ngrep \"Auto packing the repository\" fetch.out &&\n> \t\t! grep \"Should I try again\" fetch.out\n> \t)\n"},{"id":"401960","messageId":"xmqq4kpyq8wh.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"04552b1d2ed751a11eb7c50f6898cbc078b552b4.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T20:22:06Z","receivedAt":"2020-07-23T20:22:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static int maintenance_task_commit_graph(void)\n> +{\n> +\tstruct repository *r = the_repository;\n> +\tchar *chain_path;\n> +\n> +\t/* Skip commit-graph when --auto is specified. */\n> +\tif (opts.auto_flag)\n> +\t\treturn 0;\n\nStepping back a bit, back in \"git gc\" days, \"--auto\" had two\ndistinct meanings rolled into one. Check if it even needs to be\ndone, and perform only the lightweight variant if needed.\n\nFor this task, there is no \"lightweight variant\" is possible, so\nreturning without checking the need to do a lightweight one makes\nperfect sense here.\n\nBut wouldn't it suggest perhaps we could name \"auto\" field of the\noptions struct in a more meaningful way? Perhaps \"quick\" (i.e. only\nthe quicker-variant of the maintenance job) or something?\n\n> +\tclose_object_store(r->objects);\n> +\tif (run_write_commit_graph()) {\n> +\t\terror(_(\"failed to write commit-graph\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (!run_verify_commit_graph())\n> +\t\treturn 0;\n> +\n> +\twarning(_(\"commit-graph verify caught error, rewriting\"));\n> +\n> +\tchain_path = get_commit_graph_chain_filename(r->objects->odb);\n> +\tif (unlink(chain_path)) {\n> +\t\tUNLEAK(chain_path);\n> +\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n\nOK.\n\n> +\t}\n> +\tfree(chain_path);\n> +\n> +\tif (!run_write_commit_graph())\n> +\t\treturn 0;\n> +\n> +\terror(_(\"failed to rewrite commit-graph\"));\n> +\treturn 1;\n> +}\n\nError convention is \"positive for error, zero for success?\" That is\na bit unusual for our internal API.\n\n> static int maintenance_task_gc(void)\n> {\n> \tint result;\n> @@ -768,6 +836,10 @@ static void initialize_tasks(void)\n> \ttasks[num_tasks]->fn = maintenance_task_gc;\n> \ttasks[num_tasks]->enabled = 1;\n> \tnum_tasks++;\n> +\n> +\ttasks[num_tasks]->name = \"commit-graph\";\n> +\ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n> +\tnum_tasks++;\n\nAgain, I am not sure if we want to keep piling on this.\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index e4e4036e50..216ac0b19e 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -12,7 +12,7 @@ test_expect_success 'help text' '\n> \ttest_i18ngrep \"usage: git maintenance run\" err\n> '\n> \n> -test_expect_success 'gc [--auto|--quiet]' '\n> +test_expect_success 'run [--auto|--quiet]' '\n\nIt does not look like this change belongs here. If \"run\" is\nappropriate title for this test at this step, it must have been so\nin the previous step.\n\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n"},{"id":"401962","messageId":"xmqqwo2uosvz.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"3165b8916d2d80bf72dac6596a42c871ccd4cbe6.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T20:53:20Z","receivedAt":"2020-07-23T20:53:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 9204762e21..0927643247 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -53,6 +53,18 @@ since it will not expire `.graph` files that were in the previous\n> `commit-graph-chain` file. They will be deleted by a later run based on\n> the expiration delay.\n> \n> +prefetch::\n> +\tThe `fetch` task updates the object directory with the latest objects\n\ns/fetch/prefetch/ most likely.\n\n> +\tfrom all registered remotes. For each remote, a `git fetch` command\n> +\tis run. The refmap is custom to avoid updating local or remote\n\ns/remote/remote-tracking/ definitely. Do not forget the hyphen\nbetween the two words.\n\nI think it made the above unnecessarily confusing that you ended a\nsentence after \"is run\". It gives a wrong impression that you'd be\ndoing a \"real fetch\", which you need to dispel with a follow up\ndescription of the refmap.\n\n\tFor each remote, a `git fetch` command is run with a refspec\n\tto fetch their branches (those in their `refs/heads`) into\n\tour `refs/prefetch//` hierarchy and without auto\n\tfollowing tags (the configured refspec in the repository is\n\tignored).\n\n> +\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> +\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> +\tnot updated.\n> ++\n> +This means that foreground fetches are still required to update the\n> +remote refs, but the users is notified when the branches and tags are\n\ns/is notified/are notified/???\n\n> +updated on the remote.\n\nOften, when one needs to say \"X. This means Y.\", X is a suboptimal\nway to explain what needs to be conveyed to the readers. But this\nis not such a case. Unlike the \"This means\" that is often an\nattempt to rephrasing a poor explanation given first, this gives an\nimplication.\n\nBut let's not start with a negative impression (i.e. even with\nprefetch, I still have to do X? What's the point???), but let them\nfeel why it is a good thing. Perhaps (continuing my previous\nrewrite):\n\n\tThis is done to avoid disrupting the remote-tracking\n\tbranches---the end users expect them to stay unmoved unless\n\tthey initiate a fetch. With prefetch task, however, the\n\tobjects necessary to complete a later real fetch would\n\talready be obtained, so the real fetch would go faster. In\n\tthe ideal case, it will just become an update to bunch of\n\tremote-tracking branches without any object transfer.\n\nor something like that? \n\n> +\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n> +\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n> +\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n> +\targv_array_push(&cmd, refmap.buf);\n\nThe command line looks somewhat fishy, but I think it is correct.\nAt first glance it looks like a mistake to pass \"--refmap=\" and the\nrefspec \"+refs/heads/*:refs/prefetch/origin/*\" as separate arguments,\nbut I think that is exactly what you want here, i.e.\n\n - defeat any refspec in the configuration with \"--refmap=\"\n\n - give explicit refspec \"+refs/heads/*:...\", with \"--no-tags\" to\n decline auto-following, to tell what exactly are to be fetched\n and stored where.\n\nThe description in the log message about refmap needs to be\nclarified, though (I've already done so in the above suggested\nrewrite).\n\n> +static int maintenance_task_prefetch(void)\n> +{\n> +\tint result = 0;\n> +\tstruct string_list_item *item;\n> +\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n> +\n> +\tif (for_each_remote(fill_each_remote, &remotes)) {\n> +\t\terror(_(\"failed to fill remotes\"));\n> +\t\tresult = 1;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\t/*\n> +\t * Do not modify the result based on the success of the 'fetch'\n> +\t * operation, as a loss of network could cause 'fetch' to fail\n> +\t * quickly. We do not want that to stop the rest of our\n> +\t * background operations.\n> +\t */\n\nThe loop that runs different tasks abort at the first failure,\nthough. Perhaps that loop needs to be rethought as well?\n\n> +\tfor (item = remotes.items;\n> +\t item && item < remotes.items + remotes.nr;\n> +\t item++)\n> +\t\tfetch_remote(item->string);\n> +\n> +cleanup:\n> +\tstring_list_clear(&remotes, 0);\n> +\treturn result;\n> +}\n> +\n> static int maintenance_task_gc(void)\n> {\n> \tint result;\n> @@ -871,6 +929,10 @@ static void initialize_tasks(void)\n> \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n> \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n> \n> +\ttasks[num_tasks]->name = \"prefetch\";\n> +\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n> +\tnum_tasks++;\n> +\n> \ttasks[num_tasks]->name = \"gc\";\n> \ttasks[num_tasks]->fn = maintenance_task_gc;\n> \ttasks[num_tasks]->enabled = 1;\n\nTwo things.\n\n - As I said upfront, I do not see the point of preparing the table\n with code.\n\n - The reason why prefetch is placed in front is probably because\n you do not want to repack before you add more objects to the\n object store. But doesn't that imply that there is an inherent\n ordering that we, as those who are more expert on Git than the\n end users, prefer? Is it a wise decision to let the users affect\n the order of the tasks run by giving command line options in\n different order in the previous step?\n\n"},{"id":"401964","messageId":"xmqqr1t2osm3.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"83648f48655ba68126110018d81c1d2e2bcc7a6f.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T20:59:16Z","receivedAt":"2020-07-23T20:59:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> Create a 'loose-objects' task for the 'git maintenance run' command.\n> This helps clean up loose objects without disrupting concurrent Git\n> commands using the following sequence of events:\n>\n> 1. Run 'git prune-packed' to delete any loose objects that exist\n> in a pack-file. Concurrent commands will prefer the packed\n> version of the object to the loose version. (Of course, there\n> are exceptions for commands that specifically care about the\n> location of an object. These are rare for a user to run on\n> purpose, and we hope a user that has selected background\n> maintenance will not be trying to do foreground maintenance.)\n\nOK. That would make sense.\n\n> 2. Run 'git pack-objects' on a batch of loose objects. These\n> objects are grouped by scanning the loose object directories in\n> lexicographic order until listing all loose objects -or-\n> reaching 50,000 objects. This is more than enough if the loose\n> objects are created only by a user doing normal development.\n\nI haven't seen this in action, but my gut feeling is that this would\nresult in horrible locality and deltification in the resulting\npackfile. The order you feed the objects to pack-objects and the\npath hint you attach to each object matters quite a lot.\n\nI do agree that it would be useful to have a task to deal with only\nloose objects without touching existing packfiles. I just am not\nsure if 2. is a worthwhile thing to do. A poorly constructed pack\nwill also contaminate later packfiles made without \"-f\" option to\n\"git repack\".\n\n"},{"id":"401968","messageId":"xmqqeep1q4d0.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"b6328c210625e1ba98e2065208a2a478c2c64f94.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/18] maintenance: add incremental-repack task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T22:00:11Z","receivedAt":"2020-07-23T22:00:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> 1. 'git multi-pack-index write' creates a multi-pack-index file if\n> one did not exist, and otherwise will update the multi-pack-index\n> with any new pack-files that appeared since the last write. This\n> is particularly relevant with the background fetch job.\n>\n> When the multi-pack-index sees two copies of the same object, it\n> stores the offset data into the newer pack-file. This means that\n> some old pack-files could become \"unreferenced\" which I will use\n> to mean \"a pack-file that is in the pack-file list of the\n> multi-pack-index but none of the objects in the multi-pack-index\n> reference a location inside that pack-file.\"\n\nAn obvious alternative is to favor the copy in the older pack,\nright? Is the expectation that over time, most of the objects that\nare relevant would reappear in newer packs, so that eventually by\nfavoring the copies in the newer packs, we can retire and remove the\nold pack, keeping only the newer ones?\n\nBut would that assumption hold? The old packs hold objects that are\nnecessary for the older parts of the history, so unless you are\ncauterizing away the old history, these objects in the older packs\nare likely to stay with us longer than those used by the newer parts\nof the history, some of which may not even have been pushed out yet\nand can be rebased away?\n\n> 2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n> and updaes the multi-pack-index to drop those pack-files from the\n> list. This is safe to do as concurrent Git processes will see the\n> multi-pack-index and not open those packs when looking for object\n> contents. (Similar to the 'loose-objects' job, there are some Git\n> commands that open pack-files regardless of the multi-pack-index,\n> but they are rarely used. Further, a user that self-selects to\n> use background operations would likely refrain from using those\n> commands.)\n\nOK.\n\n> 3. 'git multi-pack-index repack --bacth-size=' collects a set\n> of pack-files that are listed in the multi-pack-index and creates\n> a new pack-file containing the objects whose offsets are listed\n> by the multi-pack-index to be in those objects. The set of pack-\n> files is selected greedily by sorting the pack-files by modified\n> time and adding a pack-file to the set if its \"expected size\" is\n> smaller than the batch size until the total expected size of the\n> selected pack-files is at least the batch size. The \"expected\n> size\" is calculated by taking the size of the pack-file divided\n> by the number of objects in the pack-file and multiplied by the\n> number of objects from the multi-pack-index with offset in that\n> pack-file. The expected size approximats how much data from that\n\napproximates.\n\n> pack-file will contribute to the resulting pack-file size. The\n> intention is that the resulting pack-file will be close in size\n> to the provided batch size.\n\n> +static int maintenance_task_incremental_repack(void)\n> +{\n> +\tif (multi_pack_index_write()) {\n> +\t\terror(_(\"failed to write multi-pack-index\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (multi_pack_index_verify()) {\n> +\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n> +\t\treturn rewrite_multi_pack_index();\n> +\t}\n> +\n> +\tif (multi_pack_index_expire()) {\n> +\t\terror(_(\"multi-pack-index expire failed\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tif (multi_pack_index_verify()) {\n> +\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n> +\t\treturn rewrite_multi_pack_index();\n> +\t}\n> +\tif (multi_pack_index_repack()) {\n> +\t\terror(_(\"multi-pack-index repack failed\"));\n> +\t\treturn 1;\n> +\t}\n\nHmph, I wonder if these warning should come from each helper\nfunctions that are static to this function anyway.\n\nIt also makes it easier to reason about this function by eliminating\nthe need for having a different pattern only for the verify helper.\nInstead, verify could call rewrite internally when it notices a\nbreakage. I.e.\n\n\tif (multi_pack_index_write())\n\t\treturn 1;\n\tif (multi_pack_index_verify(\"after initial write\"))\n\t\treturn 1;\n\tif (multi_pack_index_exire())\n\t\treturn 1;\n\t...\n\nAlso, it feels odd, compared to our internal API convention, that\npositive non-zero is used as an error here.\n\n> +\treturn 0;\n> +}\n> +\n> typedef int maintenance_task_fn(void);\n> \n> struct maintenance_task {\n> @@ -1037,6 +1152,10 @@ static void initialize_tasks(void)\n> \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n> \tnum_tasks++;\n> \n> +\ttasks[num_tasks]->name = \"incremental-repack\";\n> +\ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n> +\tnum_tasks++;\n> +\n> \ttasks[num_tasks]->name = \"gc\";\n> \ttasks[num_tasks]->fn = maintenance_task_gc;\n> \ttasks[num_tasks]->enabled = 1;\n\nExactly the same comment as 08/18 about natural/inherent ordering\napplies here as well.\n\nThanks.\n"},{"id":"401969","messageId":"xmqqa6zpq3nt.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"478c7f1d0b858755c2c4b98605405214910b6f4c.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T22:15:18Z","receivedAt":"2020-07-23T22:15:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When repacking during the 'incremental-repack' task, we use the\n> --batch-size option in 'git multi-pack-index repack'. The initial setting\n> used --batch-size=0 to repack everything into a single pack-file. This is\n> not sustaintable for a large repository. The amount of work required is\n\nsustainable.\n\n> also likely to use too many system resources for a background job.\n>\n> Update the 'incremental-repack' task by dynamically computing a\n> --batch-size option based on the current pack-file structure.\n\nOK.\n\n> The dynamic default size is computed with this idea in mind for a client\n> repository that was cloned from a very large remote: there is likely one\n> \"big\" pack-file that was created at clone time. Thus, do not try\n> repacking it as it is likely packed efficiently by the server.\n>\n> Instead, we select the second-largest pack-file, and create a batch size\n> that is one larger than that pack-file. If there are three or more\n> pack-files, then this guarantees that at least two will be combined into\n> a new pack-file.\n>\n> Of course, this means that the second-largest pack-file size is likely\n> to grow over time and may eventually surpass the initially-cloned\n> pack-file. Recall that the pack-file batch is selected in a greedy\n> manner: the packs are considered from oldest to newest and are selected\n> if they have size smaller than the batch size until the total selected\n> size is larger than the batch size. Thus, that oldest \"clone\" pack will\n> be first to repack after the new data creates a pack larger than that.\n>\n> We also want to place some limits on how large these pack-files become,\n> in order to bound the amount of time spent repacking. A maximum\n> batch-size of two gigabytes means that large repositories will never be\n> packed into a single pack-file using this job, but also that repack is\n> rather expensive. This is a trade-off that is valuable to have if the\n> maintenance is being run automatically or in the background. Users who\n> truly want to optimize for space and performance (and are willing to pay\n> the upfront cost of a full repack) can use the 'gc' task to do so.\n\nIt might be too late to ask this now, but how does the quality of\nthe resulting combined pack ensured, wrt locality and deltification?\n\n> +#define TWO_GIGABYTES (2147483647)\n> +#define UNSET_BATCH_SIZE ((unsigned long)-1)\n> +\n> +static off_t get_auto_pack_size(void)\n> +{\n> +\t/*\n> +\t * The \"auto\" value is special: we optimize for\n> +\t * one large pack-file (i.e. from a clone) and\n> +\t * expect the rest to be small and they can be\n> +\t * repacked quickly.\n> +\t *\n> +\t * The strategy we select here is to select a\n> +\t * size that is one more than the second largest\n> +\t * pack-file. This ensures that we will repack\n> +\t * at least two packs if there are three or more\n> +\t * packs.\n> +\t */\n> +\toff_t max_size = 0;\n> +\toff_t second_largest_size = 0;\n> +\toff_t result_size;\n> +\tstruct packed_git *p;\n> +\tstruct repository *r = the_repository;\n> +\n> +\treprepare_packed_git(r);\n> +\tfor (p = get_all_packs(r); p; p = p->next) {\n> +\t\tif (p->pack_size > max_size) {\n> +\t\t\tsecond_largest_size = max_size;\n> +\t\t\tmax_size = p->pack_size;\n> +\t\t} else if (p->pack_size > second_largest_size)\n> +\t\t\tsecond_largest_size = p->pack_size;\n> +\t}\n> +\n> +\tresult_size = second_largest_size + 1;\n\nWe won't worry about this addition wrapping around; I guess we\ncannot do anything intelligent when it happens.\n\n> +\t/* But limit ourselves to a batch size of 2g */\n> +\tif (result_size > TWO_GIGABYTES)\n> +\t\tresult_size = TWO_GIGABYTES;\n\nWell, when it happens, we'd cap to 2G, which must be a reasonable\nfallback value, so it would be OK.\n\n> +\treturn result_size;\n> +}\n> +\n> static int multi_pack_index_repack(void)\n> {\n> \tint result;\n> \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> +\tstruct strbuf batch_arg = STRBUF_INIT;\n> +\n> \targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n> \n> \tif (opts.quiet)\n> \t\targv_array_push(&cmd, \"--no-progress\");\n> \n> -\targv_array_push(&cmd, \"--batch-size=0\");\n> +\tstrbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n> +\t\t (uintmax_t)get_auto_pack_size());\n> +\targv_array_push(&cmd, batch_arg.buf);\n> \n> \tclose_object_store(the_repository->objects);\n> \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n> +\tstrbuf_release(&batch_arg);\n\nI think I saw a suggestion to use xstrfmt() with free() instead of\nthe sequence of strbuf_init(), strbuf_addf(), and strbuf_release()\nin a similar but different context. Perhaps we should follow suit\nhere, too?\n\n\nThanks. That's it for today from me.\n\n"},{"id":"401970","messageId":"xmqq5zadq3ir.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"xmqqimeeq8x4.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 06/18] maintenance: add --task option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T22:18:20Z","receivedAt":"2020-07-23T22:18:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> \tfor (i = 0; !result && i < num_tasks; i++) {\n>> -\t\tif (!tasks[i]->enabled)\n>> +\t\tif (opts.tasks_selected && !tasks[i]->selected)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n>> \t\t\tcontinue;\n>\n> I am not sure about this. Even if the task is disabled, if the\n> user says --task=, it is run anyway? Doesn't make an immediate\n> sense to me.\n\nOK, after seeing the title of 12/18, it does make sense. Even if it\nis disabled by default, you could still choose to run, which makes\nsense.\n\n"},{"id":"401973","messageId":"CAPig+cRXdrg6ceZV27yjV_jB58iDuMi=54SvtArV1BfxExRgMQ@mail.gmail.com","threadId":"53809","inReplyTo":"xmqqa6zpq3nt.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2020-07-23T23:09:55Z","receivedAt":"2020-07-23T23:10:10Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 23, 2020 at 6:15 PM Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> > + struct strbuf batch_arg = STRBUF_INIT;\n> > +\n> > - argv_array_push(&cmd, \"--batch-size=0\");\n> > + strbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n> > + (uintmax_t)get_auto_pack_size());\n> > + argv_array_push(&cmd, batch_arg.buf);\n> >\n> > + strbuf_release(&batch_arg);\n>\n> I think I saw a suggestion to use xstrfmt() with free() instead of\n> the sequence of strbuf_init(), strbuf_addf(), and strbuf_release()\n> in a similar but different context. Perhaps we should follow suit\n> here, too?\n\nPerhaps I'm missing something obvious, but wouldn't argv_array_pushf()\nbe even simpler?\n\n argv_array_pushf(&cmd, \"--batch-size=%\"PRIuMAX,\n (uintmax_t)get_auto_pack_size());\n"},{"id":"401974","messageId":"xmqqsgdholvy.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"CAPig+cRXdrg6ceZV27yjV_jB58iDuMi=54SvtArV1BfxExRgMQ@mail.gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-23T23:24:33Z","receivedAt":"2020-07-23T23:24:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> On Thu, Jul 23, 2020 at 6:15 PM Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>> > + struct strbuf batch_arg = STRBUF_INIT;\n>> > +\n>> > - argv_array_push(&cmd, \"--batch-size=0\");\n>> > + strbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n>> > + (uintmax_t)get_auto_pack_size());\n>> > + argv_array_push(&cmd, batch_arg.buf);\n>> >\n>> > + strbuf_release(&batch_arg);\n>>\n>> I think I saw a suggestion to use xstrfmt() with free() instead of\n>> the sequence of strbuf_init(), strbuf_addf(), and strbuf_release()\n>> in a similar but different context. Perhaps we should follow suit\n>> here, too?\n>\n> Perhaps I'm missing something obvious, but wouldn't argv_array_pushf()\n> be even simpler?\n>\n> argv_array_pushf(&cmd, \"--batch-size=%\"PRIuMAX,\n> (uintmax_t)get_auto_pack_size());\n\nNo, it was me who was missing an obvious and better alternative.\n"},{"id":"401992","messageId":"1501583a-0fb0-ad88-bd2f-4f64c3c66980@gmail.com","threadId":"53809","inReplyTo":"xmqqr1t2qa1n.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T12:23:03Z","receivedAt":"2020-07-24T12:23:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 3:57 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +static void initialize_tasks(void)\n>> +{\n>> +\tint i;\n>> +\tnum_tasks = 0;\n>> +\n>> +\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n>> +\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n>> +\n>> +\ttasks[num_tasks]->name = \"gc\";\n>> +\ttasks[num_tasks]->fn = maintenance_task_gc;\n>> +\ttasks[num_tasks]->enabled = 1;\n>> +\tnum_tasks++;\n> \n> Are we going to have 47 different tasks initialized by code like\n> this in the future? I would have expected that you'd have a table\n> of tasks that serves as the blueprint copy and copy it to the table\n> to be used if there is some need to mutate the table-to-be-used.\n\nMaking it a table will likely make it easier to read. I hadn't\nthought of it.\n\nAt the start, I thought that the diff would look awful as we add\nmembers to the struct. However, the members that are not specified\nare set to zero, so I should be able to craft this into something\nnot too terrible.\n\nThanks,\n-Stolee\n"},{"id":"401993","messageId":"67e49b30-9a7f-8c18-3a06-96c516027c7a@gmail.com","threadId":"53809","inReplyTo":"1501583a-0fb0-ad88-bd2f-4f64c3c66980@gmail.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T12:51:32Z","receivedAt":"2020-07-24T12:51:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/24/2020 8:23 AM, Derrick Stolee wrote:\n> On 7/23/2020 3:57 PM, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> +static void initialize_tasks(void)\n>>> +{\n>>> +\tint i;\n>>> +\tnum_tasks = 0;\n>>> +\n>>> +\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n>>> +\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n>>> +\n>>> +\ttasks[num_tasks]->name = \"gc\";\n>>> +\ttasks[num_tasks]->fn = maintenance_task_gc;\n>>> +\ttasks[num_tasks]->enabled = 1;\n>>> +\tnum_tasks++;\n>>\n>> Are we going to have 47 different tasks initialized by code like\n>> this in the future? I would have expected that you'd have a table\n>> of tasks that serves as the blueprint copy and copy it to the table\n>> to be used if there is some need to mutate the table-to-be-used.\n> \n> Making it a table will likely make it easier to read. I hadn't\n> thought of it.\n> \n> At the start, I thought that the diff would look awful as we add\n> members to the struct. However, the members that are not specified\n> are set to zero, so I should be able to craft this into something\n> not too terrible.\n\nOK, my attempt has led to this final table:\n\n\tconst struct maintenance_task default_tasks[] = {\n\t\t{\n\t\t\t\"prefetch\",\n\t\t\tmaintenance_task_prefetch,\n\t\t},\n\t\t{\n\t\t\t\"loose-objects\",\n\t\t\tmaintenance_task_loose_objects,\n\t\t\tloose_object_auto_condition,\n\t\t},\n\t\t{\n\t\t\t\"incremental-repack\",\n\t\t\tmaintenance_task_incremental_repack,\n\t\t\tincremental_repack_auto_condition,\n\t\t},\n\t\t{\n\t\t\t\"gc\",\n\t\t\tmaintenance_task_gc,\n\t\t\tneed_to_gc,\n\t\t\t1,\n\t\t},\n\t\t{\n\t\t\t\"commit-graph\",\n\t\t\tmaintenance_task_commit_graph,\n\t\t\tshould_write_commit_graph,\n\t\t}\n\t};\n\tnum_tasks = sizeof(default_tasks) / sizeof(struct maintenance_task);\n\nThis is followed by allocating and copying the data to the\n'tasks' array, allowing it to be sorted and modified according\nto command-line arguments and config.\n\nIs this what you intended?\n\nThanks,\n-Stolee\n"},{"id":"401995","messageId":"b0bc5d89-52fa-e0f4-e3af-5eb6c76d312e@gmail.com","threadId":"53809","inReplyTo":"xmqq4kpyq8wh.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T13:09:25Z","receivedAt":"2020-07-24T13:09:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 4:22 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +static int maintenance_task_commit_graph(void)\n>> +{\n>> +\tstruct repository *r = the_repository;\n>> +\tchar *chain_path;\n>> +\n>> +\t/* Skip commit-graph when --auto is specified. */\n>> +\tif (opts.auto_flag)\n>> +\t\treturn 0;\n\nNow that you point this out, this is actually a stray condition\nfrom an earlier version. We now have the \".enabled\" config and\nthe auto condition function pointer. That handles all of that\n\"should we run this when --auto is specified?\" logic outside of\nthe task itself.\n \n> Stepping back a bit, back in \"git gc\" days, \"--auto\" had two\n> distinct meanings rolled into one. Check if it even needs to be\n> done, and perform only the lightweight variant if needed.\n> \n> For this task, there is no \"lightweight variant\" is possible, so\n> returning without checking the need to do a lightweight one makes\n> perfect sense here.\n> \n> But wouldn't it suggest perhaps we could name \"auto\" field of the\n> options struct in a more meaningful way? Perhaps \"quick\" (i.e. only\n> the quicker-variant of the maintenance job) or something?\n\nBut you are discussing here how the _behavior_ can change when\n--auto is specified. And specifically, \"git gc --auto\" really\nmeant \"This is running after a foreground command, so only do\nwork if necessary and do it quickly to minimize blocking time.\"\n\nI'd be happy to replace \"--auto\" with \"--quick\" in the\nmaintenance builtin.\n\nThis opens up some extra design space for how the individual\ntasks perform depending on \"--quick\" being specified or not.\nMy intention was to create tasks that are already in \"quick\"\nmode:\n\n* loose-objects have a maximum batch size.\n* incremental-repack is capped in size.\n* commit-graph uses the --split option.\n\nBut this \"quick\" distinction might be important for some of\nthe tasks we intend to extract from the gc builtin.\n\n>> +\tclose_object_store(r->objects);\n>> +\tif (run_write_commit_graph()) {\n>> +\t\terror(_(\"failed to write commit-graph\"));\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\tif (!run_verify_commit_graph())\n>> +\t\treturn 0;\n>> +\n>> +\twarning(_(\"commit-graph verify caught error, rewriting\"));\n>> +\n>> +\tchain_path = get_commit_graph_chain_filename(r->objects->odb);\n>> +\tif (unlink(chain_path)) {\n>> +\t\tUNLEAK(chain_path);\n>> +\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n> \n> OK.\n> \n>> +\t}\n>> +\tfree(chain_path);\n>> +\n>> +\tif (!run_write_commit_graph())\n>> +\t\treturn 0;\n>> +\n>> +\terror(_(\"failed to rewrite commit-graph\"));\n>> +\treturn 1;\n>> +}\n> \n> Error convention is \"positive for error, zero for success?\" That is\n> a bit unusual for our internal API.\n\nSince the tasks are frequently running subcommands, returning\n0 for success and non-zero for error matches the error codes\nreturned by those subcommands.\n\nShould I instead change the behavior and clearly document that\ntask functions matching maintenance_task_fn follow this error\npattern?\n\n>> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n>> index e4e4036e50..216ac0b19e 100755\n>> --- a/t/t7900-maintenance.sh\n>> +++ b/t/t7900-maintenance.sh\n>> @@ -12,7 +12,7 @@ test_expect_success 'help text' '\n>> \ttest_i18ngrep \"usage: git maintenance run\" err\n>> '\n>> \n>> -test_expect_success 'gc [--auto|--quiet]' '\n>> +test_expect_success 'run [--auto|--quiet]' '\n> \n> It does not look like this change belongs here. If \"run\" is\n> appropriate title for this test at this step, it must have been so\n> in the previous step.\n\nThanks. Will fix.\n\n-Stolee\n"},{"id":"401996","messageId":"2c5e5ed3-155e-f963-02c8-2789c799f3f7@gmail.com","threadId":"53809","inReplyTo":"xmqqimeeq8x4.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 06/18] maintenance: add --task option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T13:36:35Z","receivedAt":"2020-07-24T13:36:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 4:21 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>> +static int compare_tasks_by_selection(const void *a_, const void *b_)\n>> +{\n>> +\tconst struct maintenance_task *a, *b;\n>> +\ta = (const struct maintenance_task *)a_;\n>> +\tb = (const struct maintenance_task *)b_;\n>> +\n>> +\treturn b->task_order - a->task_order;\n>> +}\n> \n> It forces the reader to know intimately that task_order *is*\n> selection order in order to understand why this is \"tasks by\n> selection\". Perhaps rename the field to match what it is\n> (i.e. \"selection_order\")?\n\nGood idea. I made this fix locally.\n\n>> static int maintenance_run(void)\n>> {\n>> \tint i;\n>> \tint result = 0;\n>> \n>> +\tif (opts.tasks_selected)\n>> +\t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n>> +\n>> \tfor (i = 0; !result && i < num_tasks; i++) {\n>> -\t\tif (!tasks[i]->enabled)\n>> +\t\tif (opts.tasks_selected && !tasks[i]->selected)\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n>> \t\t\tcontinue;\n> \n> I am not sure about this. Even if the task is disabled, if the\n> user says --task=, it is run anyway? Doesn't make an immediate\n> sense to me.\n\nYou already replied that you figured this out. However, I could make\nit easier by adding some foreshadowing in the commit message here.\n\n> As I am bad at deciphering de Morgan, I'd find it easier to read if\n> the first were written more like so:\n> \n> \t\tif (!(!opts.tasks_selected || tasks[i]->selected))\n> \t\t\tcontinue;\n> \n> That is, \"do not skip any when not limited, and do not skip the ones\n> that are selected when limited\". And that would easily extend to\n> \n> \t\tif (!tasks[i]->enabled ||\n> \t\t !(!opts.tasks_selected || tasks[i]->selected))\n> \t\t\tcontinue;\n\nThis isn't quite right, due to the confusing nature of \"enabled\".\nThe condition here will _never_ allow selecting a disabled task.\n\nPerhaps it would be better to rename 'enabled' to 'run_by_default'?\nThat would make it clear that it is choosing which tasks to run unless\nspecified otherwise with --task= here, the config option\nmaintenance..enabled later, and the --auto conditions even later.\nLooking even farther down the line (into the next series) there will be\nsimilar checks for auto-conditionschecking time-based schedules.\n\nSince this loop becomes more complicated in the future, I specifically\nwanted to group the \"skip this task\" conditions into their own if\nblocks:\n\n\t1. If the user didn't specify --task= explicitly and this\n \t task is disabled, then skip this task.\n\n\t2. If the user _did_ specify --task= explicitly and this\n\t task was not on the list, then skip this task.\n\n\t3. If the user specified --auto and the auto condition fails,\n\t then skip this task.\n\n\t4. (Later) If the user specified --scheduled and the time since\n\t the last run is too soon, then skip this task.\n\nWith this being the planned future, I'd prefer these be split out as\nseparate if conditions instead of a giant combined if. And since that\nis the plan, then I won't work too hard to combine conditions 1 and 2\ninto a single condition.\n\n>> +\n>> \t\tresult = tasks[i]->fn();\n>> \t}\n> \n>> @@ -842,6 +861,44 @@ static void initialize_tasks(void)\n>> \tnum_tasks++;\n>> }\n>> \n>> +static int task_option_parse(const struct option *opt,\n>> +\t\t\t const char *arg, int unset)\n>> +{\n>> +\tint i;\n>> +\tstruct maintenance_task *task = NULL;\n>> +\n>> +\tBUG_ON_OPT_NEG(unset);\n>> +\n>> +\tif (!arg || !strlen(arg)) {\n>> +\t\terror(_(\"--task requires a value\"));\n>> +\t\treturn 1;\n> \n> There is no need to special case an empty string that was explicitly\n> given as the value---it will be caught as \"'' is not a valid task\".\n\nSounds good. No need for this extra message.\n\nThanks,\n-Stolee\n"},{"id":"402000","messageId":"52fc3f5a-281c-a0cb-d897-3866315913a1@gmail.com","threadId":"53809","inReplyTo":"xmqqwo2uosvz.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T14:25:52Z","receivedAt":"2020-07-24T14:25:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 4:53 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n>> index 9204762e21..0927643247 100644\n>> --- a/Documentation/git-maintenance.txt\n>> +++ b/Documentation/git-maintenance.txt\n>> @@ -53,6 +53,18 @@ since it will not expire `.graph` files that were in the previous\n>> `commit-graph-chain` file. They will be deleted by a later run based on\n>> the expiration delay.\n>> \n>> +prefetch::\n>> +\tThe `fetch` task updates the object directory with the latest objects\n> \n> s/fetch/prefetch/ most likely.\n> \n>> +\tfrom all registered remotes. For each remote, a `git fetch` command\n>> +\tis run. The refmap is custom to avoid updating local or remote\n> \n> s/remote/remote-tracking/ definitely. Do not forget the hyphen\n> between the two words.\n> \n> I think it made the above unnecessarily confusing that you ended a\n> sentence after \"is run\". It gives a wrong impression that you'd be\n> doing a \"real fetch\", which you need to dispel with a follow up\n> description of the refmap.\n> \n> \tFor each remote, a `git fetch` command is run with a refspec\n> \tto fetch their branches (those in their `refs/heads`) into\n> \tour `refs/prefetch//` hierarchy and without auto\n> \tfollowing tags (the configured refspec in the repository is\n> \tignored).\n> \n>> +\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n>> +\tremote refs are stored in `refs/prefetch//`. Also, tags are\n>> +\tnot updated.\n>> ++\n>> +This means that foreground fetches are still required to update the\n>> +remote refs, but the users is notified when the branches and tags are\n> \n> s/is notified/are notified/???\n> \n>> +updated on the remote.\n> \n> Often, when one needs to say \"X. This means Y.\", X is a suboptimal\n> way to explain what needs to be conveyed to the readers. But this\n> is not such a case. Unlike the \"This means\" that is often an\n> attempt to rephrasing a poor explanation given first, this gives an\n> implication.\n> \n> But let's not start with a negative impression (i.e. even with\n> prefetch, I still have to do X? What's the point???), but let them\n> feel why it is a good thing. Perhaps (continuing my previous\n> rewrite):\n> \n> \tThis is done to avoid disrupting the remote-tracking\n> \tbranches---the end users expect them to stay unmoved unless\n> \tthey initiate a fetch. With prefetch task, however, the\n> \tobjects necessary to complete a later real fetch would\n> \talready be obtained, so the real fetch would go faster. In\n> \tthe ideal case, it will just become an update to bunch of\n> \tremote-tracking branches without any object transfer.\n> \n> or something like that? \n\nI like this clarification and have adapted it with minimal edits.\n\n>> +\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n>> +\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n>> +\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n>> +\targv_array_push(&cmd, refmap.buf);\n> \n> The command line looks somewhat fishy, but I think it is correct.\n> At first glance it looks like a mistake to pass \"--refmap=\" and the\n> refspec \"+refs/heads/*:refs/prefetch/origin/*\" as separate arguments,\n> but I think that is exactly what you want here, i.e.\n> \n> - defeat any refspec in the configuration with \"--refmap=\"\n> \n> - give explicit refspec \"+refs/heads/*:...\", with \"--no-tags\" to\n> decline auto-following, to tell what exactly are to be fetched\n> and stored where.\n> \n> The description in the log message about refmap needs to be\n> clarified, though (I've already done so in the above suggested\n> rewrite).\n\nI could have made your life easier by referring to b40a50264ac\n(fetch: document and test --refmap=\"\", 2020-01-21) in my commit\nmessage. It includes this sentence in Documentation/fetch-options.txt:\n\n Providing an empty `` to the `--refmap` option causes\n Git to ignore the configured refspecs and rely entirely on the\n refspecs supplied as command-line arguments.\n\n>> +static int maintenance_task_prefetch(void)\n>> +{\n>> +\tint result = 0;\n>> +\tstruct string_list_item *item;\n>> +\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n>> +\n>> +\tif (for_each_remote(fill_each_remote, &remotes)) {\n>> +\t\terror(_(\"failed to fill remotes\"));\n>> +\t\tresult = 1;\n>> +\t\tgoto cleanup;\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * Do not modify the result based on the success of the 'fetch'\n>> +\t * operation, as a loss of network could cause 'fetch' to fail\n>> +\t * quickly. We do not want that to stop the rest of our\n>> +\t * background operations.\n>> +\t */\n> \n> The loop that runs different tasks abort at the first failure,\n> though. Perhaps that loop needs to be rethought as well?\n\nYou're right. These maintenance tasks are intended to be\nindependent of each other, so let's try all of them and\nreport a failure after all have been given an opportunity\nto run. That makes this failure behavior unnecessary.\n\n>> +\tfor (item = remotes.items;\n>> +\t item && item < remotes.items + remotes.nr;\n>> +\t item++)\n>> +\t\tfetch_remote(item->string);\n>> +\n>> +cleanup:\n>> +\tstring_list_clear(&remotes, 0);\n>> +\treturn result;\n>> +}\n>> +\n>> static int maintenance_task_gc(void)\n>> {\n>> \tint result;\n>> @@ -871,6 +929,10 @@ static void initialize_tasks(void)\n>> \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n>> \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n>> \n>> +\ttasks[num_tasks]->name = \"prefetch\";\n>> +\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n>> +\tnum_tasks++;\n>> +\n>> \ttasks[num_tasks]->name = \"gc\";\n>> \ttasks[num_tasks]->fn = maintenance_task_gc;\n>> \ttasks[num_tasks]->enabled = 1;\n> \n> Two things.\n> \n> - As I said upfront, I do not see the point of preparing the table\n> with code.\n> \n> - The reason why prefetch is placed in front is probably because\n> you do not want to repack before you add more objects to the\n> object store. But doesn't that imply that there is an inherent\n> ordering that we, as those who are more expert on Git than the\n> end users, prefer? Is it a wise decision to let the users affect\n> the order of the tasks run by giving command line options in\n> different order in the previous step?\n\nI don't anticipate users specifying --task= very often, as\nit requires deep knowledge of the tasks. If a user _does_ use the\noption, then we should trust their order as they might have a\ngood reason to choose that order.\n\nGenerally, my philosophy is to provide expert users with flexible\nchoices while creating sensible defaults for non-expert users.\n\nThat said, if this sort is more of a problem than the value it\nprovides, then I can drop the sort and the duplicate --task=\ncheck.\n\nThanks,\n-Stolee\n"},{"id":"402001","messageId":"b861244c-af0d-581f-b1c8-6c6cb667eed8@gmail.com","threadId":"53809","inReplyTo":"xmqqr1t2osm3.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T14:50:12Z","receivedAt":"2020-07-24T14:50:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 4:59 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> Create a 'loose-objects' task for the 'git maintenance run' command.\n>> This helps clean up loose objects without disrupting concurrent Git\n>> commands using the following sequence of events:\n>>\n>> 1. Run 'git prune-packed' to delete any loose objects that exist\n>> in a pack-file. Concurrent commands will prefer the packed\n>> version of the object to the loose version. (Of course, there\n>> are exceptions for commands that specifically care about the\n>> location of an object. These are rare for a user to run on\n>> purpose, and we hope a user that has selected background\n>> maintenance will not be trying to do foreground maintenance.)\n> \n> OK. That would make sense.\n> \n>> 2. Run 'git pack-objects' on a batch of loose objects. These\n>> objects are grouped by scanning the loose object directories in\n>> lexicographic order until listing all loose objects -or-\n>> reaching 50,000 objects. This is more than enough if the loose\n>> objects are created only by a user doing normal development.\n> \n> I haven't seen this in action, but my gut feeling is that this would\n> result in horrible locality and deltification in the resulting\n> packfile. The order you feed the objects to pack-objects and the\n> path hint you attach to each object matters quite a lot.\n> \n> I do agree that it would be useful to have a task to deal with only\n> loose objects without touching existing packfiles. I just am not\n> sure if 2. is a worthwhile thing to do. A poorly constructed pack\n> will also contaminate later packfiles made without \"-f\" option to\n> \"git repack\".\n\nThere are several factors going on here:\n\n * In a partial clone, it is likely that we get loose objects only\n due to a command like \"git log -p\" that downloads blobs\n one-by-one. In such a case, this step coming in later and picking\n up those blobs _will_ find good deltas because they are present\n in the same batch.\n\n * (I know this case isn't important to core Git, but please indulge\n me) In a VFS for Git repo, the loose objects correspond to blobs\n that were faulted in by a virtual filesystem read. In this case,\n the blobs are usually from a single commit in history, so good\n deltas between the blobs don't actually exist!\n\n * My experience indicates that the packs created by the\n loose-objects task are rather small (when created daily). This\n means that they get selected by the incremental-repack task to\n repack into a new pack-file where deltas are recomputed with modest\n success. As mentioned in that task, we saw a significant compression\n factor using that step for users of the Windows OS repo, mostly due\n to recomputing tree deltas.\n\n * Some amount of \"extra\" space is expected with this incremental\n repacking scheme. The most space-efficient thing to do is a full\n repack along with a tree walk that detects the paths used for each\n blob, allowing better hints for delta compression. However, that\n operation is very _time_ consuming. The trade-off here is something\n I should make more explicit. In my experience, disk space is cheap\n but CPU time is expensive. Most repositories could probably do a\n daily repack without being a disruption to the user. These steps\n enable maintenance for repositories where a full repack is too\n disruptive.\n\nI hope this adds some context. I would love if someone who knows more\nabout delta compression could challenge my assumptions. Sharing that\nexpertise can help create better maintenance strategies. Junio's\ninitial concern here is a good first step in that direction.\n\nThanks,\n-Stolee\n"},{"id":"402002","messageId":"54bfc8a9-e3bf-4199-9c58-8ed6ad50fb36@gmail.com","threadId":"53809","inReplyTo":"xmqqeep1q4d0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 10/18] maintenance: add incremental-repack task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T15:03:26Z","receivedAt":"2020-07-24T15:03:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 6:00 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> 1. 'git multi-pack-index write' creates a multi-pack-index file if\n>> one did not exist, and otherwise will update the multi-pack-index\n>> with any new pack-files that appeared since the last write. This\n>> is particularly relevant with the background fetch job.\n>>\n>> When the multi-pack-index sees two copies of the same object, it\n>> stores the offset data into the newer pack-file. This means that\n>> some old pack-files could become \"unreferenced\" which I will use\n>> to mean \"a pack-file that is in the pack-file list of the\n>> multi-pack-index but none of the objects in the multi-pack-index\n>> reference a location inside that pack-file.\"\n> \n> An obvious alternative is to favor the copy in the older pack,\n> right? Is the expectation that over time, most of the objects that\n> are relevant would reappear in newer packs, so that eventually by\n> favoring the copies in the newer packs, we can retire and remove the\n> old pack, keeping only the newer ones?\n> \n> But would that assumption hold? The old packs hold objects that are\n> necessary for the older parts of the history, so unless you are\n> cauterizing away the old history, these objects in the older packs\n> are likely to stay with us longer than those used by the newer parts\n> of the history, some of which may not even have been pushed out yet\n> and can be rebased away?\n\nIf we created a new pack-file containing an already-packed object,\nthen shouldn't we assume that the new pack-file does a _better_ job\nof compressing that object? Or at least, doesn't make it worse?\n\nFor example, if we use 'git multi-pack-index repack --batch-size=0',\nthen this creates a new pack-file containing every previously-packed\nobject. This new pack-file should have better delta compression than\nthe previous setup across multiple pack-files. We want this new\npack-file to be used, not the old ones.\n\nThis \"pick the newest pack\" strategy is also what allows us to safely\nuse the 'expire' option to drop old pack-files. If we always keep the\nold copies, then when we try to switch the new pack-files, we cannot\ndelete the old packs safely because a concurrent Git process could\nbe trying to reference it. One race is as follows:\n\n * Process A opens the multi-pack-index referring to old pack P. It\n doesn't open the pack-file as it hasn't tried to parse objects yet.\n\n * Process B is 'git multi-pack-index expire'. It sees that pack P can\n be dropped because all objects appear in newer pack-files. It deletes\n P.\n\n * Process A tries to read from P, and this fails. A must reprepare its\n representation of the pack-files.\n\nThis is the less disruptive race since A can recover with a small cost\nto its performance. The worse race (on Windows) is this:\n\n * Process A loads the multi-pack-index and tries to parse an object by\n loading \"old\" pack P.\n\n * Process B tries to delete P. However, on Windows the handle to P by\n A prevents the deletion.\n\nAt this point, there could be two resolutions. The first is to have\nthe 'expire' fail because we can't delete A. This means we might never\ndelete A in a busy repository. The second is that the 'expire' command\ncontinues and drops A from the list in the multi-pack-index. However,\nnow all Git processes add A to the packed_git list because it isn't\nreferenced by the multi-pack-index.\n\nIn summary: the decision to pick the newer copies is a fundamental\npart of how the write->expire->repack loop was designed.\n\n>> 2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n>> and updaes the multi-pack-index to drop those pack-files from the\n>> list. This is safe to do as concurrent Git processes will see the\n>> multi-pack-index and not open those packs when looking for object\n>> contents. (Similar to the 'loose-objects' job, there are some Git\n>> commands that open pack-files regardless of the multi-pack-index,\n>> but they are rarely used. Further, a user that self-selects to\n>> use background operations would likely refrain from using those\n>> commands.)\n> \n> OK.\n> \n>> 3. 'git multi-pack-index repack --bacth-size=' collects a set\n>> of pack-files that are listed in the multi-pack-index and creates\n>> a new pack-file containing the objects whose offsets are listed\n>> by the multi-pack-index to be in those objects. The set of pack-\n>> files is selected greedily by sorting the pack-files by modified\n>> time and adding a pack-file to the set if its \"expected size\" is\n>> smaller than the batch size until the total expected size of the\n>> selected pack-files is at least the batch size. The \"expected\n>> size\" is calculated by taking the size of the pack-file divided\n>> by the number of objects in the pack-file and multiplied by the\n>> number of objects from the multi-pack-index with offset in that\n>> pack-file. The expected size approximats how much data from that\n> \n> approximates.\n> \n>> pack-file will contribute to the resulting pack-file size. The\n>> intention is that the resulting pack-file will be close in size\n>> to the provided batch size.\n> \n>> +static int maintenance_task_incremental_repack(void)\n>> +{\n>> +\tif (multi_pack_index_write()) {\n>> +\t\terror(_(\"failed to write multi-pack-index\"));\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\tif (multi_pack_index_verify()) {\n>> +\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n>> +\t\treturn rewrite_multi_pack_index();\n>> +\t}\n>> +\n>> +\tif (multi_pack_index_expire()) {\n>> +\t\terror(_(\"multi-pack-index expire failed\"));\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\tif (multi_pack_index_verify()) {\n>> +\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n>> +\t\treturn rewrite_multi_pack_index();\n>> +\t}\n>> +\tif (multi_pack_index_repack()) {\n>> +\t\terror(_(\"multi-pack-index repack failed\"));\n>> +\t\treturn 1;\n>> +\t}\n> \n> Hmph, I wonder if these warning should come from each helper\n> functions that are static to this function anyway.\n> \n> It also makes it easier to reason about this function by eliminating\n> the need for having a different pattern only for the verify helper.\n> Instead, verify could call rewrite internally when it notices a\n> breakage. I.e.\n> \n> \tif (multi_pack_index_write())\n> \t\treturn 1;\n> \tif (multi_pack_index_verify(\"after initial write\"))\n> \t\treturn 1;\n> \tif (multi_pack_index_exire())\n> \t\treturn 1;\n> \t...\n\nThis is a cleaner model. I'll work on that.\n\n> Also, it feels odd, compared to our internal API convention, that\n> positive non-zero is used as an error here.\n...\n> Exactly the same comment as 08/18 about natural/inherent ordering\n> applies here as well.\n\nI'll leave these to be resolved in the earlier messages.\n\nThanks,\n-Stolee\n\n"},{"id":"402004","messageId":"ef102be6-132e-17eb-14dd-79239fdfd04b@gmail.com","threadId":"53809","inReplyTo":"xmqqsgdholvy.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T16:09:22Z","receivedAt":"2020-07-24T16:09:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 7:24 PM, Junio C Hamano wrote:\n> Eric Sunshine writes:\n> \n>> On Thu, Jul 23, 2020 at 6:15 PM Junio C Hamano wrote:\n>>> \"Derrick Stolee via GitGitGadget\" writes:\n>>>> + struct strbuf batch_arg = STRBUF_INIT;\n>>>> +\n>>>> - argv_array_push(&cmd, \"--batch-size=0\");\n>>>> + strbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n>>>> + (uintmax_t)get_auto_pack_size());\n>>>> + argv_array_push(&cmd, batch_arg.buf);\n>>>>\n>>>> + strbuf_release(&batch_arg);\n>>>\n>>> I think I saw a suggestion to use xstrfmt() with free() instead of\n>>> the sequence of strbuf_init(), strbuf_addf(), and strbuf_release()\n>>> in a similar but different context. Perhaps we should follow suit\n>>> here, too?\n>>\n>> Perhaps I'm missing something obvious, but wouldn't argv_array_pushf()\n>> be even simpler?\n>>\n>> argv_array_pushf(&cmd, \"--batch-size=%\"PRIuMAX,\n>> (uintmax_t)get_auto_pack_size());\n> \n> No, it was me who was missing an obvious and better alternative.\n\nToday I learned about arv_array_push. Thanks!\n\n-Stolee\n\n"},{"id":"402012","messageId":"xmqq365gn1m9.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"67e49b30-9a7f-8c18-3a06-96c516027c7a@gmail.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T19:39:58Z","receivedAt":"2020-07-24T19:40:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> OK, my attempt has led to this final table:\n>\n> \tconst struct maintenance_task default_tasks[] = {\n> \t\t{\n> \t\t\t\"prefetch\",\n> \t\t\tmaintenance_task_prefetch,\n> \t\t},\n>...\n> \t\t{\n> \t\t\t\"commit-graph\",\n> \t\t\tmaintenance_task_commit_graph,\n> \t\t\tshould_write_commit_graph,\n> \t\t}\n> \t};\n> \tnum_tasks = sizeof(default_tasks) / sizeof(struct maintenance_task);\n>\n> This is followed by allocating and copying the data to the\n> 'tasks' array, allowing it to be sorted and modified according\n> to command-line arguments and config.\n>\n> Is this what you intended?\n\nI do not know how important it is for your overall design to keep\nthe blueprint/master-copy table that is separate from the working\ncopy of the table that gets sorted, enabled/chosen bit set, etc.\nIIUC, you were modifying the entries' fields at runtime, so perhaps\na pristine copy is not all that important (in which case you can\njust lose \"const\" and do without extra copying)? I dunno.\n"},{"id":"402013","messageId":"xmqqy2n8lmq3.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"b0bc5d89-52fa-e0f4-e3af-5eb6c76d312e@gmail.com","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T19:47:00Z","receivedAt":"2020-07-24T19:47:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> But you are discussing here how the _behavior_ can change when\n> --auto is specified. And specifically, \"git gc --auto\" really\n> meant \"This is running after a foreground command, so only do\n> work if necessary and do it quickly to minimize blocking time.\"\n>\n> I'd be happy to replace \"--auto\" with \"--quick\" in the\n> maintenance builtin.\n>\n> This opens up some extra design space for how the individual\n> tasks perform depending on \"--quick\" being specified or not.\n> My intention was to create tasks that are already in \"quick\"\n> mode:\n>\n> * loose-objects have a maximum batch size.\n> * incremental-repack is capped in size.\n> * commit-graph uses the --split option.\n>\n> But this \"quick\" distinction might be important for some of\n> the tasks we intend to extract from the gc builtin.\n\nYup. To be honest, I came to this topic from a completely different\ndirection. The field name \"auto\" alone (and no other field name)\nhad to have an extra cruft (i.e. \"_flag\") attached to it, which is\nunderstandable but ugly. Then I started thinking if 'auto(matic)'\nis really the right word to describe what we want out of the option,\nand came to the realization that there may be better words.\n\n> Since the tasks are frequently running subcommands, returning\n> 0 for success and non-zero for error matches the error codes\n> returned by those subcommands.\n\nAs long as these will _never_ be called from other helper functions\nbut from the cmd_foo() top-level and their return values are only\nused directly as the top-level's return value, I do not mind too\nmuch.\n\nBut whenever I am writing such a code, I find myself not brave\nenough to make such a bold promise (I saw other people call the\nhelpers I wrote in unintended ways and had to adjust the semantics\nof them to accomodate the new callers too many times), so I'd rather\nsee the caller do \"return !!helper_fn()\" to allow helper_fn() to be\nwritten more naturally (e.g. letting them return error(...)).\n\nThanks.\n"},{"id":"402015","messageId":"xmqqtuxwlmkb.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"2c5e5ed3-155e-f963-02c8-2789c799f3f7@gmail.com","subject":"Re: [PATCH v2 06/18] maintenance: add --task option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T19:50:28Z","receivedAt":"2020-07-24T19:50:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> \tfor (i = 0; !result && i < num_tasks; i++) {\n>>> -\t\tif (!tasks[i]->enabled)\n>>> +\t\tif (opts.tasks_selected && !tasks[i]->selected)\n>>> +\t\t\tcontinue;\n>>> +\n>>> +\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n>>> \t\t\tcontinue;\n>> \n>> I am not sure about this. Even if the task is disabled, if the\n>> user says --task=, it is run anyway? Doesn't make an immediate\n>> sense to me.\n>> ...\n>> \t\tif (!tasks[i]->enabled ||\n>> \t\t !(!opts.tasks_selected || tasks[i]->selected))\n>> \t\t\tcontinue;\n>\n> This isn't quite right, due to the confusing nature of \"enabled\".\n\nYes, in the message you are responding to, I was still assuming that\n--task=foo that defeat task.foo.enabled=no would be a bug. If we\nwant to run disabled tasks by selection, of course the condition\nwould need to change.\n\n"},{"id":"402017","messageId":"d50fbb33-9be3-1c48-2277-8bf894df734f@gmail.com","threadId":"53809","inReplyTo":"xmqqa6zpq3nt.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-24T19:51:55Z","receivedAt":"2020-07-24T19:52:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 6:15 PM, Junio C Hamano wrote:\n> It might be too late to ask this now, but how does the quality of\n> the resulting combined pack ensured, wrt locality and deltification?\n\nThere are two questions here, really.\n\nThe first is: given the set of objects to pack, are we packing\nthem as efficiently as possible?\n\nSince e11d86de139 (midx: teach \"git multi-pack-index repack\" honor\n\"git repack\" configurations, 2020-05-10), the 'repack' subcommand\nhonors the configured recommendations for deltas. This includes:\n\n (requires updating the arguments to pack-objects)\n * repack.useDeltaBaseOffset\n * repack.useDeltaIslands\n\n (automatically respected by pack-objects)\n * repack.packKeptsObjects\n * pack.threads\n * pack.depth\n * pack.window\n * pack.windowMemory\n * pack.deltaCacheSize\n * pack.deltaCacheLimit\n\nAll of these config settings allow the user to specify how hard\nto try for delta compression. If they know something about their\ndata or their tolerance for extra CPU time during pack-objects,\nthen they can get better deltas by changing these values.\n\nThe second question is \"how well do the deltas compress when\nonly packing incrementally versus packing the entire repo?\"\n\nOne important way to consider these things is how the pack-\nfiles are created. If we expect most pack-files coming from\n'git fetch' calls, then there are some interesting patterns\nthat arise.\n\nI started measuring by creating a local clone of the Linux\nkernel repo starting at v5.0 and then fetching an increment\nof ten commits from the first-parent history of later tags.\nEach fetch created a pack-file of ~300 MB relative to the\nbase pack-file of ~1.6 GB. Collecting ten of these in a row\nleads to almost 2 GB of \"fetched\" packs.\n\nHowever, keep in mind that we didn't fetch 2 GB of data\n\"across the wire\" but instead expanded the thin pack into\na full pack by copying the base objects. After running the\nincremental-repack step, that ~2 GB of data re-compresses\nback down to one pack-file of size ~300 MB.\n\n_Why_ did 10 pack-files all around 300 MB get repacked at\nonce? It's because there were duplicate objects across those\npack-files! Recall that the multi-pack-index repack computes\nbatch sizes by computing an \"estimated pack size\" by counting\nhow many objects in that pack-file are referenced by the\nmulti-pack-index, then computing\n\n expected size = actual size * num objects\n / num referenced objects\n\nIn this case, the \"base\" objects that are copied between the\nfetches already exist in these smaller pack-files. Thus, when\nthe batch-size is ~300 MB it still repacks all 10 \"small\"\npacks into a new pack that is still ~300 MB.\n\nNow, this is still a little wasteful. That second pack has\na significant \"extra space\" cost. However, it came at a bonus\nof writing much less data.\n\nPerhaps the Linux kernel repository is just too small to care\nabout this version of maintenance? In such a case, I can work\nto introduce a 'full-repack' task that is more aggressive with\nrepacking all pack-files. This could use the multi-pack-index\nrepack with --batch-size=0 to still benefit from the\nrepack/expire trick for safe concurrency.\n\nOther ideas are to try repacking in other ways, such as by\nobject type, to maximize easy wins. For example, perhaps we\nrepack all of the commits and trees every time, but leave the\nblobs to be repacked when we are ready to spend time on\nremoving deltas?\n\nI think the incremental-repack has value, and perhaps it is\nisolated to super-huge repositories. That can be controlled\nby limiting its use to those when an expert user configures\nGit to use it.\n\nI remain open to recommendations from others with more\nexperience with delta compression to recommend alternatives.\n\ntl,dr: the incremental-repack isn't the most space-efficient\nthing we can do, and that's by design.\n\nThanks,\n-Stolee\n"},{"id":"402018","messageId":"xmqqk0yslm82.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"b861244c-af0d-581f-b1c8-6c6cb667eed8@gmail.com","subject":"Re: [PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T19:57:49Z","receivedAt":"2020-07-24T19:57:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> * In a partial clone, it is likely that we get loose objects only\n> due to a command like \"git log -p\" that downloads blobs\n> one-by-one. In such a case, this step coming in later and picking\n> up those blobs _will_ find good deltas because they are present\n> in the same batch.\n>\n> * (I know this case isn't important to core Git, but please indulge\n> me) In a VFS for Git repo, the loose objects correspond to blobs\n> that were faulted in by a virtual filesystem read. In this case,\n> the blobs are usually from a single commit in history, so good\n> deltas between the blobs don't actually exist!\n\nLet me stop here by saying that I am now starting to worry about\noverfitting the repacking strategy to lazy clone repositories. I am\nperfectly fine with the plan to start with just one strategy overfit\nfor partially cloned repositories, as long as we make sure that we\ncan be extended it to suit other access/object acquisition patterns.\n\n\n\n"},{"id":"402021","messageId":"xmqqft9gllbz.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"d50fbb33-9be3-1c48-2277-8bf894df734f@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T20:17:04Z","receivedAt":"2020-07-24T20:17:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> tl,dr: the incremental-repack isn't the most space-efficient\n> thing we can do, and that's by design.\n\nI am not very surprised by the fact that many packfiles that were\nobtained by thin pack transfer have many duplicate objects (due to\nhaving to include the delta bases), and it is natural to expect that\ndeduplication would save many bytes. It's not all that interesting.\n\nI am more interested in making sure that we can assure that in the\ncombined single pack, (1) objects are ordered for good locality of\naccess and (2) objects are getting good delta compression.\n\n"},{"id":"402025","messageId":"xmqqr1t0k5cz.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"52fc3f5a-281c-a0cb-d897-3866315913a1@gmail.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-24T20:47:24Z","receivedAt":"2020-07-24T20:47:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I don't anticipate users specifying --task= very often, as\n> it requires deep knowledge of the tasks. If a user _does_ use the\n> option, then we should trust their order as they might have a\n> good reason to choose that order.\n>\n> Generally, my philosophy is to provide expert users with flexible\n> choices while creating sensible defaults for non-expert users.\n\nSounds sensible.\n"},{"id":"402034","messageId":"20200725012658.GA35171@syl.lan","threadId":"53809","inReplyTo":"63ec602a07756a41f8ccddd745562c567a4b3ed7.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-07-25T01:26:58Z","receivedAt":"2020-07-25T01:27:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:23PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n\nEverything in this patch looks very sensible to me, and I think that\nimplementing 'git maintenance' in the same file as the current 'git gc'\nbuiltin makes a lot of sense to me.\n\n Reviewed-by: Taylor Blau \n\nThanks,\nTaylor\n"},{"id":"402035","messageId":"20200725013317.GB35171@syl.lan","threadId":"53809","inReplyTo":"xmqqblk6q8ws.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-07-25T01:33:17Z","receivedAt":"2020-07-25T01:33:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 23, 2020 at 01:21:55PM -0700, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n> > +--[no-]maintenance::\n> > --[no-]auto-gc::\n> > -\tRun `git gc --auto` at the end to perform garbage collection\n> > -\tif needed. This is enabled by default.\n> > +\tRun `git maintenance run --auto` at the end to perform garbage\n> > +\tcollection if needed. This is enabled by default.\n>\n> Shouldn't the new synonym be called --auto-maintenance or an\n> abbreviation thereof? It is not like we will run the full\n> maintenance suite when \"--no-maintenance\" is omitted, which\n> certainly is not the impression we want to give our readers.\n>\n> > These objects may be removed by normal Git operations (such as `git commit`)\n> > -which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n> > -If these objects are removed and were referenced by the cloned repository,\n> > -then the cloned repository will become corrupt.\n> > +which automatically call `git maintenance run --auto` and `git gc --auto`.\n>\n> Hmph. Perhaps the picture may change in the end of the series but I\n> got an impression that \"gc --auto\" would eventually become just part\n> of \"maintenance --auto\" and the users won't have to be even aware of\n> its existence? Wouldn't we rather want to say something like\n>\n> \t--[no-]auto-maintenance::\n> \t--[no-]auto-gc::\n> Run `git maintenance run --auto` at the end to perform\n> garbage collection if needed (`--[no-]auto-gc` is a\n> synonym). This is enabled by default.\n>\n> > diff --git a/builtin/fetch.c b/builtin/fetch.c\n> > index 82ac4be8a5..49a4d727d4 100644\n> > --- a/builtin/fetch.c\n> > +++ b/builtin/fetch.c\n> > @@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n> > \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n> > \t\t\tN_(\"report that we have only objects reachable from this object\")),\n> > \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n>\n> > +\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n> > +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n> > \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n> > +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n>\n> OK, so this is truly a backward-compatible synonym at this point.\n\nI wouldn't be opposed to making the 'auto-gc' option an\n'OPT_HIDDEN_BOOL', but I realize that users may not want to move as\nquickly as that. Perhaps we should visit this in a couple of releases\n(or perhaps you are getting to it in a later patch that I haven't read\nyet).\n\n> > diff --git a/run-command.c b/run-command.c\n> > index 9b3a57d1e3..82ad241638 100644\n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -1865,14 +1865,17 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> > \treturn result;\n> > }\n> >\n> > -int run_auto_gc(int quiet)\n> > +int run_auto_maintenance(int quiet)\n> > {\n> > \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n> > \tint status;\n> >\n> > -\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n> > +\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n> > \tif (quiet)\n> > \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n> > +\telse\n> > +\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n> > +\n> > \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n> > \targv_array_clear(&argv_gc_auto);\n> > \treturn status;\n>\n> Don't we want to replace all _gc_ with _maintenance_ in this\n> function? I think the first business before we can do so would be\n> to rethink if spelling out \"maintenance\" fully in code is a good\n> idea in the first space. It would make names for variables,\n> structures and fields unnecessarily long without contributing to\n> ease of understanding an iota, and a easy-to-remember short-form or\n> an abbreviation may be needed. Using a short-form/abbreviation\n> wouldn't worsen the end-user experience, and not the developer\n> experience for that matter.\n>\n> If we choose \"gc\" as the short-hand, most of the change in this step\n> would become unnecessary. I also do not mind if we some other words\n> or word-fragment (perhaps \"maint\"???) is chosen.\n\nYeah, writing out 'maintenance' every time in the code and in\ncommand-line arguments is kind of a mouthful. I'm more willing to accept\nthat '--maintenance' is something that users would write or script\naround, but 'maint' makes sense to me as a shorthand in the code.\n\nI could go either way on calling the command-line flag '--maint',\nthough.\n\n>\n> > diff --git a/run-command.h b/run-command.h\n> > index 191dfcdafe..d9a800e700 100644\n> > --- a/run-command.h\n> > +++ b/run-command.h\n> > @@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args);\n> > /*\n> > * Trigger an auto-gc\n> > */\n> > -int run_auto_gc(int quiet);\n> > +int run_auto_maintenance(int quiet);\n> >\n> > #define RUN_COMMAND_NO_STDIN 1\n> > #define RUN_GIT_CMD\t 2\t/*If this is to be git sub-command */\n> > diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n> > index a66dbe0bde..9850ecde5d 100755\n> > --- a/t/t5510-fetch.sh\n> > +++ b/t/t5510-fetch.sh\n> > @@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' '\n> > \t\tgit config fetch.unpackLimit 1 &&\n> > \t\tgit config gc.autoPackLimit 1 &&\n> > \t\tgit config gc.autoDetach false &&\n> > -\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch >fetch.out 2>&1 &&\n> > +\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch --verbose >fetch.out 2>&1 &&\n> > \t\ttest_i18ngrep \"Auto packing the repository\" fetch.out &&\n> > \t\t! grep \"Should I try again\" fetch.out\n> > \t)\n\nThanks,\nTaylor\n"},{"id":"402036","messageId":"20200725013715.GA2436@danh.dev","threadId":"53809","inReplyTo":"3165b8916d2d80bf72dac6596a42c871ccd4cbe6.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-07-25T01:37:15Z","receivedAt":"2020-07-25T01:37:20Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-07-23 17:56:30+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> When working with very large repositories, an incremental 'git fetch'\n> command can download a large amount of data. If there are many other\n> users pushing to a common repo, then this data can rival the initial\n> pack-file size of a 'git clone' of a medium-size repo.\n> \n> Users may want to keep the data on their local repos as close as\n> possible to the data on the remote repos by fetching periodically in\n> the background. This can break up a large daily fetch into several\n> smaller hourly fetches.\n> \n> The task is called \"prefetch\" because it is work done in advance\n> of a foreground fetch to make that 'git fetch' command much faster.\n> \n> However, if we simply ran 'git fetch ' in the background,\n> then the user running a foregroudn 'git fetch ' would lose\n> some important feedback when a new branch appears or an existing\n> branch updates. This is especially true if a remote branch is\n> force-updated and this isn't noticed by the user because it occurred\n> in the background. Further, the functionality of 'git push\n> --force-with-lease' becomes suspect.\n> \n> When running 'git fetch ' in the background, use\n> the following options for careful updating:\n\nDoes this job interfere with FETCH_HEAD?\nFrom my quick test (by applying 01-08 on top of rc1, and messing with t7900),\nit looks like yes.\n\nI (and some other people, probably) rely on FETCH_HEAD for our scripts.\nHence, it would be nice to not touch FETCH_HEAD with prefetch job.\n\nThanks,\n-Danh\n\n> \n> 1. --no-tags prevents getting a new tag when a user wants to see\n> the new tags appear in their foreground fetches.\n> \n> 2. --refmap= removes the configured refspec which usually updates\n> refs/remotes//* with the refs advertised by the remote.\n> \n> 3. By adding a new refspec \"+refs/heads/*:refs/prefetch//*\"\n> we can ensure that we actually load the new values somewhere in\n> our refspace while not updating refs/heads or refs/remotes. By\n> storing these refs here, the commit-graph job will update the\n> commit-graph with the commits from these hidden refs.\n> \n> 4. --prune will delete the refs/prefetch/ refs that no\n> longer appear on the remote.\n> \n> We've been using this step as a critical background job in Scalar\n> [1] (and VFS for Git). This solved a pain point that was showing up\n> in user reports: fetching was a pain! Users do not like waiting to\n> download the data that was created while they were away from their\n> machines. After implementing background fetch, the foreground fetch\n> commands sped up significantly because they mostly just update refs\n> and download a small amount of new data. The effect is especially\n> dramatic when paried with --no-show-forced-udpates (through\n> fetch.showForcedUpdates=false).\n> \n> [1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 12 ++++++\n> builtin/gc.c | 64 ++++++++++++++++++++++++++++++-\n> t/t7900-maintenance.sh | 24 ++++++++++++\n> 3 files changed, 99 insertions(+), 1 deletion(-)\n> \n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 9204762e21..0927643247 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -53,6 +53,18 @@ since it will not expire `.graph` files that were in the previous\n> `commit-graph-chain` file. They will be deleted by a later run based on\n> the expiration delay.\n> \n> +prefetch::\n> +\tThe `fetch` task updates the object directory with the latest objects\n> +\tfrom all registered remotes. For each remote, a `git fetch` command\n> +\tis run. The refmap is custom to avoid updating local or remote\n> +\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n> +\tremote refs are stored in `refs/prefetch//`. Also, tags are\n> +\tnot updated.\n> ++\n> +This means that foreground fetches are still required to update the\n> +remote refs, but the users is notified when the branches and tags are\n> +updated on the remote.\n> +\n> gc::\n> \tCleanup unnecessary files and optimize the local repository. \"GC\"\n> \tstands for \"garbage collection,\" but this task performs many\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 5d99b4b805..969c127877 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -28,6 +28,7 @@\n> #include \"blob.h\"\n> #include \"tree.h\"\n> #include \"promisor-remote.h\"\n> +#include \"remote.h\"\n> \n> #define FAILED_RUN \"failed to run %s\"\n> \n> @@ -700,7 +701,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \treturn 0;\n> }\n> \n> -#define MAX_NUM_TASKS 2\n> +#define MAX_NUM_TASKS 3\n> \n> static const char * const builtin_maintenance_usage[] = {\n> \tN_(\"git maintenance run []\"),\n> @@ -781,6 +782,63 @@ static int maintenance_task_commit_graph(void)\n> \treturn 1;\n> }\n> \n> +static int fetch_remote(const char *remote)\n> +{\n> +\tint result;\n> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> +\tstruct strbuf refmap = STRBUF_INIT;\n> +\n> +\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n> +\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n> +\n> +\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n> +\targv_array_push(&cmd, refmap.buf);\n> +\n> +\tif (opts.quiet)\n> +\t\targv_array_push(&cmd, \"--quiet\");\n> +\n> +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n> +\n> +\tstrbuf_release(&refmap);\n> +\treturn result;\n> +}\n> +\n> +static int fill_each_remote(struct remote *remote, void *cbdata)\n> +{\n> +\tstruct string_list *remotes = (struct string_list *)cbdata;\n> +\n> +\tstring_list_append(remotes, remote->name);\n> +\treturn 0;\n> +}\n> +\n> +static int maintenance_task_prefetch(void)\n> +{\n> +\tint result = 0;\n> +\tstruct string_list_item *item;\n> +\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n> +\n> +\tif (for_each_remote(fill_each_remote, &remotes)) {\n> +\t\terror(_(\"failed to fill remotes\"));\n> +\t\tresult = 1;\n> +\t\tgoto cleanup;\n> +\t}\n> +\n> +\t/*\n> +\t * Do not modify the result based on the success of the 'fetch'\n> +\t * operation, as a loss of network could cause 'fetch' to fail\n> +\t * quickly. We do not want that to stop the rest of our\n> +\t * background operations.\n> +\t */\n> +\tfor (item = remotes.items;\n> +\t item && item < remotes.items + remotes.nr;\n> +\t item++)\n> +\t\tfetch_remote(item->string);\n> +\n> +cleanup:\n> +\tstring_list_clear(&remotes, 0);\n> +\treturn result;\n> +}\n> +\n> static int maintenance_task_gc(void)\n> {\n> \tint result;\n> @@ -871,6 +929,10 @@ static void initialize_tasks(void)\n> \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n> \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n> \n> +\ttasks[num_tasks]->name = \"prefetch\";\n> +\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n> +\tnum_tasks++;\n> +\n> \ttasks[num_tasks]->name = \"gc\";\n> \ttasks[num_tasks]->fn = maintenance_task_gc;\n> \ttasks[num_tasks]->enabled = 1;\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index c09a9eb90b..8b04a04c79 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -44,4 +44,28 @@ test_expect_success 'run --task duplicate' '\n> \ttest_i18ngrep \"cannot be selected multiple times\" err\n> '\n> \n> +test_expect_success 'run --task=prefetch with no remotes' '\n> +\tgit maintenance run --task=prefetch 2>err &&\n> +\ttest_must_be_empty err\n> +'\n> +\n> +test_expect_success 'prefetch multiple remotes' '\n> +\tgit clone . clone1 &&\n> +\tgit clone . clone2 &&\n> +\tgit remote add remote1 \"file://$(pwd)/clone1\" &&\n> +\tgit remote add remote2 \"file://$(pwd)/clone2\" &&\n> +\tgit -C clone1 switch -c one &&\n> +\tgit -C clone2 switch -c two &&\n> +\ttest_commit -C clone1 one &&\n> +\ttest_commit -C clone2 two &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch &&\n> +\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-prefetch.txt &&\n> +\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-prefetch.txt &&\n> +\ttest_path_is_missing .git/refs/remotes &&\n> +\ttest_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&\n> +\ttest_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&\n> +\tgit log prefetch/remote1/one &&\n> +\tgit log prefetch/remote2/two\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n\n-- \nDanh\n"},{"id":"402037","messageId":"20200725014647.GC35171@syl.lan","threadId":"53809","inReplyTo":"67e49b30-9a7f-8c18-3a06-96c516027c7a@gmail.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-07-25T01:46:47Z","receivedAt":"2020-07-25T01:46:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 24, 2020 at 08:51:32AM -0400, Derrick Stolee wrote:\n> On 7/24/2020 8:23 AM, Derrick Stolee wrote:\n> > On 7/23/2020 3:57 PM, Junio C Hamano wrote:\n> >> \"Derrick Stolee via GitGitGadget\" writes:\n> >>\n> >>> +static void initialize_tasks(void)\n> >>> +{\n> >>> +\tint i;\n> >>> +\tnum_tasks = 0;\n> >>> +\n> >>> +\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n> >>> +\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n> >>> +\n> >>> +\ttasks[num_tasks]->name = \"gc\";\n> >>> +\ttasks[num_tasks]->fn = maintenance_task_gc;\n> >>> +\ttasks[num_tasks]->enabled = 1;\n> >>> +\tnum_tasks++;\n> >>\n> >> Are we going to have 47 different tasks initialized by code like\n> >> this in the future? I would have expected that you'd have a table\n> >> of tasks that serves as the blueprint copy and copy it to the table\n> >> to be used if there is some need to mutate the table-to-be-used.\n> >\n> > Making it a table will likely make it easier to read. I hadn't\n> > thought of it.\n> >\n> > At the start, I thought that the diff would look awful as we add\n> > members to the struct. However, the members that are not specified\n> > are set to zero, so I should be able to craft this into something\n> > not too terrible.\n>\n> OK, my attempt has led to this final table:\n>\n> \tconst struct maintenance_task default_tasks[] = {\n> \t\t{\n> \t\t\t\"prefetch\",\n> \t\t\tmaintenance_task_prefetch,\n> \t\t},\n> \t\t{\n> \t\t\t\"loose-objects\",\n> \t\t\tmaintenance_task_loose_objects,\n> \t\t\tloose_object_auto_condition,\n> \t\t},\n> \t\t{\n> \t\t\t\"incremental-repack\",\n> \t\t\tmaintenance_task_incremental_repack,\n> \t\t\tincremental_repack_auto_condition,\n> \t\t},\n> \t\t{\n> \t\t\t\"gc\",\n> \t\t\tmaintenance_task_gc,\n> \t\t\tneed_to_gc,\n> \t\t\t1,\n> \t\t},\n> \t\t{\n> \t\t\t\"commit-graph\",\n> \t\t\tmaintenance_task_commit_graph,\n> \t\t\tshould_write_commit_graph,\n> \t\t}\n> \t};\n> \tnum_tasks = sizeof(default_tasks) / sizeof(struct maintenance_task);\n>\n> This is followed by allocating and copying the data to the\n> 'tasks' array, allowing it to be sorted and modified according\n> to command-line arguments and config.\n>\n> Is this what you intended?\n\nI'm not sure if Junio intended what I'm going to suggest, but I think\nthat you could make looking up these \"blueprint\" tasks a little easier\nby using the designated index initializer. For what it's worth, I wasn't\nsure if we allow this in the codebase, but some quick perusing through\nDocumentation/CodingGuidelines turns up 512f41cfac (clean.c: use\ndesignated initializer, 2017-07-14), which does use this style.\n\nMaybe something like:\n\n enum maintenance_task_kind {\n TASK_PREFETCH = 0,\n TASK_LOOSE_OBJECTS,\n /* ... */\n TASK__COUNT\n };\n\n const struct maintenance_task default_tasks[TASK__COUNT] = {\n [TASK_PREFETCH] = {\n \"prefetch\",\n maintenance_task_prefetch,\n },\n [...] = ...\n };\n\nand then you should be able to pick those out with\n'default_tasks[TASK_PREFETCH]'. I'm not sure if you are able to rely on\nthose tasks appearing in a certain order in which case you can feel free\nto discard this suggestion.\n\nIf nothing else, I'm glad that we can use the '[...] = '-style\ninitializers :-).\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"402038","messageId":"20200725014709.GB2436@danh.dev","threadId":"53809","inReplyTo":"63ec602a07756a41f8ccddd745562c567a4b3ed7.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-07-25T01:47:09Z","receivedAt":"2020-07-25T01:47:14Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-07-23 17:56:23+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> new file mode 100755\n> index 0000000000..d00641c4dd\n> --- /dev/null\n> +++ b/t/t7900-maintenance.sh\n> @@ -0,0 +1,22 @@\n> +#!/bin/sh\n> +\n> +test_description='git maintenance builtin'\n> +\n> +GIT_TEST_COMMIT_GRAPH=0\n> +GIT_TEST_MULTI_PACK_INDEX=0\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'help text' '\n> +\ttest_must_fail git maintenance -h 2>err &&\n> +\ttest_i18ngrep \"usage: git maintenance run\" err\n> +'\n\nI think this test has been tested better already in t0012,\nAnyway, if we would like to check the word \"git maintenance run\"\nexplicitly here, it would be clearer to\n\ns/test_must_fail/test_expect_code 129/\n\n> +\n> +test_expect_success 'gc [--auto]' '\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n> +\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n> +\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n> +'\n> +\n> +test_done\n> -- \n> gitgitgadget\n> \n\n-- \nDanh\n"},{"id":"402039","messageId":"xmqqv9icicvd.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"20200725013715.GA2436@danh.dev","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-25T01:48:06Z","receivedAt":"2020-07-25T01:48:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Đoàn Trần Công Danh writes:\n\n>> When running 'git fetch ' in the background, use\n>> the following options for careful updating:\n>\n> Does this job interfere with FETCH_HEAD?\n> From my quick test (by applying 01-08 on top of rc1, and messing with t7900),\n> it looks like yes.\n>\n> I (and some other people, probably) rely on FETCH_HEAD for our scripts.\n> Hence, it would be nice to not touch FETCH_HEAD with prefetch job.\n\nVery good point. For that, Derrick may want to swallow the single\npatch from 'jc/no-update-fetch-head' topic into this series and\npass the new command line option.\n\n\n\n\n"},{"id":"402040","messageId":"20200725015240.GD35171@syl.lan","threadId":"53809","inReplyTo":"xmqqy2n8lmq3.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-07-25T01:52:40Z","receivedAt":"2020-07-25T01:52:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jul 24, 2020 at 12:47:00PM -0700, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n> > But you are discussing here how the _behavior_ can change when\n> > --auto is specified. And specifically, \"git gc --auto\" really\n> > meant \"This is running after a foreground command, so only do\n> > work if necessary and do it quickly to minimize blocking time.\"\n> >\n> > I'd be happy to replace \"--auto\" with \"--quick\" in the\n> > maintenance builtin.\n> >\n> > This opens up some extra design space for how the individual\n> > tasks perform depending on \"--quick\" being specified or not.\n> > My intention was to create tasks that are already in \"quick\"\n> > mode:\n> >\n> > * loose-objects have a maximum batch size.\n> > * incremental-repack is capped in size.\n> > * commit-graph uses the --split option.\n> >\n> > But this \"quick\" distinction might be important for some of\n> > the tasks we intend to extract from the gc builtin.\n>\n> Yup. To be honest, I came to this topic from a completely different\n> direction. The field name \"auto\" alone (and no other field name)\n> had to have an extra cruft (i.e. \"_flag\") attached to it, which is\n> understandable but ugly. Then I started thinking if 'auto(matic)'\n> is really the right word to describe what we want out of the option,\n> and came to the realization that there may be better words.\n\nI wonder what the quick and slow paths are here. For the commit-graph\ncode, what you wrote here seems to match what I'd expect with passing\n'--auto' in the sense of running 'git gc'. That is, I'm leaving it up to\nthe commit-graph machinery's idea of the normal '--split' rules to\nfigure out when to roll up layers of a commit-graph, as opposed to\ncreating a new layer and extending the chain.\n\nSo, I think that makes sense if the caller gave '--auto'. But, I'm not\nsure that it makes sense if they didn't, in which case I'd imagine\nsomething quicker to happen. There, I'd expect something more like:\n\n 1. Run 'git commit-graph write --reachable --split=no-merge'.\n 2. Run 'git commit-graph verify'.\n 3. If 'git commit-graph verify' failed, drop the existing commit graph\n and rebuild it with 'git commit-graph --reachable --split=replace'.\n 4. Otherwise, do nothing.\n\nI'm biased, of course, but I think that that matches roughly what I'd\nexpect to happen in the fast/slow path. Granted, the steps to rebuild\nthe commit graph are going to be slow no matter what (depending on the\nsize of the repository), and so in that case maybe the commit-graph\nshould just be dropped. I'm not really sure what to do about that...\n\n> > Since the tasks are frequently running subcommands, returning\n> > 0 for success and non-zero for error matches the error codes\n> > returned by those subcommands.\n>\n> As long as these will _never_ be called from other helper functions\n> but from the cmd_foo() top-level and their return values are only\n> used directly as the top-level's return value, I do not mind too\n> much.\n>\n> But whenever I am writing such a code, I find myself not brave\n> enough to make such a bold promise (I saw other people call the\n> helpers I wrote in unintended ways and had to adjust the semantics\n> of them to accomodate the new callers too many times), so I'd rather\n> see the caller do \"return !!helper_fn()\" to allow helper_fn() to be\n> written more naturally (e.g. letting them return error(...)).\n>\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"402133","messageId":"421cb856-24fc-6aa0-6ab0-f513d34ddd1a@gmail.com","threadId":"53809","inReplyTo":"xmqqv9icicvd.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-27T14:07:50Z","receivedAt":"2020-07-27T14:41:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/24/2020 9:48 PM, Junio C Hamano wrote:\n> Đoàn Trần Công Danh writes:\n> \n>>> When running 'git fetch ' in the background, use\n>>> the following options for careful updating:\n>>\n>> Does this job interfere with FETCH_HEAD?\n>> From my quick test (by applying 01-08 on top of rc1, and messing with t7900),\n>> it looks like yes.\n>>\n>> I (and some other people, probably) rely on FETCH_HEAD for our scripts.\n>> Hence, it would be nice to not touch FETCH_HEAD with prefetch job.\n> \n> Very good point. For that, Derrick may want to swallow the single\n> patch from 'jc/no-update-fetch-head' topic into this series and\n> pass the new command line option.\n\nThanks for the pointer! I appreciate the attention to detail here.\n\nI'll rebase onto jc/no-update-fetch-head for the next version, since\nthat branch is based on v2.28.0-rc0, which is recent enough.\n\nThanks,\n-Stolee\n"},{"id":"402136","messageId":"xmqqy2n5gcls.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"421cb856-24fc-6aa0-6ab0-f513d34ddd1a@gmail.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-27T16:13:35Z","receivedAt":"2020-07-27T16:13:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I'll rebase onto jc/no-update-fetch-head for the next version, since\n> that branch is based on v2.28.0-rc0, which is recent enough.\n\nI do not think it is wise to base a work on top of unfinished \"you\ncould do it this way, perhaps?\" demonstration patch the original\nauthor does not have much inclination to finish, though.\n\nWhen I am really bored, I may go back to the topic to finish it, but\nI wouldn't mind if you took ownership of it at all.\n\nThanks.\n\n\n\n"},{"id":"402149","messageId":"5f201690-b090-1716-b175-562801891ee6@gmail.com","threadId":"53809","inReplyTo":"xmqqy2n5gcls.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 08/18] maintenance: add prefetch task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-27T18:27:17Z","receivedAt":"2020-07-27T18:27:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/27/2020 12:13 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> I'll rebase onto jc/no-update-fetch-head for the next version, since\n>> that branch is based on v2.28.0-rc0, which is recent enough.\n> \n> I do not think it is wise to base a work on top of unfinished \"you\n> could do it this way, perhaps?\" demonstration patch the original\n> author does not have much inclination to finish, though.\n> \n> When I am really bored, I may go back to the topic to finish it, but\n> I wouldn't mind if you took ownership of it at all.\n\nAh. I didn't understand the status of that branch. I'll pull it in\nto this topic.\n\nThanks,\n-Stolee\n\n"},{"id":"402213","messageId":"xmqqr1svegt4.fsf_-_@gitster.c.googlers.com","threadId":"53809","inReplyTo":"5f201690-b090-1716-b175-562801891ee6@gmail.com","subject":"[PATCH v2] fetch: optionally allow disabling FETCH_HEAD update","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-28T16:37:59Z","receivedAt":"2020-07-28T16:38:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 7/27/2020 12:13 PM, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>> \n>>> I'll rebase onto jc/no-update-fetch-head for the next version, since\n>>> that branch is based on v2.28.0-rc0, which is recent enough.\n>> \n>> I do not think it is wise to base a work on top of unfinished \"you\n>> could do it this way, perhaps?\" demonstration patch the original\n>> author does not have much inclination to finish, though.\n>> \n>> When I am really bored, I may go back to the topic to finish it, but\n>> I wouldn't mind if you took ownership of it at all.\n>\n> Ah. I didn't understand the status of that branch. I'll pull it in\n> to this topic.\n\nSo here is with one of the two things that I found missing in the\nfirst iteration of the patch: documentation.\n\nThe other thing that I found iffy (and still missing from this\nversion) was what should be done when \"git pull\" is explicitly given\nthe \"--no-write-fetch-head\" option.\n\nI think (but didn't check the recent code) that 'git pull' would\npass only known-to-make-sense command line options to underlying\n'git fetch', so it probably will barf with \"unknown option\", which\nis the best case. We might want to make it sure with a new test in\n5521. On the other hand, if we get anything other than \"no such\noption\", we may want to think if we want to \"fix\" it or just leave\nit inside \"if it hurts, don't do it\" territory.\n\nThanks. \n\nThe patch without doc was Reviewed-by: Taylor Blau \nbut this round has not been.\n\n-- >8 --\n\nIf you run fetch but record the result in remote-tracking branches,\nand either if you do nothing with the fetched refs (e.g. you are\nmerely mirroring) or if you always work from the remote-tracking\nrefs (e.g. you fetch and then merge origin/branchname separately),\nyou can get away with having no FETCH_HEAD at all.\n\nTeach \"git fetch\" a command line option \"--[no-]write-fetch-head\"\nand \"fetch.writeFetchHEAD\" configuration variable. Without either,\nthe default is to write FETCH_HEAD, and the usual rule that the\ncommand line option defeats configured default applies.\n\nNote that under \"--dry-run\" mode, FETCH_HEAD is never written;\notherwise you'd see list of objects in the file that you do not\nactually have. Passing `--fetch-write-head` does not force `git\nfetch` to write the file.\n\nAlso note that this option is explicitly passed when \"git pull\"\ninternally invokes \"git fetch\", so that those who configured their\n\"git fetch\" not to write FETCH_HEAD would not be able to break the\ncooperation between these two commands. \"git pull\" must see what\n\"git fetch\" got recorded in FETCH_HEAD to work correctly.\n\nSigned-off-by: Junio C Hamano \n---\n Documentation/config/fetch.txt | 7 ++++++\n Documentation/fetch-options.txt | 10 +++++++++\n builtin/fetch.c | 19 +++++++++++++---\n builtin/pull.c | 3 ++-\n t/t5510-fetch.sh | 39 +++++++++++++++++++++++++++++++--\n 5 files changed, 72 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex b20394038d..0aaa05e8c0 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -91,3 +91,10 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.writeFetchHEAD::\n+\tSetting it to false tells `git fetch` not to write the list\n+\tof remote refs fetched in the `FETCH_HEAD` file directly\n+\tunder `$GIT_DIR`. Can be countermanded from the command\n+\tline with the `--[no-]write-fetch-head` option. Defaults to\n+\ttrue.\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 6e2a160a47..6775e8499f 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -64,6 +64,16 @@ documented in linkgit:git-config[1].\n --dry-run::\n \tShow what would be done, without making any changes.\n \n+ifndef::git-pull[]\n+--[no-]write-fetch-head::\n+\tWrite the list of remote refs fetched in the `FETCH_HEAD`\n+\tfile directly under `$GIT_DIR`. This is the default unless\n+\tthe configuration variable `fetch.writeFetchHEAD` is set to\n+\tfalse. Passing `--no-write-fetch-head` from the command\n+\tline tells Git not to write the file. Under `--dry-run`\n+\toption, the file is never written.\n+endif::git-pull[]\n+\n -f::\n --force::\n \tWhen 'git fetch' is used with `:` refspec it may\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 82ac4be8a5..3ccf69753f 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -56,6 +56,7 @@ static int prune_tags = -1; /* unspecified */\n #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok;\n+static int write_fetch_head = 1;\n static int verbosity, deepen_relative, set_upstream;\n static int progress = -1;\n static int enable_auto_gc = 1;\n@@ -118,6 +119,10 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(k, \"fetch.writefetchhead\")) {\n+\t\twrite_fetch_head = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -162,6 +167,8 @@ static struct option builtin_fetch_options[] = {\n \t\t PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),\n \tOPT_BOOL(0, \"dry-run\", &dry_run,\n \t\t N_(\"dry run\")),\n+\tOPT_BOOL(0, \"write-fetch-head\", &write_fetch_head,\n+\t\t N_(\"write fetched references to the FETCH_HEAD file\")),\n \tOPT_BOOL('k', \"keep\", &keep, N_(\"keep downloaded pack\")),\n \tOPT_BOOL('u', \"update-head-ok\", &update_head_ok,\n \t\t N_(\"allow updating of HEAD ref\")),\n@@ -893,7 +900,9 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *what, *kind;\n \tstruct ref *rm;\n \tchar *url;\n-\tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head(the_repository);\n+\tconst char *filename = (!write_fetch_head\n+\t\t\t\t? \"/dev/null\"\n+\t\t\t\t: git_path_fetch_head(the_repository));\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n \n@@ -1327,7 +1336,7 @@ static int do_fetch(struct transport *transport,\n \t}\n \n \t/* if not appending, truncate FETCH_HEAD */\n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tretcode = truncate_fetch_head();\n \t\tif (retcode)\n \t\t\tgoto cleanup;\n@@ -1594,7 +1603,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \tint i, result = 0;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tint errcode = truncate_fetch_head();\n \t\tif (errcode)\n \t\t\treturn errcode;\n@@ -1795,6 +1804,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n+\t/* FETCH_HEAD never gets updated in --dry-run mode */\n+\tif (dry_run)\n+\t\twrite_fetch_head = 0;\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 8159c5d7c9..e988d92b53 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -527,7 +527,8 @@ static int run_fetch(const char *repo, const char **refspecs)\n \tstruct argv_array args = ARGV_ARRAY_INIT;\n \tint ret;\n \n-\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\", NULL);\n+\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\",\n+\t\t\t \"--write-fetch-head\", NULL);\n \n \t/* Shared options */\n \targv_push_verbosity(&args);\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex a66dbe0bde..3052c2d8d5 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -539,13 +539,48 @@ test_expect_success 'fetch into the current branch with --update-head-ok' '\n \n '\n \n-test_expect_success 'fetch --dry-run' '\n-\n+test_expect_success 'fetch --dry-run does not touch FETCH_HEAD' '\n \trm -f .git/FETCH_HEAD &&\n \tgit fetch --dry-run . &&\n \t! test -f .git/FETCH_HEAD\n '\n \n+test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success '--write-fetch-head gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --dry-run --write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --dry-run . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --no-write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch --write-fetch-head . &&\n+\ttest -f .git/FETCH_HEAD\n+'\n+\n test_expect_success \"should be able to fetch with duplicate refspecs\" '\n \tmkdir dups &&\n \t(\n-- \n2.28.0\n\n"},{"id":"402311","messageId":"20200729002231.GA1636139@coredump.intra.peff.net","threadId":"53809","inReplyTo":"04552b1d2ed751a11eb7c50f6898cbc078b552b4.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-07-29T00:22:31Z","receivedAt":"2020-07-29T00:22:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:27PM +0000, Derrick Stolee via GitGitGadget wrote:\n\n> +static int run_write_commit_graph(void)\n> +{\n> +\tint result;\n> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> +\n> +\targv_array_pushl(&cmd, \"commit-graph\", \"write\",\n> +\t\t\t \"--split\", \"--reachable\", NULL);\n> +\n> +\tif (opts.quiet)\n> +\t\targv_array_push(&cmd, \"--no-progress\");\n> +\n> +\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n> +\targv_array_clear(&cmd);\n> +\n> +\treturn result;\n> +}\n\nThis is a pretty minor nit, but since I happened to be looking at the\nmerge of all of the recent argv_array callers today... :)\n\nYou can write this a bit more succinctly by reusing the argv_array\nprovided by the child_process:\n\n struct child_process cmd = CHILD_PROCESS_INIT;\n\n cmd.git_cmd = 1;\n argv_array_pushl(&cmd.args, \"commit-graph\", \"write\",\n \"--split\", \"--reachable\", NULL);\n\n if (opts.quiet)\n argv_array_push(&cmd.args, \"--no-progress\");\n\n return run_command(&cmd);\n\nThen you don't have to worry about freeing the argv memory, because it's\nhandled automatically.\n\nLike I said, quite minor, but it looks like this pattern appears in a\nfew places, so it might be worth tweaking. And it would still work with\nthe \"pushf\" people recommended to avoid extra strbufs (I saw another one\nin fetch_remote(), too).\n\n-Peff\n"},{"id":"402328","messageId":"d1e18e79-ec06-4494-b806-38b06a0798d0@gmail.com","threadId":"53809","inReplyTo":"xmqqr1svegt4.fsf_-_@gitster.c.googlers.com","subject":"Re: [PATCH v2] fetch: optionally allow disabling FETCH_HEAD update","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-07-29T09:12:59Z","receivedAt":"2020-07-29T09:13:06Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 28/07/2020 17:37, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 7/27/2020 12:13 PM, Junio C Hamano wrote:\n>>> Derrick Stolee writes:\n>>>\n>>>> I'll rebase onto jc/no-update-fetch-head for the next version, since\n>>>> that branch is based on v2.28.0-rc0, which is recent enough.\n>>>\n>>> I do not think it is wise to base a work on top of unfinished \"you\n>>> could do it this way, perhaps?\" demonstration patch the original\n>>> author does not have much inclination to finish, though.\n>>>\n>>> When I am really bored, I may go back to the topic to finish it, but\n>>> I wouldn't mind if you took ownership of it at all.\n>>\n>> Ah. I didn't understand the status of that branch. I'll pull it in\n>> to this topic.\n> \n> So here is with one of the two things that I found missing in the\n> first iteration of the patch: documentation.\n> \n> The other thing that I found iffy (and still missing from this\n> version) was what should be done when \"git pull\" is explicitly given\n> the \"--no-write-fetch-head\" option.\n> \n> I think (but didn't check the recent code) that 'git pull' would\n> pass only known-to-make-sense command line options to underlying\n> 'git fetch', so it probably will barf with \"unknown option\", which\n> is the best case. We might want to make it sure with a new test in\n> 5521. On the other hand, if we get anything other than \"no such\n> option\", we may want to think if we want to \"fix\" it or just leave\n> it inside \"if it hurts, don't do it\" territory.\n> \n> Thanks.\n> \n> The patch without doc was Reviewed-by: Taylor Blau \n> but this round has not been.\n> \n> -- >8 --\n> \n> If you run fetch but record the result in remote-tracking branches,\n> and either if you do nothing with the fetched refs (e.g. you are\n> merely mirroring) or if you always work from the remote-tracking\n> refs (e.g. you fetch and then merge origin/branchname separately),\n> you can get away with having no FETCH_HEAD at all.\n> \n> Teach \"git fetch\" a command line option \"--[no-]write-fetch-head\"\n> and \"fetch.writeFetchHEAD\" configuration variable. Without either,\n> the default is to write FETCH_HEAD, and the usual rule that the\n> command line option defeats configured default applies.\n> \n> Note that under \"--dry-run\" mode, FETCH_HEAD is never written;\n> otherwise you'd see list of objects in the file that you do not\n> actually have. Passing `--fetch-write-head` \n\nTypo, it should be `--write-fetch-head`\n\n>does not force `git\n> fetch` to write the file.\n> \n> Also note that this option is explicitly passed when \"git pull\"\n> internally invokes \"git fetch\", so that those who configured their\n> \"git fetch\" not to write FETCH_HEAD would not be able to break the\n> cooperation between these two commands. \"git pull\" must see what\n> \"git fetch\" got recorded in FETCH_HEAD to work correctly.\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> Documentation/config/fetch.txt | 7 ++++++\n> Documentation/fetch-options.txt | 10 +++++++++\n> builtin/fetch.c | 19 +++++++++++++---\n> builtin/pull.c | 3 ++-\n> t/t5510-fetch.sh | 39 +++++++++++++++++++++++++++++++--\n> 5 files changed, 72 insertions(+), 6 deletions(-)\n> \n> diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\n> index b20394038d..0aaa05e8c0 100644\n> --- a/Documentation/config/fetch.txt\n> +++ b/Documentation/config/fetch.txt\n> @@ -91,3 +91,10 @@ fetch.writeCommitGraph::\n> \tmerge and the write may take longer. Having an updated commit-graph\n> \tfile helps performance of many Git commands, including `git merge-base`,\n> \t`git push -f`, and `git log --graph`. Defaults to false.\n> +\n> +fetch.writeFetchHEAD::\n> +\tSetting it to false tells `git fetch` not to write the list\n> +\tof remote refs fetched in the `FETCH_HEAD` file directly\n> +\tunder `$GIT_DIR`. Can be countermanded from the command\n> +\tline with the `--[no-]write-fetch-head` option. Defaults to\n> +\ttrue.\n> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\n> index 6e2a160a47..6775e8499f 100644\n> --- a/Documentation/fetch-options.txt\n> +++ b/Documentation/fetch-options.txt\n> @@ -64,6 +64,16 @@ documented in linkgit:git-config[1].\n> --dry-run::\n> \tShow what would be done, without making any changes.\n> \n> +ifndef::git-pull[]\n> +--[no-]write-fetch-head::\n> +\tWrite the list of remote refs fetched in the `FETCH_HEAD`\n> +\tfile directly under `$GIT_DIR`. This is the default unless\n> +\tthe configuration variable `fetch.writeFetchHEAD` is set to\n> +\tfalse. Passing `--no-write-fetch-head` from the command\n> +\tline tells Git not to write the file. Under `--dry-run`\n> +\toption, the file is never written.\n> +endif::git-pull[]\n> +\n> -f::\n> --force::\n> \tWhen 'git fetch' is used with `:` refspec it may\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 82ac4be8a5..3ccf69753f 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -56,6 +56,7 @@ static int prune_tags = -1; /* unspecified */\n> #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */\n> \n> static int all, append, dry_run, force, keep, multiple, update_head_ok;\n> +static int write_fetch_head = 1;\n> static int verbosity, deepen_relative, set_upstream;\n> static int progress = -1;\n> static int enable_auto_gc = 1;\n> @@ -118,6 +119,10 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n> \t\treturn 0;\n> \t}\n> \n> +\tif (!strcmp(k, \"fetch.writefetchhead\")) {\n> +\t\twrite_fetch_head = git_config_bool(k, v);\n> +\t\treturn 0;\n> +\t}\n> \treturn git_default_config(k, v, cb);\n> }\n> \n> @@ -162,6 +167,8 @@ static struct option builtin_fetch_options[] = {\n> \t\t PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),\n> \tOPT_BOOL(0, \"dry-run\", &dry_run,\n> \t\t N_(\"dry run\")),\n> +\tOPT_BOOL(0, \"write-fetch-head\", &write_fetch_head,\n> +\t\t N_(\"write fetched references to the FETCH_HEAD file\")),\n> \tOPT_BOOL('k', \"keep\", &keep, N_(\"keep downloaded pack\")),\n> \tOPT_BOOL('u', \"update-head-ok\", &update_head_ok,\n> \t\t N_(\"allow updating of HEAD ref\")),\n> @@ -893,7 +900,9 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \tconst char *what, *kind;\n> \tstruct ref *rm;\n> \tchar *url;\n> -\tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head(the_repository);\n> +\tconst char *filename = (!write_fetch_head\n> +\t\t\t\t? \"/dev/null\"\n> +\t\t\t\t: git_path_fetch_head(the_repository));\n\nI was suspicious of this as we haven't cleared write_fetch_head in the \n--dry-run case yet but the test below seems to show that we still don't \nwrite FETCH_HEAD if --dry-run is given. That makes we wonder what the \npoint of setting the filename based on the value of write_fetch_head is.\n\nThe rest looks good to me, though it might be worth having a test to \ncheck that pull does indeed reject --no-write-fetch-head\n\nBest Wishes\n\nPhillip\n\n> \tint want_status;\n> \tint summary_width = transport_summary_width(ref_map);\n> \n> @@ -1327,7 +1336,7 @@ static int do_fetch(struct transport *transport,\n> \t}\n> \n> \t/* if not appending, truncate FETCH_HEAD */\n> -\tif (!append && !dry_run) {\n> +\tif (!append && write_fetch_head) {\n> \t\tretcode = truncate_fetch_head();\n> \t\tif (retcode)\n> \t\t\tgoto cleanup;\n> @@ -1594,7 +1603,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n> \tint i, result = 0;\n> \tstruct argv_array argv = ARGV_ARRAY_INIT;\n> \n> -\tif (!append && !dry_run) {\n> +\tif (!append && write_fetch_head) {\n> \t\tint errcode = truncate_fetch_head();\n> \t\tif (errcode)\n> \t\t\treturn errcode;\n> @@ -1795,6 +1804,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tif (depth || deepen_since || deepen_not.nr)\n> \t\tdeepen = 1;\n> \n> +\t/* FETCH_HEAD never gets updated in --dry-run mode */\n> +\tif (dry_run)\n> +\t\twrite_fetch_head = 0;\n> +\n> \tif (all) {\n> \t\tif (argc == 1)\n> \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n> diff --git a/builtin/pull.c b/builtin/pull.c\n> index 8159c5d7c9..e988d92b53 100644\n> --- a/builtin/pull.c\n> +++ b/builtin/pull.c\n> @@ -527,7 +527,8 @@ static int run_fetch(const char *repo, const char **refspecs)\n> \tstruct argv_array args = ARGV_ARRAY_INIT;\n> \tint ret;\n> \n> -\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\", NULL);\n> +\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\",\n> +\t\t\t \"--write-fetch-head\", NULL);\n> \n> \t/* Shared options */\n> \targv_push_verbosity(&args);\n> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n> index a66dbe0bde..3052c2d8d5 100755\n> --- a/t/t5510-fetch.sh\n> +++ b/t/t5510-fetch.sh\n> @@ -539,13 +539,48 @@ test_expect_success 'fetch into the current branch with --update-head-ok' '\n> \n> '\n> \n> -test_expect_success 'fetch --dry-run' '\n> -\n> +test_expect_success 'fetch --dry-run does not touch FETCH_HEAD' '\n> \trm -f .git/FETCH_HEAD &&\n> \tgit fetch --dry-run . &&\n> \t! test -f .git/FETCH_HEAD\n> '\n> \n> +test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit fetch --no-write-fetch-head . &&\n> +\t! test -f .git/FETCH_HEAD\n> +'\n> +\n> +test_expect_success '--write-fetch-head gets defeated by --dry-run' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit fetch --dry-run --write-fetch-head . &&\n> +\t! test -f .git/FETCH_HEAD\n> +'\n> +\n> +test_expect_success 'fetch.writeFetchHEAD and FETCH_HEAD' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit -c fetch.writeFetchHEAD=no fetch . &&\n> +\t! test -f .git/FETCH_HEAD\n> +'\n> +\n> +test_expect_success 'fetch.writeFetchHEAD gets defeated by --dry-run' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit -c fetch.writeFetchHEAD=yes fetch --dry-run . &&\n> +\t! test -f .git/FETCH_HEAD\n> +'\n> +\n> +test_expect_success 'fetch.writeFetchHEAD and --no-write-fetch-head' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit -c fetch.writeFetchHEAD=yes fetch --no-write-fetch-head . &&\n> +\t! test -f .git/FETCH_HEAD\n> +'\n> +\n> +test_expect_success 'fetch.writeFetchHEAD and --write-fetch-head' '\n> +\trm -f .git/FETCH_HEAD &&\n> +\tgit -c fetch.writeFetchHEAD=no fetch --write-fetch-head . &&\n> +\ttest -f .git/FETCH_HEAD\n> +'\n> +\n> test_expect_success \"should be able to fetch with duplicate refspecs\" '\n> \tmkdir dups &&\n> \t(\n> \n"},{"id":"402329","messageId":"7c9592c0-1079-56b7-f456-6d2583053034@gmail.com","threadId":"53809","inReplyTo":"d1e18e79-ec06-4494-b806-38b06a0798d0@gmail.com","subject":"Re: [PATCH v2] fetch: optionally allow disabling FETCH_HEAD update","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2020-07-29T09:17:48Z","receivedAt":"2020-07-29T09:17:54Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 29/07/2020 10:12, Phillip Wood wrote:\n> On 28/07/2020 17:37, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>> On 7/27/2020 12:13 PM, Junio C Hamano wrote:\n>>>> Derrick Stolee writes:\n>>>>\n>>>>> I'll rebase onto jc/no-update-fetch-head for the next version, since\n>>>>> that branch is based on v2.28.0-rc0, which is recent enough.\n>>>>\n>>>> I do not think it is wise to base a work on top of unfinished \"you\n>>>> could do it this way, perhaps?\" demonstration patch the original\n>>>> author does not have much inclination to finish, though.\n>>>>\n>>>> When I am really bored, I may go back to the topic to finish it, but\n>>>> I wouldn't mind if you took ownership of it at all.\n>>>\n>>> Ah. I didn't understand the status of that branch. I'll pull it in\n>>> to this topic.\n>>\n>> So here is with one of the two things that I found missing in the\n>> first iteration of the patch: documentation.\n>>\n>> The other thing that I found iffy (and still missing from this\n>> version) was what should be done when \"git pull\" is explicitly given\n>> the \"--no-write-fetch-head\" option.\n>>\n>> I think (but didn't check the recent code) that 'git pull' would\n>> pass only known-to-make-sense command line options to underlying\n>> 'git fetch', so it probably will barf with \"unknown option\", which\n>> is the best case.  We might want to make it sure with a new test in\n>> 5521.  On the other hand, if we get anything other than \"no such\n>> option\", we may want to think if we want to \"fix\" it or just leave\n>> it inside \"if it hurts, don't do it\" territory.\n>>\n>> Thanks.\n>>\n>> The patch without doc was Reviewed-by: Taylor Blau \n>> but this round has not been.\n>>\n>> -- >8 --\n>>\n>> If you run fetch but record the result in remote-tracking branches,\n>> and either if you do nothing with the fetched refs (e.g. you are\n>> merely mirroring) or if you always work from the remote-tracking\n>> refs (e.g. you fetch and then merge origin/branchname separately),\n>> you can get away with having no FETCH_HEAD at all.\n>>\n>> Teach \"git fetch\" a command line option \"--[no-]write-fetch-head\"\n>> and \"fetch.writeFetchHEAD\" configuration variable.  Without either,\n>> the default is to write FETCH_HEAD, and the usual rule that the\n>> command line option defeats configured default applies.\n>>\n>> Note that under \"--dry-run\" mode, FETCH_HEAD is never written;\n>> otherwise you'd see list of objects in the file that you do not\n>> actually have.  Passing `--fetch-write-head` \n> \n> Typo, it should be `--write-fetch-head`\n> \n>> does not force `git\n>> fetch` to write the file.\n>>\n>> Also note that this option is explicitly passed when \"git pull\"\n>> internally invokes \"git fetch\", so that those who configured their\n>> \"git fetch\" not to write FETCH_HEAD would not be able to break the\n>> cooperation between these two commands.  \"git pull\" must see what\n>> \"git fetch\" got recorded in FETCH_HEAD to work correctly.\n>>\n>> Signed-off-by: Junio C Hamano \n>> ---\n>>   Documentation/config/fetch.txt  |  7 ++++++\n>>   Documentation/fetch-options.txt | 10 +++++++++\n>>   builtin/fetch.c                 | 19 +++++++++++++---\n>>   builtin/pull.c                  |  3 ++-\n>>   t/t5510-fetch.sh                | 39 +++++++++++++++++++++++++++++++--\n>>   5 files changed, 72 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/Documentation/config/fetch.txt \n>> b/Documentation/config/fetch.txt\n>> index b20394038d..0aaa05e8c0 100644\n>> --- a/Documentation/config/fetch.txt\n>> +++ b/Documentation/config/fetch.txt\n>> @@ -91,3 +91,10 @@ fetch.writeCommitGraph::\n>>       merge and the write may take longer. Having an updated commit-graph\n>>       file helps performance of many Git commands, including `git \n>> merge-base`,\n>>       `git push -f`, and `git log --graph`. Defaults to false.\n>> +\n>> +fetch.writeFetchHEAD::\n>> +    Setting it to false tells `git fetch` not to write the list\n>> +    of remote refs fetched in the `FETCH_HEAD` file directly\n>> +    under `$GIT_DIR`.  Can be countermanded from the command\n>> +    line with the `--[no-]write-fetch-head` option.  Defaults to\n>> +    true.\n>> diff --git a/Documentation/fetch-options.txt \n>> b/Documentation/fetch-options.txt\n>> index 6e2a160a47..6775e8499f 100644\n>> --- a/Documentation/fetch-options.txt\n>> +++ b/Documentation/fetch-options.txt\n>> @@ -64,6 +64,16 @@ documented in linkgit:git-config[1].\n>>   --dry-run::\n>>       Show what would be done, without making any changes.\n>> +ifndef::git-pull[]\n>> +--[no-]write-fetch-head::\n>> +    Write the list of remote refs fetched in the `FETCH_HEAD`\n>> +    file directly under `$GIT_DIR`.  This is the default unless\n>> +    the configuration variable `fetch.writeFetchHEAD` is set to\n>> +    false.  Passing `--no-write-fetch-head` from the command\n>> +    line tells Git not to write the file.  Under `--dry-run`\n>> +    option, the file is never written.\n>> +endif::git-pull[]\n>> +\n>>   -f::\n>>   --force::\n>>       When 'git fetch' is used with `:` refspec it may\n>> diff --git a/builtin/fetch.c b/builtin/fetch.c\n>> index 82ac4be8a5..3ccf69753f 100644\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -56,6 +56,7 @@ static int prune_tags = -1; /* unspecified */\n>>   #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */\n>>   static int all, append, dry_run, force, keep, multiple, update_head_ok;\n>> +static int write_fetch_head = 1;\n>>   static int verbosity, deepen_relative, set_upstream;\n>>   static int progress = -1;\n>>   static int enable_auto_gc = 1;\n>> @@ -118,6 +119,10 @@ static int git_fetch_config(const char *k, const \n>> char *v, void *cb)\n>>           return 0;\n>>       }\n>> +    if (!strcmp(k, \"fetch.writefetchhead\")) {\n>> +        write_fetch_head = git_config_bool(k, v);\n>> +        return 0;\n>> +    }\n>>       return git_default_config(k, v, cb);\n>>   }\n>> @@ -162,6 +167,8 @@ static struct option builtin_fetch_options[] = {\n>>               PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),\n>>       OPT_BOOL(0, \"dry-run\", &dry_run,\n>>            N_(\"dry run\")),\n>> +    OPT_BOOL(0, \"write-fetch-head\", &write_fetch_head,\n>> +         N_(\"write fetched references to the FETCH_HEAD file\")),\n>>       OPT_BOOL('k', \"keep\", &keep, N_(\"keep downloaded pack\")),\n>>       OPT_BOOL('u', \"update-head-ok\", &update_head_ok,\n>>               N_(\"allow updating of HEAD ref\")),\n>> @@ -893,7 +900,9 @@ static int store_updated_refs(const char *raw_url, \n>> const char *remote_name,\n>>       const char *what, *kind;\n>>       struct ref *rm;\n>>       char *url;\n>> -    const char *filename = dry_run ? \"/dev/null\" : \n>> git_path_fetch_head(the_repository);\n>> +    const char *filename = (!write_fetch_head\n>> +                ? \"/dev/null\"\n>> +                : git_path_fetch_head(the_repository));\n> \n> I was suspicious of this as we haven't cleared write_fetch_head in the \n> --dry-run case yet but the test below seems to show that we still don't \n> write FETCH_HEAD if --dry-run is given. That makes we wonder what the \n> point of setting the filename based on the value of write_fetch_head is.\n\nSorry ignore that. I misread the hunk header - we have in fact cleared \nwrite_fetch_head in the --dry-run case by the time we get here.\n\n> The rest looks good to me, though it might be worth having a test to \n> check that pull does indeed reject --no-write-fetch-head\n> \n> Best Wishes\n> \n> Phillip\n> \n>>       int want_status;\n>>       int summary_width = transport_summary_width(ref_map);\n>> @@ -1327,7 +1336,7 @@ static int do_fetch(struct transport *transport,\n>>       }\n>>       /* if not appending, truncate FETCH_HEAD */\n>> -    if (!append && !dry_run) {\n>> +    if (!append && write_fetch_head) {\n>>           retcode = truncate_fetch_head();\n>>           if (retcode)\n>>               goto cleanup;\n>> @@ -1594,7 +1603,7 @@ static int fetch_multiple(struct string_list \n>> *list, int max_children)\n>>       int i, result = 0;\n>>       struct argv_array argv = ARGV_ARRAY_INIT;\n>> -    if (!append && !dry_run) {\n>> +    if (!append && write_fetch_head) {\n>>           int errcode = truncate_fetch_head();\n>>           if (errcode)\n>>               return errcode;\n>> @@ -1795,6 +1804,10 @@ int cmd_fetch(int argc, const char **argv, \n>> const char *prefix)\n>>       if (depth || deepen_since || deepen_not.nr)\n>>           deepen = 1;\n>> +    /* FETCH_HEAD never gets updated in --dry-run mode */\n>> +    if (dry_run)\n>> +        write_fetch_head = 0;\n>> +\n>>       if (all) {\n>>           if (argc == 1)\n>>               die(_(\"fetch --all does not take a repository argument\"));\n>> diff --git a/builtin/pull.c b/builtin/pull.c\n>> index 8159c5d7c9..e988d92b53 100644\n>> --- a/builtin/pull.c\n>> +++ b/builtin/pull.c\n>> @@ -527,7 +527,8 @@ static int run_fetch(const char *repo, const char \n>> **refspecs)\n>>       struct argv_array args = ARGV_ARRAY_INIT;\n>>       int ret;\n>> -    argv_array_pushl(&args, \"fetch\", \"--update-head-ok\", NULL);\n>> +    argv_array_pushl(&args, \"fetch\", \"--update-head-ok\",\n>> +             \"--write-fetch-head\", NULL);\n>>       /* Shared options */\n>>       argv_push_verbosity(&args);\n>> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n>> index a66dbe0bde..3052c2d8d5 100755\n>> --- a/t/t5510-fetch.sh\n>> +++ b/t/t5510-fetch.sh\n>> @@ -539,13 +539,48 @@ test_expect_success 'fetch into the current \n>> branch with --update-head-ok' '\n>>   '\n>> -test_expect_success 'fetch --dry-run' '\n>> -\n>> +test_expect_success 'fetch --dry-run does not touch FETCH_HEAD' '\n>>       rm -f .git/FETCH_HEAD &&\n>>       git fetch --dry-run . &&\n>>       ! test -f .git/FETCH_HEAD\n>>   '\n>> +test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git fetch --no-write-fetch-head . &&\n>> +    ! test -f .git/FETCH_HEAD\n>> +'\n>> +\n>> +test_expect_success '--write-fetch-head gets defeated by --dry-run' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git fetch --dry-run --write-fetch-head . &&\n>> +    ! test -f .git/FETCH_HEAD\n>> +'\n>> +\n>> +test_expect_success 'fetch.writeFetchHEAD and FETCH_HEAD' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git -c fetch.writeFetchHEAD=no fetch . &&\n>> +    ! test -f .git/FETCH_HEAD\n>> +'\n>> +\n>> +test_expect_success 'fetch.writeFetchHEAD gets defeated by --dry-run' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git -c fetch.writeFetchHEAD=yes fetch --dry-run . &&\n>> +    ! test -f .git/FETCH_HEAD\n>> +'\n>> +\n>> +test_expect_success 'fetch.writeFetchHEAD and --no-write-fetch-head' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git -c fetch.writeFetchHEAD=yes fetch --no-write-fetch-head . &&\n>> +    ! test -f .git/FETCH_HEAD\n>> +'\n>> +\n>> +test_expect_success 'fetch.writeFetchHEAD and --write-fetch-head' '\n>> +    rm -f .git/FETCH_HEAD &&\n>> +    git -c fetch.writeFetchHEAD=no fetch --write-fetch-head . &&\n>> +    test -f .git/FETCH_HEAD\n>> +'\n>> +\n>>   test_expect_success \"should be able to fetch with duplicate refspecs\" '\n>>       mkdir dups &&\n>>       (\n>>\n"},{"id":"402387","messageId":"20200729220302.GA2965447@google.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/18] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-29T22:03:02Z","receivedAt":"2020-07-29T22:03:13Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:22PM +0000, Derrick Stolee via GitGitGadget wrote:\n> UPDATES in V2\n> =============\n> \n> I'm sending this between v2.28.0-rc2 adn v2.28.0 as the release things have\n> become a bit quiet.\n> \n> * The biggest disruption to the range-diff is that I removed the premature\n> use of struct repository *r and instead continue to rely on \n> the_repository. This means several patches were dropped that did prep\n> work in builtin/gc.c.\n> \n> \n> * I dropped the task hashmap and opted for a linear scan. This task list\n> will always be too small to justify the extra complication of the\n> hashmap.\n> \n> \n> * struct maintenance_opts is properly static now.\n> \n> \n> * Some tasks are renamed: fetch -> prefetch, pack-files ->\n> incremental-repack.\n> \n> \n> * With the rename, the prefetch task uses refs/prefetch/ instead of \n> refs/hidden/.\n> \n> \n> * A trace2 region around the task executions are added.\n> \n> \n> \n> Thanks, -Stolee\n\nFYI: We covered this series in review club (me, Jonathan Tan, Jonathan\nNieder) so I'll send some mails with comments from all three of us.\n\nWe noticed reviewing that it looks like it's time to focus on the\ndetails because the big picture of this series looks good. Since we were\nable to review each patch independently, the approach you took to\nseparate the commits seems effective.\n\nOne concern I (Emily) had was about the use of midx without necessarily\nchecking the config, where users can opt out of using it. What are the\nconsequences of running maintenance that creates a multipack index\nwithout the config turned on for the rest of Git?\n\n - Emily\n"},{"id":"402388","messageId":"20200729221905.GB519065@google.com","threadId":"53809","inReplyTo":"63ec602a07756a41f8ccddd745562c567a4b3ed7.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-07-29T22:19:05Z","receivedAt":"2020-07-29T22:19:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDerrick Stolee wrote:\n\n> [Subject: maintenance: create basic maintenance runner]\n\nSeems sensible, and a good way to set up for the later patches. Let's\ntake a look at how it does that.\n\n[...]\n> --- /dev/null\n> +++ b/Documentation/git-maintenance.txt\n> @@ -0,0 +1,57 @@\n[...]\n> +SUBCOMMANDS\n> +-----------\n> +\n> +run::\n> +\tRun one or more maintenance tasks.\n\n[jrnieder] How do I supply the tasks on the command line? Are they\nparameters to this subcommand? If so, it could make sense for this to\nsay something like\n\n\trun ...::\n\nWhat is the exit code convention for \"git maintenance run\"? (Many Git\ncommands don't document this, but since we're making a new command\nit seems worth building the habit.)\n\n[...]\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -699,3 +699,62 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n[...]\n> +static struct maintenance_opts {\n> +\tint auto_flag;\n> +} opts;\n\nPacking this in a struct feels a bit unusual. Is the struct going to\nbe passed somewhere, or could these be individual locals in\ncmd_maintenance?\n\n[...]\n> +\n> +static int maintenance_task_gc(void)\n> +{\n> +\tint result;\n> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n> +\n> +\targv_array_pushl(&cmd, \"gc\", NULL);\n> +\n> +\tif (opts.auto_flag)\n> +\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n\nThese are both pushing single strings, so they can use argv_array_push.\n\n[...]\n> --- /dev/null\n> +++ b/t/t7900-maintenance.sh\n> @@ -0,0 +1,22 @@\n> +#!/bin/sh\n> +\n> +test_description='git maintenance builtin'\n> +\n> +GIT_TEST_COMMIT_GRAPH=0\n> +GIT_TEST_MULTI_PACK_INDEX=0\n\nWhy does this disable commit graph and multipack index? Is that setting\nup for something that comes later?\n\n[...]\n> +test_expect_success 'gc [--auto]' '\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n> +\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n> +\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n\nThis feels a little odd in two ways:\n\n- the use of \"git gc\" isn't a user-facing detail, so this is testing\n implementation instead of the user-facing behavior. That's okay,\n though --- it can be useful to test internals sometimes.\n\n- the way that this tests for \"git gc\" feels brittle: if the trace\n emitter changes some day to include a space after the comma, for\n example, then the test would start failing. I thought that in the\n spirit of fakes, it could make sense to write a custom \"git gc\"\n command using test_write_script, but that isn't likely to work\n either since gc is a builtin.\n\nPerhaps this is suggesting that we need some central test helper for\nparsing traces so we can do this reliably in one place. What do you\nthink?\n\nThanks,\nJonathan\n"},{"id":"402389","messageId":"20200729221939.GB2965447@google.com","threadId":"53809","inReplyTo":"8e260bccf1a0b6cd799a6bc78798b31ebed8ad7e.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/18] maintenance: initialize task array","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-29T22:19:39Z","receivedAt":"2020-07-29T22:19:47Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:26PM +0000, Derrick Stolee via GitGitGadget wrote:\n> +static void initialize_tasks(void)\n> +{\n> +\tint i;\n> +\tnum_tasks = 0;\n> +\n> +\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n> +\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n\n[jonathan tan] I wonder why this is an array of pointers, not of\nobjects? If they were objects, we could use an initializer. But in a\nlater commit I see the array is sorted, OK.\n> +\n> +\ttasks[num_tasks]->name = \"gc\";\n> +\ttasks[num_tasks]->fn = maintenance_task_gc;\n> +\ttasks[num_tasks]->enabled = 1;\n> +\tnum_tasks++;\n"},{"id":"402390","messageId":"20200729222126.GC2965447@google.com","threadId":"53809","inReplyTo":"83648f48655ba68126110018d81c1d2e2bcc7a6f.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-29T22:21:26Z","receivedAt":"2020-07-29T22:21:35Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:31PM +0000, Derrick Stolee via GitGitGadget wrote:\n> +loose-objects::\n> +\tThe `loose-objects` job cleans up loose objects and places them into\n> +\tpack-files. In order to prevent race conditions with concurrent Git\n> +\tcommands, it follows a two-step process. First, it deletes any loose\n> +\tobjects that already exist in a pack-file; concurrent Git processes\n> +\twill examine the pack-file for the object data instead of the loose\n> +\tobject. Second, it creates a new pack-file (starting with \"loose-\")\n\n[jonathan tan + jonathan nieder] If you are going to document this,\nprobably it should also be tested, so the documentation does not become\nstale. Or, just don't document it.\n\n> +static int pack_loose(void)\n> +{\n> +\tstruct repository *r = the_repository;\n> +\tint result = 0;\n> +\tstruct write_loose_object_data data;\n> +\tstruct strbuf prefix = STRBUF_INIT;\n> +\tstruct child_process *pack_proc;\n> +\n> +\t/*\n> +\t * Do not start pack-objects process\n> +\t * if there are no loose objects.\n> +\t */\n> +\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n> +\t\t\t\t\t loose_object_exists,\n> +\t\t\t\t\t NULL, NULL, NULL))\n\n[emily] To me, this is unintuitive - but upon inspection, it's exiting\nthe foreach early if any loose object is found, so this is cheaper than\nactually counting. Maybe a comment would help to understand? Or we could\nname the function differently, like \"bail_if_loose()\" or something?\n\n> +test_expect_success 'loose-objects task' '\n> +\t# Repack everything so we know the state of the object dir\n> +\tgit repack -adk &&\n> +\n> +\t# Hack to stop maintenance from running during \"git commit\"\n> +\techo in use >.git/objects/maintenance.lock &&\n> +\ttest_commit create-loose-object &&\n\n[jonathan nieder] Does it make sense to use a different git command\nwhich is guaranteed to make a loose object? Is 'git commit' futureproof,\nif we decide commits should directly create packs in the future? 'git\nunpack-objects' is guaranteed to make a loose object, although it is\nclumsy because it needs a packfile to begin with...\n\n[jonathan tan] But, using 'git commit' is easier to understand in this\ncontext. Maybe commenting to say that we assume 'git commit' makes 1 or\nmore loose objects will be enough to futureproof - then we have a signal\nto whoever made a change to make this fail, and that person knows how to\nfix this test.\n"},{"id":"402391","messageId":"20200729222226.GD2965447@google.com","threadId":"53809","inReplyTo":"b6328c210625e1ba98e2065208a2a478c2c64f94.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/18] maintenance: add incremental-repack task","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-29T22:22:26Z","receivedAt":"2020-07-29T22:22:35Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:32PM +0000, Derrick Stolee via GitGitGadget wrote:\n> 1. 'git multi-pack-index write' creates a multi-pack-index file if\n> one did not exist, and otherwise will update the multi-pack-index\n> with any new pack-files that appeared since the last write. This\n> is particularly relevant with the background fetch job.\n\n[emily shaffer] Will this use midx even if the user has disabled it in\ntheir config?\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 94bb493733..3ec813979a 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -103,4 +103,41 @@ test_expect_success 'loose-objects task' '\n> \ttest_cmp packs-between packs-after\n> '\n\n[emily shaffer] Can we include a test to prove that this task is or is\nnot run if core.multipackindex is set or unset? That behavior is hard to\ndeduce from the code... we might want to be cautious.\n\n> \n> +test_expect_success 'incremental-repack task' '\n> +\tpackDir=.git/objects/pack &&\n> +\tfor i in $(test_seq 1 5)\n> +\tdo\n> +\t\ttest_commit $i || return 1\n> +\tdone &&\n> +\n> +\t# Create three disjoint pack-files with size BIG, small, small.\n> +\techo HEAD~2 | git pack-objects --revs $packDir/test-1 &&\n> +\ttest_tick &&\n> +\tgit pack-objects --revs $packDir/test-2 <<-\\EOF &&\n> +\tHEAD~1\n> +\t^HEAD~2\n> +\tEOF\n> +\ttest_tick &&\n> +\tgit pack-objects --revs $packDir/test-3 <<-\\EOF &&\n> +\tHEAD\n> +\t^HEAD~1\n> +\tEOF\n> +\trm -f $packDir/pack-* &&\n> +\trm -f $packDir/loose-* &&\n> +\tls $packDir/*.pack >packs-before &&\n> +\ttest_line_count = 3 packs-before &&\n> +\n> +\t# the job repacks the two into a new pack, but does not\n> +\t# delete the old ones.\n> +\tgit maintenance run --task=incremental-repack &&\n> +\tls $packDir/*.pack >packs-between &&\n> +\ttest_line_count = 4 packs-between &&\n> +\n> +\t# the job deletes the two old packs, and does not write\n> +\t# a new one because only one pack remains.\n> +\tgit maintenance run --task=incremental-repack &&\n> +\tls .git/objects/pack/*.pack >packs-after &&\n> +\ttest_line_count = 1 packs-after\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"402393","messageId":"20200729222334.GE2965447@google.com","threadId":"53809","inReplyTo":"478c7f1d0b858755c2c4b98605405214910b6f4c.1595527000.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Emily Shaffer","fromEmail":"emilyshaffer@google.com","sentAt":"2020-07-29T22:23:34Z","receivedAt":"2020-07-29T22:23:42Z","isPatch":true,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Thu, Jul 23, 2020 at 05:56:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index eb4b01c104..889d97afe7 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n> \treturn result;\n> }\n> \n> +#define TWO_GIGABYTES (2147483647)\n\n[jonathan tan] This would be easier to understand if it was expressed\nwith bitshift, e.g. 1 << 31\n\n> +#define UNSET_BATCH_SIZE ((unsigned long)-1)\n[jonathan tan] This looks like it's never used... and vulnerable to\ncross-platform size changes because it's referring to an implicitly\nsized int, and could behave differently if it was put into a larger\nsize, e.g. you wouldn't get 0xFFFF.. if you assigned this into a long\nlong.\n\n> +\tfor (p = get_all_packs(r); p; p = p->next) {\n> +\t\tif (p->pack_size > max_size) {\n> +\t\t\tsecond_largest_size = max_size;\n> +\t\t\tmax_size = p->pack_size;\n> +\t\t} else if (p->pack_size > second_largest_size)\n> +\t\t\tsecond_largest_size = p->pack_size;\n> +\t}\n> +\n> +\tresult_size = second_largest_size + 1;\n[jonathan tan] What happens when there's only one packfile, and when\nthere are two packfiles? Can we write tests to illustrate the behavior?\nThe edge case here (result_size=1) is hard to understand by reading the\ncode.\n\n> +\n> +\t/* But limit ourselves to a batch size of 2g */\n[emily shaffer] nit: can you please capitalize G, GB, whatever :)\n"},{"id":"402464","messageId":"5cbdb559-3897-961f-4dd3-0bab11848c5b@gmail.com","threadId":"53809","inReplyTo":"20200729221905.GB519065@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T13:12:56Z","receivedAt":"2020-07-30T13:13:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n> Hi,\n> \n> Derrick Stolee wrote:\n> \n>> [Subject: maintenance: create basic maintenance runner]\n> \n> Seems sensible, and a good way to set up for the later patches. Let's\n> take a look at how it does that.\n> \n> [...]\n>> --- /dev/null\n>> +++ b/Documentation/git-maintenance.txt\n>> @@ -0,0 +1,57 @@\n> [...]\n>> +SUBCOMMANDS\n>> +-----------\n>> +\n>> +run::\n>> +\tRun one or more maintenance tasks.\n> \n> [jrnieder] How do I supply the tasks on the command line? Are they\n> parameters to this subcommand? If so, it could make sense for this to\n> say something like\n> \n> \trun ...::\n\nHopefully this is documented to your satisfaction when the ability\nto customize the tasks is implemented.\n\n> What is the exit code convention for \"git maintenance run\"? (Many Git\n> commands don't document this, but since we're making a new command\n> it seems worth building the habit.)\n\nIs this worth doing? Do we really want every command to document\nthat \"0 means success, everything else means failure, and some of\nthose exit codes mean a specific kind of failure that is global to\nGit\"?\n\n> [...]\n>> --- a/builtin/gc.c\n>> +++ b/builtin/gc.c\n>> @@ -699,3 +699,62 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> [...]\n>> +static struct maintenance_opts {\n>> +\tint auto_flag;\n>> +} opts;\n> \n> Packing this in a struct feels a bit unusual. Is the struct going to\n> be passed somewhere, or could these be individual locals in\n> cmd_maintenance?\n\nThis will grow, and I'd rather have one global struct than many\nindividual global items. It makes it clearer when I use\n\"opts.auto_flag\" that this corresponds to whether \"--auto\" was\nprovided as a command-line option.\n\n> [...]\n>> +\n>> +static int maintenance_task_gc(void)\n>> +{\n>> +\tint result;\n>> +\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n>> +\n>> +\targv_array_pushl(&cmd, \"gc\", NULL);\n>> +\n>> +\tif (opts.auto_flag)\n>> +\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n> \n> These are both pushing single strings, so they can use argv_array_push.\n\nThanks. I noticed a few of these myself. Luckily, I'll be going\nthrough all of these invocations and replacing them with new\nmethods soon ;)\n\n[1] https://lore.kernel.org/git/30933a71-3130-5478-cbfd-0ca5bb308cf2@gmail.com/\n\n> [...]\n>> --- /dev/null\n>> +++ b/t/t7900-maintenance.sh\n>> @@ -0,0 +1,22 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='git maintenance builtin'\n>> +\n>> +GIT_TEST_COMMIT_GRAPH=0\n>> +GIT_TEST_MULTI_PACK_INDEX=0\n> \n> Why does this disable commit graph and multipack index? Is that setting\n> up for something that comes later?\n\nYes, these don't need to be here yet.\n\n> [...]\n>> +test_expect_success 'gc [--auto]' '\n>> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n>> +\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n>> +\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n>> +\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n> \n> This feels a little odd in two ways:\n> \n> - the use of \"git gc\" isn't a user-facing detail, so this is testing\n> implementation instead of the user-facing behavior. That's okay,\n> though --- it can be useful to test internals sometimes.\n\nConsider this a \"unit test\" of the builtin. I'm testing whether the\ncommand-line arguments had an effect on the child process.\n\n> - the way that this tests for \"git gc\" feels brittle: if the trace\n> emitter changes some day to include a space after the comma, for\n> example, then the test would start failing. I thought that in the\n> spirit of fakes, it could make sense to write a custom \"git gc\"\n> command using test_write_script, but that isn't likely to work\n> either since gc is a builtin.\n> \n> Perhaps this is suggesting that we need some central test helper for\n> parsing traces so we can do this reliably in one place. What do you\n> think?\n\nI'm specifically using GIT_TRACE2_EVENT, which is intended for\nconsumption by computer, not humans. Adding whitespace or otherwise\nchanging the output format would be an unnecessary disruption that\nis more likely to have downstream effects with external tools.\n\nIn some way, adding extra dependencies like this in our own test\nsuite can help ensure the output format is more stable.\n\nIf there is a better way to ask \"Did my command call 'git gc' (with\nno arguments|with these arguments)?\" then I'm happy to consider it.\n\nThanks,\n-Stolee\n"},{"id":"402465","messageId":"9aadb889-007a-c973-c59c-a471436368bb@gmail.com","threadId":"53809","inReplyTo":"xmqqblk6q8ws.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T13:29:09Z","receivedAt":"2020-07-30T13:29:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/23/2020 4:21 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +--[no-]maintenance::\n>> --[no-]auto-gc::\n>> -\tRun `git gc --auto` at the end to perform garbage collection\n>> -\tif needed. This is enabled by default.\n>> +\tRun `git maintenance run --auto` at the end to perform garbage\n>> +\tcollection if needed. This is enabled by default.\n> \n> Shouldn't the new synonym be called --auto-maintenance or an\n> abbreviation thereof? It is not like we will run the full\n> maintenance suite when \"--no-maintenance\" is omitted, which\n> certainly is not the impression we want to give our readers.\n\nAdding \"auto-\" to the argument can be informative.\n\nI would think that abbreviating the option may make understanding\nthe argument more difficult for users where English is not their\nfirst language. Of course, I'm a bad judge of that.\n\nI also don't think this option is called often by users at the\ncommand-line and instead is done by scripts and third-party tools.\nAbbreviating the word may have less value there.\n\n(This reminds me that is would be good to have a config option that\nprevents running this process _at all_. Sure, gc.auto=0 prevents the\ncommand from doing anything, but there is still an extra cost of\nstarting a process. This is more of a problem on Windows. Making a\nnote to self here.)\n\n>> These objects may be removed by normal Git operations (such as `git commit`)\n>> -which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n>> -If these objects are removed and were referenced by the cloned repository,\n>> -then the cloned repository will become corrupt.\n>> +which automatically call `git maintenance run --auto` and `git gc --auto`.\n> \n> Hmph. Perhaps the picture may change in the end of the series but I\n> got an impression that \"gc --auto\" would eventually become just part\n> of \"maintenance --auto\" and the users won't have to be even aware of\n> its existence? Wouldn't we rather want to say something like\n> \n> \t--[no-]auto-maintenance::\n> \t--[no-]auto-gc::\n> Run `git maintenance run --auto` at the end to perform\n> garbage collection if needed (`--[no-]auto-gc` is a\n> synonym). This is enabled by default.\n\nI don't completely eliminate 'git gc' at the end of this series, but\ninstead hope that we can peel it apart slowly in follow-up series.\nHowever, you are correct that I should be more careful to obliterate\nit from the documentation instead of keeping references around.\n\n>> diff --git a/builtin/fetch.c b/builtin/fetch.c\n>> index 82ac4be8a5..49a4d727d4 100644\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n>> \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n>> \t\t\tN_(\"report that we have only objects reachable from this object\")),\n>> \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> \n>> +\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n>> +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n>> \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n>> +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n> \n> OK, so this is truly a backward-compatible synonym at this point.\n> \n>> diff --git a/run-command.c b/run-command.c\n>> index 9b3a57d1e3..82ad241638 100644\n>> --- a/run-command.c\n>> +++ b/run-command.c\n>> @@ -1865,14 +1865,17 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n>> \treturn result;\n>> }\n>> \n>> -int run_auto_gc(int quiet)\n>> +int run_auto_maintenance(int quiet)\n>> {\n>> \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n>> \tint status;\n>> \n>> -\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n>> +\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n>> \tif (quiet)\n>> \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n>> +\telse\n>> +\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n>> +\n>> \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n>> \targv_array_clear(&argv_gc_auto);\n>> \treturn status;\n> \n> Don't we want to replace all _gc_ with _maintenance_ in this\n> function? I think the first business before we can do so would be\n> to rethink if spelling out \"maintenance\" fully in code is a good\n> idea in the first space. It would make names for variables,\n> structures and fields unnecessarily long without contributing to\n> ease of understanding an iota, and a easy-to-remember short-form or\n> an abbreviation may be needed. Using a short-form/abbreviation\n> wouldn't worsen the end-user experience, and not the developer\n> experience for that matter.\n> \n> If we choose \"gc\" as the short-hand, most of the change in this step\n> would become unnecessary. I also do not mind if we some other words\n> or word-fragment (perhaps \"maint\"???) is chosen.\n\nYes, I should have noticed that. Also, with Peff's feedback from\nanother thread, the method can look a bit simpler this way:\n\nint run_auto_maintenance(int quiet)\n{\n\tstruct child_process maint = CHILD_PROCESS_INIT;\n\tmaint.git_cmd = 1;\n\n\targv_array_pushl(&maint.argv, \"maintenance\", \"run\", \"--auto\", NULL);\n\tif (quiet)\n\t\targv_array_push(&maint.argv, \"--quiet\");\n\telse\n\t\targv_array_push(&maint.argv, \"--no-quiet\");\n\n\treturn run_command(&maint.argv);\n}\n\n(I will update it again to work on Peff's argv_array work, but\nhopefully it is clear how this is simpler.)\n\nThanks,\n-Stolee\n"},{"id":"402466","messageId":"5757360f-a9e5-0bd2-b933-7efddee29cf6@gmail.com","threadId":"53809","inReplyTo":"9aadb889-007a-c973-c59c-a471436368bb@gmail.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T13:31:03Z","receivedAt":"2020-07-30T13:31:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 9:29 AM, Derrick Stolee wrote:\n> On 7/23/2020 4:21 PM, Junio C Hamano wrote:\n>> Don't we want to replace all _gc_ with _maintenance_ in this\n>> function? I think the first business before we can do so would be\n>> to rethink if spelling out \"maintenance\" fully in code is a good\n>> idea in the first space. It would make names for variables,\n>> structures and fields unnecessarily long without contributing to\n>> ease of understanding an iota, and a easy-to-remember short-form or\n>> an abbreviation may be needed. Using a short-form/abbreviation\n>> wouldn't worsen the end-user experience, and not the developer\n>> experience for that matter.\n>>\n>> If we choose \"gc\" as the short-hand, most of the change in this step\n>> would become unnecessary. I also do not mind if we some other words\n>> or word-fragment (perhaps \"maint\"???) is chosen.\n> \n> Yes, I should have noticed that. Also, with Peff's feedback from\n> another thread, the method can look a bit simpler this way:\n\nIt would help if I actually _compile_ code before sending it.\nHere is the fixed version:\n\nint run_auto_maintenance(int quiet)\n{\n\tstruct child_process maint = CHILD_PROCESS_INIT;\n\tmaint.git_cmd = 1;\n\n\targv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n\tif (quiet)\n\t\targv_array_push(&maint.args, \"--quiet\");\n\telse\n\t\targv_array_push(&maint.args, \"--no-quiet\");\n\n\treturn run_command(&maint);\n}\n\n"},{"id":"402467","messageId":"b20e2e46-ff71-b84f-19da-9d8873ef08c3@gmail.com","threadId":"53809","inReplyTo":"20200725015240.GD35171@syl.lan","subject":"Re: [PATCH v2 05/18] maintenance: add commit-graph task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T13:59:21Z","receivedAt":"2020-07-30T13:59:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/24/2020 9:52 PM, Taylor Blau wrote:\n> On Fri, Jul 24, 2020 at 12:47:00PM -0700, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>> But you are discussing here how the _behavior_ can change when\n>>> --auto is specified. And specifically, \"git gc --auto\" really\n>>> meant \"This is running after a foreground command, so only do\n>>> work if necessary and do it quickly to minimize blocking time.\"\n>>>\n>>> I'd be happy to replace \"--auto\" with \"--quick\" in the\n>>> maintenance builtin.\n>>>\n>>> This opens up some extra design space for how the individual\n>>> tasks perform depending on \"--quick\" being specified or not.\n>>> My intention was to create tasks that are already in \"quick\"\n>>> mode:\n>>>\n>>> * loose-objects have a maximum batch size.\n>>> * incremental-repack is capped in size.\n>>> * commit-graph uses the --split option.\n>>>\n>>> But this \"quick\" distinction might be important for some of\n>>> the tasks we intend to extract from the gc builtin.\n>>\n>> Yup. To be honest, I came to this topic from a completely different\n>> direction. The field name \"auto\" alone (and no other field name)\n>> had to have an extra cruft (i.e. \"_flag\") attached to it, which is\n>> understandable but ugly. Then I started thinking if 'auto(matic)'\n>> is really the right word to describe what we want out of the option,\n>> and came to the realization that there may be better words.\n> \n> I wonder what the quick and slow paths are here. For the commit-graph\n> code, what you wrote here seems to match what I'd expect with passing\n> '--auto' in the sense of running 'git gc'. That is, I'm leaving it up to\n> the commit-graph machinery's idea of the normal '--split' rules to\n> figure out when to roll up layers of a commit-graph, as opposed to\n> creating a new layer and extending the chain.\n\nI had intended all of my new tasks to be the \"quick\" version of their\noperations. The \"slow\" version would abandon hope of doing a small\namount of work to create the best possible world for the repository.\nThis would include:\n\n * The commit-graph would collapse all layers into one file.\n * The multi-pack-index repack would rewrite all object data into one\n pack-file.\n * The loose-objects task would not stop at a maximum number of loose\n objects (and would probably want to repack everything, anyway).\n\nI'm open to making this possibility more explicit by renaming \"--auto\"\nand just performing a translation to 'git gc --auto'. So, what should\nthe name be? Here are a few options to consider:\n\n --quick\n --fast\n --limited\n --incremental\n -O[0|1|2...] (think GCC optimization flags, exposing granularity)\n --[non-]aggressive\n\nRegardless, this makes me rethink that the --[no-]maintenance option\nfrom PATCH 03/18 is better than --[no-]auto-maintenance, since we are\nreally saying \"run _some_ maintenance or _no_ maintenance\" and the \"how\"\nof the maintenance is left intentionally vague. I've already made the\nchange locally to add \"auto-\" so I'll wait for confirmation before\nreverting that change.\n\n> So, I think that makes sense if the caller gave '--auto'. But, I'm not\n> sure that it makes sense if they didn't, in which case I'd imagine\n> something quicker to happen. There, I'd expect something more like:\n> \n> 1. Run 'git commit-graph write --reachable --split=no-merge'.\n> 2. Run 'git commit-graph verify'.\n> 3. If 'git commit-graph verify' failed, drop the existing commit graph\n> and rebuild it with 'git commit-graph --reachable --split=replace'.\n> 4. Otherwise, do nothing.\n> \n> I'm biased, of course, but I think that that matches roughly what I'd\n> expect to happen in the fast/slow path. Granted, the steps to rebuild\n> the commit graph are going to be slow no matter what (depending on the\n> size of the repository), and so in that case maybe the commit-graph\n> should just be dropped. I'm not really sure what to do about that...\n\nI think this approach is the best we can do given the current behavior\ninside the commit-graph builtin. Perhaps in the future we could change\nthe commit-graph builtin to include a \"--verify\" option so it could do\nthe \"git commit-graph verify --shallow\" on the new layer before committing\nthe new commit-graph-chain file and expiring old layers. That way, we\nwould not need to delete and rewrite the whole thing when there is a\nproblem writing the top layer.\n\n>>> Since the tasks are frequently running subcommands, returning\n>>> 0 for success and non-zero for error matches the error codes\n>>> returned by those subcommands.\n>>\n>> As long as these will _never_ be called from other helper functions\n>> but from the cmd_foo() top-level and their return values are only\n>> used directly as the top-level's return value, I do not mind too\n>> much.\n>>\n>> But whenever I am writing such a code, I find myself not brave\n>> enough to make such a bold promise (I saw other people call the\n>> helpers I wrote in unintended ways and had to adjust the semantics\n>> of them to accomodate the new callers too many times), so I'd rather\n>> see the caller do \"return !!helper_fn()\" to allow helper_fn() to be\n>> written more naturally (e.g. letting them return error(...)).\n\nI will try to be consistent here with the behavior:\n\n * 0 is success\n * 1 is failure\n\nWhich is what I think you are implying by \"return !!helper_fn()\".\n\nThanks,\n-Stolee\n"},{"id":"402468","messageId":"4b6f5458-f4f7-119f-ec8c-933f27dd3072@gmail.com","threadId":"53809","inReplyTo":"xmqqr1svegt4.fsf_-_@gitster.c.googlers.com","subject":"Re: [PATCH v2] fetch: optionally allow disabling FETCH_HEAD update","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T15:17:31Z","receivedAt":"2020-07-30T15:17:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/28/2020 12:37 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 7/27/2020 12:13 PM, Junio C Hamano wrote:\n>>> Derrick Stolee writes:\n>>>\n>>>> I'll rebase onto jc/no-update-fetch-head for the next version, since\n>>>> that branch is based on v2.28.0-rc0, which is recent enough.\n>>>\n>>> I do not think it is wise to base a work on top of unfinished \"you\n>>> could do it this way, perhaps?\" demonstration patch the original\n>>> author does not have much inclination to finish, though.\n>>>\n>>> When I am really bored, I may go back to the topic to finish it, but\n>>> I wouldn't mind if you took ownership of it at all.\n>>\n>> Ah. I didn't understand the status of that branch. I'll pull it in\n>> to this topic.\n> \n> So here is with one of the two things that I found missing in the\n> first iteration of the patch: documentation.\n> \n> The other thing that I found iffy (and still missing from this\n> version) was what should be done when \"git pull\" is explicitly given\n> the \"--no-write-fetch-head\" option.\n> \n> I think (but didn't check the recent code) that 'git pull' would\n> pass only known-to-make-sense command line options to underlying\n> 'git fetch', so it probably will barf with \"unknown option\", which\n> is the best case. We might want to make it sure with a new test in\n> 5521. On the other hand, if we get anything other than \"no such\n> option\", we may want to think if we want to \"fix\" it or just leave\n> it inside \"if it hurts, don't do it\" territory.\n\nHere is the version I applied and updated. Please let me know what\nyou think.\n\n-->8--\n\nFrom 3f60a0f0fd388447aa9c2b805b5646039df98d0b Mon Sep 17 00:00:00 2001\nFrom: Junio C Hamano \nDate: Tue, 28 Jul 2020 09:37:59 -0700\nSubject: [PATCH] fetch: optionally allow disabling FETCH_HEAD update\n\nIf you run fetch but record the result in remote-tracking branches,\nand either if you do nothing with the fetched refs (e.g. you are\nmerely mirroring) or if you always work from the remote-tracking\nrefs (e.g. you fetch and then merge origin/branchname separately),\nyou can get away with having no FETCH_HEAD at all.\n\nTeach \"git fetch\" a command line option \"--[no-]write-fetch-head\"\nand \"fetch.writeFetchHEAD\" configuration variable. Without either,\nthe default is to write FETCH_HEAD, and the usual rule that the\ncommand line option defeats configured default applies.\n\nNote that under \"--dry-run\" mode, FETCH_HEAD is never written;\notherwise you'd see list of objects in the file that you do not\nactually have. Passing `--write-fetch-head` does not force `git\nfetch` to write the file.\n\nAlso note that this option is explicitly passed when \"git pull\"\ninternally invokes \"git fetch\", so that those who configured their\n\"git fetch\" not to write FETCH_HEAD would not be able to break the\ncooperation between these two commands. \"git pull\" must see what\n\"git fetch\" got recorded in FETCH_HEAD to work correctly.\n\nSigned-off-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 7 ++++++\n Documentation/fetch-options.txt | 10 +++++++++\n builtin/fetch.c | 19 +++++++++++++---\n builtin/pull.c | 3 ++-\n t/t5510-fetch.sh | 39 +++++++++++++++++++++++++++++++--\n t/t5521-pull-options.sh | 16 ++++++++++++++\n 6 files changed, 88 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex b20394038d1..0aaa05e8c0e 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -91,3 +91,10 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.writeFetchHEAD::\n+\tSetting it to false tells `git fetch` not to write the list\n+\tof remote refs fetched in the `FETCH_HEAD` file directly\n+\tunder `$GIT_DIR`. Can be countermanded from the command\n+\tline with the `--[no-]write-fetch-head` option. Defaults to\n+\ttrue.\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 495bc8ab5a1..6972ad2522c 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -64,6 +64,16 @@ documented in linkgit:git-config[1].\n --dry-run::\n \tShow what would be done, without making any changes.\n \n+ifndef::git-pull[]\n+--[no-]write-fetch-head::\n+\tWrite the list of remote refs fetched in the `FETCH_HEAD`\n+\tfile directly under `$GIT_DIR`. This is the default unless\n+\tthe configuration variable `fetch.writeFetchHEAD` is set to\n+\tfalse. Passing `--no-write-fetch-head` from the command\n+\tline tells Git not to write the file. Under `--dry-run`\n+\toption, the file is never written.\n+endif::git-pull[]\n+\n -f::\n --force::\n \tWhen 'git fetch' is used with `:` refspec it may\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex d8253f66e4c..1d7194aac26 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -56,6 +56,7 @@ static int prune_tags = -1; /* unspecified */\n #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok;\n+static int write_fetch_head = 1;\n static int verbosity, deepen_relative, set_upstream;\n static int progress = -1;\n static int enable_auto_gc = 1;\n@@ -118,6 +119,10 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(k, \"fetch.writefetchhead\")) {\n+\t\twrite_fetch_head = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -162,6 +167,8 @@ static struct option builtin_fetch_options[] = {\n \t\t PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),\n \tOPT_BOOL(0, \"dry-run\", &dry_run,\n \t\t N_(\"dry run\")),\n+\tOPT_BOOL(0, \"write-fetch-head\", &write_fetch_head,\n+\t\t N_(\"write fetched references to the FETCH_HEAD file\")),\n \tOPT_BOOL('k', \"keep\", &keep, N_(\"keep downloaded pack\")),\n \tOPT_BOOL('u', \"update-head-ok\", &update_head_ok,\n \t\t N_(\"allow updating of HEAD ref\")),\n@@ -895,7 +902,9 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *what, *kind;\n \tstruct ref *rm;\n \tchar *url;\n-\tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head(the_repository);\n+\tconst char *filename = (!write_fetch_head\n+\t\t\t\t? \"/dev/null\"\n+\t\t\t\t: git_path_fetch_head(the_repository));\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n \n@@ -1329,7 +1338,7 @@ static int do_fetch(struct transport *transport,\n \t}\n \n \t/* if not appending, truncate FETCH_HEAD */\n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tretcode = truncate_fetch_head();\n \t\tif (retcode)\n \t\t\tgoto cleanup;\n@@ -1596,7 +1605,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \tint i, result = 0;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tint errcode = truncate_fetch_head();\n \t\tif (errcode)\n \t\t\treturn errcode;\n@@ -1797,6 +1806,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n+\t/* FETCH_HEAD never gets updated in --dry-run mode */\n+\tif (dry_run)\n+\t\twrite_fetch_head = 0;\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 8159c5d7c96..e988d92b535 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -527,7 +527,8 @@ static int run_fetch(const char *repo, const char **refspecs)\n \tstruct argv_array args = ARGV_ARRAY_INIT;\n \tint ret;\n \n-\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\", NULL);\n+\targv_array_pushl(&args, \"fetch\", \"--update-head-ok\",\n+\t\t\t \"--write-fetch-head\", NULL);\n \n \t/* Shared options */\n \targv_push_verbosity(&args);\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 9850ecde5df..31c91d0ed2e 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -539,13 +539,48 @@ test_expect_success 'fetch into the current branch with --update-head-ok' '\n \n '\n \n-test_expect_success 'fetch --dry-run' '\n-\n+test_expect_success 'fetch --dry-run does not touch FETCH_HEAD' '\n \trm -f .git/FETCH_HEAD &&\n \tgit fetch --dry-run . &&\n \t! test -f .git/FETCH_HEAD\n '\n \n+test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success '--write-fetch-head gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --dry-run --write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --dry-run . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --no-write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch --write-fetch-head . &&\n+\ttest -f .git/FETCH_HEAD\n+'\n+\n test_expect_success \"should be able to fetch with duplicate refspecs\" '\n \tmkdir dups &&\n \t(\ndiff --git a/t/t5521-pull-options.sh b/t/t5521-pull-options.sh\nindex 159afa7ac81..1acae3b9a4f 100755\n--- a/t/t5521-pull-options.sh\n+++ b/t/t5521-pull-options.sh\n@@ -77,6 +77,7 @@ test_expect_success 'git pull -q -v --no-rebase' '\n \ttest_must_be_empty out &&\n \ttest -s err)\n '\n+\n test_expect_success 'git pull --cleanup errors early on invalid argument' '\n \tmkdir clonedcleanup &&\n \t(cd clonedcleanup && git init &&\n@@ -85,6 +86,21 @@ test_expect_success 'git pull --cleanup errors early on invalid argument' '\n \ttest -s err)\n '\n \n+test_expect_success 'git pull --no-write-fetch-head fails' '\n+\tmkdir clonedwfh &&\n+\t(cd clonedwfh && git init &&\n+\ttest_must_fail git pull --no-write-fetch-head \"../parent\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_i18ngrep \"no-write-fetch-head\" err)\n+'\n+\n+test_expect_success 'git pull succeeds with fetch.writeFetchHEAD=false' '\n+\tmkdir clonedwfhconfig &&\n+\t(cd clonedwfhconfig && git init &&\n+\tgit config fetch.writeFetchHEAD false &&\n+\tgit pull \"../parent\" >out 2>err &&\n+\tgrep FETCH_HEAD err)\n+'\n \n test_expect_success 'git pull --force' '\n \tmkdir clonedoldstyle &&\n-- \n2.27.0.366.g34746c1d11e.dirty\n\n\n\n"},{"id":"402469","messageId":"906b84c9-5182-97f9-2d13-b61cc31a4bc4@gmail.com","threadId":"53809","inReplyTo":"20200729222126.GC2965447@google.com","subject":"Re: [PATCH v2 09/18] maintenance: add loose-objects task","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T15:38:51Z","receivedAt":"2020-07-30T15:38:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/29/2020 6:21 PM, Emily Shaffer wrote:\n> On Thu, Jul 23, 2020 at 05:56:31PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> +loose-objects::\n>> +\tThe `loose-objects` job cleans up loose objects and places them into\n>> +\tpack-files. In order to prevent race conditions with concurrent Git\n>> +\tcommands, it follows a two-step process. First, it deletes any loose\n>> +\tobjects that already exist in a pack-file; concurrent Git processes\n>> +\twill examine the pack-file for the object data instead of the loose\n>> +\tobject. Second, it creates a new pack-file (starting with \"loose-\")\n> \n> [jonathan tan + jonathan nieder] If you are going to document this,\n> probably it should also be tested, so the documentation does not become\n> stale. Or, just don't document it.\n\nAdding a condition to the test is a good idea.\n\n>> +static int pack_loose(void)\n>> +{\n>> +\tstruct repository *r = the_repository;\n>> +\tint result = 0;\n>> +\tstruct write_loose_object_data data;\n>> +\tstruct strbuf prefix = STRBUF_INIT;\n>> +\tstruct child_process *pack_proc;\n>> +\n>> +\t/*\n>> +\t * Do not start pack-objects process\n>> +\t * if there are no loose objects.\n>> +\t */\n>> +\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n>> +\t\t\t\t\t loose_object_exists,\n>> +\t\t\t\t\t NULL, NULL, NULL))\n> \n> [emily] To me, this is unintuitive - but upon inspection, it's exiting\n> the foreach early if any loose object is found, so this is cheaper than\n> actually counting. Maybe a comment would help to understand? Or we could\n> name the function differently, like \"bail_if_loose()\" or something?\n\nSure. \"bail_on_loose()\" makes more sense to me.\n\n>> +test_expect_success 'loose-objects task' '\n>> +\t# Repack everything so we know the state of the object dir\n>> +\tgit repack -adk &&\n>> +\n>> +\t# Hack to stop maintenance from running during \"git commit\"\n>> +\techo in use >.git/objects/maintenance.lock &&\n>> +\ttest_commit create-loose-object &&\n> \n> [jonathan nieder] Does it make sense to use a different git command\n> which is guaranteed to make a loose object? Is 'git commit' futureproof,\n> if we decide commits should directly create packs in the future? 'git\n> unpack-objects' is guaranteed to make a loose object, although it is\n> clumsy because it needs a packfile to begin with...\n\n\"unpack-objects\" also has the problem that it won't write the loose\nobject if it exists as a packed object, so it requires moving the\npack-file out of the object directory first. (That's also required\nfor testing that the loose objects get packed by the maintenance\ntask, but I'd rather not need to remove the multi-pack-index, too.)\n\n> [jonathan tan] But, using 'git commit' is easier to understand in this\n> context. Maybe commenting to say that we assume 'git commit' makes 1 or\n> more loose objects will be enough to futureproof - then we have a signal\n> to whoever made a change to make this fail, and that person knows how to\n> fix this test.\n\nI'll add the comment for now. Thanks.\n\n-Stolee\n"},{"id":"402472","messageId":"b374cb3a-f681-d514-c465-6b3f0b674bcf@gmail.com","threadId":"53809","inReplyTo":"20200729222334.GE2965447@google.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T16:57:18Z","receivedAt":"2020-07-30T16:57:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/29/2020 6:23 PM, Emily Shaffer wrote:\n> On Thu, Jul 23, 2020 at 05:56:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> diff --git a/builtin/gc.c b/builtin/gc.c\n>> index eb4b01c104..889d97afe7 100644\n>> --- a/builtin/gc.c\n>> +++ b/builtin/gc.c\n>> @@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n>> \treturn result;\n>> }\n>> \n>> +#define TWO_GIGABYTES (2147483647)\n> \n> [jonathan tan] This would be easier to understand if it was expressed\n> with bitshift, e.g. 1 << 31\n\nThis is actually ((1 << 31) - 1) because \"unsigned long\" is 32-bits\non Windows. But it's better to not use magic numbers and instead use\noperations like this.\n\n>> +#define UNSET_BATCH_SIZE ((unsigned long)-1)\n> [jonathan tan] This looks like it's never used... and vulnerable to\n> cross-platform size changes because it's referring to an implicitly\n> sized int, and could behave differently if it was put into a larger\n> size, e.g. you wouldn't get 0xFFFF.. if you assigned this into a long\n> long.\n\nThanks for catching this cruft.\n\n>> +\tfor (p = get_all_packs(r); p; p = p->next) {\n>> +\t\tif (p->pack_size > max_size) {\n>> +\t\t\tsecond_largest_size = max_size;\n>> +\t\t\tmax_size = p->pack_size;\n>> +\t\t} else if (p->pack_size > second_largest_size)\n>> +\t\t\tsecond_largest_size = p->pack_size;\n>> +\t}\n>> +\n>> +\tresult_size = second_largest_size + 1;\n> [jonathan tan] What happens when there's only one packfile, and when\n> there are two packfiles? Can we write tests to illustrate the behavior?\n> The edge case here (result_size=1) is hard to understand by reading the\n> code.\n> \n>> +\n>> +\t/* But limit ourselves to a batch size of 2g */\n> [emily shaffer] nit: can you please capitalize G, GB, whatever :)\n\nI suppose I could (get_unit_factor() performs case-insensitive matches\non the size suffixes), except that would be inconsistent with the\nfollowing error message in parse-options.c:\n\n\treturn error(_(\"%s expects a non-negative integer value\"\n\t\t \" with an optional k/m/g suffix\"),\n\nor these options in Documentation/git-pack-objects.txt:\n\n--window-memory=::\n\tThis option provides an additional limit on top of `--window`;\n\tthe window size will dynamically scale down so as to not take\n\tup more than '' bytes in memory. This is useful in\n\trepositories with a mix of large and small objects to not run\n\tout of memory with a large window, but still be able to take\n\tadvantage of the large window for the smaller objects. The\n\tsize can be suffixed with \"k\", \"m\", or \"g\".\n\t`--window-memory=0` makes memory usage unlimited. The default\n\tis taken from the `pack.windowMemory` configuration variable.\n\n--max-pack-size=::\n\tIn unusual scenarios, you may not be able to create files\n\tlarger than a certain size on your filesystem, and this option\n\tcan be used to tell the command to split the output packfile\n\tinto multiple independent packfiles, each not larger than the\n\tgiven size. The size can be suffixed with\n\t\"k\", \"m\", or \"g\". The minimum size allowed is limited to 1 MiB.\n\tThis option\n\tprevents the creation of a bitmap index.\n\tThe default is unlimited, unless the config variable\n\t`pack.packSizeLimit` is set.\n\nPerhaps that's not really a good reason, but upper vs lower case\nseems to be arbitrary. Any tie breaker seems valid.\n\nThanks,\n-Stolee\n"},{"id":"402477","messageId":"CAPig+cT50u=she6xm-zam-Qxtt5c7RTUTme+tj4GBGnQ_nus8g@mail.gmail.com","threadId":"53809","inReplyTo":"5757360f-a9e5-0bd2-b933-7efddee29cf6@gmail.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2020-07-30T19:00:43Z","receivedAt":"2020-07-30T19:00:58Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Jul 30, 2020 at 9:31 AM Derrick Stolee wrote:\n> int run_auto_maintenance(int quiet)\n> {\n> struct child_process maint = CHILD_PROCESS_INIT;\n> maint.git_cmd = 1;\n>\n> argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n> if (quiet)\n> argv_array_push(&maint.args, \"--quiet\");\n> else\n> argv_array_push(&maint.args, \"--no-quiet\");\n\nIt's subjective, but this might be a good fit for the ternary operator:\n\n argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n argv_array_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\n\nOr even:\n\n argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\",\n quiet ? \"--quiet\" : \"--no-quiet\", NULL);\n\nThe latter is a bit less readable to me.\n"},{"id":"402478","messageId":"c0b9876a-9041-3400-25ac-e52568a02074@gmail.com","threadId":"53809","inReplyTo":"b374cb3a-f681-d514-c465-6b3f0b674bcf@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T19:02:32Z","receivedAt":"2020-07-30T19:02:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 12:57 PM, Derrick Stolee wrote:\n> On 7/29/2020 6:23 PM, Emily Shaffer wrote:\n>> On Thu, Jul 23, 2020 at 05:56:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>> diff --git a/builtin/gc.c b/builtin/gc.c\n>>> index eb4b01c104..889d97afe7 100644\n>>> --- a/builtin/gc.c\n>>> +++ b/builtin/gc.c\n>>> @@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n>>> \treturn result;\n>>> }\n>>> \n>>> +#define TWO_GIGABYTES (2147483647)\n>>\n>> [jonathan tan] This would be easier to understand if it was expressed\n>> with bitshift, e.g. 1 << 31\n> \n> This is actually ((1 << 31) - 1) because \"unsigned long\" is 32-bits\n> on Windows. But it's better to not use magic numbers and instead use\n> operations like this.\n\nNevermind. This breaks the build on 32-bit machines (even adding a few\n\"L\" characters). I'll replace my magic decimal number with a magic\nhexadecimal number.\n\nThanks,\n-Stolee\n"},{"id":"402480","messageId":"CAPx1GvfcDeCBLMFbTtCpEK13uCWQQ+u=R1x1qE2kBD3eXVwh7A@mail.gmail.com","threadId":"53809","inReplyTo":"c0b9876a-9041-3400-25ac-e52568a02074@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2020-07-30T19:24:57Z","receivedAt":"2020-07-30T19:25:11Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Thu, Jul 30, 2020 at 12:04 PM Derrick Stolee wrote:\n> > This is actually ((1 << 31) - 1) because \"unsigned long\" is 32-bits\n> > on Windows. But it's better to not use magic numbers and instead use\n> > operations like this.\n>\n> Nevermind. This breaks the build on 32-bit machines (even adding a few\n> \"L\" characters). I'll replace my magic decimal number with a magic\n> hexadecimal number.\n\nYou would need something along these lines:\n\n #define WHATEVER ((long)((1UL << 31) - 1))\n\ni.e., make the constant 1 be \"unsigned long\" so that the shift is\nwell defined, then subtract 1, then convert back to signed long.\n\nI say \"something along\" because there are several more ways to write\nit. I'm not really a big fan of most of them, and 0x7FFFFFFF (or in\nlowercase) is my own preference here too.\n\nChris\n"},{"id":"402482","messageId":"10439c4b-6f0b-6d37-d805-18a3ecc313d8@gmail.com","threadId":"53809","inReplyTo":"CAPig+cT50u=she6xm-zam-Qxtt5c7RTUTme+tj4GBGnQ_nus8g@mail.gmail.com","subject":"Re: [PATCH v2 03/18] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-30T20:21:36Z","receivedAt":"2020-07-30T20:21:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 3:00 PM, Eric Sunshine wrote:\n> On Thu, Jul 30, 2020 at 9:31 AM Derrick Stolee wrote:\n>> int run_auto_maintenance(int quiet)\n>> {\n>> struct child_process maint = CHILD_PROCESS_INIT;\n>> maint.git_cmd = 1;\n>>\n>> argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n>> if (quiet)\n>> argv_array_push(&maint.args, \"--quiet\");\n>> else\n>> argv_array_push(&maint.args, \"--no-quiet\");\n> \n> It's subjective, but this might be a good fit for the ternary operator:\n> \n> argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n> argv_array_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\n\nGood idea! Thanks.\n\n> Or even:\n> \n> argv_array_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\",\n> quiet ? \"--quiet\" : \"--no-quiet\", NULL);\n> \n> The latter is a bit less readable to me.\n\nI agree.\n-Stolee\n"},{"id":"402483","messageId":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v2.git.1595527000.gitgitgadget@gmail.com","subject":"[PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:06Z","receivedAt":"2020-07-30T22:24:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is a second attempt at redesigning Git's repository maintenance\npatterns. The first attempt [1] included a way to run jobs in the background\nusing a long-lived process; that idea was rejected and is not included in\nthis series. A future series will use the OS to handle scheduling tasks.\n\n[1] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nAs mentioned before, git gc already plays the role of maintaining Git\nrepositories. It has accumulated several smaller pieces in its long history,\nincluding:\n\n 1. Repacking all reachable objects into one pack-file (and deleting\n unreachable objects).\n 2. Packing refs.\n 3. Expiring reflogs.\n 4. Clearing rerere logs.\n 5. Updating the commit-graph file.\n 6. Pruning worktrees.\n\nWhile expiring reflogs, clearing rererelogs, and deleting unreachable\nobjects are suitable under the guise of \"garbage collection\", packing refs\nand updating the commit-graph file are not as obviously fitting. Further,\nthese operations are \"all or nothing\" in that they rewrite almost all\nrepository data, which does not perform well at extremely large scales.\nThese operations can also be disruptive to foreground Git commands when git\ngc --auto triggers during routine use.\n\nThis series does not intend to change what git gc does, but instead create\nnew choices for automatic maintenance activities, of which git gc remains\nthe only one enabled by default.\n\nThe new maintenance tasks are:\n\n * 'commit-graph' : write and verify a single layer of an incremental\n commit-graph.\n * 'loose-objects' : prune packed loose objects, then create a new pack from\n a batch of loose objects.\n * 'pack-files' : expire redundant packs from the multi-pack-index, then\n repack using the multi-pack-index's incremental repack strategy.\n * 'fetch' : fetch from each remote, storing the refs in 'refs/prefetch//'.\n\nThese tasks are all disabled by default, but can be enabled with config\noptions or run explicitly using \"git maintenance run --task=\". There are\nadditional config options to allow customizing the conditions for which the\ntasks run during the '--auto' option. ('fetch' will never run with the\n'--auto' option.)\n\n Because 'gc' is implemented as a maintenance task, the most dramatic change\nof this series is to convert the 'git gc --auto' calls into 'git maintenance\nrun --auto' calls at the end of some Git commands. By default, the only\nchange is that 'git gc --auto' will be run below an additional 'git\nmaintenance' process.\n\nThe 'git maintenance' builtin has a 'run' subcommand so it can be extended\nlater with subcommands that manage background maintenance, such as 'start',\n'stop', 'pause', or 'schedule'. These are not the subject of this series, as\nit is important to focus on the maintenance activities themselves.\n\nAn expert user could set up scheduled background maintenance themselves with\nthe current series. I have the following crontab data set up to run\nmaintenance on an hourly basis:\n\n0 * * * * git -C / maintenance run --no-quiet >>//.git/maintenance.log\n\nMy config includes all tasks except the 'gc' task. The hourly run is\nover-aggressive, but is sufficient for testing. I'll replace it with daily\nwhen I feel satisfied.\n\nHopefully this direction is seen as a positive one. My goal was to add more\noptions for expert users, along with the flexibility to create background\nmaintenance via the OS in a later series.\n\nOUTLINE\n=======\n\nPatches 1-4 remove some references to the_repository in builtin/gc.c before\nwe start depending on code in that builtin.\n\nPatches 5-7 create the 'git maintenance run' builtin and subcommand as a\nsimple shim over 'git gc' and replaces calls to 'git gc --auto' from other\ncommands.\n\nPatches 8-15 create new maintenance tasks. These are the same tasks sent in\nthe previous RFC.\n\nPatches 16-21 create more customization through config and perform other\npolish items.\n\nFUTURE WORK\n===========\n\n * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n commands run in the background. You can see my progress towards this goal\n here: https://github.com/gitgitgadget/git/pull/680\n \n \n * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n default, but might have different '--auto' conditions and more config\n options.\n \n \n * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n with use of the 'commit-graph' task.\n \n \n * Update the builtin to use struct repository *r properly, especially when\n calling subcommands.\n \n \n\nUPDATES in V2\n=============\n\nI'm sending this between v2.28.0-rc2 adn v2.28.0 as the release things have\nbecome a bit quiet.\n\n * The biggest disruption to the range-diff is that I removed the premature\n use of struct repository *r and instead continue to rely on \n the_repository. This means several patches were dropped that did prep\n work in builtin/gc.c.\n \n \n * I dropped the task hashmap and opted for a linear scan. This task list\n will always be too small to justify the extra complication of the\n hashmap.\n \n \n * struct maintenance_opts is properly static now.\n \n \n * Some tasks are renamed: fetch -> prefetch, pack-files ->\n incremental-repack.\n \n \n * With the rename, the prefetch task uses refs/prefetch/ instead of \n refs/hidden/.\n \n \n * A trace2 region around the task executions are added.\n \n \n\nUPDATES in V3\n=============\n\n * This series is now based on jk/strvec, as there are several places where\n I was adding new callers to argv_array_push* and run_command_v_opt()\n which have been replaced with strvec_push*() and run_command(), using a\n 'struct child_process'.\n \n \n * I added and updates Junio's patch from jc/no-update-fetch-head into the\n proper place before the 'prefetch' task. The 'prefetch' task now uses\n --no-write-fetch-head to avoid issues with FETCH_HEAD.\n \n \n * Since there were concerns around core.multiPackIndex, I added some extra\n care checking for that config being enabled. Since that already needed to\n adjust the config value from its existing implementation in midx.c, I\n added it to repo-settings and made it enabled by default.\n \n \n * Lots of feedback from the previous round. Thanks, all! I fully expect to\n have at least one more round of feedback, but things are improving quite\n a lot.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (19):\n maintenance: create basic maintenance runner\n maintenance: add --quiet option\n maintenance: replace run_auto_gc()\n maintenance: initialize task array\n maintenance: add commit-graph task\n maintenance: add --task option\n maintenance: take a lock on the objects directory\n maintenance: add prefetch task\n maintenance: add loose-objects task\n midx: enable core.multiPackIndex by default\n maintenance: add incremental-repack task\n maintenance: auto-size incremental-repack batch\n maintenance: create maintenance..enabled config\n maintenance: use pointers to check --auto\n maintenance: add auto condition for commit-graph task\n maintenance: create auto condition for loose-objects\n maintenance: add incremental-repack auto condition\n midx: use start_delayed_progress()\n maintenance: add trace2 regions for task execution\n\nJunio C Hamano (1):\n fetch: optionally allow disabling FETCH_HEAD update\n\n .gitignore | 1 +\n Documentation/config.txt | 2 +\n Documentation/config/core.txt | 4 +-\n Documentation/config/fetch.txt | 7 +\n Documentation/config/maintenance.txt | 32 ++\n Documentation/fetch-options.txt | 16 +-\n Documentation/git-clone.txt | 6 +-\n Documentation/git-maintenance.txt | 127 +++++\n builtin.h | 1 +\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 25 +-\n builtin/gc.c | 713 +++++++++++++++++++++++++++\n builtin/merge.c | 2 +-\n builtin/pull.c | 3 +-\n builtin/rebase.c | 4 +-\n commit-graph.c | 8 +-\n commit-graph.h | 1 +\n git.c | 1 +\n midx.c | 23 +-\n midx.h | 1 +\n object.h | 1 +\n repo-settings.c | 6 +\n repository.h | 2 +\n run-command.c | 16 +-\n run-command.h | 2 +-\n t/t5319-multi-pack-index.sh | 15 +-\n t/t5510-fetch.sh | 41 +-\n t/t5514-fetch-multiple.sh | 2 +-\n t/t5521-pull-options.sh | 16 +\n t/t7900-maintenance.sh | 215 ++++++++\n 31 files changed, 1240 insertions(+), 57 deletions(-)\n create mode 100644 Documentation/config/maintenance.txt\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\n\nbase-commit: 90dfbf331c0ade4c15c74276c466e32e384f9ceb\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-671%2Fderrickstolee%2Fmaintenance%2Fgc-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-671/derrickstolee/maintenance/gc-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/671\n\nRange-diff vs v2:\n\n 1: 63ec602a07 ! 1: 12fe73bb72 maintenance: create basic maintenance runner\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n +\n +static int maintenance_task_gc(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n +\n -+\targv_array_pushl(&cmd, \"gc\", NULL);\n ++\tchild.git_cmd = 1;\n ++\tstrvec_push(&child.args, \"gc\");\n +\n +\tif (opts.auto_flag)\n -+\t\targv_array_pushl(&cmd, \"--auto\", NULL);\n ++\t\tstrvec_push(&child.args, \"--auto\");\n +\n +\tclose_object_store(the_repository->objects);\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n -+\n -+\treturn result;\n ++\treturn run_command(&child);\n +}\n +\n +static int maintenance_run(void)\n @@ t/t7900-maintenance.sh (new)\n +\n +test_description='git maintenance builtin'\n +\n -+GIT_TEST_COMMIT_GRAPH=0\n -+GIT_TEST_MULTI_PACK_INDEX=0\n -+\n +. ./test-lib.sh\n +\n +test_expect_success 'help text' '\n -+\ttest_must_fail git maintenance -h 2>err &&\n ++\ttest_expect_code 129 git maintenance -h 2>err &&\n +\ttest_i18ngrep \"usage: git maintenance run\" err\n +'\n +\n -+test_expect_success 'gc [--auto]' '\n ++test_expect_success 'run [--auto]' '\n +\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n +\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n +\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n 2: 1d37e55cb7 ! 2: 6e533e43d7 maintenance: add --quiet option\n @@ builtin/gc.c: static const char * const builtin_maintenance_usage[] = {\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n \n \tif (opts.auto_flag)\n - \t\targv_array_pushl(&cmd, \"--auto\", NULL);\n + \t\tstrvec_push(&child.args, \"--auto\");\n +\tif (opts.quiet)\n -+\t\targv_array_pushl(&cmd, \"--quiet\", NULL);\n ++\t\tstrvec_push(&child.args, \"--quiet\");\n \n \tclose_object_store(the_repository->objects);\n - \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n + \treturn run_command(&child);\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstatic struct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n @@ t/t7900-maintenance.sh: test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n --test_expect_success 'gc [--auto]' '\n +-test_expect_success 'run [--auto]' '\n -\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n -+test_expect_success 'gc [--auto|--quiet]' '\n ++test_expect_success 'run [--auto|--quiet]' '\n +\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n +\tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n 3: f164d1a0b4 ! 3: c4674fc211 maintenance: replace run_auto_gc()\n @@ Commit message\n \n Rename run_auto_gc() to run_auto_maintenance() to be clearer what is\n happening on this call, and to expose all callers in the current diff.\n + Rewrite the method to use a struct child_process to simplify the calls\n + slightly.\n \n Since 'git fetch' already allows disabling the 'git gc --auto'\n subprocess, add an equivalent option with a different name to be more\n @@ Documentation/fetch-options.txt: ifndef::git-pull[]\n \tAllow several and arguments to be\n \tspecified. No s may be specified.\n \n -+--[no-]maintenance::\n ++--[no-]auto-maintenance::\n --[no-]auto-gc::\n -\tRun `git gc --auto` at the end to perform garbage collection\n -\tif needed. This is enabled by default.\n -+\tRun `git maintenance run --auto` at the end to perform garbage\n -+\tcollection if needed. This is enabled by default.\n ++\tRun `git maintenance run --auto` at the end to perform automatic\n ++\trepository maintenance if needed. (`--[no-]auto-gc` is a synonym.)\n ++\tThis is enabled by default.\n \n --[no-]write-commit-graph::\n \tWrite a commit-graph after fetching. This overrides the config\n @@ Documentation/git-clone.txt: repository using this option and then delete branch\n -which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n -If these objects are removed and were referenced by the cloned repository,\n -then the cloned repository will become corrupt.\n -+which automatically call `git maintenance run --auto` and `git gc --auto`.\n -+(See linkgit:git-maintenance[1] and linkgit:git-gc[1].) If these objects\n -+are removed and were referenced by the cloned repository, then the cloned\n -+repository will become corrupt.\n ++which automatically call `git maintenance run --auto`. (See\n ++linkgit:git-maintenance[1].) If these objects are removed and were referenced\n ++by the cloned repository, then the cloned repository will become corrupt.\n +\n Note that running `git repack` without the `--local` option in a repository\n cloned with `--shared` will copy objects from the source repository into a pack\n @@ builtin/fetch.c: static struct option builtin_fetch_options[] = {\n \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n \t\t\tN_(\"report that we have only objects reachable from this object\")),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n -+\tOPT_BOOL(0, \"maintenance\", &enable_auto_gc,\n ++\tOPT_BOOL(0, \"auto-maintenance\", &enable_auto_gc,\n +\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n -\t\t N_(\"run 'gc --auto' after fetching\")),\n @@ run-command.c: int run_processes_parallel_tr2(int n, get_next_task_fn get_next_t\n -int run_auto_gc(int quiet)\n +int run_auto_maintenance(int quiet)\n {\n - \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n - \tint status;\n +-\tstruct strvec argv_gc_auto = STRVEC_INIT;\n +-\tint status;\n ++\tstruct child_process maint = CHILD_PROCESS_INIT;\n \n --\targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n -+\targv_array_pushl(&argv_gc_auto, \"maintenance\", \"run\", \"--auto\", NULL);\n - \tif (quiet)\n - \t\targv_array_push(&argv_gc_auto, \"--quiet\");\n -+\telse\n -+\t\targv_array_push(&argv_gc_auto, \"--no-quiet\");\n +-\tstrvec_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n +-\tif (quiet)\n +-\t\tstrvec_push(&argv_gc_auto, \"--quiet\");\n +-\tstatus = run_command_v_opt(argv_gc_auto.items, RUN_GIT_CMD);\n +-\tstrvec_clear(&argv_gc_auto);\n +-\treturn status;\n ++\tmaint.git_cmd = 1;\n ++\tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n ++\tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\n +\n - \tstatus = run_command_v_opt(argv_gc_auto.argv, RUN_GIT_CMD);\n - \targv_array_clear(&argv_gc_auto);\n - \treturn status;\n ++\treturn run_command(&maint);\n + }\n \n ## run-command.h ##\n @@ run-command.h: int run_hook_ve(const char *const *env, const char *name, va_list args);\n 4: 8e260bccf1 ! 4: b9332c1318 maintenance: initialize task array\n @@ Commit message\n future command-line argument) along with a function pointer to its\n implementation and a boolean for whether the step is enabled.\n \n - A list of pointers to these structs are initialized with the full list\n - of implemented tasks along with a default order. For now, this list only\n - contains the \"gc\" task. This task is also the only task enabled by\n - default.\n + A list these structs are initialized with the full list of implemented\n + tasks along with a default order. For now, this list only contains the\n + \"gc\" task. This task is also the only task enabled by default.\n \n + The run subcommand will return a nonzero exit code if any task fails.\n + However, it will attempt all tasks in its loop before returning with the\n + failure. Also each failed task will send an error message.\n +\n + Helped-by: Taylor Blau \n + Helped-by: Junio C Hamano \n Signed-off-by: Derrick Stolee \n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - \treturn 0;\n - }\n - \n -+#define MAX_NUM_TASKS 1\n -+\n - static const char * const builtin_maintenance_usage[] = {\n - \tN_(\"git maintenance run []\"),\n - \tNULL\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n - \treturn result;\n + \treturn run_command(&child);\n }\n \n +typedef int maintenance_task_fn(void);\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n +\tunsigned enabled:1;\n +};\n +\n -+static struct maintenance_task *tasks[MAX_NUM_TASKS];\n -+static int num_tasks;\n ++enum maintenance_task_label {\n ++\tTASK_GC,\n ++\n ++\t/* Leave as final value */\n ++\tTASK__COUNT\n ++};\n ++\n ++static struct maintenance_task tasks[] = {\n ++\t[TASK_GC] = {\n ++\t\t\"gc\",\n ++\t\tmaintenance_task_gc,\n ++\t\t1,\n ++\t},\n ++};\n +\n static int maintenance_run(void)\n {\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n +\tint i;\n +\tint result = 0;\n +\n -+\tfor (i = 0; !result && i < num_tasks; i++) {\n -+\t\tif (!tasks[i]->enabled)\n ++\tfor (i = 0; i < TASK__COUNT; i++) {\n ++\t\tif (!tasks[i].enabled)\n +\t\t\tcontinue;\n -+\t\tresult = tasks[i]->fn();\n ++\n ++\t\tif (tasks[i].fn()) {\n ++\t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n ++\t\t\tresult = 1;\n ++\t\t}\n +\t}\n +\n +\treturn result;\n -+}\n -+\n -+static void initialize_tasks(void)\n -+{\n -+\tint i;\n -+\tnum_tasks = 0;\n -+\n -+\tfor (i = 0; i < MAX_NUM_TASKS; i++)\n -+\t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n -+\n -+\ttasks[num_tasks]->name = \"gc\";\n -+\ttasks[num_tasks]->fn = maintenance_task_gc;\n -+\ttasks[num_tasks]->enabled = 1;\n -+\tnum_tasks++;\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\t\t\t builtin_maintenance_options);\n - \n - \topts.quiet = !isatty(2);\n -+\tinitialize_tasks();\n - \n - \targc = parse_options(argc, argv, prefix,\n - \t\t\t builtin_maintenance_options,\n 5: 04552b1d2e ! 5: a4d9836bed maintenance: add commit-graph task\n @@ Documentation/git-maintenance.txt: run::\n \tstands for \"garbage collection,\" but this task performs many\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - \treturn 0;\n - }\n - \n --#define MAX_NUM_TASKS 1\n -+#define MAX_NUM_TASKS 2\n - \n - static const char * const builtin_maintenance_usage[] = {\n - \tN_(\"git maintenance run []\"),\n @@ builtin/gc.c: static struct maintenance_opts {\n \tint quiet;\n } opts;\n \n +static int run_write_commit_graph(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n +\n -+\targv_array_pushl(&cmd, \"commit-graph\", \"write\",\n -+\t\t\t \"--split\", \"--reachable\", NULL);\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"commit-graph\", \"write\",\n ++\t\t \"--split\", \"--reachable\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n -+\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n -+\treturn result;\n ++\treturn !!run_command(&child);\n +}\n +\n +static int run_verify_commit_graph(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n +\n -+\targv_array_pushl(&cmd, \"commit-graph\", \"verify\",\n -+\t\t\t \"--shallow\", NULL);\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"commit-graph\", \"verify\",\n ++\t\t \"--shallow\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n -+\n -+\treturn result;\n ++\treturn !!run_command(&child);\n +}\n +\n +static int maintenance_task_commit_graph(void)\n @@ builtin/gc.c: static struct maintenance_opts {\n +\tstruct repository *r = the_repository;\n +\tchar *chain_path;\n +\n -+\t/* Skip commit-graph when --auto is specified. */\n -+\tif (opts.auto_flag)\n -+\t\treturn 0;\n -+\n +\tclose_object_store(r->objects);\n +\tif (run_write_commit_graph()) {\n +\t\terror(_(\"failed to write commit-graph\"));\n @@ builtin/gc.c: static struct maintenance_opts {\n +\n static int maintenance_task_gc(void)\n {\n - \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \ttasks[num_tasks]->fn = maintenance_task_gc;\n - \ttasks[num_tasks]->enabled = 1;\n - \tnum_tasks++;\n -+\n -+\ttasks[num_tasks]->name = \"commit-graph\";\n -+\ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n -+\tnum_tasks++;\n - }\n + \tstruct child_process child = CHILD_PROCESS_INIT;\n +@@ builtin/gc.c: struct maintenance_task {\n + \n + enum maintenance_task_label {\n + \tTASK_GC,\n ++\tTASK_COMMIT_GRAPH,\n \n - int cmd_maintenance(int argc, const char **argv, const char *prefix)\n + \t/* Leave as final value */\n + \tTASK__COUNT\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t\tmaintenance_task_gc,\n + \t\t1,\n + \t},\n ++\t[TASK_COMMIT_GRAPH] = {\n ++\t\t\"commit-graph\",\n ++\t\tmaintenance_task_commit_graph,\n ++\t},\n + };\n + \n + static int maintenance_run(void)\n \n ## commit-graph.c ##\n @@ commit-graph.c: static char *get_split_graph_filename(struct object_directory *odb,\n @@ commit-graph.h: struct commit;\n /*\n \n ## t/t7900-maintenance.sh ##\n -@@ t/t7900-maintenance.sh: test_expect_success 'help text' '\n - \ttest_i18ngrep \"usage: git maintenance run\" err\n - '\n +@@ t/t7900-maintenance.sh: test_description='git maintenance builtin'\n \n --test_expect_success 'gc [--auto|--quiet]' '\n -+test_expect_success 'run [--auto|--quiet]' '\n - \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n - \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n - \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n + . ./test-lib.sh\n + \n ++GIT_TEST_COMMIT_GRAPH=0\n ++\n + test_expect_success 'help text' '\n + \ttest_expect_code 129 git maintenance -h 2>err &&\n + \ttest_i18ngrep \"usage: git maintenance run\" err\n 6: a09b1c1687 ! 6: dafb0d9bbc maintenance: add --task option\n @@ Commit message\n references. We use the hashmap to match the --task= arguments into\n the task struct data.\n \n + Keep in mind that the 'enabled' member of the maintenance_task struct is\n + a placeholder for a future 'maintenance..enabled' config option.\n + Thus, we use the 'enabled' member to specify which tasks are run when\n + the user does not specify any --task= arguments. The 'enabled'\n + member should be ignored if --task= appears.\n +\n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-maintenance.txt ##\n @@ builtin/gc.c: typedef int maintenance_task_fn(void);\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n -\tunsigned enabled:1;\n -+\tint task_order;\n +\tunsigned enabled:1,\n +\t\t selected:1;\n ++\tint selected_order;\n };\n \n - static struct maintenance_task *tasks[MAX_NUM_TASKS];\n - static int num_tasks;\n + enum maintenance_task_label {\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t},\n + };\n \n +static int compare_tasks_by_selection(const void *a_, const void *b_)\n +{\n +\tconst struct maintenance_task *a, *b;\n -+\ta = (const struct maintenance_task *)a_;\n -+\tb = (const struct maintenance_task *)b_;\n +\n -+\treturn b->task_order - a->task_order;\n ++\ta = (const struct maintenance_task *)&a_;\n ++\tb = (const struct maintenance_task *)&b_;\n ++\n ++\treturn b->selected_order - a->selected_order;\n +}\n +\n static int maintenance_run(void)\n @@ builtin/gc.c: typedef int maintenance_task_fn(void);\n \tint result = 0;\n \n +\tif (opts.tasks_selected)\n -+\t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n ++\t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n +\n - \tfor (i = 0; !result && i < num_tasks; i++) {\n --\t\tif (!tasks[i]->enabled)\n -+\t\tif (opts.tasks_selected && !tasks[i]->selected)\n + \tfor (i = 0; i < TASK__COUNT; i++) {\n +-\t\tif (!tasks[i].enabled)\n ++\t\tif (opts.tasks_selected && !tasks[i].selected)\n +\t\t\tcontinue;\n +\n -+\t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n ++\t\tif (!opts.tasks_selected && !tasks[i].enabled)\n \t\t\tcontinue;\n -+\n - \t\tresult = tasks[i]->fn();\n - \t}\n \n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \tnum_tasks++;\n + \t\tif (tasks[i].fn()) {\n +@@ builtin/gc.c: static int maintenance_run(void)\n + \treturn result;\n }\n \n +static int task_option_parse(const struct option *opt,\n @@ builtin/gc.c: static void initialize_tasks(void)\n +\n +\tBUG_ON_OPT_NEG(unset);\n +\n -+\tif (!arg || !strlen(arg)) {\n -+\t\terror(_(\"--task requires a value\"));\n -+\t\treturn 1;\n -+\t}\n -+\n +\topts.tasks_selected++;\n +\n -+\tfor (i = 0; i < MAX_NUM_TASKS; i++) {\n -+\t\tif (tasks[i] && !strcasecmp(tasks[i]->name, arg)) {\n -+\t\t\ttask = tasks[i];\n ++\tfor (i = 0; i < TASK__COUNT; i++) {\n ++\t\tif (!strcasecmp(tasks[i].name, arg)) {\n ++\t\t\ttask = &tasks[i];\n +\t\t\tbreak;\n +\t\t}\n +\t}\n @@ builtin/gc.c: static void initialize_tasks(void)\n +\t}\n +\n +\ttask->selected = 1;\n -+\ttask->task_order = opts.tasks_selected;\n ++\ttask->selected_order = opts.tasks_selected;\n +\n +\treturn 0;\n +}\n 7: e9260a9c3f ! 7: 1b00524da3 maintenance: take a lock on the objects directory\n @@ builtin/gc.c: static int maintenance_run(void)\n +\tfree(lock_path);\n \n \tif (opts.tasks_selected)\n - \t\tQSORT(tasks, num_tasks, compare_tasks_by_selection);\n + \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n @@ builtin/gc.c: static int maintenance_run(void)\n - \t\tresult = tasks[i]->fn();\n + \t\t}\n \t}\n \n +\trollback_lock_file(&lk);\n -: ---------- > 8: 0e94e04dcd fetch: optionally allow disabling FETCH_HEAD update\n 8: 3165b8916d ! 9: 9e38ade15c maintenance: add prefetch task\n @@ Commit message\n \n 2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n + While this looks confusing, this was documented and tested by\n + b40a50264ac (fetch: document and test --refmap=\"\", 2020-01-21),\n + including this sentence in the documentation:\n +\n + Providing an empty `` to the `--refmap` option\n + causes Git to ignore the configured refspecs and rely\n + entirely on the refspecs supplied as command-line arguments.\n \n 3. By adding a new refspec \"+refs/heads/*:refs/prefetch//*\"\n we can ensure that we actually load the new values somewhere in\n @@ Commit message\n 4. --prune will delete the refs/prefetch/ refs that no\n longer appear on the remote.\n \n + 5. --no-write-fetch-head prevents updating FETCH_HEAD.\n +\n We've been using this step as a critical background job in Scalar\n [1] (and VFS for Git). This solved a pain point that was showing up\n in user reports: fetching was a pain! Users do not like waiting to\n @@ Documentation/git-maintenance.txt: since it will not expire `.graph` files that\n the expiration delay.\n \n +prefetch::\n -+\tThe `fetch` task updates the object directory with the latest objects\n -+\tfrom all registered remotes. For each remote, a `git fetch` command\n -+\tis run. The refmap is custom to avoid updating local or remote\n ++\tThe `prefetch` task updates the object directory with the latest\n ++\tobjects from all registered remotes. For each remote, a `git fetch`\n ++\tcommand is run. The refmap is custom to avoid updating local or remote\n +\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n +\tremote refs are stored in `refs/prefetch//`. Also, tags are\n +\tnot updated.\n ++\n -+This means that foreground fetches are still required to update the\n -+remote refs, but the users is notified when the branches and tags are\n -+updated on the remote.\n ++This is done to avoid disrupting the remote-tracking branches. The end users\n ++expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n ++task, however, the objects necessary to complete a later real fetch would\n ++already be obtained, so the real fetch would go faster. In the ideal case,\n ++it will just become an update to bunch of remote-tracking branches without\n ++any object transfer.\n +\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n @@ builtin/gc.c\n \n #define FAILED_RUN \"failed to run %s\"\n \n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - \treturn 0;\n - }\n - \n --#define MAX_NUM_TASKS 2\n -+#define MAX_NUM_TASKS 3\n - \n - static const char * const builtin_maintenance_usage[] = {\n - \tN_(\"git maintenance run []\"),\n @@ builtin/gc.c: static int maintenance_task_commit_graph(void)\n \treturn 1;\n }\n \n +static int fetch_remote(const char *remote)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\tstruct strbuf refmap = STRBUF_INIT;\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n +\n -+\targv_array_pushl(&cmd, \"fetch\", remote, \"--prune\",\n -+\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n ++\t\t \"--no-write-fetch-head\", \"--refmap=\", NULL);\n +\n -+\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n -+\targv_array_push(&cmd, refmap.buf);\n ++\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--quiet\");\n ++\t\tstrvec_push(&child.args, \"--quiet\");\n +\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\n -+\tstrbuf_release(&refmap);\n -+\treturn result;\n ++\treturn !!run_command(&child);\n +}\n +\n +static int fill_each_remote(struct remote *remote, void *cbdata)\n @@ builtin/gc.c: static int maintenance_task_commit_graph(void)\n +\t\tgoto cleanup;\n +\t}\n +\n -+\t/*\n -+\t * Do not modify the result based on the success of the 'fetch'\n -+\t * operation, as a loss of network could cause 'fetch' to fail\n -+\t * quickly. We do not want that to stop the rest of our\n -+\t * background operations.\n -+\t */\n +\tfor (item = remotes.items;\n +\t item && item < remotes.items + remotes.nr;\n +\t item++)\n -+\t\tfetch_remote(item->string);\n ++\t\tresult |= fetch_remote(item->string);\n +\n +cleanup:\n +\tstring_list_clear(&remotes, 0);\n @@ builtin/gc.c: static int maintenance_task_commit_graph(void)\n +\n static int maintenance_task_gc(void)\n {\n - \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n - \t\ttasks[i] = xcalloc(1, sizeof(struct maintenance_task));\n + \tstruct child_process child = CHILD_PROCESS_INIT;\n +@@ builtin/gc.c: struct maintenance_task {\n + };\n \n -+\ttasks[num_tasks]->name = \"prefetch\";\n -+\ttasks[num_tasks]->fn = maintenance_task_prefetch;\n -+\tnum_tasks++;\n -+\n - \ttasks[num_tasks]->name = \"gc\";\n - \ttasks[num_tasks]->fn = maintenance_task_gc;\n - \ttasks[num_tasks]->enabled = 1;\n + enum maintenance_task_label {\n ++\tTASK_PREFETCH,\n + \tTASK_GC,\n + \tTASK_COMMIT_GRAPH,\n + \n +@@ builtin/gc.c: enum maintenance_task_label {\n + };\n + \n + static struct maintenance_task tasks[] = {\n ++\t[TASK_PREFETCH] = {\n ++\t\t\"prefetch\",\n ++\t\tmaintenance_task_prefetch,\n ++\t},\n + \t[TASK_GC] = {\n + \t\t\"gc\",\n + \t\tmaintenance_task_gc,\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'run --task duplicate' '\n 9: 83648f4865 ! 10: 0128fdfd1a maintenance: add loose-objects task\n @@ Documentation/git-maintenance.txt: gc::\n --auto::\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - \treturn 0;\n - }\n - \n --#define MAX_NUM_TASKS 3\n -+#define MAX_NUM_TASKS 4\n - \n - static const char * const builtin_maintenance_usage[] = {\n - \tN_(\"git maintenance run []\"),\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n - \treturn result;\n + \treturn run_command(&child);\n }\n \n +static int prune_packed(void)\n +{\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"prune-packed\", NULL);\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n ++\n ++\tchild.git_cmd = 1;\n ++\tstrvec_push(&child.args, \"prune-packed\");\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--quiet\");\n ++\t\tstrvec_push(&child.args, \"--quiet\");\n +\n -+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n ++\treturn !!run_command(&child);\n +}\n +\n +struct write_loose_object_data {\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n +\tint batch_size;\n +};\n +\n -+static int loose_object_exists(const struct object_id *oid,\n -+\t\t\t const char *path,\n -+\t\t\t void *data)\n ++static int bail_on_loose(const struct object_id *oid,\n ++\t\t\t const char *path,\n ++\t\t\t void *data)\n +{\n +\treturn 1;\n +}\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n +\tstruct repository *r = the_repository;\n +\tint result = 0;\n +\tstruct write_loose_object_data data;\n -+\tstruct strbuf prefix = STRBUF_INIT;\n -+\tstruct child_process *pack_proc;\n ++\tstruct child_process pack_proc = CHILD_PROCESS_INIT;\n +\n +\t/*\n +\t * Do not start pack-objects process\n +\t * if there are no loose objects.\n +\t */\n +\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n -+\t\t\t\t\t loose_object_exists,\n ++\t\t\t\t\t bail_on_loose,\n +\t\t\t\t\t NULL, NULL, NULL))\n +\t\treturn 0;\n +\n -+\tpack_proc = xmalloc(sizeof(*pack_proc));\n -+\n -+\tchild_process_init(pack_proc);\n ++\tpack_proc.git_cmd = 1;\n +\n -+\tstrbuf_addstr(&prefix, r->objects->odb->path);\n -+\tstrbuf_addstr(&prefix, \"/pack/loose\");\n -+\n -+\targv_array_pushl(&pack_proc->args, \"git\", \"pack-objects\", NULL);\n ++\tstrvec_push(&pack_proc.args, \"pack-objects\");\n +\tif (opts.quiet)\n -+\t\targv_array_push(&pack_proc->args, \"--quiet\");\n -+\targv_array_push(&pack_proc->args, prefix.buf);\n ++\t\tstrvec_push(&pack_proc.args, \"--quiet\");\n ++\tstrvec_pushf(&pack_proc.args, \"%s/pack/loose\", r->objects->odb->path);\n +\n -+\tpack_proc->in = -1;\n ++\tpack_proc.in = -1;\n +\n -+\tif (start_command(pack_proc)) {\n ++\tif (start_command(&pack_proc)) {\n +\t\terror(_(\"failed to start 'git pack-objects' process\"));\n -+\t\tresult = 1;\n -+\t\tgoto cleanup;\n ++\t\treturn 1;\n +\t}\n +\n -+\tdata.in = xfdopen(pack_proc->in, \"w\");\n ++\tdata.in = xfdopen(pack_proc.in, \"w\");\n +\tdata.count = 0;\n +\tdata.batch_size = 50000;\n +\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n +\n +\tfclose(data.in);\n +\n -+\tif (finish_command(pack_proc)) {\n ++\tif (finish_command(&pack_proc)) {\n +\t\terror(_(\"failed to finish 'git pack-objects' process\"));\n +\t\tresult = 1;\n +\t}\n +\n -+cleanup:\n -+\tstrbuf_release(&prefix);\n -+\tfree(pack_proc);\n +\treturn result;\n +}\n +\n @@ builtin/gc.c: static int maintenance_task_gc(void)\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \ttasks[num_tasks]->fn = maintenance_task_prefetch;\n - \tnum_tasks++;\n +@@ builtin/gc.c: struct maintenance_task {\n \n -+\ttasks[num_tasks]->name = \"loose-objects\";\n -+\ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n -+\tnum_tasks++;\n -+\n - \ttasks[num_tasks]->name = \"gc\";\n - \ttasks[num_tasks]->fn = maintenance_task_gc;\n - \ttasks[num_tasks]->enabled = 1;\n + enum maintenance_task_label {\n + \tTASK_PREFETCH,\n ++\tTASK_LOOSE_OBJECTS,\n + \tTASK_GC,\n + \tTASK_COMMIT_GRAPH,\n + \n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t\t\"prefetch\",\n + \t\tmaintenance_task_prefetch,\n + \t},\n ++\t[TASK_LOOSE_OBJECTS] = {\n ++\t\t\"loose-objects\",\n ++\t\tmaintenance_task_loose_objects,\n ++\t},\n + \t[TASK_GC] = {\n + \t\t\"gc\",\n + \t\tmaintenance_task_gc,\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n +\n +\t# Hack to stop maintenance from running during \"git commit\"\n +\techo in use >.git/objects/maintenance.lock &&\n ++\n ++\t# Assuming that \"git commit\" creates at least one loose object\n +\ttest_commit create-loose-object &&\n +\trm .git/objects/maintenance.lock &&\n +\n @@ t/t7900-maintenance.sh: test_expect_success 'prefetch multiple remotes' '\n +\ttest_cmp obj-dir-before obj-dir-between &&\n +\tls .git/objects/pack/*.pack >packs-between &&\n +\ttest_line_count = 2 packs-between &&\n ++\tls .git/objects/pack/loose-*.pack >loose-packs &&\n ++\ttest_line_count = 1 loose-packs &&\n +\n +\t# The second run deletes loose objects\n +\t# but does not create a pack-file.\n -: ---------- > 11: c2baf6e119 midx: enable core.multiPackIndex by default\n 10: b6328c2106 ! 12: 00f47c4848 maintenance: add incremental-repack task\n @@ Commit message\n size\" is calculated by taking the size of the pack-file divided\n by the number of objects in the pack-file and multiplied by the\n number of objects from the multi-pack-index with offset in that\n - pack-file. The expected size approximats how much data from that\n + pack-file. The expected size approximates how much data from that\n pack-file will contribute to the resulting pack-file size. The\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n @@ builtin/gc.c\n \n #define FAILED_RUN \"failed to run %s\"\n \n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - \treturn 0;\n - }\n - \n --#define MAX_NUM_TASKS 4\n -+#define MAX_NUM_TASKS 5\n - \n - static const char * const builtin_maintenance_usage[] = {\n - \tN_(\"git maintenance run []\"),\n @@ builtin/gc.c: static int maintenance_task_loose_objects(void)\n \treturn prune_packed() || pack_loose();\n }\n \n +static int multi_pack_index_write(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"multi-pack-index\", \"write\", NULL);\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n ++\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"multi-pack-index\", \"write\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n ++\tif (run_command(&child))\n ++\t\treturn error(_(\"failed to write multi-pack-index\"));\n +\n -+\treturn result;\n ++\treturn 0;\n +}\n +\n +static int rewrite_multi_pack_index(void)\n @@ builtin/gc.c: static int maintenance_task_loose_objects(void)\n +\tunlink(midx_name);\n +\tfree(midx_name);\n +\n -+\tif (multi_pack_index_write()) {\n -+\t\terror(_(\"failed to rewrite multi-pack-index\"));\n -+\t\treturn 1;\n -+\t}\n -+\n -+\treturn 0;\n ++\treturn multi_pack_index_write();\n +}\n +\n -+static int multi_pack_index_verify(void)\n ++static int multi_pack_index_verify(const char *message)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"multi-pack-index\", \"verify\", NULL);\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n ++\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"multi-pack-index\", \"verify\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n ++\tif (run_command(&child)) {\n ++\t\twarning(_(\"'git multi-pack-index verify' failed %s\"), message);\n ++\t\treturn 1;\n ++\t}\n +\n -+\treturn result;\n ++\treturn 0;\n +}\n +\n +static int multi_pack_index_expire(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"multi-pack-index\", \"expire\", NULL);\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n ++\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"multi-pack-index\", \"expire\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n +\tclose_object_store(the_repository->objects);\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\targv_array_clear(&cmd);\n +\n -+\treturn result;\n ++\tif (run_command(&child))\n ++\t\treturn error(_(\"'git multi-pack-index expire' failed\"));\n ++\n ++\treturn 0;\n +}\n +\n +static int multi_pack_index_repack(void)\n +{\n -+\tint result;\n -+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n ++\tstruct child_process child = CHILD_PROCESS_INIT;\n ++\n ++\tchild.git_cmd = 1;\n ++\tstrvec_pushl(&child.args, \"multi-pack-index\", \"repack\", NULL);\n +\n +\tif (opts.quiet)\n -+\t\targv_array_push(&cmd, \"--no-progress\");\n ++\t\tstrvec_push(&child.args, \"--no-progress\");\n +\n -+\targv_array_push(&cmd, \"--batch-size=0\");\n ++\tstrvec_push(&child.args, \"--batch-size=0\");\n +\n +\tclose_object_store(the_repository->objects);\n -+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n +\n -+\tif (result && multi_pack_index_verify()) {\n -+\t\twarning(_(\"multi-pack-index verify failed after repack\"));\n -+\t\tresult = rewrite_multi_pack_index();\n -+\t}\n ++\tif (run_command(&child))\n ++\t\treturn error(_(\"'git multi-pack-index repack' failed\"));\n +\n -+\treturn result;\n ++\treturn 0;\n +}\n +\n +static int maintenance_task_incremental_repack(void)\n +{\n -+\tif (multi_pack_index_write()) {\n -+\t\terror(_(\"failed to write multi-pack-index\"));\n -+\t\treturn 1;\n ++\tprepare_repo_settings(the_repository);\n ++\tif (!the_repository->settings.core_multi_pack_index) {\n ++\t\twarning(_(\"skipping incremental-repack task because core.multiPackIndex is disabled\"));\n ++\t\treturn 0;\n +\t}\n +\n -+\tif (multi_pack_index_verify()) {\n -+\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n -+\t\treturn rewrite_multi_pack_index();\n -+\t}\n -+\n -+\tif (multi_pack_index_expire()) {\n -+\t\terror(_(\"multi-pack-index expire failed\"));\n ++\tif (multi_pack_index_write())\n +\t\treturn 1;\n -+\t}\n -+\n -+\tif (multi_pack_index_verify()) {\n -+\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n ++\tif (multi_pack_index_verify(\"after initial write\"))\n +\t\treturn rewrite_multi_pack_index();\n -+\t}\n -+\n -+\tif (multi_pack_index_repack()) {\n -+\t\terror(_(\"multi-pack-index repack failed\"));\n ++\tif (multi_pack_index_expire())\n +\t\treturn 1;\n -+\t}\n -+\n ++\tif (multi_pack_index_verify(\"after expire step\"))\n ++\t\treturn !!rewrite_multi_pack_index();\n ++\tif (multi_pack_index_repack())\n ++\t\treturn 1;\n ++\tif (multi_pack_index_verify(\"after repack step\"))\n ++\t\treturn !!rewrite_multi_pack_index();\n +\treturn 0;\n +}\n +\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n - \tnum_tasks++;\n +@@ builtin/gc.c: struct maintenance_task {\n + enum maintenance_task_label {\n + \tTASK_PREFETCH,\n + \tTASK_LOOSE_OBJECTS,\n ++\tTASK_INCREMENTAL_REPACK,\n + \tTASK_GC,\n + \tTASK_COMMIT_GRAPH,\n \n -+\ttasks[num_tasks]->name = \"incremental-repack\";\n -+\ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n -+\tnum_tasks++;\n -+\n - \ttasks[num_tasks]->name = \"gc\";\n - \ttasks[num_tasks]->fn = maintenance_task_gc;\n - \ttasks[num_tasks]->enabled = 1;\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t\t\"loose-objects\",\n + \t\tmaintenance_task_loose_objects,\n + \t},\n ++\t[TASK_INCREMENTAL_REPACK] = {\n ++\t\t\"incremental-repack\",\n ++\t\tmaintenance_task_incremental_repack,\n ++\t},\n + \t[TASK_GC] = {\n + \t\t\"gc\",\n + \t\tmaintenance_task_gc,\n \n ## midx.c ##\n @@\n @@ midx.h: struct multi_pack_index {\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\n \n + ## t/t5319-multi-pack-index.sh ##\n +@@\n + test_description='multi-pack-indexes'\n + . ./test-lib.sh\n + \n ++GIT_TEST_MULTI_PACK_INDEX=0\n + objdir=.git/objects\n + \n + midx_read_expect () {\n +\n ## t/t7900-maintenance.sh ##\n +@@ t/t7900-maintenance.sh: test_description='git maintenance builtin'\n + . ./test-lib.sh\n + \n + GIT_TEST_COMMIT_GRAPH=0\n ++GIT_TEST_MULTI_PACK_INDEX=0\n + \n + test_expect_success 'help text' '\n + \ttest_expect_code 129 git maintenance -h 2>err &&\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n 11: 478c7f1d0b ! 13: ef2a231956 maintenance: auto-size incremental-repack batch\n @@ Commit message\n When repacking during the 'incremental-repack' task, we use the\n --batch-size option in 'git multi-pack-index repack'. The initial setting\n used --batch-size=0 to repack everything into a single pack-file. This is\n - not sustaintable for a large repository. The amount of work required is\n + not sustainable for a large repository. The amount of work required is\n also likely to use too many system resources for a background job.\n \n Update the 'incremental-repack' task by dynamically computing a\n @@ Commit message\n \n ## builtin/gc.c ##\n @@ builtin/gc.c: static int multi_pack_index_expire(void)\n - \treturn result;\n + \treturn 0;\n }\n \n -+#define TWO_GIGABYTES (2147483647)\n -+#define UNSET_BATCH_SIZE ((unsigned long)-1)\n ++#define TWO_GIGABYTES (0x7FFF)\n +\n +static off_t get_auto_pack_size(void)\n +{\n @@ builtin/gc.c: static int multi_pack_index_expire(void)\n +\n static int multi_pack_index_repack(void)\n {\n - \tint result;\n - \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n -+\tstruct strbuf batch_arg = STRBUF_INIT;\n -+\n - \targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\", NULL);\n - \n + \tstruct child_process child = CHILD_PROCESS_INIT;\n +@@ builtin/gc.c: static int multi_pack_index_repack(void)\n \tif (opts.quiet)\n - \t\targv_array_push(&cmd, \"--no-progress\");\n + \t\tstrvec_push(&child.args, \"--no-progress\");\n \n --\targv_array_push(&cmd, \"--batch-size=0\");\n -+\tstrbuf_addf(&batch_arg, \"--batch-size=%\"PRIuMAX,\n -+\t\t (uintmax_t)get_auto_pack_size());\n -+\targv_array_push(&cmd, batch_arg.buf);\n +-\tstrvec_push(&child.args, \"--batch-size=0\");\n ++\tstrvec_pushf(&child.args, \"--batch-size=%\"PRIuMAX,\n ++\t\t\t\t (uintmax_t)get_auto_pack_size());\n \n \tclose_object_store(the_repository->objects);\n - \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n -+\tstrbuf_release(&batch_arg);\n \n - \tif (result && multi_pack_index_verify()) {\n - \t\twarning(_(\"multi-pack-index verify failed after repack\"));\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '\n 12: a3c64930a0 ! 14: 99840c4b8f maintenance: create maintenance..enabled config\n @@ Documentation/git-maintenance.txt: SUBCOMMANDS\n \n ## builtin/gc.c ##\n @@ builtin/gc.c: static int maintenance_run(void)\n - static void initialize_tasks(void)\n - {\n - \tint i;\n -+\tstruct strbuf config_name = STRBUF_INIT;\n - \tnum_tasks = 0;\n + \treturn result;\n + }\n \n - \tfor (i = 0; i < MAX_NUM_TASKS; i++)\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \ttasks[num_tasks]->name = \"commit-graph\";\n - \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n - \tnum_tasks++;\n -+\n -+\tfor (i = 0; i < num_tasks; i++) {\n ++static void initialize_task_config(void)\n ++{\n ++\tint i;\n ++\tstruct strbuf config_name = STRBUF_INIT;\n ++\tfor (i = 0; i < TASK__COUNT; i++) {\n +\t\tint config_value;\n +\n +\t\tstrbuf_setlen(&config_name, 0);\n -+\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\", tasks[i]->name);\n ++\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n ++\t\t\t tasks[i].name);\n +\n +\t\tif (!git_config_get_bool(config_name.buf, &config_value))\n -+\t\t\ttasks[i]->enabled = config_value;\n ++\t\t\ttasks[i].enabled = config_value;\n +\t}\n +\n +\tstrbuf_release(&config_name);\n - }\n - \n ++}\n ++\n static int task_option_parse(const struct option *opt,\n + \t\t\t const char *arg, int unset)\n + {\n +@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n + \t\t\t\t builtin_maintenance_options);\n + \n + \topts.quiet = !isatty(2);\n ++\tinitialize_task_config();\n + \n + \targc = parse_options(argc, argv, prefix,\n + \t\t\t builtin_maintenance_options,\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'run [--auto|--quiet]' '\n 13: dbacc2b76c ! 15: a087c63572 maintenance: use pointers to check --auto\n @@ builtin/gc.c: static int maintenance_task_incremental_repack(void)\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n +\tmaintenance_auto_fn *auto_condition;\n - \tint task_order;\n \tunsigned enabled:1,\n \t\t selected:1;\n + \tint selected_order;\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t[TASK_GC] = {\n + \t\t\"gc\",\n + \t\tmaintenance_task_gc,\n ++\t\tneed_to_gc,\n + \t\t1,\n + \t},\n + \t[TASK_COMMIT_GRAPH] = {\n @@ builtin/gc.c: static int maintenance_run(void)\n - \t\tif (!opts.tasks_selected && !tasks[i]->enabled)\n + \t\tif (!opts.tasks_selected && !tasks[i].enabled)\n \t\t\tcontinue;\n \n +\t\tif (opts.auto_flag &&\n -+\t\t (!tasks[i]->auto_condition ||\n -+\t\t !tasks[i]->auto_condition()))\n ++\t\t (!tasks[i].auto_condition ||\n ++\t\t !tasks[i].auto_condition()))\n +\t\t\tcontinue;\n +\n - \t\tresult = tasks[i]->fn();\n - \t}\n - \n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \n - \ttasks[num_tasks]->name = \"gc\";\n - \ttasks[num_tasks]->fn = maintenance_task_gc;\n -+\ttasks[num_tasks]->auto_condition = need_to_gc;\n - \ttasks[num_tasks]->enabled = 1;\n - \tnum_tasks++;\n - \n -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\t\t\t builtin_maintenance_options);\n - \n - \topts.quiet = !isatty(2);\n + \t\tif (tasks[i].fn()) {\n + \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n + \t\t\tresult = 1;\n +@@ builtin/gc.c: static void initialize_task_config(void)\n + {\n + \tint i;\n + \tstruct strbuf config_name = STRBUF_INIT;\n +\tgc_config();\n - \tinitialize_tasks();\n ++\n + \tfor (i = 0; i < TASK__COUNT; i++) {\n + \t\tint config_value;\n \n - \targc = parse_options(argc, argv, prefix,\n \n ## t/t5514-fetch-multiple.sh ##\n @@ t/t5514-fetch-multiple.sh: test_expect_success 'git fetch --multiple (two remotes)' '\n 14: 9af2309f08 ! 16: ef3a854508 maintenance: add auto condition for commit-graph task\n @@ builtin/gc.c: static struct maintenance_opts {\n +\n static int run_write_commit_graph(void)\n {\n - \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(void)\n + \tstruct child_process child = CHILD_PROCESS_INIT;\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t[TASK_COMMIT_GRAPH] = {\n + \t\t\"commit-graph\",\n + \t\tmaintenance_task_commit_graph,\n ++\t\tshould_write_commit_graph,\n + \t},\n + };\n \n - \ttasks[num_tasks]->name = \"commit-graph\";\n - \ttasks[num_tasks]->fn = maintenance_task_commit_graph;\n -+\ttasks[num_tasks]->auto_condition = should_write_commit_graph;\n - \tnum_tasks++;\n - \n - \tfor (i = 0; i < num_tasks; i++) {\n \n ## object.h ##\n @@ object.h: struct object_array {\n 15: 42e316ca58 ! 17: 6ac3a58f2f maintenance: create auto condition for loose-objects\n @@ builtin/gc.c: struct write_loose_object_data {\n +\t\t\t\t\t NULL, NULL, &count);\n +}\n +\n - static int loose_object_exists(const struct object_id *oid,\n - \t\t\t const char *path,\n - \t\t\t void *data)\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \n - \ttasks[num_tasks]->name = \"loose-objects\";\n - \ttasks[num_tasks]->fn = maintenance_task_loose_objects;\n -+\ttasks[num_tasks]->auto_condition = loose_object_auto_condition;\n - \tnum_tasks++;\n - \n - \ttasks[num_tasks]->name = \"incremental-repack\";\n + static int bail_on_loose(const struct object_id *oid,\n + \t\t\t const char *path,\n + \t\t\t void *data)\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t[TASK_LOOSE_OBJECTS] = {\n + \t\t\"loose-objects\",\n + \t\tmaintenance_task_loose_objects,\n ++\t\tloose_object_auto_condition,\n + \t},\n + \t[TASK_INCREMENTAL_REPACK] = {\n + \t\t\"incremental-repack\",\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'loose-objects task' '\n 16: 3d527cb0dd ! 18: 801b262d1c maintenance: add incremental-repack auto condition\n @@ builtin/gc.c: static int maintenance_task_loose_objects(void)\n +\n static int multi_pack_index_write(void)\n {\n - \tint result;\n -@@ builtin/gc.c: static void initialize_tasks(void)\n - \n - \ttasks[num_tasks]->name = \"incremental-repack\";\n - \ttasks[num_tasks]->fn = maintenance_task_incremental_repack;\n -+\ttasks[num_tasks]->auto_condition = incremental_repack_auto_condition;\n - \tnum_tasks++;\n - \n - \ttasks[num_tasks]->name = \"gc\";\n + \tstruct child_process child = CHILD_PROCESS_INIT;\n +@@ builtin/gc.c: static struct maintenance_task tasks[] = {\n + \t[TASK_INCREMENTAL_REPACK] = {\n + \t\t\"incremental-repack\",\n + \t\tmaintenance_task_incremental_repack,\n ++\t\tincremental_repack_auto_condition,\n + \t},\n + \t[TASK_GC] = {\n + \t\t\"gc\",\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'incremental-repack task' '\n 17: a0f00f8ab8 = 19: 9b4cef7635 midx: use start_delayed_progress()\n 18: f24db7739f ! 20: 39eb83ad1e maintenance: add trace2 regions for task execution\n @@ Commit message\n \n ## builtin/gc.c ##\n @@ builtin/gc.c: static int maintenance_run(void)\n - \t\t !tasks[i]->auto_condition()))\n + \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n -+\t\ttrace2_region_enter(\"maintenance\", tasks[i]->name, r);\n - \t\tresult = tasks[i]->fn();\n -+\t\ttrace2_region_leave(\"maintenance\", tasks[i]->name, r);\n ++\t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n + \t\tif (tasks[i].fn()) {\n + \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n + \t\t\tresult = 1;\n + \t\t}\n ++\t\ttrace2_region_leave(\"maintenance\", tasks[i].name, r);\n \t}\n \n \trollback_lock_file(&lk);\n\n-- \ngitgitgadget\n"},{"id":"402484","messageId":"12fe73bb72bf2193967979acf68f7645b339eaa2.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 01/20] maintenance: create basic maintenance runner","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:07Z","receivedAt":"2020-07-30T22:24:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'gc' builtin is our current entrypoint for automatically maintaining\na repository. This one tool does many operations, such as repacking the\nrepository, packing refs, and rewriting the commit-graph file. The name\nimplies it performs \"garbage collection\" which means several different\nthings, and some users may not want to use this operation that rewrites\nthe entire object database.\n\nCreate a new 'maintenance' builtin that will become a more general-\npurpose command. To start, it will only support the 'run' subcommand,\nbut will later expand to add subcommands for scheduling maintenance in\nthe background.\n\nFor now, the 'maintenance' builtin is a thin shim over the 'gc' builtin.\nIn fact, the only option is the '--auto' toggle, which is handed\ndirectly to the 'gc' builtin. The current change is isolated to this\nsimple operation to prevent more interesting logic from being lost in\nall of the boilerplate of adding a new builtin.\n\nUse existing builtin/gc.c file because we want to share code between the\ntwo builtins. It is possible that we will have 'maintenance' replace the\n'gc' builtin entirely at some point, leaving 'git gc' as an alias for\nsome specific arguments to 'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-maintenance.txt | 57 +++++++++++++++++++++++++++++++\n builtin.h | 1 +\n builtin/gc.c | 56 ++++++++++++++++++++++++++++++\n git.c | 1 +\n t/t7900-maintenance.sh | 19 +++++++++++\n 6 files changed, 135 insertions(+)\n create mode 100644 Documentation/git-maintenance.txt\n create mode 100755 t/t7900-maintenance.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex ee509a2ad2..a5808fa30d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -90,6 +90,7 @@\n /git-ls-tree\n /git-mailinfo\n /git-mailsplit\n+/git-maintenance\n /git-merge\n /git-merge-base\n /git-merge-index\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nnew file mode 100644\nindex 0000000000..34cd2b4417\n--- /dev/null\n+++ b/Documentation/git-maintenance.txt\n@@ -0,0 +1,57 @@\n+git-maintenance(1)\n+==================\n+\n+NAME\n+----\n+git-maintenance - Run tasks to optimize Git repository data\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git maintenance' run []\n+\n+\n+DESCRIPTION\n+-----------\n+Run tasks to optimize Git repository data, speeding up other Git commands\n+and reducing storage requirements for the repository.\n++\n+Git commands that add repository data, such as `git add` or `git fetch`,\n+are optimized for a responsive user experience. These commands do not take\n+time to optimize the Git data, since such optimizations scale with the full\n+size of the repository while these user commands each perform a relatively\n+small action.\n++\n+The `git maintenance` command provides flexibility for how to optimize the\n+Git repository.\n+\n+SUBCOMMANDS\n+-----------\n+\n+run::\n+\tRun one or more maintenance tasks.\n+\n+TASKS\n+-----\n+\n+gc::\n+\tCleanup unnecessary files and optimize the local repository. \"GC\"\n+\tstands for \"garbage collection,\" but this task performs many\n+\tsmaller tasks. This task can be rather expensive for large\n+\trepositories, as it repacks all Git objects into a single pack-file.\n+\tIt can also be disruptive in some situations, as it deletes stale\n+\tdata.\n+\n+OPTIONS\n+-------\n+--auto::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain thresholds are met. For example, the `gc` task\n+\truns when the number of loose objects exceeds the number stored\n+\tin the `gc.auto` config setting, or when the number of pack-files\n+\texceeds the `gc.autoPackLimit` config setting.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/builtin.h b/builtin.h\nindex a5ae15bfe5..17c1c0ce49 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix);\n int cmd_ls_remote(int argc, const char **argv, const char *prefix);\n int cmd_mailinfo(int argc, const char **argv, const char *prefix);\n int cmd_mailsplit(int argc, const char **argv, const char *prefix);\n+int cmd_maintenance(int argc, const char **argv, const char *prefix);\n int cmd_merge(int argc, const char **argv, const char *prefix);\n int cmd_merge_base(int argc, const char **argv, const char *prefix);\n int cmd_merge_index(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 10346e0465..be9557452e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -699,3 +699,59 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \treturn 0;\n }\n+\n+static const char * const builtin_maintenance_usage[] = {\n+\tN_(\"git maintenance run []\"),\n+\tNULL\n+};\n+\n+static struct maintenance_opts {\n+\tint auto_flag;\n+} opts;\n+\n+static int maintenance_task_gc(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_push(&child.args, \"gc\");\n+\n+\tif (opts.auto_flag)\n+\t\tstrvec_push(&child.args, \"--auto\");\n+\n+\tclose_object_store(the_repository->objects);\n+\treturn run_command(&child);\n+}\n+\n+static int maintenance_run(void)\n+{\n+\treturn maintenance_task_gc();\n+}\n+\n+int cmd_maintenance(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_maintenance_options[] = {\n+\t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n+\t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tmemset(&opts, 0, sizeof(opts));\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t\t builtin_maintenance_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_maintenance_options,\n+\t\t\t builtin_maintenance_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tif (argc == 1) {\n+\t\tif (!strcmp(argv[0], \"run\"))\n+\t\t\treturn maintenance_run();\n+\t}\n+\n+\tusage_with_options(builtin_maintenance_usage,\n+\t\t\t builtin_maintenance_options);\n+}\ndiff --git a/git.c b/git.c\nindex 832688ca23..5bb9645403 100644\n--- a/git.c\n+++ b/git.c\n@@ -529,6 +529,7 @@ static struct cmd_struct commands[] = {\n \t{ \"ls-tree\", cmd_ls_tree, RUN_SETUP },\n \t{ \"mailinfo\", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"mailsplit\", cmd_mailsplit, NO_PARSEOPT },\n+\t{ \"maintenance\", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT },\n \t{ \"merge\", cmd_merge, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"merge-base\", cmd_merge_base, RUN_SETUP },\n \t{ \"merge-file\", cmd_merge_file, RUN_SETUP_GENTLY },\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nnew file mode 100755\nindex 0000000000..0e864eaaed\n--- /dev/null\n+++ b/t/t7900-maintenance.sh\n@@ -0,0 +1,19 @@\n+#!/bin/sh\n+\n+test_description='git maintenance builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'help text' '\n+\ttest_expect_code 129 git maintenance -h 2>err &&\n+\ttest_i18ngrep \"usage: git maintenance run\" err\n+'\n+\n+test_expect_success 'run [--auto]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"402485","messageId":"6e533e43d71580d3cd81f9b0ae2c5884a7d3ac2a.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 02/20] maintenance: add --quiet option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:08Z","receivedAt":"2020-07-30T22:24:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance activities are commonly used as steps in larger scripts.\nProviding a '--quiet' option allows those scripts to be less noisy when\nrun on a terminal window. Turn this mode on by default when stderr is\nnot a terminal.\n\nPipe the option to the 'git gc' child process.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 3 +++\n builtin/gc.c | 7 +++++++\n t/t7900-maintenance.sh | 8 +++++---\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 34cd2b4417..089fa4cedc 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -52,6 +52,9 @@ OPTIONS\n \tin the `gc.auto` config setting, or when the number of pack-files\n \texceeds the `gc.autoPackLimit` config setting.\n \n+--quiet::\n+\tDo not report progress or other information over `stderr`.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex be9557452e..3c277f9f9c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -707,6 +707,7 @@ static const char * const builtin_maintenance_usage[] = {\n \n static struct maintenance_opts {\n \tint auto_flag;\n+\tint quiet;\n } opts;\n \n static int maintenance_task_gc(void)\n@@ -718,6 +719,8 @@ static int maintenance_task_gc(void)\n \n \tif (opts.auto_flag)\n \t\tstrvec_push(&child.args, \"--auto\");\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--quiet\");\n \n \tclose_object_store(the_repository->objects);\n \treturn run_command(&child);\n@@ -733,6 +736,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstatic struct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n+\t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_END()\n \t};\n \n@@ -742,6 +747,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_maintenance_usage,\n \t\t\t\t builtin_maintenance_options);\n \n+\topts.quiet = !isatty(2);\n+\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\n \t\t\t builtin_maintenance_usage,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 0e864eaaed..f08eee0977 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,11 +9,13 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git maintenance run\" err\n '\n \n-test_expect_success 'run [--auto]' '\n-\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run &&\n+test_expect_success 'run [--auto|--quiet]' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-no-auto.txt\" git maintenance run --no-quiet &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"]\" run-auto.txt\n+\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"402486","messageId":"b9332c131863fd8d31924a348b451e03fed880ad.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 04/20] maintenance: initialize task array","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:10Z","receivedAt":"2020-07-30T22:24:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of implementing multiple maintenance tasks inside the\n'maintenance' builtin, use a list of structs to describe the work to be\ndone.\n\nThe struct maintenance_task stores the name of the task (as given by a\nfuture command-line argument) along with a function pointer to its\nimplementation and a boolean for whether the step is enabled.\n\nA list these structs are initialized with the full list of implemented\ntasks along with a default order. For now, this list only contains the\n\"gc\" task. This task is also the only task enabled by default.\n\nThe run subcommand will return a nonzero exit code if any task fails.\nHowever, it will attempt all tasks in its loop before returning with the\nfailure. Also each failed task will send an error message.\n\nHelped-by: Taylor Blau \nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 38 +++++++++++++++++++++++++++++++++++++-\n 1 file changed, 37 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3c277f9f9c..0f15162825 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -726,9 +726,45 @@ static int maintenance_task_gc(void)\n \treturn run_command(&child);\n }\n \n+typedef int maintenance_task_fn(void);\n+\n+struct maintenance_task {\n+\tconst char *name;\n+\tmaintenance_task_fn *fn;\n+\tunsigned enabled:1;\n+};\n+\n+enum maintenance_task_label {\n+\tTASK_GC,\n+\n+\t/* Leave as final value */\n+\tTASK__COUNT\n+};\n+\n+static struct maintenance_task tasks[] = {\n+\t[TASK_GC] = {\n+\t\t\"gc\",\n+\t\tmaintenance_task_gc,\n+\t\t1,\n+\t},\n+};\n+\n static int maintenance_run(void)\n {\n-\treturn maintenance_task_gc();\n+\tint i;\n+\tint result = 0;\n+\n+\tfor (i = 0; i < TASK__COUNT; i++) {\n+\t\tif (!tasks[i].enabled)\n+\t\t\tcontinue;\n+\n+\t\tif (tasks[i].fn()) {\n+\t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n+\t\t\tresult = 1;\n+\t\t}\n+\t}\n+\n+\treturn result;\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n-- \ngitgitgadget\n\n"},{"id":"402487","messageId":"c4674fc211da51f729d76ca40756479216164b10.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 03/20] maintenance: replace run_auto_gc()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:09Z","receivedAt":"2020-07-30T22:24:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe run_auto_gc() method is used in several places to trigger a check\nfor repo maintenance after some Git commands, such as 'git commit' or\n'git fetch'.\n\nTo allow for extra customization of this maintenance activity, replace\nthe 'git gc --auto [--quiet]' call with one to 'git maintenance run\n--auto [--quiet]'. As we extend the maintenance builtin with other\nsteps, users will be able to select different maintenance activities.\n\nRename run_auto_gc() to run_auto_maintenance() to be clearer what is\nhappening on this call, and to expose all callers in the current diff.\nRewrite the method to use a struct child_process to simplify the calls\nslightly.\n\nSince 'git fetch' already allows disabling the 'git gc --auto'\nsubprocess, add an equivalent option with a different name to be more\ndescriptive of the new behavior: '--[no-]maintenance'. Update the\ndocumentation to include these options at the same time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/fetch-options.txt | 6 ++++--\n Documentation/git-clone.txt | 6 +++---\n builtin/am.c | 2 +-\n builtin/commit.c | 2 +-\n builtin/fetch.c | 6 ++++--\n builtin/merge.c | 2 +-\n builtin/rebase.c | 4 ++--\n run-command.c | 16 +++++++---------\n run-command.h | 2 +-\n t/t5510-fetch.sh | 2 +-\n 10 files changed, 25 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 6e2a160a47..495bc8ab5a 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -86,9 +86,11 @@ ifndef::git-pull[]\n \tAllow several and arguments to be\n \tspecified. No s may be specified.\n \n+--[no-]auto-maintenance::\n --[no-]auto-gc::\n-\tRun `git gc --auto` at the end to perform garbage collection\n-\tif needed. This is enabled by default.\n+\tRun `git maintenance run --auto` at the end to perform automatic\n+\trepository maintenance if needed. (`--[no-]auto-gc` is a synonym.)\n+\tThis is enabled by default.\n \n --[no-]write-commit-graph::\n \tWrite a commit-graph after fetching. This overrides the config\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex c898310099..097e6a86c5 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -78,9 +78,9 @@ repository using this option and then delete branches (or use any\n other Git command that makes any existing commit unreferenced) in the\n source repository, some objects may become unreferenced (or dangling).\n These objects may be removed by normal Git operations (such as `git commit`)\n-which automatically call `git gc --auto`. (See linkgit:git-gc[1].)\n-If these objects are removed and were referenced by the cloned repository,\n-then the cloned repository will become corrupt.\n+which automatically call `git maintenance run --auto`. (See\n+linkgit:git-maintenance[1].) If these objects are removed and were referenced\n+by the cloned repository, then the cloned repository will become corrupt.\n +\n Note that running `git repack` without the `--local` option in a repository\n cloned with `--shared` will copy objects from the source repository into a pack\ndiff --git a/builtin/am.c b/builtin/am.c\nindex 3f2adb3822..2ca363f72e 100644\n--- a/builtin/am.c\n+++ b/builtin/am.c\n@@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume)\n \tif (!state->rebasing) {\n \t\tam_destroy(state);\n \t\tclose_object_store(the_repository->objects);\n-\t\trun_auto_gc(state->quiet);\n+\t\trun_auto_maintenance(state->quiet);\n \t}\n }\n \ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 01105ce8b0..9705bfb0cf 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \tgit_test_write_commit_graph_or_die();\n \n \trepo_rerere(the_repository, 0);\n-\trun_auto_gc(quiet);\n+\trun_auto_maintenance(quiet);\n \trun_commit_hook(use_editor, get_index_file(), \"post-commit\", NULL);\n \tif (amend && !no_post_rewrite) {\n \t\tcommit_post_rewrite(the_repository, current_head, &oid);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7953a1a25b..c7c8ac0861 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = {\n \tOPT_STRING_LIST(0, \"negotiation-tip\", &negotiation_tip, N_(\"revision\"),\n \t\t\tN_(\"report that we have only objects reachable from this object\")),\n \tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n+\tOPT_BOOL(0, \"auto-maintenance\", &enable_auto_gc,\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"auto-gc\", &enable_auto_gc,\n-\t\t N_(\"run 'gc --auto' after fetching\")),\n+\t\t N_(\"run 'maintenance --auto' after fetching\")),\n \tOPT_BOOL(0, \"show-forced-updates\", &fetch_show_forced_updates,\n \t\t N_(\"check for forced-updates on all updated branches\")),\n \tOPT_BOOL(0, \"write-commit-graph\", &fetch_write_commit_graph,\n@@ -1882,7 +1884,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tclose_object_store(the_repository->objects);\n \n \tif (enable_auto_gc)\n-\t\trun_auto_gc(verbosity < 0);\n+\t\trun_auto_maintenance(verbosity < 0);\n \n \treturn result;\n }\ndiff --git a/builtin/merge.c b/builtin/merge.c\nindex 7da707bf55..c068e73037 100644\n--- a/builtin/merge.c\n+++ b/builtin/merge.c\n@@ -457,7 +457,7 @@ static void finish(struct commit *head_commit,\n \t\t\t * user should see them.\n \t\t\t */\n \t\t\tclose_object_store(the_repository->objects);\n-\t\t\trun_auto_gc(verbosity < 0);\n+\t\t\trun_auto_maintenance(verbosity < 0);\n \t\t}\n \t}\n \tif (new_head && show_diffstat) {\ndiff --git a/builtin/rebase.c b/builtin/rebase.c\nindex 494107a648..d14d18191b 100644\n--- a/builtin/rebase.c\n+++ b/builtin/rebase.c\n@@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts)\n \tapply_autostash(state_dir_path(\"autostash\", opts));\n \tclose_object_store(the_repository->objects);\n \t/*\n-\t * We ignore errors in 'gc --auto', since the\n+\t * We ignore errors in 'git maintenance run --auto', since the\n \t * user should see them.\n \t */\n-\trun_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n+\trun_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE)));\n \tif (opts->type == REBASE_MERGE) {\n \t\tstruct replay_opts replay = REPLAY_OPTS_INIT;\n \ndiff --git a/run-command.c b/run-command.c\nindex 30104a4ee1..b7e1f1dd5a 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1866,15 +1866,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \treturn result;\n }\n \n-int run_auto_gc(int quiet)\n+int run_auto_maintenance(int quiet)\n {\n-\tstruct strvec argv_gc_auto = STRVEC_INIT;\n-\tint status;\n+\tstruct child_process maint = CHILD_PROCESS_INIT;\n \n-\tstrvec_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n-\tif (quiet)\n-\t\tstrvec_push(&argv_gc_auto, \"--quiet\");\n-\tstatus = run_command_v_opt(argv_gc_auto.items, RUN_GIT_CMD);\n-\tstrvec_clear(&argv_gc_auto);\n-\treturn status;\n+\tmaint.git_cmd = 1;\n+\tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n+\tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\n+\n+\treturn run_command(&maint);\n }\ndiff --git a/run-command.h b/run-command.h\nindex 8b9bfaef16..6472b38bde 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args);\n /*\n * Trigger an auto-gc\n */\n-int run_auto_gc(int quiet);\n+int run_auto_maintenance(int quiet);\n \n #define RUN_COMMAND_NO_STDIN 1\n #define RUN_GIT_CMD\t 2\t/*If this is to be git sub-command */\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex a66dbe0bde..9850ecde5d 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' '\n \t\tgit config fetch.unpackLimit 1 &&\n \t\tgit config gc.autoPackLimit 1 &&\n \t\tgit config gc.autoDetach false &&\n-\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch >fetch.out 2>&1 &&\n+\t\tGIT_ASK_YESNO=\"$D/askyesno\" git fetch --verbose >fetch.out 2>&1 &&\n \t\ttest_i18ngrep \"Auto packing the repository\" fetch.out &&\n \t\t! grep \"Should I try again\" fetch.out\n \t)\n-- \ngitgitgadget\n\n"},{"id":"402489","messageId":"a4d9836bed9a08f4ba716240a6fc18d5f13da3dc.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 05/20] maintenance: add commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:11Z","receivedAt":"2020-07-30T22:24:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe first new task in the 'git maintenance' builtin is the\n'commit-graph' job. It is based on the sequence of events in the\n'commit-graph' job in Scalar [1]. This sequence is as follows:\n\n1. git commit-graph write --reachable --split\n2. git commit-graph verify --shallow\n3. If the verify succeeds, stop.\n4. Delete the commit-graph-chain file.\n5. git commit-graph write --reachable --split\n\nBy writing an incremental commit-graph file using the \"--split\"\noption we minimize the disruption from this operation. The default\nbehavior is to merge layers until the new \"top\" layer is less than\nhalf the size of the layer below. This provides quick writes most\nof the time, with the longer writes following a power law\ndistribution.\n\nMost importantly, concurrent Git processes only look at the\ncommit-graph-chain file for a very short amount of time, so they\nwill verly likely not be holding a handle to the file when we try\nto replace it. (This only matters on Windows.)\n\nIf a concurrent process reads the old commit-graph-chain file, but\nour job expires some of the .graph files before they can be read,\nthen those processes will see a warning message (but not fail).\nThis could be avoided by a future update to use the --expire-time\nargument when writing the commit-graph.\n\nBy using 'git commit-graph verify --shallow' we can ensure that\nthe file we just wrote is valid. This is an extra safety precaution\nthat is faster than our 'write' subcommand. In the rare situation\nthat the newest layer of the commit-graph is corrupt, we can \"fix\"\nthe corruption by deleting the commit-graph-chain file and rewrite\nthe full commit-graph as a new one-layer commit graph. This does\nnot completely prevent _that_ file from being corrupt, but it does\nrecompute the commit-graph by parsing commits from the object\ndatabase. In our use of this step in Scalar and VFS for Git, we\nhave only seen this issue arise because our microsoft/git fork\nreverted 43d3561 (\"commit-graph write: don't die if the existing\ngraph is corrupt\" 2019-03-25) for a while to keep commit-graph\nwrites very fast. We dropped the revert when updating to v2.23.0.\nThe verify still has potential for catching corrupt data across\nthe layer boundary: if the new file has commit X with parent Y\nin an old file but the commit ID for Y in the old file had a\nbitswap, then we will notice that in the 'verify' command.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 18 +++++++++\n builtin/gc.c | 63 +++++++++++++++++++++++++++++++\n commit-graph.c | 8 ++--\n commit-graph.h | 1 +\n t/t7900-maintenance.sh | 2 +\n 5 files changed, 88 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 089fa4cedc..35b0be7d40 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -35,6 +35,24 @@ run::\n TASKS\n -----\n \n+commit-graph::\n+\tThe `commit-graph` job updates the `commit-graph` files incrementally,\n+\tthen verifies that the written data is correct. If the new layer has an\n+\tissue, then the chain file is removed and the `commit-graph` is\n+\trewritten from scratch.\n++\n+The verification only checks the top layer of the `commit-graph` chain.\n+If the incremental write merged the new commits with at least one\n+existing layer, then there is potential for on-disk corruption being\n+carried forward into the new file. This will be noticed and the new\n+commit-graph file will be clean as Git reparses the commit data from\n+the object database.\n++\n+The incremental write is safe to run alongside concurrent Git processes\n+since it will not expire `.graph` files that were in the previous\n+`commit-graph-chain` file. They will be deleted by a later run based on\n+the expiration delay.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 0f15162825..ec1bbc3f9e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -710,6 +710,64 @@ static struct maintenance_opts {\n \tint quiet;\n } opts;\n \n+static int run_write_commit_graph(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"commit-graph\", \"write\",\n+\t\t \"--split\", \"--reachable\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\treturn !!run_command(&child);\n+}\n+\n+static int run_verify_commit_graph(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"commit-graph\", \"verify\",\n+\t\t \"--shallow\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\treturn !!run_command(&child);\n+}\n+\n+static int maintenance_task_commit_graph(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tchar *chain_path;\n+\n+\tclose_object_store(r->objects);\n+\tif (run_write_commit_graph()) {\n+\t\terror(_(\"failed to write commit-graph\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (!run_verify_commit_graph())\n+\t\treturn 0;\n+\n+\twarning(_(\"commit-graph verify caught error, rewriting\"));\n+\n+\tchain_path = get_commit_graph_chain_filename(r->objects->odb);\n+\tif (unlink(chain_path)) {\n+\t\tUNLEAK(chain_path);\n+\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n+\t}\n+\tfree(chain_path);\n+\n+\tif (!run_write_commit_graph())\n+\t\treturn 0;\n+\n+\terror(_(\"failed to rewrite commit-graph\"));\n+\treturn 1;\n+}\n+\n static int maintenance_task_gc(void)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -736,6 +794,7 @@ struct maintenance_task {\n \n enum maintenance_task_label {\n \tTASK_GC,\n+\tTASK_COMMIT_GRAPH,\n \n \t/* Leave as final value */\n \tTASK__COUNT\n@@ -747,6 +806,10 @@ static struct maintenance_task tasks[] = {\n \t\tmaintenance_task_gc,\n \t\t1,\n \t},\n+\t[TASK_COMMIT_GRAPH] = {\n+\t\t\"commit-graph\",\n+\t\tmaintenance_task_commit_graph,\n+\t},\n };\n \n static int maintenance_run(void)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1af68c297d..9705d237e4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb,\n \t\t oid_hex);\n }\n \n-static char *get_chain_filename(struct object_directory *odb)\n+char *get_commit_graph_chain_filename(struct object_directory *odb)\n {\n \treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", odb->path);\n }\n@@ -521,7 +521,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \tstruct stat st;\n \tstruct object_id *oids;\n \tint i = 0, valid = 1, count;\n-\tchar *chain_name = get_chain_filename(odb);\n+\tchar *chain_name = get_commit_graph_chain_filename(odb);\n \tFILE *fp;\n \tint stat_res;\n \n@@ -1619,7 +1619,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t}\n \n \tif (ctx->split) {\n-\t\tchar *lock_name = get_chain_filename(ctx->odb);\n+\t\tchar *lock_name = get_commit_graph_chain_filename(ctx->odb);\n \n \t\thold_lock_file_for_update_mode(&lk, lock_name,\n \t\t\t\t\t LOCK_DIE_ON_ERROR, 0444);\n@@ -1996,7 +1996,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time = ctx->split_opts->expire_time;\n \tif (!ctx->split) {\n-\t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n+\t\tchar *chain_file_name = get_commit_graph_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n \t\tfree(chain_file_name);\n \t\tctx->num_commit_graphs_after = 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 28f89cdf3e..3c202748c3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -25,6 +25,7 @@ struct commit;\n struct bloom_filter_settings;\n \n char *get_commit_graph_filename(struct object_directory *odb);\n+char *get_commit_graph_chain_filename(struct object_directory *odb);\n int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n \n /*\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex f08eee0977..ff646abf7c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -4,6 +4,8 @@ test_description='git maintenance builtin'\n \n . ./test-lib.sh\n \n+GIT_TEST_COMMIT_GRAPH=0\n+\n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n \ttest_i18ngrep \"usage: git maintenance run\" err\n-- \ngitgitgadget\n\n"},{"id":"402488","messageId":"dafb0d9bbc4c8ddce82fa30dd0ad78439342ed66.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 06/20] maintenance: add --task option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:12Z","receivedAt":"2020-07-30T22:24:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to only run certain maintenance tasks in a certain\norder. Add the --task= option, which allows a user to specify an\nordered list of tasks to run. These cannot be run multiple times,\nhowever.\n\nHere is where our array of maintenance_task pointers becomes critical.\nWe can sort the array of pointers based on the task order, but we do not\nwant to move the struct data itself in order to preserve the hashmap\nreferences. We use the hashmap to match the --task= arguments into\nthe task struct data.\n\nKeep in mind that the 'enabled' member of the maintenance_task struct is\na placeholder for a future 'maintenance..enabled' config option.\nThus, we use the 'enabled' member to specify which tasks are run when\nthe user does not specify any --task= arguments. The 'enabled'\nmember should be ignored if --task= appears.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 4 +++\n builtin/gc.c | 59 +++++++++++++++++++++++++++++--\n t/t7900-maintenance.sh | 23 ++++++++++++\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 35b0be7d40..9204762e21 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -73,6 +73,10 @@ OPTIONS\n --quiet::\n \tDo not report progress or other information over `stderr`.\n \n+--task=::\n+\tIf this option is specified one or more times, then only run the\n+\tspecified tasks in the specified order.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ec1bbc3f9e..b7f64891cd 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -708,6 +708,7 @@ static const char * const builtin_maintenance_usage[] = {\n static struct maintenance_opts {\n \tint auto_flag;\n \tint quiet;\n+\tint tasks_selected;\n } opts;\n \n static int run_write_commit_graph(void)\n@@ -789,7 +790,9 @@ typedef int maintenance_task_fn(void);\n struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n-\tunsigned enabled:1;\n+\tunsigned enabled:1,\n+\t\t selected:1;\n+\tint selected_order;\n };\n \n enum maintenance_task_label {\n@@ -812,13 +815,29 @@ static struct maintenance_task tasks[] = {\n \t},\n };\n \n+static int compare_tasks_by_selection(const void *a_, const void *b_)\n+{\n+\tconst struct maintenance_task *a, *b;\n+\n+\ta = (const struct maintenance_task *)&a_;\n+\tb = (const struct maintenance_task *)&b_;\n+\n+\treturn b->selected_order - a->selected_order;\n+}\n+\n static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n \n+\tif (opts.tasks_selected)\n+\t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n+\n \tfor (i = 0; i < TASK__COUNT; i++) {\n-\t\tif (!tasks[i].enabled)\n+\t\tif (opts.tasks_selected && !tasks[i].selected)\n+\t\t\tcontinue;\n+\n+\t\tif (!opts.tasks_selected && !tasks[i].enabled)\n \t\t\tcontinue;\n \n \t\tif (tasks[i].fn()) {\n@@ -830,6 +849,39 @@ static int maintenance_run(void)\n \treturn result;\n }\n \n+static int task_option_parse(const struct option *opt,\n+\t\t\t const char *arg, int unset)\n+{\n+\tint i;\n+\tstruct maintenance_task *task = NULL;\n+\n+\tBUG_ON_OPT_NEG(unset);\n+\n+\topts.tasks_selected++;\n+\n+\tfor (i = 0; i < TASK__COUNT; i++) {\n+\t\tif (!strcasecmp(tasks[i].name, arg)) {\n+\t\t\ttask = &tasks[i];\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tif (!task) {\n+\t\terror(_(\"'%s' is not a valid task\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\tif (task->selected) {\n+\t\terror(_(\"task '%s' cannot be selected multiple times\"), arg);\n+\t\treturn 1;\n+\t}\n+\n+\ttask->selected = 1;\n+\ttask->selected_order = opts.tasks_selected;\n+\n+\treturn 0;\n+}\n+\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_maintenance_options[] = {\n@@ -837,6 +889,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n+\t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n+\t\t\tN_(\"run a specific task\"),\n+\t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex ff646abf7c..3cdccb24df 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -20,4 +20,27 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'run --task=' '\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" git maintenance run --task=commit-graph --task=gc &&\n+\t! grep \",\\\"gc\\\"\" run-commit-graph.txt &&\n+\tgrep \",\\\"gc\\\"\" run-gc.txt &&\n+\tgrep \",\\\"gc\\\"\" run-both.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-commit-graph.txt &&\n+\t! grep \",\\\"commit-graph\\\",\\\"write\\\"\" run-gc.txt &&\n+\tgrep \",\\\"commit-graph\\\",\\\"write\\\"\" run-both.txt\n+'\n+\n+test_expect_success 'run --task=bogus' '\n+\ttest_must_fail git maintenance run --task=bogus 2>err &&\n+\ttest_i18ngrep \"is not a valid task\" err\n+'\n+\n+test_expect_success 'run --task duplicate' '\n+\ttest_must_fail git maintenance run --task=gc --task=gc 2>err &&\n+\ttest_i18ngrep \"cannot be selected multiple times\" err\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"402490","messageId":"0e94e04dcd4e6434a70cf7b676c12cd27f859fd9.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 08/20] fetch: optionally allow disabling FETCH_HEAD update","fromName":"Junio C Hamano via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:14Z","receivedAt":"2020-07-30T22:24:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"From: Junio C Hamano \n\nIf you run fetch but record the result in remote-tracking branches,\nand either if you do nothing with the fetched refs (e.g. you are\nmerely mirroring) or if you always work from the remote-tracking\nrefs (e.g. you fetch and then merge origin/branchname separately),\nyou can get away with having no FETCH_HEAD at all.\n\nTeach \"git fetch\" a command line option \"--[no-]write-fetch-head\"\nand \"fetch.writeFetchHEAD\" configuration variable. Without either,\nthe default is to write FETCH_HEAD, and the usual rule that the\ncommand line option defeats configured default applies.\n\nNote that under \"--dry-run\" mode, FETCH_HEAD is never written;\notherwise you'd see list of objects in the file that you do not\nactually have. Passing `--write-fetch-head` does not force `git\nfetch` to write the file.\n\nAlso note that this option is explicitly passed when \"git pull\"\ninternally invokes \"git fetch\", so that those who configured their\n\"git fetch\" not to write FETCH_HEAD would not be able to break the\ncooperation between these two commands. \"git pull\" must see what\n\"git fetch\" got recorded in FETCH_HEAD to work correctly.\n\nSigned-off-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/fetch.txt | 7 ++++++\n Documentation/fetch-options.txt | 10 +++++++++\n builtin/fetch.c | 19 +++++++++++++---\n builtin/pull.c | 3 ++-\n t/t5510-fetch.sh | 39 +++++++++++++++++++++++++++++++--\n t/t5521-pull-options.sh | 16 ++++++++++++++\n 6 files changed, 88 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt\nindex b20394038d..0aaa05e8c0 100644\n--- a/Documentation/config/fetch.txt\n+++ b/Documentation/config/fetch.txt\n@@ -91,3 +91,10 @@ fetch.writeCommitGraph::\n \tmerge and the write may take longer. Having an updated commit-graph\n \tfile helps performance of many Git commands, including `git merge-base`,\n \t`git push -f`, and `git log --graph`. Defaults to false.\n+\n+fetch.writeFetchHEAD::\n+\tSetting it to false tells `git fetch` not to write the list\n+\tof remote refs fetched in the `FETCH_HEAD` file directly\n+\tunder `$GIT_DIR`. Can be countermanded from the command\n+\tline with the `--[no-]write-fetch-head` option. Defaults to\n+\ttrue.\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex 495bc8ab5a..6972ad2522 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -64,6 +64,16 @@ documented in linkgit:git-config[1].\n --dry-run::\n \tShow what would be done, without making any changes.\n \n+ifndef::git-pull[]\n+--[no-]write-fetch-head::\n+\tWrite the list of remote refs fetched in the `FETCH_HEAD`\n+\tfile directly under `$GIT_DIR`. This is the default unless\n+\tthe configuration variable `fetch.writeFetchHEAD` is set to\n+\tfalse. Passing `--no-write-fetch-head` from the command\n+\tline tells Git not to write the file. Under `--dry-run`\n+\toption, the file is never written.\n+endif::git-pull[]\n+\n -f::\n --force::\n \tWhen 'git fetch' is used with `:` refspec it may\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex c7c8ac0861..30ac57dcf6 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -56,6 +56,7 @@ static int prune_tags = -1; /* unspecified */\n #define PRUNE_TAGS_BY_DEFAULT 0 /* do we prune tags by default? */\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok;\n+static int write_fetch_head = 1;\n static int verbosity, deepen_relative, set_upstream;\n static int progress = -1;\n static int enable_auto_gc = 1;\n@@ -118,6 +119,10 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(k, \"fetch.writefetchhead\")) {\n+\t\twrite_fetch_head = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \treturn git_default_config(k, v, cb);\n }\n \n@@ -162,6 +167,8 @@ static struct option builtin_fetch_options[] = {\n \t\t PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),\n \tOPT_BOOL(0, \"dry-run\", &dry_run,\n \t\t N_(\"dry run\")),\n+\tOPT_BOOL(0, \"write-fetch-head\", &write_fetch_head,\n+\t\t N_(\"write fetched references to the FETCH_HEAD file\")),\n \tOPT_BOOL('k', \"keep\", &keep, N_(\"keep downloaded pack\")),\n \tOPT_BOOL('u', \"update-head-ok\", &update_head_ok,\n \t\t N_(\"allow updating of HEAD ref\")),\n@@ -895,7 +902,9 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *what, *kind;\n \tstruct ref *rm;\n \tchar *url;\n-\tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head(the_repository);\n+\tconst char *filename = (!write_fetch_head\n+\t\t\t\t? \"/dev/null\"\n+\t\t\t\t: git_path_fetch_head(the_repository));\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n \n@@ -1329,7 +1338,7 @@ static int do_fetch(struct transport *transport,\n \t}\n \n \t/* if not appending, truncate FETCH_HEAD */\n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tretcode = truncate_fetch_head();\n \t\tif (retcode)\n \t\t\tgoto cleanup;\n@@ -1596,7 +1605,7 @@ static int fetch_multiple(struct string_list *list, int max_children)\n \tint i, result = 0;\n \tstruct strvec argv = STRVEC_INIT;\n \n-\tif (!append && !dry_run) {\n+\tif (!append && write_fetch_head) {\n \t\tint errcode = truncate_fetch_head();\n \t\tif (errcode)\n \t\t\treturn errcode;\n@@ -1797,6 +1806,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n+\t/* FETCH_HEAD never gets updated in --dry-run mode */\n+\tif (dry_run)\n+\t\twrite_fetch_head = 0;\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\ndiff --git a/builtin/pull.c b/builtin/pull.c\nindex 858b492af3..4c66db1468 100644\n--- a/builtin/pull.c\n+++ b/builtin/pull.c\n@@ -527,7 +527,8 @@ static int run_fetch(const char *repo, const char **refspecs)\n \tstruct strvec args = STRVEC_INIT;\n \tint ret;\n \n-\tstrvec_pushl(&args, \"fetch\", \"--update-head-ok\", NULL);\n+\tstrvec_pushl(&args, \"fetch\", \"--update-head-ok\",\n+\t\t \"--write-fetch-head\", NULL);\n \n \t/* Shared options */\n \targv_push_verbosity(&args);\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 9850ecde5d..31c91d0ed2 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -539,13 +539,48 @@ test_expect_success 'fetch into the current branch with --update-head-ok' '\n \n '\n \n-test_expect_success 'fetch --dry-run' '\n-\n+test_expect_success 'fetch --dry-run does not touch FETCH_HEAD' '\n \trm -f .git/FETCH_HEAD &&\n \tgit fetch --dry-run . &&\n \t! test -f .git/FETCH_HEAD\n '\n \n+test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success '--write-fetch-head gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit fetch --dry-run --write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and FETCH_HEAD' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD gets defeated by --dry-run' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --dry-run . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --no-write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=yes fetch --no-write-fetch-head . &&\n+\t! test -f .git/FETCH_HEAD\n+'\n+\n+test_expect_success 'fetch.writeFetchHEAD and --write-fetch-head' '\n+\trm -f .git/FETCH_HEAD &&\n+\tgit -c fetch.writeFetchHEAD=no fetch --write-fetch-head . &&\n+\ttest -f .git/FETCH_HEAD\n+'\n+\n test_expect_success \"should be able to fetch with duplicate refspecs\" '\n \tmkdir dups &&\n \t(\ndiff --git a/t/t5521-pull-options.sh b/t/t5521-pull-options.sh\nindex 159afa7ac8..1acae3b9a4 100755\n--- a/t/t5521-pull-options.sh\n+++ b/t/t5521-pull-options.sh\n@@ -77,6 +77,7 @@ test_expect_success 'git pull -q -v --no-rebase' '\n \ttest_must_be_empty out &&\n \ttest -s err)\n '\n+\n test_expect_success 'git pull --cleanup errors early on invalid argument' '\n \tmkdir clonedcleanup &&\n \t(cd clonedcleanup && git init &&\n@@ -85,6 +86,21 @@ test_expect_success 'git pull --cleanup errors early on invalid argument' '\n \ttest -s err)\n '\n \n+test_expect_success 'git pull --no-write-fetch-head fails' '\n+\tmkdir clonedwfh &&\n+\t(cd clonedwfh && git init &&\n+\ttest_must_fail git pull --no-write-fetch-head \"../parent\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_i18ngrep \"no-write-fetch-head\" err)\n+'\n+\n+test_expect_success 'git pull succeeds with fetch.writeFetchHEAD=false' '\n+\tmkdir clonedwfhconfig &&\n+\t(cd clonedwfhconfig && git init &&\n+\tgit config fetch.writeFetchHEAD false &&\n+\tgit pull \"../parent\" >out 2>err &&\n+\tgrep FETCH_HEAD err)\n+'\n \n test_expect_success 'git pull --force' '\n \tmkdir clonedoldstyle &&\n-- \ngitgitgadget\n\n"},{"id":"402497","messageId":"1b00524da3c199e833acc8ce400e8ebd332908b0.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 07/20] maintenance: take a lock on the objects directory","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:13Z","receivedAt":"2020-07-30T22:24:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPerforming maintenance on a Git repository involves writing data to the\n.git directory, which is not safe to do with multiple writers attempting\nthe same operation. Ensure that only one 'git maintenance' process is\nrunning at a time by holding a file-based lock. Simply the presence of\nthe .git/maintenance.lock file will prevent future maintenance. This\nlock is never committed, since it does not represent meaningful data.\nInstead, it is only a placeholder.\n\nIf the lock file already exists, then fail silently. This will become\nvery important later when we implement the 'fetch' task, as this is our\nstop-gap from creating a recursive process loop between 'git fetch' and\n'git maintenance run'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b7f64891cd..b57bc7b0ff 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -829,6 +829,25 @@ static int maintenance_run(void)\n {\n \tint i;\n \tint result = 0;\n+\tstruct lock_file lk;\n+\tstruct repository *r = the_repository;\n+\tchar *lock_path = xstrfmt(\"%s/maintenance\", r->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) {\n+\t\t/*\n+\t\t * Another maintenance command is running.\n+\t\t *\n+\t\t * If --auto was provided, then it is likely due to a\n+\t\t * recursive process stack. Do not report an error in\n+\t\t * that case.\n+\t\t */\n+\t\tif (!opts.auto_flag && !opts.quiet)\n+\t\t\terror(_(\"lock file '%s' exists, skipping maintenance\"),\n+\t\t\t lock_path);\n+\t\tfree(lock_path);\n+\t\treturn 0;\n+\t}\n+\tfree(lock_path);\n \n \tif (opts.tasks_selected)\n \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n@@ -846,6 +865,7 @@ static int maintenance_run(void)\n \t\t}\n \t}\n \n+\trollback_lock_file(&lk);\n \treturn result;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"402491","messageId":"0128fdfd1ab286ea27437bfe93a9dbb532444277.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 10/20] maintenance: add loose-objects task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:16Z","receivedAt":"2020-07-30T22:24:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOne goal of background maintenance jobs is to allow a user to\ndisable auto-gc (gc.auto=0) but keep their repository in a clean\nstate. Without any cleanup, loose objects will clutter the object\ndatabase and slow operations. In addition, the loose objects will\ntake up extra space because they are not stored with deltas against\nsimilar objects.\n\nCreate a 'loose-objects' task for the 'git maintenance run' command.\nThis helps clean up loose objects without disrupting concurrent Git\ncommands using the following sequence of events:\n\n1. Run 'git prune-packed' to delete any loose objects that exist\n in a pack-file. Concurrent commands will prefer the packed\n version of the object to the loose version. (Of course, there\n are exceptions for commands that specifically care about the\n location of an object. These are rare for a user to run on\n purpose, and we hope a user that has selected background\n maintenance will not be trying to do foreground maintenance.)\n\n2. Run 'git pack-objects' on a batch of loose objects. These\n objects are grouped by scanning the loose object directories in\n lexicographic order until listing all loose objects -or-\n reaching 50,000 objects. This is more than enough if the loose\n objects are created only by a user doing normal development.\n We noticed users with _millions_ of loose objects because VFS\n for Git downloads blobs on-demand when a file read operation\n requires populating a virtual file. This has potential of\n happening in partial clones if someone runs 'git grep' or\n otherwise evades the batch-download feature for requesting\n promisor objects.\n\nThis step is based on a similar step in Scalar [1] and VFS for Git.\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/LooseObjectsStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 ++++\n builtin/gc.c | 97 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 39 +++++++++++++\n 3 files changed, 147 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex d134192fa8..077929b691 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -76,6 +76,17 @@ gc::\n \tIt can also be disruptive in some situations, as it deletes stale\n \tdata.\n \n+loose-objects::\n+\tThe `loose-objects` job cleans up loose objects and places them into\n+\tpack-files. In order to prevent race conditions with concurrent Git\n+\tcommands, it follows a two-step process. First, it deletes any loose\n+\tobjects that already exist in a pack-file; concurrent Git processes\n+\twill examine the pack-file for the object data instead of the loose\n+\tobject. Second, it creates a new pack-file (starting with \"loose-\")\n+\tcontaining a batch of loose objects. The batch size is limited to 50\n+\tthousand objects to prevent the job from taking too long on a\n+\trepository with many loose objects.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1f20428286..96ded73b2f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -832,6 +832,98 @@ static int maintenance_task_gc(void)\n \treturn run_command(&child);\n }\n \n+static int prune_packed(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_push(&child.args, \"prune-packed\");\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--quiet\");\n+\n+\treturn !!run_command(&child);\n+}\n+\n+struct write_loose_object_data {\n+\tFILE *in;\n+\tint count;\n+\tint batch_size;\n+};\n+\n+static int bail_on_loose(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\treturn 1;\n+}\n+\n+static int write_loose_object_to_stdin(const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_loose_object_data *d = (struct write_loose_object_data *)data;\n+\n+\tfprintf(d->in, \"%s\\n\", oid_to_hex(oid));\n+\n+\treturn ++(d->count) > d->batch_size;\n+}\n+\n+static int pack_loose(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tint result = 0;\n+\tstruct write_loose_object_data data;\n+\tstruct child_process pack_proc = CHILD_PROCESS_INIT;\n+\n+\t/*\n+\t * Do not start pack-objects process\n+\t * if there are no loose objects.\n+\t */\n+\tif (!for_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t\t bail_on_loose,\n+\t\t\t\t\t NULL, NULL, NULL))\n+\t\treturn 0;\n+\n+\tpack_proc.git_cmd = 1;\n+\n+\tstrvec_push(&pack_proc.args, \"pack-objects\");\n+\tif (opts.quiet)\n+\t\tstrvec_push(&pack_proc.args, \"--quiet\");\n+\tstrvec_pushf(&pack_proc.args, \"%s/pack/loose\", r->objects->odb->path);\n+\n+\tpack_proc.in = -1;\n+\n+\tif (start_command(&pack_proc)) {\n+\t\terror(_(\"failed to start 'git pack-objects' process\"));\n+\t\treturn 1;\n+\t}\n+\n+\tdata.in = xfdopen(pack_proc.in, \"w\");\n+\tdata.count = 0;\n+\tdata.batch_size = 50000;\n+\n+\tfor_each_loose_file_in_objdir(r->objects->odb->path,\n+\t\t\t\t write_loose_object_to_stdin,\n+\t\t\t\t NULL,\n+\t\t\t\t NULL,\n+\t\t\t\t &data);\n+\n+\tfclose(data.in);\n+\n+\tif (finish_command(&pack_proc)) {\n+\t\terror(_(\"failed to finish 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int maintenance_task_loose_objects(void)\n+{\n+\treturn prune_packed() || pack_loose();\n+}\n+\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n@@ -844,6 +936,7 @@ struct maintenance_task {\n \n enum maintenance_task_label {\n \tTASK_PREFETCH,\n+\tTASK_LOOSE_OBJECTS,\n \tTASK_GC,\n \tTASK_COMMIT_GRAPH,\n \n@@ -856,6 +949,10 @@ static struct maintenance_task tasks[] = {\n \t\t\"prefetch\",\n \t\tmaintenance_task_prefetch,\n \t},\n+\t[TASK_LOOSE_OBJECTS] = {\n+\t\t\"loose-objects\",\n+\t\tmaintenance_task_loose_objects,\n+\t},\n \t[TASK_GC] = {\n \t\t\"gc\",\n \t\tmaintenance_task_gc,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 5294396a24..27a423a4f2 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -67,4 +67,43 @@ test_expect_success 'prefetch multiple remotes' '\n \tgit log prefetch/remote2/two\n '\n \n+test_expect_success 'loose-objects task' '\n+\t# Repack everything so we know the state of the object dir\n+\tgit repack -adk &&\n+\n+\t# Hack to stop maintenance from running during \"git commit\"\n+\techo in use >.git/objects/maintenance.lock &&\n+\n+\t# Assuming that \"git commit\" creates at least one loose object\n+\ttest_commit create-loose-object &&\n+\trm .git/objects/maintenance.lock &&\n+\n+\tls .git/objects >obj-dir-before &&\n+\ttest_file_not_empty obj-dir-before &&\n+\tls .git/objects/pack/*.pack >packs-before &&\n+\ttest_line_count = 1 packs-before &&\n+\n+\t# The first run creates a pack-file\n+\t# but does not delete loose objects.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-between &&\n+\ttest_cmp obj-dir-before obj-dir-between &&\n+\tls .git/objects/pack/*.pack >packs-between &&\n+\ttest_line_count = 2 packs-between &&\n+\tls .git/objects/pack/loose-*.pack >loose-packs &&\n+\ttest_line_count = 1 loose-packs &&\n+\n+\t# The second run deletes loose objects\n+\t# but does not create a pack-file.\n+\tgit maintenance run --task=loose-objects &&\n+\tls .git/objects >obj-dir-after &&\n+\tcat >expect <<-\\EOF &&\n+\tinfo\n+\tpack\n+\tEOF\n+\ttest_cmp expect obj-dir-after &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_cmp packs-between packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"402493","messageId":"9e38ade15c3a3eeec58dc262f1835d344c4899c0.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 09/20] maintenance: add prefetch task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:15Z","receivedAt":"2020-07-30T22:24:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen working with very large repositories, an incremental 'git fetch'\ncommand can download a large amount of data. If there are many other\nusers pushing to a common repo, then this data can rival the initial\npack-file size of a 'git clone' of a medium-size repo.\n\nUsers may want to keep the data on their local repos as close as\npossible to the data on the remote repos by fetching periodically in\nthe background. This can break up a large daily fetch into several\nsmaller hourly fetches.\n\nThe task is called \"prefetch\" because it is work done in advance\nof a foreground fetch to make that 'git fetch' command much faster.\n\nHowever, if we simply ran 'git fetch ' in the background,\nthen the user running a foregroudn 'git fetch ' would lose\nsome important feedback when a new branch appears or an existing\nbranch updates. This is especially true if a remote branch is\nforce-updated and this isn't noticed by the user because it occurred\nin the background. Further, the functionality of 'git push\n--force-with-lease' becomes suspect.\n\nWhen running 'git fetch ' in the background, use\nthe following options for careful updating:\n\n1. --no-tags prevents getting a new tag when a user wants to see\n the new tags appear in their foreground fetches.\n\n2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n While this looks confusing, this was documented and tested by\n b40a50264ac (fetch: document and test --refmap=\"\", 2020-01-21),\n including this sentence in the documentation:\n\n\tProviding an empty `` to the `--refmap` option\n\tcauses Git to ignore the configured refspecs and rely\n\tentirely on the refspecs supplied as command-line arguments.\n\n3. By adding a new refspec \"+refs/heads/*:refs/prefetch//*\"\n we can ensure that we actually load the new values somewhere in\n our refspace while not updating refs/heads or refs/remotes. By\n storing these refs here, the commit-graph job will update the\n commit-graph with the commits from these hidden refs.\n\n4. --prune will delete the refs/prefetch/ refs that no\n longer appear on the remote.\n\n5. --no-write-fetch-head prevents updating FETCH_HEAD.\n\nWe've been using this step as a critical background job in Scalar\n[1] (and VFS for Git). This solved a pain point that was showing up\nin user reports: fetching was a pain! Users do not like waiting to\ndownload the data that was created while they were away from their\nmachines. After implementing background fetch, the foreground fetch\ncommands sped up significantly because they mostly just update refs\nand download a small amount of new data. The effect is especially\ndramatic when paried with --no-show-forced-udpates (through\nfetch.showForcedUpdates=false).\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 +++++++++\n builtin/gc.c | 52 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 24 ++++++++++++++\n 3 files changed, 91 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 9204762e21..d134192fa8 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -53,6 +53,21 @@ since it will not expire `.graph` files that were in the previous\n `commit-graph-chain` file. They will be deleted by a later run based on\n the expiration delay.\n \n+prefetch::\n+\tThe `prefetch` task updates the object directory with the latest\n+\tobjects from all registered remotes. For each remote, a `git fetch`\n+\tcommand is run. The refmap is custom to avoid updating local or remote\n+\tbranches (those in `refs/heads` or `refs/remotes`). Instead, the\n+\tremote refs are stored in `refs/prefetch//`. Also, tags are\n+\tnot updated.\n++\n+This is done to avoid disrupting the remote-tracking branches. The end users\n+expect these refs to stay unmoved unless they initiate a fetch. With prefetch\n+task, however, the objects necessary to complete a later real fetch would\n+already be obtained, so the real fetch would go faster. In the ideal case,\n+it will just become an update to bunch of remote-tracking branches without\n+any object transfer.\n+\n gc::\n \tCleanup unnecessary files and optimize the local repository. \"GC\"\n \tstands for \"garbage collection,\" but this task performs many\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b57bc7b0ff..1f20428286 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -28,6 +28,7 @@\n #include \"blob.h\"\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n+#include \"remote.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -769,6 +770,52 @@ static int maintenance_task_commit_graph(void)\n \treturn 1;\n }\n \n+static int fetch_remote(const char *remote)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n+\t\t \"--no-write-fetch-head\", \"--refmap=\", NULL);\n+\n+\tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--quiet\");\n+\n+\treturn !!run_command(&child);\n+}\n+\n+static int fill_each_remote(struct remote *remote, void *cbdata)\n+{\n+\tstruct string_list *remotes = (struct string_list *)cbdata;\n+\n+\tstring_list_append(remotes, remote->name);\n+\treturn 0;\n+}\n+\n+static int maintenance_task_prefetch(void)\n+{\n+\tint result = 0;\n+\tstruct string_list_item *item;\n+\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n+\n+\tif (for_each_remote(fill_each_remote, &remotes)) {\n+\t\terror(_(\"failed to fill remotes\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tfor (item = remotes.items;\n+\t item && item < remotes.items + remotes.nr;\n+\t item++)\n+\t\tresult |= fetch_remote(item->string);\n+\n+cleanup:\n+\tstring_list_clear(&remotes, 0);\n+\treturn result;\n+}\n+\n static int maintenance_task_gc(void)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -796,6 +843,7 @@ struct maintenance_task {\n };\n \n enum maintenance_task_label {\n+\tTASK_PREFETCH,\n \tTASK_GC,\n \tTASK_COMMIT_GRAPH,\n \n@@ -804,6 +852,10 @@ enum maintenance_task_label {\n };\n \n static struct maintenance_task tasks[] = {\n+\t[TASK_PREFETCH] = {\n+\t\t\"prefetch\",\n+\t\tmaintenance_task_prefetch,\n+\t},\n \t[TASK_GC] = {\n \t\t\"gc\",\n \t\tmaintenance_task_gc,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3cdccb24df..5294396a24 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -43,4 +43,28 @@ test_expect_success 'run --task duplicate' '\n \ttest_i18ngrep \"cannot be selected multiple times\" err\n '\n \n+test_expect_success 'run --task=prefetch with no remotes' '\n+\tgit maintenance run --task=prefetch 2>err &&\n+\ttest_must_be_empty err\n+'\n+\n+test_expect_success 'prefetch multiple remotes' '\n+\tgit clone . clone1 &&\n+\tgit clone . clone2 &&\n+\tgit remote add remote1 \"file://$(pwd)/clone1\" &&\n+\tgit remote add remote2 \"file://$(pwd)/clone2\" &&\n+\tgit -C clone1 switch -c one &&\n+\tgit -C clone2 switch -c two &&\n+\ttest_commit -C clone1 one &&\n+\ttest_commit -C clone2 two &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-prefetch.txt\" git maintenance run --task=prefetch &&\n+\tgrep \",\\\"fetch\\\",\\\"remote1\\\"\" run-prefetch.txt &&\n+\tgrep \",\\\"fetch\\\",\\\"remote2\\\"\" run-prefetch.txt &&\n+\ttest_path_is_missing .git/refs/remotes &&\n+\ttest_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&\n+\ttest_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&\n+\tgit log prefetch/remote1/one &&\n+\tgit log prefetch/remote2/two\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"402492","messageId":"c2baf6e11934df0d6128bcc2867146163fa1b14b.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 11/20] midx: enable core.multiPackIndex by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:17Z","receivedAt":"2020-07-30T22:24:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe core.multiPackIndex setting has been around since c4d25228ebb\n(config: create core.multiPackIndex setting, 2018-07-12), but has been\ndisabled by default. If a user wishes to use the multi-pack-index\nfeature, then they must enable this config and run 'git multi-pack-index\nwrite'.\n\nThe multi-pack-index feature is relatively stable now, so make the\nconfig option true by default. For users that do not use a\nmulti-pack-index, the only extra cost will be a file lookup to see if a\nmulti-pack-index file exists (once per process, per object directory).\n\nAlso, this config option will be referenced by an upcoming\n\"incremental-repack\" task in the maintenance builtin, so move the config\noption into the repository settings struct. Note that if\nGIT_TEST_MULTI_PACK_INDEX=1, then we want to ignore the config option\nand treat core.multiPackIndex as enabled.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/core.txt | 4 ++--\n midx.c | 11 +++--------\n repo-settings.c | 6 ++++++\n repository.h | 2 ++\n 4 files changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/config/core.txt b/Documentation/config/core.txt\nindex 74619a9c03..86c91d5381 100644\n--- a/Documentation/config/core.txt\n+++ b/Documentation/config/core.txt\n@@ -606,8 +606,8 @@ core.useReplaceRefs::\n \n core.multiPackIndex::\n \tUse the multi-pack-index file to track multiple packfiles using a\n-\tsingle index. See link:technical/multi-pack-index.html[the\n-\tmulti-pack-index design document].\n+\tsingle index. See linkgit:git-multi-pack-index[1] for more\n+\tinformation. Defaults to true.\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See linkgit:git-sparse-checkout[1]\ndiff --git a/midx.c b/midx.c\nindex a5fb797ede..ef499cf504 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -10,6 +10,7 @@\n #include \"progress.h\"\n #include \"trace2.h\"\n #include \"run-command.h\"\n+#include \"repository.h\"\n \n #define MIDX_SIGNATURE 0x4d494458 /* \"MIDX\" */\n #define MIDX_VERSION 1\n@@ -384,15 +385,9 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i\n {\n \tstruct multi_pack_index *m;\n \tstruct multi_pack_index *m_search;\n-\tint config_value;\n-\tstatic int env_value = -1;\n \n-\tif (env_value < 0)\n-\t\tenv_value = git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0);\n-\n-\tif (!env_value &&\n-\t (repo_config_get_bool(r, \"core.multipackindex\", &config_value) ||\n-\t !config_value))\n+\tprepare_repo_settings(r);\n+\tif (!r->settings.core_multi_pack_index)\n \t\treturn 0;\n \n \tfor (m_search = r->objects->multi_pack_index; m_search; m_search = m_search->next)\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0918408b34..5bd2c22726 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"config.h\"\n #include \"repository.h\"\n+#include \"midx.h\"\n \n #define UPDATE_DEFAULT_BOOL(s,v) do { if (s == -1) { s = v; } } while(0)\n \n@@ -47,6 +48,11 @@ void prepare_repo_settings(struct repository *r)\n \t\tr->settings.pack_use_sparse = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.pack_use_sparse, 1);\n \n+\tvalue = git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0);\n+\tif (value || !repo_config_get_bool(r, \"core.multipackindex\", &value))\n+\t\tr->settings.core_multi_pack_index = value;\n+\tUPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1);\n+\n \tif (!repo_config_get_bool(r, \"feature.manyfiles\", &value) && value) {\n \t\tUPDATE_DEFAULT_BOOL(r->settings.index_version, 4);\n \t\tUPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE);\ndiff --git a/repository.h b/repository.h\nindex 3c1f7d54bd..3901ce0b65 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -37,6 +37,8 @@ struct repo_settings {\n \n \tint pack_use_sparse;\n \tenum fetch_negotiation_setting fetch_negotiation_algorithm;\n+\n+\tint core_multi_pack_index;\n };\n \n struct repository {\n-- \ngitgitgadget\n\n"},{"id":"402494","messageId":"99840c4b8f28b33b1c7ec2e861ee78bf81fe0277.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 14/20] maintenance: create maintenance..enabled config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:20Z","receivedAt":"2020-07-30T22:25:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nCurrently, a normal run of \"git maintenance run\" will only run the 'gc'\ntask, as it is the only one enabled. This is mostly for backwards-\ncompatible reasons since \"git maintenance run --auto\" commands replaced\nprevious \"git gc --auto\" commands after some Git processes. Users could\nmanually run specific maintenance tasks by calling \"git maintenance run\n--task=\" directly.\n\nAllow users to customize which steps are run automatically using config.\nThe 'maintenance..enabled' option then can turn on these other\ntasks (or turn off the 'gc' task).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 2 ++\n Documentation/config/maintenance.txt | 4 ++++\n Documentation/git-maintenance.txt | 6 +++++-\n builtin/gc.c | 19 +++++++++++++++++++\n t/t7900-maintenance.sh | 12 ++++++++++++\n 5 files changed, 42 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/config/maintenance.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..2783b825f9 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -396,6 +396,8 @@ include::config/mailinfo.txt[]\n \n include::config/mailmap.txt[]\n \n+include::config/maintenance.txt[]\n+\n include::config/man.txt[]\n \n include::config/merge.txt[]\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nnew file mode 100644\nindex 0000000000..370cbfb42f\n--- /dev/null\n+++ b/Documentation/config/maintenance.txt\n@@ -0,0 +1,4 @@\n+maintenance..enabled::\n+\tThis boolean config option controls whether the maintenance task\n+\twith name `` is run when no `--task` option is specified.\n+\tBy default, only `maintenance.gc.enabled` is true.\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex a598582986..1bd105918f 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -30,7 +30,11 @@ SUBCOMMANDS\n -----------\n \n run::\n-\tRun one or more maintenance tasks.\n+\tRun one or more maintenance tasks. If one or more `--task` options\n+\tare specified, then those tasks are run in that order. Otherwise,\n+\tthe tasks are determined by which `maintenance..enabled`\n+\tconfig options are true. By default, only `maintenance.gc.enabled`\n+\tis true.\n \n TASKS\n -----\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex d94eb3e6ad..c599690591 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1171,6 +1171,24 @@ static int maintenance_run(void)\n \treturn result;\n }\n \n+static void initialize_task_config(void)\n+{\n+\tint i;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tfor (i = 0; i < TASK__COUNT; i++) {\n+\t\tint config_value;\n+\n+\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_bool(config_name.buf, &config_value))\n+\t\t\ttasks[i].enabled = config_value;\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+}\n+\n static int task_option_parse(const struct option *opt,\n \t\t\t const char *arg, int unset)\n {\n@@ -1224,6 +1242,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t\t builtin_maintenance_options);\n \n \topts.quiet = !isatty(2);\n+\tinitialize_task_config();\n \n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_maintenance_options,\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 4e9c1dfa0f..affa268dec 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -21,6 +21,18 @@ test_expect_success 'run [--auto|--quiet]' '\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n+test_expect_success 'maintenance..enabled' '\n+\tgit config maintenance.gc.enabled false &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run-config.txt\" git maintenance run &&\n+\t! grep \",\\\"fetch\\\"\" run-config.txt &&\n+\t! grep \",\\\"gc\\\"\" run-config.txt &&\n+\t! grep \",\\\"multi-pack-index\\\"\" run-config.txt &&\n+\tgrep \",\\\"commit-graph\\\"\" run-config.txt &&\n+\tgrep \",\\\"prune-packed\\\"\" run-config.txt\n+'\n+\n test_expect_success 'run --task=' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-commit-graph.txt\" git maintenance run --task=commit-graph &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-gc.txt\" git maintenance run --task=gc &&\n-- \ngitgitgadget\n\n"},{"id":"402495","messageId":"a087c63572a63bf1bcb6e002f214ccf1e2291771.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 15/20] maintenance: use pointers to check --auto","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:21Z","receivedAt":"2020-07-30T22:25:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' command has an '--auto' option. This is used\nby other Git commands such as 'git commit' or 'git fetch' to check if\nmaintenance should be run after adding data to the repository.\n\nPreviously, this --auto option was only used to add the argument to the\n'git gc' command as part of the 'gc' task. We will be expanding the\nother tasks to perform a check to see if they should do work as part of\nthe --auto flag, when they are enabled by config.\n\nFirst, update the 'gc' task to perform the auto check inside the\nmaintenance process. This prevents running an extra 'git gc --auto'\ncommand when not needed. It also shows a model for other tasks.\n\nSecond, use the 'auto_condition' function pointer as a signal for\nwhether we enable the maintenance task under '--auto'. For instance, we\ndo not want to enable the 'fetch' task in '--auto' mode, so that\nfunction pointer will remain NULL.\n\nNow that we are not automatically calling 'git gc', a test in\nt5514-fetch-multiple.sh must be changed to watch for 'git maintenance'\ninstead.\n\nWe continue to pass the '--auto' option to the 'git gc' command when\nnecessary, because of the gc.autoDetach config option changes behavior.\nLikely, we will want to absorb the daemonizing behavior implied by\ngc.autoDetach as a maintenance.autoDetach config option.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 16 ++++++++++++++++\n t/t5514-fetch-multiple.sh | 2 +-\n t/t7900-maintenance.sh | 2 +-\n 3 files changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c599690591..1c449b3776 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1074,9 +1074,17 @@ static int maintenance_task_incremental_repack(void)\n \n typedef int maintenance_task_fn(void);\n \n+/*\n+ * An auto condition function returns 1 if the task should run\n+ * and 0 if the task should NOT run. See needs_to_gc() for an\n+ * example.\n+ */\n+typedef int maintenance_auto_fn(void);\n+\n struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n+\tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1,\n \t\t selected:1;\n \tint selected_order;\n@@ -1109,6 +1117,7 @@ static struct maintenance_task tasks[] = {\n \t[TASK_GC] = {\n \t\t\"gc\",\n \t\tmaintenance_task_gc,\n+\t\tneed_to_gc,\n \t\t1,\n \t},\n \t[TASK_COMMIT_GRAPH] = {\n@@ -1161,6 +1170,11 @@ static int maintenance_run(void)\n \t\tif (!opts.tasks_selected && !tasks[i].enabled)\n \t\t\tcontinue;\n \n+\t\tif (opts.auto_flag &&\n+\t\t (!tasks[i].auto_condition ||\n+\t\t !tasks[i].auto_condition()))\n+\t\t\tcontinue;\n+\n \t\tif (tasks[i].fn()) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n@@ -1175,6 +1189,8 @@ static void initialize_task_config(void)\n {\n \tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n+\tgc_config();\n+\n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n \ndiff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh\nindex de8e2f1531..bd202ec6f3 100755\n--- a/t/t5514-fetch-multiple.sh\n+++ b/t/t5514-fetch-multiple.sh\n@@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' '\n \t GIT_TRACE=1 git fetch --multiple one two 2>trace &&\n \t git branch -r > output &&\n \t test_cmp ../expect output &&\n-\t grep \"built-in: git gc\" trace >gc &&\n+\t grep \"built-in: git maintenance\" trace >gc &&\n \t test_line_count = 1 gc\n \t)\n '\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex affa268dec..19494e6c43 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -17,7 +17,7 @@ test_expect_success 'run [--auto|--quiet]' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-auto.txt\" git maintenance run --auto &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-quiet.txt\" git maintenance run --quiet &&\n \tgrep \",\\\"gc\\\"]\" run-no-auto.txt &&\n-\tgrep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n+\t! grep \",\\\"gc\\\",\\\"--auto\\\"\" run-auto.txt &&\n \tgrep \",\\\"gc\\\",\\\"--quiet\\\"\" run-quiet.txt\n '\n \n-- \ngitgitgadget\n\n"},{"id":"402498","messageId":"ef2a2319565939bfbdbd2e898a980c61ecc39c6a.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 13/20] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:19Z","receivedAt":"2020-07-30T22:25:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen repacking during the 'incremental-repack' task, we use the\n--batch-size option in 'git multi-pack-index repack'. The initial setting\nused --batch-size=0 to repack everything into a single pack-file. This is\nnot sustainable for a large repository. The amount of work required is\nalso likely to use too many system resources for a background job.\n\nUpdate the 'incremental-repack' task by dynamically computing a\n--batch-size option based on the current pack-file structure.\n\nThe dynamic default size is computed with this idea in mind for a client\nrepository that was cloned from a very large remote: there is likely one\n\"big\" pack-file that was created at clone time. Thus, do not try\nrepacking it as it is likely packed efficiently by the server.\n\nInstead, we select the second-largest pack-file, and create a batch size\nthat is one larger than that pack-file. If there are three or more\npack-files, then this guarantees that at least two will be combined into\na new pack-file.\n\nOf course, this means that the second-largest pack-file size is likely\nto grow over time and may eventually surpass the initially-cloned\npack-file. Recall that the pack-file batch is selected in a greedy\nmanner: the packs are considered from oldest to newest and are selected\nif they have size smaller than the batch size until the total selected\nsize is larger than the batch size. Thus, that oldest \"clone\" pack will\nbe first to repack after the new data creates a pack larger than that.\n\nWe also want to place some limits on how large these pack-files become,\nin order to bound the amount of time spent repacking. A maximum\nbatch-size of two gigabytes means that large repositories will never be\npacked into a single pack-file using this job, but also that repack is\nrather expensive. This is a trade-off that is valuable to have if the\nmaintenance is being run automatically or in the background. Users who\ntruly want to optimize for space and performance (and are willing to pay\nthe upfront cost of a full repack) can use the 'gc' task to do so.\n\nReported-by: Son Luong Ngoc \nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 43 +++++++++++++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 5 +++--\n 2 files changed, 45 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 99ab1f5e9d..d94eb3e6ad 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -988,6 +988,46 @@ static int multi_pack_index_expire(void)\n \treturn 0;\n }\n \n+#define TWO_GIGABYTES (0x7FFF)\n+\n+static off_t get_auto_pack_size(void)\n+{\n+\t/*\n+\t * The \"auto\" value is special: we optimize for\n+\t * one large pack-file (i.e. from a clone) and\n+\t * expect the rest to be small and they can be\n+\t * repacked quickly.\n+\t *\n+\t * The strategy we select here is to select a\n+\t * size that is one more than the second largest\n+\t * pack-file. This ensures that we will repack\n+\t * at least two packs if there are three or more\n+\t * packs.\n+\t */\n+\toff_t max_size = 0;\n+\toff_t second_largest_size = 0;\n+\toff_t result_size;\n+\tstruct packed_git *p;\n+\tstruct repository *r = the_repository;\n+\n+\treprepare_packed_git(r);\n+\tfor (p = get_all_packs(r); p; p = p->next) {\n+\t\tif (p->pack_size > max_size) {\n+\t\t\tsecond_largest_size = max_size;\n+\t\t\tmax_size = p->pack_size;\n+\t\t} else if (p->pack_size > second_largest_size)\n+\t\t\tsecond_largest_size = p->pack_size;\n+\t}\n+\n+\tresult_size = second_largest_size + 1;\n+\n+\t/* But limit ourselves to a batch size of 2g */\n+\tif (result_size > TWO_GIGABYTES)\n+\t\tresult_size = TWO_GIGABYTES;\n+\n+\treturn result_size;\n+}\n+\n static int multi_pack_index_repack(void)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -998,7 +1038,8 @@ static int multi_pack_index_repack(void)\n \tif (opts.quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n-\tstrvec_push(&child.args, \"--batch-size=0\");\n+\tstrvec_pushf(&child.args, \"--batch-size=%\"PRIuMAX,\n+\t\t\t\t (uintmax_t)get_auto_pack_size());\n \n \tclose_object_store(the_repository->objects);\n \ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 0cc59adb21..4e9c1dfa0f 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -138,10 +138,11 @@ test_expect_success 'incremental-repack task' '\n \ttest_line_count = 4 packs-between &&\n \n \t# the job deletes the two old packs, and does not write\n-\t# a new one because only one pack remains.\n+\t# a new one because the batch size is not high enough to\n+\t# pack the largest pack-file.\n \tgit maintenance run --task=incremental-repack &&\n \tls .git/objects/pack/*.pack >packs-after &&\n-\ttest_line_count = 1 packs-after\n+\ttest_line_count = 2 packs-after\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"402496","messageId":"00f47c48484c16987592d8b39fbd70e2744e3d4a.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 12/20] maintenance: add incremental-repack task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:18Z","receivedAt":"2020-07-30T22:25:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change cleaned up loose objects using the\n'loose-objects' that can be run safely in the background. Add a\nsimilar job that performs similar cleanups for pack-files.\n\nOne issue with running 'git repack' is that it is designed to\nrepack all pack-files into a single pack-file. While this is the\nmost space-efficient way to store object data, it is not time or\nmemory efficient. This becomes extremely important if the repo is\nso large that a user struggles to store two copies of the pack on\ntheir disk.\n\nInstead, perform an \"incremental\" repack by collecting a few small\npack-files into a new pack-file. The multi-pack-index facilitates\nthis process ever since 'git multi-pack-index expire' was added in\n19575c7 (multi-pack-index: implement 'expire' subcommand,\n2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1\n(midx: implement midx_repack(), 2019-06-10).\n\nThe 'incremental-repack' task runs the following steps:\n\n1. 'git multi-pack-index write' creates a multi-pack-index file if\n one did not exist, and otherwise will update the multi-pack-index\n with any new pack-files that appeared since the last write. This\n is particularly relevant with the background fetch job.\n\n When the multi-pack-index sees two copies of the same object, it\n stores the offset data into the newer pack-file. This means that\n some old pack-files could become \"unreferenced\" which I will use\n to mean \"a pack-file that is in the pack-file list of the\n multi-pack-index but none of the objects in the multi-pack-index\n reference a location inside that pack-file.\"\n\n2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n and updaes the multi-pack-index to drop those pack-files from the\n list. This is safe to do as concurrent Git processes will see the\n multi-pack-index and not open those packs when looking for object\n contents. (Similar to the 'loose-objects' job, there are some Git\n commands that open pack-files regardless of the multi-pack-index,\n but they are rarely used. Further, a user that self-selects to\n use background operations would likely refrain from using those\n commands.)\n\n3. 'git multi-pack-index repack --bacth-size=' collects a set\n of pack-files that are listed in the multi-pack-index and creates\n a new pack-file containing the objects whose offsets are listed\n by the multi-pack-index to be in those objects. The set of pack-\n files is selected greedily by sorting the pack-files by modified\n time and adding a pack-file to the set if its \"expected size\" is\n smaller than the batch size until the total expected size of the\n selected pack-files is at least the batch size. The \"expected\n size\" is calculated by taking the size of the pack-file divided\n by the number of objects in the pack-file and multiplied by the\n number of objects from the multi-pack-index with offset in that\n pack-file. The expected size approximates how much data from that\n pack-file will contribute to the resulting pack-file size. The\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n\n The next run of the incremental-repack task will delete these\n repacked pack-files during the 'expire' step.\n\n In this version, the batch size is set to \"0\" which ignores the\n size restrictions when selecting the pack-files. It instead\n selects all pack-files and repacks all packed objects into a\n single pack-file. This will be updated in the next change, but\n it requires doing some calculations that are better isolated to\n a separate change.\n\nEach of the above steps update the multi-pack-index file. After\neach step, we verify the new multi-pack-index. If the new\nmulti-pack-index is corrupt, then delete the multi-pack-index,\nrewrite it from scratch, and stop doing the later steps of the\njob. This is intended to be an extra-safe check without leaving\na repo with many pack-files without a multi-pack-index.\n\nThese steps are based on a similar background maintenance step in\nScalar (and VFS for Git) [1]. This was incredibly effective for\nusers of the Windows OS repository. After using the same VFS for Git\nrepository for over a year, some users had _thousands_ of pack-files\nthat combined to up to 250 GB of data. We noticed a few users were\nrunning into the open file descriptor limits (due in part to a bug\nin the multi-pack-index fixed by af96fe3 (midx: add packs to\npacked_git linked list, 2019-04-29).\n\nThese pack-files were mostly small since they contained the commits\nand trees that were pushed to the origin in a given hour. The GVFS\nprotocol includes a \"prefetch\" step that asks for pre-computed pack-\nfiles containing commits and trees by timestamp. These pack-files\nwere grouped into \"daily\" pack-files once a day for up to 30 days.\nIf a user did not request prefetch packs for over 30 days, then they\nwould get the entire history of commits and trees in a new, large\npack-file. This led to a large number of pack-files that had poor\ndelta compression.\n\nBy running this pack-file maintenance step once per day, these repos\nwith thousands of packs spanning 200+ GB dropped to dozens of pack-\nfiles spanning 30-50 GB. This was done all without removing objects\nfrom the system and using a constant batch size of two gigabytes.\nOnce the work was done to reduce the pack-files to small sizes, the\nbatch size of two gigabytes means that not every run triggers a\nrepack operation, so the following run will not expire a pack-file.\nThis has kept these repos in a \"clean\" state.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 ++++\n builtin/gc.c | 112 ++++++++++++++++++++++++++++++\n midx.c | 2 +-\n midx.h | 1 +\n t/t5319-multi-pack-index.sh | 1 +\n t/t7900-maintenance.sh | 38 ++++++++++\n 6 files changed, 168 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 077929b691..a598582986 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -87,6 +87,21 @@ loose-objects::\n \tthousand objects to prevent the job from taking too long on a\n \trepository with many loose objects.\n \n+incremental-repack::\n+\tThe `incremental-repack` job repacks the object directory\n+\tusing the `multi-pack-index` feature. In order to prevent race\n+\tconditions with concurrent Git commands, it follows a two-step\n+\tprocess. First, it deletes any pack-files included in the\n+\t`multi-pack-index` where none of the objects in the\n+\t`multi-pack-index` reference those pack-files; this only happens\n+\tif all objects in the pack-file are also stored in a newer\n+\tpack-file. Second, it selects a group of pack-files whose \"expected\n+\tsize\" is below the batch size until the group has total expected\n+\tsize at least the batch size; see the `--batch-size` option for\n+\tthe `repack` subcommand in linkgit:git-multi-pack-index[1]. The\n+\tdefault batch-size is zero, which is a special case that attempts\n+\tto repack all pack-files into a single pack-file.\n+\n OPTIONS\n -------\n --auto::\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 96ded73b2f..99ab1f5e9d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -29,6 +29,7 @@\n #include \"tree.h\"\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n+#include \"midx.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -924,6 +925,112 @@ static int maintenance_task_loose_objects(void)\n \treturn prune_packed() || pack_loose();\n }\n \n+static int multi_pack_index_write(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"multi-pack-index\", \"write\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\tif (run_command(&child))\n+\t\treturn error(_(\"failed to write multi-pack-index\"));\n+\n+\treturn 0;\n+}\n+\n+static int rewrite_multi_pack_index(void)\n+{\n+\tstruct repository *r = the_repository;\n+\tchar *midx_name = get_midx_filename(r->objects->odb->path);\n+\n+\tunlink(midx_name);\n+\tfree(midx_name);\n+\n+\treturn multi_pack_index_write();\n+}\n+\n+static int multi_pack_index_verify(const char *message)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"multi-pack-index\", \"verify\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\tif (run_command(&child)) {\n+\t\twarning(_(\"'git multi-pack-index verify' failed %s\"), message);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int multi_pack_index_expire(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"multi-pack-index\", \"expire\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\tclose_object_store(the_repository->objects);\n+\n+\tif (run_command(&child))\n+\t\treturn error(_(\"'git multi-pack-index expire' failed\"));\n+\n+\treturn 0;\n+}\n+\n+static int multi_pack_index_repack(void)\n+{\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"multi-pack-index\", \"repack\", NULL);\n+\n+\tif (opts.quiet)\n+\t\tstrvec_push(&child.args, \"--no-progress\");\n+\n+\tstrvec_push(&child.args, \"--batch-size=0\");\n+\n+\tclose_object_store(the_repository->objects);\n+\n+\tif (run_command(&child))\n+\t\treturn error(_(\"'git multi-pack-index repack' failed\"));\n+\n+\treturn 0;\n+}\n+\n+static int maintenance_task_incremental_repack(void)\n+{\n+\tprepare_repo_settings(the_repository);\n+\tif (!the_repository->settings.core_multi_pack_index) {\n+\t\twarning(_(\"skipping incremental-repack task because core.multiPackIndex is disabled\"));\n+\t\treturn 0;\n+\t}\n+\n+\tif (multi_pack_index_write())\n+\t\treturn 1;\n+\tif (multi_pack_index_verify(\"after initial write\"))\n+\t\treturn rewrite_multi_pack_index();\n+\tif (multi_pack_index_expire())\n+\t\treturn 1;\n+\tif (multi_pack_index_verify(\"after expire step\"))\n+\t\treturn !!rewrite_multi_pack_index();\n+\tif (multi_pack_index_repack())\n+\t\treturn 1;\n+\tif (multi_pack_index_verify(\"after repack step\"))\n+\t\treturn !!rewrite_multi_pack_index();\n+\treturn 0;\n+}\n+\n typedef int maintenance_task_fn(void);\n \n struct maintenance_task {\n@@ -937,6 +1044,7 @@ struct maintenance_task {\n enum maintenance_task_label {\n \tTASK_PREFETCH,\n \tTASK_LOOSE_OBJECTS,\n+\tTASK_INCREMENTAL_REPACK,\n \tTASK_GC,\n \tTASK_COMMIT_GRAPH,\n \n@@ -953,6 +1061,10 @@ static struct maintenance_task tasks[] = {\n \t\t\"loose-objects\",\n \t\tmaintenance_task_loose_objects,\n \t},\n+\t[TASK_INCREMENTAL_REPACK] = {\n+\t\t\"incremental-repack\",\n+\t\tmaintenance_task_incremental_repack,\n+\t},\n \t[TASK_GC] = {\n \t\t\"gc\",\n \t\tmaintenance_task_gc,\ndiff --git a/midx.c b/midx.c\nindex ef499cf504..f0ebee34c8 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -37,7 +37,7 @@\n \n #define PACK_EXPIRED UINT_MAX\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex b18cf53bc4..baeecc70c9 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -37,6 +37,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_filename(const char *object_dir);\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 7214cab36c..2abd29a007 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -3,6 +3,7 @@\n test_description='multi-pack-indexes'\n . ./test-lib.sh\n \n+GIT_TEST_MULTI_PACK_INDEX=0\n objdir=.git/objects\n \n midx_read_expect () {\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 27a423a4f2..0cc59adb21 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -5,6 +5,7 @@ test_description='git maintenance builtin'\n . ./test-lib.sh\n \n GIT_TEST_COMMIT_GRAPH=0\n+GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n@@ -106,4 +107,41 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'incremental-repack task' '\n+\tpackDir=.git/objects/pack &&\n+\tfor i in $(test_seq 1 5)\n+\tdo\n+\t\ttest_commit $i || return 1\n+\tdone &&\n+\n+\t# Create three disjoint pack-files with size BIG, small, small.\n+\techo HEAD~2 | git pack-objects --revs $packDir/test-1 &&\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-2 <<-\\EOF &&\n+\tHEAD~1\n+\t^HEAD~2\n+\tEOF\n+\ttest_tick &&\n+\tgit pack-objects --revs $packDir/test-3 <<-\\EOF &&\n+\tHEAD\n+\t^HEAD~1\n+\tEOF\n+\trm -f $packDir/pack-* &&\n+\trm -f $packDir/loose-* &&\n+\tls $packDir/*.pack >packs-before &&\n+\ttest_line_count = 3 packs-before &&\n+\n+\t# the job repacks the two into a new pack, but does not\n+\t# delete the old ones.\n+\tgit maintenance run --task=incremental-repack &&\n+\tls $packDir/*.pack >packs-between &&\n+\ttest_line_count = 4 packs-between &&\n+\n+\t# the job deletes the two old packs, and does not write\n+\t# a new one because only one pack remains.\n+\tgit maintenance run --task=incremental-repack &&\n+\tls .git/objects/pack/*.pack >packs-after &&\n+\ttest_line_count = 1 packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"402501","messageId":"39eb83ad1efe4dc521f5fc3838f47c7cdb507fc7.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 20/20] maintenance: add trace2 regions for task execution","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:26Z","receivedAt":"2020-07-30T22:25:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n builtin/gc.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 6ccc0cca19..33e889bc71 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1312,10 +1312,12 @@ static int maintenance_run(void)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n \t\tif (tasks[i].fn()) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n \t\t}\n+\t\ttrace2_region_leave(\"maintenance\", tasks[i].name, r);\n \t}\n \n \trollback_lock_file(&lk);\n-- \ngitgitgadget\n"},{"id":"402499","messageId":"801b262d1c26625d3f174fd80d7ddd336328bdd6.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 18/20] maintenance: add incremental-repack auto condition","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:24Z","receivedAt":"2020-07-30T22:25:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe incremental-repack task updates the multi-pack-index by deleting pack-\nfiles that have been replaced with new packs, then repacking a batch of\nsmall pack-files into a larger pack-file. This incremental repack is faster\nthan rewriting all object data, but is slower than some other\nmaintenance activities.\n\nThe 'maintenance.incremental-repack.auto' config option specifies how many\npack-files should exist outside of the multi-pack-index before running\nthe step. These pack-files could be created by 'git fetch' commands or\nby the loose-objects task. The default value is 10.\n\nSetting the option to zero disables the task with the '--auto' option,\nand a negative value makes the task run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 ++++++++\n builtin/gc.c | 31 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 30 +++++++++++++++++++++++++++\n 3 files changed, 70 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a9442dd260..22229e7174 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -21,3 +21,12 @@ maintenance.loose-objects.auto::\n \tpositive value implies the command should run when the number of\n \tloose objects is at least the value of `maintenance.loose-objects.auto`.\n \tThe default value is 100.\n+\n+maintenance.incremental-repack.auto::\n+\tThis integer config option controls how often the `incremental-repack`\n+\ttask should be run as part of `git maintenance run --auto`. If zero,\n+\tthen the `incremental-repack` task will not run with the `--auto`\n+\toption. A negative value will force the task to run every time.\n+\tOtherwise, a positive value implies the command should run when the\n+\tnumber of pack-files not in the multi-pack-index is at least the value\n+\tof `maintenance.incremental-repack.auto`. The default value is 10.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3b27dc7e7f..6ccc0cca19 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"refs.h\"\n+#include \"object-store.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1029,6 +1030,35 @@ static int maintenance_task_loose_objects(void)\n \treturn prune_packed() || pack_loose();\n }\n \n+static int incremental_repack_auto_condition(void)\n+{\n+\tstruct packed_git *p;\n+\tint enabled;\n+\tint incremental_repack_auto_limit = 10;\n+\tint count = 0;\n+\n+\tif (git_config_get_bool(\"core.multiPackIndex\", &enabled) ||\n+\t !enabled)\n+\t\treturn 0;\n+\n+\tgit_config_get_int(\"maintenance.incremental-repack.auto\",\n+\t\t\t &incremental_repack_auto_limit);\n+\n+\tif (!incremental_repack_auto_limit)\n+\t\treturn 0;\n+\tif (incremental_repack_auto_limit < 0)\n+\t\treturn 1;\n+\n+\tfor (p = get_packed_git(the_repository);\n+\t count < incremental_repack_auto_limit && p;\n+\t p = p->next) {\n+\t\tif (!p->multi_pack_index)\n+\t\t\tcount++;\n+\t}\n+\n+\treturn count >= incremental_repack_auto_limit;\n+}\n+\n static int multi_pack_index_write(void)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -1218,6 +1248,7 @@ static struct maintenance_task tasks[] = {\n \t[TASK_INCREMENTAL_REPACK] = {\n \t\t\"incremental-repack\",\n \t\tmaintenance_task_incremental_repack,\n+\t\tincremental_repack_auto_condition,\n \t},\n \t[TASK_GC] = {\n \t\t\"gc\",\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 32ac5c9fb7..c8ccaa0a3d 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -182,4 +182,34 @@ test_expect_success 'incremental-repack task' '\n \ttest_line_count = 2 packs-after\n '\n \n+test_expect_success 'maintenance.incremental-repack.auto' '\n+\tgit repack -adk &&\n+\tgit config core.multiPackIndex true &&\n+\tgit multi-pack-index write &&\n+\tGIT_TRACE2_EVENT=1 git -c maintenance.incremental-repack.auto=1 \\\n+\t\tmaintenance run --auto --task=incremental-repack >out &&\n+\t! grep \"\\\"multi-pack-index\\\"\" out &&\n+\tfor i in 1 2\n+\tdo\n+\t\ttest_commit A-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-A-$i git \\\n+\t\t\t-c maintenance.incremental-repack.auto=2 \\\n+\t\t\tmaintenance run --auto --task=incremental-repack &&\n+\t\t! grep \"\\\"multi-pack-index\\\"\" trace-A-$i &&\n+\t\ttest_commit B-$i &&\n+\t\tgit pack-objects --revs .git/objects/pack/pack <<-\\EOF &&\n+\t\tHEAD\n+\t\t^HEAD~1\n+\t\tEOF\n+\t\tGIT_TRACE2_EVENT=$(pwd)/trace-B-$i git \\\n+\t\t\t-c maintenance.incremental-repack.auto=2 \\\n+\t\t\tmaintenance run --auto --task=incremental-repack >out &&\n+\t\tgrep \"\\\"multi-pack-index\\\"\" trace-B-$i >/dev/null || return 1\n+\tdone\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"402500","messageId":"6ac3a58f2fa1b7456cb80867fd62c3c462c5c858.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 17/20] maintenance: create auto condition for loose-objects","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:23Z","receivedAt":"2020-07-30T22:25:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe loose-objects task deletes loose objects that already exist in a\npack-file, then place the remaining loose objects into a new pack-file.\nIf this step runs all the time, then we risk creating pack-files with\nvery few objects with every 'git commit' process. To prevent\noverwhelming the packs directory with small pack-files, place a minimum\nnumber of objects to justify the task.\n\nThe 'maintenance.loose-objects.auto' config option specifies a minimum\nnumber of loose objects to justify the task to run under the '--auto'\noption. This defaults to 100 loose objects. Setting the value to zero\nwill prevent the step from running under '--auto' while a negative value\nwill force it to run every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 +++++++++\n builtin/gc.c | 30 ++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 25 +++++++++++++++++++++++\n 3 files changed, 64 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 9bd69b9df3..a9442dd260 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -12,3 +12,12 @@ maintenance.commit-graph.auto::\n \treachable commits that are not in the commit-graph file is at least\n \tthe value of `maintenance.commit-graph.auto`. The default value is\n \t100.\n+\n+maintenance.loose-objects.auto::\n+\tThis integer config option controls how often the `loose-objects` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `loose-objects` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\tloose objects is at least the value of `maintenance.loose-objects.auto`.\n+\tThe default value is 100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c85813fffe..3b27dc7e7f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -927,6 +927,35 @@ struct write_loose_object_data {\n \tint batch_size;\n };\n \n+static int loose_object_auto_limit = 100;\n+\n+static int loose_object_count(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\tint *count = (int*)data;\n+\tif (++(*count) >= loose_object_auto_limit)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+static int loose_object_auto_condition(void)\n+{\n+\tint count = 0;\n+\n+\tgit_config_get_int(\"maintenance.loose-objects.auto\",\n+\t\t\t &loose_object_auto_limit);\n+\n+\tif (!loose_object_auto_limit)\n+\t\treturn 0;\n+\tif (loose_object_auto_limit < 0)\n+\t\treturn 1;\n+\n+\treturn for_each_loose_file_in_objdir(the_repository->objects->odb->path,\n+\t\t\t\t\t loose_object_count,\n+\t\t\t\t\t NULL, NULL, &count);\n+}\n+\n static int bail_on_loose(const struct object_id *oid,\n \t\t\t const char *path,\n \t\t\t void *data)\n@@ -1184,6 +1213,7 @@ static struct maintenance_task tasks[] = {\n \t[TASK_LOOSE_OBJECTS] = {\n \t\t\"loose-objects\",\n \t\tmaintenance_task_loose_objects,\n+\t\tloose_object_auto_condition,\n \t},\n \t[TASK_INCREMENTAL_REPACK] = {\n \t\t\"incremental-repack\",\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 19494e6c43..32ac5c9fb7 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -119,6 +119,31 @@ test_expect_success 'loose-objects task' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'maintenance.loose-objects.auto' '\n+\tgit repack -adk &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace-lo1.txt\" \\\n+\t\tgit -c maintenance.loose-objects.auto=1 maintenance \\\n+\t\trun --auto --task=loose-objects &&\n+\t! grep \"\\\"prune-packed\\\"\" trace-lo1.txt &&\n+\tfor i in 1 2\n+\tdo\n+\t\tprintf data-A-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loA-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\t! grep \"\\\"prune-packed\\\"\" trace-loA-$i &&\n+\t\tprintf data-B-$i | git hash-object -t blob --stdin -w &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loB-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loB-$i &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-loC-$i\" \\\n+\t\t\tgit -c maintenance.loose-objects.auto=2 \\\n+\t\t\tmaintenance run --auto --task=loose-objects &&\n+\t\tgrep \"\\\"prune-packed\\\"\" trace-loC-$i || return 1\n+\tdone\n+'\n+\n test_expect_success 'incremental-repack task' '\n \tpackDir=.git/objects/pack &&\n \tfor i in $(test_seq 1 5)\n-- \ngitgitgadget\n\n"},{"id":"402502","messageId":"9b4cef7635ede3e9e597d5f8e8da24e352f34c12.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 19/20] midx: use start_delayed_progress()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:25Z","receivedAt":"2020-07-30T22:25:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nNow that the multi-pack-index may be written as part of auto maintenance\nat the end of a command, reduce the progress output when the operations\nare quick. Use start_delayed_progress() instead of start_progress().\n\nUpdate t5319-multi-pack-index.sh to use GIT_PROGRESS_DELAY=0 now that\nthe progress indicators are conditional.\n\nSigned-off-by: Derrick Stolee \n---\n midx.c | 10 +++++-----\n t/t5319-multi-pack-index.sh | 14 +++++++-------\n 2 files changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/midx.c b/midx.c\nindex f0ebee34c8..66d7053d83 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -832,7 +832,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \n \tpacks.pack_paths_checked = 0;\n \tif (flags & MIDX_PROGRESS)\n-\t\tpacks.progress = start_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n+\t\tpacks.progress = start_delayed_progress(_(\"Adding packfiles to multi-pack-index\"), 0);\n \telse\n \t\tpacks.progress = NULL;\n \n@@ -969,7 +969,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *\n \t}\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Writing chunks to multi-pack-index\"),\n+\t\tprogress = start_delayed_progress(_(\"Writing chunks to multi-pack-index\"),\n \t\t\t\t\t num_chunks);\n \tfor (i = 0; i < num_chunks; i++) {\n \t\tif (written != chunk_offsets[i])\n@@ -1104,7 +1104,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag\n \t\treturn 0;\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Looking for referenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Looking for referenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tif (prepare_midx_pack(r, m, i))\n@@ -1225,7 +1225,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tcount = xcalloc(m->num_packs, sizeof(uint32_t));\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Counting referenced objects\"),\n+\t\tprogress = start_delayed_progress(_(\"Counting referenced objects\"),\n \t\t\t\t\t m->num_objects);\n \tfor (i = 0; i < m->num_objects; i++) {\n \t\tint pack_int_id = nth_midxed_pack_int_id(m, i);\n@@ -1235,7 +1235,7 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla\n \tstop_progress(&progress);\n \n \tif (flags & MIDX_PROGRESS)\n-\t\tprogress = start_progress(_(\"Finding and deleting unreferenced packfiles\"),\n+\t\tprogress = start_delayed_progress(_(\"Finding and deleting unreferenced packfiles\"),\n \t\t\t\t\t m->num_packs);\n \tfor (i = 0; i < m->num_packs; i++) {\n \t\tchar *pack_name;\ndiff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh\nindex 2abd29a007..26f224b0e3 100755\n--- a/t/t5319-multi-pack-index.sh\n+++ b/t/t5319-multi-pack-index.sh\n@@ -173,12 +173,12 @@ test_expect_success 'write progress off for redirected stderr' '\n '\n \n test_expect_success 'write force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress write 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'write with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress write 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -335,17 +335,17 @@ test_expect_success 'git-fsck incorrect offset' '\n '\n \n test_expect_success 'repack progress off for redirected stderr' '\n-\tgit multi-pack-index --object-dir=$objdir repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n test_expect_success 'repack force progress on for stderr' '\n-\tgit multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --progress repack 2>err &&\n \ttest_file_not_empty err\n '\n \n test_expect_success 'repack with the --no-progress option' '\n-\tgit multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n+\tGIT_PROGRESS_DELAY=0 git multi-pack-index --object-dir=$objdir --no-progress repack 2>err &&\n \ttest_line_count = 0 err\n '\n \n@@ -489,7 +489,7 @@ test_expect_success 'expire progress off for redirected stderr' '\n test_expect_success 'expire force progress on for stderr' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --progress expire 2>err &&\n \t\ttest_file_not_empty err\n \t)\n '\n@@ -497,7 +497,7 @@ test_expect_success 'expire force progress on for stderr' '\n test_expect_success 'expire with the --no-progress option' '\n \t(\n \t\tcd dup &&\n-\t\tgit multi-pack-index --no-progress expire 2>err &&\n+\t\tGIT_PROGRESS_DELAY=0 git multi-pack-index --no-progress expire 2>err &&\n \t\ttest_line_count = 0 err\n \t)\n '\n-- \ngitgitgadget\n\n"},{"id":"402503","messageId":"ef3a85450800800357c4d4db8e3f0da86621a6bd.1596147867.git.gitgitgadget@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"[PATCH v3 16/20] maintenance: add auto condition for commit-graph task","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-30T22:24:22Z","receivedAt":"2020-07-30T22:25:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nInstead of writing a new commit-graph in every 'git maintenance run\n--auto' process (when maintenance.commit-graph.enalbed is configured to\nbe true), only write when there are \"enough\" commits not in a\ncommit-graph file.\n\nThis count is controlled by the maintenance.commit-graph.auto config\noption.\n\nTo compute the count, use a depth-first search starting at each ref, and\nleaving markers using the PARENT1 flag. If this count reaches the limit,\nthen terminate early and start the task. Otherwise, this operation will\npeel every ref and parse the commit it points to. If these are all in\nthe commit-graph, then this is typically a very fast operation. Users\nwith many refs might feel a slow-down, and hence could consider updating\ntheir limit to be very small. A negative value will force the step to\nrun every time.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 10 ++++\n builtin/gc.c | 76 ++++++++++++++++++++++++++++\n object.h | 1 +\n 3 files changed, 87 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 370cbfb42f..9bd69b9df3 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -2,3 +2,13 @@ maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified.\n \tBy default, only `maintenance.gc.enabled` is true.\n+\n+maintenance.commit-graph.auto::\n+\tThis integer config option controls how often the `commit-graph` task\n+\tshould be run as part of `git maintenance run --auto`. If zero, then\n+\tthe `commit-graph` task will not run with the `--auto` option. A\n+\tnegative value will force the task to run every time. Otherwise, a\n+\tpositive value implies the command should run when the number of\n+\treachable commits that are not in the commit-graph file is at least\n+\tthe value of `maintenance.commit-graph.auto`. The default value is\n+\t100.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1c449b3776..c85813fffe 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -30,6 +30,7 @@\n #include \"promisor-remote.h\"\n #include \"remote.h\"\n #include \"midx.h\"\n+#include \"refs.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -713,6 +714,80 @@ static struct maintenance_opts {\n \tint tasks_selected;\n } opts;\n \n+/* Remember to update object flag allocation in object.h */\n+#define PARENT1\t\t(1u<<16)\n+\n+static int num_commits_not_in_graph = 0;\n+static int limit_commits_not_in_graph = 100;\n+\n+static int dfs_on_ref(const char *refname,\n+\t\t const struct object_id *oid, int flags,\n+\t\t void *cb_data)\n+{\n+\tint result = 0;\n+\tstruct object_id peeled;\n+\tstruct commit_list *stack = NULL;\n+\tstruct commit *commit;\n+\n+\tif (!peel_ref(refname, &peeled))\n+\t\toid = &peeled;\n+\tif (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tcommit = lookup_commit(the_repository, oid);\n+\tif (!commit)\n+\t\treturn 0;\n+\tif (parse_commit(commit))\n+\t\treturn 0;\n+\n+\tcommit_list_append(commit, &stack);\n+\n+\twhile (!result && stack) {\n+\t\tstruct commit_list *parent;\n+\n+\t\tcommit = pop_commit(&stack);\n+\n+\t\tfor (parent = commit->parents; parent; parent = parent->next) {\n+\t\t\tif (parse_commit(parent->item) ||\n+\t\t\t commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH ||\n+\t\t\t parent->item->object.flags & PARENT1)\n+\t\t\t\tcontinue;\n+\n+\t\t\tparent->item->object.flags |= PARENT1;\n+\t\t\tnum_commits_not_in_graph++;\n+\n+\t\t\tif (num_commits_not_in_graph >= limit_commits_not_in_graph) {\n+\t\t\t\tresult = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tcommit_list_append(parent->item, &stack);\n+\t\t}\n+\t}\n+\n+\tfree_commit_list(stack);\n+\treturn result;\n+}\n+\n+static int should_write_commit_graph(void)\n+{\n+\tint result;\n+\n+\tgit_config_get_int(\"maintenance.commit-graph.auto\",\n+\t\t\t &limit_commits_not_in_graph);\n+\n+\tif (!limit_commits_not_in_graph)\n+\t\treturn 0;\n+\tif (limit_commits_not_in_graph < 0)\n+\t\treturn 1;\n+\n+\tresult = for_each_ref(dfs_on_ref, NULL);\n+\n+\tclear_commit_marks_all(PARENT1);\n+\n+\treturn result;\n+}\n+\n static int run_write_commit_graph(void)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n@@ -1123,6 +1198,7 @@ static struct maintenance_task tasks[] = {\n \t[TASK_COMMIT_GRAPH] = {\n \t\t\"commit-graph\",\n \t\tmaintenance_task_commit_graph,\n+\t\tshould_write_commit_graph,\n \t},\n };\n \ndiff --git a/object.h b/object.h\nindex 96a2105859..30d4e0f6a0 100644\n--- a/object.h\n+++ b/object.h\n@@ -74,6 +74,7 @@ struct object_array {\n * list-objects-filter.c: 21\n * builtin/fsck.c: 0--3\n * builtin/index-pack.c: 2021\n+ * builtin/maintenance.c: 16\n * builtin/pack-objects.c: 20\n * builtin/reflog.c: 10--12\n * builtin/show-branch.c: 0-------------------------------------------26\n-- \ngitgitgadget\n\n"},{"id":"402504","messageId":"xmqq4kpo7gda.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-30T23:06:09Z","receivedAt":"2020-07-30T23:06:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> * I added and updates Junio's patch from jc/no-update-fetch-head into the\n> proper place before the 'prefetch' task. The 'prefetch' task now uses\n> --no-write-fetch-head to avoid issues with FETCH_HEAD.\n\nThe update to tests for \"git pull\" looks reasonable, too.\n\nThanks.\n"},{"id":"402505","messageId":"xmqqzh7g60mk.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-30T23:31:31Z","receivedAt":"2020-07-30T23:31:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> * This series is now based on jk/strvec, as there are several places where\n> I was adding new callers to argv_array_push* and run_command_v_opt()\n> which have been replaced with strvec_push*() and run_command(), using a\n> 'struct child_process'.\n\nIt probably has been a good way to gain experience with the renamed\nAPI for you to rebase this series, but I think we'd have to spend a\nsimilar amount of effort yet again, as I expect that the jk/strvec\nseries would be rerolled at least once more, with the items[] array\nrenamed to something else, like v[].\n\nLet's see how it goes.\n\nThanks.\n\n"},{"id":"402506","messageId":"CAPx1GvcpOC2jeeB=CNR5cGh=q59j6J09J2+vRz4+8aH1_cQocQ@mail.gmail.com","threadId":"53809","inReplyTo":"ef2a2319565939bfbdbd2e898a980c61ecc39c6a.1596147867.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 13/20] maintenance: auto-size incremental-repack batch","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2020-07-30T23:36:49Z","receivedAt":"2020-07-30T23:37:03Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Thu, Jul 30, 2020 at 3:26 PM Derrick Stolee via GitGitGadget\n wrote:\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 99ab1f5e9d..d94eb3e6ad 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -988,6 +988,46 @@ static int multi_pack_index_expire(void)\n> return 0;\n> }\n>\n> +#define TWO_GIGABYTES (0x7FFF)\n\nYou meant (0x7FFFFFFF) here, right?\n\nChris\n"},{"id":"402509","messageId":"20200731003022.GA1029866@google.com","threadId":"53809","inReplyTo":"5cbdb559-3897-961f-4dd3-0bab11848c5b@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-07-31T00:30:22Z","receivedAt":"2020-07-31T00:30:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n\n>> [jrnieder] How do I supply the tasks on the command line? Are they\n>> parameters to this subcommand? If so, it could make sense for this to\n>> say something like\n>>\n>> \trun ...::\n>\n> Hopefully this is documented to your satisfaction when the ability\n> to customize the tasks is implemented.\n\nHm, do you mean that this change is too difficult to squash into the\nfirst patch?\n\n>> What is the exit code convention for \"git maintenance run\"? (Many Git\n>> commands don't document this, but since we're making a new command\n>> it seems worth building the habit.)\n>\n> Is this worth doing? Do we really want every command to document\n> that \"0 means success, everything else means failure, and some of\n> those exit codes mean a specific kind of failure that is global to\n> Git\"?\n\nI mean things like \"what exit code does it run if there is no\nmaintenance to do?\"\n\n[...]\n>>> +static struct maintenance_opts {\n>>> +\tint auto_flag;\n>>> +} opts;\n>>\n>> Packing this in a struct feels a bit unusual. Is the struct going to\n>> be passed somewhere, or could these be individual locals in\n>> cmd_maintenance?\n>\n> This will grow, and I'd rather have one global struct than many\n> individual global items. It makes it clearer when I use\n> \"opts.auto_flag\" that this corresponds to whether \"--auto\" was\n> provided as a command-line option.\n\nThat doesn't seem idiomatic within the Git codebase.\n\nCan they be locals instead?\n\n[...]\n>> Perhaps this is suggesting that we need some central test helper for\n>> parsing traces so we can do this reliably in one place. What do you\n>> think?\n>\n> I'm specifically using GIT_TRACE2_EVENT, which is intended for\n> consumption by computer, not humans. Adding whitespace or otherwise\n> changing the output format would be an unnecessary disruption that\n> is more likely to have downstream effects with external tools.\n>\n> In some way, adding extra dependencies like this in our own test\n> suite can help ensure the output format is more stable.\n\nHm, I thought the contract for GIT_TRACE2_EVENT is that it is JSON,\nwhich is a well defined format. To that end, I would sincerely hope\nthat external tools are not assuming anything beyond that the format\nis JSON.\n\nIf that wasn't the intent, I'd rather that we not use a JSON-like\nformat at all. That would make the expectations for evolving this\ninterface clearer.\n\n> If there is a better way to ask \"Did my command call 'git gc' (with\n> no arguments|with these arguments)?\" then I'm happy to consider it.\n\nMy proposal was just to factor this out into a function in\ntest-lib-functions.sh so it's easy to evolve over time in one place.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"402521","messageId":"xmqqlfj04ch6.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"xmqqzh7g60mk.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-31T02:58:29Z","receivedAt":"2020-07-31T02:58:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> * This series is now based on jk/strvec, as there are several places where\n>> I was adding new callers to argv_array_push* and run_command_v_opt()\n>> which have been replaced with strvec_push*() and run_command(), using a\n>> 'struct child_process'.\n>\n> It probably has been a good way to gain experience with the renamed\n> API for you to rebase this series, but I think we'd have to spend a\n> similar amount of effort yet again, as I expect that the jk/strvec\n> series would be rerolled at least once more, with the items[] array\n> renamed to something else, like v[].\n>\n> Let's see how it goes.\n\nSo, ... jk/strvec got updated, and just applying these twenty\npatches on top no longer compiles, so I rebased them ;-)\n\nI also adjusted the merge-fix for the 6 patches on es/config-hooks\ntopic before making tonight's final push-out of 'seen'.\n\nThanks.\n"},{"id":"402552","messageId":"20200731164001.GA1252647@google.com","threadId":"53809","inReplyTo":"5cbdb559-3897-961f-4dd3-0bab11848c5b@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-07-31T16:40:01Z","receivedAt":"2020-07-31T16:40:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n> > Derrick Stolee wrote:\n\n>>> --- /dev/null\n>>> +++ b/Documentation/git-maintenance.txt\n>>> @@ -0,0 +1,57 @@\n>> [...]\n>>> +SUBCOMMANDS\n>>> +-----------\n>>> +\n>>> +run::\n>>> +\tRun one or more maintenance tasks.\n>>\n>> [jrnieder] How do I supply the tasks on the command line? Are they\n>> parameters to this subcommand? If so, it could make sense for this to\n>> say something like\n>>\n>> \trun ...::\n>\n> Hopefully this is documented to your satisfaction when the ability\n> to customize the tasks is implemented.\n\nI think my last reply to this part missed the point a little, so let\nme try again.\n\nIn multi-patch series, requests to make improvements to early parts\nthat would be obsoleted by later parts may seem strange --- isn't this\njust wasted work? Sure, occasionally someone will stumble on the\nearly part using \"git bisect\" or \"git log\", but isn't that rare enough\nto not be worth worrying too much about?\n\nAnd I'd agree with that reaction --- for a documentation nit like this\none, that benefit alone is probably not enough to be worth the fuss of\ntweaking things.\n\nWhen I focus more on early parts of a series, it is for a different\nreason: from a release management point of view, it is useful to get\nthe early part of a series \"cooked\" as soon as possible so it can be\nexposed to users in \"next\" and then become a foundation for later\npatches in \"master\". So we try to get as much of the early part of a\nseries into that state as we can, and to merge the early part even\nwithout the later part as soon as the former is ready and valuable\nenough to users on its own.\n\nRelatedly, ensuring each commit is \"complete\" in this sense on its own\ngives us more flexibility in case a problem is discovered later: we\ncan revert a patch that is causing issues without having to revert the\npreceding part.\n\nI think that might be part of the context I didn't convey well, and\nwhy I want to get this in tip-top shape and start using it.\n\nThanks,\nJonathan\n"},{"id":"402704","messageId":"a176ddf5-b45b-fb25-8740-96efbd324edf@gmail.com","threadId":"53809","inReplyTo":"20200731003022.GA1029866@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-03T17:37:14Z","receivedAt":"2020-08-03T17:37:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n> Derrick Stolee wrote:\n>> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n> \n>>> [jrnieder] How do I supply the tasks on the command line? Are they\n>>> parameters to this subcommand? If so, it could make sense for this to\n>>> say something like\n>>>\n>>> \trun ...::\n>>\n>> Hopefully this is documented to your satisfaction when the ability\n>> to customize the tasks is implemented.\n> \n> Hm, do you mean that this change is too difficult to squash into the\n> first patch?\n\nI mean that there is only one task right now. Until the commit-graph\ntask is implemented, there is no need to have a --task= option.\n\nCreating a --task= option in this patch would unnecessarily bloat\nthe patch by implementing multiple tasks while filling in the boilerplate\nof creating a new builtin.\n\n>>> What is the exit code convention for \"git maintenance run\"? (Many Git\n>>> commands don't document this, but since we're making a new command\n>>> it seems worth building the habit.)\n>>\n>> Is this worth doing? Do we really want every command to document\n>> that \"0 means success, everything else means failure, and some of\n>> those exit codes mean a specific kind of failure that is global to\n>> Git\"?\n> \n> I mean things like \"what exit code does it run if there is no\n> maintenance to do?\"\n> \n> [...]\n>>>> +static struct maintenance_opts {\n>>>> +\tint auto_flag;\n>>>> +} opts;\n>>>\n>>> Packing this in a struct feels a bit unusual. Is the struct going to\n>>> be passed somewhere, or could these be individual locals in\n>>> cmd_maintenance?\n>>\n>> This will grow, and I'd rather have one global struct than many\n>> individual global items. It makes it clearer when I use\n>> \"opts.auto_flag\" that this corresponds to whether \"--auto\" was\n>> provided as a command-line option.\n> \n> That doesn't seem idiomatic within the Git codebase.\n>\n> Can they be locals instead?\n\nWhich part do you want to be idiomatic? The fact that the parse-options\nlibrary typically refers to static global values or the fact that the\ndata is not organized in a struct?\n\nTurning these into locals is _even worse_ than the current model of\nstatic globals because that then requires updating the prototypes of\nall the (static) local methods that operate on the arguments. Every\ntime a new option is added, that would cascade in method prototypes\nall over again.\n\nThe natural thing to do to \"fix\" that situation is to create a struct\nthat holds the information from the parsed command-line arguments. But\nthen why is it a local parameter that is passed through all of the\nlocal methods instead of a global (as presented here in this patch)?\n\n> [...]\n>>> Perhaps this is suggesting that we need some central test helper for\n>>> parsing traces so we can do this reliably in one place. What do you\n>>> think?\n>>\n>> I'm specifically using GIT_TRACE2_EVENT, which is intended for\n>> consumption by computer, not humans. Adding whitespace or otherwise\n>> changing the output format would be an unnecessary disruption that\n>> is more likely to have downstream effects with external tools.\n>>\n>> In some way, adding extra dependencies like this in our own test\n>> suite can help ensure the output format is more stable.\n> \n> Hm, I thought the contract for GIT_TRACE2_EVENT is that it is JSON,\n> which is a well defined format. To that end, I would sincerely hope\n> that external tools are not assuming anything beyond that the format\n> is JSON.\n> \n> If that wasn't the intent, I'd rather that we not use a JSON-like\n> format at all. That would make the expectations for evolving this\n> interface clearer.\n> \n>> If there is a better way to ask \"Did my command call 'git gc' (with\n>> no arguments|with these arguments)?\" then I'm happy to consider it.\n> \n> My proposal was just to factor this out into a function in\n> test-lib-functions.sh so it's easy to evolve over time in one place.\n\nThis is a valuable suggestion, but this series is already too large\nto make such a change in addition to the patches already here. I'd\nconsider this more if there were more tests that checked \"did we\nrun this subcommand with these arguments?\" Better to extract a helper\nwhen this is actually used in multiple places.\n\nThanks,\n-Stolee\n"},{"id":"402706","messageId":"aead1ca7-0477-f46a-e2a6-f5b98c774d20@gmail.com","threadId":"53809","inReplyTo":"CAPx1GvcpOC2jeeB=CNR5cGh=q59j6J09J2+vRz4+8aH1_cQocQ@mail.gmail.com","subject":"Re: [PATCH v3 13/20] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-03T17:43:09Z","receivedAt":"2020-08-03T17:43:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 7:36 PM, Chris Torek wrote:\n> On Thu, Jul 30, 2020 at 3:26 PM Derrick Stolee via GitGitGadget\n> wrote:\n>> diff --git a/builtin/gc.c b/builtin/gc.c\n>> index 99ab1f5e9d..d94eb3e6ad 100644\n>> --- a/builtin/gc.c\n>> +++ b/builtin/gc.c\n>> @@ -988,6 +988,46 @@ static int multi_pack_index_expire(void)\n>> return 0;\n>> }\n>>\n>> +#define TWO_GIGABYTES (0x7FFF)\n> \n> You meant (0x7FFFFFFF) here, right?\n\nUgh. Of course. Thanks for being diligent here. I hadn't pulled\nthis branch into microsoft/git for testing with Scalar yet, where\nI would have noticed this.\n\nI'm tempted to create an EXPENSIVE test that actually checks this\ncondition. Let's see what I can do.\n\nThanks,\n-Stolee\n\n"},{"id":"402707","messageId":"20200803174654.GA2473576@google.com","threadId":"53809","inReplyTo":"a176ddf5-b45b-fb25-8740-96efbd324edf@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-03T17:46:54Z","receivedAt":"2020-08-03T17:46:58Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n> On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n>> Derrick Stolee wrote:\n>>> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n\n>>>> [jrnieder] How do I supply the tasks on the command line? Are they\n>>>> parameters to this subcommand? If so, it could make sense for this to\n>>>> say something like\n>>>>\n>>>> \trun ...::\n>>>\n>>> Hopefully this is documented to your satisfaction when the ability\n>>> to customize the tasks is implemented.\n[...]\n> I mean that there is only one task right now. Until the commit-graph\n> task is implemented, there is no need to have a --task= option.\n\nAh, that wasn't clear to me from the docs.\n\nYou're saying that \"git maintenance run\" runs the \"gc\" task? Can the\ndocumentation say so?\n\n[...]\n>>>>> +static struct maintenance_opts {\n>>>>> +\tint auto_flag;\n>>>>> +} opts;\n>>>>\n>>>> Packing this in a struct feels a bit unusual. Is the struct going to\n>>>> be passed somewhere, or could these be individual locals in\n>>>> cmd_maintenance?\n>>>\n>>> This will grow, and I'd rather have one global struct than many\n>>> individual global items. It makes it clearer when I use\n>>> \"opts.auto_flag\" that this corresponds to whether \"--auto\" was\n>>> provided as a command-line option.\n>>\n>> That doesn't seem idiomatic within the Git codebase.\n>>\n>> Can they be locals instead?\n>\n> Which part do you want to be idiomatic? The fact that the parse-options\n> library typically refers to static global values or the fact that the\n> data is not organized in a struct?\n\nparse-options has no requirement about the values being global. Some\nolder code does that, but newer code tends to use locals when\nappropriate.\n\nPutting it in a struct is fine as long as that struct is being passed\naround. What isn't idiomatic in Git is using a global struct for\nnamespacing.\n\n[...]\n> The natural thing to do to \"fix\" that situation is to create a struct\n> that holds the information from the parsed command-line arguments. But\n> then why is it a local parameter that is passed through all of the\n> local methods instead of a global (as presented here in this patch)?\n\nI'm having trouble parsing that last sentence. You're saying a global\nis preferable over passing around a pointer to a local \"opts\" struct?\n\n[...]\n>>> If there is a better way to ask \"Did my command call 'git gc' (with\n>>> no arguments|with these arguments)?\" then I'm happy to consider it.\n>>\n>> My proposal was just to factor this out into a function in\n>> test-lib-functions.sh so it's easy to evolve over time in one place.\n>\n> This is a valuable suggestion, but this series is already too large\n> to make such a change in addition to the patches already here.\n\nHm, it's not clear to me that this would make the series significantly\nlarger.\n\nAnd on the contrary, it would make the code less fragile. I think this\nis important.\n\nhttps://chromium.googlesource.com/chromium/src/+/master/docs/cl_respect.md#remember-communication-can-be-hard:\nif you'd like to meet out of band, let me know, and I'd be happy to go\ninto this further.\n\nThanks,\nJonathan\n"},{"id":"402763","messageId":"20200803224631.GA73022@syl.lan","threadId":"53809","inReplyTo":"20200803174654.GA2473576@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T22:46:31Z","receivedAt":"2020-08-03T22:46:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 03, 2020 at 10:46:54AM -0700, Jonathan Nieder wrote:\n> Derrick Stolee wrote:\n> > On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n> >> Derrick Stolee wrote:\n> >>> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n>\n> >>>> [jrnieder] How do I supply the tasks on the command line? Are they\n> >>>> parameters to this subcommand? If so, it could make sense for this to\n> >>>> say something like\n> >>>>\n> >>>> \trun ...::\n> >>>\n> >>> Hopefully this is documented to your satisfaction when the ability\n> >>> to customize the tasks is implemented.\n> [...]\n> > I mean that there is only one task right now. Until the commit-graph\n> > task is implemented, there is no need to have a --task= option.\n>\n> Ah, that wasn't clear to me from the docs.\n>\n> You're saying that \"git maintenance run\" runs the \"gc\" task? Can the\n> documentation say so?\n>\n> [...]\n> >>>>> +static struct maintenance_opts {\n> >>>>> +\tint auto_flag;\n> >>>>> +} opts;\n> >>>>\n> >>>> Packing this in a struct feels a bit unusual. Is the struct going to\n> >>>> be passed somewhere, or could these be individual locals in\n> >>>> cmd_maintenance?\n> >>>\n> >>> This will grow, and I'd rather have one global struct than many\n> >>> individual global items. It makes it clearer when I use\n> >>> \"opts.auto_flag\" that this corresponds to whether \"--auto\" was\n> >>> provided as a command-line option.\n> >>\n> >> That doesn't seem idiomatic within the Git codebase.\n> >>\n> >> Can they be locals instead?\n> >\n> > Which part do you want to be idiomatic? The fact that the parse-options\n> > library typically refers to static global values or the fact that the\n> > data is not organized in a struct?\n>\n> parse-options has no requirement about the values being global. Some\n> older code does that, but newer code tends to use locals when\n> appropriate.\n>\n> Putting it in a struct is fine as long as that struct is being passed\n> around. What isn't idiomatic in Git is using a global struct for\n> namespacing.\n>\n> [...]\n> > The natural thing to do to \"fix\" that situation is to create a struct\n> > that holds the information from the parsed command-line arguments. But\n> > then why is it a local parameter that is passed through all of the\n> > local methods instead of a global (as presented here in this patch)?\n>\n> I'm having trouble parsing that last sentence. You're saying a global\n> is preferable over passing around a pointer to a local \"opts\" struct?\n>\n> [...]\n> >>> If there is a better way to ask \"Did my command call 'git gc' (with\n> >>> no arguments|with these arguments)?\" then I'm happy to consider it.\n> >>\n> >> My proposal was just to factor this out into a function in\n> >> test-lib-functions.sh so it's easy to evolve over time in one place.\n> >\n> > This is a valuable suggestion, but this series is already too large\n> > to make such a change in addition to the patches already here.\n>\n> Hm, it's not clear to me that this would make the series significantly\n> larger.\n\nI think what Stolee is trying to say is less about a change that would\nmake the series larger, it's about changing an already-large series.\n\n> And on the contrary, it would make the code less fragile. I think this\n> is important.\n\nI'm not sure that I see your argument. What we are really discussing is\nwhether or not we should have a static struct outside of 'cmd_gc()', or\na zero-initialized frame local struct within 'cmd_gc()'. I fully\nunderstand the arguments in favor of one or the other, but I struggle to\ngrasp that this is worth our time to debate in this much detail.\n\n> https://chromium.googlesource.com/chromium/src/+/master/docs/cl_respect.md#remember-communication-can-be-hard:\n> if you'd like to meet out of band, let me know, and I'd be happy to go\n> into this further.\n\nI assume that the part of this document you were linking to was about\nmeeting out-of-band to discuss an issue instead of emailing back and\nforth.\n\nI appreciate your willingness to resolve this expediently, but I would\nmuch prefer if these discussions (that is, the ones about the technical\ncontent of this series) happened on-list for the benefit of other\nreviewers.\n\n> Thanks,\n> Jonathan\n\nThanks,\nTaylor\n"},{"id":"402764","messageId":"20200803230134.GA58587@google.com","threadId":"53809","inReplyTo":"20200803224631.GA73022@syl.lan","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-03T23:01:34Z","receivedAt":"2020-08-03T23:01:44Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Taylor Blau wrote:\n> On Mon, Aug 03, 2020 at 10:46:54AM -0700, Jonathan Nieder wrote:\n>> Derrick Stolee wrote:\n>>> On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n>>>> Derrick Stolee wrote:\n\n>>>>> If there is a better way to ask \"Did my command call 'git gc' (with\n>>>>> no arguments|with these arguments)?\" then I'm happy to consider it.\n>>>>\n>>>> My proposal was just to factor this out into a function in\n>>>> test-lib-functions.sh so it's easy to evolve over time in one place.\n>>>\n>>> This is a valuable suggestion, but this series is already too large\n>>> to make such a change in addition to the patches already here.\n>>\n>> Hm, it's not clear to me that this would make the series significantly\n>> larger.\n>\n> I think what Stolee is trying to say is less about a change that would\n> make the series larger, it's about changing an already-large series.\n>\n>> And on the contrary, it would make the code less fragile. I think this\n>> is important.\n>\n> I'm not sure that I see your argument. What we are really discussing is\n> whether or not we should have a static struct outside of 'cmd_gc()', or\n> a zero-initialized frame local struct within 'cmd_gc()'. I fully\n> understand the arguments in favor of one or the other, but I struggle to\n> grasp that this is worth our time to debate in this much detail.\n\nSorry for the lack of clarity. The proposal Stolee is pushing back on\nabove is to have a helper in test-lib-functions.sh that tells a test\nwhether an event it cares about happened in traces (in this example,\ninvocation of \"git gc\").\n\nI consider it important because if we are getting into the habit of\nhaving our tests assume the current exact byte-for-byte trace JSON\noutput, then that is an assumption that is going to be painful to\nunravel. Factoring out a helper would also make the test easier to\nread, but that is less important than \"an ounce of prevention is worth\na pound of cure\" to me in this example.\n\nJonathan\n"},{"id":"402765","messageId":"20200803230814.GA73765@syl.lan","threadId":"53809","inReplyTo":"20200803230134.GA58587@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T23:08:14Z","receivedAt":"2020-08-03T23:08:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 03, 2020 at 04:01:34PM -0700, Jonathan Nieder wrote:\n> Taylor Blau wrote:\n> > On Mon, Aug 03, 2020 at 10:46:54AM -0700, Jonathan Nieder wrote:\n> >> Derrick Stolee wrote:\n> >>> On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n> >>>> Derrick Stolee wrote:\n>\n> >>>>> If there is a better way to ask \"Did my command call 'git gc' (with\n> >>>>> no arguments|with these arguments)?\" then I'm happy to consider it.\n> >>>>\n> >>>> My proposal was just to factor this out into a function in\n> >>>> test-lib-functions.sh so it's easy to evolve over time in one place.\n> >>>\n> >>> This is a valuable suggestion, but this series is already too large\n> >>> to make such a change in addition to the patches already here.\n> >>\n> >> Hm, it's not clear to me that this would make the series significantly\n> >> larger.\n> >\n> > I think what Stolee is trying to say is less about a change that would\n> > make the series larger, it's about changing an already-large series.\n> >\n> >> And on the contrary, it would make the code less fragile. I think this\n> >> is important.\n> >\n> > I'm not sure that I see your argument. What we are really discussing is\n> > whether or not we should have a static struct outside of 'cmd_gc()', or\n> > a zero-initialized frame local struct within 'cmd_gc()'. I fully\n> > understand the arguments in favor of one or the other, but I struggle to\n> > grasp that this is worth our time to debate in this much detail.\n>\n> Sorry for the lack of clarity. The proposal Stolee is pushing back on\n> above is to have a helper in test-lib-functions.sh that tells a test\n> whether an event it cares about happened in traces (in this example,\n> invocation of \"git gc\").\n\nServes me right for reading too quickly ;).\n\n> I consider it important because if we are getting into the habit of\n> having our tests assume the current exact byte-for-byte trace JSON\n> output, then that is an assumption that is going to be painful to\n> unravel. Factoring out a helper would also make the test easier to\n> read, but that is less important than \"an ounce of prevention is worth\n> a pound of cure\" to me in this example.\n\nI still only partially buy into this, though. I had to deal with some\nrather fragile grepping through trace2 JSON in t4216 (? or something,\nthe log-bloom tests) recently, and found it a little fragile, but not\noverly so.\n\nI'd rather just move forward, but I do feel strongly that we discuss\nthese matters on the list.\n\n> Jonathan\n\nThanks,\nTaylor\n"},{"id":"402766","messageId":"20200803231745.GB58587@google.com","threadId":"53809","inReplyTo":"20200803230814.GA73765@syl.lan","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-03T23:17:45Z","receivedAt":"2020-08-03T23:17:49Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Taylor Blau wrote:\n\n> I still only partially buy into this, though. I had to deal with some\n> rather fragile grepping through trace2 JSON in t4216 (? or something,\n> the log-bloom tests) recently, and found it a little fragile, but not\n> overly so.\n>\n> I'd rather just move forward,\n\nThis means we're promising to never change the JSON serialization, or\nthat we're willing to pay with time in the future to update all tests\nthat assume the current JSON serialization.\n\nI do not want to make that promise. I've lived through the same work\nof updating tests that assumed particular sha1s and having to change\nthem to be more generic and do not want to have to do that again.\n\nIf you're saying that you are willing to do that work when the time\ncomes, then that counts for a lot. If you're saying that we should\nnot change the JSON serialization, then it makes me worry that we made\na mistake in choosing JSON as a format, since using a well defined\nformat with an ecosystem of serialization + deserialization libraries\nwas the whole point.\n\nJonathan\n"},{"id":"402768","messageId":"20200803235211.GC58587@google.com","threadId":"53809","inReplyTo":"5cbdb559-3897-961f-4dd3-0bab11848c5b@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-03T23:52:11Z","receivedAt":"2020-08-03T23:52:16Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n\n>> Perhaps this is suggesting that we need some central test helper for\n>> parsing traces so we can do this reliably in one place. What do you\n>> think?\n>\n> I'm specifically using GIT_TRACE2_EVENT, which is intended for\n> consumption by computer, not humans. Adding whitespace or otherwise\n> changing the output format would be an unnecessary disruption that\n> is more likely to have downstream effects with external tools.\n>\n> In some way, adding extra dependencies like this in our own test\n> suite can help ensure the output format is more stable.\n\nI've discussed this a bit more at [1]: I do think this is output that\nwe cannot promise to keep stable. Can you say a little more about why\nextracting a helper would be problematic? For example, would doing\nthat in a separate patch at the end of the series be workable for you?\n\nMore generally, one theme I'm sensing in the responses to the code\nreview comments I sent is that the series may have gotten too long to\nbe able to easily amend the beginning of it. To that end, I'm happy\nto make a reroll or make this easier in any other way I can. This\nalso suggests to me that it may make sense to split off an early part\nof the series and get that into a shape you're comfortable with before\nrebasing the rest of the series on top as a separate topic (a tried\nand true method, used recently for the \"git bugreport\" command, for\nexample).\n\nThoughts?\n\nThanks,\nJonathan\n\n[1] https://lore.kernel.org/git/20200803231745.GB58587@google.com/\n"},{"id":"402770","messageId":"xmqqpn87tgsw.fsf@gitster.c.googlers.com","threadId":"53809","inReplyTo":"20200803231745.GB58587@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-04T00:07:11Z","receivedAt":"2020-08-04T00:07:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> ... If you're saying that we should\n> not change the JSON serialization, then it makes me worry that we made\n> a mistake in choosing JSON as a format, since using a well defined\n> format with an ecosystem of serialization + deserialization libraries\n> was the whole point.\n\nWell said.\n"},{"id":"402813","messageId":"aac90dbd-e885-f366-1056-0824b8b8b8fe@gmail.com","threadId":"53809","inReplyTo":"20200803231745.GB58587@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T13:32:33Z","receivedAt":"2020-08-04T13:32:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2020 7:17 PM, Jonathan Nieder wrote:\n> Taylor Blau wrote:\n> \n>> I still only partially buy into this, though. I had to deal with some\n>> rather fragile grepping through trace2 JSON in t4216 (? or something,\n>> the log-bloom tests) recently, and found it a little fragile, but not\n>> overly so.\n>>\n>> I'd rather just move forward,\n> \n> This means we're promising to never change the JSON serialization, or\n> that we're willing to pay with time in the future to update all tests\n> that assume the current JSON serialization.\n> \n> I do not want to make that promise. I've lived through the same work\n> of updating tests that assumed particular sha1s and having to change\n> them to be more generic and do not want to have to do that again.\n> \n> If you're saying that you are willing to do that work when the time\n> comes, then that counts for a lot. If you're saying that we should\n> not change the JSON serialization, then it makes me worry that we made\n> a mistake in choosing JSON as a format, since using a well defined\n> format with an ecosystem of serialization + deserialization libraries\n> was the whole point.\n\nThe benefit of JSON is that it is simple to be _additive_. We can\nalways jam more data into the format.\n\nWhat I'm doing is making the following assumptions:\n\n1. There will always be an event line corresponding to launching a\n subcommand.\n\n2. This line will contain the list of arguments used for that\n subcommand in a JSON array.\n\n3. We don't add extra whitespace in that JSON array.\n\nThe first two items are important promises made by the trace2 event\noutput. By depending on them, I'm only adding more places where we\nexpect the JSON *schema* to remain consistent. Rather, I only require\nthat the JSON schema does not _remove_ that data which is expected\nby tools that are reading and reporting that data.\n\nOnly the third item is something that adds an additional restriction\non the JSON format. But also: why would we ever add extraneous\nwhitespace characters to the output intended for computer parsing?\nThis concern raises a giant \"YAGNI\" warning in my head.\n\nWhat is seems like you are asking instead is for me to create a tool\nin the test suite that parses each JSON line, extracts a specific\nmember from that JSON object, reconstructs a command-line invocation\nfrom the JSON array, and reports whether that process worked for any\nline in the event output.\n\nThat seems overly complicated and puts even more requirements on the\nJSON schema than I have placed in my current implementation.\n\nIf this is to truly be a hard requirement for these tests to move\nforward, then I would rather do an alternate approach that would be\na better test mechanism than scraping trace data: mock the subcommands.\n\nI would create a new directory, \"t/mock\", and create a new 'git'\nexecutable there. By setting --exec-path in our test commands, we\nwould launch this mocked version of Git instead of our built version.\nThe mock could report each set of command-line arguments to a known\nfile. With some care, we could even provide an instructions file to\ntell it which error code to return or whether it should write data\nto stdout. This would allow testing things like the \"rewrite after\nverify fails\" behavior already in the maintenance builtin.\n\nIf I'm to spend time engineering something more complicated just to\ncheck \"did this subcommand run with these arguments?\" then I'd\nrather do it this way.\n\nThanks,\n-Stolee\n"},{"id":"402815","messageId":"20200804144208.GA227292@google.com","threadId":"53809","inReplyTo":"aac90dbd-e885-f366-1056-0824b8b8b8fe@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-04T14:42:08Z","receivedAt":"2020-08-04T14:42:18Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n\n> What is seems like you are asking instead is for me to create a tool\n> in the test suite that parses each JSON line, extracts a specific\n> member from that JSON object, reconstructs a command-line invocation\n> from the JSON array, and reports whether that process worked for any\n> line in the event output.\n\nNo, that isn't what I'm asking.\n\nI'm asking for this patch series to take the existing \"grep\" lines\nand put them in a function in test-lib-functions.sh, so that we can\nchange them in one place when the trace emitter changes.\n\n[...]\n> If this is to truly be a hard requirement for these tests to move\n> forward,\n\nYes, from my point of view it really is.\n\nBut that \"is this truly a hard requirement?\" comes up tells me I have\nnot done a good job of communicating in this review. A review is\nabout participants in the project working together to improve a patch,\nnot people making demands at each other.\n\n[...]\n> If I'm to spend time engineering something more complicated just to\n> check \"did this subcommand run with these arguments?\" then\n\nI don't see why this is more complicated than what is in patch 1. In\nfact, I think it would be a little more simple.\n\nJonathan\n"},{"id":"402821","messageId":"2b6da345-d004-1d83-78cd-3f1ceaf278d9@gmail.com","threadId":"53809","inReplyTo":"20200804144208.GA227292@google.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T16:32:56Z","receivedAt":"2020-08-04T16:33:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/4/2020 10:42 AM, Jonathan Nieder wrote:\n> Derrick Stolee wrote:\n> \n>> What is seems like you are asking instead is for me to create a tool\n>> in the test suite that parses each JSON line, extracts a specific\n>> member from that JSON object, reconstructs a command-line invocation\n>> from the JSON array, and reports whether that process worked for any\n>> line in the event output.\n> \n> No, that isn't what I'm asking.\n> \n> I'm asking for this patch series to take the existing \"grep\" lines\n> and put them in a function in test-lib-functions.sh, so that we can\n> change them in one place when the trace emitter changes.\n\nThanks for clarifying, clearing up my misinterpretation of your\nrequest. I'm coming around to this idea.\n\n> [...]\n>> If this is to truly be a hard requirement for these tests to move\n>> forward,\n> \n> Yes, from my point of view it really is.\n> \n> But that \"is this truly a hard requirement?\" comes up tells me I have\n> not done a good job of communicating in this review. A review is\n> about participants in the project working together to improve a patch,\n> not people making demands at each other.\n\nIt's also about a balance. A patch author and a reviewer can have\ndisagreements, and it is important to discover exactly how hard each\nis holding to their opinion.\n\n> [...]\n>> If I'm to spend time engineering something more complicated just to\n>> check \"did this subcommand run with these arguments?\" then\n> \n> I don't see why this is more complicated than what is in patch 1. In\n> fact, I think it would be a little more simple.\n\nMy interpretation (parsing JSON) _was_ more complicated, hence my\nreservations. The helper still requires non-trivial script-fu (for\nsomeone like me who is bad at scripting languages) but centralizing\nthe grep has value.\n\nHere is my attempt so far:\n\ntrace2_subcommand_data () {\n\tcommand=\"$1\" &&\n\tcommas=$(echo $command | sed s/\\ /\\\",\\\"/g) &&\n\tprintf \"[\\\"$commas\\\"]\"\n}\n\ntest_subcommand_called () {\n\tdata=$(trace2_subcommand_data $1) &&\n\tgrep $data $2\n}\n\ntest_subcommand_not_called () {\n\t! test_subcommand_called $1 $2\n}\n\nwith callers looking like\n\n\ttest_subcommand_called \"gc\" run-no-auto.txt &&\n\ttest_subcommand_not_called \"gc --auto\" run-auto.txt &&\n\ttest_subcommand_called \"gc --quiet\" run-quiet.txt\n\nThanks,\n-Stolee\n"},{"id":"402824","messageId":"20200804170220.GB227292@google.com","threadId":"53809","inReplyTo":"2b6da345-d004-1d83-78cd-3f1ceaf278d9@gmail.com","subject":"Re: [PATCH v2 01/18] maintenance: create basic maintenance runner","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2020-08-04T17:02:20Z","receivedAt":"2020-08-04T17:03:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Derrick Stolee wrote:\n\n> Here is my attempt so far:\n[...]\n> \ttest_subcommand_called \"gc\" run-no-auto.txt &&\n> \ttest_subcommand_not_called \"gc --auto\" run-auto.txt &&\n> \ttest_subcommand_called \"gc --quiet\" run-quiet.txt\n\nI like it. Some tweaks:\n- can simplify by using 'printf' instead of sed\n- passing the trace in stdin\n- following the convention that helpers like test_i18ngrep use of\n accepting ! as a parameter to negate the exit code\n\nThat would make (untested)\n\n# Check that the given command was invoked as part of the\n# trace2-format trace on stdin.\n#\n#\ttest_subcommand_called [!] ... < \n#\n# For example, to look for an invocation of \"git upload-pack\n# /path/to/repo\"\n#\n#\tGIT_TRACE2_EVENT=event.log git fetch ... &&\n#\ttest_subcommand_called git upload-pack \"$PATH\" Derrick Stolee wrote:\n> \n>> Here is my attempt so far:\n> [...]\n>> \ttest_subcommand_called \"gc\" run-no-auto.txt &&\n>> \ttest_subcommand_not_called \"gc --auto\" run-auto.txt &&\n>> \ttest_subcommand_called \"gc --quiet\" run-quiet.txt\n> \n> I like it. Some tweaks:\n> - can simplify by using 'printf' instead of sed\n> - passing the trace in stdin\n> - following the convention that helpers like test_i18ngrep use of\n> accepting ! as a parameter to negate the exit code\n> \n> That would make (untested)\n> \n> # Check that the given command was invoked as part of the\n> # trace2-format trace on stdin.\n> #\n> #\ttest_subcommand_called [!] ... < \n> #\n> # For example, to look for an invocation of \"git upload-pack\n> # /path/to/repo\"\n> #\n> #\tGIT_TRACE2_EVENT=event.log git fetch ... &&\n> #\ttest_subcommand_called git upload-pack \"$PATH\" #\n> # If the first parameter passed is !, this instead checks that\n> # the given command was not called.\n> #\n> test_subcommand_called () {\n> \tlocal negate=\n> \tif test \"$1\" = \"!\"\n> \tthen\n> \t\tnegate=t\n> \t\tshift\n> \tfi\n> \n> \tlocal expr=$(printf '\"%s\",' \"$@\")\n> \texpr=\"${expr%,}\"\n> \n> \tif test -n \"$negate\"\n> \tthen\n> \t\t! grep \"\\[$expr\\]\"\n> \telse\n> \t\tgrep \"\\[$expr\\]\"\n> \tfi\n> }\n\nThis works! Thanks. With the ! usage, I think a rename to\n'test_subcommand' makes sense. Further, I was somehow wrong\nabout not including \"git\" in the beginning, so here is my\nexample use of this method;\n\n\ttest_subcommand git gc Derrick Stolee wrote:\n>> On 7/30/2020 8:30 PM, Jonathan Nieder wrote:\n>>> Derrick Stolee wrote:\n>>>> On 7/29/2020 6:19 PM, Jonathan Nieder wrote:\n\nRevisiting the first part of this thread, as it got lost in the discussion\nabout trace2 and JSON parsing...\n\n>>>>> [jrnieder] How do I supply the tasks on the command line? Are they\n>>>>> parameters to this subcommand? If so, it could make sense for this to\n>>>>> say something like\n>>>>>\n>>>>> \trun ...::\n>>>>\n>>>> Hopefully this is documented to your satisfaction when the ability\n>>>> to customize the tasks is implemented.\n> [...]\n>> I mean that there is only one task right now. Until the commit-graph\n>> task is implemented, there is no need to have a --task= option.\n> \n> Ah, that wasn't clear to me from the docs.\n> \n> You're saying that \"git maintenance run\" runs the \"gc\" task? Can the\n> documentation say so?\n\nIn this patch, the documentation says\n\n\tRun tasks to optimize Git repository data\n\nthe plural is a bit incorrect at this point, but not a huge deal.\n\nThe run subcommand says \"Run one or more maintenance tasks\" which again\nis a bit misleading since \"or more\" is not technically possible. This\nbecomes a bit obvious when we see that the task list only contains a 'gc'\ntask.\n\nSo no, the documentation in this patch doesn't say \"Run the 'gc' task\"\nbecause that could be gathered from context _and_ it will be incorrect\nby the time this patch is integrated as part of the series.\n\nSo I struggle with finding a way to improve the documentation in this\npatch to satisfy your concerns. Any change made here will just be un-done\nby a later patch in the same series.\n\nI'd rather have something vague now and improve the documentation\nincrementally along with the implementation.\n\nAnother point you made earlier, but seems to be dropped in this email:\nyou wanted the return code documented. This also seems like wasteful\ntext, because the best I can think of is\n\n This command will return 0 unless a maintenance task failed in an\n unrecoverable way, or other typical failure cases such as incorrect\n usage or the command being run outside of a Git repository.\n\nPlease recommend something better, because that description suffers from\nbeing too generic/obvious when talking about the maintenance task and\nbeing an obvious repetition of Git's existing error code behavior.\n\nPerhaps this would be easier if there was an established pattern of\ndocumenting the return code. So, I went digging.\n\nThe examples I could find are:\n\n 1. 'git cat-file -e'\n 2. 'git check-ref-format'\n 3. 'git commit-graph --object-dir='\n 4. 'git cvsimport'\n 5. 'git fast-import'\n 6. 'git filter-branch'\n 7. 'git grep --quiet'\n 8. 'git gui citool'\n 9. 'git ls-remote --exit-code'\n10. 'git merge-base --is-ancestor'\n11. 'git push --recurse-submodules'\n12. 'git reflog exists'\n13. 'git rev-parse --parseopt'\n14. 'git update-index --test-untracked-cache' \n\nMost of these are for specific options, and the exit code is documented\nwith relation to how that specific option may lead to a non-zero exit\ncode (1, 3, 7, 8, 9, 10, 11, 12, 13, 14).\n\nThe one that is most clearly part of the main description is for\nthe plumbing command 'git check-ref-format':\n\n\tChecks if a given 'refname' is acceptable, and exits with\n\ta non-zero status if it is not.\n\nThis seems appropriate for something intended for use as a helper\nin script, with a very simple behavior. The exit code is the only\nway it provides any feedback to the caller! This use of the exit\ncode as the only output is used by some other cases (1, 7, 9, 10,\n11, 12, 13, 14).\n\nOf course, we also have examples like this in git-cvsimport.txt:\n\n\tOtherwise, success is indicated the Unix way, i.e. by simply\n\texiting with a zero exit status.\n\nThis phrase should be _assumed_ as part of Git itself, instead of\nwritten in the documentation for specific commands.\n\nThe documentation for 'git fast-import' is more useful here:\n\n\tIf fast-import is supplied invalid input it will terminate\n\twith a non-zero exit status and create a crash report...\n\nBut, you can see that the exit code is not the focus, but the\ncrash report _is_ important to document.\n\n'git filter-branch' has a special exit code for the case that no\ncommits were found to update.\n\n'git gui citool' returns an error code if the window is closed\nbefore the command can complete. This is also not a core feature,\nso is not a good example.\n\nMy only conclusion from this investigation is that the exit code\nis documented when it is useful, but otherwise the standard Unix\npattern is assumed and not worth documenting.\n\nIf this is something we want to change, then I don't think it\nvaluable to have that discussion here. Instead, let's start that\ndiscussion elsewhere with a series that updates existing docs to\ncatch up to a new standard of documenting the exit code.\n\n[1] https://git-scm.com/docs/git-cat-file#Documentation/git-cat-file.txt--e\n[2] https://git-scm.com/docs/git-check-ref-format#_description\n[3] https://git-scm.com/docs/git-commit-graph#Documentation/git-commit-graph.txt---object-dir\n[4] https://git-scm.com/docs/git-cvsimport#_output\n[5] https://git-scm.com/docs/git-fast-import#_crash_reports\n[6] https://git-scm.com/docs/git-filter-branch#_exit_status\n[7] https://git-scm.com/docs/git-grep#Documentation/git-grep.txt---quiet\n[8] https://git-scm.com/docs/git-gui#Documentation/git-gui.txt-codegitguicitoolcode\n[9] https://git-scm.com/docs/git-ls-remote#Documentation/git-ls-remote.txt---exit-code\n[10] https://git-scm.com/docs/git-merge-base#Documentation/git-merge-base.txt---is-ancestor\n[11] https://git-scm.com/docs/git-push#Documentation/git-push.txt---recurse-submodulescheckon-demandonlyno\n[12] https://git-scm.com/docs/git-reflog\n[13] https://git-scm.com/docs/git-rev-parse#_parseopt\n[14] https://git-scm.com/docs/git-update-index#Documentation/git-update-index.txt---test-untracked-cache\n\n> [...]\n>>>>>> +static struct maintenance_opts {\n>>>>>> +\tint auto_flag;\n>>>>>> +} opts;\n>>>>>\n>>>>> Packing this in a struct feels a bit unusual. Is the struct going to\n>>>>> be passed somewhere, or could these be individual locals in\n>>>>> cmd_maintenance?\n>>>>\n>>>> This will grow, and I'd rather have one global struct than many\n>>>> individual global items. It makes it clearer when I use\n>>>> \"opts.auto_flag\" that this corresponds to whether \"--auto\" was\n>>>> provided as a command-line option.\n>>>\n>>> That doesn't seem idiomatic within the Git codebase.\n>>>\n>>> Can they be locals instead?\n>>\n>> Which part do you want to be idiomatic? The fact that the parse-options\n>> library typically refers to static global values or the fact that the\n>> data is not organized in a struct?\n> \n> parse-options has no requirement about the values being global. Some\n> older code does that, but newer code tends to use locals when\n> appropriate.\n\nYou're right. Global is not required. The struct needs to be 'static'\nor initialized for compilation.\n\nWhile trying to get rid of globals based on your recommendation, I\nsee that the callback functions in the option-parsing _do_ require\nglobal state. This is used in \"maintenance: add --task option\" with\nthe global tasks array, but it also uses a global count of \"how many\ntasks have been selected\" currently in the struct. Two ways to get\naround that issue include (1) making a global int, or (2) looping\nthrough the task list to count how many have been selected so far.\n\nNeither of these solutions is clean. (In the diff below, I chose\noption 2.)\n\n> Putting it in a struct is fine as long as that struct is being passed\n> around. What isn't idiomatic in Git is using a global struct for\n> namespacing.\n\nIn my opinion, using a global struct would be better than the existing\npattern of a bunch of global variables, but I get your point about\nprecedence. More on this below.\n\n> [...]\n>> The natural thing to do to \"fix\" that situation is to create a struct\n>> that holds the information from the parsed command-line arguments. But\n>> then why is it a local parameter that is passed through all of the\n>> local methods instead of a global (as presented here in this patch)?\n> \n> I'm having trouble parsing that last sentence. You're saying a global\n> is preferable over passing around a pointer to a local \"opts\" struct?\n\nUsing a \"singleton\" like the opts struct here, the code is definitely\nmuch simpler to read. There is only ever one instance, because the struct\ndefinition comes along with the instance declaration.\n\nBy making it a local instance, we then need to pass around the struct as\na parameter to all of the static methods in the builtin, even though there\nis only ever one instance of this struct per process.\n\nThe two points above are the ones that led me to use the global instance\nin the first place.\n\nTo get back to your point about being idiomatic, these builtins use a\nstruct to group options, then pass it around as a parameter:\n\n* cat-file (struct batch_options)\n* checkout (struct checkout_opts)\n* rebase (struct rebase_options)\n* tag (struct create_tag_options)\n* worktree (struct add_opts)\n\nUnfortunately, there _is_ an example of the behavior I was following in\nthis series: my own example in the sparse-checkout builtin. So, I'm just\nrepeating my own mistakes that were not challenged before.\n\nI updated my v4-in-progress branch with that change, and the tip-to-tip\ndiff is below. It required some non-trivial care around the --task=\ncallback (which can't see the local options struct) and other cases.\n\nThanks,\n-Stolee\n\n--- \n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 1f0c519d14..3e473a1293 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -709,11 +709,10 @@ static const char * const builtin_maintenance_usage[] = {\n \tNULL\n };\n \n-static struct maintenance_opts {\n+struct maintenance_opts {\n \tint auto_flag;\n \tint quiet;\n-\tint tasks_selected;\n-} opts;\n+};\n \n /* Remember to update object flag allocation in object.h */\n #define PARENT1\t\t(1u<<16)\n@@ -789,7 +788,7 @@ static int should_write_commit_graph(void)\n \treturn result;\n }\n \n-static int run_write_commit_graph(void)\n+static int run_write_commit_graph(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -797,13 +796,13 @@ static int run_write_commit_graph(void)\n \tstrvec_pushl(&child.args, \"commit-graph\", \"write\",\n \t\t \"--split\", \"--reachable\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \treturn !!run_command(&child);\n }\n \n-static int run_verify_commit_graph(void)\n+static int run_verify_commit_graph(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -811,24 +810,24 @@ static int run_verify_commit_graph(void)\n \tstrvec_pushl(&child.args, \"commit-graph\", \"verify\",\n \t\t \"--shallow\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_commit_graph(void)\n+static int maintenance_task_commit_graph(struct maintenance_opts *opts)\n {\n \tstruct repository *r = the_repository;\n \tchar *chain_path;\n \n \tclose_object_store(r->objects);\n-\tif (run_write_commit_graph()) {\n+\tif (run_write_commit_graph(opts)) {\n \t\terror(_(\"failed to write commit-graph\"));\n \t\treturn 1;\n \t}\n \n-\tif (!run_verify_commit_graph())\n+\tif (!run_verify_commit_graph(opts))\n \t\treturn 0;\n \n \twarning(_(\"commit-graph verify caught error, rewriting\"));\n@@ -840,14 +839,14 @@ static int maintenance_task_commit_graph(void)\n \t}\n \tfree(chain_path);\n \n-\tif (!run_write_commit_graph())\n+\tif (!run_write_commit_graph(opts))\n \t\treturn 0;\n \n \terror(_(\"failed to rewrite commit-graph\"));\n \treturn 1;\n }\n \n-static int fetch_remote(const char *remote)\n+static int fetch_remote(const char *remote, struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -855,7 +854,7 @@ static int fetch_remote(const char *remote)\n \tstrvec_pushl(&child.args, \"fetch\", remote, \"--prune\", \"--no-tags\",\n \t\t \"--no-write-fetch-head\", \"--refmap=\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n \tstrvec_pushf(&child.args, \"+refs/heads/*:refs/prefetch/%s/*\", remote);\n@@ -871,7 +870,7 @@ static int fill_each_remote(struct remote *remote, void *cbdata)\n \treturn 0;\n }\n \n-static int maintenance_task_prefetch(void)\n+static int maintenance_task_prefetch(struct maintenance_opts *opts)\n {\n \tint result = 0;\n \tstruct string_list_item *item;\n@@ -886,23 +885,23 @@ static int maintenance_task_prefetch(void)\n \tfor (item = remotes.items;\n \t item && item < remotes.items + remotes.nr;\n \t item++)\n-\t\tresult |= fetch_remote(item->string);\n+\t\tresult |= fetch_remote(item->string, opts);\n \n cleanup:\n \tstring_list_clear(&remotes, 0);\n \treturn result;\n }\n \n-static int maintenance_task_gc(void)\n+static int maintenance_task_gc(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_push(&child.args, \"gc\");\n \n-\tif (opts.auto_flag)\n+\tif (opts->auto_flag)\n \t\tstrvec_push(&child.args, \"--auto\");\n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \telse\n \t\tstrvec_push(&child.args, \"--no-quiet\");\n@@ -911,14 +910,14 @@ static int maintenance_task_gc(void)\n \treturn run_command(&child);\n }\n \n-static int prune_packed(void)\n+static int prune_packed(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_push(&child.args, \"prune-packed\");\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--quiet\");\n \n \treturn !!run_command(&child);\n@@ -977,7 +976,7 @@ static int write_loose_object_to_stdin(const struct object_id *oid,\n \treturn ++(d->count) > d->batch_size;\n }\n \n-static int pack_loose(void)\n+static int pack_loose(struct maintenance_opts *opts)\n {\n \tstruct repository *r = the_repository;\n \tint result = 0;\n@@ -996,7 +995,7 @@ static int pack_loose(void)\n \tpack_proc.git_cmd = 1;\n \n \tstrvec_push(&pack_proc.args, \"pack-objects\");\n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&pack_proc.args, \"--quiet\");\n \tstrvec_pushf(&pack_proc.args, \"%s/pack/loose\", r->objects->odb->path);\n \n@@ -1027,9 +1026,9 @@ static int pack_loose(void)\n \treturn result;\n }\n \n-static int maintenance_task_loose_objects(void)\n+static int maintenance_task_loose_objects(struct maintenance_opts *opts)\n {\n-\treturn prune_packed() || pack_loose();\n+\treturn prune_packed(opts) || pack_loose(opts);\n }\n \n static int incremental_repack_auto_condition(void)\n@@ -1061,14 +1060,14 @@ static int incremental_repack_auto_condition(void)\n \treturn count >= incremental_repack_auto_limit;\n }\n \n-static int multi_pack_index_write(void)\n+static int multi_pack_index_write(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"multi-pack-index\", \"write\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \tif (run_command(&child))\n@@ -1077,7 +1076,7 @@ static int multi_pack_index_write(void)\n \treturn 0;\n }\n \n-static int rewrite_multi_pack_index(void)\n+static int rewrite_multi_pack_index(struct maintenance_opts *opts)\n {\n \tstruct repository *r = the_repository;\n \tchar *midx_name = get_midx_filename(r->objects->odb->path);\n@@ -1085,17 +1084,18 @@ static int rewrite_multi_pack_index(void)\n \tunlink(midx_name);\n \tfree(midx_name);\n \n-\treturn multi_pack_index_write();\n+\treturn multi_pack_index_write(opts);\n }\n \n-static int multi_pack_index_verify(const char *message)\n+static int multi_pack_index_verify(struct maintenance_opts *opts,\n+\t\t\t\t const char *message)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"multi-pack-index\", \"verify\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \tif (run_command(&child)) {\n@@ -1106,14 +1106,14 @@ static int multi_pack_index_verify(const char *message)\n \treturn 0;\n }\n \n-static int multi_pack_index_expire(void)\n+static int multi_pack_index_expire(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"multi-pack-index\", \"expire\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \tclose_object_store(the_repository->objects);\n@@ -1164,14 +1164,14 @@ static off_t get_auto_pack_size(void)\n \treturn result_size;\n }\n \n-static int multi_pack_index_repack(void)\n+static int multi_pack_index_repack(struct maintenance_opts *opts)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n \tchild.git_cmd = 1;\n \tstrvec_pushl(&child.args, \"multi-pack-index\", \"repack\", NULL);\n \n-\tif (opts.quiet)\n+\tif (opts->quiet)\n \t\tstrvec_push(&child.args, \"--no-progress\");\n \n \tstrvec_pushf(&child.args, \"--batch-size=%\"PRIuMAX,\n@@ -1185,7 +1185,7 @@ static int multi_pack_index_repack(void)\n \treturn 0;\n }\n \n-static int maintenance_task_incremental_repack(void)\n+static int maintenance_task_incremental_repack(struct maintenance_opts *opts)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_multi_pack_index) {\n@@ -1193,22 +1193,22 @@ static int maintenance_task_incremental_repack(void)\n \t\treturn 0;\n \t}\n \n-\tif (multi_pack_index_write())\n+\tif (multi_pack_index_write(opts))\n \t\treturn 1;\n-\tif (multi_pack_index_verify(\"after initial write\"))\n-\t\treturn rewrite_multi_pack_index();\n-\tif (multi_pack_index_expire())\n+\tif (multi_pack_index_verify(opts, \"after initial write\"))\n+\t\treturn rewrite_multi_pack_index(opts);\n+\tif (multi_pack_index_expire(opts))\n \t\treturn 1;\n-\tif (multi_pack_index_verify(\"after expire step\"))\n-\t\treturn !!rewrite_multi_pack_index();\n-\tif (multi_pack_index_repack())\n+\tif (multi_pack_index_verify(opts, \"after expire step\"))\n+\t\treturn !!rewrite_multi_pack_index(opts);\n+\tif (multi_pack_index_repack(opts))\n \t\treturn 1;\n-\tif (multi_pack_index_verify(\"after repack step\"))\n-\t\treturn !!rewrite_multi_pack_index();\n+\tif (multi_pack_index_verify(opts, \"after repack step\"))\n+\t\treturn !!rewrite_multi_pack_index(opts);\n \treturn 0;\n }\n \n-typedef int maintenance_task_fn(void);\n+typedef int maintenance_task_fn(struct maintenance_opts *opts);\n \n /*\n * An auto condition function returns 1 if the task should run\n@@ -1275,9 +1275,9 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n-static int maintenance_run(void)\n+static int maintenance_run(struct maintenance_opts *opts)\n {\n-\tint i;\n+\tint i, found_selected = 0;\n \tint result = 0;\n \tstruct lock_file lk;\n \tstruct repository *r = the_repository;\n@@ -1291,7 +1291,7 @@ static int maintenance_run(void)\n \t\t * recursive process stack. Do not report an error in\n \t\t * that case.\n \t\t */\n-\t\tif (!opts.auto_flag && !opts.quiet)\n+\t\tif (!opts->auto_flag && !opts->quiet)\n \t\t\terror(_(\"lock file '%s' exists, skipping maintenance\"),\n \t\t\t lock_path);\n \t\tfree(lock_path);\n@@ -1299,23 +1299,26 @@ static int maintenance_run(void)\n \t}\n \tfree(lock_path);\n \n-\tif (opts.tasks_selected)\n+\tfor (i = 0; !found_selected && i < TASK__COUNT; i++)\n+\t\tfound_selected = tasks[i].selected;\n+\n+\tif (found_selected)\n \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n-\t\tif (opts.tasks_selected && !tasks[i].selected)\n+\t\tif (found_selected && !tasks[i].selected)\n \t\t\tcontinue;\n \n-\t\tif (!opts.tasks_selected && !tasks[i].enabled)\n+\t\tif (!found_selected && !tasks[i].enabled)\n \t\t\tcontinue;\n \n-\t\tif (opts.auto_flag &&\n+\t\tif (opts->auto_flag &&\n \t\t (!tasks[i].auto_condition ||\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n-\t\tif (tasks[i].fn()) {\n+\t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n \t\t}\n@@ -1349,17 +1352,15 @@ static void initialize_task_config(void)\n static int task_option_parse(const struct option *opt,\n \t\t\t const char *arg, int unset)\n {\n-\tint i;\n+\tint i, num_selected = 0;\n \tstruct maintenance_task *task = NULL;\n \n \tBUG_ON_OPT_NEG(unset);\n \n-\topts.tasks_selected++;\n-\n \tfor (i = 0; i < TASK__COUNT; i++) {\n+\t\tnum_selected += tasks[i].selected;\n \t\tif (!strcasecmp(tasks[i].name, arg)) {\n \t\t\ttask = &tasks[i];\n-\t\t\tbreak;\n \t\t}\n \t}\n \n@@ -1374,13 +1375,14 @@ static int task_option_parse(const struct option *opt,\n \t}\n \n \ttask->selected = 1;\n-\ttask->selected_order = opts.tasks_selected;\n+\ttask->selected_order = num_selected + 1;\n \n \treturn 0;\n }\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n+\tstatic struct maintenance_opts opts;\n \tstatic struct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n@@ -1408,7 +1410,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (argc == 1) {\n \t\tif (!strcmp(argv[0], \"run\"))\n-\t\t\treturn maintenance_run();\n+\t\t\treturn maintenance_run(&opts);\n \t}\n \n \tusage_with_options(builtin_maintenance_usage,\n"},{"id":"402948","messageId":"20200805123721.GA23655@danh.dev","threadId":"53809","inReplyTo":"c0b9876a-9041-3400-25ac-e52568a02074@gmail.com","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-08-05T12:37:21Z","receivedAt":"2020-08-05T19:56:20Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-07-30 15:02:32-0400, Derrick Stolee wrote:\n> On 7/30/2020 12:57 PM, Derrick Stolee wrote:\n> > On 7/29/2020 6:23 PM, Emily Shaffer wrote:\n> >> On Thu, Jul 23, 2020 at 05:56:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n> >>> diff --git a/builtin/gc.c b/builtin/gc.c\n> >>> index eb4b01c104..889d97afe7 100644\n> >>> --- a/builtin/gc.c\n> >>> +++ b/builtin/gc.c\n> >>> @@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n> >>> \treturn result;\n> >>> }\n> >>> \n> >>> +#define TWO_GIGABYTES (2147483647)\n> >>\n> >> [jonathan tan] This would be easier to understand if it was expressed\n> >> with bitshift, e.g. 1 << 31\n> > \n> > This is actually ((1 << 31) - 1) because \"unsigned long\" is 32-bits\n> > on Windows. But it's better to not use magic numbers and instead use\n> > operations like this.\n> \n> Nevermind. This breaks the build on 32-bit machines (even adding a few\n> \"L\" characters). I'll replace my magic decimal number with a magic\n> hexadecimal number.\n\nWould it be better if we use this C99 feature instead:\n\n\t#define TWO_GIGABYTES INT32_MAX\n\nBut it seems like it's available everywhere, except (maybe) Windows.\n(judging from UINT32_MAX's usage)\n\n-- \nDanh\n"},{"id":"403033","messageId":"d02d5948-0b5a-2103-7a57-1bcb54db1231@gmail.com","threadId":"53809","inReplyTo":"20200805123721.GA23655@danh.dev","subject":"Re: [PATCH v2 11/18] maintenance: auto-size incremental-repack batch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-06T13:54:27Z","receivedAt":"2020-08-06T15:16:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/5/2020 8:37 AM, Đoàn Trần Công Danh wrote:\n> On 2020-07-30 15:02:32-0400, Derrick Stolee wrote:\n>> On 7/30/2020 12:57 PM, Derrick Stolee wrote:\n>>> On 7/29/2020 6:23 PM, Emily Shaffer wrote:\n>>>> On Thu, Jul 23, 2020 at 05:56:33PM +0000, Derrick Stolee via GitGitGadget wrote:\n>>>>> diff --git a/builtin/gc.c b/builtin/gc.c\n>>>>> index eb4b01c104..889d97afe7 100644\n>>>>> --- a/builtin/gc.c\n>>>>> +++ b/builtin/gc.c\n>>>>> @@ -1021,19 +1021,65 @@ static int multi_pack_index_expire(void)\n>>>>> \treturn result;\n>>>>> }\n>>>>> \n>>>>> +#define TWO_GIGABYTES (2147483647)\n>>>>\n>>>> [jonathan tan] This would be easier to understand if it was expressed\n>>>> with bitshift, e.g. 1 << 31\n>>>\n>>> This is actually ((1 << 31) - 1) because \"unsigned long\" is 32-bits\n>>> on Windows. But it's better to not use magic numbers and instead use\n>>> operations like this.\n>>\n>> Nevermind. This breaks the build on 32-bit machines (even adding a few\n>> \"L\" characters). I'll replace my magic decimal number with a magic\n>> hexadecimal number.\n> \n> Would it be better if we use this C99 feature instead:\n> \n> \t#define TWO_GIGABYTES INT32_MAX\n> \n> But it seems like it's available everywhere, except (maybe) Windows.\n> (judging from UINT32_MAX's usage)\n\nThanks for the recommendation. This appears to work in all of the\nCI builds, including Windows. I'll use that constant in my next\nversion. Perhaps we will discover a platform that doesn't have it\nby including it.\n\n-Stolee\n"},{"id":"403083","messageId":"43187293-cbc1-84e6-df5c-6970ac25e72b@gmail.com","threadId":"53809","inReplyTo":"pull.671.v3.git.1596147867.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/20] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-06T17:58:26Z","receivedAt":"2020-08-06T17:58:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/30/2020 6:24 PM, Derrick Stolee via GitGitGadget wrote:\n> This is a second attempt...\n\nJust chiming in on this version to point out that what would\nhave been v4 of this series is now split across two series:\n\n[1] https://lore.kernel.org/git/pull.695.git.1596728921.gitgitgadget@gmail.com/\n[2] https://lore.kernel.org/git/pull.696.git.1596731424.gitgitgadget@gmail.com/\n\nThanks,\n-Stolee\n"}]}