{"thread":{"id":"53164","subject":"[PATCH 01/15] run-job: create barebones builtin","startedAt":"2020-04-03T20:48:20Z","lastAt":"2020-05-28T21:13:48Z","messageCount":55,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Derrick Stolee","Phillip Wood","Danh Doan","brian m. carlson","Johannes Schindelin","Josh Steadmon","Jonathan Nieder"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"394701","messageId":"665da239774419074a9bae49b9c92b340885bfa3.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 01/15] run-job: create barebones builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:00Z","receivedAt":"2020-04-03T20:48:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git run-job' command will be used to execute a short-lived set\nof maintenance activities by a background job manager. The intention\nis to perform small batches of work that reduce the foreground time\ntaken by repository maintenance such as 'git gc --auto'.\n\nThis change does the absolute minimum to create the builtin and show\nthe usage output.\n\nProvide an explicit warning that this command is experimental. The\nset of jobs may change, and each job could alter its behavior in\nfuture versions.\n\nRFC QUESTION: This builtin is based on the background maintenance in\nScalar. Specifically, this builtin is based on the \"scalar run \"\ncommand [1] [2]. My default thought was to make this a \"git run \"\ncommand to maximize similarity. However, it seems like \"git run\" is\ntoo generic. Or, am I being overly verbose for no reason?\n\n[1] https://github.com/microsoft/scalar/blob/master/docs/advanced.md#run-maintenance-in-the-foreground\n[2] https://github.com/microsoft/scalar/blob/master/Scalar/CommandLine/RunVerb.cs\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-run-job.txt | 36 +++++++++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/run-job.c | 28 +++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t7900-run-job.sh | 15 +++++++++++++++\n 8 files changed, 84 insertions(+)\n create mode 100644 Documentation/git-run-job.txt\n create mode 100644 builtin/run-job.c\n create mode 100755 t/t7900-run-job.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex 188bd1c3de1..5dea9d3b96b 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -144,6 +144,7 @@\n /git-rev-parse\n /git-revert\n /git-rm\n+/git-run-job\n /git-send-email\n /git-send-pack\n /git-serve\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nnew file mode 100644\nindex 00000000000..0627b3ed259\n--- /dev/null\n+++ b/Documentation/git-run-job.txt\n@@ -0,0 +1,36 @@\n+git-run-job(1)\n+==============\n+\n+NAME\n+----\n+git-run-job - Run a maintenance job. Intended for background operation.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git run-job '\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Run a maintenance job on the current repository. This is available as a\n+command for a few reasons. First, the background job feature can launch\n+these commands on a schedule and each process will completely clear its\n+memory when complete. Second, an expert user could create their own job\n+schedule by running these jobs themselves.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE SET OF AVAILABLE JOBS OR THEIR EXACT\n+BEHAVIOR MAY BE ALTERED IN THE FUTURE.\n+\n+\n+JOBS\n+----\n+\n+TBD\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex ef1ff2228f0..f5f9c4d9e94 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/rev-list.o\n BUILTIN_OBJS += builtin/rev-parse.o\n BUILTIN_OBJS += builtin/revert.o\n BUILTIN_OBJS += builtin/rm.o\n+BUILTIN_OBJS += builtin/run-job.o\n BUILTIN_OBJS += builtin/send-pack.o\n BUILTIN_OBJS += builtin/shortlog.o\n BUILTIN_OBJS += builtin/show-branch.o\ndiff --git a/builtin.h b/builtin.h\nindex 2b25a80cde3..3e0ddaaf67f 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -220,6 +220,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix);\n int cmd_rev_parse(int argc, const char **argv, const char *prefix);\n int cmd_revert(int argc, const char **argv, const char *prefix);\n int cmd_rm(int argc, const char **argv, const char *prefix);\n+int cmd_run_job(int argc, const char **argv, const char *prefix);\n int cmd_send_pack(int argc, const char **argv, const char *prefix);\n int cmd_shortlog(int argc, const char **argv, const char *prefix);\n int cmd_show(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nnew file mode 100644\nindex 00000000000..2c78d053aa4\n--- /dev/null\n+++ b/builtin/run-job.c\n@@ -0,0 +1,28 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_run_job_usage[] = {\n+\tN_(\"git run-job\"),\n+\tNULL\n+};\n+\n+int cmd_run_job(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_run_job_options[] = {\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_run_job_usage,\n+\t\t\t\t builtin_run_job_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_run_job_options,\n+\t\t\t builtin_run_job_usage,\n+\t\t\t PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tusage_with_options(builtin_run_job_usage,\n+\t\t\t builtin_run_job_options);\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 20878946558..1cd2b415e46 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -156,6 +156,7 @@ git-revert mainporcelain\n git-rev-list plumbinginterrogators\n git-rev-parse plumbinginterrogators\n git-rm mainporcelain worktree\n+git-run-job plumbingmanipulators\n git-send-email foreignscminterface complete\n git-send-pack synchingrepositories\n git-shell synchelpers\ndiff --git a/git.c b/git.c\nindex b07198fe036..db5a43c8687 100644\n--- a/git.c\n+++ b/git.c\n@@ -566,6 +566,7 @@ static struct cmd_struct commands[] = {\n \t{ \"rev-parse\", cmd_rev_parse, NO_PARSEOPT },\n \t{ \"revert\", cmd_revert, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"rm\", cmd_rm, RUN_SETUP },\n+\t{ \"run-job\", cmd_run_job, RUN_SETUP },\n \t{ \"send-pack\", cmd_send_pack, RUN_SETUP },\n \t{ \"shortlog\", cmd_shortlog, RUN_SETUP_GENTLY | USE_PAGER },\n \t{ \"show\", cmd_show, RUN_SETUP },\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nnew file mode 100755\nindex 00000000000..1eac80b7ed3\n--- /dev/null\n+++ b/t/t7900-run-job.sh\n@@ -0,0 +1,15 @@\n+#!/bin/sh\n+\n+test_description='git run-job\n+\n+Testing the background jobs, in the foreground\n+'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'help text' '\n+\ttest_must_fail git run-job -h 2>err &&\n+\ttest_i18ngrep \"usage: git run-job\" err\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"394702","messageId":"pull.597.git.1585946894.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":null,"subject":"[PATCH 00/15] [RFC] Maintenance jobs and job runner","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:47:59Z","receivedAt":"2020-04-03T20:48:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Background maintenance. I've blogged about it [1]. I've talked about it [2].\nI brought it up at the contributor summit [3]. I think it's important. It's\na critical feature in Scalar and VFS for Git. It's the main reason we\ncreated the \"scalar register\" feature for existing repos (so the Scalar\nbackground maintenance would apply to a \"vanilla\" Git repo).\n\n[1] https://devblogs.microsoft.com/devops/introducing-scalar/[2] \nhttps://stolee.dev/docs/git-merge-2020.pdf[3] \nhttps://lore.kernel.org/git/35FDF767-CE0C-4C51-88A8-12965CD2D4FF@jramsay.com.au/\n\nThis RFC does the \"maintenance\" half of \"background maintenance\". It creates\ntwo new builtins to assist users in scheduling their own background\nmaintenance:\n\n * git run-job : This builtin will run a single instance of a\n maintenance job.\n \n \n * git job-runner [--repo=]: This builtin will run an infinite loop\n that executes git run-job as a subcommand.\n \n \n\nI believe that I could replace most maintenance steps in Scalar with the git\nrun-job command and all that we would lose is some logging. (The only one\nthat would remain is the one that sets recommended config settings, but that\ncould be changed to a one-time operation at service start-up.) I haven't\ntested this yet, but I plan to before sending a non-RFC option.\n\nOf course, just because we think these maintenance operations work for our\nscenario does not mean that they immediately work as the \"canonical\"\nmaintenance activities. I attempt to demonstrate the value of these jobs as\nthey are implemented. My perspective is skewed towards very large\nrepositories (2000+ full-time contributors), but even medium-size\nrepositories (100-200 full-time contributors) can benefit from these jobs.\n\nI've tried to split the series into logical commits. This includes each\nspecific maintenance job being completely described in its own commit (plus\nmaybe an extra one to allow a custom option).\n\nMost commit messages have \"RFC QUESTION(S)\" for things I was unsure about.\n\nI'm currently testing this locally by setting job.repo in my global config\nto be a few important repos on my Linux VM then running \nGIT_TRACE2_PERF= git job-runner --daemonize to launch a background\nprocess that logs the subcommands to .\n\nHere I will call out things that could definitely be improved:\n\n 1. The git job-runner process is not tested AT ALL. It's a bit tricky to\n test it since it shouldn't exit when things work as expected. I expect\n to create a --no-loop option to stop after one iteration of the job\n loop. But I would like a bit more feedback on the concept before jumping\n into those tests. (I do test the git run-job builtin for each job, but\n not the customization through arguments and config.) Perhaps we could do\n some funny business about mocking git using --exec-path to check that it\n is calling git run-job in the correct order (and more importantly, not \n calling it when certain config settings are present).\n \n \n 2. The --daemonize option at the end is shamelessly stolen from git gc\n --daemonize and git daemon, but has limited abilities on some platforms\n (I've tested on Linux and macOS). I have not done my research on how far\n this gets us to allowing users to launch this at startup or something. \n \n \n 3. As I said, this is the \"maintenance\" \"half\" of \"background maintenance\".\n The \"background\" part is harder in my opinion because it involves\n creating platform-specific ways to consistently launch background\n processes. For example, Unix systems should have one way to service\n start X while Windows has another. macOS has launchd to launch processes\n as users log in, which should be a good way forward. Scalar implements a\n Windows Service that runs as root but impersonates the latest user to\n log in, and it implements a macOS \"service\" that is running only with\n the current user. I expect to need to create these services myself as a\n follow-up, but I lack the expertise to do it (currently). If someone\n else has experience creating these things and wants to take over or\n advise that half then I would appreciate the help!\n \n \n 4. I noticed late in the RFC process that I'm not clearing my argv_arrays\n carefully in the job-runner. This will need to be rectified and\n carefully checked with valgrind before merging this code. While it leaks\n memory very slowly, it will be important that we don't leak any memory\n at all since this is a long-lived process. There's also some places\n where I was careful to not include too much of libgit.a to help keep the\n memory footprint low.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (15):\n run-job: create barebones builtin\n run-job: implement commit-graph job\n run-job: implement fetch job\n run-job: implement loose-objects job\n run-job: implement pack-files job\n run-job: auto-size or use custom pack-files batch\n config: add job.pack-files.batchSize option\n job-runner: create builtin for job loop\n job-runner: load repos from config by default\n job-runner: use config to limit job frequency\n job-runner: use config for loop interval\n job-runner: add --interval= option\n job-runner: skip a job if job..enabled is false\n job-runner: add --daemonize option\n runjob: customize the loose-objects batch size\n\n .gitignore | 2 +\n Documentation/config.txt | 2 +\n Documentation/config/job.txt | 37 +++\n Documentation/git-job-runner.txt | 63 +++++\n Documentation/git-run-job.txt | 102 +++++++\n Makefile | 2 +\n builtin.h | 2 +\n builtin/job-runner.c | 347 +++++++++++++++++++++++\n builtin/run-job.c | 458 +++++++++++++++++++++++++++++++\n cache.h | 4 +-\n command-list.txt | 2 +\n commit-graph.c | 2 +-\n commit-graph.h | 1 +\n daemon.h | 7 +\n git.c | 2 +\n midx.c | 2 +-\n midx.h | 1 +\n t/t7900-run-job.sh | 137 +++++++++\n 18 files changed, 1168 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/config/job.txt\n create mode 100644 Documentation/git-job-runner.txt\n create mode 100644 Documentation/git-run-job.txt\n create mode 100644 builtin/job-runner.c\n create mode 100644 builtin/run-job.c\n create mode 100644 daemon.h\n create mode 100755 t/t7900-run-job.sh\n\n\nbase-commit: 9fadedd637b312089337d73c3ed8447e9f0aa775\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-597%2Fderrickstolee%2Fjobs-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-597/derrickstolee/jobs-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/597\n-- \ngitgitgadget\n"},{"id":"394703","messageId":"cd3facd1e03923204bd2259ff0635a5bbf68d700.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 02/15] run-job: implement commit-graph job","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:01Z","receivedAt":"2020-04-03T20:48:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe first job to implement in the 'git run-job' command is the\n'commit-graph' job. It is based on the sequence of events in the\n'commit-graph' job in Scalar [1]. This sequence is as follows:\n\n1. git commit-graph write --reachable --split\n2. git commit-graph verify --shallow\n3. If the verify succeeds, stop.\n4. Delete the commit-graph-chain file.\n5. git commit-graph write --reachable --split\n\nBy writing an incremental commit-graph file using the \"--split\"\noption we minimize the disruption from this operation. The default\nbehavior is to merge layers until the new \"top\" layer is less than\nhalf the size of the layer below. This provides quick writes \"most\"\nof the time, with the longer writes following a power law\ndistribution.\n\nMost importantly, concurrent Git processes only look at the\ncommit-graph-chain file for a very short amount of time, so they\nwill verly likely not be holding a handle to the file when we try\nto replace it. (This only matters on Windows.)\n\nIf a concurrent process reads the old commit-graph-chain file, but\nour job expires some of the .graph files before they can be read,\nthen those processes will see a warning message (but not fail).\nThis could be avoided by a future update to use the --expire-time\nargument when writing the commit-graph.\n\nBy using 'git commit-graph verify --shallow' we can ensure that\nthe file we just wrote is valid. This is an extra safety precaution\nthat is faster than our 'write' subcommand. In the rare situation\nthat the newest layer of the commit-graph is corrupt, we can \"fix\"\nthe corruption by deleting the commit-graph-chain file and rewrite\nthe full commit-graph as a new one-layer commit graph. This does\nnot completely prevent _that_ file from being corrupt, but it does\nrecompute the commit-graph by parsing commits from the object\ndatabase. In our use of this step in Scalar and VFS for Git, we\nhave only seen this issue arise because our microsoft/git fork\nreverted 43d3561 (\"commit-graph write: don't die if the existing\ngraph is corrupt\" 2019-03-25) for a while to keep commit-graph\nwrites very fast. We dropped the revert when updating to v2.23.0.\nThe verify still has potential for catching corrupt data across\nthe layer boundary: if the new file has commit X with parent Y\nin an old file but the commit ID for Y in the old file had a\nbitswap, then we will notice that in the 'verify' command.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs\n\nRFC QUESTIONS:\n\n1. Are links like [1] helpful?\n\n2. Can anyone think of a way to test the rewrite fallback?\n It requires corrupting the latest file between two steps of\n this one command, so that is a tricky spot to inject a fault.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-run-job.txt | 21 ++++++++++--\n builtin/run-job.c | 60 ++++++++++++++++++++++++++++++++++-\n commit-graph.c | 2 +-\n commit-graph.h | 1 +\n t/t7900-run-job.sh | 32 +++++++++++++++++++\n 5 files changed, 112 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nindex 0627b3ed259..8bf2762d650 100644\n--- a/Documentation/git-run-job.txt\n+++ b/Documentation/git-run-job.txt\n@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.\n SYNOPSIS\n --------\n [verse]\n-'git run-job '\n+'git run-job commit-graph'\n \n \n DESCRIPTION\n@@ -28,7 +28,24 @@ BEHAVIOR MAY BE ALTERED IN THE FUTURE.\n JOBS\n ----\n \n-TBD\n+'commit-graph'::\n+\n+The `commit-graph` job updates the `commit-graph` files incrementally,\n+then verifies that the written data is correct. If the new layer has an\n+issue, then the chain file is removed and the `commit-graph` is\n+rewritten from scratch.\n++\n+The verification only checks the top layer of the `commit-graph` chain.\n+If the incremental write merged the new commits with at least one\n+existing layer, then there is potential for on-disk corruption being\n+carried forward into the new file. This will be noticed and the new\n+commit-graph file will be clean as Git reparses the commit data from\n+the object database.\n++\n+The incremental write is safe to run alongside concurrent Git processes\n+since it will not expire `.graph` files that were in the previous\n+`commit-graph-chain` file. They will be deleted by a later run based on\n+the expiration delay.\n \n \n GIT\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nindex 2c78d053aa4..dd7709952d3 100644\n--- a/builtin/run-job.c\n+++ b/builtin/run-job.c\n@@ -1,12 +1,65 @@\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"commit-graph.h\"\n+#include \"object-store.h\"\n #include \"parse-options.h\"\n+#include \"repository.h\"\n+#include \"run-command.h\"\n \n static char const * const builtin_run_job_usage[] = {\n-\tN_(\"git run-job\"),\n+\tN_(\"git run-job commit-graph\"),\n \tNULL\n };\n \n+static int run_write_commit_graph(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"commit-graph\", \"write\",\n+\t\t\t \"--split\", \"--reachable\", \"--no-progress\",\n+\t\t\t NULL);\n+\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int run_verify_commit_graph(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\n+\targv_array_pushl(&cmd, \"commit-graph\", \"verify\",\n+\t\t\t \"--shallow\", \"--no-progress\", NULL);\n+\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int run_commit_graph_job(void)\n+{\n+\tchar *chain_path;\n+\n+\tif (run_write_commit_graph()) {\n+\t\terror(_(\"failed to write commit-graph\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (!run_verify_commit_graph())\n+\t\treturn 0;\n+\n+\twarning(_(\"commit-graph verify caught error, rewriting\"));\n+\n+\tchain_path = get_chain_filename(the_repository->objects->odb);\n+\tif (unlink(chain_path)) {\n+\t\tUNLEAK(chain_path);\n+\t\tdie(_(\"failed to remove commit-graph at %s\"), chain_path);\n+\t}\n+\tfree(chain_path);\n+\n+\tif (!run_write_commit_graph())\n+\t\treturn 0;\n+\n+\terror(_(\"failed to rewrite commit-graph\"));\n+\treturn 1;\n+}\n+\n int cmd_run_job(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_run_job_options[] = {\n@@ -23,6 +76,11 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_run_job_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"commit-graph\"))\n+\t\t\treturn run_commit_graph_job();\n+\t}\n+\n \tusage_with_options(builtin_run_job_usage,\n \t\t\t builtin_run_job_options);\n }\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f013a84e294..6867f92d04a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -56,7 +56,7 @@ static char *get_split_graph_filename(struct object_directory *odb,\n \t\t oid_hex);\n }\n \n-static char *get_chain_filename(struct object_directory *odb)\n+char *get_chain_filename(struct object_directory *odb)\n {\n \treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", odb->path);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e87a6f63600..8b7bd5a6cb1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -13,6 +13,7 @@\n struct commit;\n \n char *get_commit_graph_filename(struct object_directory *odb);\n+char *get_chain_filename(struct object_directory *odb);\n int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n \n /*\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nindex 1eac80b7ed3..18b9bd26b3a 100755\n--- a/t/t7900-run-job.sh\n+++ b/t/t7900-run-job.sh\n@@ -5,6 +5,8 @@ test_description='git run-job\n Testing the background jobs, in the foreground\n '\n \n+GIT_TEST_COMMIT_GRAPH=0\n+\n . ./test-lib.sh\n \n test_expect_success 'help text' '\n@@ -12,4 +14,34 @@ test_expect_success 'help text' '\n \ttest_i18ngrep \"usage: git run-job\" err\n '\n \n+test_expect_success 'commit-graph job' '\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\tchain=.git/objects/info/commit-graphs/commit-graph-chain &&\n+\t\tgit checkout -b master &&\n+\t\tfor i in $(test_seq 1 15)\n+\t\tdo\n+\t\t\ttest_commit $i || return 1\n+\t\tdone &&\n+\t\tgit run-job commit-graph 2>../err &&\n+\t\ttest_must_be_empty ../err &&\n+\t\ttest_line_count = 1 $chain &&\n+\t\tfor i in $(test_seq 16 19)\n+\t\tdo\n+\t\t\ttest_commit $i || return 1\n+\t\tdone &&\n+\t\tgit run-job commit-graph 2>../err &&\n+\t\ttest_must_be_empty ../err &&\n+\t\ttest_line_count = 2 $chain &&\n+\t\tfor i in $(test_seq 20 23)\n+\t\tdo\n+\t\t\ttest_commit $i || return 1\n+\t\tdone &&\n+\t\tgit run-job commit-graph 2>../err &&\n+\t\ttest_must_be_empty ../err &&\n+\t\ttest_line_count = 1 $chain\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"394704","messageId":"5277398f189dd6e1456b0afdd6d85dbac7d98d07.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 05/15] run-job: implement pack-files job","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:04Z","receivedAt":"2020-04-03T20:48:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe previous change cleaned up loose objects using the\n'loose-objects' that can be run safely in the background. Add a\nsimilar job that performs similar cleanups for pack-files.\n\nOne issue with running 'git repack' is that it is designed to\nrepack all pack-files into a single pack-file. While this is the\nmost space-efficient way to store object data, it is not time or\nmemory efficient. This becomes extremely important if the repo is\nso large that a user struggles to store two copies of the pack on\ntheir disk.\n\nInstead, perform an \"incremental\" repack by collecting a few small\npack-files into a new pack-file. The multi-pack-index facilitates\nthis process ever since 'git multi-pack-index expire' was added in\n19575c7 (multi-pack-index: implement 'expire' subcommand,\n2019-06-10) and 'git multi-pack-index repack' was added in ce1e4a1\n(midx: implement midx_repack(), 2019-06-10).\n\nThe 'pack-files' job runs the following steps:\n\n1. 'git multi-pack-index write' creates a multi-pack-index file if\n one did not exist, and otherwise will update the multi-pack-index\n with any new pack-files that appeared since the last write. This\n is particularly relevant with the background fetch job.\n\n When the multi-pack-index sees two copies of the same object, it\n stores the offset data into the newer pack-file. This means that\n some old pack-files could become \"unreferenced\" which I will use\n to mean \"a pack-file that is in the pack-file list of the\n multi-pack-index but none of the objects in the multi-pack-index\n reference a location inside that pack-file.\"\n\n2. 'git multi-pack-index expire' deletes any unreferenced pack-files\n and updaes the multi-pack-index to drop those pack-files from the\n list. This is safe to do as concurrent Git processes will see the\n multi-pack-index and not open those packs when looking for object\n contents. (Similar to the 'loose-objects' job, there are some Git\n commands that open pack-files regardless of the multi-pack-index,\n but they are rarely used. Further, a user that self-selects to\n use background operations would likely refrain from using those\n commands.)\n\n3. 'git multi-pack-index repack --bacth-size=' collects a set\n of pack-files that are listed in the multi-pack-index and creates\n a new pack-file containing the objects whose offsets are listed\n by the multi-pack-index to be in those objects. The set of pack-\n files is selected greedily by sorting the pack-files by modified\n time and adding a pack-file to the set if its \"expected size\" is\n smaller than the batch size until the total expected size of the\n selected pack-files is at least the batch size. The \"expected\n size\" is calculated by taking the size of the pack-file divided\n by the number of objects in the pack-file and multiplied by the\n number of objects from the multi-pack-index with offset in that\n pack-file. The expected size approximats how much data from that\n pack-file will contribute to the resulting pack-file size. The\n intention is that the resulting pack-file will be close in size\n to the provided batch size.\n\n The next run of the pack-files job will delete these repacked\n pack-files during the 'expire' step.\n\n In this version, the batch size is set to \"0\" which ignores the\n size restrictions when selecting the pack-files. It instead\n selects all pack-files and repacks all packed objects into a\n single pack-file. This will be updated in the next change, but\n it requires doing some calculations that are better isolated to\n a separate change.\n\nEach of the above steps update the multi-pack-index file. After\neach step, we verify the new multi-pack-index. If the new\nmulti-pack-index is corrupt, then delete the multi-pack-index,\nrewrite it from scratch, and stop doing the later steps of the\njob. This is intended to be an extra-safe check without leaving\na repo with many pack-files without a multi-pack-index.\n\nThese steps are based on a similar background maintenance step in\nScalar (and VFS for Git) [1]. This was incredibly effective for\nusers of the Windows OS repository. After using the same VFS for Git\nrepository for over a year, some users had _thousands_ of pack-files\nthat combined to up to 250 GB of data. We noticed a few users were\nrunning into the open file descriptor limits (due in part to a bug\nin the multi-pack-index fixed by af96fe3392 (midx: add packs to\npacked_git linked list, 2019-04-29).\n\nThese pack-files were mostly small since they contained the commits\nand trees that were pushed to the origin in a given hour. The GVFS\nprotocol includes a \"prefetch\" step that asks for pre-computed pack-\nfiles containing commits and trees by timestamp. These pack-files\nwere grouped into \"daily\" pack-files once a day for up to 30 days.\nIf a user did not request prefetch packs for over 30 days, then they\nwould get the entire history of commits and trees in a new, large\npack-file. This led to a large number of pack-files that had poor\ndelta compression.\n\nBy running this pack-file maintenance step once per day, these repos\nwith thousands of packs spanning 200+ GB dropped to dozens of pack-\nfiles spanning 30-50 GB. This was done all without removing objects\nfrom the system and using a constant batch size of two gigabytes.\nOnce the work was done to reduce the pack-files to small sizes, the\nbatch size of two gigabytes means that not every run triggers a\nrepack operation, so the following run will not expire a pack-file.\nThis has kept these repos in a \"clean\" state.\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-run-job.txt | 18 ++++++-\n builtin/run-job.c | 90 ++++++++++++++++++++++++++++++++++-\n midx.c | 2 +-\n midx.h | 1 +\n t/t7900-run-job.sh | 39 +++++++++++++++\n 5 files changed, 147 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nindex 43ca1160b5a..108ed25b8bd 100644\n--- a/Documentation/git-run-job.txt\n+++ b/Documentation/git-run-job.txt\n@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.\n SYNOPSIS\n --------\n [verse]\n-'git run-job (commit-graph|fetch|loose-objects)'\n+'git run-job (commit-graph|fetch|loose-objects|pack-files)'\n \n \n DESCRIPTION\n@@ -71,6 +71,22 @@ a batch of loose objects. The batch size is limited to 50 thousand\n objects to prevent the job from taking too long on a repository with\n many loose objects.\n \n+'pack-files'::\n+\n+The `pack-files` job incrementally repacks the object directory using\n+the `multi-pack-index` feature. In order to prevent race conditions with\n+concurrent Git commands, it follows a two-step process. First, it\n+deletes any pack-files included in the `multi-pack-index` where none of\n+the objects in the `multi-pack-index` reference those pack-files; this\n+only happens if all objects in the pack-file are also stored in a newer\n+pack-file. Second, it selects a group of pack-files whose \"expected\n+size\" is below the batch size until the group has total expected size at\n+least the batch size; see the `--batch-size` option for the `repack`\n+subcommand in linkgit:git-multi-pack-index[1]. The default batch-size is\n+zero, which is a special case that attempts to repack all pack-files\n+into a single pack-file.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nindex cecf9058c51..d3543f7ccb9 100644\n--- a/builtin/run-job.c\n+++ b/builtin/run-job.c\n@@ -1,13 +1,14 @@\n #include \"builtin.h\"\n #include \"config.h\"\n #include \"commit-graph.h\"\n+#include \"midx.h\"\n #include \"object-store.h\"\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"run-command.h\"\n \n static char const * const builtin_run_job_usage[] = {\n-\tN_(\"git run-job (commit-graph|fetch|loose-objects)\"),\n+\tN_(\"git run-job (commit-graph|fetch|loose-objects|pack-files)\"),\n \tNULL\n };\n \n@@ -238,6 +239,91 @@ static int run_loose_objects_job(void)\n \treturn prune_packed() || pack_loose();\n }\n \n+static int multi_pack_index_write(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"write\",\n+\t\t\t \"--no-progress\", NULL);\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int rewrite_multi_pack_index(void)\n+{\n+\tchar *midx_name = get_midx_filename(the_repository->objects->odb->path);\n+\n+\tunlink(midx_name);\n+\tfree(midx_name);\n+\n+\tif (multi_pack_index_write()) {\n+\t\terror(_(\"failed to rewrite multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int multi_pack_index_verify(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"verify\",\n+\t\t\t \"--no-progress\", NULL);\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int multi_pack_index_expire(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"expire\",\n+\t\t\t \"--no-progress\", NULL);\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int multi_pack_index_repack(void)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\",\n+\t\t\t \"--no-progress\", \"--batch-size=0\", NULL);\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tif (result && multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after repack\"));\n+\t\tresult = rewrite_multi_pack_index();\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int run_pack_files_job(void)\n+{\n+\tif (multi_pack_index_write()) {\n+\t\terror(_(\"failed to write multi-pack-index\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after initial write\"));\n+\t\treturn rewrite_multi_pack_index();\n+\t}\n+\n+\tif (multi_pack_index_expire()) {\n+\t\terror(_(\"multi-pack-index expire failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\tif (multi_pack_index_verify()) {\n+\t\twarning(_(\"multi-pack-index verify failed after expire\"));\n+\t\treturn rewrite_multi_pack_index();\n+\t}\n+\n+\tif (multi_pack_index_repack()) {\n+\t\terror(_(\"multi-pack-index repack failed\"));\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n int cmd_run_job(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_run_job_options[] = {\n@@ -261,6 +347,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)\n \t\t\treturn run_fetch_job();\n \t\tif (!strcmp(argv[0], \"loose-objects\"))\n \t\t\treturn run_loose_objects_job();\n+\t\tif (!strcmp(argv[0], \"pack-files\"))\n+\t\t\treturn run_pack_files_job();\n \t}\n \n \tusage_with_options(builtin_run_job_usage,\ndiff --git a/midx.c b/midx.c\nindex 1527e464a7b..0f0d0a38812 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -36,7 +36,7 @@\n \n #define PACK_EXPIRED UINT_MAX\n \n-static char *get_midx_filename(const char *object_dir)\n+char *get_midx_filename(const char *object_dir)\n {\n \treturn xstrfmt(\"%s/pack/multi-pack-index\", object_dir);\n }\ndiff --git a/midx.h b/midx.h\nindex e6fa356b5ca..cf2c09dffc2 100644\n--- a/midx.h\n+++ b/midx.h\n@@ -39,6 +39,7 @@ struct multi_pack_index {\n \n #define MIDX_PROGRESS (1 << 0)\n \n+char *get_midx_filename(const char *object_dir);\n struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);\n int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);\n int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nindex 41da083257b..416ba04989d 100755\n--- a/t/t7900-run-job.sh\n+++ b/t/t7900-run-job.sh\n@@ -6,6 +6,7 @@ Testing the background jobs, in the foreground\n '\n \n GIT_TEST_COMMIT_GRAPH=0\n+GIT_TEST_MULTI_PACK_INDEX=0\n \n . ./test-lib.sh\n \n@@ -93,4 +94,42 @@ test_expect_success 'loose-objects job' '\n \ttest_cmp packs-between packs-after\n '\n \n+test_expect_success 'pack-files job' '\n+\tpackDir=.git/objects/pack &&\n+\n+\t# Create three disjoint pack-files with size BIG, small, small.\n+\n+\techo HEAD~2 | git -C client pack-objects --revs $packDir/test-1 &&\n+\n+\ttest_tick &&\n+\tgit -C client pack-objects --revs $packDir/test-2 <<-\\EOF &&\n+\tHEAD~1\n+\t^HEAD~2\n+\tEOF\n+\n+\ttest_tick &&\n+\tgit -C client pack-objects --revs $packDir/test-3 <<-\\EOF &&\n+\tHEAD\n+\t^HEAD~1\n+\tEOF\n+\n+\trm -f client/$packDir/pack-* &&\n+\trm -f client/$packDir/loose-* &&\n+\n+\tls client/$packDir/*.pack >packs-before &&\n+\ttest_line_count = 3 packs-before &&\n+\n+\t# the job repacks the two into a new pack, but does not\n+\t# delete the old ones.\n+\tgit -C client run-job pack-files &&\n+\tls client/$packDir/*.pack >packs-between &&\n+\ttest_line_count = 4 packs-between &&\n+\n+\t# the job deletes the two old packs, and does not write\n+\t# a new one because only one pack remains.\n+\tgit -C client run-job pack-files &&\n+\tls client/.git/objects/pack/*.pack >packs-after &&\n+\ttest_line_count = 1 packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"394705","messageId":"77b1da5d3063a2404cd750adfe3bb8be9b6c497d.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 03/15] run-job: implement fetch job","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:02Z","receivedAt":"2020-04-03T20:48:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen working with very large repositories, an incremental 'git fetch'\ncommand can download a large amount of data. If there are many other\nusers pushing to a common repo, then this data can rival the initial\npack-file size of a 'git clone' of a medium-size repo.\n\nUsers may want to keep the data on their local repos as close as\npossible to the data on the remote repos by fetching periodically in\nthe background. This can break up a large daily fetch into several\nsmaller hourly fetches.\n\nHowever, if we simply ran 'git fetch ' in the background,\nthen the user running a foregroudn 'git fetch ' would lose\nsome important feedback when a new branch appears or an existing\nbranch updates. This is especially true if a remote branch is\nforce-updated and this isn't noticed by the user because it occurred\nin the background. Further, the functionality of 'git push\n--force-with-lease' becomes suspect.\n\nWhen running 'git fetch ' in the background, use\nthe following options for careful updating:\n\n1. --no-tags prevents getting a new tag when a user wants to see\n the new tags appear in their foreground fetches.\n\n2. --refmap= removes the configured refspec which usually updates\n refs/remotes//* with the refs advertised by the remote.\n\n3. By adding a new refspec \"+refs/heads/*:refs/hidden//*\"\n we can ensure that we actually load the new values somewhere in\n our refspace while not updating refs/heads or refs/remotes. By\n storing these refs here, the commit-graph job will update the\n commit-graph with the commits from these hidden refs.\n\n4. --prune will delete the refs/hidden/ refs that no\n longer appear on the remote.\n\nWe've been using this step as a critical background job in Scalar\n[1] (and VFS for Git). This solved a pain point that was showing up\nin user reports: fetching was a pain! Users do not like waiting to\ndownload the data that was created while they were away from their\nmachines. After implementing background fetch, the foreground fetch\ncommands sped up significantly because they mostly just update refs\nand download a small amount of new data. The effect is especially\ndramatic when paried with --no-show-forced-udpates (through\nfetch.showForcedUpdates=false).\n\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/FetchStep.cs\n\nRFC QUESTIONS:\n\n1. One downside of the refs/hidden pattern is that 'git log' will\n decorate commits with twice as many refs if they appear at a\n remote ref (/ _and_ refs/hidden//). Is\n there an easy way to exclude a refspace from decorations? Should\n we make refs/hidden/* a \"special\" refspace that is excluded from\n decorations?\n\n2. This feature is designed for a desktop machine or equivalent\n that has a permanent wired network connection, and the machine\n stays on while the user is not present. For things like laptops\n with inconsistent WiFi connections (that may be metered) the\n feature can use the less stable connection more than the user\n wants. Of course, this feature is opt-in for Git, but in Scalar\n we have a \"scalar pause\" command [2] that pauses all maintenance\n for some amount of time. We should consider a similar mechanism\n for Git, but for the point of this series the user needs to set\n up the \"background\" part of these jobs manually.\n\n[2] https://github.com/microsoft/scalar/blob/master/Scalar/CommandLine/PauseVerb.cs\n[3] https://github.com/microsoft/scalar/blob/master/docs/advanced.md#controlling-background-maintenance\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-run-job.txt | 13 ++++-\n builtin/run-job.c | 89 ++++++++++++++++++++++++++++++++++-\n t/t7900-run-job.sh | 22 +++++++++\n 3 files changed, 122 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nindex 8bf2762d650..eb92e891915 100644\n--- a/Documentation/git-run-job.txt\n+++ b/Documentation/git-run-job.txt\n@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.\n SYNOPSIS\n --------\n [verse]\n-'git run-job commit-graph'\n+'git run-job (commit-graph|fetch)'\n \n \n DESCRIPTION\n@@ -47,6 +47,17 @@ since it will not expire `.graph` files that were in the previous\n `commit-graph-chain` file. They will be deleted by a later run based on\n the expiration delay.\n \n+'fetch'::\n+\n+The `fetch` job updates the object directory with the latest objects\n+from all registered remotes. For each remote, a `git fetch` command is\n+run. The refmap is custom to avoid updating local or remote branches\n+(those in `refs/heads` or `refs/remotes`). Instead, the remote refs are\n+stored in `refs/hidden//`. Also, no tags are updated.\n++\n+This means that foreground fetches are still required to update the\n+remote refs, but the users is notified when the branches and tags are\n+updated on the remote.\n \n GIT\n ---\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nindex dd7709952d3..e59056b2918 100644\n--- a/builtin/run-job.c\n+++ b/builtin/run-job.c\n@@ -7,7 +7,7 @@\n #include \"run-command.h\"\n \n static char const * const builtin_run_job_usage[] = {\n-\tN_(\"git run-job commit-graph\"),\n+\tN_(\"git run-job (commit-graph|fetch)\"),\n \tNULL\n };\n \n@@ -60,6 +60,91 @@ static int run_commit_graph_job(void)\n \treturn 1;\n }\n \n+static int fetch_remote(const char *remote)\n+{\n+\tint result;\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf refmap = STRBUF_INIT;\n+\n+\targv_array_pushl(&cmd, \"fetch\", remote, \"--quiet\", \"--prune\",\n+\t\t\t \"--no-tags\", \"--refmap=\", NULL);\n+\n+\tstrbuf_addf(&refmap, \"+refs/heads/*:refs/hidden/%s/*\", remote);\n+\targv_array_push(&cmd, refmap.buf);\n+\n+\tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+\n+\tstrbuf_release(&refmap);\n+\treturn result;\n+}\n+\n+static int fill_remotes(struct string_list *remotes)\n+{\n+\tint result = 0;\n+\tFILE *proc_out;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct child_process *remote_proc = xmalloc(sizeof(*remote_proc));\n+\n+\tchild_process_init(remote_proc);\n+\n+\targv_array_pushl(&remote_proc->args, \"git\", \"remote\", NULL);\n+\n+\tremote_proc->out = -1;\n+\n+\tif (start_command(remote_proc)) {\n+\t\terror(_(\"failed to start 'git remote' process\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tproc_out = xfdopen(remote_proc->out, \"r\");\n+\n+\t/* if there is no line, leave the value as given */\n+\twhile (!strbuf_getline(&line, proc_out))\n+\t\tstring_list_append(remotes, line.buf);\n+\n+\tstrbuf_release(&line);\n+\n+\tfclose(proc_out);\n+\n+\tif (finish_command(remote_proc)) {\n+\t\terror(_(\"failed to finish 'git remote' process\"));\n+\t\tresult = 1;\n+\t}\n+\n+cleanup:\n+\tfree(remote_proc);\n+\treturn result;\n+}\n+\n+static int run_fetch_job(void)\n+{\n+\tint result = 0;\n+\tstruct string_list_item *item;\n+\tstruct string_list remotes = STRING_LIST_INIT_DUP;\n+\n+\tif (fill_remotes(&remotes)) {\n+\t\terror(_(\"failed to fill remotes\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/*\n+\t * Do not modify the result based on the success of the 'fetch'\n+\t * operation, as a loss of network could cause 'fetch' to fail\n+\t * quickly. We do not want that to stop the rest of our\n+\t * background operations.\n+\t */\n+\tfor (item = remotes.items;\n+\t item && item < remotes.items + remotes.nr;\n+\t item++)\n+\t\tfetch_remote(item->string);\n+\n+cleanup:\n+\tstring_list_clear(&remotes, 0);\n+\treturn result;\n+}\n+\n int cmd_run_job(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_run_job_options[] = {\n@@ -79,6 +164,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"commit-graph\"))\n \t\t\treturn run_commit_graph_job();\n+\t\tif (!strcmp(argv[0], \"fetch\"))\n+\t\t\treturn run_fetch_job();\n \t}\n \n \tusage_with_options(builtin_run_job_usage,\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nindex 18b9bd26b3a..d3faeba135b 100755\n--- a/t/t7900-run-job.sh\n+++ b/t/t7900-run-job.sh\n@@ -44,4 +44,26 @@ test_expect_success 'commit-graph job' '\n \t)\n '\n \n+test_expect_success 'fetch job' '\n+\tgit clone \"file://$(pwd)/server\" client &&\n+\n+\t# Before fetching, build a client commit-graph\n+\tgit -C client run-job commit-graph &&\n+\tchain=client/.git/objects/info/commit-graphs/commit-graph-chain &&\n+\ttest_line_count = 1 $chain &&\n+\n+\tgit -C client branch -v --remotes >before-refs &&\n+\ttest_commit -C server 24 &&\n+\n+\tgit -C client run-job fetch &&\n+\tgit -C client branch -v --remotes >after-refs &&\n+\ttest_cmp before-refs after-refs &&\n+\ttest_cmp server/.git/refs/heads/master \\\n+\t\t client/.git/refs/hidden/origin/master &&\n+\n+\t# the hidden ref should trigger a new layer in the commit-graph\n+\tgit -C client run-job commit-graph &&\n+\ttest_line_count = 2 $chain\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"394707","messageId":"c39a35a959e485623aacdbbecc1fce7d16a956c9.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 06/15] run-job: auto-size or use custom pack-files batch","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:05Z","receivedAt":"2020-04-03T20:48:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen repacking during the 'pack-files' job, we use the --batch-size\noption in 'git multi-pack-index repack'. The initial setting used\n--batch-size=0 to repack everything into a single pack-file. This\nis not sustaintable for a large repository. The amount of work\nrequired is also likely to use too many system resources for a\nbackground job.\n\nUpdate the 'git run-job pack-files' command by allowing a direct\n--batch-size option that can change the value provided. Update the\ndefault of \"0\" to a computed size based on the existing pack-files.\nWhile computing that new size, count the existing pack-files and\nskip the repack step if there are at most two pack-files.\n\nThe dynamic default size is computed with this idea in mind for\na client repository that was cloned from a very large remote: there\nis likely one \"big\" pack-file that was created at clone time. Thus,\ndo not try repacking it as it is likely packed efficiently by the\nserver. Instead, try packing the other pack-files into a single\npack-file.\n\nThe size is then computed as follows:\n\n\tbatch size = total size - max pack size\n\nThe batch size is then also limited to be at most two gigabytes.\nThis serves two purposes. First, having a limit prevents doing too\nmuch work when the repository is extremely large. Pack-files\nlarger than two gigabytes are likely to either contain large blobs\nor have been carefully repacked by a previous repack operation.\nSecond, two gigabytes is the size limit for a signed 32-bit int.\nIt's a good limit to consider, and to keep it far away from the\nunsigned 32-bit int limit. This limit comes to mind because on\nWindows an \"unsigned long\" is 32 bits and OPT_MAGNITUDE() uses\nunsigned longs for its parsing logic.\n\nThis calculation mimics a similar calculation in Scalar [1],\nexcept for a 3% drop in the calculated batch size due to the\nround-off error that can happen with the \"expected size\"\ncalculation for a pack-file.\n\n[1] https://github.com/microsoft/scalar/blob/616e9b16dd120b8fdb652d6d5a55618c731a8aea/Scalar.Common/Maintenance/PackfileMaintenanceStep.cs#L141-L143\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-run-job.txt | 13 +++--\n builtin/run-job.c | 90 ++++++++++++++++++++++++++++++++---\n t/t7900-run-job.sh | 6 ++-\n 3 files changed, 96 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nindex 108ed25b8bd..cdd6417f7c9 100644\n--- a/Documentation/git-run-job.txt\n+++ b/Documentation/git-run-job.txt\n@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.\n SYNOPSIS\n --------\n [verse]\n-'git run-job (commit-graph|fetch|loose-objects|pack-files)'\n+'git run-job (commit-graph|fetch|loose-objects|pack-files) []'\n \n \n DESCRIPTION\n@@ -82,9 +82,14 @@ only happens if all objects in the pack-file are also stored in a newer\n pack-file. Second, it selects a group of pack-files whose \"expected\n size\" is below the batch size until the group has total expected size at\n least the batch size; see the `--batch-size` option for the `repack`\n-subcommand in linkgit:git-multi-pack-index[1]. The default batch-size is\n-zero, which is a special case that attempts to repack all pack-files\n-into a single pack-file.\n+subcommand in linkgit:git-multi-pack-index[1].\n++\n+The default batch size is computed to optimize for having a single large\n+pack-file and many small pack-files. When there are two or fewer\n+pack-files, the job does not attempt to repack. Otherwise, the batch\n+size is the sum of all pack-file sizes minus the largest pack-file size.\n+The batch size is capped at two gigabytes. This intends to pack all\n+small pack-files into a single pack-file.\n \n \n GIT\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nindex d3543f7ccb9..2ccc3bbae2d 100644\n--- a/builtin/run-job.c\n+++ b/builtin/run-job.c\n@@ -3,12 +3,18 @@\n #include \"commit-graph.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"packfile.h\"\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"run-command.h\"\n \n static char const * const builtin_run_job_usage[] = {\n-\tN_(\"git run-job (commit-graph|fetch|loose-objects|pack-files)\"),\n+\tN_(\"git run-job (commit-graph|fetch|loose-objects|pack-files) []\"),\n+\tNULL\n+};\n+\n+static char const * const builtin_run_job_pack_file_usage[] = {\n+\tN_(\"git run-job pack-files [--batch-size=]\"),\n \tNULL\n };\n \n@@ -278,15 +284,74 @@ static int multi_pack_index_expire(void)\n \treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n }\n \n-static int multi_pack_index_repack(void)\n+#define TWO_GIGABYTES (2147483647)\n+\n+static off_t get_auto_pack_size(int *count)\n+{\n+\t/*\n+\t * The \"auto\" value is special: we optimize for\n+\t * one large pack-file (i.e. from a clone) and\n+\t * expect the rest to be small and they can be\n+\t * repacked quickly. Find the sum of the sizes\n+\t * other than the largest pack-file, then use\n+\t * that as the batch size.\n+\t */\n+\toff_t total_size = 0;\n+\toff_t max_size = 0;\n+\toff_t result_size;\n+\tstruct packed_git *p;\n+\n+\t*count = 0;\n+\n+\treprepare_packed_git(the_repository);\n+\tfor (p = get_all_packs(the_repository); p; p = p->next) {\n+\t\t(*count)++;\n+\t\ttotal_size += p->pack_size;\n+\n+\t\tif (p->pack_size > max_size)\n+\t\t\tmax_size = p->pack_size;\n+\t}\n+\n+\tresult_size = total_size - max_size;\n+\n+\t/* But limit ourselves to a batch size of 2g */\n+\tif (result_size > TWO_GIGABYTES)\n+\t\tresult_size = TWO_GIGABYTES;\n+\n+\treturn result_size;\n+}\n+\n+#define UNSET_BATCH_SIZE ((unsigned long)-1)\n+static int multi_pack_index_repack(unsigned long batch_size)\n {\n \tint result;\n \tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\tstruct strbuf batch_arg = STRBUF_INIT;\n+\tint count;\n+\toff_t default_size = get_auto_pack_size(&count);\n+\n+\tif (count <= 2)\n+\t\treturn 0;\n+\n+\tstrbuf_addstr(&batch_arg, \"--batch-size=\");\n+\n+\tif (batch_size != UNSET_BATCH_SIZE)\n+\t\tstrbuf_addf(&batch_arg, \"\\\"%\"PRIuMAX\"\\\"\", (uintmax_t)batch_size);\n+\telse\n+\t\tstrbuf_addf(&batch_arg, \"%\"PRIuMAX,\n+\t\t\t (uintmax_t)default_size);\n+\n \targv_array_pushl(&cmd, \"multi-pack-index\", \"repack\",\n-\t\t\t \"--no-progress\", \"--batch-size=0\", NULL);\n+\t\t\t \"--no-progress\", batch_arg.buf, NULL);\n \tresult = run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n \n-\tif (result && multi_pack_index_verify()) {\n+\tstrbuf_release(&batch_arg);\n+\n+\t/*\n+\t * Verify here to avoid verifying again when there are two\n+\t * or fewer pack-files.\n+\t */\n+\tif (!result && multi_pack_index_verify()) {\n \t\twarning(_(\"multi-pack-index verify failed after repack\"));\n \t\tresult = rewrite_multi_pack_index();\n \t}\n@@ -294,8 +359,19 @@ static int multi_pack_index_repack(void)\n \treturn result;\n }\n \n-static int run_pack_files_job(void)\n+static int run_pack_files_job(int argc, const char **argv)\n {\n+\tstatic unsigned long batch_size = UNSET_BATCH_SIZE;\n+\tstatic struct option builtin_run_job_pack_file_options[] = {\n+\t\tOPT_MAGNITUDE(0, \"batch-size\", &batch_size,\n+\t\t\t N_(\"specify a batch-size for the incremental repack\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_run_job_pack_file_options,\n+\t\t\t builtin_run_job_pack_file_usage, 0);\n+\n \tif (multi_pack_index_write()) {\n \t\terror(_(\"failed to write multi-pack-index\"));\n \t\treturn 1;\n@@ -316,7 +392,7 @@ static int run_pack_files_job(void)\n \t\treturn rewrite_multi_pack_index();\n \t}\n \n-\tif (multi_pack_index_repack()) {\n+\tif (multi_pack_index_repack(batch_size)) {\n \t\terror(_(\"multi-pack-index repack failed\"));\n \t\treturn 1;\n \t}\n@@ -348,7 +424,7 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)\n \t\tif (!strcmp(argv[0], \"loose-objects\"))\n \t\t\treturn run_loose_objects_job();\n \t\tif (!strcmp(argv[0], \"pack-files\"))\n-\t\t\treturn run_pack_files_job();\n+\t\t\treturn run_pack_files_job(argc, argv);\n \t}\n \n \tusage_with_options(builtin_run_job_usage,\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nindex 416ba04989d..2d9f6cdf328 100755\n--- a/t/t7900-run-job.sh\n+++ b/t/t7900-run-job.sh\n@@ -128,8 +128,10 @@ test_expect_success 'pack-files job' '\n \t# the job deletes the two old packs, and does not write\n \t# a new one because only one pack remains.\n \tgit -C client run-job pack-files &&\n-\tls client/.git/objects/pack/*.pack >packs-after &&\n-\ttest_line_count = 1 packs-after\n+\tls client/$packDir/*.pack >packs-after &&\n+\ttest_line_count = 2 packs-after &&\n+\tcat packs-after | grep \"pack/test-1-\" &&\n+\tcat packs-after | grep \"pack/pack-\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"394706","messageId":"4c4aeb55dea9dd7b97bd7f08c8a767b498886109.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 04/15] run-job: implement loose-objects job","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:03Z","receivedAt":"2020-04-03T20:48:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOne goal of background maintenance jobs is to allow a user to\ndisable auto-gc (gc.auto=0) but keep their repository in a clean\nstate. Without any cleanup, loose objects will clutter the object\ndatabase and slow operations. In addition, the loose objects will\ntake up extra space because they are not stored with deltas against\nsimilar objects.\n\nCreate a 'loose-objects' job for the 'git run-job' command. This\nhelps clean up loose objects without disrupting concurrent Git\ncommands using the following sequence of events:\n\n1. Run 'git prune-packed' to delete any loose objects that exist\n in a pack-file. Concurrent commands will prefer the packed\n version of the object to the loose version. (Of course, there\n are exceptions for commands that specifically care about the\n location of an object. These are rare for a user to run on\n purpose, and we hope a user that has selected background\n maintenance will not be trying to do foreground maintenance.)\n\n2. Run 'git pack-objects' on a batch of loose objects. These\n objects are grouped by scanning the loose object directories in\n lexicographic order until listing all loose objects -or-\n reaching 50,000 objects. This is more than enough if the loose\n objects are created only by a user doing normal development.\n We noticed users with _millions_ of loose objects because VFS\n for Git downloads blobs on-demand when a file read operation\n requires populating a virtual file. This has potential of\n happening in partial clones if someone runs 'git grep' or\n otherwise evades the batch-download feature for requesting\n promisor objects.\n\nThis step is based on a similar step in Scalar [1] and VFS for Git.\n[1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/LooseObjectsStep.cs\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-run-job.txt | 14 ++++-\n builtin/run-job.c | 97 ++++++++++++++++++++++++++++++++++-\n t/t7900-run-job.sh | 27 ++++++++++\n 3 files changed, 136 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-run-job.txt b/Documentation/git-run-job.txt\nindex eb92e891915..43ca1160b5a 100644\n--- a/Documentation/git-run-job.txt\n+++ b/Documentation/git-run-job.txt\n@@ -9,7 +9,7 @@ git-run-job - Run a maintenance job. Intended for background operation.\n SYNOPSIS\n --------\n [verse]\n-'git run-job (commit-graph|fetch)'\n+'git run-job (commit-graph|fetch|loose-objects)'\n \n \n DESCRIPTION\n@@ -59,6 +59,18 @@ This means that foreground fetches are still required to update the\n remote refs, but the users is notified when the branches and tags are\n updated on the remote.\n \n+'loose-objects'::\n+\n+The `loose-objects` job cleans up loose objects and places them into\n+pack-files. In order to prevent race conditions with concurrent Git\n+commands, it follows a two-step process. First, it deletes any loose\n+objects that already exist in a pack-file; concurrent Git processes will\n+examine the pack-file for the object data instead of the loose object.\n+Second, it creates a new pack-file (starting with \"loose-\") containing\n+a batch of loose objects. The batch size is limited to 50 thousand\n+objects to prevent the job from taking too long on a repository with\n+many loose objects.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/run-job.c b/builtin/run-job.c\nindex e59056b2918..cecf9058c51 100644\n--- a/builtin/run-job.c\n+++ b/builtin/run-job.c\n@@ -7,7 +7,7 @@\n #include \"run-command.h\"\n \n static char const * const builtin_run_job_usage[] = {\n-\tN_(\"git run-job (commit-graph|fetch)\"),\n+\tN_(\"git run-job (commit-graph|fetch|loose-objects)\"),\n \tNULL\n };\n \n@@ -145,6 +145,99 @@ static int run_fetch_job(void)\n \treturn result;\n }\n \n+static int prune_packed(void)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"prune-packed\", NULL);\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+struct write_loose_object_data {\n+\tFILE *in;\n+\tint count;\n+\tint batch_size;\n+};\n+\n+static int loose_object_exists(const struct object_id *oid,\n+\t\t\t const char *path,\n+\t\t\t void *data)\n+{\n+\treturn 1;\n+}\n+\n+static int write_loose_object_to_stdin(const struct object_id *oid,\n+\t\t\t\t const char *path,\n+\t\t\t\t void *data)\n+{\n+\tstruct write_loose_object_data *d = (struct write_loose_object_data *)data;\n+\n+\tfprintf(d->in, \"%s\\n\", oid_to_hex(oid));\n+\n+\treturn ++(d->count) > d->batch_size;\n+}\n+\n+static int pack_loose(void)\n+{\n+\tint result = 0;\n+\tstruct write_loose_object_data data;\n+\tstruct strbuf prefix = STRBUF_INIT;\n+\tstruct child_process *pack_proc;\n+\n+\t/*\n+\t * Do not start pack-objects process\n+\t * if there are no loose objects.\n+\t */\n+\tif (!for_each_loose_file_in_objdir(the_repository->objects->odb->path,\n+\t\t\t\t\t loose_object_exists,\n+\t\t\t\t\t NULL, NULL, NULL))\n+\t\treturn 0;\n+\n+\tpack_proc = xmalloc(sizeof(*pack_proc));\n+\n+\tchild_process_init(pack_proc);\n+\n+\tstrbuf_addstr(&prefix, the_repository->objects->odb->path);\n+\tstrbuf_addstr(&prefix, \"/pack/loose\");\n+\n+\targv_array_pushl(&pack_proc->args, \"git\", \"pack-objects\",\n+\t\t\t \"--quiet\", prefix.buf, NULL);\n+\n+\tpack_proc->in = -1;\n+\n+\tif (start_command(pack_proc)) {\n+\t\terror(_(\"failed to start 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n+\tdata.in = xfdopen(pack_proc->in, \"w\");\n+\tdata.count = 0;\n+\tdata.batch_size = 50000;\n+\n+\tfor_each_loose_file_in_objdir(the_repository->objects->odb->path,\n+\t\t\t\t write_loose_object_to_stdin,\n+\t\t\t\t NULL,\n+\t\t\t\t NULL,\n+\t\t\t\t &data);\n+\n+\tfclose(data.in);\n+\n+\tif (finish_command(pack_proc)) {\n+\t\terror(_(\"failed to finish 'git pack-objects' process\"));\n+\t\tresult = 1;\n+\t}\n+\n+cleanup:\n+\tstrbuf_release(&prefix);\n+\tfree(pack_proc);\n+\treturn result;\n+}\n+\n+static int run_loose_objects_job(void)\n+{\n+\treturn prune_packed() || pack_loose();\n+}\n+\n int cmd_run_job(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_run_job_options[] = {\n@@ -166,6 +259,8 @@ int cmd_run_job(int argc, const char **argv, const char *prefix)\n \t\t\treturn run_commit_graph_job();\n \t\tif (!strcmp(argv[0], \"fetch\"))\n \t\t\treturn run_fetch_job();\n+\t\tif (!strcmp(argv[0], \"loose-objects\"))\n+\t\t\treturn run_loose_objects_job();\n \t}\n \n \tusage_with_options(builtin_run_job_usage,\ndiff --git a/t/t7900-run-job.sh b/t/t7900-run-job.sh\nindex d3faeba135b..41da083257b 100755\n--- a/t/t7900-run-job.sh\n+++ b/t/t7900-run-job.sh\n@@ -66,4 +66,31 @@ test_expect_success 'fetch job' '\n \ttest_line_count = 2 $chain\n '\n \n+test_expect_success 'loose-objects job' '\n+\tls client/.git/objects >obj-dir-before &&\n+\ttest_file_not_empty obj-dir-before &&\n+\tls client/.git/objects/pack/*.pack >packs-before &&\n+\ttest_line_count = 1 packs-before &&\n+\n+\t# The first run creates a pack-file\n+\t# but does not delete loose objects.\n+\tgit -C client run-job loose-objects &&\n+\tls client/.git/objects >obj-dir-between &&\n+\ttest_cmp obj-dir-before obj-dir-between &&\n+\tls client/.git/objects/pack/*.pack >packs-between &&\n+\ttest_line_count = 2 packs-between &&\n+\n+\t# The second run deletes loose objects\n+\t# but does not create a pack-file.\n+\tgit -C client run-job loose-objects &&\n+\tls client/.git/objects >obj-dir-after &&\n+\tcat >expect <<-\\EOF &&\n+\tinfo\n+\tpack\n+\tEOF\n+\ttest_cmp expect obj-dir-after &&\n+\tls client/.git/objects/pack/*.pack >packs-after &&\n+\ttest_cmp packs-between packs-after\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"394711","messageId":"9104ae46371a08967805f6ce27586ccade37972a.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 09/15] job-runner: load repos from config by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:08Z","receivedAt":"2020-04-03T20:48:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git job-runner' command was introduced with a '--repo='\noption so a user could start a process with an explicit list of\nrepos. However, it is more likely that we will want 'git\njob-runner' to start without any arguments and discover the set of\nrepos from another source.\n\nA natural source is the Git config. The 'git job-runner' does not\nneed to run in a Git repository, but the config could be located in\nthe global or system config.\n\nCreate the job.repo config setting as a multi-valued config option.\n\nRead all values for job.repo whenever triggering an iteration of\nthe job loop. This allows a user to add or remove repos dynamically\nwithout restarting the job-runner.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/job.txt | 7 +++++++\n builtin/job-runner.c | 20 ++++++++++++++++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/Documentation/config/job.txt b/Documentation/config/job.txt\nindex efdb76afad3..2dfed3935fa 100644\n--- a/Documentation/config/job.txt\n+++ b/Documentation/config/job.txt\n@@ -4,3 +4,10 @@ job.pack-files.batchSize::\n \tpart of `git run-job pack-files`. If not specified, then a\n \tdynamic size calculation is run. See linkgit:git-run-job[1]\n \tfor more details.\n+\n+job.repo::\n+\tStore an absolute path to a repository that wants background\n+\tjobs run by `git job-runner`. This is a multi-valued config\n+\toption, but it must be stored in a config file seen by the\n+\tbackground runner. This may be the global or system config\n+\tdepending on how `git job-runner` is launched on your system.\ndiff --git a/builtin/job-runner.c b/builtin/job-runner.c\nindex 135288bcaae..bed401f94bf 100644\n--- a/builtin/job-runner.c\n+++ b/builtin/job-runner.c\n@@ -20,6 +20,9 @@ static int arg_repos_append(const struct option *opt,\n \n static int load_active_repos(struct string_list *repos)\n {\n+\tstruct string_list_item *item;\n+\tconst struct string_list *config_repos;\n+\n \tif (arg_repos.nr) {\n \t\tstruct string_list_item *item;\n \t\tfor (item = arg_repos.items;\n@@ -29,6 +32,23 @@ static int load_active_repos(struct string_list *repos)\n \t\treturn 0;\n \t}\n \n+\tconfig_repos = git_config_get_value_multi(\"job.repo\");\n+\n+\tfor (item = config_repos->items;\n+\t item && item < config_repos->items + config_repos->nr;\n+\t item++) {\n+\t\tDIR *dir = opendir(item->string);\n+\n+\t\tif (!dir)\n+\t\t\tcontinue;\n+\n+\t\tclosedir(dir);\n+\t\tstring_list_append(repos, item->string);\n+\t}\n+\n+\tstring_list_sort(repos);\n+\tstring_list_remove_duplicates(repos, 0);\n+\n \treturn 0;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"394715","messageId":"90379bad303e77cc82ef79fc2ba505805a571483.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 08/15] job-runner: create builtin for job loop","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:07Z","receivedAt":"2020-04-03T20:48:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nGiven the existing 'git run-job ' builtin, users _could_\nconstruct their own scheduling mechanism for running jobs in the\nbackground. However, it is much easier to have a dedicated process\nthat manages all jobs across multiple repos.\n\nThe 'git job-runner' builtin is specifically built to handle this\nscenario. It will be customized further in later changes, but for\nnow it does the following:\n\n* Given a list of '--repo=' arguments, construct a list of\n repositories to manage with jobs.\n\n* Every 30 minutes, iterate over all jobs and all repos to run\n\n\tgit -C run-job \n\nThis builtin needs to be careful about how much of the Git internals\nit consumes. The intention is that this is a long-lived process that\ncould be launched upon login and only closed on logout. For that\nreason, we will avoid instantiating any object store or index data.\n\nRun the maintenance jobs by running subcommands. We will update how\nwe enable or disable these jobs and separate their runs in a later\nchange.\n\nRFC QUESTIONS:\n\n1. The hardest part of this builtin is \"how do we test it?\" In\n Scalar, we can unit test the scheduler with mocks. What is the\n equivalent here for \"make sure 'git job-runner' runs 'git run-job\n pack-files' on repo X? I expect to add a \"--no-loop\" option that\n ensures the logic only runs one iteration of the loop.\n\n2. The difference between 'git run-job' and 'git job-runner' is\n subtle and probably confusing. Are there better names?\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-job-runner.txt | 52 +++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/job-runner.c | 111 +++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 168 insertions(+)\n create mode 100644 Documentation/git-job-runner.txt\n create mode 100644 builtin/job-runner.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 5dea9d3b96b..24c377b2883 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -83,6 +83,7 @@\n /git-init-db\n /git-interpret-trailers\n /git-instaweb\n+/git-job-runner\n /git-log\n /git-ls-files\n /git-ls-remote\ndiff --git a/Documentation/git-job-runner.txt b/Documentation/git-job-runner.txt\nnew file mode 100644\nindex 00000000000..010b2d05f9b\n--- /dev/null\n+++ b/Documentation/git-job-runner.txt\n@@ -0,0 +1,52 @@\n+git-job-runner(1)\n+=================\n+\n+NAME\n+----\n+git-job-runner - run jobs on multiple repos according to a schedlue.\n+Intended for background operation.\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git job-runner [--repo=]'\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Run jobs in a loop with some frequency. Intended for running in the\n+background.\n+\n+The `git run-job ` command runs a specific maintenance task.\n+The `job-runner` command is a long-running process that calls the\n+`run-job` command on a set of repositories at some frequency. The\n+`job.*` config values customize the frequencies of these jobs.\n+\n+\n+OPTIONS\n+-------\n+\n+--repo=::\n+\tIf at least one `--repo` option is provided, the runner only\n+\tattempts running jobs on repositories located at the provided\n+\t`` values. This option can be specified multiple times.\n+\n+\n+CONFIGURATION\n+-------------\n+\n+The `git job-runner` command is intended to run as a long-running\n+process. The following config options are checked periodically during\n+the process and will modify its behavior:\n+\n+The below documentation is the same as what's found in\n+linkgit:git-config[1]:\n+\n+include::config/job.txt[]\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex f5f9c4d9e94..ee98a91af0f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1082,6 +1082,7 @@ BUILTIN_OBJS += builtin/help.o\n BUILTIN_OBJS += builtin/index-pack.o\n BUILTIN_OBJS += builtin/init-db.o\n BUILTIN_OBJS += builtin/interpret-trailers.o\n+BUILTIN_OBJS += builtin/job-runner.o\n BUILTIN_OBJS += builtin/log.o\n BUILTIN_OBJS += builtin/ls-files.o\n BUILTIN_OBJS += builtin/ls-remote.o\ndiff --git a/builtin.h b/builtin.h\nindex 3e0ddaaf67f..5008c7096b3 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -176,6 +176,7 @@ int cmd_help(int argc, const char **argv, const char *prefix);\n int cmd_index_pack(int argc, const char **argv, const char *prefix);\n int cmd_init_db(int argc, const char **argv, const char *prefix);\n int cmd_interpret_trailers(int argc, const char **argv, const char *prefix);\n+int cmd_job_runner(int argc, const char **argv, const char *prefix);\n int cmd_log(int argc, const char **argv, const char *prefix);\n int cmd_log_reflog(int argc, const char **argv, const char *prefix);\n int cmd_ls_files(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/job-runner.c b/builtin/job-runner.c\nnew file mode 100644\nindex 00000000000..135288bcaae\n--- /dev/null\n+++ b/builtin/job-runner.c\n@@ -0,0 +1,111 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static char const * const builtin_job_runner_usage[] = {\n+\tN_(\"git job-runner []\"),\n+\tNULL\n+};\n+\n+static struct string_list arg_repos = STRING_LIST_INIT_DUP;\n+\n+static int arg_repos_append(const struct option *opt,\n+\t\t\t const char *arg, int unset)\n+{\n+\tstring_list_append(&arg_repos, arg);\n+\treturn 0;\n+}\n+\n+static int load_active_repos(struct string_list *repos)\n+{\n+\tif (arg_repos.nr) {\n+\t\tstruct string_list_item *item;\n+\t\tfor (item = arg_repos.items;\n+\t\t item && item < arg_repos.items + arg_repos.nr;\n+\t\t item++)\n+\t\t\tstring_list_append(repos, item->string);\n+\t\treturn 0;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int run_job(const char *job, const char *repo)\n+{\n+\tstruct argv_array cmd = ARGV_ARRAY_INIT;\n+\targv_array_pushl(&cmd, \"-C\", repo, \"run-job\", job, NULL);\n+\treturn run_command_v_opt(cmd.argv, RUN_GIT_CMD);\n+}\n+\n+static int run_job_loop_step(struct string_list *list)\n+{\n+\tint result = 0;\n+\tstruct string_list_item *job;\n+\tstruct string_list repos = STRING_LIST_INIT_DUP;\n+\n+\tif ((result = load_active_repos(&repos)))\n+\t\treturn result;\n+\n+\tfor (job = list->items;\n+\t !result && job && job < list->items + list->nr;\n+\t job++) {\n+\t\tstruct string_list_item *repo;\n+\t\tfor (repo = repos.items;\n+\t\t !result && repo && repo < repos.items + repos.nr;\n+\t\t repo++)\n+\t\t\tresult = run_job(job->string,\n+\t\t\t\t\t repo->string);\n+\t}\n+\n+\tstring_list_clear(&repos, 0);\n+\treturn result;\n+}\n+\n+static unsigned int get_loop_interval(void)\n+{\n+\t/* Default: 30 minutes */\n+\treturn 30 * 60;\n+}\n+\n+static int initialize_jobs(struct string_list *list)\n+{\n+\tstring_list_append(list, \"commit-graph\");\n+\tstring_list_append(list, \"fetch\");\n+\tstring_list_append(list, \"loose-objects\");\n+\tstring_list_append(list, \"pack-files\");\n+\treturn 0;\n+}\n+\n+int cmd_job_runner(int argc, const char **argv, const char *prefix)\n+{\n+\tint result;\n+\tstruct string_list job_list = STRING_LIST_INIT_DUP;\n+\tstatic struct option builtin_job_runner_options[] = {\n+\t\tOPT_CALLBACK_F(0, \"repo\",\n+\t\t\t NULL,\n+\t\t\t N_(\"\"),\n+\t\t\t N_(\"run jobs on the repository at \"),\n+\t\t\t PARSE_OPT_NONEG, arg_repos_append),\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_job_runner_usage,\n+\t\t\t\t builtin_job_runner_options);\n+\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_job_runner_options,\n+\t\t\t builtin_job_runner_usage,\n+\t\t\t 0);\n+\n+\tresult = initialize_jobs(&job_list);\n+\n+\twhile (!(result = run_job_loop_step(&job_list))) {\n+\t\tunsigned int interval = get_loop_interval();\n+\t\tsleep(interval);\n+\t}\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 1cd2b415e46..cb835c74693 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -110,6 +110,7 @@ git-init mainporcelain init\n git-instaweb ancillaryinterrogators complete\n git-interpret-trailers purehelpers\n gitk mainporcelain\n+git-job-runner plumbingmanipulators\n git-log mainporcelain info\n git-ls-files plumbinginterrogators\n git-ls-remote plumbinginterrogators\ndiff --git a/git.c b/git.c\nindex db5a43c8687..3a2da2c232f 100644\n--- a/git.c\n+++ b/git.c\n@@ -517,6 +517,7 @@ static struct cmd_struct commands[] = {\n \t{ \"init\", cmd_init_db },\n \t{ \"init-db\", cmd_init_db },\n \t{ \"interpret-trailers\", cmd_interpret_trailers, RUN_SETUP_GENTLY },\n+\t{ \"job-runner\", cmd_job_runner, RUN_SETUP_GENTLY },\n \t{ \"log\", cmd_log, RUN_SETUP },\n \t{ \"ls-files\", cmd_ls_files, RUN_SETUP },\n \t{ \"ls-remote\", cmd_ls_remote, RUN_SETUP_GENTLY },\n-- \ngitgitgadget\n\n"},{"id":"394708","messageId":"2c3c9d28a903e23c37e44f9783299f3c001f574e.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 11/15] job-runner: use config for loop interval","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:10Z","receivedAt":"2020-04-03T20:48:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git job-runner' process uses a sleep(X) call to pause its\noperation between iterations of the job loop. This defaults to 30\nminutes, but is now available to be done with longer or shorter\nintervals according to the job.loopInterval config option. For\nexample, a user may want the job loop to run once every five\nminutes while another wants the job loop to run once every six\nhours.\n\nThis config value is checked immediately before the sleep(X) call,\nwhich allows users to see the effect without restarting the\njob-runner process. However, the process will be paused until the\nprevious sleep(X) call returns and the job loop is executed.\n\nRFC QUESITON: Is this use of sleep(X) the best way to do this?\nIs there a better way to delay the process for a time interval, or\nuntil a specified time? This just seemed like the simplest option.\nThe job-runner is doing low-priority work on an unpredictable\nschedule by design, so sleep(X) seemd appropriate.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/job.txt | 4 ++++\n builtin/job-runner.c | 6 +++++-\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config/job.txt b/Documentation/config/job.txt\nindex 7c799d66221..772001e6744 100644\n--- a/Documentation/config/job.txt\n+++ b/Documentation/config/job.txt\n@@ -1,3 +1,7 @@\n+job.loopInterval::\n+\tThe number of seconds to sleep between rounds of running\n+\tbackground jobs in `git job-runner`.\n+\n job..interval::\n \tThe minimum number of seconds between runs of\n \t`git run-job ` when running `git job-runner`.\ndiff --git a/builtin/job-runner.c b/builtin/job-runner.c\nindex aee55c106e8..7e37b122d99 100644\n--- a/builtin/job-runner.c\n+++ b/builtin/job-runner.c\n@@ -261,7 +261,11 @@ static int run_job_loop_step(struct string_list *list)\n static unsigned int get_loop_interval(void)\n {\n \t/* Default: 30 minutes */\n-\treturn 30 * 60;\n+\ttimestamp_t interval = 30 * 60;\n+\n+\ttry_get_timestamp(NULL, \".\", \"loopinterval\", &interval);\n+\n+\treturn interval;\n }\n \n static int initialize_jobs(struct string_list *list)\n-- \ngitgitgadget\n\n"},{"id":"394713","messageId":"18429182ffcf4cb56fcf3d8c76e41652a6c45adc.1585946894.git.gitgitgadget@gmail.com","threadId":"53164","inReplyTo":"pull.597.git.1585946894.gitgitgadget@gmail.com","subject":"[PATCH 10/15] job-runner: use config to limit job frequency","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-03T20:48:09Z","receivedAt":"2020-04-03T20:48:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe want to run different maintenance tasks at different rates. That\nmeans we cannot rely only on the time between job-runner loop pauses\nto reduce the frequency of specific jobs. This means we need to\npersist a timestamp for the previous run somewhere. A natural place\nis the Git config file for that repo.\n\nCreate the job..lastRun config option to store the\ntimestamp of the previous run of that job. Set this config option\nafter a successful run of 'git -C run-job '.\n\nTo maximize code reuse, we dynamically construct the config key\nand parse the config value into a timestamp in a generic way. This\nmakes the introduction of another config option extremely trivial:\n\nThe job..interval option allows a user to specify a\nminimum number of seconds between two calls to 'git run-job\n' on a given repo. This could be stored in the global\nor system config to provide an update on the default for all repos,\nor could be updated on a per-repo basis. This is checked on every\niteration of the job loop, so a user could update this and see the\neffect without restarting the job-runner process.\n\nRFC QUESTION: I'm using a 'git -C config