[PATCH v3 0/4] Maintenance IV: Platform-specific background maintenance
- From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
- Date
- Nov 13, 2020, 14:00 UTC
- Message-ID
- <pull.776.v3.git.1605276024.gitgitgadget@gmail.com>
- In-Reply-To
- <pull.776.v2.git.1604520368.gitgitgadget@gmail.com>
This is based on ds/maintenance-part-3.
After sitting with the background maintenance as it has been cooking, I wanted to come back around and implement the background maintenance for Windows. However, I noticed that there were some things bothering me with background maintenance on my macOS machine. These are detailed in PATCH 3, but the tl;dr is that 'cron' is not recommended by Apple and instead 'launchd' satisfies our needs.
This series implements the background scheduling so git maintenance (start|stop) works on those platforms. I've been operating with these schedules for a while now without the problems described in the patches.
There is a particularly annoying case about console windows popping up on Windows, but PATCH 4 describes a plan to get around that.
Updates in V3 =============
* This actually includes the feedback responses I had intended for v2. Sorry about that! * One major change is the use of a 'struct child_process' instead of just run_command_v_opt() so we can suppress error messages from the schedule helpers. We will rely on exit code and present our own error messages, as necessary. * Some doc and test fixes.
Updates in V2 =============
* This is a faster turnaround for a v2 than I would normally like, but Eric inspired extra documentation about how to customize background schedules. * New extensions to git-maintenance.txt include guidelines for inspecting what git maintenance start does and how to customize beyond that. This includes a new PATCH 2 that includes documentation for 'cron' on non-macOS non-Windows systems. * Several improvements, especially in the tests, are included. * While testing manually, I noticed that somehow I had incorrectly had an opening <dict> tag instead of a closing </dict> tag in the hourly format on macOS. I found that the xmllint tool can verify the XML format of a file, which catches the bug. This seems like a good approach since the test is macOS-only. Does anyone have concerns about adding this dependency?
Thanks, -Stolee
cc: jrnieder@gmail.com [jrnieder@gmail.com], jonathantanmy@google.com [jonathantanmy@google.com], sluongng@gmail.com [sluongng@gmail.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com]cc: Đoàn Trần Công Danh congdanhqx@gmail.com [congdanhqx@gmail.com]cc: Martin Ågren martin.agren@gmail.com [martin.agren@gmail.com]cc: Eric Sunshine sunshine@sunshineco.com [sunshine@sunshineco.com]cc: Derrick Stolee stolee@gmail.com [stolee@gmail.com]
Derrick Stolee (4): maintenance: extract platform-specific scheduling maintenance: include 'cron' details in docs maintenance: use launchctl on macOS maintenance: use Windows scheduled tasks
Documentation/git-maintenance.txt | 116 +++++++++ builtin/gc.c | 417 ++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 75 +++++- t/test-lib.sh | 4 + 4 files changed, 592 insertions(+), 20 deletions(-)
base-commit: 0016b618182f642771dc589cf0090289f9fe1b4f Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-776%2Fderrickstolee%2Fmaintenance%2FmacOS-v3 Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-776/derrickstolee/maintenance/macOS-v3 Pull-Request: https://github.com/gitgitgadget/git/pull/776
Range-diff vs v2:
1: d35f1aa162 = 1: d35f1aa162 maintenance: extract platform-specific scheduling
2: 709a173720 ! 2: 0dfe53092e maintenance: include 'cron' details in docs
@@ Commit message
baseline can provide a way forward for users who have never worked with
cron schedules.
+ Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
## Documentation/git-maintenance.txt ##
@@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c
+---------------------------------------
+
+The standard mechanism for scheduling background tasks on POSIX systems
-+is `cron`. This tool executes commands based on a given schedule. The
++is cron(8). This tool executes commands based on a given schedule. The
+current list of user-scheduled tasks can be found by running `crontab -l`.
+The schedule written by `git maintenance start` is similar to this:
+
@@ Documentation/git-maintenance.txt: Further, the `git gc` command should not be c
+Any modifications within this region will be completely deleted by
+`git maintenance stop` or overwritten by `git maintenance start`.
+
-+The `<path>` string is loaded to specifically use the location for the
-+`git` executable used in the `git maintenance start` command. This allows
-+for multiple versions to be compatible. However, if the same user runs
-+`git maintenance start` with multiple Git executables, then only the
-+latest executable will be used.
++The `crontab` entry specifies the full path of the `git` executable to
++ensure that the executed `git` command is the same one with which
++`git maintenance start` was issued independent of `PATH`. If the same user
++runs `git maintenance start` with multiple Git executables, then only the
++latest executable is used.
+
+These commands use `git for-each-repo --config=maintenance.repo` to run
+`git maintenance run --schedule=<frequency>` on each repository listed in
+the multi-valued `maintenance.repo` config option. These are typically
-+loaded from the user-specific global config located at `~/.gitconfig`.
-+The `git maintenance` process then determines which maintenance tasks
-+are configured to run on each repository with each `<frequency>` using
-+the `maintenance.<task>.schedule` config options. These values are loaded
-+from the global or repository config values.
++loaded from the user-specific global config. The `git maintenance` process
++then determines which maintenance tasks are configured to run on each
++repository with each `<frequency>` using the `maintenance.<task>.schedule`
++config options. These values are loaded from the global or repository
++config values.
+
+If the config values are insufficient to achieve your desired background
+maintenance schedule, then you can create your own schedule. If you run
+`crontab -e`, then an editor will load with your user-specific `cron`
+schedule. In that editor, you can add your own schedule lines. You could
+start by adapting the default schedule listed earlier, or you could read
-+https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation]
-+for advanced scheduling techniques. Please do use the full path and
-+`--exec-path` techniques from the default schedule to ensure you are
-+executing the correct binaries in your schedule.
++the crontab(5) documentation for advanced scheduling techniques. Please
++do use the full path and `--exec-path` techniques from the default
++schedule to ensure you are executing the correct binaries in your
++schedule.
+
GIT
3: 0fafd75d10 ! 3: 1629bcfcf8 maintenance: use launchctl on macOS
@@ Commit message
of macOS 10.11, which was released in September 2015. Before that
release the 'launchctl load' subcommand was recommended. The best
source of information on this transition I have seen is available
- at [2].
+ at [2]. The current design does not preclude a future version that
+ detects the available fatures of 'launchctl' to use the older
+ commands. However, it is best to rely on the newest version since
+ Apple might completely remove the deprecated version on short
+ notice.
[2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/
@@ Commit message
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
## Documentation/git-maintenance.txt ##
-@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do use the full path and
- executing the correct binaries in your schedule.
+@@ Documentation/git-maintenance.txt: schedule to ensure you are executing the correct binaries in your
+ schedule.
+BACKGROUND MAINTENANCE ON MACOS SYSTEMS
+---------------------------------------
+
+While macOS technically supports `cron`, using `crontab -e` requires
-+elevated privileges and the executed process do not have a full user
++elevated privileges and the executed process does not have a full user
+context. Without a full user context, Git and its credential helpers
+cannot access stored credentials, so some maintenance tasks are not
+functional.
+
+Instead, `git maintenance start` interacts with the `launchctl` tool,
-+which is the recommended way to
-+https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html[schedule timed jobs in macOS].
-+
-+Scheduling maintenance through `git maintenance (start|stop)` requires
-+some `launchctl` features available only in macOS 10.11 or later.
++which is the recommended way to schedule timed jobs in macOS. Scheduling
++maintenance through `git maintenance (start|stop)` requires some
++`launchctl` features available only in macOS 10.11 or later.
+
+Your user-specific scheduled tasks are stored as XML-formatted `.plist`
+files in `~/Library/LaunchAgents/`. You can see the currently-registered
+tasks using the following command:
+
+-----------------------------------------------------------------------
-+$ ls ~/Library/LaunchAgents/ | grep org.git-scm.git
++$ ls ~/Library/LaunchAgents/org.git-scm.git*
+org.git-scm.git.daily.plist
+org.git-scm.git.hourly.plist
+org.git-scm.git.weekly.plist
@@ Documentation/git-maintenance.txt: for advanced scheduling techniques. Please do
+and delete the `.plist` files.
+
+To create more advanced customizations to your background tasks, see
-+https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html#//apple_ref/doc/uid/TP40001762-104142[the `launchctl` documentation]
-+for more information.
++launchctl.plist(5) for more information.
+
+
GIT
@@ builtin/gc.c: static int maintenance_unregister(void)
+ return strbuf_detach(&output, NULL);
+}
+
-+static int bootout(const char *filename)
++static int boot_plist(int enable, const char *filename)
+{
+ int result;
-+ struct strvec args = STRVEC_INIT;
++ struct child_process child = CHILD_PROCESS_INIT;
+ char *uid = get_uid();
+ const char *launchctl = getenv("GIT_TEST_CRONTAB");
+ if (!launchctl)
+ launchctl = "/bin/launchctl";
+
-+ strvec_split(&args, launchctl);
-+ strvec_push(&args, "bootout");
-+ strvec_pushf(&args, "gui/%s", uid);
-+ strvec_push(&args, filename);
++ strvec_split(&child.args, launchctl);
+
-+ result = run_command_v_opt(args.v, 0);
++ if (enable)
++ strvec_push(&child.args, "bootstrap");
++ else
++ strvec_push(&child.args, "bootout");
++ strvec_pushf(&child.args, "gui/%s", uid);
++ strvec_push(&child.args, filename);
+
-+ strvec_clear(&args);
-+ free(uid);
-+ return result;
-+}
++ child.no_stderr = 1;
++ child.no_stdout = 1;
+
-+static int bootstrap(const char *filename)
-+{
-+ int result;
-+ struct strvec args = STRVEC_INIT;
-+ char *uid = get_uid();
-+ const char *launchctl = getenv("GIT_TEST_CRONTAB");
-+ if (!launchctl)
-+ launchctl = "/bin/launchctl";
++ if (start_command(&child))
++ die(_("failed to start launchctl"));
+
-+ strvec_split(&args, launchctl);
-+ strvec_push(&args, "bootstrap");
-+ strvec_pushf(&args, "gui/%s", uid);
-+ strvec_push(&args, filename);
++ result = finish_command(&child);
+
-+ result = run_command_v_opt(args.v, 0);
-+
-+ strvec_clear(&args);
+ free(uid);
+ return result;
+}
@@ builtin/gc.c: static int maintenance_unregister(void)
+ const char *frequency = get_frequency(schedule);
+ char *name = get_service_name(frequency);
+ char *filename = get_service_filename(name);
-+ int result = bootout(filename);
++ int result = boot_plist(0, filename);
++ unlink(filename);
+ free(filename);
+ free(name);
+ return result;
@@ builtin/gc.c: static int maintenance_unregister(void)
+
+ if (safe_create_leading_directories(filename))
+ die(_("failed to create directories for '%s'"), filename);
-+ plist = fopen(filename, "w");
-+
-+ if (!plist)
-+ die(_("failed to open '%s'"), filename);
++ plist = xfopen(filename, "w");
+
+ preamble = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
+ "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n"
@@ builtin/gc.c: static int maintenance_unregister(void)
+ fprintf(plist, "</array>\n</dict>\n</plist>\n");
+
+ /* bootout might fail if not already running, so ignore */
-+ bootout(filename);
-+ if (bootstrap(filename))
++ boot_plist(0, filename);
++ if (boot_plist(1, filename))
+ die(_("failed to bootstrap service %s"), filename);
+
+ fclose(plist);
@@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' '
'
+test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' '
-+ echo "#!/bin/sh\necho \$@ >>args" >print-args &&
-+ chmod a+x print-args &&
++ write_script print-args "#!/bin/sh\necho \$* >>args" &&
+
+ rm -f args &&
+ GIT_TEST_CRONTAB="./print-args" git maintenance start &&
@@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' '
+ for frequency in hourly daily weekly
+ do
+ PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" &&
-+ xmllint "$PLIST" >/dev/null &&
++ xmllint --noout "$PLIST" &&
+ grep schedule=$frequency "$PLIST" &&
+ echo "bootout gui/$UID $PLIST" >>expect &&
+ echo "bootstrap gui/$UID $PLIST" >>expect || return 1
@@ t/t7900-maintenance.sh: test_expect_success 'stop from existing schedule' '
+ test_cmp expect args &&
+
+ rm -f args &&
-+ GIT_TEST_CRONTAB="./print-args" git maintenance stop &&
++ GIT_TEST_CRONTAB="./print-args" git maintenance stop &&
+
+ # stop does not unregister the repo
+ git config --get --global maintenance.repo "$(pwd)" &&
+
-+ # stop does not remove plist files, but boots them out
-+ rm expect &&
-+ for frequency in hourly daily weekly
-+ do
-+ PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" &&
-+ grep schedule=$frequency "$PLIST" &&
-+ echo "bootout gui/$UID $PLIST" >>expect || return 1
-+ done &&
-+ test_cmp expect args
++ printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \
++ hourly daily weekly >expect &&
++ test_cmp expect args &&
++ ls "$HOME/Library/LaunchAgents" >actual &&
++ test_line_count = 0 actual
+'
+
test_expect_success 'register preserves existing strategy' '
4: 84eb44de31 ! 4: ed7a61978f maintenance: use Windows scheduled tasks
@@ Commit message
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
## Documentation/git-maintenance.txt ##
-@@ Documentation/git-maintenance.txt: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSy
- for more information.
+@@ Documentation/git-maintenance.txt: To create more advanced customizations to your background tasks, see
+ launchctl.plist(5) for more information.
+BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS
@@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd)
+static int schedule_task(const char *exec_path, enum schedule_priority schedule)
+{
+ int result;
-+ struct strvec args = STRVEC_INIT;
++ struct child_process child = CHILD_PROCESS_INIT;
+ const char *xml, *schtasks;
-+ char *xmlpath;
++ char *xmlpath, *tempDir;
+ FILE *xmlfp;
+ const char *frequency = get_frequency(schedule);
+ char *name = get_task_name(frequency);
+
-+ xmlpath = xstrfmt("%s/schedule-%s.xml",
-+ the_repository->objects->odb->path,
-+ frequency);
-+ xmlfp = fopen(xmlpath, "w");
-+ if (!xmlfp)
-+ die(_("failed to open '%s'"), xmlpath);
++ tempDir = xstrfmt("%s/temp", the_repository->objects->odb->path);
++ xmlpath = xstrfmt("%s/schedule-%s.xml", tempDir, frequency);
++ safe_create_leading_directories(xmlpath);
++ xmlfp = xfopen(xmlpath, "w");
+
+ xml = "<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n"
+ "<Task version=\"1.4\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\n"
@@ builtin/gc.c: static int platform_update_schedule(int run_maintenance, int fd)
+ schtasks = getenv("GIT_TEST_CRONTAB");
+ if (!schtasks)
+ schtasks = "schtasks";
-+ strvec_split(&args, schtasks);
-+ strvec_pushl(&args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL);
++ strvec_split(&child.args, schtasks);
++ strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL);
+
-+ result = run_command_v_opt(args.v, 0);
++ child.no_stdout = 1;
++ child.no_stderr = 1;
++
++ if (start_command(&child))
++ die(_("failed to start schtasks"));
++ result = finish_command(&child);
+
-+ strvec_clear(&args);
+ unlink(xmlpath);
++ rmdir(tempDir);
+ free(xmlpath);
+ free(name);
+ return result;
@@ t/t7900-maintenance.sh: test_expect_success !MACOS_MAINTENANCE 'stop from existi
GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start &&
grep "Important information!" cron.txt
@@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' '
- test_cmp expect args
+ test_line_count = 0 actual
'
+test_expect_success MINGW 'start and stop Windows maintenance' '
@@ t/t7900-maintenance.sh: test_expect_success MACOS_MAINTENANCE 'start and stop ma
+ # start registers the repo
+ git config --get --global maintenance.repo "$(pwd)" &&
+
-+ for frequency in hourly daily weekly
-+ do
-+ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \
-+ $frequency $frequency
-+ done >expect &&
++ printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/temp/schedule-%s.xml\n" \
++ hourly hourly daily daily weekly weekly >expect &&
+ test_cmp expect args &&
+
+ rm -f args &&-- gitgitgadget