Re: [PATCH v2 3/7] maintenance: add --scheduled option and config

On 8/25/2020 6:01 PM, Junio C Hamano wrote:

My intention for the *.schedule is that it is not an _exact_ frequency, but instead a lower bound on the frequency. That can be shelved for now as we discuss this setup:

I had briefly considered setting up crontab entries for each task (and possibly each repo) but ended up with these complications:

 1. Maintenance frequency differs by task, so we need to split the
    crontab by task. But we can't just split everything because we
    do not want multiple tasks running at the same time on one
    repository. We would need to group the tasks and have one entry
    saying "git maintenance run --task=<task1> --task=<task2> ..."
    for all tasks in the group.

 2. Different repositories might want different tasks at different
    frequencies, so we might need to split the crontab by repository.
    Again, we likely want to group repositories by these frequencies
    because a user could have 100 registered repositories and we don't
    really want to launch 100 parallel processes.

 3. If we want to stop maintenance, then restart it, we need to
    clear the crontab and repopulate it, which would require iterating
    through all "registered" repositories to read their config for
    frequencies.

 4. On macOS, editing the crontab doesn't require "sudo" but it _does_
    create a pop-up(!) to get permission from the user. It would be
    good to minimize how often we edit the crontab and instead use
    config edits to change frequencies.

With these things in mind, here is a suggested alternative design:

Let users specify a schedule frequency among this list: hourly, daily, weekly, monthly. We then set the following* crontab:

	0 * * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=hourly
	0 0 * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=daily
	0 0 * * 0 git for-each-repo --config=maintenance.repos maintenance run --scheduled=weekly
	0 0 0 * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=monthly

*Of course, there is some care around "$path/git --exec-path=$path" that I drop for ease here.

Then, "git maintenance (start|stop)" can be just as simple as we have now: write a fixed schedule every time.

The problem here is that cron will launch these processes in parallel, and then our object-database lock will cause some to fail! If anyone knows a simple way to tell cron "run hourly _except_ not at midnight" then we could let the "daily" schedule also run the "hourly" jobs, for instance. Hopefully that pattern could be extended to the weekly and monthly collisions.

Alternatively, we could run every hour and then interpret from config if the current "hour" matches one of the schedules ourselves. So, the crontab would be this simple:

	0 * * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled

and then we would internally decide "is this the midnight hour?" and "is this the first day of the week?" and "is this the first day of the month?" to detect if we should run the daily/weekly/monthly tasks. While it adds more time-awareness into Git, it does avoid the parallel task collisions. There are some concerns here related to long-running tasks delaying sequential runs of "git -C <repo> maintenance run --scheduled" causing the "is this the midnight hour?" queries to fail and having nightly/weekly/monthly maintenance be skipped accidentally. This motivates the *.lastRun config giving us some guarantee of _eventually_ running the tasks, just _not too frequently_.

I hope this launches a good discussion to help us find a good cron schedule strategy. After we land on a suitable strategy, I'll summarize all of these subtleties in the commit message for posterity.

Hopefully, the current way that I integrate with crontab and test that integration (in PATCH 6/7) could also be reviewed in parallel with this discussion. I'm very curious to see how that could be improved.

Thanks, -Stolee