Re: Prefetch maintenance might lead to thundering herd issues
On Sat, Sep 21, 2024 at 1:20 AM Derrick Stolee <stolee@gmail.com> wrote:
Show 26 quoted lines
>
> On 9/20/24 11:15 AM, Shubham Kanodia wrote:
> > The current git-maintenance prefetch command allows daily, weekly, and
> > hourly schedules.
> > However, in repositories with many active developers, timing `fetch`
> > to happen at the same wall clock time can lead to a thundering herd
> > problem for the backend server.
> >
> > Ideally, these fetch requests would be spaced out by a small amount,
> > so as not to lead to a co-ordinated load on the server.
> >
> > I don't see a way to configure this at the moment in a cross-platform
> > way at the moment. Perhaps it might be sensible to offset the cron
> > schedules slightly for users so as to not cause such an issue?
>
> This is already handled by using a random minute of the hour, as
> implemented in 9b43399057 (maintenance: use random minute in cron
> scheduler, 2023-08-10), for example. There are similar uses for
> systemd, launchctl, and schtasks schedulers (look for uses of
> the get_random_minute() method).
>
> Are you noticing that this isn't working as expected?
>
> Thanks,
> -Stolee
>
Ah, thanks for pointing that out. It wasn't really clear from the
multiple examples and cron expressions on the git
maintenance documentation page (which have examples such as `0 1-23 *
* *` for hourly). I should've looked
more deeply into the implementation.
Perhaps adding a line to the existing doc might be of help? What do you think about adding a line to the section on scheduling —
diff --git a/Documentation/git-maintenance.txt
b/Documentation/git-maintenance.txt
index 51d0f7e94b..34828cdfe6 100644
--- a/Documentation/git-maintenance.txt
+++ b/Documentation/git-maintenance.txt
@@ -217,11 +217,13 @@ on an hourly basis. Each run executes the
"hourly" tasks. At midnight,
that process also executes the "daily" tasks. At midnight on the first day
of the week, that process also executes the "weekly" tasks. A single
process iterates over each registered repository, performing the scheduled
-tasks for that frequency. Depending on the number of registered
-repositories and their sizes, this process may take longer than an hour.
-In this case, multiple `git maintenance run` commands may run on the same
-repository at the same time, colliding on the object database lock. This
-results in one of the two tasks not running.
+tasks for that frequency. The processes are scheduled to a random minute of
+the hour per client to spread out load from multiple clients (e.g. from
+prefetching). Depending on the number of registered repositories and their
+sizes, this process may take longer than an hour. In this case, multiple
+`git maintenance run` commands may run on the same repository at the same
+time, colliding on the object database lock. This results in one of the two
+tasks not running.
If you find that some maintenance windows are taking longer than one hour
to complete, then consider reducing the complexity of your maintenance
--
Shubham Kanodia