{"thread":{"id":"62155","subject":"Prefetch maintenance might lead to thundering herd issues","startedAt":"2024-09-20T15:16:04Z","lastAt":"2024-09-23T01:26:43Z","messageCount":4,"participants":["Shubham Kanodia","Derrick Stolee"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"503139","messageId":"CAG=Um+3C1JMKoW+j03nF6uS2DjcJnKD7+0PSaiXGB-td9qyc8Q@mail.gmail.com","threadId":"62155","inReplyTo":null,"subject":"Prefetch maintenance might lead to thundering herd issues","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-09-20T15:15:26Z","receivedAt":"2024-09-20T15:16:04Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"The current git-maintenance prefetch command allows daily, weekly, and\nhourly schedules.\nHowever, in repositories with many active developers, timing `fetch`\nto happen at the same wall clock time can lead to a thundering herd\nproblem for the backend server.\n\nIdeally, these fetch requests would be spaced out by a small amount,\nso as not to lead to a co-ordinated load on the server.\n\nI don't see a way to configure this at the moment in a cross-platform\nway at the moment. Perhaps it might be sensible to offset the cron\nschedules slightly for users so as to not cause such an issue?\n\nThis might come at a small cost of predictability of the run, but that\nshould be okay as predictability isn't the primary goal for\nmaintenance.\n"},{"id":"503157","messageId":"b67c7b6b-f11b-4306-b3ba-c440e8cf7cab@gmail.com","threadId":"62155","inReplyTo":"CAG=Um+3C1JMKoW+j03nF6uS2DjcJnKD7+0PSaiXGB-td9qyc8Q@mail.gmail.com","subject":"Re: Prefetch maintenance might lead to thundering herd issues","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-09-20T19:20:47Z","receivedAt":"2024-09-20T19:20:49Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/20/24 11:15 AM, Shubham Kanodia wrote:\n> The current git-maintenance prefetch command allows daily, weekly, and\n> hourly schedules.\n> However, in repositories with many active developers, timing `fetch`\n> to happen at the same wall clock time can lead to a thundering herd\n> problem for the backend server.\n> \n> Ideally, these fetch requests would be spaced out by a small amount,\n> so as not to lead to a co-ordinated load on the server.\n> \n> I don't see a way to configure this at the moment in a cross-platform\n> way at the moment. Perhaps it might be sensible to offset the cron\n> schedules slightly for users so as to not cause such an issue?\n\nThis is already handled by using a random minute of the hour, as\nimplemented in 9b43399057 (maintenance: use random minute in cron\nscheduler, 2023-08-10), for example. There are similar uses for\nsystemd, launchctl, and schtasks schedulers (look for uses of\nthe get_random_minute() method).\n\nAre you noticing that this isn't working as expected?\n\nThanks,\n-Stolee\n\n"},{"id":"503225","messageId":"CAG=Um+2g-pn3XmC4MVkA+UAGeWpHw5U6fe=zXR=6-ibu3d0qsg@mail.gmail.com","threadId":"62155","inReplyTo":"b67c7b6b-f11b-4306-b3ba-c440e8cf7cab@gmail.com","subject":"Re: Prefetch maintenance might lead to thundering herd issues","fromName":"Shubham Kanodia","fromEmail":"shubham.kanodia10@gmail.com","sentAt":"2024-09-22T18:40:22Z","receivedAt":"2024-09-22T18:40:59Z","isPatch":false,"sender":{"key":"shubham.kanodia10@gmail.com","avatar":"https://avatars.githubusercontent.com/u/8946207?v=4"},"body":"On Sat, Sep 21, 2024 at 1:20 AM Derrick Stolee <stolee@gmail.com> wrote:\n>\n> On 9/20/24 11:15 AM, Shubham Kanodia wrote:\n> > The current git-maintenance prefetch command allows daily, weekly, and\n> > hourly schedules.\n> > However, in repositories with many active developers, timing `fetch`\n> > to happen at the same wall clock time can lead to a thundering herd\n> > problem for the backend server.\n> >\n> > Ideally, these fetch requests would be spaced out by a small amount,\n> > so as not to lead to a co-ordinated load on the server.\n> >\n> > I don't see a way to configure this at the moment in a cross-platform\n> > way at the moment. Perhaps it might be sensible to offset the cron\n> > schedules slightly for users so as to not cause such an issue?\n>\n> This is already handled by using a random minute of the hour, as\n> implemented in 9b43399057 (maintenance: use random minute in cron\n> scheduler, 2023-08-10), for example. There are similar uses for\n> systemd, launchctl, and schtasks schedulers (look for uses of\n> the get_random_minute() method).\n>\n> Are you noticing that this isn't working as expected?\n>\n> Thanks,\n> -Stolee\n>\n\nAh, thanks for pointing that out. It wasn't really clear from the\nmultiple examples and cron expressions on the git\nmaintenance documentation page (which have examples such as `0 1-23 *\n* *` for hourly). I should've looked\nmore deeply into the implementation.\n\nPerhaps adding a line to the existing doc might be of help? What do\nyou think about adding a line to the\nsection on scheduling —\n\ndiff --git a/Documentation/git-maintenance.txt\nb/Documentation/git-maintenance.txt\nindex 51d0f7e94b..34828cdfe6 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -217,11 +217,13 @@ on an hourly basis. Each run executes the\n\"hourly\" tasks. At midnight,\nthat process also executes the \"daily\" tasks. At midnight on the first day\nof the week, that process also executes the \"weekly\" tasks. A single\nprocess iterates over each registered repository, performing the scheduled\n-tasks for that frequency. Depending on the number of registered\n-repositories and their sizes, this process may take longer than an hour.\n-In this case, multiple `git maintenance run` commands may run on the same\n-repository at the same time, colliding on the object database lock. This\n-results in one of the two tasks not running.\n+tasks for that frequency. The processes are scheduled to a random minute of\n+the hour per client to spread out load from multiple clients (e.g. from\n+prefetching). Depending on the number of registered repositories and their\n+sizes, this process may take longer than an hour. In this case, multiple\n+`git maintenance run` commands may run on the same repository at the same\n+time, colliding on the object database lock. This results in one of the two\n+tasks not running.\nIf you find that some maintenance windows are taking longer than one hour\nto complete, then consider reducing the complexity of your maintenance\n\n--\n"},{"id":"503230","messageId":"91bf8c63-b2fc-48ca-a6de-37a576ddc1b1@gmail.com","threadId":"62155","inReplyTo":"CAG=Um+2g-pn3XmC4MVkA+UAGeWpHw5U6fe=zXR=6-ibu3d0qsg@mail.gmail.com","subject":"Re: Prefetch maintenance might lead to thundering herd issues","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-09-23T01:26:40Z","receivedAt":"2024-09-23T01:26:43Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/22/24 2:40 PM, Shubham Kanodia wrote:\n> On Sat, Sep 21, 2024 at 1:20 AM Derrick Stolee <stolee@gmail.com> wrote:\n\n>> This is already handled by using a random minute of the hour, as\n>> implemented in 9b43399057 (maintenance: use random minute in cron\n>> scheduler, 2023-08-10), for example. There are similar uses for\n>> systemd, launchctl, and schtasks schedulers (look for uses of\n>> the get_random_minute() method).\n\n> Ah, thanks for pointing that out. It wasn't really clear from the\n> multiple examples and cron expressions on the git\n> maintenance documentation page (which have examples such as `0 1-23 *\n> * *` for hourly). I should've looked\n> more deeply into the implementation.\n> \n> Perhaps adding a line to the existing doc might be of help? What do\n> you think about adding a line to the\n> section on scheduling —\n\nI do think a doc change would help. Please submit it as a patch for\nreview!\n\nThanks,\n-Stolee\n\n"}]}