{"thread":{"id":"60271","subject":"git-retry tool or git.retry config (built-in implementation)?","startedAt":"2023-09-26T17:05:59Z","lastAt":"2023-09-27T13:29:14Z","messageCount":4,"participants":["Yaroslav Halchenko","Bagas Sanjaya"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482348","messageId":"ZRMLNyHXoWOj6K-l@bilena","threadId":"60271","inReplyTo":null,"subject":"git-retry tool or git.retry config (built-in implementation)?","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2023-09-26T16:47:51Z","receivedAt":"2023-09-26T17:05:59Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"Dear Git Gurus,\n\nIn DataLad (https://datalad.org) we are doing lots of automated cloning,\nfetching etc as part of our CI etc jobs.  Once in a while git operations\nfail [see e.g. 1], and beg us to retry but we need to know when to\ndo so, and not do it upon every failed git invocation since some\nfailures could be legit (repository is gone). While looking how others\nsolve it we found\nhttps://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed\nwhich pointed to tools like git-retry and later part of\nhttps://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go\nwhich serve as a collection of regexes to be on lookout for to retry.\n\nWould that be the \"best\" strategy currently?  \n\nAs regex matching might eventually break whenever `git` changes\nanything in the output messages, I wondered if there could be a more\nrobust internal implementation in git itself?  Similarly git-annex has\nannex.retry config setting which sets the count of retries for\n\"retriable\" operations.  \n\nDo you think something like that could be implemented in git in some\nforeseable future?\n\n[1] https://github.com/datalad/datalad/issues/7485\n\nThank you in advance!\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nWWW:   http://www.linkedin.com/in/yarik        \n\n"},{"id":"482359","messageId":"ZRNv-n_VlIDPX0oi@debian.me","threadId":"60271","inReplyTo":"ZRMLNyHXoWOj6K-l@bilena","subject":"Re: git-retry tool or git.retry config (built-in implementation)?","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2023-09-26T23:57:46Z","receivedAt":"2023-09-27T00:44:49Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:\n> Dear Git Gurus,\n> \n> In DataLad (https://datalad.org) we are doing lots of automated cloning,\n> fetching etc as part of our CI etc jobs.  Once in a while git operations\n> fail [see e.g. 1], and beg us to retry but we need to know when to\n> do so, and not do it upon every failed git invocation since some\n> failures could be legit (repository is gone). While looking how others\n> solve it we found\n> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed\n> which pointed to tools like git-retry and later part of\n> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go\n> which serve as a collection of regexes to be on lookout for to retry.\n> \n> Would that be the \"best\" strategy currently?  \n\nLooking at the actual git_retry.py script [1], it really just wraps\nactual Git commands. IMO, git-retry(1) shell script as you mentioned\nonly calls the python version, which adds another level of indirection\n(why not doing it in pure shell?).\n\nAFAIK, to solve the retrying problem, we need to have a way to tell\ntransport backend (curl/ssh) to resume transfer from the faulty point.\n\n> \n> As regex matching might eventually break whenever `git` changes\n> anything in the output messages, I wondered if there could be a more\n> robust internal implementation in git itself?  Similarly git-annex has\n> annex.retry config setting which sets the count of retries for\n> \"retriable\" operations.  \n\nDo you use porcelain interfaces instead of plumbing ones?\n\nThanks.\n\n[1]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/01d2cde990f22d409e74e239de7e4d347102d6f6/git_retry.py\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"482371","messageId":"ZRQjKYBLwNg42xAd@bilena","threadId":"60271","inReplyTo":"ZRNv-n_VlIDPX0oi@debian.me","subject":"Re: git-retry tool or git.retry config (built-in implementation)?","fromName":"Yaroslav Halchenko","fromEmail":"yoh@onerussian.com","sentAt":"2023-09-27T12:42:17Z","receivedAt":"2023-09-27T12:42:36Z","isPatch":false,"sender":{"key":"yoh@onerussian.com","avatar":"https://gravatar.com/avatar/8901b82415ae451a83aea49409708912726e53620e3ac92320bf1f86548d97e9?d=mp&s=160"},"body":"\nOn Wed, 27 Sep 2023, Bagas Sanjaya wrote:\n\n> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:\n> > Dear Git Gurus,\n\n> > In DataLad (https://datalad.org) we are doing lots of automated cloning,\n> > fetching etc as part of our CI etc jobs.  Once in a while git operations\n> > fail [see e.g. 1], and beg us to retry but we need to know when to\n> > do so, and not do it upon every failed git invocation since some\n> > failures could be legit (repository is gone). While looking how others\n> > solve it we found\n> > https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed\n> > which pointed to tools like git-retry and later part of\n> > https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go\n> > which serve as a collection of regexes to be on lookout for to retry.\n\n> > Would that be the \"best\" strategy currently?  \n\n> Looking at the actual git_retry.py script [1], it really just wraps\n> actual Git commands. IMO, git-retry(1) shell script as you mentioned\n> only calls the python version, which adds another level of indirection\n> (why not doing it in pure shell?).\n\nMy guess would be that it is just easier to code in Python usually for\nsuch cases with a \"registry\" of hits etc.  But why not just to strip .py\nfrom python script which has shebang already and not require bash\nwrapper at all? ;)\n\n> AFAIK, to solve the retrying problem, we need to have a way to tell\n> transport backend (curl/ssh) to resume transfer from the faulty point.\n\nsome times it seems not even getting connected (https) entirely and that\n(?) leading to error in the caller above, e.g. from\nhttps://github.com/datalad/datalad/issues/7485#issuecomment-1735619755\n\nerror: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...\n\n\n> > As regex matching might eventually break whenever `git` changes\n> > anything in the output messages, I wondered if there could be a more\n> > robust internal implementation in git itself?  Similarly git-annex has\n> > annex.retry config setting which sets the count of retries for\n> > \"retriable\" operations.  \n\n> Do you use porcelain interfaces instead of plumbing ones?\n\nI would say -- a \"mix\". \n\n-- \nYaroslav O. Halchenko\nCenter for Open Neuroscience     http://centerforopenneuroscience.org\nDartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755\nWWW:   http://www.linkedin.com/in/yarik        \n\n"},{"id":"482373","messageId":"58c0dd77-200f-45d2-837b-89b611ec711b@gmail.com","threadId":"60271","inReplyTo":"ZRQjKYBLwNg42xAd@bilena","subject":"Re: git-retry tool or git.retry config (built-in implementation)?","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2023-09-27T13:29:07Z","receivedAt":"2023-09-27T13:29:14Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 27/09/2023 19:42, Yaroslav Halchenko wrote:\n> \n> On Wed, 27 Sep 2023, Bagas Sanjaya wrote:\n> \n>> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:\n>>> Dear Git Gurus,\n> \n>>> In DataLad (https://datalad.org) we are doing lots of automated cloning,\n>>> fetching etc as part of our CI etc jobs.  Once in a while git operations\n>>> fail [see e.g. 1], and beg us to retry but we need to know when to\n>>> do so, and not do it upon every failed git invocation since some\n>>> failures could be legit (repository is gone). While looking how others\n>>> solve it we found\n>>> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed\n>>> which pointed to tools like git-retry and later part of\n>>> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go\n>>> which serve as a collection of regexes to be on lookout for to retry.\n> \n>>> Would that be the \"best\" strategy currently?  \n> \n>> Looking at the actual git_retry.py script [1], it really just wraps\n>> actual Git commands. IMO, git-retry(1) shell script as you mentioned\n>> only calls the python version, which adds another level of indirection\n>> (why not doing it in pure shell?).\n> \n> My guess would be that it is just easier to code in Python usually for\n> such cases with a \"registry\" of hits etc.  But why not just to strip .py\n> from python script which has shebang already and not require bash\n> wrapper at all? ;)\n> \n>> AFAIK, to solve the retrying problem, we need to have a way to tell\n>> transport backend (curl/ssh) to resume transfer from the faulty point.\n> \n> some times it seems not even getting connected (https) entirely and that\n> (?) leading to error in the caller above, e.g. from\n> https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755\n> \n> error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...\n> \n\nThen the question is how can the server cache generated on-the-fly bundles\nin hope that future clients can resume the transfer.\n \n> \n>>> As regex matching might eventually break whenever `git` changes\n>>> anything in the output messages, I wondered if there could be a more\n>>> robust internal implementation in git itself?  Similarly git-annex has\n>>> annex.retry config setting which sets the count of retries for\n>>> \"retriable\" operations.  \n> \n>> Do you use porcelain interfaces instead of plumbing ones?\n> \n> I would say -- a \"mix\". \n> \n\nJunio, what do you think of above ideas from Yaroslav?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n\n"}]}