git-retry tool or git.retry config (built-in implementation)?

> Dear Git Gurus,
> 
> In DataLad (https://datalad.org) we are doing lots of automated cloning,
> fetching etc as part of our CI etc jobs.  Once in a while git operations
> fail [see e.g. 1], and beg us to retry but we need to know when to
> do so, and not do it upon every failed git invocation since some
> failures could be legit (repository is gone). While looking how others
> solve it we found
> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
> which pointed to tools like git-retry and later part of
> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
> which serve as a collection of regexes to be on lookout for to retry.
> 
> Would that be the "best" strategy currently?  

> 
> As regex matching might eventually break whenever `git` changes
> anything in the output messages, I wondered if there could be a more
> robust internal implementation in git itself?  Similarly git-annex has
> annex.retry config setting which sets the count of retries for
> "retriable" operations.  

> > In DataLad (https://datalad.org) we are doing lots of automated cloning,
> > fetching etc as part of our CI etc jobs.  Once in a while git operations
> > fail [see e.g. 1], and beg us to retry but we need to know when to
> > do so, and not do it upon every failed git invocation since some
> > failures could be legit (repository is gone). While looking how others
> > solve it we found
> > https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
> > which pointed to tools like git-retry and later part of
> > https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
> > which serve as a collection of regexes to be on lookout for to retry.

> > As regex matching might eventually break whenever `git` changes
> > anything in the output messages, I wondered if there could be a more
> > robust internal implementation in git itself?  Similarly git-annex has
> > annex.retry config setting which sets the count of retries for
> > "retriable" operations.  

> 
> On Wed, 27 Sep 2023, Bagas Sanjaya wrote:
> 
>> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
>>> Dear Git Gurus,
> 
>>> In DataLad (https://datalad.org) we are doing lots of automated cloning,
>>> fetching etc as part of our CI etc jobs.  Once in a while git operations
>>> fail [see e.g. 1], and beg us to retry but we need to know when to
>>> do so, and not do it upon every failed git invocation since some
>>> failures could be legit (repository is gone). While looking how others
>>> solve it we found
>>> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
>>> which pointed to tools like git-retry and later part of
>>> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
>>> which serve as a collection of regexes to be on lookout for to retry.
> 
>>> Would that be the "best" strategy currently?  
> 
>> Looking at the actual git_retry.py script [1], it really just wraps
>> actual Git commands. IMO, git-retry(1) shell script as you mentioned
>> only calls the python version, which adds another level of indirection
>> (why not doing it in pure shell?).
> 
> My guess would be that it is just easier to code in Python usually for
> such cases with a "registry" of hits etc.  But why not just to strip .py
> from python script which has shebang already and not require bash
> wrapper at all? ;)
> 
>> AFAIK, to solve the retrying problem, we need to have a way to tell
>> transport backend (curl/ssh) to resume transfer from the faulty point.
> 
> some times it seems not even getting connected (https) entirely and that
> (?) leading to error in the caller above, e.g. from
> https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755
> 
> error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...
> 

> 
>>> As regex matching might eventually break whenever `git` changes
>>> anything in the output messages, I wondered if there could be a more
>>> robust internal implementation in git itself?  Similarly git-annex has
>>> annex.retry config setting which sets the count of retries for
>>> "retriable" operations.  
> 
>> Do you use porcelain interfaces instead of plumbing ones?
> 
> I would say -- a "mix". 
>