Re: git-retry tool or git.retry config (built-in implementation)?

From: Yaroslav Halchenko <yoh@onerussian.com>
Date: Sep 27, 2023, 12:42 UTC
Message-ID: <ZRQjKYBLwNg42xAd@bilena>
In-Reply-To: <ZRNv-n_VlIDPX0oi@debian.me>

On Wed, 27 Sep 2023, Bagas Sanjaya wrote:

> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
> > Dear Git Gurus,

Show 10 quoted lines

> > In DataLad (https://datalad.org) we are doing lots of automated cloning,
> > fetching etc as part of our CI etc jobs.  Once in a while git operations
> > fail [see e.g. 1], and beg us to retry but we need to know when to
> > do so, and not do it upon every failed git invocation since some
> > failures could be legit (repository is gone). While looking how others
> > solve it we found
> > https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
> > which pointed to tools like git-retry and later part of
> > https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
> > which serve as a collection of regexes to be on lookout for to retry.

> > Would that be the "best" strategy currently?

> Looking at the actual git_retry.py script [1], it really just wraps
> actual Git commands. IMO, git-retry(1) shell script as you mentioned
> only calls the python version, which adds another level of indirection
> (why not doing it in pure shell?).

My guess would be that it is just easier to code in Python usually for such cases with a "registry" of hits etc. But why not just to strip .py from python script which has shebang already and not require bash wrapper at all? ;)

> AFAIK, to solve the retrying problem, we need to have a way to tell
> transport backend (curl/ssh) to resume transfer from the faulty point.

some times it seems not even getting connected (https) entirely and that (?) leading to error in the caller above, e.g. from https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755

error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...

Show 5 quoted lines

> > As regex matching might eventually break whenever `git` changes
> > anything in the output messages, I wondered if there could be a more
> > robust internal implementation in git itself?  Similarly git-annex has
> > annex.retry config setting which sets the count of retries for
> > "retriable" operations.

> Do you use porcelain interfaces instead of plumbing ones?

I would say -- a "mix".

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
WWW:   http://www.linkedin.com/in/yarik