Re: Concurrent fetch commands

From: Patrick Steinhardt <ps@pks.im>
Date: Jan 3, 2024, 10:40 UTC
Message-ID: <ZZU5s4LKQF1NLgnC@tanuki>
In-Reply-To: <ZZU1TCyQdLqoLxPw@ugly>

On Wed, Jan 03, 2024 at 11:22:04AM +0100, Oswald Buddenhagen wrote:

Show 12 quoted lines

> On Wed, Jan 03, 2024 at 09:11:07AM +0100, Patrick Steinhardt wrote:
> > Ah, one thing I didn't think of is parallel fetches. It's expected that
> > all of the fetches write into FETCH_HEAD at the same point in time
> > concurrently
> > 
> is it, though? given that the contents could be already randomly scrambled,
> it would not seem particularly bad if the behavior changed.
> 
> the one real complication i see is the --append option, which requires using
> a waiting lock after the actual fetch, rather than acquiring it immediately
> and erroring out on failure (and ideally giving a hint to use
> --no-write-fetch-head).

I should probably clarify, but with "parallel fetches" I meant `git fetch --jobs=`, not two separate executions of git-fetch(1). And these do in fact use `--append` internally: the main process first truncates FETCH_HEAD and then spawns its children, which will then append to FETCH_HEAD in indeterministic order.

But even though the order is indeterministic, I wouldn't go as far as claiming that the complete feature is broken. It works and records all updated refs in FETCH_HEAD just fine, even if it's not particularly elegant. Which to me shows that we should try hard not to break it.

> an extra complication is that concurrent runs with and without --append
> should be precluded, because that would again result in undefined behavior.
> it generally seems tricky to get --append straight if parallel fetches are
> supposed to work.

Yeah, the `--append` flag indeed complicates things. There are two ways to handle this:

  - `--append` should refrain from running when there is a lockfile.
    This breaks `git fetch --jobs` without extra infra to handle this
    case, and furthermore a user may (rightfully?) expect that two
    manually spawned `git fetch --append` processes should work just
    fine.

  - `--append` should handle concurrency just fine, that is it knows to
    append to a preexisting lockfile. This is messy though, and the
    original creator of the lockfile wouldn't know when it can commit it
    into place.

Both options are kind of ugly, so I'm less sure now whether lockfiles are the way to go.

Patrick