From: Taylor Blau <ttaylorr@openai.com>
Date: Tue, 14 Jul 2026 04:07:46 GMT
Subject: Re: [PATCH 1/2] http: use unique tempfiles for packfile URI downloads
Message-ID: <alW2EnNR21VmkESW@com-79390>
In-Reply-To: <alWXwAGWgXSXoRJv@com-76773>

On Mon, Jul 13, 2026 at 06:58:24PM -0700, Ted Nyman wrote:
> > While that does sound like a safe and correct approach, stepping
> > back briefly, would it not be wasteful for the second process to
> > download the same packfile that the first has already started
> > downloading?
>
> Yes. If two fetches overlap, the second download is redundant.
>
> > Are there better ways for these processes to coordinate with each
> > other? Instead of appending to the file, what if the second process
> > uses a predictable temporary name (which we already use) to open a
> > new file with O_CREAT | O_EXCL to avoid this redundant work?
>
> Using the existing pack-<hash>.pack.temp name with O_CREAT | O_EXCL
> would prevent concurrent writes, but EEXIST alone would not
> distinguish an in-progress download from one left by an earlier
> failed or interrupted invocation. The existing .pack.temp name is not
> covered by the tmp_* pruning path, so simply waiting for it to
> disappear could leave a fetch stuck after a crash.

Exactly. If two processes are downloading the same pack at the same time
to different locations, the effort is of course redundant. But I don't
think we can reliably distinguish between that case and one where an
earlier process died in the middle of downloading a pack but was unable
to clean up after itself.

Thanks,
Taylor