git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: concurrent fetches to update same mirror

From
NKNeal Kreitzinger <nkreitzinger@gmail.com>
Date
Jan 5, 2011, 22:34 UTC
Message-ID
<4D24F1FE.5060400@gmail.com>
In-Reply-To
<20110105211313.GB7808@sigill.intra.peff.net>
On 1/5/2011 3:13 PM, Jeff King wrote:
Show 52 quoted lines
> On Wed, Jan 05, 2011 at 03:53:25PM -0500, Jeff King wrote:
>
>>> If both fetch processes try to update the same ref at the same time,
>>> one will get the lock and continue, and the other will crash with an
>>> error (because the lock was busy).  If one is slightly slower than the
>>> other, they will probably update the refs twice, with the slower fetch
>>> updating what the faster one had just updated.  :-)
>>
>> I assumed it would take the "old" value at the very beginning of the
>> fetch (before talking with the remote), and then see that the ref was
>> changed under our feet. Or does it simply do it at the end?
>
> Hmm. Weirder even, builtin/fetch.c:s_update_ref takes a "check_old"
> flag, and we do always use it for branch updates. But not for tag
> updates. I can't think of why. The code blames all the way back to the
> original builtin-fetch.
>
> Anyway, when we do check, we check the value from the beginning of the
> fetch. So you can get lock conflicts. For example, doing this:
>
>    mkdir repo&&  cd repo&&  git init
>    echo contents>foo&&  git add .&&  git commit -m one
>    git update-ref refs/remotes/origin/master refs/heads/master
>    git remote add origin some-remote-repo-that-takes-a-few-seconds
>    xterm -e 'git fetch -v; read'&  xterm -e 'git fetch -v; read'
>
> I.e., putting some cruft into the ref and then updating it. One fetch
> will force-write over the ref properly:
>
>     + ac32203...4e64590 master     ->  origin/master  (forced update)
>
> but the other one will barf on the lock:
>
>    error: Ref refs/remotes/origin/master is at 4e6459052ab329914c7712a926773e566b8c821d but expected ac32203727daa3bcb5fc041786aa45adbbe86299
>    ...
>     ! ac32203...4e64590 master     ->  origin/master  (unable to update local ref)
>
> Interestingly, in the case of ref _creation_, not update, like this:
>
>    mkdir repo&&  cd repo&&  git init
>    git remote add origin some-remote-repo-that-takes-a-few-seconds
>    xterm -e 'git fetch -v; read'&  xterm -e 'git fetch -v; read'
>
> then both will happily update, the second one overwriting the results of
> the first. It seems in the case of locking a ref which previously didn't
> exist, we don't enforce that it still doesn't exist.
>
> I wonder if we should, but perhaps there is some corner case I am not
> considering. The code is in lock_ref_sha1_basic, but blaming didn't turn
> up anything helpful.
>
> -Peff

This was actually the case in my test. Updates to the mirror are always new branches except for master. The only pre-existing branch that might get updated is master, but in that test it didn't. The new branches and tags were updated. The new tags always point to the new branches. I'm running 1.7.1 on both servers.

v/r, Neal

Previous: Jeff KingNext: Neal Kreitzinger
Message 6 of 12 in “concurrent fetches to update same mirror”
  1. Neal KreitzingerJan 5, 2011
  2. Jeff KingJan 5, 2011
  3. Shawn PearceJan 5, 2011
  4. Jeff KingJan 5, 2011
  5. Jeff KingJan 5, 2011
  6. Neal KreitzingerJan 5, 2011
  7. Neal KreitzingerJan 5, 2011
  8. Jeff KingJan 5, 2011
  9. Junio C HamanoJan 5, 2011
  10. Jeff KingJan 6, 2011
  11. Marc BranchaudJan 7, 2011
  12. Marc BranchaudJan 7, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.