Git push race condition?

Hi folks,

I run a private Git repository (using Gitlab) with about 200 users doing about 100 pushes per day.

I've noticed that a few times in the past several weeks, we've had events where pushes have been lost when two people pushed at just about the same time. The scenario is that two users both have commits based on commit A, call them B and B'. The user with commit B pushes at about the same time as the user who pushes B'. Both pushes are determined to be fast-forwards and both succeed, but B' overwrites B and B is no longer on origin/master. The server does have B in its .git directory but the commit isn't on any branch.

I'm confident nobody is force pushing (we have a hook to disallow it on master branches and I've seen screenshots of both user's clients after they pushed). Both git clients say "successfully pushed A..B master -> master" (or A..B') in the output of their push commands. However, when the user that had B does a fetch, it shows master as having been force updated.

We have a few pre-receive hooks and post-receive hooks that run on pushes, and Gitlab has an update hook as well. My original theory was that this was happening because Git checks if it's a fast-forward before running hooks, and that the hooks taking a few seconds creates more opportunity for a race condition to occur.

However, after reading http://git.661346.n2.nabble.com/push-race-td7569254.html and doing some of my own testing (creating a hook that runs for 60 seconds and pushing from two locations to a test repo) this theory seems to be wrong. With the 60 second sleep hook (tried as an update hook and a pre-receive hook), I wasn't able to reproduce the problem. The second pusher always got an error like this:

error: Ref refs/heads/master is at 4584c1f34e07cea2df6abc8e0d407fe016017130 but expected 61b79b6d35b066d054fb3deab550f1c51598cf5f remote: error: failed to lock refs/heads/master

Which looks like exactly what I'd want Git to be doing in this scenario, and supports what that archived thread says about how this should work.

So the question is, how might this be happening and what can I do about it?

Thanks, Scott