{"thread":{"id":"36284","subject":"Git push race condition?","startedAt":"2014-03-24T19:18:14Z","lastAt":"2014-04-11T07:46:28Z","messageCount":17,"participants":["Scott Sandler","Matthieu Moy","Ævar Arnfjörð Bjarmason","Junio C Hamano","Jeff King","Nasser Grainawi"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"237483","messageId":"CAAyEjTN53+5B9Od9wW698wODNL3hR6Upot8-ZLwEksn3ir_zjA@mail.gmail.com","threadId":"36284","inReplyTo":null,"subject":"Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-03-24T19:18:14Z","receivedAt":"2014-03-24T19:18:14Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"Hi folks,\n\nI run a private Git repository (using Gitlab) with about 200 users\ndoing about 100 pushes per day.\n\nI've noticed that a few times in the past several weeks, we've had\nevents where pushes have been lost when two people pushed at just\nabout the same time. The scenario is that two users both have commits\nbased on commit A, call them B and B'. The user with commit B pushes\nat about the same time as the user who pushes B'. Both pushes are\ndetermined to be fast-forwards and both succeed, but B' overwrites B\nand B is no longer on origin/master. The server does have B in its\n.git directory but the commit isn't on any branch.\n\nI'm confident nobody is force pushing (we have a hook to disallow it\non master branches and I've seen screenshots of both user's clients\nafter they pushed). Both git clients say \"successfully pushed A..B\nmaster -> master\" (or A..B') in the output of their push commands.\nHowever, when the user that had B does a fetch, it shows master as\nhaving been force updated.\n\nWe have a few pre-receive hooks and post-receive hooks that run on\npushes, and Gitlab has an update hook as well. My original theory was\nthat this was happening because Git checks if it's a fast-forward\nbefore running hooks, and that the hooks taking a few seconds creates\nmore opportunity for a race condition to occur.\n\nHowever, after reading\nhttp://git.661346.n2.nabble.com/push-race-td7569254.html and doing\nsome of my own testing (creating a hook that runs for 60 seconds and\npushing from two locations to a test repo) this theory seems to be\nwrong. With the 60 second sleep hook (tried as an update hook and a\npre-receive hook), I wasn't able to reproduce the problem. The second\npusher always got an error like this:\n\nerror: Ref refs/heads/master is at\n4584c1f34e07cea2df6abc8e0d407fe016017130 but expected\n61b79b6d35b066d054fb3deab550f1c51598cf5f\nremote: error: failed to lock refs/heads/master\n\nWhich looks like exactly what I'd want Git to be doing in this\nscenario, and supports what that archived thread says about how this\nshould work.\n\nSo the question is, how might this be happening and what can I do about it?\n\nThanks,\nScott\n"},{"id":"237485","messageId":"vpq61n3bcve.fsf@anie.imag.fr","threadId":"36284","inReplyTo":"CAAyEjTN53+5B9Od9wW698wODNL3hR6Upot8-ZLwEksn3ir_zjA@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-03-24T19:44:21Z","receivedAt":"2014-03-24T19:44:21Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Scott Sandler writes:\n\n> Both pushes are\n> determined to be fast-forwards and both succeed, but B' overwrites B\n> and B is no longer on origin/master. The server does have B in its\n> .git directory but the commit isn't on any branch.\n\nIs the reflog enabled on the server? If so, does it say anything about B\nand B'?\n\nWhat filesystem do you use on the server? Is there any kind of NFS, and\nif so are you sure that there's only one machine accessing the\nfilesystem at the same time?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"237486","messageId":"CAAyEjTNPqPHswbrrV9pRyXUUqD8dYzJaXQpWr+g3kuBERNLMRw@mail.gmail.com","threadId":"36284","inReplyTo":"vpq61n3bcve.fsf@anie.imag.fr","subject":"Re: Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-03-24T20:01:32Z","receivedAt":"2014-03-24T20:01:32Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"It's a bare repo and I didn't realize server-side reflogs were a\nthing. Just ran \"git config core.logallrefupdates true\" in the repo on\nthe server which seems to be what I should do to enable that.\n\nThe server does know about B, it shows up when you do \"git show B\".\nHowever \"git branch --contains B\" returns nothing.\n\nThe filesystem is ext4 on linux. It's on a virtual machine in our own\ndatacenter. It's not an NFS share or anything like that, there is\ndefinitely only one server accessing the filesystem at a time.\n\nGitlab's update hook maintains an event log when any push event\nhappens, who pushed and which commits. The most recent time this\nhappened, the first push which was lost occured at 2014-03-24 19:04:51\nand the one that overwrote it happened at 2014-03-24 19:05:04. That's\nwhen the update hook ran, not necessarily when the user hit \"git\npush\", but it is notable that it's 13 seconds apart which is a pretty\nlong time. We do run several hooks for checking coding syntax and\nvarious other things so it's believable to me that the hooks would\ntake more than 13 seconds on occasion, but based on the testing I did\nwith the sleep hook it didn't seem like the hooks were actually the\nproblem.\n\nOn Mon, Mar 24, 2014 at 3:44 PM, Matthieu Moy\n wrote:\n> Scott Sandler writes:\n>\n>> Both pushes are\n>> determined to be fast-forwards and both succeed, but B' overwrites B\n>> and B is no longer on origin/master. The server does have B in its\n>> .git directory but the commit isn't on any branch.\n>\n> Is the reflog enabled on the server? If so, does it say anything about B\n> and B'?\n>\n> What filesystem do you use on the server? Is there any kind of NFS, and\n> if so are you sure that there's only one machine accessing the\n> filesystem at the same time?\n>\n> --\n> Matthieu Moy\n> http://www-verimag.imag.fr/~moy/\n"},{"id":"237491","messageId":"CACBZZX4ZEPA3sBp4-3QF6de0EWXzPkcOiqSxH3_CXV27Z=gxtw@mail.gmail.com","threadId":"36284","inReplyTo":"CAAyEjTN53+5B9Od9wW698wODNL3hR6Upot8-ZLwEksn3ir_zjA@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2014-03-24T21:16:52Z","receivedAt":"2014-03-24T21:16:52Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Mar 24, 2014 at 8:18 PM, Scott Sandler\n wrote:\n> I run a private Git repository (using Gitlab) with about 200 users\n> doing about 100 pushes per day.\n\nDitto but about 2x those numbers.\n\n> error: Ref refs/heads/master is at\n> 4584c1f34e07cea2df6abc8e0d407fe016017130 but expected\n> 61b79b6d35b066d054fb3deab550f1c51598cf5f\n> remote: error: failed to lock refs/heads/master\n\nI also see this error once in a while. I read the code a while back\nand it's basically because there's two levels of locks that\nreceive-pack tries to get, and it's possible for two pushers to get\nthe first lock due to a race condition.\n\nI've never seen data loss due to this though, because the inner lock is atomic.\n"},{"id":"237497","messageId":"CAAyEjTM8SSoJQ7+hb_nGEsbb6MC+z7=Y8SP9YQ8-i=_tRxLyuA@mail.gmail.com","threadId":"36284","inReplyTo":"CACBZZX4ZEPA3sBp4-3QF6de0EWXzPkcOiqSxH3_CXV27Z=gxtw@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-03-24T21:29:21Z","receivedAt":"2014-03-24T21:29:21Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"Right. Receiving that error is what happens during my testing with a\nhook that sleeps for 60s, and that outcome makes sense. But whatever\nis occurring in production must be different, since both users see\nsuccessful pushes with the first one just being overwritten.\n\nOn Mon, Mar 24, 2014 at 5:16 PM, Ævar Arnfjörð Bjarmason\n wrote:\n> On Mon, Mar 24, 2014 at 8:18 PM, Scott Sandler\n> wrote:\n>> I run a private Git repository (using Gitlab) with about 200 users\n>> doing about 100 pushes per day.\n>\n> Ditto but about 2x those numbers.\n>\n>> error: Ref refs/heads/master is at\n>> 4584c1f34e07cea2df6abc8e0d407fe016017130 but expected\n>> 61b79b6d35b066d054fb3deab550f1c51598cf5f\n>> remote: error: failed to lock refs/heads/master\n>\n> I also see this error once in a while. I read the code a while back\n> and it's basically because there's two levels of locks that\n> receive-pack tries to get, and it's possible for two pushers to get\n> the first lock due to a race condition.\n>\n> I've never seen data loss due to this though, because the inner lock is atomic.\n"},{"id":"237512","messageId":"vpqa9cf9rkv.fsf@anie.imag.fr","threadId":"36284","inReplyTo":"CAAyEjTNPqPHswbrrV9pRyXUUqD8dYzJaXQpWr+g3kuBERNLMRw@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-03-24T22:09:36Z","receivedAt":"2014-03-24T22:09:36Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Scott Sandler writes:\n\n> It's a bare repo and I didn't realize server-side reflogs were a\n> thing. Just ran \"git config core.logallrefupdates true\" in the repo on\n> the server which seems to be what I should do to enable that.\n\nThat should be it, yes.\n\n> The server does know about B, it shows up when you do \"git show B\".\n> However \"git branch --contains B\" returns nothing.\n\nThat is \"normal\": git push sends objects to the object store in a\nlockless manner, and then updates the reference corresponding to the\nbranch you're pushing to. So in case of concurrent access, the objects\nmay be sent, but the reference update will fail. Objects would be\ngarbage collected by a further \"git gc [--prune]\".\n\nThe \"not normal\" part is that the race condition on the ref update does\nactually break for you.\n\n> Gitlab's update hook maintains an event log when any push event\n> happens, who pushed and which commits. The most recent time this\n> happened, the first push which was lost occured at 2014-03-24 19:04:51\n> and the one that overwrote it happened at 2014-03-24 19:05:04. That's\n> when the update hook ran, not necessarily when the user hit \"git\n> push\", but it is notable that it's 13 seconds apart which is a pretty\n> long time. We do run several hooks for checking coding syntax and\n> various other things so it's believable to me that the hooks would\n> take more than 13 seconds on occasion, but based on the testing I did\n> with the sleep hook it didn't seem like the hooks were actually the\n> problem.\n\nAre you really, really, really sure that there's no force-push involved?\n(either \"push --force\" or \"push remotename +branchname\")\n\nWhat you describe really looks like a force-push, or a hook doing a ref\nupdate (e.g. a hook on a dev branch that updates master if the code\npasses tests or so).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"237515","messageId":"xmqqob0vnrnf.fsf@gitster.dls.corp.google.com","threadId":"36284","inReplyTo":"vpqa9cf9rkv.fsf@anie.imag.fr","subject":"Re: Git push race condition?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-24T22:44:20Z","receivedAt":"2014-03-24T22:44:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matthieu Moy writes:\n\n> What you describe really looks like a force-push, or a hook doing a ref\n> update (e.g. a hook on a dev branch that updates master if the code\n> passes tests or so).\n\n... or a filesystem that is broken. But I thought this is just a\nplain-vanilla ext4, nothing exotic, so.... Puzzled.\n"},{"id":"237516","messageId":"20140324225136.GA17080@sigill.intra.peff.net","threadId":"36284","inReplyTo":"CACBZZX4ZEPA3sBp4-3QF6de0EWXzPkcOiqSxH3_CXV27Z=gxtw@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-24T22:51:36Z","receivedAt":"2014-03-24T22:51:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 24, 2014 at 10:16:52PM +0100, Ævar Arnfjörð Bjarmason wrote:\n\n> > error: Ref refs/heads/master is at\n> > 4584c1f34e07cea2df6abc8e0d407fe016017130 but expected\n> > 61b79b6d35b066d054fb3deab550f1c51598cf5f\n> > remote: error: failed to lock refs/heads/master\n> \n> I also see this error once in a while. I read the code a while back\n> and it's basically because there's two levels of locks that\n> receive-pack tries to get, and it's possible for two pushers to get\n> the first lock due to a race condition.\n> \n> I've never seen data loss due to this though, because the inner lock is atomic.\n\nThe reason is that there are not 2 locks. Each side remembers the \"old\"\nvalue when it started the operation, and only takes a lock when it comes\ntime to write the ref (and then checks that the old value is still\ncurrent). Two pushes happening simultaneously do not have any idea that\nthe other is occurring.\n\n-Peff\n"},{"id":"237517","messageId":"20140324225434.GB17080@sigill.intra.peff.net","threadId":"36284","inReplyTo":"CAAyEjTN53+5B9Od9wW698wODNL3hR6Upot8-ZLwEksn3ir_zjA@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-24T22:54:34Z","receivedAt":"2014-03-24T22:54:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 24, 2014 at 03:18:14PM -0400, Scott Sandler wrote:\n\n> I've noticed that a few times in the past several weeks, we've had\n> events where pushes have been lost when two people pushed at just\n> about the same time. The scenario is that two users both have commits\n> based on commit A, call them B and B'. The user with commit B pushes\n> at about the same time as the user who pushes B'. Both pushes are\n> determined to be fast-forwards and both succeed, but B' overwrites B\n> and B is no longer on origin/master. The server does have B in its\n> .git directory but the commit isn't on any branch.\n\nWhat version of git are you running on the server? Is it possible that\nthere is a simultaneous process running `git pack-refs` (e.g., a `git\ngc` run by a cron job or similar)?\n\nThere were some race conditions fixed last year wherein git could see\nstale values of refs, but I do not think they could impact writing to a\nref like this. When we take the lock on the ref, we always go straight\nto the filesystem, so the value we see is up-to-date.\n\n-Peff\n"},{"id":"237518","messageId":"557DE2F7-1024-42A5-8192-ACE910CE6C81@codeaurora.org","threadId":"36284","inReplyTo":"20140324225434.GB17080@sigill.intra.peff.net","subject":"Re: Git push race condition?","fromName":"Nasser Grainawi","fromEmail":"nasser@codeaurora.org","sentAt":"2014-03-24T22:59:57Z","receivedAt":"2014-03-24T22:59:57Z","isPatch":false,"sender":{"key":"nasser@codeaurora.org","avatar":"https://avatars.githubusercontent.com/u/757421?v=4"},"body":"On Mar 24, 2014, at 4:54 PM, Jeff King wrote:\n\n> On Mon, Mar 24, 2014 at 03:18:14PM -0400, Scott Sandler wrote:\n> \n>> I've noticed that a few times in the past several weeks, we've had\n>> events where pushes have been lost when two people pushed at just\n>> about the same time. The scenario is that two users both have commits\n>> based on commit A, call them B and B'. The user with commit B pushes\n>> at about the same time as the user who pushes B'. Both pushes are\n>> determined to be fast-forwards and both succeed, but B' overwrites B\n>> and B is no longer on origin/master. The server does have B in its\n>> .git directory but the commit isn't on any branch.\n> \n> What version of git are you running on the server? Is it possible that\n> there is a simultaneous process running `git pack-refs` (e.g., a `git\n> gc` run by a cron job or similar)?\n\n`git gc --auto` could be getting triggered as well, so if you suspect\nthat you could set gc.auto=0 on the server side.\n\n> \n> There were some race conditions fixed last year wherein git could see\n> stale values of refs, but I do not think they could impact writing to a\n> ref like this. When we take the lock on the ref, we always go straight\n> to the filesystem, so the value we see is up-to-date.\n> \n> -Peff\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n\n-- \nThe Qualcomm Innovation Center, Inc. is a member of Code Aurora \nForum, hosted by The Linux Foundation\n"},{"id":"237656","messageId":"CAAyEjTPtaKExJJSc3yrxVNzx0DmOyeUFH-Uxz3dn0iezqc5VKA@mail.gmail.com","threadId":"36284","inReplyTo":"557DE2F7-1024-42A5-8192-ACE910CE6C81@codeaurora.org","subject":"Re: Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-03-25T13:45:20Z","receivedAt":"2014-03-25T13:45:20Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"Version of git on the server? git version 1.8.3-rc0\n\nIs there a hook or cron job that updates or gcs this repository or any\nrefs? No. No cron jobs touching the repo at all, and all the hooks are\nread-only. There are pre-receive hooks that either reject a push or\ndon't based on some checks, there are post-receive hooks that curl to\nnotify external services like jenkins, and there is gitlab's update\nhook which just either allows or denies a push based on gitlab's own\npermissions check, and then updates a redis queue\n(https://github.com/gitlabhq/gitlab-shell/blob/master/lib/gitlab_update.rb).\n\nAm I absolutely positively sure that it's not a force push? I'm pretty\nconfident they're not force pushing. This has happened now to 5\ndifferent pairs of developers using a variety of Git clients (Git\nbash, git cli on mac and linux, sourcetree) and most of them didn't\neven know what force push was until I asked them if they had done it.\nThey're all using ssh URLs for the git remote, not http, in case\nthat's relevant, and they showed me their git configurations and\nnothing looks amiss.\n\nAfter the first time it happened I also put a pre-receive hook in the\nrepository to prevent force pushes to master, since I thought that's\nwhat had happened:\n\nwhile read OLDREV NEWREV REFNAME\ndo\n if [ \"$REFNAME\" == \"refs/heads/master\" -a \"$OLDREV\" != $(git\nmerge-base $OLDREV $NEWREV) ]; then\n echo \"ERROR: It seems like you are trying to force-push on master.\"\n exit 1\n fi\ndone\n\nI did this so that people could still force push on other branches if\nthey really wanted to (since many mentioned they do this on their\nremote branches sometimes). I'm under the impression this hook works\nproperly since it rejects my attempted force pushes to master.\n\n\nOn Mon, Mar 24, 2014 at 6:59 PM, Nasser Grainawi wrote:\n> On Mar 24, 2014, at 4:54 PM, Jeff King wrote:\n>\n>> On Mon, Mar 24, 2014 at 03:18:14PM -0400, Scott Sandler wrote:\n>>\n>>> I've noticed that a few times in the past several weeks, we've had\n>>> events where pushes have been lost when two people pushed at just\n>>> about the same time. The scenario is that two users both have commits\n>>> based on commit A, call them B and B'. The user with commit B pushes\n>>> at about the same time as the user who pushes B'. Both pushes are\n>>> determined to be fast-forwards and both succeed, but B' overwrites B\n>>> and B is no longer on origin/master. The server does have B in its\n>>> .git directory but the commit isn't on any branch.\n>>\n>> What version of git are you running on the server? Is it possible that\n>> there is a simultaneous process running `git pack-refs` (e.g., a `git\n>> gc` run by a cron job or similar)?\n>\n> `git gc --auto` could be getting triggered as well, so if you suspect\n> that you could set gc.auto=0 on the server side.\n>\n>>\n>> There were some race conditions fixed last year wherein git could see\n>> stale values of refs, but I do not think they could impact writing to a\n>> ref like this. When we take the lock on the ref, we always go straight\n>> to the filesystem, so the value we see is up-to-date.\n>>\n>> -Peff\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n> --\n> The Qualcomm Innovation Center, Inc. is a member of Code Aurora\n> Forum, hosted by The Linux Foundation\n>\n"},{"id":"237658","messageId":"vpqvbv2pe8e.fsf@anie.imag.fr","threadId":"36284","inReplyTo":"CAAyEjTPtaKExJJSc3yrxVNzx0DmOyeUFH-Uxz3dn0iezqc5VKA@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-03-25T14:03:29Z","receivedAt":"2014-03-25T14:03:29Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Scott Sandler writes:\n\n> Is there a hook or cron job that updates or gcs this repository or any\n> refs? No. No cron jobs touching the repo at all, and all the hooks are\n> read-only.\n\nIf you activated the reflog, you can double-check that. Running git\nreflog on the server should give you something like this:\n\n$ git reflog show master\nbf40764 (HEAD, master) master@{0}: push\n2c4fc6d master@{1}: push\ne72211a master@{2}: push\n...\n\nIt should be possible to check the reflog for non-fast forward. I don't\nfind an obvious way, but a script going through the sha1 list and\nchecking that each line is an ancestor of the previous should be easy.\n\nI can't exclude the hypothesis of a bug in Git, but my feeling is that\nthere's an issue with your setup.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"237660","messageId":"CAAyEjTOy-DyeU96_DWWydgcpb+x5DMRkf1NHBfn+eNZ-yDrZUw@mail.gmail.com","threadId":"36284","inReplyTo":"vpqvbv2pe8e.fsf@anie.imag.fr","subject":"Re: Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-03-25T14:16:44Z","receivedAt":"2014-03-25T14:16:44Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"I'm definitely open to the possibility there's a problem with my\nsetup. I've got the reflog on now and will check what that looks like\nnext time the issue happens. So far it looks like you described:\n\nb2b202d master@{0}: push\n7c01312 master@{1}: push\n3635312 master@{2}: push\naea29bf master@{3}: push\n8bfe464 master@{4}: push\nfb35676 master@{5}: push\n267114e master@{6}: push\n2b5c822 master@{7}: push\n9d7206f master@{8}: push\n8fbeaf9 master@{9}: push\n\nI'd like to see it happen again under these conditions and get that\ninformation, then enable receive.denyNonFastForwards explicitly just\nto be sure it's not force pushes, and see if it still happens. If\nanyone has other ideas of things to look into or test, let me know.\n\nThanks,\nScott\n\nOn Tue, Mar 25, 2014 at 10:03 AM, Matthieu Moy\n wrote:\n> Scott Sandler writes:\n>\n>> Is there a hook or cron job that updates or gcs this repository or any\n>> refs? No. No cron jobs touching the repo at all, and all the hooks are\n>> read-only.\n>\n> If you activated the reflog, you can double-check that. Running git\n> reflog on the server should give you something like this:\n>\n> $ git reflog show master\n> bf40764 (HEAD, master) master@{0}: push\n> 2c4fc6d master@{1}: push\n> e72211a master@{2}: push\n> ...\n>\n> It should be possible to check the reflog for non-fast forward. I don't\n> find an obvious way, but a script going through the sha1 list and\n> checking that each line is an ancestor of the previous should be easy.\n>\n> I can't exclude the hypothesis of a bug in Git, but my feeling is that\n> there's an issue with your setup.\n>\n> --\n> Matthieu Moy\n> http://www-verimag.imag.fr/~moy/\n"},{"id":"237661","messageId":"vpqmwgepdc5.fsf@anie.imag.fr","threadId":"36284","inReplyTo":"CAAyEjTOy-DyeU96_DWWydgcpb+x5DMRkf1NHBfn+eNZ-yDrZUw@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-03-25T14:22:50Z","receivedAt":"2014-03-25T14:22:50Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"(please, don't top-post on this list)\n\nScott Sandler writes:\n\n> I'd like to see it happen again under these conditions and get that\n> information, then enable receive.denyNonFastForwards explicitly just\n> to be sure it's not force pushes, and see if it still happens.\n\nTo be really sure, you also have to set receive.denyDeletes to true.\nOtherwise, a workaround to perform a non-fast forward is to delete the\nbranch, and re-create it somewhere else in history.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"237662","messageId":"20140325145700.GA10132@sigill.intra.peff.net","threadId":"36284","inReplyTo":"CAAyEjTPtaKExJJSc3yrxVNzx0DmOyeUFH-Uxz3dn0iezqc5VKA@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-03-25T14:57:00Z","receivedAt":"2014-03-25T14:57:00Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Mar 25, 2014 at 09:45:20AM -0400, Scott Sandler wrote:\n\n> Version of git on the server? git version 1.8.3-rc0\n\nThere was significant work done between v1.8.3 and v1.8.4 on handling\nraces in the ref code. As I said before, I don't think the symptoms you\nare describing are anything we have seen, or that could be triggered by\nthe races we found (which were mostly to do with ref enumeration, not\nref writing). But I would suggest upgrading to a newer version of git as\na precaution.\n\nYou mentioned elsewhere turning on the reflog, which I think is a good\nidea. If there is a race of this sort, you will see a \"hole\" in the\nreflog, where a ref goes from A->B, then again from A->B' (whereas with\nnormal writes, it would be B->B').\n\n-Peff\n"},{"id":"238641","messageId":"CAAyEjTO3JTNDxDWpX+k_Z-O9=8-Vu5uyT_1eK-A8nFXWVcyD6w@mail.gmail.com","threadId":"36284","inReplyTo":"20140325145700.GA10132@sigill.intra.peff.net","subject":"Re: Git push race condition?","fromName":"Scott Sandler","fromEmail":"scott.m.sandler@gmail.com","sentAt":"2014-04-10T19:14:10Z","receivedAt":"2014-04-10T19:14:10Z","isPatch":false,"sender":{"key":"scott.m.sandler@gmail.com","avatar":"https://gravatar.com/avatar/98c9b68040f50624bddb62f0d58d9046ec21f454b4e323ad0e27301d09997cba?d=mp&s=160"},"body":"On Tue, Mar 25, 2014 at 10:57 AM, Jeff King wrote:\n> On Tue, Mar 25, 2014 at 09:45:20AM -0400, Scott Sandler wrote:\n>\n>> Version of git on the server? git version 1.8.3-rc0\n>\n> There was significant work done between v1.8.3 and v1.8.4 on handling\n> races in the ref code. As I said before, I don't think the symptoms you\n> are describing are anything we have seen, or that could be triggered by\n> the races we found (which were mostly to do with ref enumeration, not\n> ref writing). But I would suggest upgrading to a newer version of git as\n> a precaution.\n>\n> You mentioned elsewhere turning on the reflog, which I think is a good\n> idea. If there is a race of this sort, you will see a \"hole\" in the\n> reflog, where a ref goes from A->B, then again from A->B' (whereas with\n> normal writes, it would be B->B').\n>\n> -Peff\n\nThis finally happened again. Here's what the reflog looks like:\n\n2805f68 master@{0}: push\n96eebc0 master@{1}: push\n75bd4a6 master@{2}: push\nabc30da master@{3}: push\neba874f master@{4}: push\n10981e7 master@{5}: push\n76b3957 master@{6}: push\n2e3ea06 master@{7}: push\n9d4e778 master@{8}: push\ndbd70ae master@{9}: push\n508ab4f master@{10}: push\n36a0ce4 master@{11}: push\nddc258e master@{12}: push\ncf025de master@{13}: push\ndbd70ae master@{14}: push\n95d33eb master@{15}: push\n75b8e9a master@{16}: push\n\nThe commit that was lost does not show in the reflog at all, its short\nhash was e0de949aa. The commit that \"won the race\" against it is\n9d4e778 (I'm inferring this based on timing since it was pushed at\nabout the same time as the lost commit).\n\nOne interesting thing to note is that dbd70ae shows up at two separate\npoints in the reflog though, one being directly before the 9d4e778\ncommit that won the race. According to Gitlab's event log that commit\nwas just pushed once, right after 95d33eb and before cf025de as it\nshows in master@{14} there. The fact that the same commit shows up\nagain in master@{9} is interesting.\n\nNow that it has happened again and I've got this data, I'm going to\nupgrade git but let me know if this provides any insight in the mean\ntime.\n\n-Scott\n"},{"id":"238669","messageId":"vpqob08qpez.fsf@anie.imag.fr","threadId":"36284","inReplyTo":"CAAyEjTO3JTNDxDWpX+k_Z-O9=8-Vu5uyT_1eK-A8nFXWVcyD6w@mail.gmail.com","subject":"Re: Git push race condition?","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2014-04-11T07:46:28Z","receivedAt":"2014-04-11T07:46:28Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"\n> This finally happened again. Here's what the reflog looks like:\n>\n> 2805f68 master@{0}: push\n> 96eebc0 master@{1}: push\n> 75bd4a6 master@{2}: push\n> abc30da master@{3}: push\n> eba874f master@{4}: push\n> 10981e7 master@{5}: push\n> 76b3957 master@{6}: push\n> 2e3ea06 master@{7}: push\n> 9d4e778 master@{8}: push\n> dbd70ae master@{9}: push\n> 508ab4f master@{10}: push\n> 36a0ce4 master@{11}: push\n> ddc258e master@{12}: push\n> cf025de master@{13}: push\n> dbd70ae master@{14}: push\n> 95d33eb master@{15}: push\n> 75b8e9a master@{16}: push\n\nYou can have a look at the actual reflog (.git/logs/refs/heads/master)\nwhich contains a bit more information. It will show you the pairs\n(source, destination) for each change.\n\nNormally, the destination of a push is the source of the next one, but\nthat would be worth checking.\n\n> One interesting thing to note is that dbd70ae shows up at two separate\n> points in the reflog though, one being directly before the 9d4e778\n> commit that won the race. According to Gitlab's event log that commit\n> was just pushed once, right after 95d33eb and before cf025de as it\n> shows in master@{14} there. The fact that the same commit shows up\n> again in master@{9} is interesting.\n\nMy interpretation is that someone/something did a non-fast forward push\nat master@{9}, which reverted the history back to dbd70ae, and then\nmaster@{8} did a fast-forward, non-race condition, absolutely normal\npush.\n\nLook at the complete reflog line corresponding to master@{9}, it may\ngive you more information.\n\n> Now that it has happened again and I've got this data, I'm going to\n> upgrade git but let me know if this provides any insight in the mean\n> time.\n\nIf I were you, I'd keep a copy of the complete repo (including reflog &\nall) in case.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"}]}