{"thread":{"id":"44716","subject":"Races on ref .lock files?","startedAt":"2016-12-16T16:48:24Z","lastAt":"2016-12-21T21:08:29Z","messageCount":6,"participants":["Andreas Krey","Junio C Hamano","Bryan Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"307888","messageId":"20161216164751.GA12174@inner.h.apk.li","threadId":"44716","inReplyTo":null,"subject":"Races on ref .lock files?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2016-12-16T16:47:51Z","receivedAt":"2016-12-16T16:48:24Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"Hi all,\n\nWe're occasionally seeing a lot of \n\n error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.\n\nfrom the server side with fetches as well as pushes. (Bitbucket server.)\n\nWhat I find strange is that neither the fetches nor the pushes even\ntouch these refs (but the bitbucket triggers underneath might).\n\nBut my question is whether there are race conditions that can cause\nsuch messages in regular operation - they continue with 'If no other git\nprocess is currently running, this probably means a git process crashed\nin this repository earlier.' which indicates some level of anticipation.\n\n- Andreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds \nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"307889","messageId":"xmqqpokru6yg.fsf@gitster.mtv.corp.google.com","threadId":"44716","inReplyTo":"20161216164751.GA12174@inner.h.apk.li","subject":"Re: Races on ref .lock files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-16T17:20:07Z","receivedAt":"2016-12-16T17:21:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Krey writes:\n\n> We're occasionally seeing a lot of \n>\n> error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.\n>\n> from the server side with fetches as well as pushes. (Bitbucket server.)\n>\n> What I find strange is that neither the fetches nor the pushes even\n> touch these refs (but the bitbucket triggers underneath might).\n>\n> But my question is whether there are race conditions that can cause\n> such messages in regular operation - they continue with 'If no other git\n> process is currently running, this probably means a git process crashed\n> in this repository earlier.' which indicates some level of anticipation.\n\nI think (and I think you also think) these messages come from the\nBitbucket side, not your \"git push\" (or \"git fetch\"). Not having\nseen Bitbucket's sources, I can only guess, but assuming that its\npull-request is triggered from their Web frontend like GitHub's\ndoes, it is quite possible when you try to \"push\" into (or \"fetch\"\nfrom, for that matter) a repository, somebody is clicking a button\nto create that ref. We do not know what their \"receive-pack\" that\nresponds to your \"git push\" (or \"upload-pack\" for \"git fetch\") does\nwhen there are locked refs. I'd naively think that unless you are\npushing to that ref you showed an error message for, the receiving\nend shouldn't care if the ref is being written by somebody else, but\nwho knows ;-) They may have their own reasons wanting to lock that\nref that we think would be irrelevant for the operation, causing\nerrors.\n\n\n\n"},{"id":"307929","messageId":"CAGyf7-EvbV4XWCsfLpCUDo5V4_wM3SSJcHxVh9Rp78JUC6S-yw@mail.gmail.com","threadId":"44716","inReplyTo":"xmqqpokru6yg.fsf@gitster.mtv.corp.google.com","subject":"Re: Races on ref .lock files?","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2016-12-16T23:34:22Z","receivedAt":"2016-12-16T23:34:29Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"Andreas,\n\nBitbucket Server developer here. Typically these errors on your client\nare indicative of git gc --auto being triggered by git-receive-pack on\nthe server. Auto GC directly attached to a push in a repository with\npull requests often fails due to concurrent ref updates linked to\nbackground pull request processing.\n\nIf you'd like to investigate more in depth, I'd encourage you to\ncreate a ticket on support.atlassian.com so we can work with you.\nOtherwise, if you just want to prevent seeing these messages, you can\neither fork the relevant repository in Bitbucket Server (which\ndisables auto GC), or run \"git config gc.auto 0\" in\n/opt/apps/.../repositories/68. Once auto GC is disabled, Bitbucket\nServer will automatically take over managing GC for the repository\nwithout any additional configuration required.\n\nNote that we're working on revamping our GC handling such that auto GC\nwill always be disabled for all repositories and managed explicitly\nwithin Bitbucket Server instead, so a future upgrade should\nautomatically prevent these messages from appearing on clients.\n\nBest regards,\nBryan Turner\n\nOn Fri, Dec 16, 2016 at 9:20 AM, Junio C Hamano wrote:\n> Andreas Krey writes:\n>\n>> We're occasionally seeing a lot of\n>>\n>> error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.\n>>\n>> from the server side with fetches as well as pushes. (Bitbucket server.)\n>>\n>> What I find strange is that neither the fetches nor the pushes even\n>> touch these refs (but the bitbucket triggers underneath might).\n>>\n>> But my question is whether there are race conditions that can cause\n>> such messages in regular operation - they continue with 'If no other git\n>> process is currently running, this probably means a git process crashed\n>> in this repository earlier.' which indicates some level of anticipation.\n>\n> I think (and I think you also think) these messages come from the\n> Bitbucket side, not your \"git push\" (or \"git fetch\"). Not having\n> seen Bitbucket's sources, I can only guess, but assuming that its\n> pull-request is triggered from their Web frontend like GitHub's\n> does, it is quite possible when you try to \"push\" into (or \"fetch\"\n> from, for that matter) a repository, somebody is clicking a button\n> to create that ref. We do not know what their \"receive-pack\" that\n> responds to your \"git push\" (or \"upload-pack\" for \"git fetch\") does\n> when there are locked refs. I'd naively think that unless you are\n> pushing to that ref you showed an error message for, the receiving\n> end shouldn't care if the ref is being written by somebody else, but\n> who knows ;-) They may have their own reasons wanting to lock that\n> ref that we think would be irrelevant for the operation, causing\n> errors.\n>\n>\n>\n"},{"id":"308194","messageId":"20161221083335.GA1206@inner.h.apk.li","threadId":"44716","inReplyTo":"CAGyf7-EvbV4XWCsfLpCUDo5V4_wM3SSJcHxVh9Rp78JUC6S-yw@mail.gmail.com","subject":"Re: Races on ref .lock files?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2016-12-21T08:33:35Z","receivedAt":"2016-12-21T08:33:47Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Fri, 16 Dec 2016 15:34:22 +0000, Bryan Turner wrote:\n...\n> Bitbucket Server developer here.\n\nSocial media rock. :-)\n\n> If you'd like to investigate more in depth, I'd encourage you to\n> create a ticket on support.atlassian.com so we can work with you.\n\nThat is going to be postponed until we update our bitbucket instance\nto the current state.\n\n> Otherwise, if you just want to prevent seeing these messages, you can\n> either fork the relevant repository in Bitbucket Server (which\n> disables auto GC), or run \"git config gc.auto 0\" in\n\nDoing that for now. Will come back either if it doesn't help,\nor after the upgrade.\n\n> within Bitbucket Server instead, so a future upgrade should\n> automatically prevent these messages from appearing on clients.\n\nI still wonder if git itself should prevent these, or is there\na (git level) recommendation not to enable auto-gc in repos where\npeople regularly push to?\n\n- Andreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds \nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"308197","messageId":"20161221100033.GB1206@inner.h.apk.li","threadId":"44716","inReplyTo":"xmqqpokru6yg.fsf@gitster.mtv.corp.google.com","subject":"Re: Races on ref .lock files?","fromName":"Andreas Krey","fromEmail":"a.krey@gmx.de","sentAt":"2016-12-21T10:00:33Z","receivedAt":"2016-12-21T10:00:44Z","isPatch":false,"sender":{"key":"a.krey@gmx.de","avatar":"https://avatars.githubusercontent.com/u/37810?v=4"},"body":"On Fri, 16 Dec 2016 09:20:07 +0000, Junio C Hamano wrote:\n...\n> > error: cannot lock ref 'stash-refs/pull-requests/18978/to': Unable to create '/opt/apps/..../repositories/68/stash-refs/pull-requests/18978/to.lock': File exists.\n...\n> I think (and I think you also think) these messages come from the\n> Bitbucket side, not your \"git push\" (or \"git fetch\").\n\nI *know* that this is the case - we don't have refs like that\non the local side. (Our users are more scared about them.)\n\n...\n> when there are locked refs. I'd naively think that unless you are\n> pushing to that ref you showed an error message for, the receiving\n> end shouldn't care if the ref is being written by somebody else, but\n> who knows ;-) They may have their own reasons wanting to lock that\n> ref that we think would be irrelevant for the operation, causing\n> errors.\n\nPossible. I'm going Byrans way for now, disabling the gc there.\n\nBut:\n\nIn a different instance, we have a simple bare git repo that we\nuse for backup purposes. Which means there are lots of pushes\ngoing there (all to disjunct refs), and I now cared to look\ninto those logfiles:\n\n----snip\nWed Dec 21 05:08:14 CET 2016\nfatal: Unable to create '/data/git-backup/backup.git/packed-refs.lock': File exists.\n\nIf no other git process is currently running, this probably means a\ngit process crashed in this repository earlier. Make sure no other git\nprocess is running and remove the file manually to continue.\nerror: failed to run pack-refs\nTo git-backup-user@socrepo.advantest.com:backup.git\n + 8aac9ae...2df6d56 refs/zz/current -> refs/backup/socvm217/ZworkspacesZsocvm217ZjohanabtZws-release_tools.Ycurr (forced update)\n----snip\n\nI interpret this as \"I updated the refs files, but packing them\ndidn't work because someone else was also packing right now.\"\n\nIs that happening as designed, or do I need to be afraid\nthat some refs didn't make the push?\n\nTo ask differently, is git relying on people reading such\nmessages and following up on them? And thus isn't that\neasy to use in automated processes? (Additional problem:\nThe user in question, besides being an automat, doesn't\nhave the capability to work in the target repository.)\n\nAndreas\n\n-- \n\"Totally trivial. Famous last words.\"\nFrom: Linus Torvalds \nDate: Fri, 22 Jan 2010 07:29:21 -0800\n"},{"id":"308216","messageId":"xmqq4m1xhtx6.fsf@gitster.mtv.corp.google.com","threadId":"44716","inReplyTo":"20161221100033.GB1206@inner.h.apk.li","subject":"Re: Races on ref .lock files?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-21T21:08:21Z","receivedAt":"2016-12-21T21:08:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Krey writes:\n\n> In a different instance, we have a simple bare git repo that we\n> use for backup purposes. Which means there are lots of pushes\n> going there (all to disjunct refs), and I now cared to look\n> into those logfiles:\n>\n> ----snip\n> Wed Dec 21 05:08:14 CET 2016\n> fatal: Unable to create '/data/git-backup/backup.git/packed-refs.lock': File exists.\n>\n> If no other git process is currently running, this probably means a\n> git process crashed in this repository earlier. Make sure no other git\n> process is running and remove the file manually to continue.\n> error: failed to run pack-refs\n> To git-backup-user@socrepo.advantest.com:backup.git\n> + 8aac9ae...2df6d56 refs/zz/current -> refs/backup/socvm217/ZworkspacesZsocvm217ZjohanabtZws-release_tools.Ycurr (forced update)\n> ----snip\n>\n> I interpret this as \"I updated the refs files, but packing them\n> didn't work because someone else was also packing right now.\"\n\nCorrect.\n\n> Is that happening as designed, or do I need to be afraid\n> that some refs didn't make the push?\n\nCorrect and No. Packing refs into the packed-refs file is merely a\nperformance thing and done under the lock (needless to say, updating\nindividual refs is also done under the lock). Your push may have\ncompeted with somebody else's push that started earlier and you may\nhave given up packing refs, but no ill effect should be left behind.\n\nWhen the lock holder (the other guy who competes with your push)\nstuffs refs into a packed-refs file, the values for the refs you\npushed may not be in the packed-refs file, because the other guy may\nhave observed and captured the value before your push updated them.\nThose refs updated by you that are missed by the other guy will be\nleft as loose refs. Because whenever Git tries to find the value\nfor a ref, it always checks the loose refs first, there is no issue\ndue to this.\n\n"}]}