{"thread":{"id":"43927","subject":"on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","startedAt":"2016-08-24T21:00:52Z","lastAt":"2016-08-26T00:09:13Z","messageCount":5,"participants":["Alex Nauda","Jeff King","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"300114","messageId":"CAMQLHmAraXPL=8SmMG4X_424FAzx4q2Byk8pva5wEOg7vNSqLw@mail.gmail.com","threadId":"43927","inReplyTo":null,"subject":"on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","fromName":"Alex Nauda","fromEmail":"alex@alexnauda.com","sentAt":"2016-08-24T20:52:33Z","receivedAt":"2016-08-24T21:00:52Z","isPatch":false,"sender":{"key":"alex@alexnauda.com","avatar":null},"body":"Elastic File System (EFS) is Amazon's scalable filesystem product that\nis exposed to the OS as an NFS mount. We're using EFS to host the\nfilesystem used by a Jenkins CI server. Sometimes when Jenkins tries\nto git fetch, we get this error:\n$ git -c core.askpass=true fetch --tags --progress\ngit@github.com:mediasilo/dodo.git\n+refs/pull/*:refs/remotes/origin/pr/*\nfatal: Reference directory conflict: refs/heads/\n$ echo $? 128\n\nHas anyone seen anything like this before? Any tips on how to troubleshoot it?\n\nRelated Jenkins issue: https://issues.jenkins-ci.org/browse/JENKINS-37653\n"},{"id":"300120","messageId":"20160824213900.gcnwxu46zvgpjr5a@sigill.intra.peff.net","threadId":"43927","inReplyTo":"CAMQLHmAraXPL=8SmMG4X_424FAzx4q2Byk8pva5wEOg7vNSqLw@mail.gmail.com","subject":"Re: on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-24T21:39:01Z","receivedAt":"2016-08-24T21:40:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 24, 2016 at 04:52:33PM -0400, Alex Nauda wrote:\n\n> Elastic File System (EFS) is Amazon's scalable filesystem product that\n> is exposed to the OS as an NFS mount. We're using EFS to host the\n> filesystem used by a Jenkins CI server. Sometimes when Jenkins tries\n> to git fetch, we get this error:\n> $ git -c core.askpass=true fetch --tags --progress\n> git@github.com:mediasilo/dodo.git\n> +refs/pull/*:refs/remotes/origin/pr/*\n> fatal: Reference directory conflict: refs/heads/\n> $ echo $? 128\n> \n> Has anyone seen anything like this before? Any tips on how to troubleshoot it?\n\nNo, I haven't seen it before. That's an internal assertion in the refs\ncode that shouldn't ever happen. It looks like it happens when the loose\nrefs end up with duplicate directory entries. While a bug in git is an\nobvious culprit, I wonder if it's possible that your filesystem might\nexpose the same name twice in one set of readdir() results.\n\n+cc Michael, who added this assertion long ago (and since this is the\nfirst report in all these years, it does make me suspect that the\nfilesystem is a critical part of reproducing).\n\n-Peff\n"},{"id":"300133","messageId":"67cde0aa-89fc-b574-d5f5-7e754d77b5bb@alum.mit.edu","threadId":"43927","inReplyTo":"20160824213900.gcnwxu46zvgpjr5a@sigill.intra.peff.net","subject":"Re: on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-08-25T06:28:43Z","receivedAt":"2016-08-25T06:28:58Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 08/24/2016 11:39 PM, Jeff King wrote:\n> On Wed, Aug 24, 2016 at 04:52:33PM -0400, Alex Nauda wrote:\n> \n>> Elastic File System (EFS) is Amazon's scalable filesystem product that\n>> is exposed to the OS as an NFS mount. We're using EFS to host the\n>> filesystem used by a Jenkins CI server. Sometimes when Jenkins tries\n>> to git fetch, we get this error:\n>> $ git -c core.askpass=true fetch --tags --progress\n>> git@github.com:mediasilo/dodo.git\n>> +refs/pull/*:refs/remotes/origin/pr/*\n>> fatal: Reference directory conflict: refs/heads/\n>> $ echo $? 128\n>>\n>> Has anyone seen anything like this before? Any tips on how to troubleshoot it?\n> \n> No, I haven't seen it before. That's an internal assertion in the refs\n> code that shouldn't ever happen. It looks like it happens when the loose\n> refs end up with duplicate directory entries. While a bug in git is an\n> obvious culprit, I wonder if it's possible that your filesystem might\n> expose the same name twice in one set of readdir() results.\n> \n> +cc Michael, who added this assertion long ago (and since this is the\n> first report in all these years, it does make me suspect that the\n> filesystem is a critical part of reproducing).\n\nThanks for the CC.\n\nI've never heard of this problem before.\n\nWhat Git version are you using?\n\nI tried to provoke the problem by hand-corrupting the packed-refs file,\nbut wasn't successful.\n\nSo Peff's suggestion that the problem originates in your filesystem\nseems to be to be the most likely cause. A quick Google search found,\nfor example,\n\n    https://bugzilla.redhat.com/show_bug.cgi?id=739222\n\nhttp://superuser.com/questions/640419/how-can-i-have-two-files-with-the-same-name-in-a-directory-when-mounted-with-nfs\n\nthough these reports seem connected with having lots of files in the\ndirectory, which seems unlikely for `$GIT_DIR/refs/`. But I didn't do a\nmore careful search, and it is easily possible that there are other bugs\nin NFS (or EFS) that could be affecting you.\n\nIf this were repeatable, you could run Git under strace to test Peff's\nhypothesis. But I suppose it only happens rarely, right?\n\nIs it possible that multiple clients have the same NFS filesystem\nmounted while Git is running? That would seem like an especially bad\nidea and I could imagine it leading to problems like this.\n\nIt's surprising that you are seeing this problem in directory `refs`,\nbecause (1) that directory is unlikely to have very many entries, and\n(2) as far as I remember, Git will never delete the directories\n`refs/heads` and `refs/tags`.\n\nMichael\n\n"},{"id":"300174","messageId":"CAMQLHmA=b0HMO9s0nt3VTdOYe51oE3UrrrCDH3Z_Tq0Nuhe-kQ@mail.gmail.com","threadId":"43927","inReplyTo":"67cde0aa-89fc-b574-d5f5-7e754d77b5bb@alum.mit.edu","subject":"Re: on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","fromName":"Alex Nauda","fromEmail":"alex@alexnauda.com","sentAt":"2016-08-25T16:01:57Z","receivedAt":"2016-08-25T16:02:05Z","isPatch":false,"sender":{"key":"alex@alexnauda.com","avatar":null},"body":"On Thu, Aug 25, 2016 at 2:28 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:\n> On 08/24/2016 11:39 PM, Jeff King wrote:\n>> On Wed, Aug 24, 2016 at 04:52:33PM -0400, Alex Nauda wrote:\n>>\n>>> Elastic File System (EFS) is Amazon's scalable filesystem product that\n>>> is exposed to the OS as an NFS mount. We're using EFS to host the\n>>> filesystem used by a Jenkins CI server. Sometimes when Jenkins tries\n>>> to git fetch, we get this error:\n>>> $ git -c core.askpass=true fetch --tags --progress\n>>> git@github.com:mediasilo/dodo.git\n>>> +refs/pull/*:refs/remotes/origin/pr/*\n>>> fatal: Reference directory conflict: refs/heads/\n>>> $ echo $? 128\n>>>\n>>> Has anyone seen anything like this before? Any tips on how to troubleshoot it?\n>>\n>> No, I haven't seen it before. That's an internal assertion in the refs\n>> code that shouldn't ever happen. It looks like it happens when the loose\n>> refs end up with duplicate directory entries. While a bug in git is an\n>> obvious culprit, I wonder if it's possible that your filesystem might\n>> expose the same name twice in one set of readdir() results.\n>>\n>> +cc Michael, who added this assertion long ago (and since this is the\n>> first report in all these years, it does make me suspect that the\n>> filesystem is a critical part of reproducing).\n>\n> Thanks for the CC.\n>\n> I've never heard of this problem before.\n>\n> What Git version are you using?\nGit client 2.7.4 against GitHub (Git 2.6.5)\n\n>\n> I tried to provoke the problem by hand-corrupting the packed-refs file,\n> but wasn't successful.\n>\n> So Peff's suggestion that the problem originates in your filesystem\n> seems to be to be the most likely cause. A quick Google search found,\n> for example,\n>\n>     https://bugzilla.redhat.com/show_bug.cgi?id=739222\n>\n> http://superuser.com/questions/640419/how-can-i-have-two-files-with-the-same-name-in-a-directory-when-mounted-with-nfs\n>\n> though these reports seem connected with having lots of files in the\n> directory, which seems unlikely for `$GIT_DIR/refs/`. But I didn't do a\n> more careful search, and it is easily possible that there are other bugs\n> in NFS (or EFS) that could be affecting you.\n>\n> If this were repeatable, you could run Git under strace to test Peff's\n> hypothesis. But I suppose it only happens rarely, right?\nActually it seems to be reproducible. Here's the last portion of an strace:\n\n[...]\nstat(\".git/refs/remotes/origin/pr/7/head\", {st_mode=S_IFREG|0644,\nst_size=41, ...}) = 0\nlstat(\".git/refs/remotes/origin/pr/7/head\", {st_mode=S_IFREG|0644,\nst_size=41, ...}) = 0\nopen(\".git/refs/remotes/origin/pr/7/head\", O_RDONLY) = 4\nread(4, \"5d82811a248900efd8e201c6d9232de5\"..., 256) = 41\nread(4, \"\", 215)                        = 0\nclose(4)                                = 0\ngetdents(3, /* 0 entries */, 32768)     = 0\nclose(3)                                = 0\nopen(\".git/refs/remotes/origin/pr/16/\",\nO_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3\nfstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0\ngetdents(3, /* 3 entries */, 32768)     = 72\nstat(\".git/refs/remotes/origin/pr/16/head\", {st_mode=S_IFREG|0644,\nst_size=41, ...}) = 0\nlstat(\".git/refs/remotes/origin/pr/16/head\", {st_mode=S_IFREG|0644,\nst_size=41, ...}) = 0\nopen(\".git/refs/remotes/origin/pr/16/head\", O_RDONLY) = 4\nread(4, \"2886c4f3ba8c3b5c2306029f6e39498d\"..., 256) = 41\nread(4, \"\", 215)                        = 0\nclose(4)                                = 0\ngetdents(3, /* 0 entries */, 32768)     = 0\nclose(3)                                = 0\nopen(\".git/refs/tags/\", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3\nfstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0\ngetdents(3, /* 2 entries */, 32768)     = 48\ngetdents(3, /* 0 entries */, 32768)     = 0\nclose(3)                                = 0\nopen(\".git/refs/bisect/\", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) =\n-1 ENOENT (No such file or directory)\nopen(\".git/packed-refs\", O_RDONLY)      = -1 ENOENT (No such file or directory)\nfstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0\nwrite(2, \"fatal: Reference directory confl\"..., 58fatal: Reference\ndirectory conflict: refs/remotes/origin/\n) = 58\nexit_group(128)                         = ?\n+++ exited with 128 +++\n\n>\n> Is it possible that multiple clients have the same NFS filesystem\n> mounted while Git is running? That would seem like an especially bad\n> idea and I could imagine it leading to problems like this.\n>\n> It's surprising that you are seeing this problem in directory `refs`,\n> because (1) that directory is unlikely to have very many entries, and\n> (2) as far as I remember, Git will never delete the directories\n> `refs/heads` and `refs/tags`.\nSeems like sometimes it happens on other directories:\nrefs/remotes/origin/ or refs/remotes/origin/pr/1\nThen as I was stracing it again, suddenly it succeeded. Some kind of\nrace condition?\n\n>\n> Michael\n>\n"},{"id":"300222","messageId":"1d783e10-c1ce-4f17-4a54-8798b16697b2@alum.mit.edu","threadId":"43927","inReplyTo":"CAMQLHmA=b0HMO9s0nt3VTdOYe51oE3UrrrCDH3Z_Tq0Nuhe-kQ@mail.gmail.com","subject":"Re: on Amazon EFS (NFS): \"Reference directory conflict: refs/heads/\" with status code 128","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2016-08-26T00:06:43Z","receivedAt":"2016-08-26T00:09:13Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 08/25/2016 06:01 PM, Alex Nauda wrote:\n> On Thu, Aug 25, 2016 at 2:28 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:\n>> On 08/24/2016 11:39 PM, Jeff King wrote:\n>>> On Wed, Aug 24, 2016 at 04:52:33PM -0400, Alex Nauda wrote:\n>>>\n>>>> Elastic File System (EFS) is Amazon's scalable filesystem product that\n>>>> is exposed to the OS as an NFS mount. We're using EFS to host the\n>>>> filesystem used by a Jenkins CI server. Sometimes when Jenkins tries\n>>>> to git fetch, we get this error:\n>>>> $ git -c core.askpass=true fetch --tags --progress\n>>>> git@github.com:mediasilo/dodo.git\n>>>> +refs/pull/*:refs/remotes/origin/pr/*\n>>>> fatal: Reference directory conflict: refs/heads/\n>>>> $ echo $? 128\n>>>>\n>>>> Has anyone seen anything like this before? Any tips on how to troubleshoot it?\n>>>\n>>> No, I haven't seen it before. That's an internal assertion in the refs\n>>> code that shouldn't ever happen. It looks like it happens when the loose\n>>> refs end up with duplicate directory entries. While a bug in git is an\n>>> obvious culprit, I wonder if it's possible that your filesystem might\n>>> expose the same name twice in one set of readdir() results.\n>>>\n>>> +cc Michael, who added this assertion long ago (and since this is the\n>>> first report in all these years, it does make me suspect that the\n>>> filesystem is a critical part of reproducing).\n>>\n>> Thanks for the CC.\n>>\n>> I've never heard of this problem before.\n>>\n>> What Git version are you using?\n> Git client 2.7.4 against GitHub (Git 2.6.5)\n> \n>>\n>> I tried to provoke the problem by hand-corrupting the packed-refs file,\n>> but wasn't successful.\n>>\n>> So Peff's suggestion that the problem originates in your filesystem\n>> seems to be to be the most likely cause. A quick Google search found,\n>> for example,\n>>\n>>     https://bugzilla.redhat.com/show_bug.cgi?id=739222\n>>\n>> http://superuser.com/questions/640419/how-can-i-have-two-files-with-the-same-name-in-a-directory-when-mounted-with-nfs\n>>\n>> though these reports seem connected with having lots of files in the\n>> directory, which seems unlikely for `$GIT_DIR/refs/`. But I didn't do a\n>> more careful search, and it is easily possible that there are other bugs\n>> in NFS (or EFS) that could be affecting you.\n>>\n>> If this were repeatable, you could run Git under strace to test Peff's\n>> hypothesis. But I suppose it only happens rarely, right?\n> Actually it seems to be reproducible. Here's the last portion of an strace:\n> \n> [...]\n> stat(\".git/refs/remotes/origin/pr/7/head\", {st_mode=S_IFREG|0644,\n> st_size=41, ...}) = 0\n> lstat(\".git/refs/remotes/origin/pr/7/head\", {st_mode=S_IFREG|0644,\n> st_size=41, ...}) = 0\n> open(\".git/refs/remotes/origin/pr/7/head\", O_RDONLY) = 4\n> read(4, \"5d82811a248900efd8e201c6d9232de5\"..., 256) = 41\n> read(4, \"\", 215)                        = 0\n> close(4)                                = 0\n> getdents(3, /* 0 entries */, 32768)     = 0\n> close(3)                                = 0\n> open(\".git/refs/remotes/origin/pr/16/\",\n> O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3\n> fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0\n> getdents(3, /* 3 entries */, 32768)     = 72\n> stat(\".git/refs/remotes/origin/pr/16/head\", {st_mode=S_IFREG|0644,\n> st_size=41, ...}) = 0\n> lstat(\".git/refs/remotes/origin/pr/16/head\", {st_mode=S_IFREG|0644,\n> st_size=41, ...}) = 0\n> open(\".git/refs/remotes/origin/pr/16/head\", O_RDONLY) = 4\n> read(4, \"2886c4f3ba8c3b5c2306029f6e39498d\"..., 256) = 41\n> read(4, \"\", 215)                        = 0\n> close(4)                                = 0\n> getdents(3, /* 0 entries */, 32768)     = 0\n> close(3)                                = 0\n> open(\".git/refs/tags/\", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3\n> fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0\n> getdents(3, /* 2 entries */, 32768)     = 48\n> getdents(3, /* 0 entries */, 32768)     = 0\n> close(3)                                = 0\n> open(\".git/refs/bisect/\", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) =\n> -1 ENOENT (No such file or directory)\n> open(\".git/packed-refs\", O_RDONLY)      = -1 ENOENT (No such file or directory)\n> fstat(2, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 3), ...}) = 0\n> write(2, \"fatal: Reference directory confl\"..., 58fatal: Reference\n> directory conflict: refs/remotes/origin/\n> ) = 58\n> exit_group(128)                         = ?\n> +++ exited with 128 +++\n\nThanks for the additional information.\n\nFrom the strace output it is clear that there is no packed-refs file at\nthe time of the problem, so the problem must be among the loose refs.\n\nThe error is a \"Reference directory conflict\", which suggests that\n\"refs/remotes/origin/\" appears in two entries; once as a reference\ndirectory and once as a reference. But in fact it could also mean that\n\"refs/remotes/origin/\" appears twice, both as directories. Neither one\nshould happen in normal operation.\n\nUnfortunately there is not enough strace output to see whether (in this\ncase) path `refs/remotes/origin` was reported twice by `getdents()`. I\nthink that is still the most likely hypothesis, especially since the\nreports of this problem all seem to be on NFS + EBS.\n\nOne other long shot:\n\nI see that you are fetching into refs/remotes/origin/pr/*. What is the\nfull refspec configuration for origin?\n\nI know it used to be recommended to set up *two* refspecs when fetching\nfrom GitHub:\n\n    +refs/heads/*:refs/remotes/origin/*\n    +refs/pull/*:refs/remotes/origin/pr/*\n\nThis is not such a good idea (and I think it is no longer recommended)\nbecause, for example, a remote reference named `refs/heads/pr/5/head`\nwould want to end up at the same path as a remote PR branch\n`refs/pull/5/head`, namely `refs/remotes/origin/pr/5/head`. That\nshouldn't be allowed by Git, but since this sounds a little bit similar\nto your problem I thought I'd ask anyway.\n\nAre you using such a configuration? If so, can you reproduce the problem\nwith the `refs/pull/*` refspec disabled?\n\nMichael\n\n"}]}