{"thread":{"id":"33318","subject":"Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","startedAt":"2013-03-28T10:22:08Z","lastAt":"2013-04-07T18:56:36Z","messageCount":7,"participants":["Kenneth Ölwing","Thomas Rast","Jason Pyeron"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"212500","messageId":"515419D0.7030107@olwing.se","threadId":"33318","inReplyTo":null,"subject":"Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Kenneth Ölwing","fromEmail":"kenneth@olwing.se","sentAt":"2013-03-28T10:22:08Z","receivedAt":"2013-03-28T10:22:08Z","isPatch":false,"sender":{"key":"kenneth@olwing.se","avatar":null},"body":"Hi,\n\nI'm hoping to hear some wisdom on the subject so I can decide if I'm \nchasing a pipe dream or if it should be expected to work and I just need \nto work out the kinks.\n\nFinding things like this makes it sound possible:\n   http://permalink.gmane.org/gmane.comp.version-control.git/122670\nbut then again, in threads like this:\n   http://kerneltrap.org/mailarchive/git/2010/11/14/44799\nopinions are that it's just not reliable to trust.\n\nMy experience so far is that I eventually get repo corruption when I \nstress it with concurrent read/write access from multiple hosts (beyond \nany sort of likely levels, but still). Maybe I'm doing something wrong, \nmissing a configuration setting somewhere, put an unfair stress on the \nsystem, there's a bona fide bug - or, given the inherent difficulty in \nachieving perfect coherency between machines on what's visible on the \nmount, it's just impossible (?) to truly get it working under all \nsituations.\n\nMy eventual usecase is to set up a bunch of (gitolite) hosts that all \nare effectively identical and all seeing the same storage and clients \ncan then be directed to any of them to get served. For the purpose of \nthis query I assume it can be boiled down to be the same as n users \nworking on their own workstations and accessing a central repo on an NFS \nshare they all have mounted, using regular file paths. Correct?\n\nI have a number of load-generating test scripts (that from humble \nbeginnings have grown to beasts), but basically, they fork a number of \ncode pieces that proceed to hammer the repo with concurrent reading and \nwriting. If necessary, the scripts can be started on different hosts, \ntoo. It's all about the central repo so clients will retry in various \nmodes whenever they get an error, up to re-cloning and starting over. \nAll in all, they can yield quite a load.\n\nOn a local filesystem everything seems to be holding up fine which is \nexpected. In the scenario with concurrency on top of shared NFS storage \nhowever, the scripts eventually fails with various problems (when the \ntiming finally finds a hole, I guess) - possible to clone but checkouts \nfails, errors about refs pointing wrong and errors where the original \nrepo is actually corrupted and can't be cloned from. Depending on test \nstrategy, I've also had everything going fine (ops looks fine in my \nlogs), but afterwards the repo has lost a branch or two.\n\nI've experimented with various NFS settings (e.g. turning off the \nattribute cache), but haven't reached a conclusion. I do suspect that it \njust is a fact of life with a remote filesystem to have coherency \nproblems with high concurrency like this but I'd be happily proven wrong \n- I'm not an expert in either NFS or git.\n\nSo, any opionions either way would be valuable, e.g. 'it should work' or \n'it can never work 100%' is fine, as well as any suggestions. Obviously, \nthe concurrency needed to make it probable to hit this seems so unlikely \nthat maybe I just shouldn't worry...\n\nI'd be happy to share scripts and whatever if someone would like to try \nit out themselves.\n\nThanks for your time,\n\nken1\n"},{"id":"213299","messageId":"515EC51C.9070206@olwing.se","threadId":"33318","inReplyTo":"515419D0.7030107@olwing.se","subject":"Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Kenneth Ölwing","fromEmail":"kenneth@olwing.se","sentAt":"2013-04-05T12:35:40Z","receivedAt":"2013-04-05T12:35:40Z","isPatch":false,"sender":{"key":"kenneth@olwing.se","avatar":null},"body":"Hi\n\nBasically, I'm at a place where I'm considering giving up getting this \nto work reliably. In general, my setup work really fine, except for the \nitty-bitty detail that when I put pressure on things I tend to get into \nvarious kinds of trouble with the central repo being corrupted.\n\nCan anyone authoritatively state anything either way?\n\nTIA,\n\nken1\n\nOn 2013-03-28 11:22, Kenneth Ölwing wrote:\n> Hi,\n>\n> I'm hoping to hear some wisdom on the subject so I can decide if I'm \n> chasing a pipe dream or if it should be expected to work and I just \n> need to work out the kinks.\n>\n> Finding things like this makes it sound possible:\n>   http://permalink.gmane.org/gmane.comp.version-control.git/122670\n> but then again, in threads like this:\n>   http://kerneltrap.org/mailarchive/git/2010/11/14/44799\n> opinions are that it's just not reliable to trust.\n>\n> My experience so far is that I eventually get repo corruption when I \n> stress it with concurrent read/write access from multiple hosts \n> (beyond any sort of likely levels, but still). Maybe I'm doing \n> something wrong, missing a configuration setting somewhere, put an \n> unfair stress on the system, there's a bona fide bug - or, given the \n> inherent difficulty in achieving perfect coherency between machines on \n> what's visible on the mount, it's just impossible (?) to truly get it \n> working under all situations.\n>\n> My eventual usecase is to set up a bunch of (gitolite) hosts that all \n> are effectively identical and all seeing the same storage and clients \n> can then be directed to any of them to get served. For the purpose of \n> this query I assume it can be boiled down to be the same as n users \n> working on their own workstations and accessing a central repo on an \n> NFS share they all have mounted, using regular file paths. Correct?\n>\n> I have a number of load-generating test scripts (that from humble \n> beginnings have grown to beasts), but basically, they fork a number of \n> code pieces that proceed to hammer the repo with concurrent reading \n> and writing. If necessary, the scripts can be started on different \n> hosts, too. It's all about the central repo so clients will retry in \n> various modes whenever they get an error, up to re-cloning and \n> starting over. All in all, they can yield quite a load.\n>\n> On a local filesystem everything seems to be holding up fine which is \n> expected. In the scenario with concurrency on top of shared NFS \n> storage however, the scripts eventually fails with various problems \n> (when the timing finally finds a hole, I guess) - possible to clone \n> but checkouts fails, errors about refs pointing wrong and errors where \n> the original repo is actually corrupted and can't be cloned from. \n> Depending on test strategy, I've also had everything going fine (ops \n> looks fine in my logs), but afterwards the repo has lost a branch or two.\n>\n> I've experimented with various NFS settings (e.g. turning off the \n> attribute cache), but haven't reached a conclusion. I do suspect that \n> it just is a fact of life with a remote filesystem to have coherency \n> problems with high concurrency like this but I'd be happily proven \n> wrong - I'm not an expert in either NFS or git.\n>\n> So, any opionions either way would be valuable, e.g. 'it should work' \n> or 'it can never work 100%' is fine, as well as any suggestions. \n> Obviously, the concurrency needed to make it probable to hit this \n> seems so unlikely that maybe I just shouldn't worry...\n>\n> I'd be happy to share scripts and whatever if someone would like to \n> try it out themselves.\n>\n> Thanks for your time,\n>\n> ken1\n>\n> -- \n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"213287","messageId":"87li8xrt5f.fsf@linux-k42r.v.cablecom.net","threadId":"33318","inReplyTo":"515EC51C.9070206@olwing.se","subject":"Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-04-05T13:42:04Z","receivedAt":"2013-04-05T13:42:04Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Kenneth Ölwing <kenneth@olwing.se> writes:\n\n> Basically, I'm at a place where I'm considering giving up getting this\n> to work reliably. In general, my setup work really fine, except for\n> the itty-bitty detail that when I put pressure on things I tend to get\n> into various kinds of trouble with the central repo being corrupted.\n>\n> Can anyone authoritatively state anything either way?\n\nMy non-authoritative impression was that it's supposed to work\nconcurrently.  Obviously something breaks:\n\n>> My experience so far is that I eventually get repo corruption when I\n>> stress it with concurrent read/write access from multiple hosts\n>> (beyond any sort of likely levels, but still). Maybe I'm doing\n>> something wrong, missing a configuration setting somewhere, put an\n>> unfair stress on the system, there's a bona fide bug - or, given the\n>> inherent difficulty in achieving perfect coherency between machines\n>> on what's visible on the mount, it's just impossible (?) to truly\n>> get it working under all situations.\n\nCan you run the same tests under strace or similar, and gather the\nrelevant outputs?  Otherwise it's probably very hard to say what is\ngoing wrong.\n\nIn particular we've had some reports on lustre that boiled down to\n\"impossible\" returns from libc functions, not git issues.  It's hard to\nsay without some evidence.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"213280","messageId":"515EE38D.2000502@olwing.se","threadId":"33318","inReplyTo":"87li8xrt5f.fsf@linux-k42r.v.cablecom.net","subject":"Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Kenneth Ölwing","fromEmail":"kenneth@olwing.se","sentAt":"2013-04-05T14:45:33Z","receivedAt":"2013-04-05T14:45:33Z","isPatch":false,"sender":{"key":"kenneth@olwing.se","avatar":null},"body":"On 2013-04-05 15:42, Thomas Rast wrote:\n> Can you run the same tests under strace or similar, and gather the \n> relevant outputs? Otherwise it's probably very hard to say what is \n> going wrong. In particular we've had some reports on lustre that \n> boiled down to \"impossible\" returns from libc functions, not git \n> issues. It's hard to say without some evidence. \nThomas, thanks for your reply.\n\nI'm assuming I should strace the git commands as they're issued? I'm \nalready collecting regular stdout/err output in a log as I go. Is there \nany debugging things I can turn on to make the calls issue internal \ntracing of some sort?\n\nThe main issue I see is that I suspect it will generate so much data \nthat it'll overflow my disk ;-). Consider that my hammer consists of a \nPerl script that forks a number of tasks (e.g. 15) that each loops doing \nclone/commit/push/pull, with retrying on a few levels as errors occur \n(usually expected ones due to the concurrency, i.e. someone else pushed \nso a pull is necessary first, but occasionally the central repo is \nbroken enough that it can't be cloned from, or at least not checked out \nmaster from...sometimes with printed errors that still give me a zero \nexit code...). That is then also run on several machines to the same \nrepo to hopefully cause a breakage by sheer pounding...it's going to \ngenerate huge collections of strace output I expect...\n\nI have some variations of this (e.g. all tasks are working on different \nbranches, improving concurrency in some respects, but effects there have \nbeen that at the end I was missing a branch or so...). The likelihood of \nproblems seems to increase when I actually use ssh in my ultimate setup, \nso a loadbalancer roundrobins each call to any of several hosts. In that \ncase I must admit I don't know how to get in on the action since I guess \nI would need to strace the git-upload/receive-pack processes on the \nserver side...?\n\nLastly, I don't know how much this will impact timings etc, or load. To \nget a broken result I have sometimes needed to run for many hours, \nothers fairly quickly.\n\nWell...I will try, it'll probably be a blast :-)\n\nBTW, this is mostly done on Centos 6.3 and 6.4, locally built git-1.8.2.\n\nken1\n"},{"id":"213321","messageId":"871uaorscq.fsf@linux-k42r.v.cablecom.net","threadId":"33318","inReplyTo":"515EE38D.2000502@olwing.se","subject":"Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-04-06T08:11:33Z","receivedAt":"2013-04-06T08:11:33Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Kenneth Ölwing <kenneth@olwing.se> writes:\n\n> On 2013-04-05 15:42, Thomas Rast wrote:\n>> Can you run the same tests under strace or similar, and gather the\n>> relevant outputs? Otherwise it's probably very hard to say what is\n>> going wrong. In particular we've had some reports on lustre that\n>> boiled down to \"impossible\" returns from libc functions, not git\n>> issues. It's hard to say without some evidence. \n> Thomas, thanks for your reply.\n>\n> I'm assuming I should strace the git commands as they're issued? I'm\n> already collecting regular stdout/err output in a log as I go. Is\n> there any debugging things I can turn on to make the calls issue\n> internal tracing of some sort?\n\nI don't think there's any internal debugging that helps at this point.\nUsually errors pointing to corruption are caused by a chain of syscalls\nfailing in some way, and the final error shows only the last one, so\nstrace() output is very interesting.\n\n> The main issue I see is that I suspect it will generate so much data\n> that it'll overflow my disk ;-).\n\nWell, assuming you have some automated way of detecting when it fails,\nyou can just overwrite the same strace output file repeatedly; we're\nonly interested in the last one (or all the last ones if several gits\nfail concurrently).\n\nFiddling with strace will unfortunately change the timings somewhat\n(causing a bunch of extra context switches per syscall), but I hope that\nyou can still get it to reproduce.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"213336","messageId":"CB4C1FB3EB914D079EE0534228DE372D@black","threadId":"33318","inReplyTo":"871uaorscq.fsf@linux-k42r.v.cablecom.net","subject":"RE: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2013-04-06T11:49:20Z","receivedAt":"2013-04-06T11:49:20Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> -----Original Message-----\n> From: Thomas Rast\n> Sent: Saturday, April 06, 2013 4:12\n> \n> Kenneth Ölwing <kenneth@olwing.se> writes:\n> \n> > On 2013-04-05 15:42, Thomas Rast wrote:\n> >> Can you run the same tests under strace or similar, and gather the \n> >> relevant outputs? Otherwise it's probably very hard to say what is \n> >> going wrong. In particular we've had some reports on lustre that \n> >> boiled down to \"impossible\" returns from libc functions, not git \n> >> issues. It's hard to say without some evidence.\n> > Thomas, thanks for your reply.\n> >\n> > I'm assuming I should strace the git commands as they're \n> issued? I'm \n> > already collecting regular stdout/err output in a log as I go. Is \n> > there any debugging things I can turn on to make the calls issue \n> > internal tracing of some sort?\n> \n> I don't think there's any internal debugging that helps at this point.\n> Usually errors pointing to corruption are caused by a chain \n> of syscalls failing in some way, and the final error shows \n> only the last one, so\n> strace() output is very interesting.\n> \n> > The main issue I see is that I suspect it will generate so \n> much data \n> > that it'll overflow my disk ;-).\n> \n> Well, assuming you have some automated way of detecting when \n> it fails, you can just overwrite the same strace output file \n> repeatedly; we're only interested in the last one (or all the \n> last ones if several gits fail concurrently).\n\nWe use tmpwatch for this type of issue, especially with oracle traces. Set up a\ndirectory and tell tmpwatch to delete files older than X. This will keep the\nfiles at bay and when you detect a problem stop  the clean up script.\n\n> \n> Fiddling with strace will unfortunately change the timings \n> somewhat (causing a bunch of extra context switches per \n> syscall), but I hope that you can still get it to reproduce.\n\n\n\n--\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n-                                                               -\n- Jason Pyeron                      PD Inc. http://www.pdinc.us -\n- Principal Consultant              10 West 24th Street #100    -\n- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -\n-                                                               -\n-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\nThis message is copyright PD Inc, subject to license 20080407P00.\n\n \n"},{"id":"213451","messageId":"5161C164.7020502@olwing.se","threadId":"33318","inReplyTo":"CB4C1FB3EB914D079EE0534228DE372D@black","subject":"Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)","fromName":"Kenneth Ölwing","fromEmail":"kenneth@olwing.se","sentAt":"2013-04-07T18:56:36Z","receivedAt":"2013-04-07T18:56:36Z","isPatch":false,"sender":{"key":"kenneth@olwing.se","avatar":null},"body":"Thanks for suggestions,\n\n>> I don't think there's any internal debugging that helps at this \n>> point. Usually errors pointing to corruption are caused by a chain of \n>> syscalls failing in some way, and the final error shows only the last \n>> one, so strace() output is very interesting. \n\nRight - a problem could be, in my understanding, that it will be quite \nhard to figure out which of the traces are actually interesting. First, \njust because of the intense concurrency, there will be a lot of false \nerrors on the way, and as far as I can tell many of those errors are \neffectively indistinguishable from a real error; i.e. a clone can report \nwording to the effect of 'possibly the remote repo is broken', when it's \njust in transition by another process. So, a lot of retries eventually \nwill work. Except when the repo actually is broken, but the retries are \ndone until they're exhausted. I do keep all such logs anyway, and adding \nstrace to the output should be fine - it's just a lot to go through. \nWhich is the second thing - I noticed that I can get strace to put in \ntimestamps in it's output which will likely be necessary to try to find \nwhere two or more processes interfere.\n\nOh, BTW - I'm also uncertain whether it is the actual regular ops (e.g. \npush) or perhaps auto-gc's that sometimes kick in that causes problems. \nWhile I can set gc.auto=0 to take those out of the equation, it's \nobviously not a solution in the long run. Hm, maybe I should go the \nother way, like doing gc --aggressive very often while doing pushes and \nsee if that more quickly\nprovokes an error. Even Linus in my first link suggests 'avoiding \nconcurrent gc --prune' (I know, not the same as aggressive), which is \nunderstandable, but since, again as I understand it, git will \noccasionally decide to do it on it's own, frankly I would expect this to \nwork. Not optimally from any viewpoint of course, but still, I simply \nshouldn't be able to break a repo as long as I use regular git commands. \nOr is that an unreasonable expectation? Given that I'm probably way \nbeyond any reasonable normal use, I guess it could be considered to \nchase ghosts...but then again, if there's even a tiny hole, it would be \nnice to close it.\n\nWell, I'll just have to try to battle on with it. Is there a hint of \ndocs anywhere that would describe the locking behavior git uses to \nbattle concurrency, and/or some (preferably single) points in the source \ncode that I could look at?\n\n>>> The main issue I see is that I suspect it will generate so\n>> much data\n>>> that it'll overflow my disk ;-).\n>> Well, assuming you have some automated way of detecting when\n>> it fails, you can just overwrite the same strace output file\n>> repeatedly; we're only interested in the last one (or all the\n>> last ones if several gits fail concurrently).\n> We use tmpwatch for this type of issue, especially with oracle traces. Set up a\n> directory and tell tmpwatch to delete files older than X. This will keep the\n> files at bay and when you detect a problem stop  the clean up script.\n>\nThanks - as described above I do keep track of all tasks that eventually \nrun out of steam and die from exhaustion (logs and the final copy of \ntheir clone), and can I get the strace in there things should be fine. \nIt'll still be a lot of data since I as described I haven't yet figured \nout how to accurately detect the point where the real error actually \noccurs. I'll look into if I can have some checkpoints in my tasks where \nthey all calm down so I can test the repo for correctness, so limiting \nthe data and more quickly discovering the fail state.\n\nken1\n"}]}