{"thread":{"id":"25746","subject":"Multiple clients accessing git over NFS","startedAt":"2010-11-14T21:24:01Z","lastAt":"2010-11-16T13:47:02Z","messageCount":10,"participants":["Khawaja Shams","Greg Troxel","Jonathan Nieder","Sitaram Chamarty","J. Bruce Fields","Jan Hudec","Drew Northup","Alex"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155878","messageId":"AANLkTimyFkVFAw4s2fiWKZFPvnx15K6U6GbxmRgznx7Z@mail.gmail.com","threadId":"25746","inReplyTo":null,"subject":"Multiple clients accessing git over NFS","fromName":"Khawaja Shams","fromEmail":"kshams@usc.edu","sentAt":"2010-11-14T21:24:01Z","receivedAt":"2010-11-14T21:24:01Z","isPatch":false,"sender":{"key":"kshams@usc.edu","avatar":null},"body":"  Is it a recommended practice to share a repository over NFS, where\nmultiple clients can be pushing changes simultaneously?  In our\nproduction environment, we have a Git repository setup behind\ngit-http-backend. We would like to place multiple Apache servers\nbehind a load balancer to maximize availability and performance.\nBefore we proceed, we wanted to check to see if this practice has a\npotential to cause repository corruption. If there are other ways\nothers have solved this problem, we would be very interested in\nlearning about those as well. Thank you.\n"},{"id":"155894","messageId":"rmi39r3mrf6.fsf@fnord.ir.bbn.com","threadId":"25746","inReplyTo":"AANLkTimyFkVFAw4s2fiWKZFPvnx15K6U6GbxmRgznx7Z@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Greg Troxel","fromEmail":"gdt@ir.bbn.com","sentAt":"2010-11-14T23:11:41Z","receivedAt":"2010-11-14T23:11:41Z","isPatch":false,"sender":{"key":"gdt@ir.bbn.com","avatar":null},"body":"\nKhawaja Shams <kshams@usc.edu> writes:\n\n> Is it a recommended practice to share a repository over NFS, where\n> multiple clients can be pushing changes simultaneously?  In our\n> production environment, we have a Git repository setup behind\n> git-http-backend. We would like to place multiple Apache servers\n> behind a load balancer to maximize availability and performance.\n> Before we proceed, we wanted to check to see if this practice has a\n> potential to cause repository corruption. If there are other ways\n> others have solved this problem, we would be very interested in\n> learning about those as well. Thank you.\n\nNFS locking has historically been problematic, and my impression is that\nmost people avoid it.  Perhaps it's ok on Solaris, but without serious\ntesting, I'd be worried.\n\nCan you explain what you have set up, and what your performance\nsituation is, and why you think adding a second or third apache over NFS\nwill help?  How many users?  How many pushes/day?\n\nOne option is to have a multi-core box with tons of RAM running apache;\nI've done that for trac (8 core, 16G, RAID5) because trac/python is so\npiggy, and buying a $3K box was cheaper than making trac go faster.\nThat doesn't get you into remote FS locking issues.\n"},{"id":"155895","messageId":"AANLkTim1bUbofDzC5HJnB--0WkT45ewbWCa25RebEgae@mail.gmail.com","threadId":"25746","inReplyTo":"rmi39r3mrf6.fsf@fnord.ir.bbn.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Khawaja Shams","fromEmail":"kshams@usc.edu","sentAt":"2010-11-14T23:42:29Z","receivedAt":"2010-11-14T23:42:29Z","isPatch":false,"sender":{"key":"kshams@usc.edu","avatar":null},"body":"Hi Greg,\n   Thank you for the insightful response. We have multiple automated\nclients pushing and pulling changes from git as events occur. We have\nnot hit any real performance issues just yet. Our main goal is to\nimprove the availability of the repository in case the box running the\napache server has an outage during a mission critical period. Any\nother ideas on how to accomplish this? From your remarks, it sounds\nlike putting the git repository on NFS, even with a single client, can\nbe problematic due to the locking issues. Is that what you meant?\n\n   I am still interested in knowing if git can handle multiple\nsimultaneous pushes on the same repository without encountering\ncorruption issues. Thank you.\n"},{"id":"155896","messageId":"rmipqu7lb8f.fsf@fnord.ir.bbn.com","threadId":"25746","inReplyTo":"AANLkTinX-XR2TaZPGPeWyekMq3e8wEDkfcmi_o6pTvMK@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Greg Troxel","fromEmail":"gdt@ir.bbn.com","sentAt":"2010-11-14T23:46:40Z","receivedAt":"2010-11-14T23:46:40Z","isPatch":false,"sender":{"key":"gdt@ir.bbn.com","avatar":null},"body":"\nIf only one computer is accessing the repository, then failure to lock\nmay be ok.  But you'll still need atomic rename etc. to work.\n\nI may be overly conservative, but I would not (and do not) allow anyone\nto access a repository (cvs, svn, git, whatever) over NFS, ever.\n\nMy expectation is that multiple git processes on one machine with a repo\non local disk works fine.  If not it's a bug.  When you add a remote FS\nyou have to wonder if the unix filesystem sematics are preserved.\n\nAnother approach would be cloned repositories that constantly pull from\nthe main one or each other, and have people use those.  That will get\nyou delayed merge conflicts, but they might be useful as RO replicas.\n\n\n"},{"id":"155903","messageId":"20101115003259.GC26104@burratino","threadId":"25746","inReplyTo":"AANLkTim1bUbofDzC5HJnB--0WkT45ewbWCa25RebEgae@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-15T00:32:59Z","receivedAt":"2010-11-15T00:32:59Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Khawaja Shams wrote:\n\n>    I am still interested in knowing if git can handle multiple\n> simultaneous pushes on the same repository without encountering\n> corruption issues.\n\nYes, concurrent attempts to update a branch are serialized.  (But\nplease don't ask me to answer about NFS semantics.  See\n\nhttp://stackoverflow.com/questions/750765/concurrency-in-a-git-repo-on-a-network-shared-folder\n\nfor some notes.)  See the note about fast-fowards in the git push\nmanual for how integrity is preserved.\n\nAfter reading that, you might wonder: if there are many, many clients\npushing to the same branch, how is starvation avoided?  Good question!\nIt isn't.  If you have so many clients wanting to push to a single\nbranch, I would suggest having a single person or a few people\nmaintaining it, pulling from others.  Life will be better for many\nreasons, especially quality control.\n\nHope that helps.\nJonathan\n"},{"id":"155904","messageId":"AANLkTi=4D+yoeN9=aKNw4y_Um_F9WhUCPDaU5FZkY4QF@mail.gmail.com","threadId":"25746","inReplyTo":"AANLkTimyFkVFAw4s2fiWKZFPvnx15K6U6GbxmRgznx7Z@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2010-11-15T01:26:32Z","receivedAt":"2010-11-15T01:26:32Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Mon, Nov 15, 2010 at 2:54 AM, Khawaja Shams <kshams@usc.edu> wrote:\n>   Is it a recommended practice to share a repository over NFS, where\n> multiple clients can be pushing changes simultaneously?  In our\n\nhttp://permalink.gmane.org/gmane.comp.version-control.git/122670\n\nmay be useful...\n\n> production environment, we have a Git repository setup behind\n> git-http-backend. We would like to place multiple Apache servers\n> behind a load balancer to maximize availability and performance.\n> Before we proceed, we wanted to check to see if this practice has a\n> potential to cause repository corruption. If there are other ways\n> others have solved this problem, we would be very interested in\n> learning about those as well. Thank you.\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\n\n\n-- \nSitaram\n"},{"id":"155930","messageId":"20101115162432.GA6258@fieldses.org","threadId":"25746","inReplyTo":"rmi39r3mrf6.fsf@fnord.ir.bbn.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"J. Bruce Fields","fromEmail":"bfields@fieldses.org","sentAt":"2010-11-15T16:24:32Z","receivedAt":"2010-11-15T16:24:32Z","isPatch":false,"sender":{"key":"bfields@citi.umich.edu","avatar":null},"body":"On Sun, Nov 14, 2010 at 06:11:41PM -0500, Greg Troxel wrote:\n> \n> Khawaja Shams <kshams@usc.edu> writes:\n> \n> > Is it a recommended practice to share a repository over NFS, where\n> > multiple clients can be pushing changes simultaneously?  In our\n> > production environment, we have a Git repository setup behind\n> > git-http-backend. We would like to place multiple Apache servers\n> > behind a load balancer to maximize availability and performance.\n> > Before we proceed, we wanted to check to see if this practice has a\n> > potential to cause repository corruption. If there are other ways\n> > others have solved this problem, we would be very interested in\n> > learning about those as well. Thank you.\n> \n> NFS locking has historically been problematic, and my impression is that\n> most people avoid it.  Perhaps it's ok on Solaris, but without serious\n> testing, I'd be worried.\n\nDoes git actually do file locking when people push to a bare repo?\n\nIf all it needs is for rename and/or O_EXCL to be atomic--that should be\nfine over NFS.\n\n--b.\n\n> \n> Can you explain what you have set up, and what your performance\n> situation is, and why you think adding a second or third apache over NFS\n> will help?  How many users?  How many pushes/day?\n> \n> One option is to have a multi-core box with tons of RAM running apache;\n> I've done that for trac (8 core, 16G, RAID5) because trac/python is so\n> piggy, and buying a $3K box was cheaper than making trac go faster.\n> That doesn't get you into remote FS locking issues.\n"},{"id":"155943","messageId":"20101115195641.GG30870@efreet.light.src","threadId":"25746","inReplyTo":"AANLkTim1bUbofDzC5HJnB--0WkT45ewbWCa25RebEgae@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2010-11-15T19:56:41Z","receivedAt":"2010-11-15T19:56:41Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote:\n> Hi Greg,\n>    Thank you for the insightful response. We have multiple automated\n> clients pushing and pulling changes from git as events occur. We have\n> not hit any real performance issues just yet. Our main goal is to\n> improve the availability of the repository in case the box running the\n> apache server has an outage during a mission critical period.\n\nIf you are out for availability, NFS isn't an answer, because the NFS server\nremains a single point of failure. There are distributed filesystems\n(Gluster, Lustre etc.) that can provide redundancy of storage nodes too or\nyou could have shared storage array with appropriate filesystem (GlobalFS,\nOCFS2, etc.), but that requires special hardware. These will probably give\nyou better performance too -- git network protocol is optimized to send\nminimal data, but that often means a lot more needs to be read from the disk.\n\nI don't have personal experience with them though, so I can't give you more\nspecific recommendation.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec <bulb@ucw.cz>\n"},{"id":"155948","messageId":"1289853858.9996.14.camel@drew-northup.unet.maine.edu","threadId":"25746","inReplyTo":"20101115195641.GG30870@efreet.light.src","subject":"Re: Multiple clients accessing git over NFS","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-11-15T20:44:18Z","receivedAt":"2010-11-15T20:44:18Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Mon, 2010-11-15 at 20:56 +0100, Jan Hudec wrote:\n> On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote:\n> > Hi Greg,\n> >    Thank you for the insightful response. We have multiple automated\n> > clients pushing and pulling changes from git as events occur. We have\n> > not hit any real performance issues just yet. Our main goal is to\n> > improve the availability of the repository in case the box running the\n> > apache server has an outage during a mission critical period.\n> \n> If you are out for availability, NFS isn't an answer, because the NFS server\n> remains a single point of failure. There are distributed filesystems\n> (Gluster, Lustre etc.) that can provide redundancy of storage nodes too or\n> you could have shared storage array with appropriate filesystem (GlobalFS,\n> OCFS2, etc.), but that requires special hardware. These will probably give\n> you better performance too -- git network protocol is optimized to send\n> minimal data, but that often means a lot more needs to be read from the disk.\n> \n> I don't have personal experience with them though, so I can't give you more\n> specific recommendation.\n\nKhawaja,\nI haven't tried setting a server up with it yet, but perhaps DRDB\nmirrored devices may be of use? At that point then you have a way of\nmaking all of your HTTPd instances \"see\" the same filesystem (and will\nhave notification options for when they do not). It probably isn't\nperfect, but may be worth looking into if your SAN cannot provide\ndowntime-less NFS. As an added benefit, there is no longer a requirement\nthat all of your front-ends be co-located (physically or logically).\n\n-- \n-Drew Northup N1XIM\n   AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"155997","messageId":"loom.20101116T144422-559@post.gmane.org","threadId":"25746","inReplyTo":"AANLkTimyFkVFAw4s2fiWKZFPvnx15K6U6GbxmRgznx7Z@mail.gmail.com","subject":"Re: Multiple clients accessing git over NFS","fromName":"Alex","fromEmail":"ajb44.geo@yahoo.com","sentAt":"2010-11-16T13:47:02Z","receivedAt":"2010-11-16T13:47:02Z","isPatch":false,"sender":{"key":"ajb44.geo@yahoo.com","avatar":null},"body":"\nKhawaja Shams <kshams <at> usc.edu> writes:\n\n> \n>   Is it a recommended practice to share a repository over NFS, where\n> multiple clients can be pushing changes simultaneously?  In our\n> production environment, we have a Git repository setup behind\n> git-http-backend. We would like to place multiple Apache servers\n> behind a load balancer to maximize availability and performance.\n> Before we proceed, we wanted to check to see if this practice has a\n> potential to cause repository corruption. If there are other ways\n> others have solved this problem, we would be very interested in\n> learning about those as well. Thank you.\n> \n\n\nOthers have commented on the git aspects of this, but FYI there is a handy\nprogram here: http://www.unixcoding.org/NFSCoding#NFS_Cache_Tester\nthat tests aspects of your NFS implementation. (Sadly the one we have\nat work is crap, or at least it was last time I ran the program).\n\nAlex\n \n"}]}