Is it a recommended practice to share a repository over NFS, where multiple clients can be pushing changes simultaneously? In our production environment, we have a Git repository setup behind git-http-backend. We would like to place multiple Apache servers behind a load balancer to maximize availability and performance. Before we proceed, we wanted to check to see if this practice has a potential to cause repository corruption. If there are other ways others have solved this problem, we would be very interested in learning about those as well. Thank you.
threads / discuss / 25746
Multiple clients accessing git over NFS
Subject: Multiple clients accessing git over NFS
Re: Multiple clients accessing git over NFS
Khawaja Shams <kshams@usc.edu> writes:
Show 9 quoted lines
> Is it a recommended practice to share a repository over NFS, where > multiple clients can be pushing changes simultaneously? In our > production environment, we have a Git repository setup behind > git-http-backend. We would like to place multiple Apache servers > behind a load balancer to maximize availability and performance. > Before we proceed, we wanted to check to see if this practice has a > potential to cause repository corruption. If there are other ways > others have solved this problem, we would be very interested in > learning about those as well. Thank you.
NFS locking has historically been problematic, and my impression is that most people avoid it. Perhaps it's ok on Solaris, but without serious testing, I'd be worried.
Can you explain what you have set up, and what your performance situation is, and why you think adding a second or third apache over NFS will help? How many users? How many pushes/day?
One option is to have a multi-core box with tons of RAM running apache; I've done that for trac (8 core, 16G, RAID5) because trac/python is so piggy, and buying a $3K box was cheaper than making trac go faster. That doesn't get you into remote FS locking issues.
Re: Multiple clients accessing git over NFS
Hi Greg, Thank you for the insightful response. We have multiple automated clients pushing and pulling changes from git as events occur. We have not hit any real performance issues just yet. Our main goal is to improve the availability of the repository in case the box running the apache server has an outage during a mission critical period. Any other ideas on how to accomplish this? From your remarks, it sounds like putting the git repository on NFS, even with a single client, can be problematic due to the locking issues. Is that what you meant?
I am still interested in knowing if git can handle multiple simultaneous pushes on the same repository without encountering corruption issues. Thank you.
Jonathan NiederRe: Multiple clients accessing git over NFS
Khawaja Shams wrote:
> I am still interested in knowing if git can handle multiple > simultaneous pushes on the same repository without encountering > corruption issues.
Yes, concurrent attempts to update a branch are serialized. (But please don't ask me to answer about NFS semantics. See
http://stackoverflow.com/questions/750765/concurrency-in-a-git-repo-on-a-network-shared-folder
for some notes.) See the note about fast-fowards in the git push manual for how integrity is preserved.
After reading that, you might wonder: if there are many, many clients pushing to the same branch, how is starvation avoided? Good question! It isn't. If you have so many clients wanting to push to a single branch, I would suggest having a single person or a few people maintaining it, pulling from others. Life will be better for many reasons, especially quality control.
Hope that helps. Jonathan
Re: Multiple clients accessing git over NFS
On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote:
Show 6 quoted lines
> Hi Greg, > Thank you for the insightful response. We have multiple automated > clients pushing and pulling changes from git as events occur. We have > not hit any real performance issues just yet. Our main goal is to > improve the availability of the repository in case the box running the > apache server has an outage during a mission critical period.
If you are out for availability, NFS isn't an answer, because the NFS server remains a single point of failure. There are distributed filesystems (Gluster, Lustre etc.) that can provide redundancy of storage nodes too or you could have shared storage array with appropriate filesystem (GlobalFS, OCFS2, etc.), but that requires special hardware. These will probably give you better performance too -- git network protocol is optimized to send minimal data, but that often means a lot more needs to be read from the disk.
I don't have personal experience with them though, so I can't give you more specific recommendation.
-- Jan 'Bulb' Hudec <bulb@ucw.cz>
Drew NorthupRe: Multiple clients accessing git over NFS
On Mon, 2010-11-15 at 20:56 +0100, Jan Hudec wrote:
Show 18 quoted lines
> On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote: > > Hi Greg, > > Thank you for the insightful response. We have multiple automated > > clients pushing and pulling changes from git as events occur. We have > > not hit any real performance issues just yet. Our main goal is to > > improve the availability of the repository in case the box running the > > apache server has an outage during a mission critical period. > > If you are out for availability, NFS isn't an answer, because the NFS server > remains a single point of failure. There are distributed filesystems > (Gluster, Lustre etc.) that can provide redundancy of storage nodes too or > you could have shared storage array with appropriate filesystem (GlobalFS, > OCFS2, etc.), but that requires special hardware. These will probably give > you better performance too -- git network protocol is optimized to send > minimal data, but that often means a lot more needs to be read from the disk. > > I don't have personal experience with them though, so I can't give you more > specific recommendation.
Khawaja, I haven't tried setting a server up with it yet, but perhaps DRDB mirrored devices may be of use? At that point then you have a way of making all of your HTTPd instances "see" the same filesystem (and will have notification options for when they do not). It probably isn't perfect, but may be worth looking into if your SAN cannot provide downtime-less NFS. As an added benefit, there is no longer a requirement that all of your front-ends be co-located (physically or logically).
-- -Drew Northup N1XIM AKA RvnPhnx on OPN ________________________________________________ "As opposed to vegetable or mineral error?" -John Pescatore, SANS NewsBites Vol. 12 Num. 59
Re: Multiple clients accessing git over NFS
On Sun, Nov 14, 2010 at 06:11:41PM -0500, Greg Troxel wrote:
Show 16 quoted lines
> > Khawaja Shams <kshams@usc.edu> writes: > > > Is it a recommended practice to share a repository over NFS, where > > multiple clients can be pushing changes simultaneously? In our > > production environment, we have a Git repository setup behind > > git-http-backend. We would like to place multiple Apache servers > > behind a load balancer to maximize availability and performance. > > Before we proceed, we wanted to check to see if this practice has a > > potential to cause repository corruption. If there are other ways > > others have solved this problem, we would be very interested in > > learning about those as well. Thank you. > > NFS locking has historically been problematic, and my impression is that > most people avoid it. Perhaps it's ok on Solaris, but without serious > testing, I'd be worried.
Does git actually do file locking when people push to a bare repo?
If all it needs is for rename and/or O_EXCL to be atomic--that should be fine over NFS.
--b.
Show 9 quoted lines
> > Can you explain what you have set up, and what your performance > situation is, and why you think adding a second or third apache over NFS > will help? How many users? How many pushes/day? > > One option is to have a multi-core box with tons of RAM running apache; > I've done that for trac (8 core, 16G, RAID5) because trac/python is so > piggy, and buying a $3K box was cheaper than making trac go faster. > That doesn't get you into remote FS locking issues.
Sitaram ChamartyRe: Multiple clients accessing git over NFS
On Mon, Nov 15, 2010 at 2:54 AM, Khawaja Shams <kshams@usc.edu> wrote:
> Is it a recommended practice to share a repository over NFS, where > multiple clients can be pushing changes simultaneously? In our
http://permalink.gmane.org/gmane.comp.version-control.git/122670
may be useful...
Show 12 quoted lines
> production environment, we have a Git repository setup behind > git-http-backend. We would like to place multiple Apache servers > behind a load balancer to maximize availability and performance. > Before we proceed, we wanted to check to see if this practice has a > potential to cause repository corruption. If there are other ways > others have solved this problem, we would be very interested in > learning about those as well. Thank you. > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html >
-- Sitaram
Re: Multiple clients accessing git over NFS
Khawaja Shams <kshams <at> usc.edu> writes:
Show 11 quoted lines
> > Is it a recommended practice to share a repository over NFS, where > multiple clients can be pushing changes simultaneously? In our > production environment, we have a Git repository setup behind > git-http-backend. We would like to place multiple Apache servers > behind a load balancer to maximize availability and performance. > Before we proceed, we wanted to check to see if this practice has a > potential to cause repository corruption. If there are other ways > others have solved this problem, we would be very interested in > learning about those as well. Thank you. >
Others have commented on the git aspects of this, but FYI there is a handy program here: http://www.unixcoding.org/NFSCoding#NFS_Cache_Tester that tests aspects of your NFS implementation. (Sadly the one we have at work is crap, or at least it was last time I ran the program).
Alex
Re: Multiple clients accessing git over NFS
If only one computer is accessing the repository, then failure to lock may be ok. But you'll still need atomic rename etc. to work.
I may be overly conservative, but I would not (and do not) allow anyone to access a repository (cvs, svn, git, whatever) over NFS, ever.
My expectation is that multiple git processes on one machine with a repo on local disk works fine. If not it's a bug. When you add a remote FS you have to wonder if the unix filesystem sematics are preserved.
Another approach would be cloned repositories that constantly pull from the main one or each other, and have people use those. That will get you delayed merge conflicts, but they might be useful as RO replicas.