{"thread":{"id":"20002","subject":"what's the current wisdom on git over NFS/CIFS?","startedAt":"2009-07-02T13:11:04Z","lastAt":"2009-07-03T16:08:08Z","messageCount":10,"participants":["Sitaram Chamarty","Martin Langhoff","Linus Torvalds","Johannes Sixt","Dmitry Potapov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"117363","messageId":"slrnh4pcf8.484.sitaramc@sitaramc.homelinux.net","threadId":"20002","inReplyTo":null,"subject":"what's the current wisdom on git over NFS/CIFS?","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-07-02T13:11:04Z","receivedAt":"2009-07-02T13:11:04Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"Hi all,\n\nA couple of us were beating each other senseless on this\nissue over on #git, so I thought I'd ask: is it OK to do\nthis? Or would there potential be race conditions due to\nthe lack of proper locking?\n\nThe git faq says SMBFS is not recommended. Searching for\n\"git on NFS\" gets me too many hits about the NFS _code_ :-)\n\nThanks,\n\nSitaram\n"},{"id":"117365","messageId":"46a038f90907020658g67cc0fcdi99c8455842b7ad43@mail.gmail.com","threadId":"20002","inReplyTo":"slrnh4pcf8.484.sitaramc@sitaramc.homelinux.net","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-07-02T13:58:34Z","receivedAt":"2009-07-02T13:58:34Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Thu, Jul 2, 2009 at 3:11 PM, Sitaram Chamarty wrote:\n> The git faq says SMBFS is not recommended.  Searching for\n> \"git on NFS\" gets me too many hits about the NFS _code_ :-)\n\nYou can use marc.theaimsgroup.info to search the git list archives.\n\nhth\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"117366","messageId":"46a038f90907020700w607f733fw76dfb68b88effa8@mail.gmail.com","threadId":"20002","inReplyTo":"46a038f90907020658g67cc0fcdi99c8455842b7ad43@mail.gmail.com","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-07-02T14:00:03Z","receivedAt":"2009-07-02T14:00:03Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Thu, Jul 2, 2009 at 3:58 PM, Martin\nLanghoff wrote:\n> You can use marc.theaimsgroup.info to search the git list archives.\n\nActually, marc.theaimsgroup.com - which redirects to marc.info . See\n\n http://marc.info/?l=git&w=2&r=1&s=nfs&q=b\n\nhth,\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"117389","messageId":"alpine.LFD.2.01.0907021345440.3210@localhost.localdomain","threadId":"20002","inReplyTo":"slrnh4pcf8.484.sitaramc@sitaramc.homelinux.net","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-02T20:58:25Z","receivedAt":"2009-07-02T20:58:25Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 2 Jul 2009, Sitaram Chamarty wrote:\n> \n> A couple of us were beating each other senseless on this\n> issue over on #git, so I thought I'd ask: is it OK to do\n> this? Or would there potential be race conditions due to\n> the lack of proper locking?\n\nIt should all work fine.\n\nWe've had a few NFS issues due to oddities with renaming across \ndirectories, but modern git avoids the cross-directory renames, and that \nwhole issue only hit some very specific cases anyway.\n\nAnd git doesn't have \"proper locking\", because it doesn't need it for \ndatabase ops: git objects are stable. For refs, git should be using the \nproper NFS-safe \"create and atomic rename\" ops.\n\nYou do need to be a bit careful if you do maintenance operations \nconcurrently (I would suggest avoiding doing concurrent \"git gc --prune\", \nfor example), but any normal git workflow should be fine.\n\nThere is one big thing to look out for: performance. You may want to add\n\n\t[core]\n\t\tPreloadIndex = true\n\nin your .git/config file, especially if you have a high-latency NFS server \n(but if you have a fast network and a high-end NFS server, you might be \nbetter off without it, so do your own testing).\n\nBtw, I think we fixed the problem we had with CIFS. That one was a cifs \nfilesystem problem on Linux, but it should be fixed in 2.6.30+ (commit \n0f4d634c: \"cifs: flush data on any setattr\"). If you have an older kernel \n(or are just uncertain), you can also work around it with\n\n\t[core]\n\t\tfsyncobjectfiles = true\n\nwhich may be a good thing in general (regardless of any cifs issues), but \nin most cases the performance loss isn't worth it if your filesystem is \nstable and sane.\n\n\t\t\tLinus\n"},{"id":"117396","messageId":"2e24e5b90907021752t10243468sc07be88cd88ac5c1@mail.gmail.com","threadId":"20002","inReplyTo":"alpine.LFD.2.01.0907021345440.3210@localhost.localdomain","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-07-03T00:52:01Z","receivedAt":"2009-07-03T00:52:01Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Fri, Jul 3, 2009 at 2:28 AM, Linus\nTorvalds wrote:\n\n> On Thu, 2 Jul 2009, Sitaram Chamarty wrote:\n\n>> A couple of us were beating each other senseless on this\n>> issue over on #git, so I thought I'd ask: is it OK to do\n>> this?  Or would there potential be race conditions due to\n>> the lack of proper locking?\n>\n> It should all work fine.\n>\n> We've had a few NFS issues due to oddities with renaming across\n> directories, but modern git avoids the cross-directory renames, and that\n> whole issue only hit some very specific cases anyway.\n>\n> And git doesn't have \"proper locking\", because it doesn't need it for\n> database ops: git objects are stable. For refs, git should be using the\n> proper NFS-safe \"create and atomic rename\" ops.\n>\n> You do need to be a bit careful if you do maintenance operations\n> concurrently (I would suggest avoiding doing concurrent \"git gc --prune\",\n> for example), but any normal git workflow should be fine.\n>\n> There is one big thing to look out for: performance. You may want to add\n>\n>        [core]\n>                PreloadIndex = true\n\nGreat! Thanks for the really detailed reply; appreciate it very much.\n\nI knew about core.PreloadIndex simply due to searching \"man\ngit-config\" for NFS etc., but not the rest of it.\n\n> in your .git/config file, especially if you have a high-latency NFS server\n> (but if you have a fast network and a high-end NFS server, you might be\n> better off without it, so do your own testing).\n>\n> Btw, I think we fixed the problem we had with CIFS. That one was a cifs\n> filesystem problem on Linux, but it should be fixed in 2.6.30+ (commit\n> 0f4d634c: \"cifs: flush data on any setattr\"). If you have an older kernel\n> (or are just uncertain), you can also work around it with\n>\n> [core]\n> fsyncobjectfiles = true\n>\n> which may be a good thing in general (regardless of any cifs issues), but\n> in most cases the performance loss isn't worth it if your filesystem is\n> stable and sane.\n\nThough I asked this following from a debate on IRC, it now looks as if\nthis will solve another of my problems too.\n\nLet me explain.\n\nI'm evangelising git at work, and although most projects are happy,\neven eager to setup a proper server for git, and those that can't are\nhappy to just use mine, there are a couple of projects that are almost\nexclusively Windows _and_ cannot add another machine _and_ have client\nconfidentiality issues so they can't just use my server.\n\nMy alternatives for them so far were (1) VirtualBox running Fedora or\nsomething within one of their beefier Windows servers or (2) the whole\ncygwin install, which is painful compared to msysgit.\n\nSounds like we can just do it with traditional Windows fileshares, as\nlong as we make sure no one does a \"git gc\" on the bare repo that is\nbeing shared. That's a very small price to pay!\n\nThanks once again, and I'm copying Gitte in case he has anything to\nadd, seeing I have sneakily changed the focus of the question now :-)\n[It wasn't pre-meditated, honest!]\n\nSitaram\n"},{"id":"117401","messageId":"4A4DA1D6.40301@viscovery.net","threadId":"20002","inReplyTo":"2e24e5b90907021752t10243468sc07be88cd88ac5c1@mail.gmail.com","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-07-03T06:14:46Z","receivedAt":"2009-07-03T06:14:46Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Sitaram Chamarty schrieb:\n> On Fri, Jul 3, 2009 at 2:28 AM, Linus\n> Torvalds wrote:\n>> Btw, I think we fixed the problem we had with CIFS. That one was a cifs\n>> filesystem problem on Linux, but it should be fixed in 2.6.30+ (commit\n>> 0f4d634c: \"cifs: flush data on any setattr\"). If you have an older kernel\n>> (or are just uncertain), you can also work around it with\n>>\n>> [core]\n>> fsyncobjectfiles = true\n>>\n>> which may be a good thing in general (regardless of any cifs issues), but\n>> in most cases the performance loss isn't worth it if your filesystem is\n>> stable and sane.\n> \n> Though I asked this following from a debate on IRC, it now looks as if\n> this will solve another of my problems too.\n> \n> Let me explain.\n> \n> I'm evangelising git at work, and although most projects are happy,\n> even eager to setup a proper server for git, and those that can't are\n> happy to just use mine, there are a couple of projects that are almost\n> exclusively Windows _and_ cannot add another machine _and_ have client\n> confidentiality issues so they can't just use my server.\n\nI don't exactly understand why your reply with this Windows story to the\nfsyncobjectfiles paragraph. But if you think that fsyncobjectfiles+msysgit\nis your killer argument, then you think wrong: fsyncobjectfiles is ignored\nin msysgit. (We don't yet have an implementation of fsync() :-( )\n\nThe issue that Linus mentioned happened when a Linux client operates in a\nrepository that lives on a CIFS mount.\n\n> My alternatives for them so far were (1) VirtualBox running Fedora or\n> something within one of their beefier Windows servers or (2) the whole\n> cygwin install, which is painful compared to msysgit.\n> \n> Sounds like we can just do it with traditional Windows fileshares, as\n> long as we make sure no one does a \"git gc\" on the bare repo that is\n> being shared. That's a very small price to pay!\n\nIt will work with the caveat you mention. I have such a setup myself.\n\n-- Hannes\n"},{"id":"117403","messageId":"2e24e5b90907022337k55b9c984rf3b5a985283e18e4@mail.gmail.com","threadId":"20002","inReplyTo":"4A4DA1D6.40301@viscovery.net","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-07-03T06:37:47Z","receivedAt":"2009-07-03T06:37:47Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Fri, Jul 3, 2009 at 11:44 AM, Johannes Sixt wrote:\n> Sitaram Chamarty schrieb:\n>> On Fri, Jul 3, 2009 at 2:28 AM, Linus\n>> Torvalds wrote:\n\n[snipped all of Linus' email]\n\n>> Though I asked this following from a debate on IRC, it now looks as if\n>> this will solve another of my problems too.\n\n[snipped my \"Windows story\" ;-)]\n\n> I don't exactly understand why your reply with this Windows story to the\n> fsyncobjectfiles paragraph. But if you think that fsyncobjectfiles+msysgit\n\nNot at all. Since I was adding Gitte to the CC, I kept _all_ of\nLinus' email quoted to provide him the full context. My Windows story\npertains in general to the whole email, really.\n\n>> Sounds like we can just do it with traditional Windows fileshares, as\n>> long as we make sure no one does a \"git gc\" on the bare repo that is\n>> being shared.  That's a very small price to pay!\n>\n> It will work with the caveat you mention. I have such a setup myself.\n\nAnd _that_ is what I was hoping to hear! Thank you! I can use it and\ntest it all I like, but hearing the guys who know say \"it will work\"\nmakes a big difference.\n"},{"id":"117407","messageId":"20090703085608.GA18757@dpotapov.dyndns.org","threadId":"20002","inReplyTo":"4A4DA1D6.40301@viscovery.net","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-07-03T08:56:08Z","receivedAt":"2009-07-03T08:56:08Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Fri, Jul 03, 2009 at 08:14:46AM +0200, Johannes Sixt wrote:\n> (We don't yet have an implementation of fsync() :-( )\n\nWhy? It appears rather straightforward to me. Here is a patch that\nimplements it. Or did I miss something?\n\nWarning: I do not use MinGW/Git, so I have not tested this patch.\n\n-- >8 --\nFrom 0a5e712ff8775e0f27923bedcb3c234288592eaa Mon Sep 17 00:00:00 2001\nFrom: Dmitry Potapov \nDate: Fri, 3 Jul 2009 12:45:30 +0400\nSubject: [PATCH] mingw: fsync implementation for Windows\n\nSigned-off-by: Dmitry Potapov \n---\n compat/mingw.c | 16 ++++++++++++++++\n compat/mingw.h | 3 +--\n 2 files changed, 17 insertions(+), 2 deletions(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex bed4178..65c9e8e 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1231,3 +1231,19 @@ struct dirent *mingw_readdir(DIR *dir)\n \treturn (struct dirent*)&dir->dd_dir;\n }\n #endif // !NO_MINGW_REPLACE_READDIR\n+\n+int fsync(int fd)\n+{\n+\tHANDLE h = (HANDLE) _get_osfhandle(fd);\n+\tif (h == INVALID_HANDLE_VALUE)\n+\t{\n+\t\terrno = EBADF;\n+\t\treturn -1;\n+\t}\n+\tif (!FlushFileBuffers(h))\n+\t{\n+\t\terrno = err_win_to_posix(GetLastError());\n+\t\treturn -1;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 4f7ba4c..14d53c0 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -75,8 +75,7 @@ static inline int fork(void)\n { errno = ENOSYS; return -1; }\n static inline unsigned int alarm(unsigned int seconds)\n { return 0; }\n-static inline int fsync(int fd)\n-{ return 0; }\n+int fsync(int fd);\n static inline int getppid(void)\n { return 1; }\n static inline void sync(void)\n-- \n1.6.2.3\n\n-- >8 --\n"},{"id":"117408","messageId":"4A4DCCDD.9030901@viscovery.net","threadId":"20002","inReplyTo":"20090703085608.GA18757@dpotapov.dyndns.org","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-07-03T09:18:21Z","receivedAt":"2009-07-03T09:18:21Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Dmitry Potapov schrieb:\n> On Fri, Jul 03, 2009 at 08:14:46AM +0200, Johannes Sixt wrote:\n>> (We don't yet have an implementation of fsync() :-( )\n> \n> Why?\n\nNobody cared. ;)\n\n> It appears rather straightforward to me. Here is a patch that\n> implements it. Or did I miss something?\n\nYes, but it's not critical; see below ;)\n\n> +int fsync(int fd)\n> +{\n> +\tHANDLE h = (HANDLE) _get_osfhandle(fd);\n> +\tif (h == INVALID_HANDLE_VALUE)\n> +\t{\n> +\t\terrno = EBADF;\n> +\t\treturn -1;\n> +\t}\n> +\tif (!FlushFileBuffers(h))\n> +\t{\n> +\t\terrno = err_win_to_posix(GetLastError());\n> +\t\treturn -1;\n> +\t}\n\nDo we ever call fsync on a pipe? In this case, fsync should fail with\nEINVAL, but your implementation would wait until the reader has drained\nall data.\n\n> -static inline int fsync(int fd)\n> -{ return 0; }\n> +int fsync(int fd);\n\nThis declaration is now in the wrong section of mingw.h. And someone who\ncares should test it first; so I won't bless this patch as is.\n\nThanks anyway,\n-- Hannes\n"},{"id":"117421","messageId":"alpine.LFD.2.01.0907030858540.3210@localhost.localdomain","threadId":"20002","inReplyTo":"4A4DCCDD.9030901@viscovery.net","subject":"Re: what's the current wisdom on git over NFS/CIFS?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-07-03T16:08:08Z","receivedAt":"2009-07-03T16:08:08Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 3 Jul 2009, Johannes Sixt wrote:\n> \n> Do we ever call fsync on a pipe? In this case, fsync should fail with\n> EINVAL, but your implementation would wait until the reader has drained\n> all data.\n\nI'm pretty sure we never call fsync() on anything but a file. So I suspect \nthe patch for windows is fine - it only needs to care about regular files.\n\nThat said, there's a reason we don't enable fsync by default - it's \ngenerally not needed. I'm not sure what the NTFS crash semantics are, but \nI _think_ NTFS does everything with a proper log, and then fsync probably \ndoesn't matter on windows.\n\nThe \"fsync on CIFS\" was not about \"windows is crap, and because it's a \nwindows filesystem we need to fsync\" - it was literally because there was \na bug in the Linux CIFS implementation that meant that pending writes \ncould be lost if the file was made read-only while they were still \npending.\n\nAnd on non-journaling filesystems, fsync is obviously a good idea, but \neven then the state is often recoverable even in the face of a crash. Now, \nwithout fsync, you might have to work at it a bit and know what to do - \nlike _manually_ resetting to the previous state and re-doing the last \ncorrupt commit, throwing out any corrupt objects etc.\n\nSo enabling fsync can make things _much_ easier for recovery (to the point \nof making it possible at all for somebody who doesn't know the git object \nmodel), but is generally not a hard requirement.\n\nGiving windows git the capability to do it sounds like a good idea, \nthough.\n\nKeep in mind: enabling fsync _will_ make object creation slower. Most \npeople probably won't care much, though. It really only matters under some \nfairly rare circumstances - I care, beause it matters for things like \n\"apply 200 patches in one go as fast as you can\", but for most 'normal' \nworkflows you'd probably never even notice.\n\n\t\t\tLinus\n"}]}