{"thread":{"id":"13892","subject":"Consolidate SHA1 object file close","startedAt":"2008-06-11T01:47:18Z","lastAt":"2008-06-12T16:00:12Z","messageCount":11,"participants":["Linus Torvalds","Andreas Ericsson","Pierre Habouzit"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"79415","messageId":"alpine.LFD.1.10.0806101842460.3101@woody.linux-foundation.org","threadId":"13892","inReplyTo":null,"subject":"Consolidate SHA1 object file close","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T01:47:18Z","receivedAt":"2008-06-11T01:47:18Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nThis consolidates the common operations for closing the new temporary file \nthat we have written, before we move it into place with the final name. \n\nThere's some common code there (make it read-only and check for errors on \nclose), but more importantly, this also gives a single place to add an \nfsync_or_die() call if we want to add a safe mode.\n\nThis was triggered due to Denis Bueno apparently twice being able to \ncorrupt his git repository on OS X due to an unlucky combination of kernel \ncrashes and a not-very-robust filesystem.\n\nSigned-off-by: Linus Torvalds <torvalds@linux-foundation.org>\n---\n\nJunio, this is just the scaffolding, without the actual fsync_or_die call. \n\nI think it's a worthy place-holder regardless of whether we really want to \ndo the fsync (whether conditionally with a config option or not, and \nwhether there are more clever options like aio_fsync()).\n\n sha1_file.c |   17 +++++++++++------\n 1 files changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex adcf37c..f311c79 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2105,6 +2105,15 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n \treturn 0;\n }\n \n+/* Finalize a file on disk, and close it. */\n+static void close_sha1_file(int fd)\n+{\n+\t/* For safe-mode, we could fsync_or_die(fd, \"sha1 file\") here */\n+\tfchmod(fd, 0444);\n+\tif (close(fd) != 0)\n+\t\tdie(\"unable to write sha1 file\");\n+}\n+\n static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \t\t\t      void *buf, unsigned long len, time_t mtime)\n {\n@@ -2170,9 +2179,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \n \tif (write_buffer(fd, compressed, size) < 0)\n \t\tdie(\"unable to write sha1 file\");\n-\tfchmod(fd, 0444);\n-\tif (close(fd))\n-\t\tdie(\"unable to write sha1 file\");\n+\tclose_sha1_file(fd);\n \tfree(compressed);\n \n \tif (mtime) {\n@@ -2350,9 +2357,7 @@ int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,\n \t} while (1);\n \tinflateEnd(&stream);\n \n-\tfchmod(local, 0444);\n-\tif (close(local) != 0)\n-\t\tdie(\"unable to write sha1 file\");\n+\tclose_sha1_file(local);\n \tSHA1_Final(real_sha1, &c);\n \tif (ret != Z_STREAM_END) {\n \t\tunlink(tmpfile);\n"},{"id":"79437","messageId":"484F81FE.7070801@op5.se","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806101842460.3101@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2008-06-11T07:42:54Z","receivedAt":"2008-06-11T07:42:54Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Linus Torvalds wrote:\n> This consolidates the common operations for closing the new temporary file \n> that we have written, before we move it into place with the final name. \n> \n> There's some common code there (make it read-only and check for errors on \n> close), but more importantly, this also gives a single place to add an \n> fsync_or_die() call if we want to add a safe mode.\n> \n> This was triggered due to Denis Bueno apparently twice being able to \n> corrupt his git repository on OS X due to an unlucky combination of kernel \n> crashes and a not-very-robust filesystem.\n> \n> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>\n> ---\n> \n> Junio, this is just the scaffolding, without the actual fsync_or_die call. \n> \n> I think it's a worthy place-holder regardless of whether we really want to \n> do the fsync (whether conditionally with a config option or not, and \n> whether there are more clever options like aio_fsync()).\n> \n>  sha1_file.c |   17 +++++++++++------\n>  1 files changed, 11 insertions(+), 6 deletions(-)\n> \n> diff --git a/sha1_file.c b/sha1_file.c\n> index adcf37c..f311c79 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -2105,6 +2105,15 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n>  \treturn 0;\n>  }\n>  \n> +/* Finalize a file on disk, and close it. */\n> +static void close_sha1_file(int fd)\n\n\nWhy close_sha1_file() when it operates on any old file?\nI'd name it crash_safe_close() or perhaps close_and_fsync() or\nsomething instead, as it's got nothing to do with sha1's and\neverything to do with plain old files.\n\nOther than that, I'm all for it.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"79438","messageId":"20080611074309.GB28629@artemis.madism.org","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806101842460.3101@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-06-11T07:43:09Z","receivedAt":"2008-06-11T07:43:09Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jun 11, 2008 at 01:47:18AM +0000, Linus Torvalds wrote:\n> \n> This consolidates the common operations for closing the new temporary file \n> that we have written, before we move it into place with the final name. \n> \n> There's some common code there (make it read-only and check for errors on \n> close), but more importantly, this also gives a single place to add an \n> fsync_or_die() call if we want to add a safe mode.\n> \n> This was triggered due to Denis Bueno apparently twice being able to \n> corrupt his git repository on OS X due to an unlucky combination of kernel \n> crashes and a not-very-robust filesystem.\n\n  Could this be the source of a problem we often meet at work ? Let me\ntry to describe it.\n\n  We work with our git repositories (storages I should say) on NFS\nhomes, with workdirs on a local directory (NFS homes are backuped daily,\nhence everything commited get backuped, and developers have shorter\ncompilation times thanks to the local FS). I don't think the workdir use\nis relevant because I use it almost the same without NFS and haven't any\nissues, but I mention it just in case.\n\n  Quite often, when people commit, they have corrupt repositories. The\nsymptom is a `cannot read <sha1>` error message (or many at times). The\nusual way to \"fix\" it is to git fsck, and git reset (because after the\nfsck the index is totally screwed and all local files are marked new),\nand usually everything is fine then.\n\n  This is not really a hard corruption, and it's really hard to\nreproduce, I don't know why it happens, and I wonder if this patch could\nhelp, or if it's unrelated. I can only bring speculations as it's really\nhard to reproduce, and it quite depends on the load of the NFS server :/\n\n-- \n·O·  Pierre Habouzit\n··O                                                madcoder@debian.org\nOOO                                                http://www.madism.org\n"},{"id":"79459","messageId":"alpine.LFD.1.10.0806110755190.3101@woody.linux-foundation.org","threadId":"13892","inReplyTo":"20080611074309.GB28629@artemis.madism.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T15:17:04Z","receivedAt":"2008-06-11T15:17:04Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 11 Jun 2008, Pierre Habouzit wrote:\n> \n>   Could this be the source of a problem we often meet at work ? Let me\n> try to describe it.\n\nThe fsync() *should* make no difference unless you actually crash. So my \ninitial reaction is no, but non-coherent client-side write caching over \nNFS may actually make a difference.\n\n>   We work with our git repositories (storages I should say) on NFS\n> homes, with workdirs on a local directory (NFS homes are backuped daily,\n> hence everything commited get backuped, and developers have shorter\n> compilation times thanks to the local FS).\n\nOk, so your actual git object directory is on NFS?\n\n>   Quite often, when people commit, they have corrupt repositories. The\n> symptom is a `cannot read <sha1>` error message (or many at times). The\n> usual way to \"fix\" it is to git fsck, and git reset (because after the\n> fsck the index is totally screwed and all local files are marked new),\n> and usually everything is fine then.\n\nHmm. Very interesting. That definitely sounds like a cache coherency \nissue (ie the \"fsck\" probably doesn't really _do_ anything, it just \ndelays things and possibly causes memory pressure to throw some stuff out \nof the cache).\n\nWhat clients, what server?\n\nNFS clients (I assume v2, which is not coherent) _should_ be doing what is \ncalled open-close consistent, which means that while clients can cache \ndata locally, they should aim to be consistent between two clients over a \nan open-close pair (ie if two clients have the same file open at the same \ntime, there are no consistency guarantees, but if you close on one client \nand then open on another, the data should be consistent).\n\nIf open-close consistency doesn't work, then things like various parallel \nload distribution things (clusters with a NFS filesystem doing parallel \nmakes, etc) don't tend to work all that well either (ie an object file is \nwritten on one client, and then used for linking on another).\n\nAnd that is what git does: even without the fsync(), git will \"close()\" \nthe file before it actually does the link + unlink to move it to the new \nposition. So it all _should_ be perfectly consistent even in the absense \nof explicit syncs.\n\nThat said, if there is some problem with that whole thing, then yes, the \nfsync() may well hide it. So yes, adding the fsync() is certainly worth \ntesting.\n\n>   This is not really a hard corruption, and it's really hard to\n> reproduce, I don't know why it happens, and I wonder if this patch could\n> help, or if it's unrelated. I can only bring speculations as it's really\n> hard to reproduce, and it quite depends on the load of the NFS server :/\n\nYes, that sounds very much like a cache coherency issue. The \"corruption\" \ngoes away when the cache gets flushed and the clients see the real state \nagain. But as mentioned, git should already do things in a way that this \nshould all work, but hey, that's using certain assumptions that perhaps \naren't true in your environment.\n\n\t\t\tLinus\n"},{"id":"79461","messageId":"20080611154020.GE28629@artemis.madism.org","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806110755190.3101@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-06-11T15:40:20Z","receivedAt":"2008-06-11T15:40:20Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jun 11, 2008 at 03:17:04PM +0000, Linus Torvalds wrote:\n> \n> \n> On Wed, 11 Jun 2008, Pierre Habouzit wrote:\n> > \n> >   Could this be the source of a problem we often meet at work ? Let me\n> > try to describe it.\n> \n> The fsync() *should* make no difference unless you actually crash. So my \n> initial reaction is no, but non-coherent client-side write caching over \n> NFS may actually make a difference.\n\n  That's what I thought as well but … one never knows ;)\n\n> >   We work with our git repositories (storages I should say) on NFS\n> > homes, with workdirs on a local directory (NFS homes are backuped daily,\n> > hence everything commited get backuped, and developers have shorter\n> > compilation times thanks to the local FS).\n> \n> Ok, so your actual git object directory is on NFS?\n\n  Yes.\n\n> >   Quite often, when people commit, they have corrupt repositories. The\n> > symptom is a `cannot read <sha1>` error message (or many at times). The\n> > usual way to \"fix\" it is to git fsck, and git reset (because after the\n> > fsck the index is totally screwed and all local files are marked new),\n> > and usually everything is fine then.\n> \n> Hmm. Very interesting. That definitely sounds like a cache coherency \n> issue (ie the \"fsck\" probably doesn't really _do_ anything, it just \n> delays things and possibly causes memory pressure to throw some stuff out \n> of the cache).\n> \n> What clients, what server?\n\n  Server uses NFSv3 kernel server from Debian's 2.6.18 etch (up to\ndate).  Clients are various Unbuntu/Debian's with at least 2.6.18\nkernels, some .22 .24 and .25.  It's a really simple setup, no clusters\nare involved. The server exports an ext3 over dm-crypt partition though,\nbut I would be surprised it matters.\n\n> That said, if there is some problem with that whole thing, then yes, the \n> fsync() may well hide it. So yes, adding the fsync() is certainly worth \n> testing.\n\nOkay, I'll try to make my colleagues use that to see if they still have\nthe issues. I work on a laptop and not NFS, so I'm not the one having\nthe issues, only the one having to fix them on other's machines ;P\n\n> >   This is not really a hard corruption, and it's really hard to\n> > reproduce, I don't know why it happens, and I wonder if this patch could\n> > help, or if it's unrelated. I can only bring speculations as it's really\n> > hard to reproduce, and it quite depends on the load of the NFS server :/\n> \n> Yes, that sounds very much like a cache coherency issue. The \"corruption\" \n> goes away when the cache gets flushed and the clients see the real state \n> again. But as mentioned, git should already do things in a way that this \n> should all work, but hey, that's using certain assumptions that perhaps \n> aren't true in your environment.\n\n  Well we have the issue for quite a long time actually, and given that\nit's hard to reproduce, I'm never in a state to be able to give more\nuseful informations :/ We'll see if the fsync() helps or not…\n\n-- \n·O·  Pierre Habouzit\n··O                                                madcoder@debian.org\nOOO                                                http://www.madism.org\n"},{"id":"79466","messageId":"alpine.LFD.1.10.0806110952290.3101@woody.linux-foundation.org","threadId":"13892","inReplyTo":"20080611154020.GE28629@artemis.madism.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T17:25:28Z","receivedAt":"2008-06-11T17:25:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 11 Jun 2008, Pierre Habouzit wrote:\n> > \n> > Hmm. Very interesting. That definitely sounds like a cache coherency \n> > issue (ie the \"fsck\" probably doesn't really _do_ anything, it just \n> > delays things and possibly causes memory pressure to throw some stuff out \n> > of the cache).\n> > \n> > What clients, what server?\n> \n>   Server uses NFSv3 kernel server from Debian's 2.6.18 etch (up to\n> date).\n\nOk, it's almost impossible to be a server-side issue then - I could \nimagine that if you had some fancy cluster server or something, but in \nthat kind os straightforward situation the only thing that is going to \nmatter is the client-side caching.\n\nI stopped using NFS so long ago that the only case I ever worried about \nand knew anything about was the traditional v2 issues. But iirc, v3 does \nnothing much to change the caching rules (v4, on the other hand, does add \ndelegations and explicit caching support).\n\n> Clients are various Unbuntu/Debian's with at least 2.6.18 kernels, some \n> .22 .24 and .25.\n\nI'll ask Trond if he has any comments on this from the NFS client side. We \n_did_ hit some other NFS client bug with git long ago, I forget what it \nwas all about (pread/pwrite?).\n\nWhat is quite odd, though, is that exactly because of how git works, I \nwould normally expect each client to not even *try* to look up objects \nthat are written by other clients until long long after they have been \nwritten.\n\nIOW, access to new objects is not something a git client will do just \nbecause the object suddenly appears in a directory - after the file is \nwritten and closed, it will not just be moved to the right position, but \nthere has to be *other* files modified (ie the refs) to tell other clients \nabout the changes too!\n\nAnd that matters because even though there can be things like local \ndirectory caches (and Linux does support negative caching - ie the caching \nof the fact that a file was *not* there), those caches should be empty \nsimply because other clients that didn't create the file shouldn't even \nhave tried to look up non-existent objects!\n\nIf it's a directory content caching issue, then adding an fsync() won't \nmatter. In fact, the fsync() should matter only if the client who wrote \nthe object didn't write it out to the server at close() time, and that \nreally sounds very unlikely indeed. So my personal guess is that the \nfsync() won't make any difference at all.\n\nDo you have people using special flags for your NFS mounts? And do you \nknow if there is some pattern to the client kernel versions when the \nproblem happens?\n\n\t\tLinus\n"},{"id":"79469","messageId":"alpine.LFD.1.10.0806111037530.3101@woody.linux-foundation.org","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806110952290.3101@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T17:46:00Z","receivedAt":"2008-06-11T17:46:00Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 11 Jun 2008, Linus Torvalds wrote:\n> \n> Do you have people using special flags for your NFS mounts? And do you \n> know if there is some pattern to the client kernel versions when the \n> problem happens?\n\nOh, before I even go there - let's get the _really_ obvious case out of \nthe way first.\n\nIf you are using a shared git object repository (why are you doing that, \nbtw?), are people perhaps doing things like \"git gc --auto\" etc at the \nsame time? Perhaps even unknowingly, thanks to autogc?\n\nThat's an absolute no-no. It works on a real POSIX filesystem, because \neven if you unlink a file that is in use by another process, the other \nprocess still has access to the data and the file won't be *really* \nremoved until all users have gone away.\n\nThat's also true within a _single_ NFS client thanks to so-called \n\"silly-renaming\", but it is *not* true across multiple clients.\n\nSo if one client is doing some kind of gc and creates a new pack-file and \nthen removes old loose objects, and another client has already looked up \nand opened that loose object (but not finished reading it), then when the \nfile gets removed, you will literally lose the data on the other client \nand get a short read!\n\nAnd nothing we can do can ever fix this very fundamental issue of NFS. NFS \nsimply isn't an even remotely POSIX filesystem, even though it's set up to \nmostly _look_ like one when accessed from a single client.\n\nIn general, I would discourage people ever sharing object directories \namong multiple users except in a server kind of environment (eg \nkernel.org). \n\n\t\t\tLinus\n"},{"id":"79514","messageId":"20080611222534.GC16439@artemis.madism.org","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806111037530.3101@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-06-11T22:25:34Z","receivedAt":"2008-06-11T22:25:34Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jun 11, 2008 at 05:34:53PM +0000, Linus Torvalds wrote:\n> \n> \n> > > >   Quite often, when people commit, they have corrupt repositories. The\n> > > > symptom is a `cannot read <sha1>` error message (or many at times).\n> \n> Btw, do you have exact error messages?\n\n  Well, like said I don't have had the problem myself, and it's hard to\nreproduce, people have it like once a week average, for people that\nperform around 50 to 100 commits a week.\n\n> That plain \"cannot read <sha1>\" sounds unlikely. It exists in archive-tar, \n> archive-zip and builtin-tag (and in two of the tests), but not in any \n> commit paths that I can tell.\n\n  Well that was what I recalled, I'll pass the word that I want a copy\nof the message so that people don't trash their history when it happens.\n\nOn Wed, Jun 11, 2008 at 05:25:28PM +0000, Linus Torvalds wrote:\n> Do you have people using special flags for your NFS mounts? And do you \n> know if there is some pattern to the client kernel versions when the \n> problem happens?\n\n  not that I'm aware of. One machine with issues has this:\n192.168.2.2:/home on /home type nfs (rw,noatime,bg,intr,hard,tcp,rsize=65536,wsize=65536,addr=192.168.2.2)\n\n  IOW nothing fancy that I can see.\n\nOn Wed, Jun 11, 2008 at 05:46:00PM +0000, Linus Torvalds wrote:\n> On Wed, 11 Jun 2008, Linus Torvalds wrote:\n> > \n> > Do you have people using special flags for your NFS mounts? And do you \n> > know if there is some pattern to the client kernel versions when the \n> > problem happens?\n> \n> Oh, before I even go there - let's get the _really_ obvious case out of \n> the way first.\n> \n> If you are using a shared git object repository (why are you doing that, \n> btw?), are people perhaps doing things like \"git gc --auto\" etc at the \n> same time? Perhaps even unknowingly, thanks to autogc?\n\n  No, we're not using a shared git object repository, each developper\nhas a git checkout in his /home (on NFS) but works for real in a workdir\nthat lives on his local hard drive (to get faster compilation times,\nbecause NFS really sucks at speed for compilation). Though, people\nworking on plain NFS have had the same problems.\n\n  I also had the same reaction as you do, and I've seen such problems\noccur, and the operation that we were doing was just \"git commit -as\" or\nsomething very similar. No nothing in progress in any other terminal at\nthe same time.\n\n> So if one client is doing some kind of gc and creates a new pack-file and \n> then removes old loose objects, and another client has already looked up \n> and opened that loose object (but not finished reading it), then when the \n> file gets removed, you will literally lose the data on the other client \n> and get a short read!\n\n  I don't think that is what happens.\n\n> And nothing we can do can ever fix this very fundamental issue of NFS. NFS \n> simply isn't an even remotely POSIX filesystem, even though it's set up to \n> mostly _look_ like one when accessed from a single client.\n> \n> In general, I would discourage people ever sharing object directories \n> among multiple users except in a server kind of environment (eg \n> kernel.org). \n\n  It's not shared repositories, really, it's just for conveniency so\nthat the object store is on NFS (and backuped like all that is on the\nNFS server) but the checkouts on local hard drives, so that all\noperations to the checkout are local. And when people commit, it goes\ninto the NFS and all is perfectly fine. Those repositories are *ONLY*\nused by one physical user exclusively.\n\n\n  I'll try to do some commits on NFS repeatedly tonight and trigger the\nissue, I'll keep you posted.\n\n\n-- \n·O·  Pierre Habouzit\n··O                                                madcoder@debian.org\nOOO                                                http://www.madism.org\n"},{"id":"79521","messageId":"20080611230355.GE16439@artemis.madism.org","threadId":"13892","inReplyTo":"20080611222534.GC16439@artemis.madism.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-06-11T23:03:55Z","receivedAt":"2008-06-11T23:03:55Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Wed, Jun 11, 2008 at 10:25:34PM +0000, Pierre Habouzit wrote:\n> On Wed, Jun 11, 2008 at 05:34:53PM +0000, Linus Torvalds wrote:\n> > \n> > \n> > > > >   Quite often, when people commit, they have corrupt repositories. The\n> > > > > symptom is a `cannot read <sha1>` error message (or many at times).\n> > \n> > Btw, do you have exact error messages?\n> \n>   Well, like said I don't have had the problem myself, and it's hard to\n> reproduce, people have it like once a week average, for people that\n> perform around 50 to 100 commits a week.\n\n  I should say that the problem happens more often when people use the script I\ninlined in the end (this scripts does what git pull --rebase nowadays\ndoes, with a dirty workdir, roughly, I'm not its author, the goal isn't\nto discuss its lack of beauty or whatever, just to point out some\npossible things that leads to the NFS issue).\n\n  Next time something breaks, I'll be able to give you a log.\n\n-------------------------------------\n#!/bin/bash\n\nOPTIONS_SPEC=\nSUBDIRECTORY_OK=t\n. git-sh-setup\nrequire_work_tree\n\ncase $# in\n    0)\n        branch=origin/$(git symbolic-ref HEAD|sed -e 's!refs/heads/!!')\n        ;;\n    1)\n        branch=\"$1\"\n        ;;\n    *)\n        echo 1>&2 \"$(basename $0) [<branch>]\"\n        exit 1\n        ;;\nesac\n\nif git rev-parse --verify HEAD > /dev/null &&\n    git update-index --refresh &&\n    git diff-files --quiet &&\n    git diff-index --cached --quiet HEAD --\nthen\n    NEED_TEMPO=\nelse\n    NEED_TEMPO=t\nfi\n\ngit fetch\ntest -z \"$NEED_TEMPO\" || git commit -a -s -m'tempo'\ngit rebase \"$branch\"\n\nif test -n \"$NEED_TEMPO\"; then\n    if test  -d \"$(dirname \"$(git rev-parse --git-dir)\")/.dotest\"; then\n        echo \"\"\necho \"run 'git reset HEAD~1' when rebase is finished\"\n    else\n        git reset HEAD~1\n    fi\nfi\n-------------------------------------\n\n  Also note that now people use an enhanced version of that script (the\none in attachment). It still breaks, but less often. I'm not sure it's\nvaluable information at all, but one never knows.\n-- \n·O·  Pierre Habouzit\n··O                                                madcoder@debian.org\nOOO                                                http://www.madism.org\n\n\n#!/bin/sh\n\nOPTIONS_SPEC=\"\\\n$(basename $0) [options] [<remote> [<branch>]]\n--\nk,gitk      visualize unmerged differences\nr,rebase    perform a rebase\nm,merge     perform a merge\n\"\nSUBDIRECTORY_OK=t\n. git-sh-setup\nrequire_work_tree\n\nlbranch=$(git symbolic-ref HEAD | sed -e s~refs/heads/~~)\nremote=$(git config --get \"branch.$lbranch.remote\" || echo origin)\nbranch=$(git config --get \"branch.$lbranch.merge\" || echo \"refs/heads/$lbranch\")\n\ncase \"$(git config --bool --get madcoder.up-gitk)\" in\n    true) gitk=gitk;;\n    *)    gitk=:\nesac\ncase \"$(git config --bool --get \"branch.$lbranch.rebase\")\" in\n    true) action=rebase;;\n    *)    action=;;\nesac\n\nwhile test $# != 0; do\n    case \"$1\" in\n        -k|--gitk)\n            shift; gitk=gitk;;\n        --no-gitk)\n            shift; gitk=:;;\n        -r|--rebase)\n            shift; action=rebase;;\n        --no-rebase)\n            shift; rebase=${rebase#rebase};;\n        -m|--merge)\n            shift; action=merge;;\n        --no-merge)\n            shift; rebase=${rebase#merge};;\n        --)\n            shift; break;;\n        *)\n            usage;;\n    esac\ndone\n\ncase $# in\n    0) ;;\n    1) remote=\"$1\";;\n    2) remote=\"$1\"; branch=\"$2\";;\n    *) usage;;\nesac\n\ngit fetch \"${remote}\"\nif test `git rev-list ..\"${remote}/${branch#refs/heads/}\" -- | wc -l` = 0; then\n    echo \"Current branch $lbranch is up to date.\"\n    exit 0\nfi\n\n$gitk ..\"${remote}/${branch#refs/heads/}\" --\nif test -z \"$action\"; then\n    echo -n \"(r)ebase/(m)erge/(q)uit ? \"\n    read ans\n    case \"$ans\" in\n        r*) action=rebase;;\n        m*) action=merge;;\n        *)  exit 0;;\n    esac\nfi\n\nunclean=\ngit rev-parse --verify HEAD > /dev/null && \\\n    git update-index --refresh && \\\n    git diff-files --quiet && \\\n    git diff-index --cached --quiet HEAD -- || unclean=t\n\ncase \"$action\" in\n    rebase)\n        test -z \"$unclean\" || git stash save \"git-up stash\"\n        git rebase \"${remote}/${branch#refs/heads/}\"\n        ;;\n    merge)\n        test -z \"$unclean\" || git stash save \"git-up stash\"\n        git merge \"${remote}/${branch#refs/heads/}\"\n        ;;\n    *)\n        echo 1>&2 \"no action specified\"\n        exit 1\n        ;;\nesac\n\nif test -n \"$unclean\"; then\n    if test  -d \"$(git rev-parse --git-dir)/../.dotest\"; then\n        echo \"\"\n        echo \"run 'git stash apply' when rebase is finished\"\n    else\n        git stash apply\n    fi\nfi\n"},{"id":"79602","messageId":"alpine.LFD.1.10.0806120758090.3041@woody.linux-foundation.org","threadId":"13892","inReplyTo":"20080611222534.GC16439@artemis.madism.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-12T15:33:53Z","receivedAt":"2008-06-12T15:33:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 12 Jun 2008, Pierre Habouzit wrote:\n> \n>   No, we're not using a shared git object repository, each developper\n> has a git checkout in his /home (on NFS) but works for real in a workdir\n> that lives on his local hard drive (to get faster compilation times,\n> because NFS really sucks at speed for compilation). Though, people\n> working on plain NFS have had the same problems.\n\nAhhh..\n\nIn that case it's not going to be a client caching issue - at least not in \nthe sense that two different clients are out-of-sync with each other wrt \ncaches. It sounds as it you only ever have one client that reads and \nwrites to the same git repository at a time.\n\nSo scratch all the previous theory.\n\nQuite frankly, in that case, it sounds more like simply some NFS problem. \nAnd we _have_ had NFS problems before. See the threads\n\n - bug: git-repack -a -d produces broken pack on NFS\n\n   Turned out to apparently be ethernet packet corruption that was not \n   detected by the hardware and was due to a badly seated ethernet card!\n\n - git 1.5.3.5 error over NFS\n\n   Some unexplained corruption due to problms with pread() on NFS not \n   returning data that was previously written.\n\nfor example.\n\nBasically, NFS has many serious failure cases that can go undetected, and \nit _could_ be that you actually have flaky NFS but never noticed it before \nbecause most tools don't care as deeply as git does (ie if a bit is \nflipped in some random data, a lot of tools will never notice). There are \nsupposed to be checksums etc on the network packets that NFS uses, but:\n\n - the ethernet checksum (which is a fairly strong CRC) is sadly often not \n   even checked by some switches and/or cards, and especially if it's a \n   store-and-forward switch that doesn't check the CRC properly, it can \n   end up re-sending a corrupt packet with a recomputed ethernet CRC that \n   now matches the _corrupt_ data. Oops.\n\n - Perhaps worse, the ethernet checksum is purely a physical layer one, \n   not an end-to-end checksum, which not only explains how a switch can \n   re-generate a broken one, but also means that even if the ethernet card \n   checks it properly, it doesn't actually account for any corruption that \n   happens _afterwards_. So if there is corruption going from the card to \n   memory (which was apparently the problem in the first git thread \n   above), the CRC got checked earlier and the new corruption isn't found.\n\n - there _is_ an TCP/IP-level packet check, with a checksum of the IP \n   header, and a separate checksum of UDP and TCP data. HOWEVER. All these \n   checksums are very very weak, and to make things worse, the UDP \n   checksum can be entirely disabled, and quite often \"better\" ethernet \n   cards will do checksumming for you in hardware, which again means that \n   it's not an end-to-end checksum, and you have the exact same failure \n   case as with the ethernet CRC.\n\nIOW, there are safety nets in place, but they tend to be fairly easily \nbroken under certain circumstances.\n\nAdd to the above the possibility of just a kernel NFS bug (or a NFSd one), \nand it would really be very interesting to hear:\n\n - do the errors seem to happen more at certain clients than others?\n\n   If it's a client-side problem, it really should happen more for certain \n   kernel versions or certain hardware.\n\n - have you had any other anecdotal evidence of problems with non-git \n   usage? Unexplained SIGSEGV's if you have binaries over NFS, for \n   example? Strange syntax errors when compiling over NFS?\n\nI'm not discounting a git bug, but quite frankly, it really is worth \nchecking that your network/NFS setup is solid.\n\n\t\t\tLinus\n"},{"id":"79605","messageId":"20080612160012.GA13533@artemis.madism.org","threadId":"13892","inReplyTo":"alpine.LFD.1.10.0806120758090.3041@woody.linux-foundation.org","subject":"Re: Consolidate SHA1 object file close","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-06-12T16:00:12Z","receivedAt":"2008-06-12T16:00:12Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Thu, Jun 12, 2008 at 03:33:53PM +0000, Linus Torvalds wrote:\n> IOW, there are safety nets in place, but they tend to be fairly easily \n> broken under certain circumstances.\n\n  Right, though this is a LAN with really few switches between the\nclients and the server, and the issue happens with any client with that\nsetup.\n\n> Add to the above the possibility of just a kernel NFS bug (or a NFSd one), \n> and it would really be very interesting to hear:\n> \n>  - do the errors seem to happen more at certain clients than others?\n\n  Not really, it happens more often with the script I inlined in one of\nmy mails, than with the one I attached to it. And people working on\n_pure_ NFS have the issue a bit less than the ones using their workdir\non a separate local device. That's all I can tell for now.\n\n  One of the developper told me that this pattern of use triggers the\nproblem more for him: he has a 'master' branch checkouted in his NFS\nhome, and a 'local' branch on his local hard drive workdir[0]. The issue\nhappens more when he's working on the two workdirs at the same time\n(for a value of \"at the same time\" that is like in the same minute, not\nat the same nanosecond of course, he never commits in both workdirs at\nthe same time).  When he only works in his NFS 'master' or the local\n'local' branch, it happens really less often.\n\n>    If it's a client-side problem, it really should happen more for certain \n>    kernel versions or certain hardware.\n\n  It doesn't afaict. Clients are heterogenous in kernel versions (.18,\n.22, .24, .25 for whate I've seen), and in hardware (all machines are\nDell computers, but from really different years, hence different mobos\nand NICs. Some even have non Dell gigabit NICs in them).\n\n\n>  - have you had any other anecdotal evidence of problems with non-git \n>    usage? Unexplained SIGSEGV's if you have binaries over NFS, for \n>    example? Strange syntax errors when compiling over NFS?\n\n  Not really no. Our NFS server is remarkably stable with anything else\n(it's also remarkably slow compared to local drives but that's not\nreally relevant ;p).\n\n> I'm not discounting a git bug, but quite frankly, it really is worth \n> checking that your network/NFS setup is solid.\n\n  Well to date I'd say it's quite solid. We are a software company and\neven have tested our software (that does heavy use of mmap, pread,\npwrite, and other things that NFS is not often dealing with very well)\non that very server without a glitch that could have been attributed to\nNFS (I mean we've had tons of bugs, but it was always in our software in\nthe end ;p). Though we really rarely pread what we just pwrite-d or\nthings like that, so maybe we never triggered a possible kernel bug\neither :)\n\n\n  [0] the reason of that setup is that when we work on topic branches,\n      we sometimes spot big bugs that are small to fix, and we use the\n      \"NFS\" master branch to push those bugfixes as soon as we find\n      them, whereas local is pushed only when the feature is ready.\n\n-- \n·O·  Pierre Habouzit\n··O                                                madcoder@debian.org\nOOO                                                http://www.madism.org\n"}]}