Re: [PATCH/RFC] Allow writing loose objects that are corrupted in a pack file

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Jan 7, 2009, 04:54 UTC
Message-ID: <alpine.LFD.2.00.0901062026500.3057@localhost.localdomain>
In-Reply-To: <1231292360.8870.61.camel@starfruit>

On Tue, 6 Jan 2009, R. Tyler Ballance wrote:

Show 5 quoted lines

> 
> I'll back the patch out and redeploy, it's worth mentioning that a
> coworker of mine just got the issue as well (on 1.6.1). He was able to
> `git pull` and the error went away, but I doubt that it "magically fixed
> itself"

Quite frankly, that behaviour sounds like a disk _cache_ corruption issue. The fact that some corruption "comes and goes" and sometimes magically heals itself sounds very much like some disk cache problem, and then that particular part of the cache gets replaced and then when re-populated it is magically correct.

We had that in one case with a Linux NFS client, where a rename across directories caused problems.

This was a networked filesystem on OS X, right? File caching is much more "interesting" in networked filesystems than it is in normal private on-disk ones.

> I've tarred one of the repositories that had it in a reproducible state
> so I can create a build and extract the tar and run against that to
> verify any patches anybody might have, but unfortunately at 7GB of
> company code and assets, I can't exactly share ;)

The thing to do is

 - untar it on some trusted machine with a local disk and a known-good 
   filesystem.

   IOW, not that networked samba share.

 - verify that it really does happen on that machine, with that untarred 
   image. Because maybe it doesn't.

   The hope is that you caught the corruption in the cache, and it 
   actually got written out to the tar-file. But if it _is_ a disk cache 
   (well, network cache) issue, maybe the IO required to tar everything up 
   was enough to flush it, and the tar-file actually _works_ because it 
   got repopulated correctly.

   So that's why you should double-check that it really ends up being 
   corrupt after being untarred again.

 - go back and test the original git repo on the network share, preferably 
   on another client. See if the error has gone away.

 - If so, try to compare that known-corrupt filesystem with the original 
   one:  and preferably do this on another machine over the network mount.

   See if they differ. They obviously should *not* differ, since it's an 
   tar/untar of the same files, but ...

The fact that you seem to get a _lot_ of these errors really does make it sound like something in your environment. It's actually really hard to get git to corrupt anything. Especially objects that got packed. They've been quiescent for a long time, they got repacked in a very simple way, they are totally read-only.

But it is _not_ hard to corrupt network filesystems. It's downright trivial with some of them, especially with some hardware (eg there's no end-to-end checksumming except for the _extremely_ weak 16-bit IP csum, and even that has been known to be disabled, or screwed up by ethernet cards that do IP packet offloading and thus computing the csum not on the data that tee user actually wrote, but the data that the card received, which is not necessarily at all the same thing).

And while ethernet uses a stronger CRC, that one is not end-to-end, so corruption on the card or in a switch in between easily defeats that too.

Just google for something like

	"OS X" SMB "file corruption"

and you'll find quite a bit of hits. Not all that unusual.

				Linus