Re: reducing prune sync()s

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: May 30, 2008, 02:17 UTC
Message-ID: <alpine.LFD.1.10.0805291905360.3141@woody.linux-foundation.org>
In-Reply-To: <1212112295.3094.3.camel@obelisk.thedillows.org>

On Thu, 29 May 2008, David Dillow wrote:

Show 12 quoted lines

> 
> On Thu, 2008-05-29 at 17:27 -0700, Linus Torvalds wrote:
> > That would be much better. The code was ported from shell script, and 
> > there is no fsync() in shell, but the rule should basically be that you 
> > can remove all the objects that correspond to a pack-file after you have 
> > made sure that the pack-file (and it's index - we can re-generate the pack 
> > index, but realistically speaking it's *much* better to not have to) is 
> > stable on disk.
> 
> Even if the data is stable on disk, don't we also need to ensure the
> pack's connectivity to the namespace is also stable? Without an fsync()
> of the directory that contains it, could it go away?

In theory, yes. That said, it would always be in lost+found, so the data wouldn't ever get really lost. In that sense it is no different from a lot of other theoretical git corruption issues - git in general does *not* guarantee that the repository will not need to be "fixed up", it just makes a strong case for

 - git will always at least *see* the corruption

   (ie it is by design is very hard to corrupt a git repo silently and 
   subtly!)

 - git makes it very hard to lose data

   Old data is not overwritten, but that doesn't mean that you may not 
   have to _look_ for it!

An example of the latter is how a crash in the middle of "git commit" may actually cause partial *new* objects to be on disk (the objects themselves are not fsync'ed when written!) and may end up with the ref being updated but some object it points to was never written (again, a "git commit" does not wait until things are stable!), and the index file may be totally corrupt (again, no fsync anywhere!).

So if you have a system crash at a really bad time, you may have a git repository that needs manual intervention to actually be *usable*. I hope nobody ever believed anything else. That manual intervention may be things like:

 - throw away a corrupt .git/index file and re-create it (git read-tree)

 - possibly have to recover refs manually ("git fsck" + look at dangling 
   commits)

 - actually throw away broken commits, and re-create them (ie basically 
   doing a "git reset <known-good-state>" plus re-committing the working 
   tree or perhaps re-doing a whole "git am" series or something)

 - and yes, possibly recover lost inodes from /lost+found

Now, quite frankly, all of these are extremely rare. In most cases they will never happen at all, simply because the filesystem itself is more or less transactional.

		Linus