Re: fsck errors on newly cloned, newly imported git repository

> On Mon, Oct 25, 2010 at 6:58 AM, Drew Northup <drew.northup@maine.edu> wrote:
>>
>> On Sun, 2010-10-24 at 11:54 -0400, Mike Herrick wrote:
>>> This weekend we're cutting over to use git for our source code control
>>> system.  I've imported about 20 years worth of previous history using
>>> "git cvsimport" (takes about four hours).  I then cloned the resulting
>>> repository onto five different machines (four Linux, one Solaris).
>>> I've set up a cron job to do a nightly "git fsck" on each of the five
>>> machines, and last night, two of the machines reported fsck errors on
>>> their initial run.
>> <snip>
>>
>>> The errors reported on these two machines were different, but what's
>>> interesting is that all of the missing blobs refer to various
>>> revisions of the same file, namely our "Changes" file (which is
>>> updated with each change).  It's also the largest file in our
>>> repository (3.3M).  I immediately started looking at logs to see if
>>> there was any indication of disk corruption and found none (no SMART
>>> errors either).  Both of these machines have been stable over a
>>> multi-year period of time (no unexplained crashes).  They're also
>>> older Linux machines (running  2.6.5-1.358 and  2.6.1-1.65, with
>>> relatively little memory: 1GB and .5GB), but with newly installed
>>> version of git (1.7.3.1).  I initially used git-daemon for the clone
>>> process, but even using ssh, I still see fsck errors on the resulting
>>> clones on these two machines.
>>
>> Did you "git fsck" BEFORE you attempted to clone? Is it ONLY clones
>> showing errors? Alas, no blatant evidence of disk corruption is not
>> evidence of no disk corruption as well.
>
> Thanks for your reply.
>
> Only two of the five clones exhibit fsck errors and the server
> repository has no fsck errors.
>
> The two machines report different sets of missing blobs, but always in
> the "Changes" file (which has the somewhat unique characteristics that
> it is the "most changed" file in the repository, the largest, and one
> which is almost always only added to).
>
> I've since created two more clones on one of the machines (one using
> git-daemon and the other ssh) and both of these clones have the exact
> same set of missing blobs!  For me this rules out disk corruption.
>
> The good(?) news is that the process is repeatable on one machine:
> cloning from a known good repository results in different (but
> repeatable) errors.  Performing a second clone on the other "bad"
> machine also results in missing blobs, but different ones than the
> first (although all in the Changes file).
>
> My current thought is that somehow it's related to very old kernels?
> Apparently these machines are FC2 vintage.