Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)

From: Thomas Rast <trast@inf.ethz.ch>
Date: Apr 6, 2013, 08:11 UTC
Message-ID: <871uaorscq.fsf@linux-k42r.v.cablecom.net>
In-Reply-To: <515EE38D.2000502@olwing.se>

Kenneth Ölwing <kenneth@olwing.se> writes:

Show 12 quoted lines

> On 2013-04-05 15:42, Thomas Rast wrote:
>> Can you run the same tests under strace or similar, and gather the
>> relevant outputs? Otherwise it's probably very hard to say what is
>> going wrong. In particular we've had some reports on lustre that
>> boiled down to "impossible" returns from libc functions, not git
>> issues. It's hard to say without some evidence. 
> Thomas, thanks for your reply.
>
> I'm assuming I should strace the git commands as they're issued? I'm
> already collecting regular stdout/err output in a log as I go. Is
> there any debugging things I can turn on to make the calls issue
> internal tracing of some sort?

I don't think there's any internal debugging that helps at this point. Usually errors pointing to corruption are caused by a chain of syscalls failing in some way, and the final error shows only the last one, so strace() output is very interesting.

> The main issue I see is that I suspect it will generate so much data
> that it'll overflow my disk ;-).

Well, assuming you have some automated way of detecting when it fails, you can just overwrite the same strace output file repeatedly; we're only interested in the last one (or all the last ones if several gits fail concurrently).

Fiddling with strace will unfortunately change the timings somewhat (causing a bunch of extra context switches per syscall), but I hope that you can still get it to reproduce.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch