Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)

From: Kenneth Ölwing <kenneth@olwing.se>
Date: Apr 5, 2013, 14:45 UTC
Message-ID: <515EE38D.2000502@olwing.se>
In-Reply-To: <87li8xrt5f.fsf@linux-k42r.v.cablecom.net>

On 2013-04-05 15:42, Thomas Rast wrote:

Show 5 quoted lines

> Can you run the same tests under strace or similar, and gather the 
> relevant outputs? Otherwise it's probably very hard to say what is 
> going wrong. In particular we've had some reports on lustre that 
> boiled down to "impossible" returns from libc functions, not git 
> issues. It's hard to say without some evidence.

Thomas, thanks for your reply.

I'm assuming I should strace the git commands as they're issued? I'm already collecting regular stdout/err output in a log as I go. Is there any debugging things I can turn on to make the calls issue internal tracing of some sort?

The main issue I see is that I suspect it will generate so much data that it'll overflow my disk ;-). Consider that my hammer consists of a Perl script that forks a number of tasks (e.g. 15) that each loops doing clone/commit/push/pull, with retrying on a few levels as errors occur (usually expected ones due to the concurrency, i.e. someone else pushed so a pull is necessary first, but occasionally the central repo is broken enough that it can't be cloned from, or at least not checked out master from...sometimes with printed errors that still give me a zero exit code...). That is then also run on several machines to the same repo to hopefully cause a breakage by sheer pounding...it's going to generate huge collections of strace output I expect...

I have some variations of this (e.g. all tasks are working on different branches, improving concurrency in some respects, but effects there have been that at the end I was missing a branch or so...). The likelihood of problems seems to increase when I actually use ssh in my ultimate setup, so a loadbalancer roundrobins each call to any of several hosts. In that case I must admit I don't know how to get in on the action since I guess I would need to strace the git-upload/receive-pack processes on the server side...?

Lastly, I don't know how much this will impact timings etc, or load. To get a broken result I have sometimes needed to run for many hours, others fairly quickly.

Well...I will try, it'll probably be a blast :-)

BTW, this is mostly done on Centos 6.3 and 6.4, locally built git-1.8.2.

ken1