Troubleshoot clone issue to NFS.

In setting up some new git servers I was trying to test the performance of some NFS mounted volumes and when compared to local disk (although this is a vitualized server so not truly local) cloning to NFS was taking a long time.

Here are some timings:

~ $ time bin/git clone https://github.com/git/git test Cloning into 'test'... remote: Counting objects: 185964, done. remote: Compressing objects: 100% (276/276), done. remote: Total 185964 (delta 203), reused 32 (delta 32), pack-reused 185656 Receiving objects: 100% (185964/185964), 61.42 MiB | 26.16 MiB/s, done. Resolving deltas: 100% (135454/135454), done. Checking connectivity... done.

real 0m8.156s user 0m10.569s sys 0m3.857s

~ $ time bin/git clone https://github.com/git/git /sami/test Cloning into '/sami/test'... remote: Counting objects: 185964, done. remote: Compressing objects: 100% (276/276), done. remote: Total 185964 (delta 203), reused 32 (delta 32), pack-reused 185656 Receiving objects: 100% (185964/185964), 61.42 MiB | 10.15 MiB/s, done. Resolving deltas: 100% (135454/135454), done. Checking connectivity... done. Checking out files: 100% (2795/2795), done.

real 0m58.685s user 0m12.153s sys 0m7.619s

So cloning to NFS is 50 seconds slower on average (I've run this a lot over the last few days and it does appear to be consistent and not a network / github connectivity issue).  Tests creating files on the NFS with dd didn't show that much difference to the local disk (and were sometimes quicker).

Volume mount differences are:

/dev/mapper/rootvg-homelv on /home type ext4 (rw,nodev) nfsserver:/vol/sami/repos on /sami type nfs (rw,bg,nfsvers=3,tcp,hard,nointr,timeo=600,rsize=32768,wsize=32768,addr=10.1.1.1)

And there doesn't appear to be any issue with NFS retransmissions:

/sami $ nfsstat Client rpc stats: calls retrans authrefrsh 11719847 0 11720190

This morning I did some more digging as when I tried this on a newly build server the NFS times were slower than local disk, but only buy around 6-10 seconds.  The new server had git 1.7.1  installed on it compared to 2.4.0 on the machine I've been testing on.  So I build a number of versions of git to test each one to try and find the point where it changed:

v1.8.0 11.363 s v1.8.3 13.597 s v1.8.4 13.958 s v1.8.4.1 14.563 s v1.8.4.2 1m 0s v1.8.4.3 1m 9s v1.8.4.5 1m 1s v1.8.5 1m 0s v1.8.5.6 1m 0s v1.9.0 1m 38

v2.4.0 58s v2.4.1 58s

So there appears to be a change in 1.8.4.2 that made this issue appear for me.  Looking at the release notes the only thing that I can see that might be related could be:

* When an object is not found after checking the packfiles and then
   loose object directory, read_sha1_file() re-checks the packfiles to
   prevent racing with a concurrent repacker; teach the same logic to
   has_sha1_file().

So the questions are:

1) Should I expect a clone to NFS to be that much slower?
2) Is there anything I could do to speed it up (I've tried --bare as that is what the repositories will be when stored on NFS and there wasn't really a difference).
3) What else can I use in git to compare the performance on local to NFS to see if it is just clones that are affected?
4) I assume I could bisect between 1.8.4.1 and 1.8.4.2 to find exactly where things get worse for me?

Thanks for any help,

Steve

________________________________

This e-mail is for the sole use of the intended recipient and contains information that may be privileged and/or confidential. If you are not an intended recipient, please notify the sender by return e-mail and delete this e-mail and any attachments. Certain required legal entity disclosures can be accessed on our website.<http://thomsonreuters.com/prof_disclosures/>