High locking contention during repack?

From: Iucha, Florin <florin.iucha@amd.com>
Date: Dec 12, 2018, 03:01 UTC
Message-ID: <SN1PR12MB23840AFE62E41D908A40D1B095A70@SN1PR12MB2384.namprd12.prod.outlook.com>

Greetings,

I am running “git-repack  -A -d -f -F --window=250 --depth=250” on a Git repository converted using git-svn.

The repository contains more than 10 years of development, and a mixture of source code and media assets. The size of the objects directory is around 50GB, and there are around 700k objects. There are several svn branches.

I have ran the repack with Git 2.17.1 (original package from Ubuntu 18.04), 2.19.1 (package from Ubuntu Git PPA) and with git next (bc1bbc6f855c3b5ef7fcbd0f688f647c4e5b208b).

The system is a 16 core / 32 thread Threadripper with 128GB of RAM and NVMe storage. The repack starts strong, with 32 threads but it fairly quickly gets to 99% done and the number of threads drops to 4 then 3 then 2. However, running “dstat 5” I see lots of “sys” time without any IO time (the network traffic you see is caused by SSH).

--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
24  24  48   4   0|   0     0 | 362B 8392B|   0     0 |  64k 5042
27  25  44   4   0|   0     0 | 289B 4830B|   0     0 |  36k 5017
24  24  48   4   0|   0     0 | 362B 8392B|   0     0 |  64k 5042
27  25  44   4   0|   0     0 | 289B 4830B|   0     0 |  36k 5017
20  22  54   4   0|   0     0 | 330B 4826B|   0     0 |  31k 4657
22  23  51   3   0|   0     0 | 286B 4834B|   0     0 |  27k 4536
17  23  56   4   0|   0     0 | 299B 4820B|   0     0 |  29k 4099
24  23  49   4   0|   0     0 | 347B 4830B|   0     0 |  32k 4436
27  24  45   4   0|   0     0 | 292B 4820B|   0     0 |  22k 4513
26  27  44   3   0|   0     0 | 292B 4820B|   0     0 |  24k 4878
25  28  44   4   0|   0     0 | 329B 4837B|   0     0 |  45k 5023
23  30  44   3   0|   0  9830B| 370B 4836B|   0     0 |  40k 5064
23  26  47   4   0|   0     0 | 373B 4848B|   0     0 |  52k 4746
25  26  45   4   0|   0     0 | 323B 4833B|   0     0 |  31k 4874
24  28  45   3   0|   0     0 | 269B 4842B|   0     0 |  30k 4820
25  28  43   4   0|   0     0 | 280B 4826B|   0     0 |  45k 5237
25  29  43   3   0|   0     0 | 263B 4826B|   0     0 |  26k 5037
22  30  45   4   0|   0     0 | 287B 4823B|   0     0 |  45k 4154
  0  53  42   5   0|   0     0 | 275B 1210B|   0     0 |5017  1331
  0  53  42   5   0|   0     0 |  97B    0 |   0     0 |5006  1272
  0  53  42   5   0|   0     0 |  70B    0 |   0     0 |5008  1272
  0  53  42   5   0|   0     0 |  24B    0 |   0     0 |5007  1283
11  43  43   4   0|   0     0 | 644B   19k|   0     0 |  20k 2995
25  28  43   4   0|   0   123k| 304B 4802B|   0     0 |  29k 5118
25  27  44   4   0|   0     0 | 297B 4800B|   0     0 |  41k 5121
26  27  43   4   0|   0     0 | 320B 4786B|   0     0 |  29k 4973
22  30  44   3   0|   0     0 | 332B 4786B|   0     0 |  32k 4610
24  27  46   3   0|   0     0 | 296B 4786B|   0     0 |  56k 4632
35  17  44   3   0|   0     0 | 292B 4776B|   0     0 | 100k 4889
34  20  43   4   0|   0  9830B| 610B 4820B|   0     0 |  59k 4811

This when it gets to two threads phase:

  1  40  59   0   0|   0    38k| 373B 2458B|   0     0 |  50k 2874
  0  40  59   0   0|   0     0 | 268B 2892B|   0     0 |  10k 2858
  0  40  59   0   0|   0     0 | 299B 3102B|   0     0 |8392  2919
  1  40  59   0   0|   0     0 | 297B 3176B|   0     0 |  21k 2831
  0  40  59   0   0|   0     0 | 281B 3414B|   0     0 |9192  2856
  1  40  59   0   0|   0     0 | 264B 3493B|   0     0 |  17k 2905
  1  40  59   0   0|   0    35k| 340B 3571B|   0     0 |  13k 2885

The repack still progresses, as I see the count of repacked objects being incremented every several minutes. But I suspect there is some terrible lock contention, because otherwise I can’t explain the high system usage with no IO.

Running a strace on the running git-repack process shows only these: --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---

Any idea on how to debug this? I have ran git-repack under gdb, but it seems to spin on builtin/repack.c line 409.

Thank you for Git,

florin