{"thread":{"id":"55652","subject":"`git index-pack --strict` is *very* slow during pushes to large repos","startedAt":"2021-05-09T20:52:35Z","lastAt":"2021-05-09T20:52:35Z","messageCount":1,"participants":["Craig de Stigter"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"423977","messageId":"CAF1M8pcRuYXXORdZ7gdKHN2mxBi6feYppB31bULpZUif6e7APw@mail.gmail.com","threadId":"55652","inReplyTo":"CAF1M8pepgrnZWhx+CeMH85J-5oWx+w6r0w3KCcsG8dWgCT9K9Q@mail.gmail.com","subject":"`git index-pack --strict` is *very* slow during pushes to large repos","fromName":"Craig de Stigter","fromEmail":"craig.destigter@koordinates.com","sentAt":"2021-05-09T20:52:19Z","receivedAt":"2021-05-09T20:52:35Z","isPatch":false,"sender":{"key":"craig.destigter@koordinates.com","avatar":"https://gravatar.com/avatar/5f34ac1c172de83e085896805c18eb8ea55b7194fe992d17c3e0eccce1efffee?d=mp&s=160"},"body":"Hey folks\n\n(apologies if repost; my first post seemed to disappear entirely)\n\nWe're hosting a service with some fairly large repos (created by\nKart[1] ), and I've been looking into some poor\nperformance of `git push` on our service.\n\nBackground: We host repositories with a specific layout. I'll try and avoid\nmost of the technical details but a brief description of the repo layout\nmight be helpful:\n\n- At each revision we have 256 trees\n      - each containing 256 trees (so 65536 trees at this level)\n      - each subtree contains a number of objects (distributed via a hash\n      scheme, evenly across the subtrees)\n- Some repos have up to 100 million blobs active in a given revision.\nIn that case each of the 65536 subtrees would contain ~1500 blobs.\n- Blobs are usually a few bytes to a few KB in size.\n- For various reasons we have disabled deltas entirely.\n- Most repos have a few hundred commits, and a typical commit might\nmodify 100,000 features (again spread evenly across the 65536 trees),\nthus modifying most of the trees also.\n- Our largest repos are currently a few hundred GB on disk.\n\nWe've come across a curious performance issue with `git index-pack` when\ninvoked by `receive-pack` during a push operation. We have\n`transfer.fsckObjects=true` in the server config, so the index-pack\ninvocation looks like:\n\n```\ngit --shallow-file shallow_filename index-pack \\\n   --stdin --keep='receive-pack 1234 on <servername>' \\\n   --show-resolving-progress --report-end-of-input --fix-thin \\\n   --strict\n```\n\nFor our largest repos, when pushing ~100K blobs and associated trees, this\ntakes a *long* time - sometimes over 12 hours. The process uses enormous\namounts of disk IO (all reads; I haven't measured how much per process, but\nthe server was doing many terabytes of IO in total)\n\nHere is one that \"only\" took 45 minutes with a few tracing environment vars\nenabled:\n\n```\n$ cat craig.pack | /opt/sno/libexec/git-core/git --shallow-file\nmyfilename index-pack --stdin --keep='receive-pack 159567 on\nservername' --show-resolving-progress --report-end-of-input --fix-thin\n--strict\n07:48:20.781099 common-main.c:48                  version 2.29.2\n07:48:20.781111 common-main.c:48             | d0 | main\n      | version      |     |           |           |              |\n2.29.2\n07:48:20.781127 common-main.c:49                  start\n/opt/sno/libexec/git-core/git --shallow-file indexed.pack index-pack\n--stdin '--keep=receive-pack 159567 on cave-7dc7798cc9-qcvxd'\n--show-resolving-progress --report-end-of-input --fix-thin --strict\n07:48:20.781133 common-main.c:49             | d0 | main\n      | start        |     |  0.000264 |           |              |\n/opt/sno/libexec/git-core/git --shallow-file indexed.pack index-pack\n--stdin '--keep=receive-pack 159567 on cave-7dc7798cc9-qcvxd'\n--show-resolving-progress --report-end-of-input --fix-thin --strict\n07:48:20.781296 git.c:444               trace: built-in: git\nindex-pack --stdin '--keep=receive-pack 159567 on\ncave-7dc7798cc9-qcvxd' --show-resolving-progress --report-end-of-input\n--fix-thin --strict\n07:48:20.781306 git.c:445                         cmd_name index-pack\n(index-pack)\n07:48:20.781312 git.c:445                    | d0 | main\n      | cmd_name     |     |           |           |              |\nindex-pack (index-pack)\n07:48:20.781530 midx.c:184                   | d0 | main\n      | data         | r0  |  0.000670 |  0.000670 | midx         |\nload/num_packs:1\n07:48:20.781542 midx.c:185                   | d0 | main\n      | data         | r0  |  0.000683 |  0.000683 | midx         |\nload/num_objects:42658742\npack    5aa14bbb43187b7dfd5f996514854c3dcdc66d71\n08:27:33.724306 git.c:700                         exit\nelapsed:2352.943441 code:0\n08:27:33.724321 git.c:700                    | d0 | main\n      | exit         |     | 2352.943441 |           |              |\ncode:0\n08:27:33.724336 trace2/tr2_tgt_normal.c:123       atexit\nelapsed:2352.943475 code:0\n08:27:33.724341 trace2/tr2_tgt_perf.c:213    | d0 | main\n      | atexit       |     | 2352.943475 |           |              |\ncode:0\n```\n\nRemoving the `--strict` from the invocation by disabling\n`transfer.fsckObjects` solves the problem - the process completes in less\nthan a minute, and uses less than a GB of read IO.\n\nI can theorise why this operation is slightly expensive:\n\n   - `--strict` causes `index-pack` to call `fsck_object()` on each object\n   pushed\n   - these large pushes that push 100K+ blobs actually touch almost every\n   *tree* as well - so most/all of the 65K trees are pushed too.\n   - calling `fsck_object` on a tree looks up all its children (blobs and\n   trees) to ensure they're reachable [2]\n\nWhat I can't understand is why that makes it take quite *so* much longer\nand use so much IO. I think it *should* probably not be checking much about\nobjects that are already in the repo, other than that they exist. We\nhave multi-pack indexes enabled, so my assumption is that a \"does\nobject xyz exist?\" check should be very inexpensive.\nWhat could I be missing here?\n\nAs a start of a possible theory, we found when using libgit2 that our\npeculiar repo structure with so many trees requires that we expand the size\nof the tree cache[3] - otherwise repeated operations on blobs would\ncause tree cache misses\nevery time their path was traversed. I wonder if there is a similar tree\ncache structure in git itself, and if so could it be relevant here?\n\nMany thanks and sorry about the long winded post :)\n\nCraig de Stigter\nPlatform Engineer\nKoordinates\n\n\nreferences:\n[1]: https://kartproject.org\n[2]: fsck_walk_tree:\nhttps://github.com/git/git/blob/a0dda6023ed82b927fa205c474654699a5b07a82/fsck.c#L300\n[3] GIT_OPT_SET_CACHE_OBJECT_LIMIT:\nhttps://github.com/libgit2/libgit2/blob/508361401fbb5d87118045eaeae3356a729131aa/include/git2/common.h#L266-L272\n"}]}