Partitioned packs

I've been running some experiments, as hinted earlier by the discussion about just how much git-index-pack sucks (which, really, isn't much since the gaping memleak is gone now).

These experiments include trying to see if there's a noticeable performance improvement by splitting out objects of different types into different packs. So far, it definitely seems to make a difference, though not the one I was initially expecting. For all of these tests, I did 'sysctl -w vm.drop_caches=3' before running, to effectively simulate a cold-cache run.

Single 3.1GB pack file containing all commits, blobs, and trees First run (cold cache): git-rev-list --all > /dev/null 5.52s user 0.32s system 45% cpu 12.872 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.01s system 0% cpu 40.218s total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.48s user 0.10s system 5% cpu 10.143 total

Subsequent runs (warm cache): git-rev-list --all > /dev/null 5.19s user 0.48s system 98% cpu 5.750 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.00s system 0% cpu 11.960 total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.43s user 0.04s system 100% cpu 0.472 total

Single pack for commit objects and another pack for the rest First run (cold cache): git-rev-list --all > /dev/null 5.84s user 0.34s system 31% cpu 19.427 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.00s system 0% cpu 9:42.74 total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.50s user 0.26s system 0% cpu 1:35.44 total

Subsequent runs (warm cache): git-rev-list --all > /dev/null 5.94s user 0.26s system 99% cpu 6.204 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.00s system 0% cpu 12.394 total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.41s user 0.07s system 98% cpu 0.486 total

Fully-partitioned separate packs for commit, tree, and blob objects First run (cold cache): git-rev-list --all > /dev/null 6.24s user 0.32s system 25% cpu 25.689 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.00s system 0% cpu 1:08.76 total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.38s user 0.30s system 0% cpu 1:35.89 total

Subsequent runs (warm cache): git-rev-list --all > /dev/null 6.28s user 0.24s system 99% cpu 6.527 total git-blame -- kdelibs/kdeui/kmenubar.cpp 0.00s user 0.00s system 0% cpu 13.895 total git-archive --format=tar HEAD -- kdelibs >> /dev/null 0.42s user 0.06s system 99% cpu 0.476 total

I packed all of these using --delta-base-offset, with a window of 100 and a depth of 10.

-clee