Re: Resolving deltas dominates clone time
- From
- Martin Fick <mfick@codeaurora.org>
- Date
- Apr 23, 2019, 20:09 UTC
- Message-ID
- <3329645.KIYB9vJKXd@mfick-lnx>
- In-Reply-To
- <CACsJy8B7tjjpUZK+zH4rvOSk=uTLOHCOy6hk4FkkHXqCzNZU9g@mail.gmail.com>
On Tuesday, April 23, 2019 5:08:40 PM MDT Duy Nguyen wrote:
Show 34 quoted lines
> On Tue, Apr 23, 2019 at 11:45 AM Jeff King <peff@peff.net> wrote:
> > On Mon, Apr 22, 2019 at 09:55:38PM -0400, Jeff King wrote:
> > > Here are my p5302 numbers on linux.git, by the way.
> > >
> > > Test jk/p5302-repeat-fix
> > > ------------------------------------------------------------------
> > > 5302.2: index-pack 0 threads 307.04(303.74+3.30)
> > > 5302.3: index-pack 1 thread 309.74(306.13+3.56)
> > > 5302.4: index-pack 2 threads 177.89(313.73+3.60)
> > > 5302.5: index-pack 4 threads 117.14(344.07+4.29)
> > > 5302.6: index-pack 8 threads 112.40(607.12+5.80)
> > > 5302.7: index-pack default number of threads 135.00(322.03+3.74)
> > >
> > > which still imply that "4" is a win over "3" ("8" is slightly better
> > > still in wall-clock time, but the total CPU rises dramatically; that's
> > > probably because this is a quad-core with hyperthreading, so by that
> > > point we're just throttling down the CPUs).
> >
> > And here's a similar test run on a 20-core Xeon w/ hyperthreading (I
> > tweaked the test to keep going after eight threads):
> >
> > Test HEAD
> > ----------------------------------------------------
> > 5302.2: index-pack 1 threads 376.88(364.50+11.52)
> > 5302.3: index-pack 2 threads 228.13(371.21+17.86)
> > 5302.4: index-pack 4 threads 151.41(387.06+21.12)
> > 5302.5: index-pack 8 threads 113.68(413.40+25.80)
> > 5302.6: index-pack 16 threads 100.60(511.85+37.53)
> > 5302.7: index-pack 32 threads 94.43(623.82+45.70)
> > 5302.8: index-pack 40 threads 93.64(702.88+47.61)
> >
> > I don't think any of this is _particularly_ relevant to your case, but
> > it really seems to me that the default of capping at 3 threads is too
> > low.Here are my index-pack results (I only ran them once since they take a while) using vgit 1.8.3.2:
Threads real user sys 1 108m46.151s 106m14.420s 1m57.192s 2 58m14.274s 106m23.158s 5m32.736s 3 40m33.351s 106m42.281s 5m40.884s 4 31m40.342s 107m20.278s 5m40.675s 5 26m0.454s 106m54.370s 5m35.827s 12 13m25.304s 107m57.271s 6m26.493s 16 10m56.866s 107m46.107s 6m41.330s 18 10m18.112s 109m50.893s 7m1.369s 20 9m54.010s 113m51.028s 7m53.082s 24 9m1.104s 115m8.245s 7m57.156s 28 8m26.058s 116m46.311s 8m34.752s 32 8m42.967s 140m33.280s 9m59.514s 36 8m52.228s 151m28.939s 11m55.590s 40 8m22.719s 153m4.496s 12m36.041s 44 8m12.419s 166m41.594s 14m7.717s 48 8m0.377s 172m3.597s 16m32.041s 56 8m22.320s 188m31.426s 17m48.274s
> Looking back at the multithread commit, I think the trend was the same > and I capped it because the gain was not proportional to the number of > cores we threw at index-pack anymore. I would not be opposed to > raising the cap though (or maybe just remove it)
I think that if there were no default limit during a clone it could have disastrous effects on people using the repo tool from the android project, or any other "submodule like" tool that might clone many projects in parallel. With the repo tool, people often use a large -j number such as 24 which means they end up cloning around 24 projects at a time, and they may do this for around 1000 projects. If git clone suddenly started as many threads as there are CPUs for each clone, this would likely paralyze the machine.
I do suspect it would be nice to have a switch though that repo could use to adjust this intelligently, is there some way to adjust threads from a clone, I don't see one? I tried using 'GIT_FORCE_THREADS=28 git clone ...' and it didn't seem to make a difference?
-Martin
-- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation