git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Poor push performance with large number of refs

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
Dec 10, 2014, 23:34 UTC
Message-ID
<20141210233443.GA130424@vauxhall.crustytoothpaste.net>
In-Reply-To
<CAJo=hJvKBvQvN=EV4y=ACz5pou9A0tD+txAn_8VR9L3KKtQSiA@mail.gmail.com>
On Tue, Dec 09, 2014 at 09:41:28PM -0800, Shawn Pearce wrote:
Show 16 quoted lines
> On Tue, Dec 9, 2014 at 4:37 PM, brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > Most of the time is spent between the "Pushing to remote machine" and
> > "Counting objects", running git pack-objects:
> >
> >   git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
> >
> > Unfortunately, -vvv doesn't provide any helpful output.  I have some
> > suspicions what's going on here, but no hard data.  Where should I
> > be looking to determine the bottleneck?
> 
> My guess is the revision queue is struggling to insert 20,000 commits
> that the remote side "has", are uninteresting, and should not be
> transmitted. This queue insertion usually requires parsing the commit
> object out of the local object store to get the commit timestamp, then
> bubble sort inserting that commit into the queue.

I looked at this more in depth today and I found that the bottleneck is --thin. I tried git send-pack, which does not use --thin by default, which led me to further testing. A particular push went from 24 seconds with --thin to 4 seconds without.

I agree that the large number of refs is at least part of the problem, because reducing the number of refs has a slight but noticeable impact. It's also the factor I can least control.

I have a patch which allows per-remote configuration of whether to use thin packs (which I will send shortly), but I'm wondering if we can do better, especially since --thin is the default. It looks like --thin forces pack-objects to do its own lookup (essentially a rev-list) instead of using the values provided on stdin.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
Previous: Shawn PearceNext: brian m. carlson
Message 3 of 14 in “Poor push performance with large number of refs”
  1. brian m. carlsonDec 10, 2014
  2. Shawn PearceDec 10, 2014
  3. brian m. carlsonDec 10, 2014
  4. push: add remote.pushThin to control thin pack generationbrian m. carlson, Dec 10, 2014
  5. brian m. carlsonDec 11, 2014
  6. Duy NguyenDec 11, 2014
  7. brian m. carlsonDec 11, 2014
  8. list-objects: mark fewer commits as edges for non-shallow clonesbrian m. carlson, Dec 11, 2014
  9. Duy NguyenDec 11, 2014
  10. brian m. carlsonDec 20, 2014
  11. Junio C HamanoDec 11, 2014
  12. Junio C HamanoDec 11, 2014
  13. brian m. carlsonDec 20, 2014
  14. Duy NguyenDec 10, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.