git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Git push performance problems with ~100K refs

From
MFMartin Fick <mfick@codeaurora.org>
Date
Mar 30, 2012, 00:18 UTC
Message-ID
<201203291818.49933.mfick@codeaurora.org>
Hello,

I am back to talk about git performance with lots of refs: ~100K. This time I am investigating pushes to a repo with about ~100K ref, a gerrit mirror which includes the gerrit changes. When I push just a simple one line change to a file, the push takes about ~43s. If I delete the changes from the destination repo, this push takes about 6s. This seems rather excessive to me, in fact given that the repo with 100K refs has more data in it and is more likely to have the objects I am pushing in it, if things are done right, it should be a faster push (in theory).

So, upon early investigation, I noticed that the time to push seems mostly determined by the receiving end which is processing all out for 100% on a CPU. During this time period, the receiving end git commands look like this:

  git-receive-pack path/to/repo.git
and:
  git rev-list --objects --stdin --not --all
The latter of these two commands is the one burning CPU.

Does anyone have any hints as to what might be wrong with the receiving end algorithm that would cause a small change to use so much CPU? Is there anything that can be done about it? I noticed that the --all option will effectively feed all the 100K refs to rev-list, is this really necessary? Are there any tests that I can perform to help debug this?

I am using git 1.7.8.3 and I also tried 1.7.10.rc3, same results.

Thanks,
-Martin
-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum
Next: Junio C Hamano
Message 1 of 37 in “Git push performance problems with ~100K refs”
  1. Martin FickMar 30, 2012
  2. Junio C HamanoMar 30, 2012
  3. Martin FickMar 30, 2012
  4. Jeff KingMar 30, 2012
  5. Jeff KingMar 30, 2012
  6. Martin FickMar 30, 2012
  7. 1/3 add mergesort() for linked listsRené Scharfe, Mar 31, 2012
  8. Junio C HamanoApr 5, 2012
  9. René ScharfeApr 8, 2012
  10. Junio C HamanoApr 9, 2012
  11. Stephen BoydApr 11, 2012
  12. Junio C HamanoApr 11, 2012
  13. 2/3 commit: use mergesort() in commit_list_sort_by_date()René Scharfe, Mar 31, 2012
  14. 3/3 revision: insert unsorted, then sort in prepare_revision_walk()René Scharfe, Mar 31, 2012
  15. Martin FickMar 31, 2012
  16. Junio C HamanoMar 31, 2012
  17. Martin FickApr 2, 2012
  18. Shawn PearceApr 2, 2012
  19. Martin FickApr 2, 2012
  20. Shawn PearceApr 2, 2012
  21. Jeff KingApr 2, 2012
  22. Jeff KingApr 2, 2012
  23. Martin FickApr 2, 2012
  24. Nguyen Thai Ngoc DuyApr 3, 2012
  25. Martin FickApr 3, 2012
  26. 0/3 Commit cacheNguyễn Thái Ngọc Duy, Apr 3, 2012
  27. 1/3 parse_commit_buffer: rename a confusing variable nameNguyễn Thái Ngọc Duy, Apr 3, 2012
  28. 2/3 Add commit cache to help speed up commit traversalNguyễn Thái Ngọc Duy, Apr 3, 2012
  29. 3/3 Add parse_commit_for_rev() to take advantage of sha1-cacheNguyễn Thái Ngọc Duy, Apr 3, 2012
  30. Nguyen Thai Ngoc DuyApr 5, 2012
  31. Shawn PearceApr 6, 2012
  32. Nguyen Thai Ngoc DuyApr 7, 2012
  33. Nguyen Thai Ngoc DuyApr 3, 2012
  34. Jeff KingApr 2, 2012
  35. René ScharfeApr 2, 2012
  36. Jeff KingApr 3, 2012
  37. Jeff KingApr 3, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.