git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Make the git codebase thread-safe

From
SZStefan Zager <szager@chromium.org>
Date
Feb 12, 2014, 18:12 UTC
Message-ID
<CAHOQ7J8gvwpwJV2mBPDaARu3cQ54-ZDQ6iGOwKuJRr9Z+XBL7g@mail.gmail.com>
In-Reply-To
<CACsJy8Bsc6sywL9L5QC-SKKmh9J+CKnoG5i78WfUbAG9BdZ8Rw@mail.gmail.com>
On Tue, Feb 11, 2014 at 6:11 PM, Duy Nguyen <pclouds@gmail.com> wrote:
Show 5 quoted lines
>
> I have no comments about thread safety improvements (well, not yet).
> If you have investigated about git performance on chromium
> repositories, could you please sum it up? Threading may be an option
> to improve performance, but it's probably not the only option.

Well, the painful operations that we use frequently are pack-objects, checkout, status, and blame. Anything on Windows that touches a lot of files is miserable due to the usual file system slowness on Windows, and luafv.sys (the UAC file virtualization driver) seems to make it much worse.

With threading turned on, pack-objects on Windows now takes about twice as long as on Linux, which is still more than a 2x improvement over the non-threaded operation.

Checkout is really bad on Windows. The blink repository is ~200K files, and a full clean checkout from the index takes about 10 seconds on Linux, and about 3:30 on Windows. I used the Very Sleepy profiler to see where all the time was spent on Windows: 55% of the time was spent in OpenFile, and 25% in CloseFile (both in win32). My immediate goal is to add threading to checkout, so those file system calls can be done in parallel.

Enabling the fscache speeds up status quite a bit. I'm optimistic that parallelizing the stat calls will yield a further improvement. Beyond that, it may not be possible to do much more without using a file system watcher daemon, like facebook does with mercurial. (https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/)

Blame is something that chromium and blink developers use heavily, and it is not unusual for a blame invocation on the blink repository to run for 30 seconds. It seems like it should be possible to parallelize blame, but it requires pack file operations to be thread-safe.

Stefan
Previous: Duy NguyenNext: Matthieu Moy
Message 9 of 55 in “Make the git codebase thread-safe”
  1. Stefan ZagerFeb 12, 2014
  2. Robin H. JohnsonFeb 12, 2014
  3. Duy NguyenFeb 12, 2014
  4. Karsten BleesFeb 12, 2014
  5. Mike HommeyFeb 12, 2014
  6. Karsten BleesFeb 13, 2014
  7. Stefan ZagerFeb 12, 2014
  8. Duy NguyenFeb 12, 2014
  9. Stefan ZagerFeb 12, 2014
  10. Matthieu MoyFeb 12, 2014
  11. Stefan ZagerFeb 12, 2014
  12. David KastrupFeb 12, 2014
  13. Stefan ZagerFeb 12, 2014
  14. David KastrupFeb 12, 2014
  15. Mike HommeyFeb 12, 2014
  16. David KastrupFeb 13, 2014
  17. Mike HommeyFeb 13, 2014
  18. Mike HommeyFeb 13, 2014
  19. David KastrupFeb 13, 2014
  20. Junio C HamanoFeb 12, 2014
  21. Stefan ZagerFeb 12, 2014
  22. Junio C HamanoFeb 12, 2014
  23. Erik Faye-LundFeb 12, 2014
  24. Stefan ZagerFeb 12, 2014
  25. Erik Faye-LundFeb 12, 2014
  26. Stefan ZagerFeb 12, 2014
  27. Erik Faye-LundFeb 12, 2014
  28. Karsten BleesFeb 12, 2014
  29. Stefan ZagerFeb 12, 2014
  30. Johannes SixtFeb 13, 2014
  31. David KastrupFeb 13, 2014
  32. Stefan ZagerFeb 13, 2014
  33. Zachary TurnerFeb 13, 2014
  34. Karsten BleesFeb 13, 2014
  35. Stefan ZagerFeb 13, 2014
  36. Zachary TurnerFeb 13, 2014
  37. Karsten BleesFeb 14, 2014
  38. Stefan ZagerFeb 14, 2014
  39. brian m. carlsonFeb 13, 2014
  40. Matheus TavaresApr 2, 2019
  41. Duy NguyenApr 2, 2019
  42. David KastrupApr 2, 2019
  43. Duy NguyenApr 2, 2019
  44. David KastrupApr 2, 2019
  45. Matheus Tavares BernardinoApr 2, 2019
  46. Zachary TurnerFeb 14, 2014
  47. Karsten BleesFeb 14, 2014
  48. Duy NguyenFeb 15, 2014
  49. Stefan ZagerFeb 15, 2014
  50. Duy NguyenFeb 15, 2014
  51. Zachary TurnerFeb 15, 2014
  52. Duy NguyenFeb 15, 2014
  53. Junio C HamanoFeb 18, 2014
  54. Zachary TurnerFeb 18, 2014
  55. Stefan ZagerFeb 14, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.