git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Git is not scalable with too many refs/*

From
Julian Phillips <julian@quantumfyre.co.uk>
Date
Sep 27, 2011, 07:59 UTC
Message-ID
<2ad152520590530dd1bdcfae9941ef9d@quantumfyre.co.uk>
In-Reply-To
<3539dab7-0fb2-4759-baaf-8e22efab2904@email.android.com>
On Mon, 26 Sep 2011 20:34:02 -0600, Martin Fick wrote:
Show 22 quoted lines
>> Julian Phillips <julian@quantumfyre.co.uk> wrote:
>>On Mon, 26 Sep 2011 18:12:31 -0600, Martin Fick wrote:
>>That sounds a lot better.  Hopefully other commands should be faster
>>now too.
>
> Yeah, I will try this in a few other places to see.
>
>>> Thanks way much!!!
>>
>>No problem.  Thank you for all the time you've put in to help chase
>>this down.  Makes it so much easier when the person with original
>>problem mucks in with the investigation.
>>Just think how much time you've saved for anyone with a large number 
>> of
>>
>>those Gerrit change refs ;)
>
>  Perhaps this is a naive question, but why are all these refs being
> put into a list to be sorted, only to be discarded soon thereafter
> anyway?  After all, git branch knows that it isn't going to print
> these, and the refs are stored precategorized, so why not only grab
> the refs which matter upfront?

I can't say that I am aware of a specific decision having been taken on the subject, but I'll have a guess at the reason:

The extra code it would take to have an API for getting a list of only a subset of the refs has never been considered worth the cost. It would take effort to implement, test and maintain - and it would have to be done separately for packed and unpacked cases to avoid still loading and discarding unwanted refs. All that to not do something that no-one has noticed taking any time? Until now, I doubt anyone has considered it something that was a problem - and now that even with 100k refs it takes less than a second, I doubt anyone will feel all that inclined to have a crack at it now either.

-- 
Julian
Previous: Martin FickNext: Sverre Rabbelier
Message 106 of 126 in “Git is not scalable with too many refs/*”
  1. NAKAMURA TakumiJun 9, 2011
  2. Sverre RabbelierJun 9, 2011
  3. Shawn PearceJun 9, 2011
  4. A Large Angry SCMJun 9, 2011
  5. Shawn PearceJun 9, 2011
  6. Jeff KingJun 9, 2011
  7. NAKAMURA TakumiJun 10, 2011
  8. Jeff KingJun 13, 2011
  9. Andreas EricssonJun 14, 2011
  10. Jeff KingJun 14, 2011
  11. Junio C HamanoJun 14, 2011
  12. Sverre RabbelierJun 14, 2011
  13. Johan HerlandJun 14, 2011
  14. Sverre RabbelierJun 14, 2011
  15. Jeff KingJun 14, 2011
  16. Shawn PearceJun 14, 2011
  17. Jeff KingJun 14, 2011
  18. Shawn PearceJun 14, 2011
  19. Martin FickSep 8, 2011
  20. Martin FickSep 9, 2011
  21. Thomas RastSep 9, 2011
  22. Thomas RastSep 9, 2011
  23. Jens LehmannSep 9, 2011
  24. Martin FickSep 25, 2011
  25. Christian CouderSep 26, 2011
  26. Martin FickSep 26, 2011
  27. Christian CouderSep 26, 2011
  28. Martin FickSep 30, 2011
  29. Martin FickSep 30, 2011
  30. Martin FickSep 30, 2011
  31. Martin FickSep 30, 2011
  32. Junio C HamanoOct 1, 2011
  33. Michael HaggertyOct 2, 2011
  34. Martin FickOct 3, 2011
  35. Michael HaggertyOct 4, 2011
  36. Martin FickOct 3, 2011
  37. Junio C HamanoOct 3, 2011
  38. Michael HaggertyOct 4, 2011
  39. Martin FickOct 8, 2011
  40. Michael HaggertyOct 9, 2011
  41. Martin FickSep 28, 2011
  42. Martin FickSep 28, 2011
  43. Julian PhillipsSep 29, 2011
  44. Martin FickSep 29, 2011
  45. Julian PhillipsSep 29, 2011
  46. Martin FickSep 29, 2011
  47. Julian PhillipsSep 29, 2011
  48. René ScharfeSep 29, 2011
  49. Junio C HamanoSep 29, 2011
  50. refs: Use binary search to lookup refs fasterJulian Phillips, Sep 29, 2011
  51. Junio C HamanoSep 29, 2011
  52. refs: Use binary search to lookup refs fasterJulian Phillips, Sep 29, 2011
  53. Junio C HamanoSep 29, 2011
  54. refs: Use binary search to lookup refs fasterJulian Phillips, Sep 29, 2011
  55. Junio C HamanoSep 29, 2011
  56. Michael HaggertySep 30, 2011
  57. Junio C HamanoSep 30, 2011
  58. refs: Remove duplicates after sorting with qsortJulian Phillips, Sep 30, 2011
  59. Michael HaggertyOct 2, 2011
  60. Junio C HamanoOct 2, 2011
  61. Junio C HamanoOct 4, 2011
  62. Martin FickSep 30, 2011
  63. Junio C HamanoSep 30, 2011
  64. Julian PhillipsSep 30, 2011
  65. Martin FickSep 30, 2011
  66. Martin FickSep 29, 2011
  67. Julian PhillipsSep 29, 2011
  68. Martin FickSep 29, 2011
  69. René ScharfeSep 30, 2011
  70. Martin FickSep 30, 2011
  71. Junio C HamanoSep 30, 2011
  72. René ScharfeSep 30, 2011
  73. René ScharfeOct 1, 2011
  74. 1/8 checkout: check for "Previous HEAD" notice in t2020René Scharfe, Oct 1, 2011
  75. Sverre RabbelierOct 1, 2011
  76. 2/8 revision: factor out add_pending_sha1René Scharfe, Oct 1, 2011
  77. 3/8 checkout: use add_pending_{object,sha1} in orphan checkRené Scharfe, Oct 1, 2011
  78. 4/8 revision: add leak_pending flagRené Scharfe, Oct 1, 2011
  79. 5/8 bisect: use leak_pending flagRené Scharfe, Oct 1, 2011
  80. 6/8 bundle: use leak_pending flagRené Scharfe, Oct 1, 2011
  81. 7/8 checkout: use leak_pending flagRené Scharfe, Oct 1, 2011
  82. 8/8 commit: factor out clear_commit_marks_for_object_arrayRené Scharfe, Oct 1, 2011
  83. Martin FickSep 26, 2011
  84. Sverre RabbelierSep 26, 2011
  85. Martin FickSep 26, 2011
  86. Sverre RabbelierSep 26, 2011
  87. Martin FickSep 26, 2011
  88. Julian PhillipsSep 26, 2011
  89. Martin FickSep 26, 2011
  90. Julian PhillipsSep 26, 2011
  91. Martin FickSep 26, 2011
  92. Junio C HamanoSep 26, 2011
  93. Julian PhillipsSep 26, 2011
  94. Martin FickSep 26, 2011
  95. Martin FickSep 26, 2011
  96. Julian PhillipsSep 26, 2011
  97. David Michael BarrSep 26, 2011
  98. refs.c: Fix slowness with numerous loose refsDavid Barr, Sep 27, 2011
  99. David Michael BarrSep 27, 2011
  100. Junio C HamanoSep 26, 2011
  101. Don't sort ref_list too earlyJulian Phillips, Sep 27, 2011
  102. Michael HaggertyOct 2, 2011
  103. Martin FickSep 27, 2011
  104. Julian PhillipsSep 27, 2011
  105. Martin FickSep 27, 2011
  106. Julian PhillipsSep 27, 2011
  107. Sverre RabbelierSep 27, 2011
  108. Julian PhillipsSep 27, 2011
  109. Sverre RabbelierSep 27, 2011
  110. Nguyen Thai Ngoc DuySep 27, 2011
  111. Michael HaggertySep 27, 2011
  112. Julian PhillipsSep 27, 2011
  113. Julian PhillipsSep 26, 2011
  114. Michael HaggertySep 26, 2011
  115. Martin FickSep 26, 2011
  116. Thomas RastSep 26, 2011
  117. Michael HaggertySep 9, 2011
  118. Michael HaggertySep 9, 2011
  119. Jens LehmannSep 9, 2011
  120. Andreas EricssonJun 10, 2011
  121. Shawn PearceJun 10, 2011
  122. Jakub NarebskiJun 10, 2011
  123. Jeff KingJun 10, 2011
  124. Andreas EricssonJun 13, 2011
  125. Jakub NarebskiJun 9, 2011
  126. Stephen BashJun 9, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.