git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2] add test to demonstrate that shallow recursive clones fail

From
Jeff King <peff@peff.net>
Date
Nov 19, 2015, 13:58 UTC
Message-ID
<20151119135808.GA9353@sigill.intra.peff.net>
In-Reply-To
<CAOE36qj2m4e3hw73-QoLbbpGv4RiyhBt_ou7eN4i4q8pF15rdA@mail.gmail.com>
On Wed, Nov 18, 2015 at 03:40:02PM -0800, Terry Parker wrote:
Show 14 quoted lines
> > +Terry, who did optimize the JGit implementation for bitmaps,
> > as we also had a "lots of refs" hoarder repo, which underperformed
> > before.
> 
> The performance issue with the "hoarder" repo was that the bitmap
> commit selection algorithm in JGit was selecting too many commits and
> the hoarder repo caused OOMs.
> 
> JGit's selection algorithm walks each branch and makes sure commits
> are selected every 100-200 commits for "recent" history and every
> 5000-5100 commits for more distant history. IIUC cgit sorts all commits
> by date and does a similar selection. Since it doesn't attempt to get
> even coverage on each branch, the selected commits may have
> large gaps for a particular branch.

Thanks for the data point. I agree that cgit seems to have the opposite problem: selecting fewer commits, which gives worse performance at read time, but no OOM at write time. :)

I suspect there's some low-hanging fruit in tweaking cgit's implementation, where we might be able to do a better job of hitting near the tips of tags without having to drastically increase the number of bitmapped commits. But I haven't looked closely.

-Peff
Previous: Jeff King
Message 35 of 35 in “add test to demonstrate that shallow recursive clones fail”
  1. add test to demonstrate that shallow recursive clones faillarsxschneider@gmail.com, Nov 12, 2015
  2. Stefan BellerNov 12, 2015
  3. Lars SchneiderNov 15, 2015
  4. Jeff KingNov 13, 2015
  5. Stefan BellerNov 13, 2015
  6. Stefan BellerNov 13, 2015
  7. Jeff KingNov 13, 2015
  8. Jeff KingNov 13, 2015
  9. Stefan BellerNov 14, 2015
  10. Jens LehmannNov 16, 2015
  11. Stefan BellerNov 16, 2015
  12. Jens LehmannNov 16, 2015
  13. Stefan BellerNov 16, 2015
  14. Jens LehmannNov 17, 2015
  15. Stefan BellerNov 17, 2015
  16. Jens LehmannNov 17, 2015
  17. Stefan BellerNov 17, 2015
  18. Jens LehmannNov 17, 2015
  19. Duy NguyenNov 17, 2015
  20. Jeff KingNov 17, 2015
  21. Duy NguyenNov 18, 2015
  22. Jeff KingNov 18, 2015
  23. Stefan BellerNov 18, 2015
  24. Junio C HamanoNov 30, 2015
  25. Stefan BellerDec 1, 2015
  26. Duy NguyenDec 1, 2015
  27. Junio C HamanoDec 1, 2015
  28. Junio C HamanoDec 1, 2015
  29. Stefan BellerDec 3, 2015
  30. Junio C HamanoDec 4, 2015
  31. Junio C HamanoDec 4, 2015
  32. Duy NguyenDec 4, 2015
  33. Lars SchneiderNov 15, 2015
  34. Jeff KingNov 17, 2015
  35. Jeff KingNov 19, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.