git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv5 00/14] git notes

From
Johan Herland <johan@herland.net>
Date
Sep 12, 2009, 18:35 UTC
Message-ID
<200909122035.47051.johan@herland.net>
In-Reply-To
<20090912181150.GN1033@spearce.org>
On Saturday 12 September 2009, Shawn O. Pearce wrote:
Show 13 quoted lines
> Johan Herland <johan@herland.net> wrote:
> > Shawn, do you have any additional defence for the date-based fanout?
> 
> No.
> 
> The only defense I have for it is "it sounds like a nice theory
> given access patterns", and the note about memory usage you made,
> but which I clipped to keep this email shorter. :-)
> 
> It was only a theory I tossed out there in a back-seat-driver
> sort of way.  Your results show my hunch was correct, it may help.
> But they also say it may not help enough to justify the complexity,
> so I now agree with you that SHA-1 fan out may be good enough.

Ok, so I guess we can drop the flexible part of notes code. Junio: Feel free to drop the two last patches from the jh/notes series.

Show 11 quoted lines
> > How does the plan for notes usage in your code-review thingy
> > compare to my test scenario?
> 
> I think your tests may still have been too low in volume, 115k notes
> isn't a lot.  Based on the distributions I was looking at before,
> I could be seeing a growth of >100k notes/year.  Ask me again in
> 5 years if 115k notes is a lot. :-)
> 
> But we all know that SHA-1 distributes data quite well, so the SHA-1
> fan-out may just need to change from 2_38 to 2_2_2_34 (or something)
> to handle that larger volume.

Yes, I expect that the optimal number of entries per tree level is ~256, so if we add an upper threshold at ~300 (where we start using another fanout level), and a lower threshold at ~200 (where we consolidate subtrees and put all into this level), the (still-to-be-written) writing part of the notes code should automatically adjust the notes tree to the optimal layout.

With those assumptions, and a growth of 100k notes/year, a 2/2/36 fanout should last you ~150 years, and a 2/2/2/34 fanout should be enough for the next ~40,000 years... ;)

Have fun! :)
...Johan
-- 
Johan Herland, <johan@herland.net>
www.herland.net
Previous: Shawn O. PearceNext: Geert Bosch
Message 33 of 58 in “[PATCHv5 00/14] git notes”
  1. Johan HerlandSep 8, 2009
  2. 01/14 Introduce commit notesJohan Herland, Sep 8, 2009
  3. 02/14 Add a script to edit/inspect notesJohan Herland, Sep 8, 2009
  4. 03/14 Speed up git notes lookupJohan Herland, Sep 8, 2009
  5. 04/14 Add an expensive test for git-notesJohan Herland, Sep 8, 2009
  6. 05/14 Teach "-m <msg>" and "-F <file>" to "git notes edit"Johan Herland, Sep 8, 2009
  7. 06/14 fast-import: Add support for importing commit notesJohan Herland, Sep 8, 2009
  8. 07/14 t3302-notes-index-expensive: Speed up create_repo()Johan Herland, Sep 8, 2009
  9. 08/14 Add flags to get_commit_notes() to control the format of the note stringJohan Herland, Sep 8, 2009
  10. 09/14 Add '%N'-format for pretty-printing commit notesJohan Herland, Sep 8, 2009
  11. 10/14 Teach notes code to free its internal data structures on request.Johan Herland, Sep 8, 2009
  12. 11/14 Teach the notes lookup code to parse notes trees with various fanout schemesJohan Herland, Sep 8, 2009
  13. 12/14 Selftests verifying semantics when loading notes trees with various fanoutsJohan Herland, Sep 8, 2009
  14. 13/14 Allow flexible organization of notes trees, using both commit date and SHA1Johan Herland, Sep 8, 2009
  15. 14/14 Add test cases for date-based fanoutsJohan Herland, Sep 8, 2009
  16. Johan HerlandSep 8, 2009
  17. Junio C HamanoSep 8, 2009
  18. Johan HerlandSep 8, 2009
  19. Johannes SchindelinSep 8, 2009
  20. Johan HerlandSep 8, 2009
  21. Johannes SchindelinSep 8, 2009
  22. Johan HerlandSep 8, 2009
  23. Stephen R. van den BergSep 10, 2009
  24. Johan HerlandSep 10, 2009
  25. Junio C HamanoSep 8, 2009
  26. Shawn O. PearceSep 8, 2009
  27. Sverre RabbelierSep 8, 2009
  28. Shawn O. PearceSep 8, 2009
  29. Sverre RabbelierSep 8, 2009
  30. Johan HerlandSep 8, 2009
  31. Johan HerlandSep 12, 2009
  32. Shawn O. PearceSep 12, 2009
  33. Johan HerlandSep 12, 2009
  34. Geert BoschSep 10, 2009
  35. Michael J GruberSep 10, 2009
  36. Geert BoschSep 10, 2009
  37. Junio C HamanoSep 12, 2009
  38. Johan HerlandSep 12, 2009
  39. 00/14 git notesJohan Herland, Sep 12, 2009
  40. 01/14 Introduce commit notesJohan Herland, Sep 12, 2009
  41. 02/14 Add a script to edit/inspect notesJohan Herland, Sep 12, 2009
  42. 03/14 Speed up git notes lookupJohan Herland, Sep 12, 2009
  43. 04/14 Add an expensive test for git-notesJohan Herland, Sep 12, 2009
  44. 05/14 Teach "-m <msg>" and "-F <file>" to "git notes edit"Johan Herland, Sep 12, 2009
  45. 06/14 fast-import: Add support for importing commit notesJohan Herland, Sep 12, 2009
  46. 07/14 t3302-notes-index-expensive: Speed up create_repo()Johan Herland, Sep 12, 2009
  47. 08/14 Add flags to get_commit_notes() to control the format of the note stringJohan Herland, Sep 12, 2009
  48. 09/14 Add '%N'-format for pretty-printing commit notesJohan Herland, Sep 12, 2009
  49. 10/14 Teach notes code to free its internal data structures on request.Johan Herland, Sep 12, 2009
  50. Junio C HamanoSep 12, 2009
  51. Johan HerlandSep 12, 2009
  52. 11/14 Teach the notes lookup code to parse notes trees with various fanout schemesJohan Herland, Sep 12, 2009
  53. 12/14 Selftests verifying semantics when loading notes trees with various fanoutsJohan Herland, Sep 12, 2009
  54. 13/14 Allow flexible organization of notes trees, using both commit date and SHA1Johan Herland, Sep 12, 2009
  55. Junio C HamanoSep 12, 2009
  56. Johan HerlandSep 12, 2009
  57. Junio C HamanoSep 12, 2009
  58. 14/14 Add test cases for various date-based fanoutsJohan Herland, Sep 12, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.