git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 12/19] rev-list: add bitmap mode to speed up object lists

From
Jeff King <peff@peff.net>
Date
Oct 30, 2013, 08:12 UTC
Message-ID
<20131030081201.GL11317@sigill.intra.peff.net>
In-Reply-To
<CAJo=hJvKG2sjWasO4YfExbbSN+U-q_it6gb43v_YJoL60XKwUg@mail.gmail.com>
On Fri, Oct 25, 2013 at 02:00:48PM +0000, Shawn O. Pearce wrote:
Show 9 quoted lines
> On Thu, Oct 24, 2013 at 6:06 PM, Jeff King <peff@peff.net> wrote:
> > Note that most of the time we spend for --count invocations is on
> > generating the pack revindex. It may be worth storing a revidx (either
> > in a separate file, as part of the .idx, or as an optional section in
> > the .bitmap file).
> 
> This was discussed before, and it may make sense to cache the revidx.
> I'm glad to see the timings are already better with bitmaps, even
> though the revidx has to be computed on the fly for each invocation.

The radix sort makes a _huge_ difference there. But I would still like an mmap-able .revidx cache. I think we can get at least another order-of-magnitude speedup.

I'm intentionally leaving that out for now, though. I want to get this series merged before trying to go much further.

-Peff
Previous: Shawn PearceNext: Jeff King
Message 39 of 87 in “pack bitmaps”
  1. 0/19 pack bitmapsJeff King, Oct 24, 2013
  2. 01/19 sha1write: make buffer const-correctJeff King, Oct 24, 2013
  3. 02/19 revindex: Export new APIsJeff King, Oct 24, 2013
  4. 03/19 pack-objects: Refactor the packing listJeff King, Oct 24, 2013
  5. 04/19 pack-objects: factor out name_hashJeff King, Oct 24, 2013
  6. 05/19 revision: allow setting custom limiter functionJeff King, Oct 24, 2013
  7. 06/19 sha1_file: export `git_open_noatime`Jeff King, Oct 24, 2013
  8. 07/19 compat: add endianness helpersJeff King, Oct 24, 2013
  9. Thomas RastOct 26, 2013
  10. Jeff KingOct 30, 2013
  11. Vicent MartíOct 30, 2013
  12. 08/19 ewah: compressed bitmap implementationJeff King, Oct 24, 2013
  13. Junio C HamanoOct 24, 2013
  14. Jeff KingOct 25, 2013
  15. Thomas RastOct 26, 2013
  16. 09/19 documentation: add documentation for the bitmap formatJeff King, Oct 24, 2013
  17. Duy NguyenOct 25, 2013
  18. Jeff KingOct 25, 2013
  19. Duy NguyenOct 25, 2013
  20. Shawn PearceOct 25, 2013
  21. Jeff KingOct 30, 2013
  22. Shawn PearceOct 30, 2013
  23. Vicent MartiOct 30, 2013
  24. Vicent MartiOct 30, 2013
  25. 10/19 pack-bitmap: add support for bitmap indexesJeff King, Oct 24, 2013
  26. Shawn PearceOct 25, 2013
  27. Jeff KingOct 30, 2013
  28. Shawn PearceOct 30, 2013
  29. Vicent MartiOct 30, 2013
  30. Shawn PearceOct 30, 2013
  31. Jeff KingOct 30, 2013
  32. 11/19 pack-objects: use bitmaps when packing objectsJeff King, Oct 24, 2013
  33. Shawn PearceOct 25, 2013
  34. Jeff KingOct 30, 2013
  35. Shawn PearceOct 30, 2013
  36. Vicent MartiOct 30, 2013
  37. 12/19 rev-list: add bitmap mode to speed up object listsJeff King, Oct 24, 2013
  38. Shawn PearceOct 25, 2013
  39. Jeff KingOct 30, 2013
  40. 13/19 pack-objects: implement bitmap writingJeff King, Oct 24, 2013
  41. Duy NguyenOct 25, 2013
  42. Jeff KingOct 25, 2013
  43. 14/19 repack: stop using magic number for ARRAY_SIZE(exts)Jeff King, Oct 24, 2013
  44. 15/19 repack: turn exts array into array-of-structJeff King, Oct 24, 2013
  45. 16/19 repack: handle optional files created by pack-objectsJeff King, Oct 24, 2013
  46. 17/19 repack: consider bitmaps when performing repacksJeff King, Oct 24, 2013
  47. 18/19 t: add basic bitmap functionality testsJeff King, Oct 24, 2013
  48. 19/19 pack-bitmap: implement optional name_hash cacheJeff King, Oct 24, 2013
  49. Junio C HamanoOct 24, 2013
  50. Junio C HamanoOct 25, 2013
  51. 0/19 pack bitmapsJeff King, Oct 25, 2013
  52. 01/19 sha1write: make buffer const-correctJeff King, Oct 25, 2013
  53. 02/19 revindex: Export new APIsJeff King, Oct 25, 2013
  54. 03/19 pack-objects: Refactor the packing listJeff King, Oct 25, 2013
  55. 04/19 pack-objects: factor out name_hashJeff King, Oct 25, 2013
  56. 05/19 revision: allow setting custom limiter functionJeff King, Oct 25, 2013
  57. 06/19 sha1_file: export `git_open_noatime`Jeff King, Oct 25, 2013
  58. 07/19 compat: add endianness helpersJeff King, Oct 25, 2013
  59. 08/19 ewah: compressed bitmap implementationJeff King, Oct 25, 2013
  60. 09/19 documentation: add documentation for the bitmap formatJeff King, Oct 25, 2013
  61. 10/19 pack-bitmap: add support for bitmap indexesJeff King, Oct 25, 2013
  62. Junio C HamanoOct 25, 2013
  63. Jeff KingOct 26, 2013
  64. Jeff KingOct 26, 2013
  65. Junio C HamanoOct 28, 2013
  66. Jeff KingOct 30, 2013
  67. Duy NguyenOct 26, 2013
  68. Jeff KingOct 30, 2013
  69. 11/19 pack-objects: use bitmaps when packing objectsJeff King, Oct 25, 2013
  70. Duy NguyenOct 26, 2013
  71. Jeff KingOct 30, 2013
  72. Duy NguyenOct 30, 2013
  73. Jeff KingOct 30, 2013
  74. Duy NguyenOct 31, 2013
  75. 12/19 rev-list: add bitmap mode to speed up object listsJeff King, Oct 25, 2013
  76. 13/19 pack-objects: implement bitmap writingJeff King, Oct 25, 2013
  77. 14/19 repack: stop using magic number for ARRAY_SIZE(exts)Jeff King, Oct 25, 2013
  78. 15/19 repack: turn exts array into array-of-structJeff King, Oct 25, 2013
  79. 16/19 repack: handle optional files created by pack-objectsJeff King, Oct 25, 2013
  80. 17/19 repack: consider bitmaps when performing repacksJeff King, Oct 25, 2013
  81. 18/19 t: add basic bitmap functionality testsJeff King, Oct 25, 2013
  82. SZEDER GáborOct 28, 2013
  83. Jeff KingOct 30, 2013
  84. 19/19 pack-bitmap: implement optional name_hash cacheJeff King, Oct 25, 2013
  85. 20/19 count-objects: consider .bitmap without .pack/.idx pair garbageNguyễn Thái Ngọc Duy, Oct 26, 2013
  86. Jeff KingOct 30, 2013
  87. Junio C HamanoOct 30, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.