git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 03/16] pack-objects: use a faster hash table

From
Thomas Rast <trast@inf.ethz.ch>
Date
Jun 25, 2013, 14:03 UTC
Message-ID
<87a9mdnae3.fsf@linux-k42r.v.cablecom.net>
In-Reply-To
<1372116193-32762-4-git-send-email-tanoku@gmail.com>
Vicent Marti <tanoku@gmail.com> writes:
Show 28 quoted lines
> This commit brings `khash.h`, a header only hash table implementation
> that while remaining rather simple (uses quadratic probing and a
> standard hashing scheme) and self-contained, offers a significant
> performance improvement in both insertion and lookup times.
>
> `khash` is a generic hash table implementation that can be 'templated'
> for any given type while maintaining good performance by using preprocessor
> macros. This specific version has been modified to define by default a
> `khash_sha1` type, a map of SHA1s (const unsigned char[20]) to void *
> pointers.
>
> When replacing the old hash table implementation in `pack-objects` with
> the khash_sha1 table, the insertion time is greatly reduced:
>
> 	kh_put_sha1 :: 284.011ms
> 	add_object_entry_1 : 36.06ms
> 	hashcmp :: 24.045ms
>
> This reduction of more than 50% in the insertion and lookup times,
> although nice, is not particularly noticeable for normal `pack-objects`
> operation: `pack-objects` performs massive batch insertions and
> relatively few lookups, so `khash` doesn't get a chance to shine here.
>
> The big win here, however, is in the massively reduced amount of hash
> collisions (as you can see from the huge reduction of time spent in
> `hashcmp` after the change). These greatly improved lookup times
> will result critical once we implement the writing algorithm for bitmap
> indxes in a later patch of this series.

Is that reduction in collisions purely because it uses quadratic probing, or is there some other magic trick involved? Is the same also applicable to the other users of the "big" object hash table? (I assume Peff has already tried applying it there, but I'm still curious...)

-- 
Thomas Rast
trast@{inf,student}.ethz.ch
Previous: Vicent MartiNext: Jeff King
Message 6 of 64 in “Speed up Counting Objects with bitmap data”
  1. 00/16 Speed up Counting Objects with bitmap dataVicent Marti, Jun 24, 2013
  2. 01/16 list-objects: mark tree as unparsed when we free its bufferVicent Marti, Jun 24, 2013
  3. 02/16 sha1_file: refactor into `find_pack_object_pos`Vicent Marti, Jun 24, 2013
  4. Thomas RastJun 25, 2013
  5. 03/16 pack-objects: use a faster hash tableVicent Marti, Jun 24, 2013
  6. Thomas RastJun 25, 2013
  7. Jeff KingJun 26, 2013
  8. Jeff KingJun 26, 2013
  9. Ramkumar RamachandraJun 25, 2013
  10. Junio C HamanoJun 25, 2013
  11. Vicent MartíJun 25, 2013
  12. 04/16 pack-objects: make `pack_name_hash` globalVicent Marti, Jun 24, 2013
  13. 05/16 revision: allow setting custom limiter functionVicent Marti, Jun 24, 2013
  14. 06/16 sha1_file: export `git_open_noatime`Vicent Marti, Jun 24, 2013
  15. 07/16 compat: add endinanness helpersVicent Marti, Jun 24, 2013
  16. Peter KreftingJun 25, 2013
  17. Vicent MartíJun 25, 2013
  18. Peter KreftingJun 27, 2013
  19. 08/16 ewah: compressed bitmap implementationVicent Marti, Jun 24, 2013
  20. Junio C HamanoJun 25, 2013
  21. Junio C HamanoJun 25, 2013
  22. Thomas RastJun 25, 2013
  23. 09/16 documentation: add documentation for the bitmap formatVicent Marti, Jun 24, 2013
  24. Shawn PearceJun 25, 2013
  25. Vicent MartíJun 25, 2013
  26. Junio C HamanoJun 25, 2013
  27. Vicent MartíJun 25, 2013
  28. Shawn PearceJun 27, 2013
  29. Vicent MartíJun 27, 2013
  30. Jeff KingJun 27, 2013
  31. Shawn PearceJun 27, 2013
  32. Jeff KingJun 27, 2013
  33. Colby RangerJul 1, 2013
  34. Shawn PearceJul 1, 2013
  35. Jeff KingJul 7, 2013
  36. Shawn PearceJul 7, 2013
  37. Jeff KingJun 26, 2013
  38. Colby RangerJun 26, 2013
  39. Colby RangerJun 26, 2013
  40. Colby RangerJun 27, 2013
  41. Shawn PearceJun 27, 2013
  42. Shawn PearceJun 27, 2013
  43. Thomas RastJun 25, 2013
  44. Vicent MartíJun 25, 2013
  45. Thomas RastJun 26, 2013
  46. Thomas RastJun 26, 2013
  47. 10/16 pack-objects: use bitmaps when packing objectsVicent Marti, Jun 24, 2013
  48. Ramkumar RamachandraJun 25, 2013
  49. Thomas RastJun 25, 2013
  50. Junio C HamanoJun 25, 2013
  51. Vicent MartíJun 25, 2013
  52. 11/16 rev-list: add bitmap mode to speed up listsVicent Marti, Jun 24, 2013
  53. Thomas RastJun 25, 2013
  54. Vicent MartíJun 26, 2013
  55. Thomas RastJun 26, 2013
  56. Jeff KingJun 26, 2013
  57. 12/16 pack-objects: implement bitmap writingVicent Marti, Jun 24, 2013
  58. 13/16 repack: consider bitmaps when performing repacksVicent Marti, Jun 24, 2013
  59. Junio C HamanoJun 25, 2013
  60. Vicent MartíJun 25, 2013
  61. 14/16 sha1_file: implement `nth_packed_object_info`Vicent Marti, Jun 24, 2013
  62. 15/16 write-bitmap: implement new git command to write bitmapsVicent Marti, Jun 24, 2013
  63. 16/16 rev-list: Optimize --count using bitmaps tooVicent Marti, Jun 24, 2013
  64. Thomas RastJun 25, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.