git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] fsck: use oidset for skiplist

From
Jeff King <peff@peff.net>
Date
Aug 27, 2018, 23:03 UTC
Message-ID
<20180827230314.GB10402@sigill.intra.peff.net>
In-Reply-To
<030bac66-eeb4-7bc9-8f27-1e6b4124fd76@web.de>
On Sun, Aug 26, 2018 at 01:37:41PM +0200, René Scharfe wrote:
Show 32 quoted lines
> Am 14.08.2018 um 03:58 schrieb Jeff King:
> > Your suggestion can be implemented using khash (my patch below).
> > 
> >> Before:
> >> Benchmark #1: ./git-cat-file --batch-all-objects --buffer --unordered --batch-check='%(objectname)'
> >>
> >>   Time (mean ± σ):     269.5 ms ±  26.7 ms    [User: 247.7 ms, System: 21.4 ms]
> >>
> >>   Range (min … max):   240.3 ms … 339.3 ms
> >>
> >> After:
> >> Benchmark #1: ./git-cat-file --batch-all-objects --buffer --unordered --batch-check='%(objectname)'
> >>
> >>   Time (mean ± σ):     224.2 ms ±  18.2 ms    [User: 201.7 ms, System: 22.1 ms]
> >>
> >>   Range (min … max):   205.0 ms … 259.0 ms
> > 
> > Yeah. My best-of-five dropped from 300ms to 247ms. That 300 was using
> > the memory pool, though khash's deletion strategy isn't all that
> > different (the wasted memory hangs around until the next hash resize,
> > but if you're evenly dropping and adding, you likely won't need to
> > resize).
> 
> With your khash patch:
> 
> Benchmark #1: ./git-cat-file --batch-all-objects --buffer --unordered --batch-check='%(objectname)'
> 
>   Time (mean ± σ):     159.1 ms ±  20.5 ms    [User: 140.3 ms, System: 18.5 ms]
> 
>   Range (min … max):   140.0 ms … 214.0 ms
> 
> So it seems worth it.

Hmm, that really does. Which is a shame, because I hoped that one day we could get rid of the nasty macro-infestation that is khash.h. But it really is a lot faster than the alternatives.

> [...]

I agree with all of your comments here. What I posted was just trying to do the least amount of work to get something we could time.

Do you want to pick up this topic and carry it forward?
-Peff
Previous: René ScharfeNext: René Scharfe
Message 10 of 31 in “fsck: use strbuf_getline() to read skiplist file”
  1. 1/2 fsck: use strbuf_getline() to read skiplist fileRené Scharfe, Aug 11, 2018
  2. 2/2 fsck: use oidset for skiplistRené Scharfe, Aug 11, 2018
  3. Jeff KingAug 11, 2018
  4. Jeff KingAug 11, 2018
  5. René ScharfeAug 11, 2018
  6. René ScharfeAug 13, 2018
  7. Jeff KingAug 14, 2018
  8. Jeff KingAug 14, 2018
  9. René ScharfeAug 26, 2018
  10. Jeff KingAug 27, 2018
  11. René ScharfeOct 1, 2018
  12. Jeff KingOct 1, 2018
  13. René ScharfeOct 2, 2018
  14. Jeff KingOct 2, 2018
  15. René ScharfeAug 13, 2018
  16. Jeff KingAug 14, 2018
  17. Ævar Arnfjörð BjarmasonAug 11, 2018
  18. René ScharfeAug 25, 2018
  19. Ramsay JonesAug 11, 2018
  20. René ScharfeAug 25, 2018
  21. Junio C HamanoAug 13, 2018
  22. René ScharfeAug 13, 2018
  23. Junio C HamanoAug 13, 2018
  24. René ScharfeAug 13, 2018
  25. Jeff KingAug 11, 2018
  26. René ScharfeAug 11, 2018
  27. 2/2 fsck: use oidset for skiplistRené Scharfe, Aug 25, 2018
  28. Ævar Arnfjörð BjarmasonAug 27, 2018
  29. René ScharfeAug 27, 2018
  30. 1/2 fsck: use strbuf_getline() to read skiplist fileRené Scharfe, Aug 25, 2018
  31. Jeff KingAug 27, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.