git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/7] thread lazy_init_name_hash

From
Junio C Hamano <gitster@pobox.com>
Date
Mar 23, 2017, 17:52 UTC
Message-ID
<xmqqh92jev8d.fsf@gitster.mtv.corp.google.com>
In-Reply-To
<1490276825-41544-1-git-send-email-git@jeffhostetler.com>
git@jeffhostetler.com writes:
Show 34 quoted lines
> From: Jeff Hostetler <jeffhost@microsoft.com>
>
> Version 2 of this patch series addresses the coding
> style issues, compile errors in non-threaded builds,
> and updated API documentation.
>
>
> This patch series is a performance optimization for
> lazy_init_name_hash() in name-hash.c on very large
> repositories.
>
> This change allows lazy_init_name_hash() to optionally
> use multiple threads when building the the_index.dir_hash
> and the_index.name_hash hashmaps.  The original code path
> has been preserved and is used when the repo is small or
> the system does not have sufficient CPUs.
>
> A helper command (t/helper/test-lazy-init-name-hash) was
> created to demonstrate performance differences and validate
> output.  For example, use the '-p' option to compare both
> code paths on a large repo.
>
> During our testing on the Windows source tree (3.1M
> files, 500K folders, 450MB index) with an 8 (logical)
> core machine, we reduced the runtime of lazy_init_name_hash()
> from 1.4 to 0.27 seconds.
>
> This patch series replaces my earlier
>      * jh/memihash-opt (2017-02-17) 5 commits
> patch series.  This series is an improvement over the
> original proposal because it completely isolates the changes
> in name name-hash.c (rather than having parts in preload-index.c)
> and eliminates the need to update/invalidate precomputed hash
> values as cache_entries are changed.

Having the above helps readers by giving them what to expect to see to prepare them before they dive into the patches ;-)

The API document update in 4/7 is a nice addition and it comes at the right spot in the series, just after API enhancement is done. I gave a quick reading on it twice, and all looked reasonable. Nicely done.

I queued the sparse things Ramsay pointed out in the form of "SQUASH???" immediately that follows your original patch while applying the series, so please check what I push out (when it happens) and if there is no further change needed, just tell me to actually squash them in, if you think these SQUASH??? patches are good. That way we do not have to have another reroll only to fix these up ;-)

I'll need to re-read "name-hash: perf improvement for lazy_init_name_hash" later again to convince myself, but during my initial read (from the last round) I didn't spot anything wrong there.

Thanks.
Previous: git@jeffhostetler.comNext: Jeff Hostetler
Message 15 of 20 in “thread lazy_init_name_hash”
  1. 0/7 thread lazy_init_name_hashgit@jeffhostetler.com, Mar 23, 2017
  2. 2/7 hashmap: allow memihash computation to be continuedgit@jeffhostetler.com, Mar 23, 2017
  3. 6/7 name-hash: add test-lazy-init-name-hashgit@jeffhostetler.com, Mar 23, 2017
  4. Ramsay JonesMar 23, 2017
  5. 1/7 name-hash: specify initial size for istate.dir_hash tablegit@jeffhostetler.com, Mar 23, 2017
  6. 7/7 name-hash: add perf test for lazy_init_name_hashgit@jeffhostetler.com, Mar 23, 2017
  7. 5/7 name-hash: perf improvement for lazy_init_name_hashgit@jeffhostetler.com, Mar 23, 2017
  8. Ramsay JonesMar 23, 2017
  9. Stefan BellerMar 23, 2017
  10. Jeff HostetlerMar 24, 2017
  11. Junio C HamanoMar 27, 2017
  12. Jeff HostetlerMar 27, 2017
  13. 3/7 hashmap: Add disallow_rehash settinggit@jeffhostetler.com, Mar 23, 2017
  14. 4/7 hashmap: document memihash_cont, hashmap_disallow_rehash apigit@jeffhostetler.com, Mar 23, 2017
  15. Junio C HamanoMar 23, 2017
  16. Jeff HostetlerMar 24, 2017
  17. Stefan BellerMar 24, 2017
  18. Junio C HamanoMar 24, 2017
  19. Duy NguyenApr 6, 2017
  20. Jeff HostetlerApr 6, 2017

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.