Re: [PATCH v5 2/5] read-cache: load cache extensions on a worker thread
- From
Ben Peart <peartben@gmail.com>
- Date
- Sep 17, 2018, 16:38 UTC
- Message-ID
- <bfb8d836-86a4-a4f0-cdaf-aba25424d4cf@gmail.com>
- In-Reply-To
- <CACsJy8D-sM2SSfTMmsR0uKnP0FM9fGrN0Z-pH5irMsH1=-jrmQ@mail.gmail.com>
On 9/15/2018 6:24 AM, Duy Nguyen wrote:
Show 23 quoted lines
> On Sat, Sep 15, 2018 at 12:22 PM Duy Nguyen <pclouds@gmail.com> wrote:
>>> @@ -1944,6 +1993,26 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)
>>> istate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));
>>> istate->initialized = 1;
>>>
>>> + p.istate = istate;
>>> + p.mmap = mmap;
>>> + p.mmap_size = mmap_size;
>>> +
>>> +#ifndef NO_PTHREADS
>>> + nr_threads = git_config_get_index_threads();
>>> + if (!nr_threads)
>>> + nr_threads = online_cpus();
>>> +
>>> + if (nr_threads >= 2) {
>>> + extension_offset = read_eoie_extension(mmap, mmap_size);
>>> + if (extension_offset) {
>
> One more thing I forgot. If the extension area is small enough, then
> we should not need to create a thread to parse extensions in parallel.
> We should know roughly how much work we need because we know the total
> size of all extensions.
> The only extensions I found to be significant enough to be helped by a separate thread was the cache tree. Since the size of the cache tree is driven by the number of files in the repo, I think the existing THREAD_COST logic (that comes in the next patch of the series) is a sufficient proxy. Basically, if you have enough cache entries to be benefited by threading, your extensions (driven by the cache tree) are probably also big enough to warrant a thread.
Show 15 quoted lines
>>> + /* create a thread to load the index extensions */
>>
>> Pointless comment. It's pretty clear from the pthread_create() below
>> thanks to good function naming. Please remove.
>>
>>> + p.src_offset = extension_offset;
>>> + if (pthread_create(&p.pthread, NULL, load_index_extensions, &p))
>>> + die(_("unable to create load_index_extensions_thread"));
>>> + }
>>> + }
>>> +#endif
>>> +
>>> if (istate->version == 4) {
>>> previous_name = &previous_name_buf;
>>> mem_pool_init(&istate->ce_mem_pool,