Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff

From: Duy Nguyen <pclouds@gmail.com>
Date: Apr 15, 2016, 00:16 UTC
Message-ID: <CACsJy8C9wA=Yig6n4evqYnpOTqOWKbH4kXH5OaYTRvS_5a+a8A@mail.gmail.com>
In-Reply-To: <xmqqd1psgiuc.fsf@gitster.mtv.corp.google.com>

On Fri, Apr 15, 2016 at 1:12 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 18 quoted lines

> Duy Nguyen <pclouds@gmail.com> writes:
>
>> On Wed, Apr 13, 2016 at 1:05 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>>>>  - access is slow (unless cached, but we can't be sure)
>>>>
>>>> We could solve this (and the other problem) with mlock.
>>>
>>> Probably you meant madvise(2)?
>>>
>>> For something of a size comparable to the index file held by
>>> index-helper-daemon in-core, I'd expect we wouldn't page too
>>> badly.
>>
>> I had a look at linux implementation of madvise(MADV_WILLNEED). All it
>> does is force populating the entire memory region, which is good. But
>> I suspect when memory is under pressure, some pages may be reclaimed.
>
> I share that suspicion.  Why is such a reclamation bad thing, though?

Because the index is read entirely in memory in the next git process, it will be slowed down a bit (and the disk error factor creeps in), but the next one would be fast again. So no, not so bad.

Show 10 quoted lines

>> index files in monster repo case can go up to a few hundred megabytes,
>> chances of being reclaimed rise accordingly. But we can reconsider
>> mlock() later when/if real problems happen.
>
> Holding onto "a few hundred megabytes" just so that occasional Git
> operations will not stall with the daemon and slowing down the
> overall work of the user by panalizing other elements in the user's
> workflow does not sound like a good trade-off to me.  Wouldn't the
> user better off by not using the daemon at that point, which would
> give the few hundred megabytes back to the system for better uses?

In an ideal world, I would go with multiple index backends and use something else, maybe lmdb again, more suitable for indexes this size. But that touches many many places.

We don't hold this memory forever though. If the daemon is idle for a while, it exits, releasing memory back to system. And not using mlock() already gives the OS more freedom in memory usage.

-- 
Duy