Re: [PATCH v3 03/16] index-helper: new daemon for caching index and related stuff

From: Duy Nguyen <pclouds@gmail.com>
Date: Apr 12, 2016, 09:40 UTC
Message-ID: <CACsJy8C2FtdetQ_NJSKR_JCZ5Ju0E3rV7Du=J4f2_kn5qrcHxg@mail.gmail.com>
In-Reply-To: <1460417232.5540.53.camel@twopensource.com>

On Tue, Apr 12, 2016 at 6:27 AM, David Turner <dturner@twopensource.com> wrote:

Show 24 quoted lines

> On Fri, 2016-04-08 at 18:16 -0400, David Turner wrote:
>> And SHM on Macs works a bit differently than on Linux in at least two
>> irritating ways.
>>
>> So, uh, new version to come once I actually make it work on Mac.
>> Probably Monday.
>
> I was chatting with a friend about this and he mentioned that SHM does
> not really fit well into the Unix "everything is a file" model.  It
> lives in a separate namespace, and still requires most of the file-like
> operations just with funny names and a separate namespace: shm_open,
> shm_unlink.  This weirdness is something I noticed in my porting work:
> on OS X, a shm name can only be 32 bytes long, requiring weird hacks.
> And on OSX, fstat on a shm fd is rounded up to the page size (!).
>  There may also be other portability issues that I have not yet
> discovered.
>
> Instead, my friend suggests that we should just use files.  For
> instance, we could do $TMPDIR/$index_helper_pid/shm-index.$sha.
>
> (I'm proposing $TMPDIR because it's cleaned up on reboot so we don't
> need any manual intervention or complicated gc schemes)
>
> What do folks think of this?

I avoided actual files for two reasons

 - disk error rate is higher than memory one, and we might need
trailing SHA-1 back
 - access is slow (unless cached, but we can't be sure)

If we can keep index-helper stuff on tmpfs or similar, then it would address both, but that's even more OS-specific than shm. If we have a good abstraction layer then people can put stuff on $TMPDIR on Mac, for example. But then it's not full POSIX file interface anymore... And you forgot Windows which does not strictly follow UNIX design.

-- 
Duy