Re: Fast access git-rev-list output: some OS knowledge required

From: Shawn Pearce <spearce@spearce.org>
Date: Dec 6, 2006, 20:18 UTC
Message-ID: <20061206201816.GF20320@spearce.org>
In-Reply-To: <e5bfff550612061208g6e4003e7ifa7dbd5ed69180c9@mail.gmail.com>

Marco Costalba <mcostalba@gmail.com> wrote:

Show 14 quoted lines

> On 12/6/06, Shawn Pearce <spearce@spearce.org> wrote:
> >Shawn Pearce <spearce@spearce.org> wrote:
> >
> >Perhaps there is some fast IPC API supported by Qt that you could
> >use to run the revision listing outside of the main UI process,
> >to eliminate the bottlenecks you are seeing and remove the problems
> >noted above?  One that doesn't involve reading from a pipe I mean...
> >
> 
> Qt it's very fast in reading from files, also git-rev-list is fast in
> write to a file...the problem is I would not want the file to be saved
> on disk, but stay cached in the OS memory for the few seconds needed
> to be written and read back, and then deleted. It's a kind of shared
> memory at the end. But I don't know how to realize it.

On a modern Linux (probably your largest target audience) a small file which has a very short lifespan (few seconds) is unlikey to hit the platter. Most filesystems will put the data into buffer cache and delay writing to disk because temporary files are so common on UNIX.

Though our resident Linux experts may chime in with more details...

> Also let git-rev-list to write directly in qgit process address space
> would be nice, indeed very nice.

And ugly.  :-)

SysV IPC (shared memory, semaphores) are messy and difficult to get right. mmap against a random file in the filesystem tends to work better on those systems which support it well, provided that the file isn't on a network mount. But again you still need semaphores or something like them to control access to the data in the mmap'd region.

I was thinking that maybe if Qt had a bounded buffer available for use between a process and its child, that you could use that to run your own "qgit-rev-list" child and get the data back more quickly, without the need for a temporary file. But it doesn't look like they have one. Oh well.

Your current temporary file approach is probably the best you can get, and has the simplest possible implementation. Doing better would require linking against libgit.a, and getting the core Git hackers to make at least the revision machinery more useful in a library setting.