Re: Fast access git-rev-list output: some OS knowledge required

From: Marco Costalba <mcostalba@gmail.com>
Date: Dec 7, 2006, 06:46 UTC
Message-ID: <e5bfff550612062246m194ce235nf97149f8b041f486@mail.gmail.com>
In-Reply-To: <Pine.LNX.4.64.0612061642440.3542@woody.osdl.org>

On 12/7/06, Linus Torvalds <torvalds@osdl.org> wrote:

Show 13 quoted lines

>
>
> On Thu, 7 Dec 2006, Johannes Schindelin wrote:
> >
> > Because, depending on what you do, the revision machinery is not
> > reentrable. For example, if you filter by filename, the history is
> > rewritten in-memory to simulate a history where just that filename was
> > tracked, and nothing else. These changes are not cleaned up after calling
> > the internal revision machinery.
>
> Well, it really wouldn't be that hard to add a new library interface to
> "reset object state". We could fairly trivially either:
>

So the library approach sounds like the best?

Of course in this case the producer git-rev-list and the receiver use the same address space.

In the case of a temporary file data is first copied to OS disk cache buffers and then again to userspace, in qgit address space. But the real pain is that the temporary file is always flushed to disk after 4-5 seconds from creation, also if under heavy read/write activity. This is a problem for big repos. I really don't know how to workaround this useless disk flush.

Finally, what about using some kind of shared memory at run time, instead of _sharing_ developer libraries ;-) ? is it too messy?

Probably the concurrent reading while writing is possible without syncro if the reader understands that a sequence of _two_ or more \0 it means the end of current write stream if producer is still running or the end of data if producer is not running anymore. I use a similar approach in the 'temporary file' patch where receiver is able to read while producer writes without explicit synchronization. In that case a read() of a block smaller then maximum with producer still running is used as the 'break' condition in the receiver while loop.