Re: win2k/cygwin cannot handle even moderately sized packs

From: Alex Riesen <fork0@t-online.de>
Date: Nov 8, 2006, 21:33 UTC
Message-ID: <20061108213314.GA4437@steel.home>
In-Reply-To: <20061108171131.GA13487@spearce.org>

Shawn Pearce, Wed, Nov 08, 2006 18:11:31 +0100:

Show 13 quoted lines

> > >All true.  However what happens when the header spans two windows?
> > >Lets say I have the first 4 MiB mapped and the next 4 MiB mapped in
> > >a different window; these are not necessarily at the same locations
> > >within memory.  Now if an object header is split over these two
> > >then some bytes are at the end of the first window and the rest
> > >are at the start of the next window.
> > 
> > Assuming these are adjacent windows, we can just increment counters on the
> > all touched pages (at least the two together) and return the pointer into
> > the lowest page. Otherwise - time for garbage collection (why produce the
> > garbage at all, btw?) and remap.
> 
> They are adjacent in the pack file but not necessarily in virtual memory!

Oh, right! Don't know why I thought the mapped regions would be connected.

Show 6 quoted lines

> The garbage creation is to account for the 2-4 windows required
> by most applications.  Most of the time each window is unused;
> we really only have two windows in use during delta decompression,
> at all other times we really only have 1 window in use.  The commit
> parsing applications don't keep the commit window in use when they
> go access a tree or a blob.

So they actually can call unuse_pack to unmap the window, but it's kept for caching reasons?

Show 5 quoted lines

> Consequently we want the garbage there.  Actually I shouldn't have
> used garbage: the correct term would be LRU managed cache.  :-)
> When we need a new window and we would exceed our maximum limit
> (128 MiB in my implementation) we unmap the least recently used
> window which is not currently in use.

Yep, noticed that :) Just wondered why.

> I could be wrong.  It may not matter.  But I think its crazy to
> unmap otherwise valid mappings just because 2 bytes are on the
> wrong side of an arbitrary boundary.

You're right, would be unfortunate to remap too often.

use_pack always maps at least 20 bytes, if I understand in_window and its use correctly. Actually, now I'm staring at it longer, I think the interface I suggested does almost the same, just allows to configure (well, hint at) the amount of bytes to be mapped in.

I still can't let go of the idea to get as much data as possible with just one call to sliding window code. Calling use_pack for every byte just does not seem right.