Re: [PATCH/RFC] fast-import: insert new object entries at start of hash bucket

From: Shawn Pearce <spearce@spearce.org>
Date: Nov 23, 2010, 23:29 UTC
Message-ID: <AANLkTimG2V2h=z3t0kHxOy65W7JjayNb1RA8E4AabCV+@mail.gmail.com>
In-Reply-To: <20101123231718.GA4317@burratino>

On Tue, Nov 23, 2010 at 3:17 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

Show 8 quoted lines

> Shawn Pearce wrote:
>> On Mon, Nov 22, 2010 at 11:53 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>>                                                           The way I
>> read this store_tree() code, every subdirectory is recursed into even
>> if no modifications were made inside of that subdirectory during the
>> current commit.
>
> Doesn't the is_null_sha1 check avoid that?

Oh, right, yes. That check avoids processing a tree that wasn't modified. Damn that code isn't very clear. :-(

> To further explain the workload: svn-fe receives its blobs from svn
> in the form of deltas.  So the conversation might go like this:

...

Show 6 quoted lines

> Current svn-fe in vcs-svn-pu requests the preimage blobs using marks,
> but the idea is the same.
>
> If this proves a bottleneck I suppose we could cache the content of
> frequently-requested old blobs and keep pointers to that in the
> in-core tree.

OK, now I get it.

fast-import already uses sha1_file.c's delta base cache for the delta bases that are frequently accessed. It *may* help an importer that needs to keep refetching the same blob to cache the blob itself rather than its delta base (and reapply every time). It might not help too, it may hurt to try and keep recently accessed objects that the importer won't need to request again.

-- 
Shawn.