From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Mon, 09 Apr 2012 14:43:56 GMT
Subject: Re: [POC PATCH 5/5] sha1_file: make the pack machinery thread-safe
Message-ID: <CACsJy8AyphUD-vFwwgaW0eWG3ekgHA+tcAwV2zk5YGorkW0TzQ@mail.gmail.com>
In-Reply-To: <dd2bf650b382f5aca727b7d93a48598fb1a2f7d9.1323419666.git.trast@student.ethz.ch>

On Fri, Dec 9, 2011 at 3:39 PM, Thomas Rast <trast@student.ethz.ch> wrote:
> More precisely speaking, this pushes the locking down from
> read_object() into bits of the pack machinery that cannot (yet) run in
> parallel.
>
> There are several hacks here:
>
> a) prepare_packed_git() must be called before any parallel accesses
>   happen.  It now unconditionally opens and maps all index files.
>
> b) similarly, prepare_replace_object() must be called before any
>   parallel read_sha1_file() happens
>
> This simplification lets us avoid locking outright to guard the index
> accesses; locking is then mainly required for open_packed_git(),
> [un]use_pack(), and such.
>
> The ultimate goal would of course be to let at least _some_ pack
> accesses happen without any locking whatsoever.  But grep already
> benefits from it with a nice speed boost on non-worktree greps.

(I'm running into multithread pack access problem in rev-list..)

Why not put the global pointer "struct packed_git *packed_git" to
"struct pack_context" and avoid locking entirely? Resource usage is
like we run <n> different processes, I think, which is not too bad. We
may want to share a few static pack_* variables such as
pack_open_fds.. to avoid hitting system limits too fast.
-- 
Duy