git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [POC PATCH 5/5] sha1_file: make the pack machinery thread-safe

From
Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date
Apr 9, 2012, 14:43 UTC
Message-ID
<CACsJy8AyphUD-vFwwgaW0eWG3ekgHA+tcAwV2zk5YGorkW0TzQ@mail.gmail.com>
In-Reply-To
<dd2bf650b382f5aca727b7d93a48598fb1a2f7d9.1323419666.git.trast@student.ethz.ch>
On Fri, Dec 9, 2011 at 3:39 PM, Thomas Rast <trast@student.ethz.ch> wrote:
Show 19 quoted lines
> More precisely speaking, this pushes the locking down from
> read_object() into bits of the pack machinery that cannot (yet) run in
> parallel.
>
> There are several hacks here:
>
> a) prepare_packed_git() must be called before any parallel accesses
>   happen.  It now unconditionally opens and maps all index files.
>
> b) similarly, prepare_replace_object() must be called before any
>   parallel read_sha1_file() happens
>
> This simplification lets us avoid locking outright to guard the index
> accesses; locking is then mainly required for open_packed_git(),
> [un]use_pack(), and such.
>
> The ultimate goal would of course be to let at least _some_ pack
> accesses happen without any locking whatsoever.  But grep already
> benefits from it with a nice speed boost on non-worktree greps.
(I'm running into multithread pack access problem in rev-list..)

Why not put the global pointer "struct packed_git *packed_git" to "struct pack_context" and avoid locking entirely? Resource usage is like we run <n> different processes, I think, which is not too bad. We may want to share a few static pack_* variables such as pack_open_fds.. to avoid hitting system limits too fast.

-- 
Duy
Previous: Thomas RastNext: Thomas Rast
Message 7 of 11 in “Threaded loose object and pack access”
  1. 0/5 Threaded loose object and pack accessThomas Rast, Dec 9, 2011
  2. 1/5 Turn grep's use_threads into a global flagThomas Rast, Dec 9, 2011
  3. 2/5 grep: push locking into read_sha1_*Thomas Rast, Dec 9, 2011
  4. 3/5 sha1_file_name_buf(): sha1_file_name in caller's bufferThomas Rast, Dec 9, 2011
  5. 4/5 sha1_file: stuff various pack reading variables into a structThomas Rast, Dec 9, 2011
  6. 5/5 sha1_file: make the pack machinery thread-safeThomas Rast, Dec 9, 2011
  7. Nguyen Thai Ngoc DuyApr 9, 2012
  8. Thomas RastApr 10, 2012
  9. Nguyen Thai Ngoc DuyApr 10, 2012
  10. Thomas RastDec 9, 2011
  11. Nguyen Thai Ngoc DuyDec 10, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.