git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] Enable index-pack threading in msysgit.

From
Duy Nguyen <pclouds@gmail.com>
Date
Mar 21, 2014, 01:33 UTC
Message-ID
<CACsJy8A_CZYtHvVo5iMzgmP5t7XLxHNRD6n4D=_jiOLOvBgmWQ@mail.gmail.com>
In-Reply-To
<CAHOQ7J-sUt3HGYNE7n=X3ZmV3Q-n+n9hMDAtzLbH3YU8iAqoqA@mail.gmail.com>
On Fri, Mar 21, 2014 at 4:56 AM, Stefan Zager <szager@chromium.org> wrote:
Show 7 quoted lines
>> Considering all that, Duy's solution of opening separate file descriptors per thread seems to be the best pattern for future multi-threaded work.
>
> Does that mean you would endorse the (N threads) * (M pack files)
> approach to threading checkout and status?  That seems kind of
> crazy-town to me.  Not to mention that pack windows are not shared, so
> this approach to multi-threading can have the side-effect of blowing
> out memory consumption.

Maybe we could protect and share the delta cache. Pack windows are mmap'd so we should not need to worry about their memory consumption.

> We have already had to dial back settings for
> pack.threads and core.deltaBaseCacheLimit, because threaded index-pack
> was causing OOM errors on 32-bit platforms.

Hm.. I don't think index-pack uses sha1_file.c heavily. Local pack access is only needed for verifying identical objects (and that should never happen often). Something is fishy with these OOM errors.

Show 8 quoted lines
> Cygwin (and MSVC) should be able to share a "mostly" compliant pread
> implementation.  I don't have any insight into NonstopKernel; does is
> really not have a thread-safe pread implementation?  If so, then I
> suppose we have to #ifdef NO_PREAD, just as we do now.
>
> I realize that these are deep changes.  However, the performance of
> msysgit on the chromium repositories is pretty awful, enough so to
> motivate this work.
-- 
Duy
Previous: Stefan ZagerNext: Karsten Blees
Message 12 of 19 in “Enable index-pack threading in msysgit.”
  1. Enable index-pack threading in msysgit.szager@chromium.org, Mar 19, 2014
  2. Duy NguyenMar 19, 2014
  3. Stefan ZagerMar 19, 2014
  4. Duy NguyenMar 19, 2014
  5. Stefan ZagerMar 19, 2014
  6. Stefan ZagerMar 19, 2014
  7. Junio C HamanoMar 19, 2014
  8. Karsten BleesMar 20, 2014
  9. Stefan ZagerMar 20, 2014
  10. Karsten BleesMar 20, 2014
  11. Stefan ZagerMar 20, 2014
  12. Duy NguyenMar 21, 2014
  13. Karsten BleesMar 21, 2014
  14. Duy NguyenMar 21, 2014
  15. Duy NguyenMar 21, 2014
  16. Stefan ZagerMar 21, 2014
  17. Karsten BleesMar 21, 2014
  18. index-pack: work around thread-unsafe pread()Nguyễn Thái Ngọc Duy, Mar 25, 2014
  19. Johannes SixtMar 26, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.