git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: cloning the kernel - why long time in "Resolving 313037 deltas"

From
Shawn Pearce <spearce@spearce.org>
Date
Dec 19, 2006, 06:39 UTC
Message-ID
<20061219063930.GA2511@spearce.org>
In-Reply-To
<20061219051108.GA29405@thunk.org>
Theodore Tso <tytso@mit.edu> wrote:
Show 19 quoted lines
> On Mon, Dec 18, 2006 at 07:13:40PM -0500, Nicolas Pitre wrote:
> > Maybe.  However the mmap() may occur on section of the pack file which 
> > has just been written to in order to write even more, always to the same 
> > file.  On Linux this is fast because the mmap'd data is likely to still 
> > be in the cache.
> > 
> > I guess this could be turned into a malloc()/read()/free() with no 
> > trouble.
> 
> Actually, depending on the size of the chunk, even on Linux
> malloc/read/free can be faster than the mmap/munmap, because
> mmap/munmap calls involve page table manipulations, and even on Linux
> that is often slower or dead even with the memory copy involved with
> using malloc/read.  Even when reading huge chunks of Canon Raw File
> data at a time, I found (experimentally) that it was no faster to use
> mmap() compared to read().  And for small chunks of data, malloc/read
> will definitely win out over mmap(), since the page table operations
> and resulting page faults completely trump the cost of copying the
> bytes from the page cache to the read() buffer.

This is why git-fast-import mmaps 128 MiB blocks from the file at a time. The mmap region is usually much larger than the file itself; the application appends to the file via write() then goes back and rereads data when necessary via the already established mmap. Its rare for the application to need to unmap/remap a different block so there really isn't very much page table manipulation overhead.

Why isn't git-index-pack doing the same? Is there some hidden glitch in some OS somewhere that has a problem with overmapping a file and appending into it via write()? I've done that on Mac OS X, Linux, BSDi, Solaris... never had a problem.

Previous: Theodore TsoNext: Linus Torvalds
Message 17 of 51 in “Re: [PATCH] fetch-pack: avoid fixing thin packs when unnecessary”
  1. Johannes SchindelinDec 18, 2006
  2. Nicolas PitreDec 18, 2006
  3. Randal L. SchwartzDec 18, 2006
  4. Nicolas PitreDec 18, 2006
  5. Randal L. SchwartzDec 18, 2006
  6. Nicolas PitreDec 18, 2006
  7. Linus TorvaldsDec 18, 2006
  8. Randal L. SchwartzDec 18, 2006
  9. Martin LanghoffDec 18, 2006
  10. Kyle MoffettDec 22, 2006
  11. Shawn PearceDec 22, 2006
  12. Marco RoelandDec 22, 2006
  13. Andreas EricssonJan 3, 2007
  14. Linus TorvaldsDec 18, 2006
  15. Nicolas PitreDec 19, 2006
  16. Theodore TsoDec 19, 2006
  17. Shawn PearceDec 19, 2006
  18. Linus TorvaldsDec 19, 2006
  19. Shawn PearceDec 19, 2006
  20. Marco RoelandDec 19, 2006
  21. Shawn PearceDec 19, 2006
  22. Shawn PearceDec 19, 2006
  23. Marco RoelandDec 19, 2006
  24. Shawn PearceDec 19, 2006
  25. Marco RoelandDec 19, 2006
  26. Alex RiesenDec 19, 2006
  27. Juergen RuehleDec 21, 2006
  28. Theodore TsoDec 19, 2006
  29. Linus TorvaldsDec 19, 2006
  30. Shawn PearceDec 20, 2006
  31. Shawn PearceDec 20, 2006
  32. Linus TorvaldsDec 19, 2006
  33. Johannes SchindelinDec 19, 2006
  34. Junio C HamanoDec 19, 2006
  35. Jeff KingDec 19, 2006
  36. Andy WhitcroftDec 19, 2006
  37. index-pack usage of mmap() is unacceptably slower on many OSes other than LinuxNicolas Pitre, Dec 19, 2006
  38. Junio C HamanoDec 19, 2006
  39. Nicolas PitreDec 19, 2006
  40. Linus TorvaldsDec 19, 2006
  41. Randal L. SchwartzDec 19, 2006
  42. Randal L. SchwartzDec 19, 2006
  43. Jeff GarzikDec 19, 2006
  44. Junio C HamanoDec 20, 2006
  45. Linus TorvaldsDec 20, 2006
  46. Jeff GarzikDec 20, 2006
  47. Junio C HamanoDec 20, 2006
  48. Junio C HamanoDec 20, 2006
  49. Linus TorvaldsDec 20, 2006
  50. Junio C HamanoDec 20, 2006
  51. Nikolai WeibullDec 20, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.