git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Compression speed for large files

From
JHJoachim Berdal Haga <c.j.b.haga@fys.uio.no>
Date
Jul 4, 2006, 05:42 UTC
Message-ID
<44A9FFAB.1010708@fys.uio.no>
In-Reply-To
<Pine.LNX.4.64.0607031556480.12404@g5.osdl.org>
Linus Torvalds wrote:
Show 10 quoted lines
> 
> On Tue, 4 Jul 2006, Joachim Berdal Haga wrote:
>> Here's a test with "time gzip -[169] -c file >/dev/null". Random data
>> from /dev/urandom, kernel headers are concatenation of *.h in kernel
>> sources. All times in seconds, on my puny home computer (1GHz Via Nehemiah)
> 
> That "Via Nehemiah" is probably a big part of it.
> 
> I think the VIA Nehemiah just has a 64kB L2 cache, and I bet performance 
> plummets if the tables end up being used past that.

Not really. The numbers in my original post were from a Intel core-duo, they were: 158/18/6 s for comparable (but larger) data.

And on a P4 1.8GHz with 512kB L2, the same 23MB data file compresses in 28.1/5.9/1.3 s. That's a factor 22 slowest/fastest; the VIA was only factor 18, so the difference is actually *larger*.

-j.
Previous: Linus Torvalds
Message 21 of 21 in “Compression speed for large files”
  1. Joachim B HagaJul 3, 2006
  2. Alex RiesenJul 3, 2006
  3. ElrondJul 3, 2006
  4. Joachim B HagaJul 3, 2006
  5. Joachim Berdal HagaJul 3, 2006
  6. Nicolas PitreJul 3, 2006
  7. Yakov LernerJul 3, 2006
  8. Johannes SchindelinJul 3, 2006
  9. Linus TorvaldsJul 3, 2006
  10. Make zlib compression level configurable, and change default.Joachim B Haga, Jul 3, 2006
  11. Linus TorvaldsJul 3, 2006
  12. Linus TorvaldsJul 3, 2006
  13. Joachim B HagaJul 3, 2006
  14. Use configurable zlib compression level everywhere.Joachim B Haga, Jul 3, 2006
  15. Junio C HamanoJul 3, 2006
  16. David LangJul 7, 2006
  17. Johannes SchindelinJul 8, 2006
  18. Jeff KingJul 3, 2006
  19. Joachim Berdal HagaJul 3, 2006
  20. Linus TorvaldsJul 3, 2006
  21. Joachim Berdal HagaJul 4, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.