git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Compression speed for large files

From
Linus Torvalds <torvalds@osdl.org>
Date
Jul 3, 2006, 23:02 UTC
Message-ID
<Pine.LNX.4.64.0607031556480.12404@g5.osdl.org>
In-Reply-To
<44A99961.8090504@fys.uio.no>
On Tue, 4 Jul 2006, Joachim Berdal Haga wrote:
> 
> Here's a test with "time gzip -[169] -c file >/dev/null". Random data
> from /dev/urandom, kernel headers are concatenation of *.h in kernel
> sources. All times in seconds, on my puny home computer (1GHz Via Nehemiah)
That "Via Nehemiah" is probably a big part of it.

I think the VIA Nehemiah just has a 64kB L2 cache, and I bet performance plummets if the tables end up being used past that.

And I think a large part of the higher compressions is that they allow the compression window and tables to grow bigger.

		Linus
Previous: Joachim Berdal HagaNext: Joachim Berdal Haga
Message 20 of 21 in “Compression speed for large files”
  1. Joachim B HagaJul 3, 2006
  2. Alex RiesenJul 3, 2006
  3. ElrondJul 3, 2006
  4. Joachim B HagaJul 3, 2006
  5. Joachim Berdal HagaJul 3, 2006
  6. Nicolas PitreJul 3, 2006
  7. Yakov LernerJul 3, 2006
  8. Johannes SchindelinJul 3, 2006
  9. Linus TorvaldsJul 3, 2006
  10. Make zlib compression level configurable, and change default.Joachim B Haga, Jul 3, 2006
  11. Linus TorvaldsJul 3, 2006
  12. Linus TorvaldsJul 3, 2006
  13. Joachim B HagaJul 3, 2006
  14. Use configurable zlib compression level everywhere.Joachim B Haga, Jul 3, 2006
  15. Junio C HamanoJul 3, 2006
  16. David LangJul 7, 2006
  17. Johannes SchindelinJul 8, 2006
  18. Jeff KingJul 3, 2006
  19. Joachim Berdal HagaJul 3, 2006
  20. Linus TorvaldsJul 3, 2006
  21. Joachim Berdal HagaJul 4, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.