git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Compression speed for large files

From
JHJoachim B Haga <cjhaga@fys.uio.no>
Date
Jul 3, 2006, 11:13 UTC
Message-ID
<loom.20060703T124601-969@post.gmane.org>

I'm looking at doing version control of data files, potentially very large, often binary. In git, committing of large files is very slow; I have tested with a 45MB file, which takes about 1 minute to check in (on an intel core-duo 2GHz).

Now, most of the time is spent in compressing the file. Would it be a good idea to change the Z_BEST_COMPRESSION flag to zlib, at least for large files? I have measured the time spent by git-commit with different flags in sha1_file.c:

  method                 time (s)  object size (kB)
  Z_BEST_COMPRESSION     62.0      17136
  Z_DEFAULT_COMPRESSION  10.4      16536
  Z_BEST_SPEED            4.8      17071

In this case Z_BEST_COMPRESSION also compresses worse, but that's not the major issue: the time is. Here's a couple of other data points, measured with gzip -9, -6 and -1 (comparable to the Z_ flags above):

129MB ascii data file
  method    time (s)  object size (kB)
  gzip -9   158       23066
  gzip -6    18       23619
  gzip -1     6       32304
3MB ascii data file
  gzip -9   2.2        887
  gzip -6   0.7        912
  gzip -1   0.3       1134
So: is it a good idea to change to faster compression, at least for larger
files? From my (limited) testing I would suggest using Z_BEST_COMPRESSION only
for small files (perhaps <1MB?) and Z_DEFAULT_COMPRESSION/Z_BEST_SPEED for
larger ones.
-j.
Next: Alex Riesen
Message 1 of 21 in “Compression speed for large files”
  1. Joachim B HagaJul 3, 2006
  2. Alex RiesenJul 3, 2006
  3. ElrondJul 3, 2006
  4. Joachim B HagaJul 3, 2006
  5. Joachim Berdal HagaJul 3, 2006
  6. Nicolas PitreJul 3, 2006
  7. Yakov LernerJul 3, 2006
  8. Johannes SchindelinJul 3, 2006
  9. Linus TorvaldsJul 3, 2006
  10. Make zlib compression level configurable, and change default.Joachim B Haga, Jul 3, 2006
  11. Linus TorvaldsJul 3, 2006
  12. Linus TorvaldsJul 3, 2006
  13. Joachim B HagaJul 3, 2006
  14. Use configurable zlib compression level everywhere.Joachim B Haga, Jul 3, 2006
  15. Junio C HamanoJul 3, 2006
  16. David LangJul 7, 2006
  17. Johannes SchindelinJul 8, 2006
  18. Jeff KingJul 3, 2006
  19. Joachim Berdal HagaJul 3, 2006
  20. Linus TorvaldsJul 3, 2006
  21. Joachim Berdal HagaJul 4, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.