git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Compression and dictionaries

From
Jon Smirl <jonsmirl@gmail.com>
Date
Aug 14, 2006, 03:37 UTC
Message-ID
<9e4733910608132037t4297c3bbq9b0cd6ebaa03b979@mail.gmail.com>
>From what I remember from long ago most compression schemes build

dictionaries as a way of achieving significant compression. If so, since we zlib compress each entry in a pack individually, are there many copies of very similar dictionaries in the pack?

Some compression schemes support being initialized with a fixed dictionary and sharing it over all entries. A fixed dictionary could be built by analysing a large pack file. Sharing a compression dictionary would probably be a win for me since I have 1M+ entries in a pack.

Poking around in the zlib it appears that zlib supports precomputed dictionaries. http://www.zlib.net/manual.html

  int deflateSetDictionary (z_streamp strm, const Bytef *dictionary,
uInt dictLength);
-- 
Jon Smirl
jonsmirl@gmail.com
Next: Shawn Pearce
Message 1 of 20 in “Compression and dictionaries”
  1. Jon SmirlAug 14, 2006
  2. Shawn PearceAug 14, 2006
  3. Jon SmirlAug 14, 2006
  4. Shawn PearceAug 14, 2006
  5. Alex RiesenAug 14, 2006
  6. Erik MouwAug 14, 2006
  7. Johannes SchindelinAug 14, 2006
  8. Jon SmirlAug 14, 2006
  9. Johannes SchindelinAug 14, 2006
  10. Jon SmirlAug 14, 2006
  11. David LangAug 14, 2006
  12. Jakub NarebskiAug 14, 2006
  13. Jeff GarzikAug 14, 2006
  14. David LangAug 14, 2006
  15. Jeff GarzikAug 14, 2006
  16. Jon SmirlAug 14, 2006
  17. David LangAug 14, 2006
  18. Johannes SchindelinAug 14, 2006
  19. Alex RiesenAug 14, 2006
  20. Johannes SchindelinAug 14, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.