git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Huge win, compressing a window of delta runs as a unit

From
Jon Smirl <jonsmirl@gmail.com>
Date
Aug 18, 2006, 13:15 UTC
Message-ID
<9e4733910608180615q4895334bw57c55e59a4ac5482@mail.gmail.com>
In-Reply-To
<Pine.LNX.4.64.0608172323520.11359@localhost.localdomain>
On 8/18/06, Nicolas Pitre <nico@cam.org> wrote:
> A better way to get such a size saving is to increase the window and
> depth parameters.  For example, a window of 20 and depth of 20 can
> usually provide a pack size saving greater than 11% with none of the
> disadvantages mentioned above.

Our window size is effectively infinite. I am handing him all of the revisions from a single file in optimal order. This includes branches. He takes these revisions, runs xdiff on them, and then puts the entire result into a single zlib blob.

I suspect the size reduction is directly proportional to the age of the repository. The kernel repository only has three years worth of data in it. Linus has the full history in another repository that is not in general distribution. We can get it from him when he gets back from vacation.

If the repository doesn't contain long delta chains the optimization doesn't help that much. On the other hand it doesn't hurt either since the chains weren't long. My repository is four times as old as the kernel one and I am getting 4x the benefit.

This is a good format for archival data that is infrequency accessed. That is why I proposed a two pack system. One pack would contain the archival data and be highly optimized for size and the second pack would contain recent changes and be optimized for speed. The size optimization is important for controlling bandwidth costs.

-- 
Jon Smirl
jonsmirl@gmail.com
Previous: Jon SmirlNext: Johannes Schindelin
Message 25 of 36 in “Huge win, compressing a window of delta runs as a unit”
  1. Jon SmirlAug 16, 2006
  2. Shawn PearceAug 17, 2006
  3. Johannes SchindelinAug 17, 2006
  4. Johannes SchindelinAug 17, 2006
  5. Jon SmirlAug 17, 2006
  6. Johannes SchindelinAug 17, 2006
  7. Nicolas PitreAug 17, 2006
  8. Johannes SchindelinAug 17, 2006
  9. Jon SmirlAug 17, 2006
  10. Nicolas PitreAug 17, 2006
  11. Jon SmirlAug 17, 2006
  12. Nicolas PitreAug 17, 2006
  13. Jon SmirlAug 17, 2006
  14. Nicolas PitreAug 17, 2006
  15. Jon SmirlAug 17, 2006
  16. Nicolas PitreAug 17, 2006
  17. Nicolas PitreAug 18, 2006
  18. Jon SmirlAug 18, 2006
  19. Nicolas PitreAug 18, 2006
  20. Jon SmirlAug 18, 2006
  21. Nicolas PitreAug 21, 2006
  22. Shawn PearceAug 21, 2006
  23. Jakub NarebskiAug 21, 2006
  24. Jon SmirlAug 21, 2006
  25. Jon SmirlAug 18, 2006
  26. Johannes SchindelinAug 18, 2006
  27. Jon SmirlAug 18, 2006
  28. Linus TorvaldsAug 19, 2006
  29. Nicolas PitreAug 18, 2006
  30. Shawn PearceAug 21, 2006
  31. Jon SmirlAug 21, 2006
  32. Nicolas PitreAug 21, 2006
  33. Jon SmirlAug 21, 2006
  34. Nicolas PitreAug 21, 2006
  35. Nicolas PitreAug 21, 2006
  36. Nicolas PitreAug 21, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.