git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Huge win, compressing a window of delta runs as a unit

From
Nicolas Pitre <nico@cam.org>
Date
Aug 17, 2006, 16:33 UTC
Message-ID
<Pine.LNX.4.64.0608171220260.11359@localhost.localdomain>
In-Reply-To
<Pine.LNX.4.63.0608171738490.28360@wbgn013.biozentrum.uni-wuerzburg.de>
On Thu, 17 Aug 2006, Johannes Schindelin wrote:
Show 16 quoted lines
> Hi,
> 
> On Thu, 17 Aug 2006, Jon Smirl wrote:
> 
> > On 8/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > At least, the delta-chains should be limited by size (_not_ by number of
> > > deltas: you can have huge deltas, and if you have to unpack 5 huge deltas
> > > before getting to the huge delta you really need, it takes really long).
> > 
> > This is not an obvious conclusion.
> 
> The big win is bought by having _one_ zlib stream for multiple deltas, 
> right?
> 
> So, everytime you want to access the _last_ delta in the chain, you unpack 
> _all_ of them.
This is the case whether deltas are separately deflated or not.
> This quite obvious conclusion is obviously your reason to 
> propose two packs, one for the archive of "old" objects, and one for the 
> "new" objects.

Old objects are usually further down the delta chain and also stored further from the beginning of the pack. Hence "new" objects could still have quick access since even if a delta chain is all in the same zlib stream, it is likely that inflating the whole of the zlib stream to get "new" objects won't be necessary, just like it is done now where only the needed deltas are inflated.

Show 8 quoted lines
> > As for public servers there is an immediate win in shifting to the new
> > pack format.  Three hour downloads vs one hour, plus the bandwidth
> > bills. Are the tools smart enough to say this is a newer pack format,
> > upgrade? It takes far less than two hours to upgrade your git install.
> 
> Have you thought about a non-upgraded client? The server has to repack in 
> that case, and if the client wants a clone, you might not even have the 
> time to kiss your server good-bye before it goes belly up.

Nah. The only overhead for a server to feed an "old" pack format from a "new" pack file is to inflate and deflate some data. That is not _that_ costly.

Nicolas
Previous: Johannes SchindelinNext: Johannes Schindelin
Message 7 of 36 in “Huge win, compressing a window of delta runs as a unit”
  1. Jon SmirlAug 16, 2006
  2. Shawn PearceAug 17, 2006
  3. Johannes SchindelinAug 17, 2006
  4. Johannes SchindelinAug 17, 2006
  5. Jon SmirlAug 17, 2006
  6. Johannes SchindelinAug 17, 2006
  7. Nicolas PitreAug 17, 2006
  8. Johannes SchindelinAug 17, 2006
  9. Jon SmirlAug 17, 2006
  10. Nicolas PitreAug 17, 2006
  11. Jon SmirlAug 17, 2006
  12. Nicolas PitreAug 17, 2006
  13. Jon SmirlAug 17, 2006
  14. Nicolas PitreAug 17, 2006
  15. Jon SmirlAug 17, 2006
  16. Nicolas PitreAug 17, 2006
  17. Nicolas PitreAug 18, 2006
  18. Jon SmirlAug 18, 2006
  19. Nicolas PitreAug 18, 2006
  20. Jon SmirlAug 18, 2006
  21. Nicolas PitreAug 21, 2006
  22. Shawn PearceAug 21, 2006
  23. Jakub NarebskiAug 21, 2006
  24. Jon SmirlAug 21, 2006
  25. Jon SmirlAug 18, 2006
  26. Johannes SchindelinAug 18, 2006
  27. Jon SmirlAug 18, 2006
  28. Linus TorvaldsAug 19, 2006
  29. Nicolas PitreAug 18, 2006
  30. Shawn PearceAug 21, 2006
  31. Jon SmirlAug 21, 2006
  32. Nicolas PitreAug 21, 2006
  33. Jon SmirlAug 21, 2006
  34. Nicolas PitreAug 21, 2006
  35. Nicolas PitreAug 21, 2006
  36. Nicolas PitreAug 21, 2006

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.