git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] multi item packed files

From
Linus Torvalds <torvalds@osdl.org>
Date
Apr 21, 2005, 15:41 UTC
Message-ID
<Pine.LNX.4.58.0504210832490.2344@ppc970.osdl.org>
In-Reply-To
<200504211113.13630.mason@suse.com>
On Thu, 21 Apr 2005, Chris Mason wrote:
Show 6 quoted lines
> 
> There have been a few threads on making git more space efficient, and 
> eventually someone mentions tiny files and space fragmentation.  Now that git 
> object names are decoupled from their compression, it's easier to consider a 
> a variety of compression algorithms.  I whipped up a really silly "pack files 
> together" compression.
Careful.

This is something that needs history to tell whether it's effective. In particular, if one file changes and another one does not, your packed archive now ends up being a new blob, so while you "saved space" by having just one blob for the object, in reality you didn't save any space at all because with the <x> files changing, you just guaranteed that the packed blob changes <x> times more often.

See? Your "packing in space" ends up also resulting in "packing in time", and you didn't actually win anything.

(If you did a good job of packing, you hopefully didn't _lose_ anything either - you needed 1:<x> number of objects that took 1:<x> the space if the packing ended up perfect - but since you needed <x> times more of these objects unless they all change together, you end up with exactly the same space usage).

So the argument is: you can't lose with the method, and you _can_ win. Right?

Wrong. You most definitely _can_ lose: you end up having to optimize for one particular filesystem blocking size, and you'll lose on any other filesystem. And you'll lose on the special filesystem of "network traffic", which is byte-granular.

I don't want to pee on peoples parades, and I'm all for gathering numbers, but the thing is, the current git isn't actually all that bad, and I guarantee that it's hard to make it better without using delta representation. And the current thing is really really simple.

		Linus
Previous: Chris MasonNext: Chris Mason
Message 2 of 17 in “multi item packed files”
  1. multi item packed filesChris Mason, Apr 21, 2005
  2. Linus TorvaldsApr 21, 2005
  3. Chris MasonApr 21, 2005
  4. Krzysztof HalasaApr 21, 2005
  5. Linus TorvaldsApr 21, 2005
  6. Krzysztof HalasaApr 22, 2005
  7. Martin UeckerApr 22, 2005
  8. Chris MasonApr 21, 2005
  9. Linus TorvaldsApr 21, 2005
  10. Chris MasonApr 22, 2005
  11. Linus TorvaldsApr 22, 2005
  12. Chris MasonApr 22, 2005
  13. Linus TorvaldsApr 22, 2005
  14. Chris MasonApr 22, 2005
  15. Chris MasonApr 22, 2005
  16. Chris MasonApr 25, 2005
  17. Krzysztof HalasaApr 22, 2005

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.