git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: chunking (Re: [ANNOUNCEMENT] /Arch/ embraces `git')

From
C. Scott Ananian <cscott@cscott.net>
Date
Apr 20, 2005, 23:42 UTC
Message-ID
<Pine.LNX.4.61.0504201934380.2630@cag.csail.mit.edu>
In-Reply-To
<Pine.LNX.4.58.0504201510520.6467@ppc970.osdl.org>
On Wed, 20 Apr 2005, Linus Torvalds wrote:
Show 6 quoted lines
> What's the disk usage results? I'm on ext3, for example, which means that
> even small files invariably take up 4.125kB on disk (with the inode).
>
> Even uncompressed, most source files tend to be small. Compressed, I'm
> seeing the median blob size being ~1.6kB in my trivial checks. That's
> blobs only, btw.

I'm working on it. The format was chosen so that blobs under 1 block long *stay* 1 block long; i.e. there's no 'chunk plus index file' overhead. So the chunking should only kick in on multiple-block files. I hacked 'convert-cache' to do the conversion and it's running out of memory on linux-2.6.git, however --- I found a few memory leaks in your code =) but I certainly seem to be missing a big one still (maybe it's in my code!).

When I get this working properly, my plan is to do a number of runs over the linux-2.6 archive trying out various combinations of chunking parameters. I *will* be watching both 'real' disk usage (bunged up to block boundaries) and 'ideal' disk usage (on a reiserfs-type system). The goal is to improve both, but if I can improve 'ideal' usage significantly with a minimal penalty in 'real' usage then I would argue it's still worth doing, since that will improve network times.

The handshaking penalties you mention are significant, but that's why rsync uses a pipelined approach. The 'upstream' part of your full-duplex pipe is 'free' while you've got bits clogging your 'downstream' pipe. The wonders of full-duplex...

Anyway, "numbers talk, etc".  I'm working on them.
  --scott
LIONIZER LCPANES shortwave MKSEARCH ESGAIN Saddam Hussein Rijndael 
WASHTUB Morwenstow ZPSEMANTIC SKIMMER cryptographic FJHOPEFUL assassination
                          ( http://cscott.net/ )
Previous: Linus TorvaldsNext: C. Scott Ananian
Message 22 of 23 in “[ANNOUNCEMENT] /Arch/ embraces `git'”
  1. Tom LordApr 20, 2005
  2. Miles BaderApr 20, 2005
  3. duchier@ps.uni-sb.deApr 20, 2005
  4. Tomas MrazApr 20, 2005
  5. Denys DuchierApr 21, 2005
  6. Tomas MrazApr 21, 2005
  7. duchier@ps.uni-sb.deApr 21, 2005
  8. Tomas MrazApr 20, 2005
  9. Tom LordApr 21, 2005
  10. Tom LordApr 21, 2005
  11. Tom LordApr 20, 2005
  12. Denys DuchierApr 21, 2005
  13. Tom LordApr 21, 2005
  14. Tomas MrazApr 21, 2005
  15. Tom LordApr 21, 2005
  16. Tom LordApr 21, 2005
  17. Linus TorvaldsApr 22, 2005
  18. Edésio Costa e SilvaApr 22, 2005
  19. Petr BaudisApr 20, 2005
  20. C. Scott AnanianApr 20, 2005
  21. chunking (Re: [ANNOUNCEMENT] /Arch/ embraces `git')Linus Torvalds, Apr 20, 2005
  22. C. Scott AnanianApr 20, 2005
  23. blowing chunks (quick update)C. Scott Ananian, Apr 22, 2005

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.