Re: [PATCH] multi item packed files

On Thu, 21 Apr 2005, Chris Mason wrote:

I really disagree. 

On the other hand, git does a lot of things that are just _lovely_ for a filesystem:

 - it never rewrites a file. Rewriting a file is unquestionably _the_ 
   single worst access pattern for any filesystem. In contrast, never
   writing to a file again means that filesystems can optimize their
   layout and that things like defragmentation actually works.

 - it caches beautifully, and efficiently. Part of it comes from never 
   modifying files after they are written (which means that any network 
   filesystem automatically breathes a huge sign of relief), but part of 
   it is that it always reads full files, and the layout is done so that 
   it really actually _uses_ everything it reads.

   It also caches beautifully on a memory subsystem level, largely for the
   same reasons.

 - it doesn't use tons of directories.

   You say that "subdirectories with lots of files" is painful, but that's 
   not really the whole story. A _deep_ directory structure tends to 
   actually be worse in many ways, because it's much easier to optimize a 
   flat directory structure than a deep one. In other words, git ends up 
   making name hashing etc _productive_. 

So yes, it's a bit wasteful. But it's wasteful of what is absolutely the cheapest resource around: disk space. It's not a huge downside, and in fact I really do believe that the biggest downside _by_far_ in diskspace utilization is the _seek_ costs, not the space itself. Let's face it, anybody who wants three years of kernel archives and thinks that 3GB of disk is too much, has some serious problems.

The _seek_ issue is real, but git actually has a very nice architecture even there: not only dos it cache really really well (and you can do a simple "ls-tree $(cat .git/HEAD)" and populate the case from the results), but the low level of indirection in a git archive means that it's almost totally prefetchable with near-perfect access patterns.

In seeking, the real cost is synchronization, and the git model actually means that there are very few seeks that have to be synchronized. You could literally do the "ls-tree" thing and make an absolutely trivial prefetcher that did the prefetching with enough parallellism that the filesystem could probably get decent IO performance out of a disk.

In other words, we really could have a "git prefetch" command that would populate the cache of the current head quite efficiently. Because the data layout supports that.

		Linus