Re: Repacking many disconnected blobs

On Wed, 14 Jun 2006, Keith Packard wrote:

There's actually a secondary benefit to packing that turned out to be much bigger from a performance standpoint: the size benefit coupled with the fact that it's all in one file ends up meaning that accessing packed objects is _much_ faster than accessing individual files.

The Linux system call overhead is one of the lowest ones out there, but it's still much bigger than just a function call, and doing a full pathname walk and open/close is bigger yet. In contrast, if you access lots of objects and they are all in a pack, you only end up doing one mmap and a page fault for each 4kB entry, and that's it.

So packing has a large performance benefit outside of the actual disk use one, and to some degree that performance benefit is then further magnified by good locality (ie you get more effective objects per page fault), but in your case that locality issue is secondary.

I assume that you never actually end up looking at the _contents_ of the objects any more ever afterwards, because in a very real sense you're really interested in the SHA1 names, right? All the latter phases of parsecvs will just use the SHA1 names directly, and never actually even open the data (packed or not).

So in that sense, you only care about the disksize and a much improved directory walk from fewer files (until the repository has actually been fully created, at which point a repack will do the right thing).

			Linus