two questions about the format of loose object

Hi,

In current implementation the loose objects are compressed:

     loose object = deflate(typename + <space> + size + '\0' + data)

In sha1_file.c:unpack_sha1_file():
	1) unpack_sha1_header() inflates first 8KB
        2) parse_sha1_header() gets object's size
        3) unpack_sha1_reset() allocates a (1+size) bytes buffer and
           copy the first 8KB without header to it.

* Question 1:

Why not use the format below for loose object?
    loose object = typename + <space> + size + '\0' + deflate(data)

So the size of loose object can be known before inflating it, in step 3 above the 8KB memcpy isn't required.

In general, deflate() can decrease file size by 70% for text file, I checked the git source and linux-2.6 source and got the statistical data below:

.------------------+--------------+--------.
|                  | <= (8/0.3)KB | <= 8KB |
|------------------+--------------+--------|
| git-1.6.03       |          97% |    84% |
| linux-2.6.27-rc6 |          90% |    66% |
`------------------+--------------+--------'

* Question 2:

Why not use uncompressed loose object? That's to say:
   loose object = typename + <space> + size + '\0' + data

I did a simple benchmark on my notebook and a server in my company, writing a big file to disk is faster than compressing it first and writing the result out. The former's performance for reading should also be better because of file cache.

The current implementation caches objects in one process, the objects can't be shared by many processes because they are uncompressed to heap memory area of each process.

Uncompressed loose objects are better for sharing objects among multiple git processes because they can be used directly after being mmap-ed.

And I guess the most frequently used objects are loose objects when you do some coding(git add, git diff, git diff --cached, git merge), using uncompressed loose objects avoids uncompressing loose objects again and again.

Below is the result of my simple benchmark:

########################################
# on my notebook
$ perl b.pl git-1.5.6/Makefile 1000
               Rate   compressed uncompressed
compressed    198/s           --         -92%
uncompressed 2463/s        1147%           --

$ perl b.pl git-1.5.6/parse-options.c 2000
               Rate   compressed uncompressed
compressed    341/s           --         -88%
uncompressed 2845/s         734%           --

$ find git-1.5.6/ -name "*.[ch]" -exec cat {} + > all.c
$ perl b.pl all.c 1000
               Rate   compressed uncompressed
compressed   3.39/s           --         -97%
uncompressed  111/s        3182%           --

#######################################
# on a server
$ perl b.pl Makefile 6000
            (warning: too few iterations for a reliable count)
                Rate   compressed uncompressed
compressed     447/s           --         -98%
uncompressed 18750/s        4094%           --

$ perl b.pl parse-options.c 8000
            (warning: too few iterations for a reliable count)
                Rate   compressed uncompressed
compressed    1130/s           --         -97%
uncompressed 33333/s        2850%           --

$ perl b.pl all.c 1000
               Rate   compressed uncompressed
compressed   5.48/s           --         -95%
uncompressed  115/s        1997%           

##################################################### # b.pl #!/usr/bin/perl use strict; use warnings; use Benchmark qw(:hireswallclock cmpthese); use File::Slurp; use IO::Compress::Deflate qw(deflate $DeflateError);

my $text = read_file($ARGV[0], binmode => ':raw');

cmpthese($ARGV[1], {'compressed' => \&zip, 'uncompressed' => \&output});

sub zip {
    deflate \$text => 'all.c.z' || die "$!\n";
}

sub output {
    write_file("all2.c", {binmode => ':raw'}, $text);
}