Hi all,
I've seen the same effect, so i tried to reduce the size of the revdb and made a new format: First, in the bin files the sha1 are stored as hexvalues not as ascii, this reduces the a single sha1 from 41 bytes to 20. Second, only save the non-zero commits, thats what the idx are used for. A idx file has three 32bit integers per entry. The first integer represents the first zero svn revision, the second the last zero revision and the last integer is the position of the next non-zero revision in the bin.
Example: Revision 0-373006 are zero revision and 373007 is the first actualy used revision and 373008-373623 are again zero revisions the idx has the following content: 0 373006 0 373007 373007 1
and the bin only saves 59037b8043268c9ca0d87ba86519ed0b5358c8a1 eef3f7e25993a46e3c4242aa502d93e909b08c57
The format currently used produce a 373624*41bytes large file.
Used on a git-svn clone here, i get: The results are: old: 1,1G hadoop (1004M svn/) new: 47M hadoop (5,9M svn/)
in detail:
.git/svn/trunk: old: -rw-r--r-- 1 david david 24M 2007-11-29 10:26 .rev_db.13f79535-47bb-0310-9956-ffa450edef68 -rw-r--r-- 1 david david 75K 2007-11-29 10:26 .rev_db.7fecf15c-03ad-4724-994c-e980afa7160c new: -rw-r--r-- 1 david david 32K 2007-12-03 18:40 revdb-13f79535-47bb-0310-9956-ffa450edef68.bin -rw-r--r-- 1 david david 18K 2007-12-03 18:40 revdb-13f79535-47bb-0310-9956-ffa450edef68.idx -rw-r--r-- 1 david david 32K 2007-12-03 18:44 revdb-7fecf15c-03ad-4724-994c-e980afa7160c.bin -rw-r--r-- 1 david david 2,0K 2007-12-03 18:44 revdb-7fecf15c-03ad-4724-994c-e980afa7160c.idx
Here a example sourcecode to test this idea:
I try to integrate this in git-svn this week.
NOTE: I'm not a perl hacker, so use at your own risk.
Bye David ps.: I'm not a member of this list please reply directly to me.
migrate.pl: $uuid = "7fecf15c-03ad-4724-994c-e980afa7160c";
open (NONZERO, '.rev_db.'.$uuid); open (IDX, '>revdb-'.$uuid.'.idx'); open (BIN, '>revdb-'.$uuid.'.bin');
$first_zero = 0; $pos = 0; $rev = 0;
while ($sha1 = <NONZERO>) {chomp($sha1);
if ($sha1 ne ("0" x 40))
{
print BIN pack("H40", $sha1); if ($first_zero != $rev)
{
print IDX pack("N N N", $first_zero, ($rev-1), $pos);
} $first_zero=$rev+1;
$pos++;
}$rev++;
}
close(BIN); close(IDX); close(NONZERO);
parse.pl: use strict; use Fcntl;
my(@index, $buf, $i); my($uuid) = "13f79535-47bb-0310-9956-ffa450edef68"; my($db_path) = "revdb-$uuid.bin";
sysopen(IDX, "revdb-$uuid.idx", O_RDONLY);
while (sysread(IDX, $buf, 12)) {
my($minrev, $maxrev, $pos) = unpack("N N N", $buf); push @index, [$minrev, $maxrev, $pos];
}
close(IDX);
my($lastindex) = scalar(@index)-1; my($lastindexpos) = $index[$lastindex][2]; my($lastindexrev) = $index[$lastindex][1];
my @stat = stat $db_path; ($stat[7] % 20) == 0 or die "$db_path inconsistent size: $stat[7]\n"; my ($maxrev) = ($stat[7]/20)-($lastindexpos)+($lastindexrev); my ($minrev) = $index[0][1]+1;
my($cachestep) = int((scalar(@index))/9);
my(@cache);
for (my($i)=0; $i < scalar(@index); $i += $cachestep) {
push @cache, [$index[$i][0], $i];
}my($lastsearch) = 0;
sub pos2sha1 {
my($pos) = @_;
sysopen(BINDB, $db_path, O_RDONLY); sysseek(BINDB, $pos, 0);
my($buf);
sysread(BINDB, $buf, 20);
return unpack ("H40", $buf); close(BINDB);
}
sub get_sha1 {
my($rev) = @_;
my($i) = 0; if (($rev <= 0) || ($rev > $maxrev) || $rev <= $index[0][1]) {
return ("0" x 40);
} if ($rev > $lastindexrev) {
my($pos) = (((($rev-1) - $lastindexrev)+$lastindexpos))*20; return pos2sha1($pos);
}
if(($rev >= $index[$lastsearch][0] && $rev <= $index[$lastsearch][1]) ||
($rev >= $index[$lastsearch+1][0] && $rev <= $index[$lastsearch+1][1])) {
return ("0" x 40);
}
elsif ($rev > $index[$lastsearch][1] && $rev < $index[$lastsearch+1][0]) {
my($pos) = (($rev-1) - $index[$lastsearch][1] + $index[$lastsearch][2]) * 20; return pos2sha1($pos);
}
elsif($rev > $index[$lastsearch+1][1] && $rev < $index[$lastsearch+2][0]) {
$lastsearch++; my($pos) = (($rev-1) - $index[$lastsearch][1] + $index[$lastsearch][2]) * 20;
return pos2sha1($pos);
}
elsif($lastsearch != 0 && $rev > $index[$lastsearch-1][1] && $rev <
$index[$lastsearch][0]) {
$lastsearch--; my($pos) = (($rev-1) - $index[$lastsearch][1] + $index[$lastsearch][2]) * 20;
return pos2sha1($pos);
}
my($l, $r);
$l = 0;
$r = scalar(@index)-1;
my($lastcache) = scalar(@cache)-1;
for (my($i)=0; $i <= $lastcache; $i++) {
if ($rev >= $cache[$i][0]) {
$l = $cache[$i][1];
} if ($rev < $cache[$lastcache-$i][0]) {
$r = $cache[$lastcache-$i][1];
}
} if ($rev <= $index[$l][1]) {
return ("0" x 40);
} while ($l <= $r) {
$i = int(($l + $r)/2); if ($rev >= $index[$i][0] && $rev <= $index[$i][1]) {
$lastsearch = $i; return ("0" x 40);
}
elsif ($rev <= $index[$i][0]) {
$r = $i-1;
}
elsif ($rev >= $index[$i+1][0]) {
$l = $i+1;
}
else {
$lastsearch = $i;
my($pos) = (($rev-1) - $index[$i][1] + $index[$i][2]) * 20; return pos2sha1($pos);
}
} return ("0" x 40);
}for (my($i)=$maxrev; $i >= $minrev; $i--) {
my($sha1) = get_sha1($i);
if ($sha1 ne ("0" x 40)) {
print "$i = $sha1\n";
}
}
