Re: Update on SoC proposal: git-remote-svn

From: David Michael Barr <david.barr@cordelta.com>
Date: Apr 14, 2010, 17:15 UTC
Message-ID: <21A95727-F438-48FB-9A16-132B4DE85B6E@cordelta.com>
In-Reply-To: <F0CD5B83-2D28-47E1-A336-5C88E2803CBE@gmail.com>

Hi Steve,

> If I were to check out the svn root of that repository, I would use well
> over 3TB of disk space to have that checked out ...

This stirred my thoughts and I whipped up a bash script that uses SVK, find, shasum and ln to build a filesystem view of the root of an svn repository that consumes moderate storage:

SVK_DEPOT="" MAX_REV=12345

CO_DIR=validation HASH_DIR=hashes

svk co -r1 /$SVK_DEPOT/ $CO_DIR
mkdir -p $HASH_DIR
for (( REV=1 ; REV<=MAX_REV ; ++REV )) do
  svk up -r$REV $CO_DIR
  # Hashify working copy
  find $CO_DIR -type d -cmin -5 -prune -o \
    -type f -links 1 -exec shasum '{}' + | (
    while read HASH FILE ; do
      [ -x "$FILE" ] && HASH="$HASH"x
      ln "$FILE" $HASH_DIR/$HASH 2>/dev/null || \
        ln -f $HASH_DIR/$HASH "$FILE"
    done
  )
done

Important assumptions are that each update will take less than 5 minutes and that SVK uses writes to a temporary file and then renames to perform a modification. I've used this to build a simple validation script for my project. I estimate it will use about 20GB to represent my 1GB repo and that it will take about 3 hours.

-- David Barr