git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]

From: Tarmigan <tarmigan+git@gmail.com>
Date: Jul 8, 2008, 05:48 UTC
Message-ID: <905315640807072248w44ccdc4y2f1cf54a10c50c43@mail.gmail.com>

(trimmed cc list to folks who've touched http-push.c)

On Mon, Jul 7, 2008 at 9:19 PM, Shawn O. Pearce <spearce@spearce.org> wrote:

Show 9 quoted lines

> Using Marek's pack generation code I added support for push over
> the dumb sftp:// and amazon-s3:// protocols, with the latter also
> supporting transparent client side encryption.
>
> I chose to add these features to jgit partly as an exercise to prove
> that Marek's code was built well enough to be reused for this task,
> partly because I wanted to backup some private personal repositories
> to Amazon S3, and partly to prove that multiple dumb transports
> could implement push support.

That sounds cool. I've been looking into adding s3 push into cgit, and was looking into modifying http-push.c, but got in over my head. I had trouble trying to make it fit into the DAV model that http-push is built around, in part because s3 doesn't seem to support any locking and a lot of the http-push code seems to be around the locking.

Can you describe the s3 support that you added? Did you do any locking when you pushed? The objects and packs seem likely to be naturally OK, but I was worried about refs/ and especially objects/info/packs and info/refs (fetch over http works currently out of the box with publicly accessable s3 repos).

Thanks, Tarmigan

PS For anyone else who's interested, here's some instructions on how I got started with s3 and git:

Start by creating an amazon s3 account

Next download and install "aws" from http://timkay.com/aws/ Set it up and install your amazon keys as specified.

# I setup a bucket named git_test. s3mkdir git_test

#  Run this script to upload a git repo to amazon (run
update-server-info first):
#!/bin/bash
for i in $(tree -fi --noreport git_test_orig.git) ; do
    #exclude directories
    if [ ! -d $i ] ; then
        echo $i
        s3put "x-amz-acl: public-read" git_test/ $i
    fi
done

# Then you can clone (really, feel free to clone from this url. It should just work): git clone http://s3.amazonaws.com/git_test/git_test_orig.git

# This experimenting on the git.git repo will set you back about US$0.05