Re: git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]

Tarmigan <tarmigan+git@gmail.com> wrote:

It behaves like http push does in C git in that it is pretty transparent to the end-user:

  # Create a bucket using other S3 tools.
  # I used jets3t's cockpit tool to creat "gitney".

  # Create a configuration file for jgit's S3 client:
  #
  $ touch ~/.jgit_s3_public
  $ chmod 600 ~/.jgit_s3_public
  $ cat >>~/.jgit_s3_public
  accesskey: AWSAccessKeyId
  secretkey: AWSSecretAccessKey
  acl: public
  EOF

  # Configure the remote and push
  #
  $ git remote add s3 amazon-s3://.jgit_s3_public@gitney/projects/egit.git/
  $ jgit push s3 refs/heads/master
  $ jgit push --tags s3

  # Future incremental updates are just as easy
  #
  $ jgit push s3 refs/heads/master

  (or)
  $ git config remote.s3.push refs/heads/master
  $ jgit push s3

This is now cloneable[*1*]:

  $ git clone http://gitney.s3.amazonaws.com/projects/egit.git

Pushes are incremental, rather than the approach you outlined, as that causes a full re-upload of the repository. Consequently there is relatively little bandwidth usage during subsequent pushes.

A jgit amazon-s3 URL is organized as:

  amazon-s3://$config@$bucket/$prefix

where:

  $config = path to configuration in $GIT_DIR/$config or $HOME/$config
  $bucket = name of the Amazon S3 bucket holding the objects
  $prefix = prefix to apply to all objects, implicitly ends in "/"

Amazon S3 atomically replaces a file, but offers no locking support. Our crude remote VFS abstraction offers two types of file write operations:

  - Atomic write for small (in-memory) files <~1M
  - Stream write for large (e.g. pack) files >~1M

In the S3 implementation both operations are basically the same code, since even large streams are atomic updates. But in sftp:// our remote VFS writes to a "$path.lock" for the atomic case and renames to "$path". This is not the same as a real lock, but it avoids readers from seeing an in-progress update.

We are very carefully to order the update operations to try and avoid any sorts of race conditions:

  - Delete loose refs which are being deleted.
  - Upload new pack:
    - If same pack already exists:
      - (atomic) Remove it from objects/info/packs
      - Delete .idx
      - Delete .pack
    - Upload new .pack
    - Upload new .idx
    - (atomic) Add to front of objects/info/packs.
  - (atomic) Create/update loose refs.
  - (atomic) Update (if necessary) packed-refs.
  - (atomic) Update info/refs.

Since we are pushing over a dumb transport we assume readers are pulling over the same dumb transport and thus rely upon the objects/info/packs and info/refs files to obtain the listing of what is available. This isn't true though for jgit's sftp:// and amazon-s3:// protocols as both support navigation of the objects/packs and refs/{heads,tags} tree directly.

Locking on S3 is difficult. Multi-object writes may not sync across the S3 cluster immediately. This means you can write to A, then to B, then read A and see the old content still there, then seconds later read A again and see the new content suddenly arrive. It all depends upon when the replicas update and which replica the load-balancer sends you into during the request. So despite our attempts to order writes to S3 it is still possible for an S3 write to appear "late" and for a client to see a ref in info/refs for which the corresponding pack is not listed in object/info/packs.

However, this is the same mirroring problem that kernel.org has for its git trees. I believe they are moved out to the public mirrors by dumb rsync and not some sort of smart git-aware transport. As rsync is free to order the writes out of order kernel.org has the same issue. ;-)

Actually I suspect the S3 replica update occurs more quickly than the kernel.org mirrors update, so the window under which a client can see out-of-order writes is likely smaller.

*1* You need a bug fix in jgit to correctly initialize HEAD during
    push to a new, non-existant repository stored on S3.  The patch
    is going to be posted later this evening, its still in my tree.