{"thread":{"id":"14342","subject":"git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]","startedAt":"2008-07-08T05:48:59Z","lastAt":"2008-07-09T03:26:34Z","messageCount":4,"participants":["Tarmigan","Mike Hommey","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"82557","messageId":"905315640807072248w44ccdc4y2f1cf54a10c50c43@mail.gmail.com","threadId":"14342","inReplyTo":null,"subject":"git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]","fromName":"Tarmigan","fromEmail":"tarmigan+git@gmail.com","sentAt":"2008-07-08T05:48:59Z","receivedAt":"2008-07-08T05:48:59Z","isPatch":false,"sender":{"key":"tarmigan+git@gmail.com","avatar":null},"body":"(trimmed cc list to folks who've touched http-push.c)\n\nOn Mon, Jul 7, 2008 at 9:19 PM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Using Marek's pack generation code I added support for push over\n> the dumb sftp:// and amazon-s3:// protocols, with the latter also\n> supporting transparent client side encryption.\n>\n> I chose to add these features to jgit partly as an exercise to prove\n> that Marek's code was built well enough to be reused for this task,\n> partly because I wanted to backup some private personal repositories\n> to Amazon S3, and partly to prove that multiple dumb transports\n> could implement push support.\n\nThat sounds cool.  I've been looking into adding s3 push into cgit,\nand was looking into modifying http-push.c, but got in over my head.\nI had trouble trying to make it fit into the DAV model that http-push\nis built around, in part because s3 doesn't seem to support any\nlocking and a lot of the http-push code seems to be around the\nlocking.\n\nCan you describe the s3 support that you added?  Did you do any\nlocking when you pushed?  The objects and packs seem likely to be\nnaturally OK, but I was worried about refs/ and especially\nobjects/info/packs and info/refs (fetch over http works currently out\nof the box with publicly accessable s3 repos).\n\nThanks,\nTarmigan\n\nPS For anyone else who's interested, here's some instructions on how I\ngot started with s3 and git:\n\nStart by creating an amazon s3 account\n\nNext download and install \"aws\" from http://timkay.com/aws/\nSet it up and install your amazon keys as specified.\n\n# I setup a bucket named git_test.\ns3mkdir git_test\n\n#  Run this script to upload a git repo to amazon (run\nupdate-server-info first):\n#!/bin/bash\nfor i in $(tree -fi --noreport git_test_orig.git) ; do\n    #exclude directories\n    if [ ! -d $i ] ; then\n        echo $i\n        s3put \"x-amz-acl: public-read\" git_test/ $i\n    fi\ndone\n\n# Then you can clone (really, feel free to clone from this url.  It\nshould just work):\ngit clone http://s3.amazonaws.com/git_test/git_test_orig.git\n\n# This experimenting on the git.git repo will set you back about US$0.05\n"},{"id":"82558","messageId":"20080708055610.GA12591@glandium.org","threadId":"14342","inReplyTo":"905315640807072248w44ccdc4y2f1cf54a10c50c43@mail.gmail.com","subject":"Re: git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2008-07-08T05:56:10Z","receivedAt":"2008-07-08T05:56:10Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Mon, Jul 07, 2008 at 10:48:59PM -0700, Tarmigan wrote:\n> (trimmed cc list to folks who've touched http-push.c)\n> \n> On Mon, Jul 7, 2008 at 9:19 PM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> > Using Marek's pack generation code I added support for push over\n> > the dumb sftp:// and amazon-s3:// protocols, with the latter also\n> > supporting transparent client side encryption.\n> >\n> > I chose to add these features to jgit partly as an exercise to prove\n> > that Marek's code was built well enough to be reused for this task,\n> > partly because I wanted to backup some private personal repositories\n> > to Amazon S3, and partly to prove that multiple dumb transports\n> > could implement push support.\n> \n> That sounds cool.  I've been looking into adding s3 push into cgit,\n> and was looking into modifying http-push.c, but got in over my head.\n> I had trouble trying to make it fit into the DAV model that http-push\n> is built around, in part because s3 doesn't seem to support any\n> locking and a lot of the http-push code seems to be around the\n> locking.\n> \n> Can you describe the s3 support that you added?  Did you do any\n> locking when you pushed?  The objects and packs seem likely to be\n> naturally OK, but I was worried about refs/ and especially\n> objects/info/packs and info/refs (fetch over http works currently out\n> of the box with publicly accessable s3 repos).\n\nFWIW, I'm starting to work again on the http backend overhaul. My idea\nis to provide a generic dumb protocol vfs-like interface, so that other\ndumb protocols could be built out of it.\n\nMike\n"},{"id":"82671","messageId":"20080709032222.GA18520@spearce.org","threadId":"14342","inReplyTo":"905315640807072248w44ccdc4y2f1cf54a10c50c43@mail.gmail.com","subject":"Re: git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-07-09T03:22:22Z","receivedAt":"2008-07-09T03:22:22Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Tarmigan <tarmigan+git@gmail.com> wrote:\n> (trimmed cc list to folks who've touched http-push.c)\n> On Mon, Jul 7, 2008 at 9:19 PM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> > Using Marek's pack generation code I added support for push over\n> > the dumb sftp:// and amazon-s3:// protocols, with the latter also\n> > supporting transparent client side encryption.\n> \n> Can you describe the s3 support that you added?  Did you do any\n> locking when you pushed?  The objects and packs seem likely to be\n> naturally OK, but I was worried about refs/ and especially\n> objects/info/packs and info/refs (fetch over http works currently out\n> of the box with publicly accessable s3 repos).\n\nIt behaves like http push does in C git in that it is pretty\ntransparent to the end-user:\n\n  # Create a bucket using other S3 tools.\n  # I used jets3t's cockpit tool to creat \"gitney\".\n\n  # Create a configuration file for jgit's S3 client:\n  #\n  $ touch ~/.jgit_s3_public\n  $ chmod 600 ~/.jgit_s3_public\n  $ cat >>~/.jgit_s3_public\n  accesskey: AWSAccessKeyId\n  secretkey: AWSSecretAccessKey\n  acl: public\n  EOF\n\n  # Configure the remote and push\n  #\n  $ git remote add s3 amazon-s3://.jgit_s3_public@gitney/projects/egit.git/\n  $ jgit push s3 refs/heads/master\n  $ jgit push --tags s3\n\n  # Future incremental updates are just as easy\n  #\n  $ jgit push s3 refs/heads/master\n\n  (or)\n  $ git config remote.s3.push refs/heads/master\n  $ jgit push s3\n\nThis is now cloneable[*1*]:\n\n  $ git clone http://gitney.s3.amazonaws.com/projects/egit.git\n\nPushes are incremental, rather than the approach you outlined, as\nthat causes a full re-upload of the repository.  Consequently there\nis relatively little bandwidth usage during subsequent pushes.\n\nA jgit amazon-s3 URL is organized as:\n\n  amazon-s3://$config@$bucket/$prefix\n\nwhere:\n\n  $config = path to configuration in $GIT_DIR/$config or $HOME/$config\n  $bucket = name of the Amazon S3 bucket holding the objects\n  $prefix = prefix to apply to all objects, implicitly ends in \"/\"\n\n\nAmazon S3 atomically replaces a file, but offers no locking support.\nOur crude remote VFS abstraction offers two types of file write\noperations:\n\n  - Atomic write for small (in-memory) files <~1M\n  - Stream write for large (e.g. pack) files >~1M\n\nIn the S3 implementation both operations are basically the same\ncode, since even large streams are atomic updates.  But in sftp://\nour remote VFS writes to a \"$path.lock\" for the atomic case and\nrenames to \"$path\".  This is not the same as a real lock, but it\navoids readers from seeing an in-progress update.\n\nWe are very carefully to order the update operations to try and\navoid any sorts of race conditions:\n\n  - Delete loose refs which are being deleted.\n  - Upload new pack:\n    - If same pack already exists:\n      - (atomic) Remove it from objects/info/packs\n      - Delete .idx\n      - Delete .pack\n    - Upload new .pack\n    - Upload new .idx\n    - (atomic) Add to front of objects/info/packs.\n  - (atomic) Create/update loose refs.\n  - (atomic) Update (if necessary) packed-refs.\n  - (atomic) Update info/refs.\n\nSince we are pushing over a dumb transport we assume readers\nare pulling over the same dumb transport and thus rely upon the\nobjects/info/packs and info/refs files to obtain the listing of\nwhat is available.  This isn't true though for jgit's sftp://\nand amazon-s3:// protocols as both support navigation of the\nobjects/packs and refs/{heads,tags} tree directly.\n\nLocking on S3 is difficult.  Multi-object writes may not sync\nacross the S3 cluster immediately.  This means you can write to A,\nthen to B, then read A and see the old content still there, then\nseconds later read A again and see the new content suddenly arrive.\nIt all depends upon when the replicas update and which replica\nthe load-balancer sends you into during the request.  So despite\nour attempts to order writes to S3 it is still possible for an S3\nwrite to appear \"late\" and for a client to see a ref in info/refs\nfor which the corresponding pack is not listed in object/info/packs.\n\nHowever, this is the same mirroring problem that kernel.org has for\nits git trees.  I believe they are moved out to the public mirrors\nby dumb rsync and not some sort of smart git-aware transport.\nAs rsync is free to order the writes out of order kernel.org has\nthe same issue.  ;-)\n\nActually I suspect the S3 replica update occurs more quickly than\nthe kernel.org mirrors update, so the window under which a client\ncan see out-of-order writes is likely smaller.\n\n\n*1* You need a bug fix in jgit to correctly initialize HEAD during\n    push to a new, non-existant repository stored on S3.  The patch\n    is going to be posted later this evening, its still in my tree.\n\n-- \nShawn.\n"},{"id":"82672","messageId":"20080709032634.GB18520@spearce.org","threadId":"14342","inReplyTo":"20080708055610.GA12591@glandium.org","subject":"Re: git push to amazon s3 [was: [GSoC] What is status of Git's Google Summer of Code 2008 projects?]","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-07-09T03:26:34Z","receivedAt":"2008-07-09T03:26:34Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Mike Hommey <mh@glandium.org> wrote:\n> FWIW, I'm starting to work again on the http backend overhaul. My idea\n> is to provide a generic dumb protocol vfs-like interface, so that other\n> dumb protocols could be built out of it.\n\njgit has a vfs abstraction for the different dumb protocols.  Not sure\nif you would find it of any value to read as we are also able to use a\nnumber of Java standard abstractions like InputStream/OutputStream,\nbut here it is:\n\n  WalkRemoteObjectDatabase:\n  http://repo.or.cz/w/egit.git?a=blob;f=org.spearce.jgit/src/org/spearce/jgit/transport/WalkRemoteObjectDatabase.java;h=915faac9eb85e59c0ed2c08b9631d03cbc4c6bf8;hb=8d085723b260f3b51a70d11b723608779160b090\n\nThus far this abstraction has worked for sftp:// and amazon-s3://.\nWebDAV may make it more complicated due to locking being available\n(and something we would want to use to protect writes) but S3 uses\nHTTP PUT much like DAV does to upload content so there wouldn't be\ntoo many changes to actually implement DAV support.\n\n-- \nShawn.\n"}]}