{"thread":{"id":"22486","subject":"Git and Amazon S3","startedAt":"2010-02-02T13:56:41Z","lastAt":"2010-02-03T12:05:42Z","messageCount":6,"participants":["Aneurin Price","Marc Weber","Tay Ray Chuan","Jay Soffian","Petr Baudis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"133353","messageId":"501db8661002020556k2f65add2rf06b289f2c9cbcac@mail.gmail.com","threadId":"22486","inReplyTo":null,"subject":"Git and Amazon S3","fromName":"Aneurin Price","fromEmail":"aneurin.price@gmail.com","sentAt":"2010-02-02T13:56:41Z","receivedAt":"2010-02-02T13:56:41Z","isPatch":false,"sender":{"key":"aneurin.price@gmail.com","avatar":null},"body":"Hello all,\n\nI have several (mostly private) git repositories which I'm using for various\npurposes including versioned backup of moderate amounts of data, and I'm trying\nto work out the cheapest way of having them remotely backed-up. Since I'm not\ndoing collaborative development, the git hosting options I've found aren't a\ngood fit - for this amount of data they tend to assume you must have a huge\nproject with many contributors, and charge accordingly.\n\nIt looks like the cheapest option from a pure storage and data-transfer point of\nview would be S3, so I'm looking at the best way to use it with git. So far, the\noptions I've found are either using jgit, which I've never used but appears to\nhave a native S3 transport, or using one of the FUSE options to mount S3 as a\nfilesystem.\n\nI'm not particularly happy about the idea of using jgit since it would require\njava on all the machines I might want to use it with, and it would mean learning\nto use a different command for fetch and push. It does have the bonus that it's\npossible to publish repositories for read access via dumb http though. On the\nother hand, I'm concerned about the fuse option because a) I'm not sure how\nreliable it is, and b) I'm concerned that the abstraction might leak if, for\nexample, git assumes that it is accessing a local filesystem and acts\ndifferently.\n\nDoes anyone have any remarks about these options? Is there a better option - how\ndifficult would it be to add native support to git? Are there any other options\nfor more git-friendly remote storage at a comparable price? Or maybe I should\njust give up, spend more and get a Linode; then I'd have the flexibility to do\nwhatever I want with it.\n\nThanks for your time,\nAneurin Price\n"},{"id":"133354","messageId":"1265119362-sup-1889@nixos","threadId":"22486","inReplyTo":"501db8661002020556k2f65add2rf06b289f2c9cbcac@mail.gmail.com","subject":"Re: Git and Amazon S3","fromName":"Marc Weber","fromEmail":"marco-oweber@gmx.de","sentAt":"2010-02-02T14:04:39Z","receivedAt":"2010-02-02T14:04:39Z","isPatch":false,"sender":{"key":"marco-oweber@gmx.de","avatar":null},"body":"> Does anyone have any remarks about these options? Is there a better option - how\n> difficult would it be to add native support to git? Are there any other options\n\nThere are also tools such as curlftpfs. Then you can mount any FTP\naccount. However git will be slow. There are some .git options.\nMaybe it's worth a try - Don't know excactly.\n\nThere should be fuse like filesystems for Amazon S3 as well. Google\nshowns some hits. Don't know which one works best.\n\nGood luck.\nMarc Weber\n"},{"id":"133357","messageId":"be6fef0d1002020701u77910be2k95815ad887d03524@mail.gmail.com","threadId":"22486","inReplyTo":"501db8661002020556k2f65add2rf06b289f2c9cbcac@mail.gmail.com","subject":"Re: Git and Amazon S3","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-02-02T15:01:07Z","receivedAt":"2010-02-02T15:01:07Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Tue, Feb 2, 2010 at 9:56 PM, Aneurin Price wrote:\n> Does anyone have any remarks about these options? Is there a better option - how\n> difficult would it be to add native support to git? Are there any other options\n> for more git-friendly remote storage at a comparable price? Or maybe I should\n> just give up, spend more and get a Linode; then I'd have the flexibility to do\n> whatever I want with it.\n\nbeing a backup, I believe you can just sync your .git folder one-way,\nwith your local copy being the authoritative one, and the S3 one just\nfollowing it.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"133365","messageId":"76718491002020834k5a666a2eh935d6aced9f8d216@mail.gmail.com","threadId":"22486","inReplyTo":"501db8661002020556k2f65add2rf06b289f2c9cbcac@mail.gmail.com","subject":"Re: Git and Amazon S3","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2010-02-02T16:34:49Z","receivedAt":"2010-02-02T16:34:49Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Tue, Feb 2, 2010 at 8:56 AM, Aneurin Price wrote:\n> It looks like the cheapest option from a pure storage and data-transfer point of\n> view would be S3, so I'm looking at the best way to use it with git. So far, the\n> options I've found are either using jgit, which I've never used but appears to\n> have a native S3 transport, or using one of the FUSE options to mount S3 as a\n> filesystem.\n>\n> Does anyone have any remarks about these options? Is there a better option -\n\nI guess it might exceed your costs, but you could use a small EC2\ninstance backed by an EBS volume. The instance would have git\ninstalled.\n\nWhen you need to push, fire up the instance, push to git running on\nthat instance, then shutdown the instance and snapshot the EBS volume\nto S3.\n\nHmm, maybe that's over-engineered. :-)\n\nj.\n"},{"id":"133380","messageId":"20100202194455.GO9553@machine.or.cz","threadId":"22486","inReplyTo":"501db8661002020556k2f65add2rf06b289f2c9cbcac@mail.gmail.com","subject":"Re: Git and Amazon S3","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2010-02-02T19:44:55Z","receivedAt":"2010-02-02T19:44:55Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Tue, Feb 02, 2010 at 01:56:41PM +0000, Aneurin Price wrote:\n> Does anyone have any remarks about these options? Is there a better option - how\n> difficult would it be to add native support to git?\n\n I'm not really familiar with Amazon S3 _or_ the current transport\ncode, but by cursory examination of both, it seems that it would be\nfairly easy to add support for another transfer. And it might be even\nbetter idea to actually just add generic support to invoke an external\nhelper to perform all the heavy lifting.\n\n Basically, all the abstraction is already pre-cooked in the form of\nrsync protocol support. I would just cut'n'paste that and replace rsync\nmagic with simple calls to external helper along some sensible simple\nAPI, then code up an easy wrapper for S3 there. Or just add S3 API\nsupport directly to core Git - it doesn't seem to be licence-encumbered.\nShould take just a couple of hours including debugging, if you just\ncopy the existing rsync support functions.\n\n Another idea might be to actually use the rsync protocol support\nitself. ;-) There seems to be some sort of commercial rsync-S3 interface,\nthough I can't tell from their terribly strange pricing policy how\nexpensive it is to use it in practice.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nIf you can't see the value in jet powered ants you should turn in\nyour nerd card. -- Dunbal (464142)\n"},{"id":"133467","messageId":"501db8661002030405h23e940e7r6e8fd0149818c80f@mail.gmail.com","threadId":"22486","inReplyTo":"20100202194455.GO9553@machine.or.cz","subject":"Re: Git and Amazon S3","fromName":"Aneurin Price","fromEmail":"aneurin.price@gmail.com","sentAt":"2010-02-03T12:05:42Z","receivedAt":"2010-02-03T12:05:42Z","isPatch":false,"sender":{"key":"aneurin.price@gmail.com","avatar":null},"body":"Thanks for the replies everyone.\n\nOn Tue, Feb 2, 2010 at 19:44, Petr Baudis wrote:\n> On Tue, Feb 02, 2010 at 01:56:41PM +0000, Aneurin Price wrote:\n>> Does anyone have any remarks about these options? Is there a better option - how\n>> difficult would it be to add native support to git?\n>\n> I'm not really familiar with Amazon S3 _or_ the current transport\n> code, but by cursory examination of both, it seems that it would be\n> fairly easy to add support for another transfer. And it might be even\n> better idea to actually just add generic support to invoke an external\n> helper to perform all the heavy lifting.\n>\n> Basically, all the abstraction is already pre-cooked in the form of\n> rsync protocol support. I would just cut'n'paste that and replace rsync\n> magic with simple calls to external helper along some sensible simple\n> API, then code up an easy wrapper for S3 there. Or just add S3 API\n> support directly to core Git - it doesn't seem to be licence-encumbered.\n> Should take just a couple of hours including debugging, if you just\n> copy the existing rsync support functions.\n>\n\nThis sounds like the most interesting option, if not necessarily the most\npractical. I've also discovered s3cmd (http://s3tools.org/s3cmd) which seems to\nbe widely packaged and could probably serve nicely as that wrapper.\n\nIf I can manage to get those couple of hours free at some point I'll give it a\ngo. Is this something that might be a mergeable feature?\n\nThanks again,\nAneurin Price\n"}]}