{"thread":{"id":"30430","subject":"recipe to use git for deployment","startedAt":"2012-05-05T03:51:21Z","lastAt":"2012-05-16T20:08:00Z","messageCount":9,"participants":["Neal Kreitzinger","Marc Weber","Junio C Hamano","Ævar Arnfjörð Bjarmason","Sitaram Chamarty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"190817","messageId":"jo283q$kna$1@dough.gmane.org","threadId":"30430","inReplyTo":null,"subject":"recipe to use git for deployment","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2012-05-05T03:51:21Z","receivedAt":"2012-05-05T03:51:21Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"I'm trying to cook up automated mass deployment using git as the main \ningredient.  Here's a recipe idea:\n\ninstall git on deployed server under the home dir of the 'gittech' user \nand add that path only to the PATH of 'gittech'.\n\nhave the git repos on deployed server be under the 'gittech' home dir \nwith worktree(s) assigned to deployment locations.  If people mess with \nthe worktrees I will be able to tell with git status via 'gittech'.\n\nI can ssh to deployed servers from a deployment server and do git fetch \nto get updates.\n\nI can have a 'deployment manager' repo which contains script to apply \npatches from remote tracking branch to worktree.  It checks out commits \nat waypoints and at waypoint it executes needed file/data conversion \nprograms, creates dir/file structures, etc.  If conversion validation \nprogram is successful it will continue to checkout next waypoint, etc, \nuntil worktree matches HEAD of remote tracking branch.\n\nData/files are commited to data tracking git repo at waypoints.  If \nwaypoint validation fails then software version repo and data version \nrepo are checked out to last good waypoint or back to startpoint.\n\nProblem(1): cron patches.  The cron jobs portion I'm not sure of.  Need \nto learn more about cron.  Perhaps cron can run scripts that are \nsymlinks to a worktree.  Cron worktree can have waypoints just like \nsoftware and data worktrees.  The cron tabs, cron.daily, etc can be \nworktrees perhaps also.\n\nProblem(2a): diskspace.  I think I will have to pull shallow clones to \navoid growing history.  For deployment I only care about recent \nrevisions.  Not sure if the above ideas will work with shallow clone.\n\nProblem(2b): network latency.  Maybe I can have the git repo object \nstore on the deployment server and the deployed git repo reference it to \nsave diskpace on the deployed server.  That would probably be really slow.\n\nI'm thinking the ingredients of this recipe could make a tasty git \ndeployment.  Maybe different ways of mixing and cooking them and other \ningredients can make a good dish.  Please let me know if you think this \ncan be edible for security, palatable for reliability, and digestible by \ngit.\n\nv/r,\nneal\n"},{"id":"190818","messageId":"1336190286-sup-3813@nixos","threadId":"30430","inReplyTo":"jo283q$kna$1@dough.gmane.org","subject":"Re: recipe to use git for deployment","fromName":"Marc Weber","fromEmail":"marco-oweber@gmx.de","sentAt":"2012-05-05T04:10:46Z","receivedAt":"2012-05-05T04:10:46Z","isPatch":false,"sender":{"key":"marco-oweber@gmx.de","avatar":null},"body":"I always did \"git pull\" on the servers - but the history of my projects\nis not that huge.. Thus I never cared.\nGreat: You can keep some server specific config settings and rebase\nthem - and you can do a fast git status to check whether file contents\nhave been modified (eg determine whether you've been hacked ..)\n\nIf you really care that much about history why not push a zip file using\ngit archive --format=zip and unpack that on the deployment server\ninstead?\n\nFor FTP access only this did a great job for small projects:\nhttps://github.com/MarcWeber/git-ftp (-> git-ftp-minimal.sh)\nIt only copies changed files *and* checks whether they have been\nmodified on the server first (detecting work of others).\nBut that's probably not a good thing for automatic deployment.\n\nMarc Weber\n"},{"id":"190821","messageId":"7v4nrv8czo.fsf@alter.siamese.dyndns.org","threadId":"30430","inReplyTo":"jo283q$kna$1@dough.gmane.org","subject":"Re: recipe to use git for deployment","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-05T04:31:55Z","receivedAt":"2012-05-05T04:31:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Neal Kreitzinger <nkreitzinger@gmail.com> writes:\n\n> I'm trying to cook up automated mass deployment using git as the main\n> ingredient.\n\nI think the standard answer to this is \"don't do it\".\n\nYou could of course add hooks to hack around it, but fundamentally Git was\ndesigned to track source files (so the only sane behaviour for \"checking\nout\" is to keep the timestamp of the working tree file to whatever the\nunderlying OS gives it when \"checkout\" happens, for example), so anything\nthat your \"deployment strategy\" wants conflicts with what the usual source\ncoutrol operation should do, you need to tweak around what Git does (the\nsame thing can be said for CVS or SVN, by the way).\n\nUnless you do this with an understanding that you are merely using git \"as\na better rsync\" and take responsiblity of everything else that git-the-scm\ndoes _not_ do for you, I suspect that you would be volunteering for a lot\nof pain.\n"},{"id":"190822","messageId":"4FA4AFE4.6040701@gmail.com","threadId":"30430","inReplyTo":"1336190286-sup-3813@nixos","subject":"Re: recipe to use git for deployment","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2012-05-05T04:43:16Z","receivedAt":"2012-05-05T04:43:16Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 5/4/2012 11:10 PM, Marc Weber wrote:\n> I always did \"git pull\" on the servers - but the history of my projects\n> is not that huge.. Thus I never cared.\n> Great: You can keep some server specific config settings and rebase\n> them -\nSo I could rebase apache virtual host conf files with generic template \nchanges like symlink support, but keep the networking ip's.  Merge \nconflicts would be bad.  I guess that would be cause to reset --hard to \na previous commit.\n\nand you can do a fast git status to check whether file contents\n> have been modified (eg determine whether you've been hacked ..)\n>\nWith 'source is run' like html, css, etc. that would be a good way to \nenforce no-hacking-allowed support agreements.\n\n> If you really care that much about history why not push a zip file using\n> git archive --format=zip and unpack that on the deployment server\n> instead?\n>\nThat is what I'm currently doing.  By paying attention to tracked \nexecutable bits I don't even have to fix permissions on deployment.\n\nThe scriptable patching of a git repo (scripted checkout of remote \ntracking branch commits to worktree with inserted conversions and \nvalidations -- an interactive cherry-pick so to speak) allows for \nflexible conversions of varying patch levels.  Maybe git-sequencer will \ndo this when it comes out.\n\nAlso, I could have a deployment tracking repo on one of my servers that \nremotes to all deployed servers and pulls to refresh my monitor view of \nall deployed revisions in gitk.\n\n> For FTP access only this did a great job for small projects:\n> https://github.com/MarcWeber/git-ftp (->  git-ftp-minimal.sh)\n> It only copies changed files *and* checks whether they have been\n> modified on the server first (detecting work of others).\n> But that's probably not a good thing for automatic deployment.\n>\nI'll take a look to see what I can steal, uh I mean reuse.  Thanks!\n\nv/r,\nneal\n"},{"id":"190823","messageId":"1336194247-sup-6608@nixos","threadId":"30430","inReplyTo":"4FA4AFE4.6040701@gmail.com","subject":"Re: recipe to use git for deployment","fromName":"Marc Weber","fromEmail":"marco-oweber@gmx.de","sentAt":"2012-05-05T05:04:15Z","receivedAt":"2012-05-05T05:04:15Z","isPatch":false,"sender":{"key":"marco-oweber@gmx.de","avatar":null},"body":"I've been deploying .php files this way only..\n\nIf you want to deploy full configuration of hosts you may also want to\nlearn about nixos.org (which handles restarting apache etc) or chef\nlike systemms:\nhttp://www.rubyinside.com/chef-tasty-server-configuraiton-2162.html\nusing nixos you can easily automatically generate additional cron job\nensuring that permissions are correct etc.\n\nMaybe talking about your exact use case will make others list more\noptions.\n\nMarc Weber\n"},{"id":"190825","messageId":"4FA4BAF3.10601@gmail.com","threadId":"30430","inReplyTo":"1336190286-sup-3813@nixos","subject":"Re: recipe to use git for deployment","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2012-05-05T05:30:27Z","receivedAt":"2012-05-05T05:30:27Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 5/4/2012 11:10 PM, Marc Weber wrote:\n>\n> If you really care that much about history why not push a zip file\n> using git archive --format=zip and unpack that on the deployment\n> server instead?\n>\nI summon forth the ents to do my bidding\nwith one command to find them:  --format=tar\none command to transform them:  --prefix=deployed/path\none command to arm them:  config tar.umask (executable bits)\none command to gather them:  tar -A\nand one command to bind them:  gzip\n\n(We don't have the new tar with the transform option)\n\nThey are faithful and true and have won many battles, but I fear they \nmay be too slow and lumbering for the gathering whirlwind of patches \nabout to descend from above and legion of servers encircling them on all \nsides.\n\nInterestingly, git-archive was not intended for deployment, but it does \npretty good at it.\n\n(Sorry, I haven't even read LOTR, but couldn't resist the opportunity to \nuse the word 'ent' after reading about it in git-glossary ;)\n\nv/r,\nneal\n"},{"id":"190828","messageId":"CACBZZX50LAu+XNWzHJjQB1kgK3W418FVW--DdsYchDvg8bEztw@mail.gmail.com","threadId":"30430","inReplyTo":"jo283q$kna$1@dough.gmane.org","subject":"Re: recipe to use git for deployment","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2012-05-05T08:25:55Z","receivedAt":"2012-05-05T08:25:55Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, May 5, 2012 at 5:51 AM, Neal Kreitzinger <nkreitzinger@gmail.com> wrote:\n> I'm trying to cook up automated mass deployment using git as the main\n> ingredient.  Here's a recipe idea:\n\nAt work we use and have open sourced\nhttps://github.com/git-deploy/git-deploy which we use for our\ndeployments.\n\nIt doesn't actually do the work of syncing out anything, we\neffectively do that with distributed rsync.\n"},{"id":"190837","messageId":"CAMK1S_i0bsO0fp69Ae_75dySsN9r=z1+c2psEMm4urw+UpYGXw@mail.gmail.com","threadId":"30430","inReplyTo":"4FA4BAF3.10601@gmail.com","subject":"Re: recipe to use git for deployment","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2012-05-05T12:14:16Z","receivedAt":"2012-05-05T12:14:16Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Sat, May 5, 2012 at 11:00 AM, Neal Kreitzinger\n<nkreitzinger@gmail.com> wrote:\n\n> Interestingly, git-archive was not intended for deployment, but it does\n> pretty good at it.\n\nExcept it doesn't delete files that should be deleted (the way 'git\ncheckout' does or rsync would if you gave it --delete).\n\nOn a lark I once wrote a very efficient way of dealing with this; see\nattached.  It's probably won't work on anything but Linux and requires\nbash, and it's only been minimally tested and I do not actually use\nit, but someone might like it.\n"},{"id":"191605","messageId":"4FB40920.6010106@gmail.com","threadId":"30430","inReplyTo":"CAMK1S_i0bsO0fp69Ae_75dySsN9r=z1+c2psEMm4urw+UpYGXw@mail.gmail.com","subject":"Re: recipe to use git for deployment","fromName":"Neal Kreitzinger","fromEmail":"nkreitzinger@gmail.com","sentAt":"2012-05-16T20:08:00Z","receivedAt":"2012-05-16T20:08:00Z","isPatch":false,"sender":{"key":"nkreitzinger@gmail.com","avatar":null},"body":"On 5/5/2012 7:14 AM, Sitaram Chamarty wrote:\n> On Sat, May 5, 2012 at 11:00 AM, Neal Kreitzinger\n> <nkreitzinger@gmail.com>  wrote:\n>\n>> Interestingly, git-archive was not intended for deployment, but it does\n>> pretty good at it.\n> Except it doesn't delete files that should be deleted (the way 'git\n> checkout' does or rsync would if you gave it --delete).\n>\nOh, yeah.  Thats why I move everything to a backup directory before \nuntarring it.  That also satisfies our pre-git practice of saving off \nthe old version before replacing it so we have a quick recovery point if \nthe new version turns out to be worse than the old one!\n\nv/r,\nneal\n"}]}