recipe to use git for deployment

I'm trying to cook up automated mass deployment using git as the main ingredient. Here's a recipe idea:

install git on deployed server under the home dir of the 'gittech' user and add that path only to the PATH of 'gittech'.

have the git repos on deployed server be under the 'gittech' home dir with worktree(s) assigned to deployment locations. If people mess with the worktrees I will be able to tell with git status via 'gittech'.

I can ssh to deployed servers from a deployment server and do git fetch to get updates.

I can have a 'deployment manager' repo which contains script to apply patches from remote tracking branch to worktree. It checks out commits at waypoints and at waypoint it executes needed file/data conversion programs, creates dir/file structures, etc. If conversion validation program is successful it will continue to checkout next waypoint, etc, until worktree matches HEAD of remote tracking branch.

Data/files are commited to data tracking git repo at waypoints. If waypoint validation fails then software version repo and data version repo are checked out to last good waypoint or back to startpoint.

Problem(1): cron patches. The cron jobs portion I'm not sure of. Need to learn more about cron. Perhaps cron can run scripts that are symlinks to a worktree. Cron worktree can have waypoints just like software and data worktrees. The cron tabs, cron.daily, etc can be worktrees perhaps also.

Problem(2a): diskspace. I think I will have to pull shallow clones to avoid growing history. For deployment I only care about recent revisions. Not sure if the above ideas will work with shallow clone.

Problem(2b): network latency. Maybe I can have the git repo object store on the deployment server and the deployed git repo reference it to save diskpace on the deployed server. That would probably be really slow.

I'm thinking the ingredients of this recipe could make a tasty git deployment. Maybe different ways of mixing and cooking them and other ingredients can make a good dish. Please let me know if you think this can be edible for security, palatable for reliability, and digestible by git.

v/r, neal

I always did "git pull" on the servers - but the history of my projects
is not that huge.. Thus I never cared.
Great: You can keep some server specific config settings and rebase
them - and you can do a fast git status to check whether file contents
have been modified (eg determine whether you've been hacked ..)

If you really care that much about history why not push a zip file using git archive --format=zip and unpack that on the deployment server instead?

For FTP access only this did a great job for small projects: https://github.com/MarcWeber/git-ftp (-> git-ftp-minimal.sh) It only copies changed files *and* checks whether they have been modified on the server first (detecting work of others). But that's probably not a good thing for automatic deployment.

Marc Weber

On 5/4/2012 11:10 PM, Marc Weber wrote:

So I could rebase apache virtual host conf files with generic template changes like symlink support, but keep the networking ip's. Merge conflicts would be bad. I guess that would be cause to reset --hard to a previous commit.

and you can do a fast git status to check whether file contents

With 'source is run' like html, css, etc. that would be a good way to enforce no-hacking-allowed support agreements.

That is what I'm currently doing. By paying attention to tracked executable bits I don't even have to fix permissions on deployment.

The scriptable patching of a git repo (scripted checkout of remote tracking branch commits to worktree with inserted conversions and validations -- an interactive cherry-pick so to speak) allows for flexible conversions of varying patch levels. Maybe git-sequencer will do this when it comes out.

Also, I could have a deployment tracking repo on one of my servers that remotes to all deployed servers and pulls to refresh my monitor view of all deployed revisions in gitk.

I'll take a look to see what I can steal, uh I mean reuse.  Thanks!

v/r, neal

I've been deploying .php files this way only..

If you want to deploy full configuration of hosts you may also want to learn about nixos.org (which handles restarting apache etc) or chef like systemms: http://www.rubyinside.com/chef-tasty-server-configuraiton-2162.html using nixos you can easily automatically generate additional cron job ensuring that permissions are correct etc.

Maybe talking about your exact use case will make others list more options.

Marc Weber

On 5/4/2012 11:10 PM, Marc Weber wrote:

I summon forth the ents to do my bidding with one command to find them: --format=tar one command to transform them: --prefix=deployed/path one command to arm them: config tar.umask (executable bits) one command to gather them: tar -A and one command to bind them: gzip

(We don't have the new tar with the transform option)

They are faithful and true and have won many battles, but I fear they may be too slow and lumbering for the gathering whirlwind of patches about to descend from above and legion of servers encircling them on all sides.

Interestingly, git-archive was not intended for deployment, but it does pretty good at it.

(Sorry, I haven't even read LOTR, but couldn't resist the opportunity to use the word 'ent' after reading about it in git-glossary ;)

v/r, neal

On Sat, May 5, 2012 at 11:00 AM, Neal Kreitzinger <nkreitzinger@gmail.com> wrote:

Except it doesn't delete files that should be deleted (the way 'git checkout' does or rsync would if you gave it --delete).

On a lark I once wrote a very efficient way of dealing with this; see attached. It's probably won't work on anything but Linux and requires bash, and it's only been minimally tested and I do not actually use it, but someone might like it.

On 5/5/2012 7:14 AM, Sitaram Chamarty wrote:

Oh, yeah. Thats why I move everything to a backup directory before untarring it. That also satisfies our pre-git practice of saving off the old version before replacing it so we have a quick recovery point if the new version turns out to be worse than the old one!

v/r, neal

Neal Kreitzinger <nkreitzinger@gmail.com> writes:

I think the standard answer to this is "don't do it".

You could of course add hooks to hack around it, but fundamentally Git was designed to track source files (so the only sane behaviour for "checking out" is to keep the timestamp of the working tree file to whatever the underlying OS gives it when "checkout" happens, for example), so anything that your "deployment strategy" wants conflicts with what the usual source coutrol operation should do, you need to tweak around what Git does (the same thing can be said for CVS or SVN, by the way).

Unless you do this with an understanding that you are merely using git "as a better rsync" and take responsiblity of everything else that git-the-scm does _not_ do for you, I suspect that you would be volunteering for a lot of pain.

On Sat, May 5, 2012 at 5:51 AM, Neal Kreitzinger <nkreitzinger@gmail.com> wrote:

At work we use and have open sourced https://github.com/git-deploy/git-deploy which we use for our deployments.

It doesn't actually do the work of syncing out anything, we effectively do that with distributed rsync.