git with large files...

I use git for many things, but I am trying to work out the workflow to use git for deployment. I have a wordpress site that i need to track multiple branches of and it has many many resource files, images, video, etc... how can i streamline the performance. ive read about git-annex and also something about git-bigfiles being merged in but i dont know how any of that has progressed, and I am just trying to work out the best way to handle this large site.

Happy Cake Oven -->  -[__]-

Darek> I use git for many things, but I am trying to work out the Darek> workflow to use git for deployment.

Don't.

On Fri, Jul 20, 2012 at 6:54 PM, Randal L. Schwartz <merlyn@stonehenge.com> wrote:

Heh. Best to keep in mind that it just doesn't work very well. git-bigfiles, git-annex might help you, but look at the docs and caveats carefully.

Perhaps use rsync, unison work better for you.

m

On Fri, Jul 20, 2012 at 4:28 PM, Martin Langhoff <martin.langhoff@gmail.com> wrote:

I'm not sure if it was the "big files" part that Randal was responding to. IIUC it was the "using git for deployment" part.

Packaging tools (Makefiles, .rpm, .deb, etc) are a better suited for deploying software.

Well, many folks use puppet in serverless configuration pushing the manifest from a central git server via cron and applying locally the configuration fetched. In this sense git IS used for deployement. And, for a configuration management system as puppet this could be a sensible thing to do - reduce load, scalability ecc.

Best regards

2012/7/21, David Aguilar <davvid@gmail.com>:

On Sat, Jul 21, 2012 at 3:11 AM, Elia Pinto <gitter.spiros@gmail.com> wrote:

That's a great thing to know, and timely too. I am looking at Puppet, and worried a bit about reported memory load on the server side (and some rumours of memory footprint issues on the client side too).

Can you point me to more information & discussion?

thanks!

m

Sure . There is a complete paragraph " creating decentralized puppet architecture" in "puppet 2.7 cookbook". I am also sure to have read the same topic on web on the site bitfieldconsulting.com , not casually : the book's author have his blog there. Sorry i have not the complete url now - old smartphone, no wifi etc. But i hope this can help you the same.

Best regards

2012/7/21, Martin Langhoff <martin.langhoff@gmail.com>:

I forgot to tell that using git for deploying puppet have also some disadvantages : you cant'use some puppet feature as external node classifiers or stored configurations. As always, there are tradeoff. For more

http: //bitfieldconsulting.com/scaling-puppet-with-distributed-version-control (the missing link)

cheers

2012/7/21, Elia Pinto <gitter.spiros@gmail.com>:

While my example isn't about puppet, it's about another templating technology - that is by technical design hence also completely open source. We have a completely modulirized and completely distributed deployment scenario that is based on git.

The fundamental core how git addresses the solution is explained here
- especially on the visualization.
http://abstractiondev.wordpress.com/git-based-distribution/

I'd like to add a point of view that will underline git behaviour as a core requirement on distributed depoloyment - if there is any issue with big files, that's a good concern to address.

While this can be well justified and argued to benefit for possibly platform specific deployment tools such as rpm or msi modules, the audit trail of distributed software (the core requirement for source code andyway) still remains. If there is a "big file" that is part of the audit process, it's consistency needs to be guaranteed. And there is no way around it but the git way of doing complete secure hashed history trail.

This is not "nice to have" feature, but critical requirement to deploy embedded software for automated machinery to suffice safety regulations. Hospital, aviation, nuclear device software all apply here. Distributed software processes and projects all benefit from this - and trying to "cut short" some part of the complete audit trail of development will cause larger pain points elsewhere.

So is there real problem with big files performance or storage files? Anything else than "uncommon" slowness on calculating and comparing large file sha1 hashes?

Kalle

On 21/07/12 15:42, Martin Langhoff wrote:

We use "masterless" puppet, deployed using gitolite post-receive hooks. The most useful clues I found are here.

 http://groups.google.com/group/puppet-users/browse_thread/thread/ed9a4032b31bd8d4/e1a68aa8ea91305d

 http://semicomplete.com/presentations/puppet-at-loggly/puppet-at-loggly.pdf.html

 http://current.workingdirectory.net/posts/2011/puppet-without-masters/

 http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

We had to join the dots ourselves. Works for us so far, but it's only about six months old. We don't have lots of servers or very exacting requirements, just a inclination against the pull orthodoxy and the freedom to experiment. Can't comment about memory footprint, except it's not been a problem for us. (On the other hand I am not that enamoured with Puppet's DSL design, I might prefer to avoid it if I could.)

N

On Fri, Jul 20, 2012 at 11:47 PM, David Aguilar <davvid@gmail.com> wrote:

Fair enough. On that topic, I have to say that git, with suitable scripts controlling it, has been good to me for deployment to webservers (ie: synchronized deployment on server clusters). Limited to interpreted languages (py, php, etc).

I am not afraid of Makefiles, rpms or debs but sometimes they are a bit too much overhead.

cheers,

m

merlyn@stonehenge.com (Randal L. Schwartz) writes:

Yeah, "don't think 'git checkout' is a way to 'deploy'". Using Git as a transport measure is probably fine.

On Mon, Jul 23, 2012 at 2:24 AM, Junio C Hamano <gitster@pobox.com> wrote:

You can also try http://sitaramc.github.com/the-list-and-irc/deploy.html. Whether it's saying you *can* use git for deploying something, or you *can* but *should not*, or you *cannot*, will depend on your own thoughts on the matter ;-)

2012/7/23 Sitaram Chamarty <sitaramc@gmail.com>:

Nice summary list of options!

If you combine that with several key concepts:
1. You plan and design to deploy - hence you have separate deploy
repositories dedicated for that
2. You design for modularity and complete audit trail, hence you have this:
http://abstractiondev.files.wordpress.com/2011/12/git-distribution-overview.png

You can combine the staging with any approach, that *tries* to maintain the local version copy. But if any problem arises in pull/fetch, simply trash that part and get it from fresh (or just use the git archive approach).

Now this model would introduce complete and as detailed security enforcement - as the deployment can validate with certificates (from the additional catalogue-metadata binding, whether there is authorized party confirmed signature available for the wished deployable version).

I don't see much overhead in any of the steps here - and the deployment is as detailed and as securely controlled as desired. With just Git (and well, possibly GnuPG or alike common tool for digital certificate work).

Everything is also transparent - which is very important in having that complete control and audit trail.

On Mon, Jul 23, 2012 at 12:23 AM, Sitaram Chamarty <sitaramc@gmail.com> wrote:

After a couple of false starts, I think that Sitaram came closest to what Darek was asking about.

Now, as somebody who is using Git currently to stage things to "deployment" (I may change to SVN due to office politics--which will double my workload on rollout of updates, but I'm not saying any more than that in public) on production web servers, I have a few comments.

We have several WordPress instances @$work where we are using Git to stage template changes out to our development server (where I've contemplated putting the lessons in Sitaram's article to use) before merging those changes back into the "Production" branch (after suitable testing) and pulling them from a central gitolite into the live server. It works and it respects the posix extended ACLs on the destination host (which is what you actually want on a live web server). Even better, it provides a safe way of tracking and merging back in any "opportunistic" changes that were made directly in the development or production servers so that they are not lost.

Thought must be applied to do this safely, but that's the way it usually is on web servers. To those who say admins should be using RPM, DEB, or any other "generic package management" software to deploy non-system updates to in-house web servers may I kindly indicate that it often doesn't make sense to do so unless each and every website has its own server and IP address--and you are deploying tens of thousands of them. Most of us can't afford that. (Yes, there is an overhead to building packages. I've done it enough times to know about that quite intimately.)

Packages and package management are great for system software but they are not a good solution for installing client code into a webspace on a shared server (yes, heresy, I know). For this common use case Git is not a half-bad ADDITION to the toolkit of a website development and maintenance team.