git as a versioned filesystem

Hi all,

First off, I'm *not* using git as a typical VCS on the front-end; I'm using it for dist control on the back-end. I'm also fairly new to git (about a week and a half into it).

The Scene =========

For source control, we're using CVS (migrating off of it, btw--I only have limited influence around here). We build our software, etc, etc. and then we have the developers scp/rsync/untar their builds on a *master disk image*.

This master disk image is disted via rsync to a few thousand servers to keep them all up to date and in sync, etc. This works mostly fine and I can't really change this system.

The Problem ===========

Our problem has been that occasionally bad stuff gets put in the master image and we have no easy way to revert it or to allow the QA team to cherry-pick/revert changes to that master image.

The Solution ============

Git seems like the perfect tool for this, but I'm still not sure how to adapt it to our situation. I'm building a tool that uses git to let the developers commit their binary changes to this master image into the git repository, which hopefully will allow me to offer the QA team some ability to cherry-pick updates or revert regressions and make a clean dist image from week to week.

The Question ============

What I need to know from y'all is: is there a better way, a more git-like way, to accomplish this. Here's the model I *want* to follow:

-----a----b--T1-------c--------d-e---f------g [master]
               \   (a)  \
                ----|----c'---                [B1]

Here is branch B1 created from the master at some point in time T1. On branch B1, I revert commit (a) and cherry-pick commit (c):

  git checkout master
  git branch B1
  git checkout B1
  git revert a
  git cherry-pick c

At this point, B1 is our "perfect image" and we're ready to dist it. I check it out elsewhere and rsync it, etc. Wonderful.

But now it's a few weeks later and we're ready to do another dist. What I *want* to do is create a *copy* of branch B1 to give the release manager a reference point for him to bring things up to date. What is the best way to do that?

If I branch off of B1, now I have the burden of doing a whole lot of cherry-picks and having a challenging time getting things back in sync:

-----a----b--T1-------c--------d-e---f------g [master]
               \   (a)  \         \   \
                ----|----c'---     \   \      [B1]
                               \    \   \
                                -----e'--f'---[B2]

Ugh. Now B2 is kind of a mess. If I rebase it on master, I'll get (d) and maybe (a) again, which I don't want. [side question: unless there's a way to rebase on master but still exclude commits... possible?]. B3 and B4 are going to look even worse and the risk of drifting so far away from the master is unappealing.

Ideally I'd want each week's release to come directly from the master, kind of the flying-fish approach:

                               ----e'--f'---  [B2]
                             /    /   /
-----a----b--T1-------c--------d-e---f------g [master]
               \   (a)  \
                ----|----c'---                [B1]

The problem with this is that now B2 contains (a), so I'll have to revert that again--which I can do happily--but I just wonder if there's a better way. If it's possible to simply *copy* branch B1 to B2 without making B2 a branch off of B1.

In the absence of a git-branch-copy, is there something that would help me do set intersection and subtraction between branches? Something like this:

  git log B1
  ... bunch of commit ids ...
  git log B2
  ... bunch of commit ids ...

  ## find the intersection(B1, B2)

  ## revert all the things missing in B1 from B2

  ## now B2 is the same as B1--assuming git is idempotent (is it?)

  ## is there way besides rebase to clean out a revert as if it never
  ## happened? I suppose I could branch again and repeat this as
  ## needed.

Am I even thinking about this correctly?

Keep in mind that these commits are not source code commits; they're file system changes of all kinds: updated binaries and libraries, new directory trees, removed directory trees, etc. It's much closer to a package manager in spirit than a VCS.

I feel like I'm missing something grand in git-rev-list or git-log or git-bisect some other tool that will make all my troubles disappear. I've read an awful lot of the man pages, but am still very new to git and I'm certain I've missed some subtleties.

Any ideas? I'm not even sure I'm asking the right questions. I'll accept any advice on this subject.

Scott

On Fri, Oct 2, 2009 at 12:49 PM, Scott Wiersdorf <scott@perlcode.org> wrote:

Beware that git performs rather badly on binary files, especially huge ones, which it tries to load entirely into RAM. It also keeps every revision of every file that was ever committed (and every user who checks it out needs to download the whole thing), so your giant binary repository is going to get very big, very fast.

I've looked into using git for this kind of situation myself. It's close, but not quite there (for my purposes anyway). It basically just needs some optimizations and some improved support for "shallow clones."

But on to your actual question:

If you rebase your "release" changes onto current master, you'll get the revert-a patch applied, so (a) will still be gone. Rebase will also probably be smart enough to throw away c', since it's identical to (c). You will indeed end up with the unwanted (d), but you can just revert that in B2.

"revert-a" is a patch on its own. Git doesn't think of reverting (a) as anything special; it's just a change that happens to reverse what (a) does. So if you rebase B1 onto master, it will get copied. It sounds rebase will produce exactly the results you're looking for here.

Now, that said, this release process seems extremely suspicious to me.

To summarize what I'm hearing: you have a 'master' branch that people put stuff into, but which doesn't actually work correctly. At the last minute before a release, you make a new branch, drop out the stuff that doesn't work, and put it into production.

This sounds problematic. If (a) and (d) don't work, why are they in master at all? Git makes branching really easy: get people to put their not-quite-working features into a different branch, and let the release manager merge those branches into master when they're actually ready.

If you do that, you'll always be releasing straight out of master, and your life will be a lot simpler. And if you "merge --squash" from the feature branches into master, you can throw away the interim versions of the feature branches, which should help keep your repository from growing so quickly with tons of binary file revisions that never even got released.

You probably want "git revert -i".

Have fun,

Avery