Coping with the pull-before-you-push model

  After a deployment of Git on a centralized server at my place of 
business, the largest amount of grumbling has been with the 
pull-before-you-push model.  Coming from the file-centric Perforce where 
you need only have latest of just the files you are submitting, the 
pull-before-you-push model has really been a pain in the neck for a 
large team.

Even with topic branches being used, merges to master occur frequently. It can really be a frustrating battle to get your merged branch pushed to the central master branch. In the time it took you to pull, test, and push, someone has probably already pushed before you. To cope with this, people will pull, not bother testing, and immediately push their changes. Yes, this could result in build instability, but it is considered better than never being able to make your change live.

(Let's ignore what we should or shouldn't be doing as far as 'development practices'. :) We're solving the problems one step at a time...)

Gerrit provides a compelling model where branches are pushed to the code review server in the form refs/for/master, and the given push will always succeed. Code reviews are performed, someone sets the verified bit, and the change is submitted and merged to master by Gerrit itself in a queued fashion. Unfortunately, its general "requirement" to squash your branch down to a single commit is, possibly, a showstopper. If it treated a branch merge as a group of commits that MUST stay together, that would be perfect.

What other tools are out there that would let users successfully push their branch to the server (without having the HEAD master commit), and the push would be automatically merged to the master branch?

Is there another workflow that is successful for your large(-ish) enterprise team?

Thanks for your insights!

Josh

On Thu, Sep 9, 2010 at 04:47, Joshua Jensen <jjensen@workspacewhiz.com> wrote:

Let's not ignore that.

Presumably you had exactly the same problem in perforce, i.e. because you only had have the files you were changing checked out in Perforce in the time between `hack && pull && test && push` someone else might have already pushed. Thus what you just submitted wasn't guaranteed to pass tests.

So is the flow in Git where you don't run the tests again, rebase and push and hope for the best any different?

This sounds like something that's configurable in Gerrit, or should be.

Linux manages to deal with a huge number of commits, but does so by having subsystems.

Maybe that's something you can use in your codebase?

  ----- Original Message -----
From: Ævar Arnfjörð Bjarmason
Date: 9/9/2010 7:06 AM

Fair.

The end result is the same; submitted code is never really tested against latest in Perforce either. The primary difference between the two is that the Perforce submit is successful the majority of the time (odds of someone editing and submitting the same file as you are low), and the Git push fails the majority of the time.

Don't get me wrong. I've given training on why Git's enforced pull-before-you-push model can be better than what we had before (reproducible state, fewer broken builds, etc). Nevertheless, the issue is very frequent, and that's why I am querying others.

Agreed, but it appears it is currently a missing feature. There have been discussions about it on their mailing list over the past months, and a feature request is in their tracker. I am unsure of their progress or even interest.

Unless I'm mistaken, though, it seems to work in reverse of what our working model has been for years.

I'm grossly oversimplifying the process, but the Linux model seems to be built on hierarchical 'pull requests'. I can tell a subsystem maintainer I have some changes and then ask that maintainer to pull them from a certain location. When that person has time/inclination, the change is pulled, merged in, and then another pull request is sent to the upstream hierarchy.

This _is_ compelling, but even if it would work within the company I work for, it is such a dramatic shift in workflow that I am certain it could not be done in one fell swoop.

Thanks for your insights!

Josh

On Fri, Sep 10, 2010 at 12:43 AM, Joshua Jensen <jjensen@workspacewhiz.com> wrote:>>

I am not sure that git does enforce a pull before push model, except if you attempt to use git without the equivalent of a maintainer - in essence devolving responsibility of maintaining the tip of the shared branch to the whole team.

This is probably the cultural shift that is hardest for enterprises to accept - why do I need a _person_ to do this _manual_ work when tools like {insert favourite non-DVCS here} can do this for me? To management, this looks like a step-backwards.

In our case, the team leaders act like the subsystem leads and the build team acts like the maintainer. Team leads have a more review oriented focus, where as the build team doesn't understand the code that well and will reject merge conflicts if they are non-trivial. Their focus is just to keep the daily build rhythm going.

If you don't have a maintainer role, then everyone is obliged to pull, test, pull-again, then push because there is no-one else there to do it and no other time to do it. The delays inherent in the test process inevitably mean you have to cycle several times until your commit wins.

But with a maintainer role on your project, you don't need to do this. Everyone develops on yesterday's baseline and verifies that they don't regress anything with respect to that baseline - there is no requirement to certify against the bleeding edge of the integration stream - that's the role of the maintainer/build team once the daily build is done, thereby amortising test execution effort across the whole team.

Note that one difference between the maintainer and maintainer-less model is that the tip of the reference branch is published relatively rarely - it is quite stable. This is quite helpful because the baseline doesn't keep shifting every 30 minutes. In the maintainer-less model, you never have stability, nor any opportunity to acquire it.

The big difference between commercial work and open source projects like git and linux is that the latter have just one constraint - quality whereas the former also have budgets and schedules to worry about. Unintegrated crap code has cost the open source project almost nothing. Commercial enterprises, on the other hand, pay lots of money for people to develop code whether it is crap or otherwise. The idea of leaving expensive code unintegrated causes management paroxysms of concern that are hard to ignore - a tool that blindly integrates crap code automatically is a soothing balm for such people. Hence the resistance to tools like git that encourage a maintainer role and, implicit in that, the possibility of review that might result in crap code being exposed for what it is.

jon seymour.

On Fri, Sep 10, 2010 at 03:35:21PM +1000, Jon Seymour wrote:

Bear in mind that you can still shift to a maintainer model, but keep the maintainer automated. That is, you can queue up "to-pull" heads, and then have an automated process pull them one by one and do some basic QA (does it merge, does it build, does it pass automated tests, etc). Which is not that different from what many shops do in the non-maintainer model, except that when you break the build, the maintainer process notices _before_ publishing the merged tip, so everybody won't try to build on your broken crap.

I seem to recall that Gerrit does something like this, but I may be mis-remembering. I haven't actually used it for real work.

I still prefer a human maintainer, because they can do things like reorder the queue manually (or outright reject flaky topics) to get more sensible merges, or do easy but non-trivial merges themselves to avoid kicking code back to the developer.

-Peff

  ----- Original Message -----
From: Jeff King
Date: 9/10/2010 8:15 AM

Do you know of any existing software that does this? This may be ideal in the short term.

I agree with you concerning how valuable a human maintainer is.

Josh

On Mon, Sep 13, 2010 at 10:47:30PM -0600, Joshua Jensen wrote:

I think that Avery Pennarun's gitbuilder may do what you want:

  http://github.com/apenwarr/gitbuilder/

but I've never used it.

I seem to recall from one of Shawn's presentations on Gerrit Code Review that it does something like this, too, but I can't seem to find any docs about it in my brief search:

  http://code.google.com/p/gerrit/

It may be that Gerrit doesn't handle building itself, but that the Android project is running something alongside it. Shawn may be able to say more.

Basically, what we are talking about is continuous integration, with the slight twist that instead of developers pushing commits to a mainline branch which is built and tested, we would build and test their commits and then merge them to the mainline branch.

Systems like Hudson that do continuous integration and support git may handle a workflow like this, but I don't know (I've only ever used Hudson in the everything-goes-to-svn-trunk model).

There is also a small continuous integration system that lives in the contrib/continuous directory of git.git itself. It's quite old at this point, but I wouldn't be surprised if its descendant is what Gerrit uses.

-Peff

On Mon, Sep 13, 2010 at 10:24 PM, Jeff King <peff@peff.net> wrote:

gitbuilder doesn't do this.

However, it does let you automatically do user-defined things for each branch, and it stores the result for each one. So it would be a pretty short trip from there to auto-merging when a branch is ready.

Have fun,

Avery

On Tue, Sep 14, 2010 at 01:24:51AM -0400, Jeff King wrote:

Gerrit doesn't do the build and test, but it isn't all that difficult to hook into. It also allows the results of the build and test to record their state so that the developer can track what is happening.

The nice part about it, compared to other CI systems I've seen is that it catches the problems before they are merged into master, rather than after.

Internally, we actually do multiple levels of a CI-type thing. Every time a developer uploads a change, one machine performs a sanity build on it (with multiple configurations). These results are visible in Gerrit, and it keeps people from putting too much effort into reviewing code that doesn't even compile.

Once the code has gotten through two levels of code review, a more throughough build and test system pulls the whole project (from numerous git repos) builds and runs tests. This takes long enough that it doesn't do individual changes, so failures take work to track down, but it does generally assure that the result of each 'git merge' somewhat works.

The only real annoying part I've found with the gerrit model is that the tree is filled with lots of merge comments (generally a merge for every real commit). The other option is to let Gerrit rebase, which then gets annoying when a developer has pulled changes from other developers.

It also has a nice link to a 'git pull' command to pull down an individual change.

David

On Sep 14, 2010, at 12:47 AM, Joshua Jensen wrote:

Our workflow at $WORK involves pushing changes to gerrit to various "effort branches", and then once they are approved, we have a "Mergitator" script that will attempt to merge the effort branch with the merged master branch, and then attempt to do a build.  If the build succeeds, then the changes will get pushed back to the publically visible merged master branch, and then the Mergitator will move on to the next effort branch that requires merging.   If there is a merge conflict, the Mergitator will refuse the merge, and then give instructions on how to fix up the tree to avoid merge conflicts.

The Mergitator code hasn't been released, and I suspect the main reason is that there's relatively little code that could be used outside of our environment, and a large amount of code which contains lots of details about our internal build system that would have to be stripped out and generalized before it could be released --- and no one has time to do it.

So this probably doesn't help you since I suspect you meant to ask the question, "do you know of any existing publically available software", but I can tell you that it certainly can be done, and that software exists.  Making it be software which is useful and usable to you would definitely take more work...

-- Ted

  ----- Original Message -----
From: Theodore Tso
Date: 9/14/2010 6:12 AM

How does the integration with Gerrit work here? The only thing that comes to mind is that you do something like:

git push gerrit HEAD:refs/for/merged-master

Then the approvals are done. Afterward, Gerrit merges to the merged-master branch.

I would suppose an external script is performing regular fetches. When it sees new code, it builds.

No, this can't be right, but I'll leave my incorrect workflow here.

It's the branch queueing issue that is my current hang up. Gerrit's method is slick, but that won't work outside of JGit. I'm not opposed to JGit; I just haven't touched Java in years.

So, perhaps, a web interface where the branch owner selects the (pushed to central server) branch name that is ready to go. A merge is attempted. If it succeeds, great. If it fails, then the merge is reset?

Josh

On Tue, Sep 14, 2010 at 11:51 AM, Joshua Jensen <jjensen@workspacewhiz.com> wrote:

In our case we have moved from CVS to git and had pretty much same problems. The only difference was that we are working with java and trying to have every project in separate repository. So we don't have many commits into common codebase. We have a server keeping bare mainline repos. In order to integrate the project with others we are using Hudson CI which performs builds upon each change detected on the server side (i.e. mainline repository) and also builds the projects that are dependent on the one you just changed. Those dependencies can be described inside each hudson job, but we are using Ivy for dependency management. If you have good testing infrastructure that is even better.

So generally the amount of contributors to one project is pretty small. Some projects are solo, some have small teams coworking on one project code base.

We are using two main branch model. The development goes in topic branches. The qa branch is a staging branch for the changes to be accumulated for the release. People are using direct pushes to qa branch to integrate or a maintainer. Hudson builds this codebase. When all release functionality is done the person responsible for the release is merging qa branch into the master, which gets released (this must be a fast-forward).

So, i would say component oriented approach is the key here.

HTH, Eugene

On Tue, Sep 14, 2010 at 09:51:22AM -0600, Joshua Jensen wrote:

So we'll push to something to, say, refs/heads/fs/ext4/for-merged on the gerrit server, and let gerrit do its thing. After a colleague approves all of the patches in the branch, gerrit will release the commits to the branch, and only then will the mergitator script attempt to do a trial merge of the effort branch with the merged/master branch. If the trial merge succeeds, then it will attempt to do a trial compile. If the trial compile succeeds then the merged/master branch will be updated with the commit id of the trial merge, and then the mergitator script will move on to the next effort branch will has been updated. If the mergitator fails to merge a particular branch, then an e-mail is sent out explanining the cause of the merge failure, so a human can fix things up. This could be done by cherry-picking a commit from merged/master which caused the merge conflict, and then fixing up the merge conflict, for example.

A wise developer will do a trial merge on their own *before* submitting their effort branch to gerrit for code review, but this is not strictly speaking required.

						- Ted

  ----- Original Message -----
From: Jon Seymour
Date: 9/9/2010 11:35 PM

My apologies for the late response. I'm not ignoring this. I am trying to process it in the context of my organization as I establish the workflows we'll be using. I may have some follow up questions soon.

Josh