Parallell Development / Switching to GIT

Hello,

we are using SVN right now and with the way we do / need to develop, it seems we are constantly get in a merging horror. I did quite some reading about git now, but I am still not really sure if that what we try to accomplish can be done with git, Or if we are really doing something a too odd way.

Let my try to describe – I also added an image. 

---- repo 1
  |
   - repo 2 (=branch of repo 1 - for our external developers)

We have the main branch and 2nd branch for external developers. 

We work inside the repo1, which are usually features/updates that go life after a short turn. Our external developer work on different features that will be merged into repo1 from time to time.

Usually during development, we sometimes need to push features from repo1 to repo2, and later the features developed on repo2 will be pushed back to repo1, And also smaller bug fixes come from repo2 that needs to go into repo1.

But this is a constant process, meaning, that both branches will exist, especially repo2 will exist after this feature has finished for smaller updates/bugfixes. We don’t want to do a new branch for each bugfix, for each new small feature, but have different branches for different developer teams.

So I was wondering, if this could cause troubles with GIT in case of merging around without closing a branch.

I am adding an link to an image that might show what I tried to explain. http://temp.in.futureweb.at/parallell-development.png

Thanks Patrick

Patrick Neuner wrote:

Conflicts will still happen with git, but "merge horror" no longer applies,
for a couple of reasons (I come from a similar background, but switched to
git in late 2005).
1. In SVN (and CVS), you're merging *unknown changes* into *unsaved state*.
   You haven't committed your changes to the repository before you merge,
   and you haven't (usually) looked at the upstream changes before you
   try to merge. Git doesn't have this problem (and neither does any other
   distributed version control system), since you first fetch changes from
   someone else and then merge them into an already saved state. When a
   merge conflict resolution goes wahoonie-shaped, you can easily restore
   either of the previously saved states with zero hassle.
2. Git has "rerere", which records and reuses previously resolved merge
   conflicts, so you won't get the same merge-conflict more than once, if
   you enable rerere.
3. SVN (and CVS) won't remember which changes are already merged in, so
   they will fail horribly at repeated same-branch merges. Git (and other
   DAG-based scm's) can and do calculate the merge-base for you so you'll
   never have to think about that yourself.

Git doesn't take away merge conflicts, but it does make it (a LOT) easier to handle them when they appear, for the reasons stated above. What you want to do sounds pretty reasonable, although I'd personally use feature-branches for both internal and external developers, since they make it possible to pick which features and fixes you want to release while allowing developers to make as many commits as they feel is necessary for each feature.

We're quite fond of ascii-art here on git@vger. Since I don't know what the different colors mean in the picture, ascii would probably have made more sense (since I then could have commented on it).

Hi,

I have been installing and trying out GIT by now and like it. 

I am not sure if I understood everything correctly though. Let me point out 2 parts I just couldn't find a solution for so far:

We are still having the scenario of 2 branches. Master Development.

1) What if I only want to merge a specific file/directly, but not the whole branch, is there a way?

I found that I can git checkout development directory/ but this only works for new added files, it would overwrite changes within the same file of master, if it exists. So it's pretty similar to just copiing files over and commit.

The reason is, that external developers will only commit to development branch. They are working on new features, and sometimes some small bugfixes, or design templates. Those need to be merged separately, and we try to not have more branches. As developers can access our testserver and then see what they have done and test functionality.

If we do different branches, testing on the testserver would always involve merging the branch into development branch, and especially for templates, this isn't a one time update, but might be there are quite some commits, until everything fits.

Is there a way?
  
2) We are using gitosis to give external developers access to the branches and have some kind of access restriction. 
But we are only able to limit push rights, not pull rights. In most cases, that's not a problem, if they see master
And development, but sometimes (like for external designers), we might want them to only be able to checkout some directories. 

Important here is, that merging will still work (like later into the master), and that files go directly into development branch. Same as above, they need to test functionality, so switching branches on the testserver wouldn't work.

As I am coming from an svn background, where both things were possible, I wonder if I just didn't hit the right trail to get this accomplished with git?

Thanks

Patrick

-----Ursprüngliche Nachricht-----
Von: Andreas Ericsson [mailto:ae@op5.se] 
Gesendet: Donnerstag, 25. Juni 2009 12:12
An: Patrick Neuner - Futureweb.at
Cc: git@vger.kernel.org
Betreff: Re: Parallell Development / Switching to GIT

Patrick Neuner wrote:

Conflicts will still happen with git, but "merge horror" no longer applies,
for a couple of reasons (I come from a similar background, but switched to
git in late 2005).
1. In SVN (and CVS), you're merging *unknown changes* into *unsaved state*.
   You haven't committed your changes to the repository before you merge,
   and you haven't (usually) looked at the upstream changes before you
   try to merge. Git doesn't have this problem (and neither does any other
   distributed version control system), since you first fetch changes from
   someone else and then merge them into an already saved state. When a
   merge conflict resolution goes wahoonie-shaped, you can easily restore
   either of the previously saved states with zero hassle.
2. Git has "rerere", which records and reuses previously resolved merge
   conflicts, so you won't get the same merge-conflict more than once, if
   you enable rerere.
3. SVN (and CVS) won't remember which changes are already merged in, so
   they will fail horribly at repeated same-branch merges. Git (and other
   DAG-based scm's) can and do calculate the merge-base for you so you'll
   never have to think about that yourself.

Git doesn't take away merge conflicts, but it does make it (a LOT) easier to handle them when they appear, for the reasons stated above. What you want to do sounds pretty reasonable, although I'd personally use feature-branches for both internal and external developers, since they make it possible to pick which features and fixes you want to release while allowing developers to make as many commits as they feel is necessary for each feature.

We're quite fond of ascii-art here on git@vger. Since I don't know what the different colors mean in the picture, ascii would probably have made more sense (since I then could have commented on it).

On Sun, Jun 28, 2009 at 07:51:26PM +0200, Patrick Neuner - Futureweb.at wrote:

For the situation you describe, it is not about merging a specific _file_, but rather you want to pick specific _commits_ from the development branch that have the bugfixes (or whatever) that you need, and merge the changes introduced by those commits (but not the rest of the history).

And that is easy to do; it is called cherry-picking, and you can use "git cherry-pick" to pick specific commits from development to master.

There are two ways you can split access, and one will work but the other will not.

In git, you generally cannot split your data by _tree_. That is, you cannot say "here is all of the history for the master branch, but you are only allowed to look at some subset of the files." Because at a fundamental level, git is about tracking changes to the _whole_ set of files over time, and it makes the assumption that if you have commit X, which points to tree Y, which points to files A, B, and C, that you will have the data for X, Y, A, B, and C in your repository.

However, if you have your data split by _history_, that might work. That is, if you have a "master" branch and a "development" branch, you can in theory say "you may look at the history of master, but not of development". The usual way to do that is to actually keep "master" and "development" in two different repositories, and only grant read permission in the filesystem for the "master" one (which obviously implies doing your reading over something authenticated, like ssh).

Hope that helps, -Peff

Hi,

thanks for your answers, I appreciate that. I read about cherry-picking, but I am not quite sure if that's really what we need. Lets assume, you do a new feature:

/featureX

You will commit it, check it out on the testserver and probably see a bug, fix it, commit and push it again. (and probably more commits after the testing person ran over other issues). 

With cherry-picking, I would need to know all commits I have to pick. But as there have been serveral commits, so wouldn't it be a pain to check all commits to that file or directory to have the same version?

Just trying to find the right way to handle that. 

About the 2nd point - I am not sure if I get the different repositories thing. Do you talk about to different clones of the rep, and give different directory permissions on it, or is there a way to have like to completly different git rep's running and still merge things over (both ways)? I just thought this approach would break correct mergin, as it doesn't know where it's comming from.

The only thing I ran over so far is probably doing a hook for that (like a pre-pull hook if that exists). didn't get to read too much about hooks yet, just did the update hook that checks if the user with specific ssh key is allowed to push to a specific branch. That works pretty good and is more important in fact.

But having 2 completly different repos would be another solution, but I kinda wonder that mergin would work correctly this way (if both sides have changes). 

Thanks Patrick

-----Ursprüngliche Nachricht-----
Von: Jeff King [mailto:peff@peff.net] 
Gesendet: Sonntag, 28. Juni 2009 20:47
An: Patrick Neuner - Futureweb.at
Cc: Andreas Ericsson; git@vger.kernel.org
Betreff: Re: Parallell Development / Switching to GIT

On Sun, Jun 28, 2009 at 07:51:26PM +0200, Patrick Neuner - Futureweb.at wrote:

For the situation you describe, it is not about merging a specific _file_, but rather you want to pick specific _commits_ from the development branch that have the bugfixes (or whatever) that you need, and merge the changes introduced by those commits (but not the rest of the history).

And that is easy to do; it is called cherry-picking, and you can use "git cherry-pick" to pick specific commits from development to master.

There are two ways you can split access, and one will work but the other will not.

In git, you generally cannot split your data by _tree_. That is, you cannot say "here is all of the history for the master branch, but you are only allowed to look at some subset of the files." Because at a fundamental level, git is about tracking changes to the _whole_ set of files over time, and it makes the assumption that if you have commit X, which points to tree Y, which points to files A, B, and C, that you will have the data for X, Y, A, B, and C in your repository.

However, if you have your data split by _history_, that might work. That is, if you have a "master" branch and a "development" branch, you can in theory say "you may look at the history of master, but not of development". The usual way to do that is to actually keep "master" and "development" in two different repositories, and only grant read permission in the filesystem for the "master" one (which obviously implies doing your reading over something authenticated, like ssh).

Hope that helps, -Peff

On Sun, Jun 28, 2009 at 10:08:45PM +0200, Patrick Neuner - Futureweb.at wrote:

You are doing parallel development with a centralized resource: the test server.

That is not going to fly, and there's no amount of tooling that will resolve this conflict.

The only solution is spend some time on infrastructure and use virtual machines (virtualbox and other systems are known to work well; I'm sure you can find better recommendations from others here) to remove your central test server bottleneck. You don't even need virtual machines. It sounds like you're developing a webapp, in which case every normal workstation would do just fine for hosting a test server.

Once you have the test environment duplicated on each developers' machine then you will find that testing and managing new features using "feature branches" will come easily and naturally.

This exercise will also improve your deployment strategy substantially. There is no longer anything particularly special about $production_server or $test_server. They would all be "just another instance" of the easily replicatable environment that you've created for the development team.

I've intentionally described things at a high level. There are plenty of good resources that go into the mechanics of how to use feature branches in git.

Once all of your features and bugfixes are happening on branches then knowing which commits went into a particular feature or bugfix becomes trivial.

In summary: use branches. They are a simple yet very powerful construct.

Patrick Neuner - Futureweb.at wrote:

I can't help but think you're going about this the entirely wrong way. It sounds as if every developer has to be able to log in to the test server to try out their stuff, which sounds just plain insane to me (unless you're developing supercomputer applications, but a company named "futureweb" probably doesn't do that).

Anyways, the way you *merge* specific features in git is to develop them on a topic-branch. If you absolutely do not want that (though I can't think why, since branching is both cheap and easy in git), you have to resort to cherry-picking. You cannot merge and say "I want only changes to these and those files" because you'd end up with either a disconnected DAG (meaning git wouldn't know where to merge from when you want to merge next), or with a connected DAG that *still* doesn't have all the changes.

In short; Your workflow seems built around the capabilities and shortcomings of SVN. Git has a different set of capabilities and shortcomings, so it's natural that some things will feel awkward. It's like bringing a skateboard to a formula 1 race, really.

With SVN, noone uses feature branches, because merging is a serious pain. With SVN, every active contributor has commit access to the central repository, because without it they'd have to juggle patches, which is boring, and that would hinder them in their work.

With git, a lot of people use feature-branches, because merging is really easy. With git, there's no real need to let every developer have write access to the "officially blessed" repository. Instead it's often useful to have each developer set up his/her own public repository and then issue "please pull" requests, allowing the maintainer(s) to fetch their changes, inspect them and then decide which of the changes to make public. A lot of active git contributors have their own repositories, and nearly all of the linux subsystem maintainers have them too. Git also makes it really easy to send patches by email (and apply such emailed patches). Since git is a distributed version control system, each developer still gains the full benefit of using an SCM, while the trusted people act as gate- keepers for patches that get sent in. But I digress...

I'm sure he is, yes.

Yes, there is. That's what happens as soon as you have a public repo and a local repo, and it's how all distributed version control systems work. They're both separate repositories, but you can merge til your heart's content between them, ofcourse.

bzzt! wrong. You're thinking SVN. Git has a DAG, with each revision having a unique name, based on its content and all its history, so separate copies of the same repository can be merged without problem at all. This is how Linux development happens; The subsystem maintainers all have their own repositories, and Linus merges from them during each merge-window. I think there are about 30 "official" Linux repositories if you count all the subsystem ones. Git was *designed* for scenarios like that, so it does it very, very well.

git help merge-base

In SVN, you need to know exactly which revision you've merged before in order to once again merge the same branch. Git holds that knowledge already. Spend some time with gitk after following the git tutorial and you'll probably get a much better grasp of how the DAG works and how git uses it.

I'd advise you to clone the linux kernel and inspecting its history using gitk. Every merge-commit you see which has a line saying something like "merge foo bar frotz of git://example.com/path/to/repo.git" is a merge with branches from different repositories. I wouldn't be the least surprised if you find more than 5000 such merges in the linux kernel history.

On Mon, Jun 29, 2009 at 4:35 AM, Andreas Ericsson wrote:

You got me curious, so I looked:

~/linux-2.6$ git log | grep -c "Merge.*git:" 4431

Not quite 5000, but still an average of roughly 2.88 such merges per day, every single day, since the kernel was moved to git in 2005.

Peter Harris

Hello,

thanks for your really helpful and detailled information I got, it helped us a lot to get started. And I got the point of using branches and thinking different about the workflow.

I stumbled over something I just can't figure out, I kinda feel stupid as it must be something really simple, but going up and down docs, just can't figure it out. 

We use the update-hook to check into which branches pushes are allowed per different ssh keys. Now, I wonder how I am able to create branches that are below another branch.

Like Refs/heads/master Refs/heads/dev Refs/heads/dev/featureA Refs/heads/dev/featureB

Instead of Refs/heads/featureA

Anything I tried either results in an error or creates the branch under /refs/heads/. 

For the update hook this would be helpful as we won't always be able to pull from a developer, so they would need to push their feature branches up to our public one. (as we prefer it this way instead of receiving patches). 

I found on following page what we need, so I bet there is a way I just couldn't figure out how? Basically the refs/heads/bw/.* example is what I am seeking, so each developer can push feature branches up under development. I do understand it's nothing different if the branch is under /refs/heads, but first it would be cleaner, 2nd access control is better handled.

++ http://www.kernel.org/pub/software/scm/git/docs/howto/update-hook-example.txt

        refs/heads/master	junio
	+refs/heads/pu		junio
        refs/heads/cogito$	pasky
        refs/heads/bw/.*	linus
        refs/heads/tmp/.*	.*
        refs/tags/v[0-9].*	junio

With this, *Linus can push or create "bw/penguin" or "bw/zebra" or "bw/panda" branches*, Pasky can do only "cogito", and JC can do master and pu branches and make versioned tags. And anybody can do tmp/blah branches. The '+' sign at the pu record means that JC can make non-fast-forward pushes on it. ++

Thanks

Patrick 

-----Ursprüngliche Nachricht-----
Von: Andreas Ericsson [mailto:ae@op5.se] 
Gesendet: Montag, 29. Juni 2009 10:35
An: Patrick Neuner - Futureweb.at
Cc: Jeff King; git@vger.kernel.org
Betreff: Re: AW: Parallell Development / Switching to GIT

Patrick Neuner - Futureweb.at wrote:

I can't help but think you're going about this the entirely wrong way. It sounds as if every developer has to be able to log in to the test server to try out their stuff, which sounds just plain insane to me (unless you're developing supercomputer applications, but a company named "futureweb" probably doesn't do that).

Anyways, the way you *merge* specific features in git is to develop them on a topic-branch. If you absolutely do not want that (though I can't think why, since branching is both cheap and easy in git), you have to resort to cherry-picking. You cannot merge and say "I want only changes to these and those files" because you'd end up with either a disconnected DAG (meaning git wouldn't know where to merge from when you want to merge next), or with a connected DAG that *still* doesn't have all the changes.

In short; Your workflow seems built around the capabilities and shortcomings of SVN. Git has a different set of capabilities and shortcomings, so it's natural that some things will feel awkward. It's like bringing a skateboard to a formula 1 race, really.

With SVN, noone uses feature branches, because merging is a serious pain. With SVN, every active contributor has commit access to the central repository, because without it they'd have to juggle patches, which is boring, and that would hinder them in their work.

With git, a lot of people use feature-branches, because merging is really easy. With git, there's no real need to let every developer have write access to the "officially blessed" repository. Instead it's often useful to have each developer set up his/her own public repository and then issue "please pull" requests, allowing the maintainer(s) to fetch their changes, inspect them and then decide which of the changes to make public. A lot of active git contributors have their own repositories, and nearly all of the linux subsystem maintainers have them too. Git also makes it really easy to send patches by email (and apply such emailed patches). Since git is a distributed version control system, each developer still gains the full benefit of using an SCM, while the trusted people act as gate- keepers for patches that get sent in. But I digress...

I'm sure he is, yes.

Yes, there is. That's what happens as soon as you have a public repo and a local repo, and it's how all distributed version control systems work. They're both separate repositories, but you can merge til your heart's content between them, ofcourse.

bzzt! wrong. You're thinking SVN. Git has a DAG, with each revision having a unique name, based on its content and all its history, so separate copies of the same repository can be merged without problem at all. This is how Linux development happens; The subsystem maintainers all have their own repositories, and Linus merges from them during each merge-window. I think there are about 30 "official" Linux repositories if you count all the subsystem ones. Git was *designed* for scenarios like that, so it does it very, very well.

git help merge-base

In SVN, you need to know exactly which revision you've merged before in order to once again merge the same branch. Git holds that knowledge already. Spend some time with gitk after following the git tutorial and you'll probably get a much better grasp of how the DAG works and how git uses it.

I'd advise you to clone the linux kernel and inspecting its history using gitk. Every merge-commit you see which has a line saying something like "merge foo bar frotz of git://example.com/path/to/repo.git" is a merge with branches from different repositories. I wouldn't be the least surprised if you find more than 5000 such merges in the linux kernel history.

Patrick Neuner - Futureweb.at schrieb:

You cannot have refs/heads/dev and refs/heads/dev/featureA at the same time, just like you cannot have a file and a directory with the same name at the same time. In fact, the refs "database" is implemented as physical files on the file system.

-- Hannes

Hi,

ok, I see, well then this howto (at the end of page) seems to be misleading. http://www.kernel.org/pub/software/scm/git/docs/howto/update-hook-example.txt as it actually works with the update hook, but not with git itself.

Well, anyway, no problem, we will do it different. 

Thanks

Patrick Neuner Futureweb.at Innsbrucker Str. 4 6380 St. Johann

neuner@futureweb.at www.futureweb.at

Fon: +43 (0) 5352 65335
Fax: +43 (0) 5352 63148
Gratis über Skype anrufen | Skype-ID: futureweb

Please don't top-post on this list.

Patrick Neuner - Futureweb.at schrieb:

Are you refering to the example line such as

        refs/heads/bw/.*	linus

? Of course, you can have refs/heads/dev/featureA in your repository (and the shortened branch name would obviously be "dev/featureA"), but then you cannot have a branch named "dev", i.e. refs/heads/dev.

I don't see how the cited document would be misleading in this regard.

-- Hannes

On Sun, Jun 28, 2009 at 10:08:45PM +0200, Patrick Neuner - Futureweb.at wrote:

I don't quite understand what you are asking. You make some commits pushing a new feature forward. While testing, you see some bugs. You fix the bugs and make new commits. Now you realize you want those bugfixes on some other branch. So you cherry-pick them away. Yes, you have to figure out which commits you want. You can use "git log" or "git log <set of files>" to look through the list of commits and pick them out.

When you say "wouldn't it be a pain to check all commits to that file or directory to have the same version?" I can't quite parse what you are trying to say. Can you rephrase it?

No, it doesn't break merging at all. You will have two different repositories, but they may actually contain quite a similar subset of commits. That subset will be the shared part of the history graph, and then each one will have commits on top. Periodically features from development will get merged to master, which will make those merged bits part of the shared history.

To git, two branches in the same repo is exactly the same as two repos, each with its own branch.

Yes, that is the hook you would need, but it doesn't exist yet.

Of course it is still possible to have merge conflicts, but it is no different than merging two branches from the same repository.

-Peff