RE: git pull on Linux/ACPI release tree

Perhaps the tools should try to support what "a lot of people" expect, rather than making "a lot of people" do extra work because of the tools?

Call me old fashioned, but I believe that tools are supposed to make work easier, not harder.

-Len - To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html

On Sun, 8 Jan 2006, Brown, Len wrote:

They DO.

Len, you're doing EXTRA WORK that is pointless.

Just stop doing the automated merges. Problems solved. It really is that easy. Don't do what David suggests - he does it because he's apparently _so_ comfortable with things that he prefers to do extra work just to keep his trees extra clean (I actually would disagree - but git makes that fairly easy to do, so if you prefer to have as linear a history as possible, you can do it with git pretty easily).

Now, I'm only complaining about _automated_ merges. If you have a reason to worry about my tree having clashes with your tree, do a real merge. For example, in your latest pull, you had a

	"pull linus into release branch"

merge, where you merged my v2.6.15 tree. That makes perfect sense.

What I object to is that there were _also_ two automated merges within ten hours or each other, with absolutely _zero_ development in your tree in between. Why did you do that in your development tree? By _definition_ you had done zero development. You just tracked the development in _my_ tree.

In case you wonder, the two commits I'm talking about are:

	add5b5ee992e40c9cd8697ea94c223628be162a7
	25da0974601fc8096461f3d3f7ca3aab8e79adfb

and neither of them have any reason to be in a development tree. You didn't develop them.

They are real merges, because you had a trivial patch in your tree (changing the acpi-devel mailing list address) that I didn't have, so when you pulled, your end result was thus always different from something I had (so you did a real "merge", even though it was totally trivial), but the point is that there is a difference between "the ACPI development tree" and "the tree that has random ACPI patches and then tracks Linus' tree as closely as possible".

See?

That's the most egregious example. There's two unnecessary pulls on December 28 and 29th too (commits 0a5296dc and c1a959d8).

You can do

	gitk 0aec63e..f9a204e1 

to see exactly what I see when I pulled from you. 11 commits, 5 of which are just trivial merges that are no development, just tracking _my_ tree. Of those, one makes sense (tracking a release).

(NOTE NOTE NOTE! It does make sense to track my tree in case you do big changes and you worry about clashes. Then you would want to synchronize those big changes with my changes, so that you can resolve any clashes early. So I'm not saying that tracking trees is always bad: I'm saying that doing so _unnecessarily_ is bad, because it adds no value, and it just makes the history harder to read).

Now, most people don't read the history. It gets messy enough quickly enough that it's hard to read anyway over time. My tree has tons of _real_ merges anyway, since it's by definition the one that is used for most synchronization, so my tree is always pretty hard to follow.

But my guess is that this probably makes it harder for _you_ to see what you've done too. If you didn't merge with me, then "git log" would show just your own changes at the top, and that's likely what you care most about anyway, no?

Also, if you didn't pull from me, and you decided that you needed to re-do your tree (let's say that you notice that one of your commits was bad _before_ you ask me to pull from your tree), then you'd also have an easier time re-creating your own development without that buggy change, exactly because _your_ tree wouldn't have my changed mixed up in it.

So your merges likely make git harder to use for you, not easier.

		Linus

On 1/9/06, Brown, Len <len.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

I think it does. All the tricky stuff that David and Junio have been discussing is actually done very transparently by

    git-rebase <upstream>

Now, git-rebase uses git-format-patch <options> | git-am <options> so it sometimes has problems merging. In that case, you can choose to either resolve the problem (see the doco for how to signal to git-am that you've resolved a conflict) or to cancel the rebase. If you choose to cancel the rebase, do

   cp .git/refs/heads/{<headname>,<headnamebadrebase>}
   cat .git/HEAD_ORIG > .git/refs/heads/<headname>
   git-reset --hard
   rm -fr .dotest

and you'll be back to where you started. Perhaps this could be rolled into something like git-rebase --cancel to make it easier, but that's about it. The toolchain definitely supports it.

cheers,

martin - To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, 9 Jan 2006, Martin Langhoff wrote:

Yes, it's fairly easy to do. That said, I would actually discourage it. I haven't said anything to David, because he is obviously very comfy with the git usage, and it _does_ result in cleaner trees, so especially since the networking code ends up being the source of a lot of changes, the extra cleanup stage that David does might actually be worth it for that case.

But git is actually designed to have parallel development, and what David does is to basically artificially linearize it. We merge between us often enough that it doesn't really end up losing any historical information (since David can't linearize the stuff that we already merged), but in _theory_ what David does actually does remove the historical context.

So "git-rebase" is a tool that is designed to allow maintainers to (as the command says) rebase their own development and re-linearize it, so that they don't see the real history. It's basically the reverse of what Len is doing - Len mixes up his history with other peoples history in order to keep them in sync, while David bassically "re-does" his history to be on top of mine (to keep it _separate_).

The "git-rebase" means that David will always see the development he has done/merged as being "on top" of whatever my most recent tree is. It's actually a bit scary, because if something goes wrong when David re-bases things, he'll have to clean things up by hand, and git won't help him much, but hey, it works for him because (a) things seldom go wrong and (b) he appears so comfortable with the tool that he _can_ fix things up when they do go wrong.

And yes, git-rebase can be very convenient. It has some problems too (which is the other reason I don't try to convince other maintainers to use it): because it re-writes history, a change that _might_ have worked in its original place in history might no longer work after a rebase if it depended on something subtle that used to be true but no longer is in the new place that it has been rebased to.

Which just means that a commit that was tested and found to be working might suddenly not work any more, which can be very surprising ("But I didn't change anything!").

On the other hand, this is no different from doing a merge of two independent streams of development, and getting a new bug that didn't exist in either of the two, just because they changed the assumptions of each other (ie not a _mismerge_, but simply two developers changing something that the other depended on it, and the bug only appears when both the working trees are merged and the end result no longer works).

So my suggested git usage is to _not_ play games. Neither do too-frequent merges _nor_ play games with git-rebase.

That said, git-rebase (and associated tools like "git-cherry-pick" etc) can be a very powerful tool, especially if you've screwed something up, and want to clean things up. Re-doing history because you realized that a you did something stupid that you don't want to admit to anybody else.

So trying out git-rebase and git-cherry-pick just in case you decide to want to use them might be worthwhile. Making it part of your daily routine like David has done? Somewhat questionable, but hey, it seems to be working for David, and it does make some things much easier, so..

			Linus

From: Linus Torvalds <torvalds@osdl.org>
Date: Sun, 8 Jan 2006 11:56:21 -0800 (PST)

The time at which I do the by-hand rebasing the most are the weeks leading up to a major release. The reason is to integrate bug fixes that I know conflict with the 80-odd patches I have queued up for the next development phase, or that I simply want integrated so that no _future_ development patches create conflicts.

I think merges with conflicts that need to get resolved by hand create a lot of noise and useless information and therefore to me they are pointless. But this is just my opinion. It simply works easier to me to shuffle the patches in by hand and deal with the rejects one by one. It's very much akin to how Andrew's -mm tree works.

I think a clean history is worth an extra few minutes of someone's time. And note that subsystem development is largely linear anyways.

--- Linus Torvalds <torvalds@osdl.org> wrote:

How about this usage (branch == tree):

Tree A    (your tree)
  Tree B     (project B, dependent on Tree A)
     Tree C     (project C, dependent on project B)

(i.e. diff(C-A) = diff(C-B) + diff(B-A))

Your tree is pulled into Tree A as often as your tree changes and it just fast forwards.

If I want to run project B with your latest tree, then I resolve/merge from tree A to tree B, compile B and run it.

If I want to run project C and project B with your latest tree, I resolve/merge from tree A to tree B and from tree B to tree C, compile C and run it.

In such cases, are you saying that you'd prefer to pull from Tree B and Tree C (depending on your needs)?

Another question: Sometimes, a fix for project B finds its way into tree C (project C) (since C depended on that fix in B). Now I'd like to pull that particular fix, identified by its SHA, into project B, and nothing else, for this I can use git-cherry-pick, right?

And lastly, is there a tool whereby I can "see" changes between repos, kind of like git-diff but being able to give URLs too?

    Luben

On Sun, 8 Jan 2006, Luben Tuikov wrote:

No.

If tree B is based on _some_point_in_ A, then you just test that.

Because development line B is _independent_ of development line A. The fact that A changes doesn't change B - unless they have some real dependencies (which we should try to avoid).

So when you update ("fetch" in git parlance) branch A from me, that shouldn't affect branch B _nor_ branch C in any way. They clearly do not depend on the new stuff in A, since they do their own independent development. The fact that they _started_ at some random point during the development of A doesn't change that fact.

Now, if you want to _test_ the combined "new stuff in branch A and new stuff in branch B", feel free to do that. But realize that that is _not_ appropriate in either branch A _nor_ branch B.

So you'd be much better off with a separate "test" branch that you test stuff out in, and you then resolve ("pull" in git parlance) both branch A and branch B into that test branch.

See? Testing the combination of two branches doesn't actually have anything to do with either branch.

At some point, you decide that you want to merge what you've done in branch B. That's a _different_ and independent thing from deciding that you want to test the combination of two development branches. Clearly, it's great to test often, but that has nothing to do with releasing a branch.

I'm saying that mixing up the "let's test the combination" and "let's merge the two branches" are totally different things and should not be mixed up.

One is a random event (and then it makes sense to have, for example, a "automated test branch" that automatically merges every day and tests the results. I don't think you should expose those random merges to others, because they actually hinder the readability of the history for _both_ sides.

The other is a _directed_ event. It's the event of saying "branch B" is now ready to be merged. Usually that's best done by just saying "please pull now" - ie not by merging branch A into branch B (because that's not what you actually want, is it? What you want is for the development in branch B to show up in branch A - so you want branch A to do the pull).

Now, there's a third kind of event, which is again independent of the other two. It's more of a "let's try to keep the 'topic branch' development up-to-date with the branch we eventually want to merge the topic changes into". That's where you can now do two things:

 - David often "rebases" all of the changes in his "topic branch" (ie 
   conceptually "branch B") to the new top-of-head of "branch A". In other 
   words, he re-writes branch B entirely _as_if_ it was based on the newer 
   state "branch A". This is what "git rebase" is all about.

 - You can just pull from branch A into branch B, as a way to keep branch 
   B more up-to-date with the work in the "main trunk" or whatever. This 
   is ok, but it shouldn't be a common event. It should be something that 
   happens when you (for example) notice during testing that the test 
   merge no longer works cleanly. Or it might be "It's now been two weeks 
   since I synchronized, let's just synchronize to be safe".

See? I'm not objecting to topic branches pulling from my tree in general. It's just that they should have a _reason_. There's never any reason to pull into a development tree that you haven't done any development in, just because you also want to use that development tree for testing.

That's one way. It's often the best way, especially if it's a really obvious bugfix. Or you could just fix it in your tree yourself. It will mean that the two branches have the same fix, but especially if it really is an identical fix, it won't be a merge problem.

You _can_ just decide to pull branch B into branch C, but that has a real problem, namely that it inexorably links the two together, so that nobody can then pull branch C without pulling indirectly branch B at the time that B->C merge happened. Sometimes that is ok. But it's nice to avoid it if you can.

But for example, if somebody fixed something in the trunk, and you actually do need that fix from the trunk for your topic branch development, then just doing a pull is _fine_. Now we're back to doing a merge that actually has a perfectly good reason.

IOW, don't cherry-pick to avoid merges when the merge really does make tons of sense. Merges are good, it's just that _too_ much of a good thing is bad.

No, all the good tools really are based on fetching (NOT "pulling") the other branch into your local tree as a separate branch. At that point, there are tons of wonderful tools you can use.

In other words, say that you want to know what has happened in another repository, at git://git.kernel.org/xyzzy. You aren't interested in the stuff that is already part of the trunk, you're just interested in what is only in that "xyzzy" branch, and how it relates to your code.

What you'd do is

	git fetch git://git.kernel.org/xyzzy master:xyzzy-snapshot

which says "fetch the 'master' branch from that xyzzy repository, and call it 'xyzzy-snapshot' locally.

You can then (for example) fetch the code that is in _my_ tree by doign the same time (just call that branch 'linus'), and you can now do

	gitk linus..xyzzy-snapshot HEAD

which looks strange (you give "gitk" _both_ a range from the "linus" branch to the "xyzzy-snapshot" _and_ your own HEAD at this time), but what it basically does is that the "linus.." syntax tells git that you're not interested in anything that is already in the 'linus' branch.

So the above command line will actually graphically show _both_ your current HEAD branch _and_ the 'xyzzy-snapshot' branch, in parallel. You can see how (if at all) they are related to each other, ignoring all the commits that have already made it into my tree.

(You can also do "linus..HEAD" instead of just HEAD and effectively repeat the "don't show 'linus' branch any more" twice. It's perfectly equivalent, of course. You may also want to use the "-d" flag to "gitk" which tells it to show things in date order, instead of a simplified history order).

Or just do "what has xyzzy-snapshot that I do not have in my HEAD":

	git log HEAD..xyzzy-snapshot

(or gitk), or the other way around: what do _I_ have in my HEAD that hasn't been pushed to xyzzy-snapshot yet:

	git log xyzzy-snapshot..HEAD

(or do diffs, "git whatchanged -p", or whatever).

In other words, using a few different branches (you can make them up dynamically) can be very powerful.

		Linus

Martin Langhoff <martin.langhoff-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

Careful.  I do not think rebase works across merges at all.

- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html

On Sun, 8 Jan 2006, Junio C Hamano wrote:

Right. You have to do one or the other (rebase your changes to another tree _or_ merge another tree into your changes), but not mix the two.

		Linus

I'll try to update the using-topic-branches document to capture this. Some of the problem is that it doesn't quite capture what I'm doing with my test/release branches.

My release branch really is just used as a transfer point to Linus. I usually[1] don't leave patches sitting in "release" for long enough that I'll be tempted to merge in from Linus ... once I decide that some patches are ready to go to Linus I'll update "release" from Linus (which will be a fast-forward, so no history) merge in the topic branches, do one final sanity build, push to kernel.org and send the "please pull" e-mail.

The huge majority of my "automatic update from upstream" merges go into my test branch ... which never becomes part of the real history as I never ask Linus to pull from it.

-Tony

[1] Sometimes I goof on this because I forget that I've applied a trivial patch directly to the release branch without going through a topic branch. I think I'll fix my update script to check for this case.

On Sun, Jan 08, 2006 at 11:41:28AM -0800, Linus Torvalds wrote:

My impression is that you and Len are talking at different levels.

I can't speak for Len, but let me try to describe a problem in this area I don't know the solution for:

Consider I want to do the following:
1. update my tree daily from your tree
2. include 10 patches per week into my tree
3. ask you once a month to pull from my tree

How should step 1 be done?

In CVS, I'd do a "cvs update -dP ." In cogito, the equivalent command seems to be "cg-update".

CVS has no problems if I have changed MAINTAINERS in one place and it changes daily in your tree in other places, but how do I do the same in git/cogito without creating the merges you don't want to see?

The solution might be described somewhere in TFM, but this is the class of problems people like me run into when the goal is simply a git tree to both track your tree and send changes to you without any interest in advanced SCM knowledge.

cu Adrian

Hi Adrian,

On Mon, Jan 09, 2006 at 12:06:11AM +0100, Adrian Bunk wrote:

I believe we all have the same problem. The only solution I found for this was to proceed like David described. I know even have a 'git-patches' directory next to my git repo to keep the resulting patches after I do a 'git-format-patch --mbox'.

When Linus called it the 'stupid content tracker', he was half right. In fact, it's more a 'changes tracker' than a 'content tracker'. It logs everything you and others do, so if you don't want your operations to appear on others' history, you have to hide them by working on temporary trees to generate the patches you will use later.

At first I found this very annoying, but finally, it's a way to ensure that I always have clean and ordered patches. It's not much different from what I was doing by hand previously, it's just that all git-xxx operations take much longer time, possibly because of the compression. On the other hand, its ability to understand mbox saves me some time.

What sometimes worries me is that some operations seem so much complicated that there is a high risk of doing the wrong thing, and I'm not certain that this will reduce the number of mistakes I do in a month, compared to manual patching. I became a real addict to 'git-reset --hard' ...

Regards, Willy

- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, 9 Jan 2006, Adrian Bunk wrote:

I'd do

	git fetch linus

to fetch my tree as a branch (obviously, this assumes you've set up a '.git/remotes/linus' file for the shorthand).

Then, keep your checked-out working-tree that you also do you development in your 'devel' branch (or whatever).

And then do

	git-rebase linus

to rebase your development branch to mine.

THIS is what "rebase" is for. It sounds like what you really want to do is not have a development branch at all, but you just want to track my tree and then keep track of a few branches of your own. In other words, you don't really have a "real" branch - you've got an odd collection of patches that you really want to carry around on top of _my_ branch. No?

Now, in this model, you're not really using git as a distributed system. In this model, you're using git to track somebody elses tree, and track a few patches on top of it, and then "git rebase" is a way to move the base that you're tracking your patches against forwards..

It's also entirely possible that you may want to look at "stacked git" (stg), which is really more about a "quilt on top of git" approach. Which again, may or may not suit your needs better.

Now, the other alternative is to use git as a "real" distributed system, and then you might keep my "linus" branch around perhaps as a reference point, but you don't care too much about it. What you care a lot more about is your "real development" branch, and you simply don't rebase that, or try to track my branch all that closely. You work in your real development branch, and that's your bread and butter. You _don't_ merge my tree every day, because you simply don't care - that's a separate issue.

You might have a totally different directory that you use _just_ to track my tree, and that is my virgin tree checked out and ready to go to test what _I_ am doing - totally independently of your tree.

See? Two totally different usage schenarios. In one, you keep a couple of "odd-ball" patches around (hey, it could be many, but I say a couple just because the patches aren't a huge deal - they are kind of a small side project to you, and not the main focus). And you use "git rebase" to move those patches forward to match the "real" tree, aka mine.

In the other schenario, your development tree is a full-fledged real branch, with a life of its own, and _not_ slaved to what happens to be going on in my tree. You may care about my tree for _other_ reasons, but the two aren't really joined at the hip.

The third schenario is somewhere in between: you do pull from my tree, but you do it occasionally enough that the merges don't get annoying.

Most people I work with are actually in that gray area. EVERYBODY pulls some. Nobody tends to do black-and-white either-or schenario. The question is just how much.

My gut rule: if you have almost as many merges as you have "real work" commits, you're doing something bad, and you should pull less often (perhaps by using "rebase", perhaps by just realizing that you don't need to be tailgating _quite_ that closely).

But 30 "real" commits and 5 merges because the 30 real commits happened over four weeks time? Sounds fine to me.

		Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sun, Jan 08, 2006 at 07:26:50PM -0800, Linus Torvalds wrote:

Yes.

I am using the workaround of carrying the patches in a mail folder, applying them in a batch, and not pulling from your tree between applying a batch of patches and you pulling from my tree.

After a quick look, stg seems to be an interesting project that might suit my needs.

I'd say the main problem is that git with several other projects like cogito and stg on top of it allow many different workflows. But finding the one that suits one's needs without doing something in a wrong way is non-trivial.

It might help if someone could write some kind of "Git for dummies" that focusses only on the usage and advantages/disadvantages of user interfaces like cogito, stg and (H)GCT and restricts the discussion of git internals to a short paragraph in the introduction.

cu Adrian

On 1/11/06, Adrian Bunk <bunk@stusta.de> wrote:

In that case, there's a mostly automated way of doing that if you read the last couple lines of git-rebase, using something along the lines of

      git-format-patch <yours> <linus> | git-am -3 -k

You are right about that, but much of the space (of what workflows are interesting) is still being explored, and git and the porcelains reacting to people's interests. So it's still a moving target. A fast moving target.

cheers,

martin

Martin Langhoff wrote:

Isn't this rebase in a nutshell ?

Good thing there are competent people around to snipe those targets in mid-stride. :)

I for one was amazed at how much easier git was to work with than any of the other scm's I've tried (quite a few, I never really liked any of them), and I really like the fact that it's flexible enough to suit (almost) all our needs. The only thing I haven't really found it to be satisfactory for is our collection of RPM spec-files and their respective patches, where we not so much change files as continuously replace them completely. Perhaps that's changed now that most git-commands can be run from subdirs.

So, kudos to Linus for inventing it, Junio for nursing it, and the other 129 developers that have so far contributed to the current release.

On Tue, Jan 10, 2006 at 09:19:09PM +0100, Adrian Bunk wrote:

Ick, I'd strongly recommend using quilt for this. It works great for just this kind of workflow.

thanks,

greg k-h

On Wed, Jan 11, 2006 at 05:37:06PM -0800, Greg KH wrote:

It works in my case because I'm only going through the folder with the trivial patches in batches and ask Linus to pull from my tree immediately after I'm finished.

That would certainly not be a recommended practice for a subsystem maintainer, but I'm handling only trivial patches.

cu Adrian

On 1/9/06, Linus Torvalds <torvalds-3NddpPZAyC0@public.gmane.org> wrote:

FWIW, I determine whether I should rebase or merge based on

 + Whether the branch/head I maintain is public. For public repos, I
*must* merge carefully as rebase "rewinds" the head and that makes a
mess of any repositor tracking me.

 + Whether the changes on my both sides are significant, and it is
semantically meaningful to have a merge. If either side had just a
couple of minor commits, rebase makes life a lot easier down the path.
If both side clearly saw parallel development, it is more sincere to
merge and let that be recorded.

 + If my attempt to rebase leads to any non-trivial conflicts or
co-dependencies, then I definitely cancel the rebase and merge.

I'd argue that it is not about distributed or not. It's all in what you want to record in your history. As such, it is a communication device -- and I want to make effective use of it. I guess the question I ask myself is: what will communicate what's happened here most clearly? What will be useful for people to read? In that context, a white-lie here and there simplifying the history a bit where it's not interesting counts as a good thing.

cheers,

martin - To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html

On Tue, 10 Jan 2006, Adrian Bunk wrote:

Yes, that also works.

I think "quilt" is really the right thing here, although stg may be even easier due to the more direct git integration. But with a smallish number of patches, just doing patch management by hand is obviously simply not a huge problem either, so extra tools may just end up confusing the issue.

		Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Greg KH <greg-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> wrote:

Or StGIT :-). Similar workflow (and similar commands) but better integrated with GIT and better at dealing with conflicts since it uses a three-way merge when pushing patches rather than applying them with "patch".