What's in git.git

I haven't heard major breakage around the new features scheduled for 1.2.0 so far, except for the two-tree "diff-tree --cc" Linus has already fixed, so the previous "What's new" is pretty much unchanged.

One *major* change I am thinking about doing is to change my workflow a bit. So far, the proposed updates branch "pu" was almost impossible to follow unless you are really a devoted git developer, because it is always rebased to the latest master and then topic branches are merged onto it. While that keeps the number of unnecessary merge nodes between master and pu to the minimum, it actively discouraged for the branch to be followed by developers.

I would like to rectify that.

So I have created another branch, "next". This is managed quite differently from "pu". I'd promise these things:

 * It is to contain planned updates and merge from topic
   branches, just like "pu" currently does.  However, the topics
   merged there will not contain majorly whacky / unproven ones
   like bind commits and shallow clones, until the basic part
   proves sound during the list discussion.

 * I will not rewind or rebase the "next" branch.  Also I will
   not rebase the topic branches that are merged into it.

 * It would occasionally merge from "master" if only to prevent
   conflicts.

 * If there are patches sent to improve a topic branch in it,
   they will be applied to the topic branch, and then the topic
   branch is merged into "next", without any funny rewinding or
   rebasing of "next".  This will make the "next" branch
   cluttered with repeated merges from the same topic branch,
   but that is OK.  "next" will not be merged into "master",
   ever.

 * Once a topic is fully cooked, the topic branch will be merged
   into "master".

What this means is that "next" should be as easy to follow as "master", but still is slightly ahead of "master" with not so wildly experimental features.

Although there theoretically is no reason not to follow the above principles I set for "next" to manage "pu", it will stay wild for now until I get more comfortable with this workflow.

Now, what's in "next"? Currently I have two topic branches merged to it.

    * jc/nostat:
      ls-files: debugging aid for CE_VALID changes.
      "Assume unchanged" git: do not set CE_VALID with --refresh
      "Assume unchanged" git

    * jc/empty-commit:
      t6000: fix a careless test library add-on.
      Do not allow empty name or email.

On Wed, 08 Feb 2006 22:47:54 -0800 Junio C Hamano <junkio@cox.net> wrote:

I've always followed it okay by just using "git branch -d pu" each time before pulling from you. Your "next" branch does sound like an improvement though.

Sean

sean wrote:

I thought

	Pull: +pu:pu

was supposed to handle such things automatically. It has always pulled properly for me anyways.

On Thu, 09 Feb 2006 10:04:53 +0100 Andreas Ericsson <ae@op5.se> wrote:

The only problem with that is that Junio rebases and discards commits periodically that will still be in your local pu branch. The fetch/merge logic doesn't notice that commits have disappeared from Junio's pu branch. So you'll end up with a union of all the pu branches in your local repo with commits that were dropped and never merged into mainline by Junio.

Unless you add changes to the pu branch locally you should never need anything but a fast forward when pulling from Junio. Except it breaks when he rebases things. The easy hackish "fix" is just to delete and repull the branch which is always small anyway.

Sean

Andreas Ericsson <ae@op5.se> writes:

Yes, fetching to look at is no problem, but what I wanted to solve is that you cannot easily _touch_ it. The point of this is to make improving on top of what is still _not_ in master easier for the contributors.

If you want to improve upon what is in the current "pu", the natural thing for you to do would be:

	$ git fetch git://git.kernel.org/pub/scm/git/git +pu:pu
	$ git checkout -b my-pu pu ;# initial
        $ hack on it and git commit many times
        $ git format-patch --stdout pu..my-pu |
          git send-email --to junkio@cox.net --cc git@vger.kernel.org

(Side note: I do not know git-send-email would work like the above, but if it did that might be handy. Ryan?)

But sometimes you may take more time than how my "pu" progresses, and you would want to sync your work to my updated "pu". A natural thing you would want to do is this:

        $ git pull git://git.kernel.org/pub/scm/git/git +pu:pu

Unfortunately, this would _not_ work very well, because by the time you pull from my "pu" again, it would have rewound and rebased. You would end up seeing unnecessary merge conflicts.

Another possibility would be:

        $ git fetch git://git.kernel.org/pub/scm/git/git +pu:pu
        $ git rebase pu

This helps somewhat because "git rebase" uses "git cherry" to detect the same patch with different commit ID in "pu" that you already have in "my-pu". But my topic branches have been sometimes rewound and even rewritten to fix minor points (using "reset --soft HEAD^" followed by "commit -a -c ORIG_HEAD"), and when that happens "git rebase" would not be of much help.

The updated workflow on my part is trying to reduce these problems by (1) not rewinding nor rebasing "next" and (2) not rewinding nor rebasing the topic branches merged into "next".

Strictly speaking, the latter is not necessary (I would need to resolve conflicts when merging the rewound/rebased topic branches into "next", but after that is done, contributors who pulled "next" do not have to deal with that, as long as "next" itself is not rewound/rebased), but that way you could disect component topic branches more easily out of "next".

For example, as of this writhing, my "master" and "next" look like this:

    $ git show-branch --topo-order master next
    * [master] .gitignore git-rerere and config.mak
     ! [next] Merge branch 'jc/nostat'
    --
     - [next] Merge branch 'jc/nostat'
     + [next^2] "Assume unchanged" git: --really-refresh fix.
     - [next^] Merge branch 'jc/ls-files-o'
     + [next^^2] ls-files: honour per-directory ignore file ...
     - [next~2] Merge branches 'jc/nostat' and 'jc/empty-commit'
     + [next~2^3] t6000: fix a careless test library add-on.
     + [next~2^3^] Do not allow empty name or email.
     + [next^2^] ls-files: debugging aid for CE_VALID changes.
     + [next^2~2] "Assume unchanged" git: do not set CE_VALID...
     + [next^2~3] "Assume unchanged" git
    *+ [master] .gitignore git-rerere and config.mak

If you want to help fixing my thinko in jc/nostat branch, you could:

	$ git checkout -b jc/nostat next^2
        $ fix fix fix; git commit

By convention, merge records what was the tip of the branch as the first parent, and the second parent (and subsequent ones if it is an Octopus) is the tip of the branch that was merged in, so you can tell "next^2" is what was merged into the branch to advance "next"; in other words, that is the tip of the jc/nostat branch. Similarly, you can tell the tip of jc/empty-commit was merged to next~2 in an Octopus as the second merged-in branch, so you can tell that its tip is next~2^3.

You could even publish your jc/nostat branch after you built on it and tell me to pull from it to fix my stupidity.

Junio C Hamano wrote:

This is exactly what I do when I improve upon things in master, and according to numerous emails this is the recommended workflow.

With my (still un-published) git-send-patch you could do

	$ work, work, work
	$ git send-patch -s "Some subject for a prelude message" pu

and it would do the right thing.

I guess I'll have to get around to sending that thing in sooner or later.

Do you mean
	$ git pull git://git.kernel.org/pub/scm/git/git +pu:my-pu

? Otherwise, I don't see how I can end up with merge-conflicts.

Using my own topic-branch, this is what I always do. Conflicts that occur that way are always in my patches, so they would have to be reworked anyway. The new rerere tool should help if I dally too long.

Perhaps I'm just weird, but I never touch published branches.

Andreas Ericsson <ae@op5.se> writes:

Yes.

I do mean "+pu:pu". In my illustration, "pu" is used in your repository to track "pu" retrieved from me, and "my-pu" is a fork you created from it and you build your changes upon.

	$ git pull $URL +pu:my-pu

is a shorthand for:

	$ git fetch $URL +pu:my-pu
        $ git merge "auto merge message" HEAD my-pu

and you definitely do not want to _fetch_ into my-pu when you are on my-pu.

The problem is exactly why you need the plus sign when you fetch, i.e. "+pu:pu". My "pu" rebases.

Suppose I had this:

             o--o--o
            /      "pu"
	o--o
           "master"     

You do fetch +pu:pu, branch my-pu, and build on top of it:

                     o--o--o--o--o--o--o
                    /                  "my-pu"
             o--o--o
            /      "pu"
	o--o
           "master"

I add some to my "master" and rebuild "pu", maybe while adding another commit on "pu". You fetch +pu:pu again:

                     o--o--o--o--o--o--o
                    /                  "my-pu"
             o--o--o        o--o--o--o
            /              /         "pu" 
	o--o--o--o--o--o--o
                          "master"

Now, what happens when you merge "pu" into "my-pu"? The three commits I had on my previous "pu" are not part of the history of the updated "pu" anymore, but is considered to be part of your development trail. If these had an addition of a file, and if your development on top of the previous "pu" modified it, the merge would result in:

 * originally the file did not exist.
 * "pu" adds it one way.
 * "my-pu" adds it in another way.

This requires a hand merge. What should be done is for me to instead of rebasing "pu", merge the updated master to "pu".

                     o--o--o--o--o--o--o
                    /                  "my-pu"
             o--o--o--------*--o
            /              /   "pu" 
	o--o--o--o--o--o--o
                          "master"

Then merge between "my-pu" and "pu" become easier. You do not have to worry about the earlier three commits, because the point you forked from the previous "pu" becomes the merge base.

The reason I have not done it that way so far is primarily I am lazy and also I do not like to see too many merges in the log. Also "pu" tends to have really wacky stuff, so separating out only usable bits, excluding wacky ones is slightly easier if I rebuild it from scratch.

The new "next" aka "not too close to bleeding or broken edge" branch will be managed like the last picture above, in order to make working with it easier to manage. This is only usable if I do not include too bleeding-edge topic branch in it.

Junio C Hamano wrote:

But wouldn't rebase detect the commits as being the same, unless you've made changes to them? If it doesn't, can we teach it to discard parent info and re-hash the commits if they conflict? That should solve most such merge-conflicts, really.

Good thinking. You're a marvel at explaining things.

Andreas Ericsson <ae@op5.se> writes:

Yes, rebase would help somewhat (I said that didn't I?). But the thing is, with the "pu" workflow of mine so far, I _did_ rewrite/replace commits on my topic branches, especially when they are young and not in a good shape.

For example, the topic branch jc/nostat ("Assume unchanged" git) has four commits since it forked from the mainline:

 + [jc/nostat] "Assume unchanged" git: --really-refresh fix.
 + [jc/nostat^] ls-files: debugging aid for CE_VALID changes.
 + [jc/nostat~2] "Assume unchanged" git: do not set CE_VALID with --refresh
 + [jc/nostat~3] "Assume unchanged" git

With the "pu" workflow, I would have merged the "do not set CE_VALID" commit and "--really-refresh fix" commit into the first "Assume unchanged" commit after I found out about these two small mistakes. So one day "pu" would have contained what is there right now as "jc/nostat~3", but the next day it would have a commit, perhaps with slightly modified log message from what is there as "jc/nostat~3" to contain fixes jc/nostat~2 and jc/nostat have right now (the ls-files one is a debugging aid so I would have left it separate even with the rewriting-history workflow).

That kind of rewriting history is not being honest, but the end result is that people do not have to see intermediate states and earlier mistakes when things are fully cooked and ready to be merged into the mainline. By promising not to rewind "next" and topic branches that go to "next", I am closing the door for me to do this kind of history rewrite freely. I can still rewrite things I have not pushed out yet, though.

BTW, it is always a judgement call if it is a good thing to squash commits into one like this. Being too honest hurts the usability of the history. Especially if you have a trivial "Oh, what I checked in does not even compile" kind of mistakes left in the development trail, that would inconvenience bisection. Being too sanitized OTOH tends to drop a single big ball of wax into the history, and makes correcting things harder if it is found later that only parts of that change are desired and the other parts are not.

Hi,

On Wed, 8 Feb 2006, Junio C Hamano wrote:

I never quite understood why you not just publish your topic branches. IMHO what you intend to put into "next" should be put into "master" anyway: everyone interested in git development should try the new features as early as possible. If there is a "whacky" feature, you can put it into "whacky/archexport" or something along the lines.

Ciao, Dscho

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

Yes, but I've been trying to be _very_ conservative to keep "master" clean and stable, as I said in my inauguration speech.

Since git is still young and we are building features that are needed in the field every day, it is very beneficial for users to keep up-to-date with "master", and I would really like to encourage that. It saddens me to see git patches posted to the kernel list marked with 0.99.9.GIT by prominent kernel people.

However, I do not want to see their time wasted on getting bitten by stupid bugs I carelessly place on the "master" branch. So I'd like to keep "master" conservative, stable and boring, at least for now.

Instead of introducing "next", I could treat "pu" the way I said I would do "next". But even if I rid of its constant rewinding nature, "pu" tends to have intrusive stuff near its tip and is very hard to build on top of it. Patches against the tip of "pu" to fix things unrelated to the whacky ones often would be inapplicable to "master". This is especially true with what are currently pending near the tip of "pu" (bind commits and shallow clones). I do not forsee them to graduate to "master" any time soon. Not in their current shape.

The promised "next" should be much easier to build on top of, without disecting it into component topic branches, and it would be the branch to track for people interested in git development if you want to stay closer to the edge without touching bleeding or even broken edge. Making it easier to participate in git development by people interested is what I am aiming at here.

I've considered publishing the topic branches individually. Branches are cheap from the storage point of view (not really, one inode and a filesystem block wasted to store only 41-bytes ;-)), but it needs management time and care (I will need to remember to go to the repository and remove stale ones once they are merged up). Since branches in "next" are meant to be short-lived, I am hoping it is easier for me to bundle them up like I am planning.

On the other hand, long-lived whacky intrusive ones might be better published as individual branches.

Hi,

On Thu, 9 Feb 2006, Junio C Hamano wrote:

Not really. I use reiserfs which is quite efficient on these small files.

Hth, Dscho

On 2/8/06, Junio C Hamano <junkio@cox.net> wrote:

This is pretty much the workflow in my test/release branches (mostly documented in Documentation/howto/using-topic-branches.txt).

I've sometimes wondered about re-creating the topic branches in the case where there have been a series of follow-on commits before pulling them into the release branch. The goal would be to present history not as it was, but as it should have been if we didn't have all the dumb mistakes and typos.

So is there an easy way in git to take the series of commits from a topic branch, make a new branch with all those commits as just one commit ... with an open editor on the concatenated commit comments (If there were just typo fixes the comment from the first commit would apply, but sometimes the follow-on commits would have substantive changes).

-Tony

On Thu, Feb 09, 2006 at 03:14:59PM -0800, Tony Luck wrote:

git checkout topic git format-patch --stdout origin > topic-diff $VISUAL topic-diff # Fix comments git checkout master git checkout -b new-topic master git-am topic-diff

.. done?

(I typically do something akin to this before sending patches to Junio, by looking at the output of format-patch in a directory, editing, combining a few changes if necessary, then re-committing, re-running format-patch, and sending the output upstream.)

Tony Luck <tony.luck@intel.com> writes:

I do not have a script to do so but my guess is that would be a 20-30 line shell script. Use cherry to find which ones to pick, run diff-tree on them to extract patches, run apply --index on each of them in turn, and dump "git log master..topic" to your editor and you are done. The editor would have the commit log to be edited while working tree and the index would have a ready-to-commit tree with all the changes applied.

Tony Luck <tony.luck@intel.com> writes:

Yup.  Sorry I did not make that clear.  You deserve the credit.

I am beginning to feel this workflow might benefit from some tool support, but I haven't had enough experience to talk about exactly what they are yet.

For example, listing topics that have ever been merged into a particular branch, listing topics that have not been fully merged into a particular branch, etc. are things I find myself doing frequently. I vaguely recall seeing your post that has these things.