What's cooking in git/spearce.git (topics)

Here are the topics that have been cooking. Commits prefixed with '-' are only in 'pu' while commits prefixed with '+' are in 'next'. The topics list the commits in reverse chronological order.

I know there are a number of other topics that I don't have in my tree yet (bisect improvements, push refspec matching improvements, option parser). I plan on loading those into 'pu' tomorrow. In the meantime we have this in next.

* db/fetch-pack (Thu Oct 11 01:47:55 2007 +0100) 58 commits
 + fetch: if not fetching from default remote, ignore default merge
 + Support 'push --dry-run' for http transport
 + Support 'push --dry-run' for rsync transport
 + Fix 'push --all branch...' error handling
 + Fix compilation when NO_CURL is defined
 + Added a test for fetching remote tags when there is not tags.
 + Fix a crash in ls-remote when refspec expands into nothing
 + Remove duplicate ref matches in fetch
 ... and many more ...

 The above commits are all new since Junio's last published next.
 About half of them fix known bugs in the builtin-fetch series.
 If you are running next, you want these fixes.

 I'm actually really happy with this series, but given the set of
 bugs that the above commits had to fix I think it needs to cook
 more in next before the topic can safely can graduate into master.

* jc/am-quiet (Mon Oct 1 00:27:51 2007 -0700) 2 commits
 + git-am: fix typo in the previous one.
 + git-am: make the output quieter.

 I'm also really happy with this change.  If nobody objects I may
 move it over to master later this week.

* lt/diff-rename (Tue Oct 2 19:28:19 2007 -0700) 1 commit
 + optimize diffcore-delta by sorting hash entries.

* kh/commit (Mon Sep 17 20:06:47 2007 -0400) 4 commits
 + Export rerere() and launch_editor().
 + Introduce entry point add_interactive and add_files_to_cache
 + Enable wt-status to run against non-standard index file.
 + Enable wt-status output to a given FILE pointer.

* js/stash-create (Mon Jul 9 00:51:23 2007 -0700) 2 commits
 + rebase: allow starting from a dirty tree.
 + stash: implement "stash create"

I inherited the above three topics from Junio. I haven't had a chance to really look at these yet myself to understand what their status is and how ready they are for master (or not ready as the case may be).

Hi,

first let me thank you for being the interim maintainer. I know it is much work, and I frankly do not have the time, or nerve, to do it. Out of curiousity: did you use the scripts in "todo" to send these emails?

On Tue, 16 Oct 2007, Shawn O. Pearce wrote:

AFAIR this was ready to go to master, with a 5-10% speedup or so, just needing a bit of testing. Which it should have gotten by now.

This is the beginning of the builtin-commit. The option parser has to go in before that (it was split out from the builtin-commit series), and the (minimal) adjustments to builtin-commit.c for the now-changed option parser have to be done.

So I think this topic should stay in master until builtin-commit is there, too.

This needs more work in rebase-i, and Junio indicated that he's not completely happy with it.

It would serve to be able to rebase in a dirty tree, by first stashing away the changes, and then applying them on top of the rebased branch.

I think that this would avoid many "Huh?" effects, but it should try to "git stash apply --index" first, falling back to "git stash apply".

Something like that would be very nice for git-pull, too, I guess.

However, I have not thought through all implications.

Ciao, Dscho

On Tue, Oct 16, 2007 at 12:16:28PM +0100, Johannes Schindelin wrote:

I've recently started trying to use some of the scripts in "todo" to send similar "What's cooking" messages, and started wondering if they were what Junio actually used in production to send his notes. For example, the scripts don't work particularly well if the refs have been packed. So I had to make changes such as these so they would work for me.

I was waiting for Junio to get back from vacation to ask if he was interested in accepting such changes, and/or changes to make the scripts more general purpose, and possibly useful for other projects.

		     	      	     	       - Ted

Theodore Tso <tytso@mit.edu> wrote:

I think Junio just makes sure he doesn't pack his refs.  :-)
 

Perhaps `git for-each-ref --format='%(refname)' refs/heads` is the better change here as then you can avoid the pipe into awk. We still need the sed to strip the refs/heads/ off the results though.

I'm actually applying it with the show-ref variant and will wind up pushing it into my tree later tonight. Because I do pack my refs.

Theodore Tso <tytso@mit.edu> writes:

Sorry, WI is for "what's in", WC is for "what's cooking". I should remove PU and RB from there.

On Mon, Oct 22, 2007 at 06:15:09PM -0700, Junio C Hamano wrote:

I assume PU is what you used to build your proposed-update branch? I'm actually trying to use it, and it is useful, although it hasn't been working for me completely perfectly. I've still been trying to figure out if the reason why it hasn't been working quite right is due to my not understanding how to use it correctly, or whether you don't use it these days.

One question which I have had about the WC script is that if I manually add a commit to the next branch, it ends up showing up in all of the topic branches as a commit that was part of that topic branch which is in next. So I can manually edit the output of WC to remove the spurious commit, or I could make it be a rule that all new commits either must go against the master branch, or be put in a topic branch based off of master, and then merged into next. Or maybe I'm not understanding how to make the WC and git-topic.perl script work and sing for me perfectly?

					- Ted

Theodore Tso <tytso@mit.edu> writes:

Ah, these days I almost always do:

	git checkout pu
	git reset --hard next

and then merge the topics that haven't been merged anywhere by hand, using output from

        Meta/git-topic.perl

as the guide.

Well, the policy is never to commit directly on top of next (iow, only merge other topics and nothing else). Otherwise it becomes hard to allow individual topics graduate to 'master' independently.

On Mon, Oct 22, 2007 at 06:29:59PM -0700, Junio C Hamano wrote:

I see. So if it's non-trivial enough that you want it to "cook" in next for a cycle, you'll create a topic branch for it (based off of 'master'), and then force a merge into 'next'?

					- Ted

Theodore Tso <tytso@mit.edu> wrote:

Yes. Because 'next' always has commits in it that never appear in 'master'. So any topic forked from master must merge into next. It can't be a fast-forward. No forced merging required.

Of course this isn't true for a new project. That first topic that forked from master to *create* next will be a fast-forward as it creates next. But that's no big deal. The second topic will merge into next, and that first topic can still be merged back into master without merging next (or the second topic).

I was also doing the same thing Junio already explained to manage next and pu while he was away. Except I shortcut his:

	git checkout pu
	git reset --hard next

as:

	git branch -f pu next
	git checkout pu

as I'm was usually already sitting on next. This saved my poor little laptop from a second of IO chugging as it slewed around between the two versions. There were no files to update as it switched from next to pu, and pu was already setup for merging the proposed topics. :-)

On Tue, Oct 23, 2007 at 12:05:22AM -0400, Shawn O. Pearce wrote:

Why is it the case that 'next' always commits that never appear in 'master'. So far in how I've been doing things that hasn't been the case. When I do a "git checkout master; git merge next", it's always been a fast-forward merge.

Oh, I see. That's because if you put some trivial changes in 'master', and then pull those changes into next, there will be merge commits in 'next' that will never be in 'master. Is that it?

I had been trying to avoid that case by always putting new commits, even trivial ones, into 'next', and then having them drop into 'master' at the next cycle; so 'master' was always trailing 'next', but they were always the same commit string (i.e., 'master' was always a subset of 'next').

Aside from the WC script not working right, are there other disadvantages to my doing things that way as opposed to the way the Junio has been running the git repository?

						- Ted

Theodore Tso <tytso@mit.edu> wrote:

Exactly.  This of course means that next has been growing in distance
from master for quite some time.  It has well over 1000 commits now
in git.git that aren't in master.  Most of those will never merge
there either.
 

The reason Junio does what he does is flexibility.

By merging only individual topics forked from master into next you can merge those individual topics into master at different points in time. For example db/fetch-pack has been in next for many weeks and hasn't yet merged into master, yet jc/am-quiet was forked after db/fetch-pack started and has already merged into master.

Your way would make jc/am-quiet wait until db/fetch-pack was ready. That's a big risk in the sense that your tree is "blocked" and even simple changes are held up by ones that suddenly became a lot more complex then you originally thought they were going to be.

On Tue, Oct 23, 2007 at 12:46:57AM -0400, Shawn O. Pearce wrote:

Yes, true. Alternatively, what I've been doing is that if I wasn't sure that a particular topic was ready to go to 'master' very shortly after it went into 'next', I would never let it go into 'next', but rather keep it in 'pu' (which is OK, because pu is constantly getting rewound). But I guess the downside of that is you might get fewer testers for the code, because fewer people are probably tracking and testing 'pu' as compared to 'next'.

Right?

					- Ted

Theodore Tso <tytso@mit.edu> wrote:

Yes, that's a good point.

I think in Git part of the reason less people track pu is because its very volatile. Not because of the rewind policy, but becuase sometimes the code there doesn't work properly so using it for real "production" work is pretty risky. On the other hand most of the code that merges into next has been reasonbly well reviewed and tested, so following it for "production" work is not as risky.

Junio has in the past proposed rewinding next, especially after a significant release (e.g. 1.5.3). A bunch of folks (myself included if I recall correctly) didn't want to do this, as we create topic branches locally from things in next and sometimes make commits over them to improve the topic further. But I also make topic branches for things in pu, so I might as well just shut up and not complain. :-)

Of course another thought that just came to mind is it is very easy for me to review next with a

	git log -p --reverse origin/next..build-next

just before merging it into my build branch and compiling it locally. If next rewound frequently (as pu does) this would be more difficult.

On Tue, Oct 23, 2007 at 01:07:26AM -0400, Shawn O. Pearce wrote:

Hmm, yes. I think I'd want to rewind next after a while; the thought of next drifting hundreds or thousands of commits away from master just gives me the heebee-jeebies. I'm sure it mostly works, but it just feels wrong. :-)

I guess I don't see why this would be a hardship; would a quick rebase on the topic branches more or less take care of the problem?

I guess that brings up another question; I've been regularly rebasing the topics branches as master and next advances... probably more out of superstition than anything else. Is that a bad idea for any reason?

Hmm... I guess some of this would be really good to get into the Howto section of the user guide when talking about git workflows!

	       	    	       	       	     	 - Ted

Theodore Tso <tytso@mit.edu> wrote:

There's been a couple of times in git history where Junio has basically done this to whack next back into line:

	git checkout next
	git diff next master | git apply --index
	git commit -m "Whack next back in line"

Because we've found a change or two lurking in there that shouldn't
have been there after a while.  I think it was related to a merge
conflict that happened in next but didn't in master or something
like that.  But usually this difference exists as there's usually
always something cooking in next.
 

Yes. But you need the prior value of the branch so you can do something easy like:

	git checkout yourtopic
	git rebase --onto $newtopic $oldtopic

which means you probably need to look through the logs for not just pu but also pu@{1}. A script to break out the topic branches from pu post fetch and store them as proper tracking branches would make this easier, but that much. If you plan ahead you can save that $oldtopic point so you can do something like this:

	git log pu ; # find $newtopic
	git checkout yourtopic
	git rebase --onto $newtopic base-yourtopic
	git tag -f base-yourtopic $newtopic

It keeps the history shorter in gitk. But otherwise it isn't bad. Unless you are running into a lot of conflicts every time you rebase and its wasting your time. ;-)

I prefer to rebase the topics until they've merged to an integration branch that doesn't rewind (e.g. master or next in git.git). That way they have the shortest line possible in gitk between the final merge and the start point.

There are good reasons why there's an "author" and a "committer"
field in commits.  Rebasing will change the committer field's
timestamp, but not the author field.  And author comes from the
email, to preserve the original date of development.
 

Yea, I think so too. We've adopted this model in git.git because it works for our community. A lot of other communities aren't too far away, as we have a lot of crossover in members. E.g. we learned a lot from the kernel community.

On Tue, Oct 23, 2007 at 01:42:38AM -0400, Shawn O. Pearce wrote:

Yeah, I had thought about writing a little script that would take my project's topic branches, and then push them out into the public repository under topics/ad/extents-testcases or topics/tt/badblocks-cleanup. That would make it easy to find the head of your topic, and once you find that, the base of your-topic isn't that hard to find, since it would just be the result of "git-rev-list topic ^master | tail -1".

One of the reasons I was thinking the above is because most of the patches are coming into my end as emailed patch series, and I end up tweaking them a lot as I carry them around in the topics branch. So if other people want to see what I've done to a branch after I've done a git rebase --interactive, it's easier if they can get access to the individual topics branch, so they can extract out the patch series while it's being tweaked by me (and possibly others).

This is probably because my view of git has been colored by kernel community practices, where patches are normally perfected and get rebased a lot (normally in a sub-maintainer or maintainer's tree) before they get pushed to Linus, and in my mental model a topic branch represents the maintainer's git tree in the central repository.

The extreme end of this would be the classical BitKeeper model, where Larry McVoy once argued to me that he didn't like history to *ever* be rewound/rewritten, since not only did this interfere with other people trees once they had been pushed, but it causes development history to be lost, which is always valuable. (Of course, in the end he did write "bk fold", which squashes the last N commits into 1, mainly due to customer pressure.) The kernel viewpoint is to rebase all the time, because the history is so huge that we don't *want* to see the development history of the rough drafts of features before they get merged into mainline.

It sounds like what you are saying here is that the git.git tree takes a viewpoint which is slightly between the extremes of the kernel model (which does involve resolving rebasing a lot and resolving lots of conflicts, but heh, that's not Linus's problem, that's been pushed out to the leafs of the developer community, and besides, it strongly encourages topics to get merged into the mainline fast), and the classical Bitkeeper model, which says that philosophical goodness means you should keep *all* development history once it enters the BK SCM.

With git.git, we are essentially throwing away development history while it is in 'pu', but once a commit graduates to 'next', we do keep the development history forever. The downside to this is that development 'crud' can build up in next; even if all substantive commits in 'next' end up graduating to 'master', there will still be lots of merge commits that will only be in 'next'.

I have an emotional bias which tends to treat that excess history as toxic waste to be avoided at all costs, but that's probably because when you have a git tree as huge as the kernel, life is easier if the history is kept as clean as possible.

Which I suppose is easy enough to do in the git.git model; if you throw away the 'next' branch and then rewind it so it is forked off of 'master' all of that history essentially gets flushed. The downside is that people maintaining topics branches which were forked against the old 'next' will need to do some grotty work to rebase their patches, so any attempt to rewind next would probably require the central maintainer to give plenty of notice, and then on the flag day, save 'next' as 'old-next' before rewinding to allow the other developers to more easily rebase any private branches they might have.

Hmm, interesting. A lot of this is quite subtle, or at least the impacts of different choices in the git workflow really didn't become obvious to me until I started trying I stepped into the central maintainer role for a project using git!

							- Ted

I keep thinking that there should be a better mechanism to use for "pu" than a branch. Normally what you see in "pu" is a sequential merge of "next" and a number of topic branches, where the series of merges is either entirely uninteresting or only interesting in a schematic sense (that is, it is interesting what topics appear, and in what order, but the snapshot of each topic's head when it got merged isn't interesting).

That is, the work which "pu" consists of, and therefore the history is a sequence of steps, each of which is one or more of: "add this topic", "update this topic", "remove this topic", "update to a new next". And we don't keep a record of this history, but it's not what's discarded by rewinding anyway.

I think that, if we actually care about this sort of thing, we'd want to make "pu" a series of commits, each with the previous "pu" as the sole parent, with a series object given in a new header. The series object would start with a "next" commit, and then list the topics merged by name and head-as-merged. Of course, the "pu" commits would contain the resulting tree as normal, so that people without a git that understands this would see "pu" as consisting of a straight line of commits, each of which simply shows the net effect of the changes. Or something like that. I suppose "pu" could also be represented as a superproject where each subproject is "next" or a topic branch, if we really want to avoid introducing new objects, but that seems unweildy somehow.

Is this worth doing? It might be; I bet it would make debugging -mm a whole lot nicer. (First bisect through -mm to find the action Andrew took that accepted the breakage, then bisect the history within that action.) I bet the status quo is a real pain when the feature that broke is only in -mm and later in Andrew's list than the tree whose change triggered the failure (i.e., -mm4 works; -mm5 doesn't work, and everything in -mm5 is either broken or untestable).

And, of course, "origin/pu[db/builtin-fetch]" would be an easy thing to build into git, and it could even generate extra magic, where it knows what the topic was last rebased on, so git could lead people through "rebase everything in a pu collection on the new base for the collection" and "make my local topic branch agree with my topic branch as present in origin/pu".

	-Daniel
*This .sig left intentionally blank*

Theodore Tso <tytso@mit.edu> writes:

You can view 'next' as if it is sort of -mm. Following 'master' is like following Linus tree, whose development is without those numerous 'merge improvements again' merges into 'next'.

An alternative is to give an easier access to the tips of individual topic branch head, at least to the ones that have been merged to 'next' as they will never be rewound once they are in 'next', and encourage people to fork off of them, instead of 'next' directly. Then 'next' will be more like 'pu'.

Very true.

On Tue, Oct 23, 2007 at 12:00:38PM -0700, Junio C Hamano wrote:

Actually -mm is much closer to 'pu', since it can and is rewound all the time. Patches can disappear if they are causing problems, they can be replaced and reworked, etc.

						- Ted

Theodore Tso <tytso@mit.edu> wrote:

Actually I found PU of some use:

	git branch -f pu next
	git checkout pu
	./Meta/PU --continue

its a little sluggish to list, but made it pretty easy to pick topics for merging into pu. git-rerere really makes it easy to recover conflicts in pu during future rebuilds of pu.

The other tidbits I managed to learn by trial and error here was to make sure I did the following:

	- make sure master is fully merged into next
	- make sure next is fully merged into pu
	- Run "Meta/git-topic.perl --base=master | less"

It wasn't uncommon for me to merge master->next, next->pu, run git-topic, then reset both next and pu *back* to what I had last published (remote tracking branches updated during push make this easy) before moving on with my next and pu updating activities.

Of course take my notes above with a grain of salt; I only worked this way for a week and it took me a couple of days to come up with the above. Junio may very well have it streamlined even more.

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Heh. In the past 24 hours I've really learned how much I appreciate the work that Junio does, and how infrequently I make it known that I'm happy he's doing it for us. Nothing like understanding what the guy goes through then to walk a day in his shoes. :-)

Yes.  Took me a few minutes to figure out which scripts did
what magic.  Junio likes two character script names, because uh,
they are short to type.  Then I hand modified the output to say
git/spearce.git and appear from me, not Junio.  But otherwise they
are the same scripts available from his Meta repository (aka the
todo branch).
 

OK.  I'll look at it tomorrow and consider moving it.  I recall the
context of this discussion now that you mention it, and I've been
running this in production use since Junio committed it to next
so I'm certainly not seeing any downsides to having this patch in
the tree.
 

Thanks for the summary on these two topics. They are going to stay parked in next for a while then. :-)