Git, merging, and News/Relnotes files

As a policy on a project that I manage, almost every commit warrants a change to our NEWS (changelog) file, which end-users can browse to get an in-depth idea of the changes that have happened from the last release. If it's an added feature, the changelog includes a description of how to use it; if it's a fixed bug, it briefly describes what happened. Internal changes may or may not get added, depending on the visibility of the APIs affected.

Something that I've noticed recently, as we've started migrating away from the ghetto SVN development model to the Git branchy model, is that this NEWS file ends up being the source of a lot of conflicts. Granted, they're easy conflicts to resolve, but still, they make a pull a little more complicated than it should be.

What would you guys, as experienced Git users, recommend in this case? Scrapping a NEWS file and simply drawing up the release-notes shortly before release (as the Git project does)? Aggregating the Git commit messages into one monster release log? Having the release manager add the NEWS entries himself, and mandate that no patch have it in them?

Thanks!

On Sat, 5 Jul 2008, Edward Z. Yang wrote:

I don't think anybody really _uses_ the functionality, but git does have the capability to specify special merge drivers based on filenames.

So you can

 (a) create a merge strategy that automaticaly does what you want. There's 
     a built-in driver called "union" that may or may not work for your 
     use case.

     See "Defining a custom merge driver" in "man gitattributes" for more 
     details about this.

 (b) Say which files you want to merge with this driver, by having 
     something like

	NEWS merge=news-merge

     in your .gitattributes file (or in ".git/info/attributes", if you 
     want to keep this all private to your own setup rather than in a 
     committed file that gets distributed to everybody else too).

and now your NEWS file will be merged using your special "news-merge" custom merge function.

Of course, the custom merge driver is only done for non-trivial merges. Git will do all the trivial fast-forward merges on its own, and only call the custom merge driver for things that have actual possible data conflicts (ie changes in both branches).

NOTE! Keeping an ordered list (like a ChangeLog or a NEWS file) is fundamentally not an easy thing to do in a distributed environment. The "union" merge strategy may well work for you (and if it does, this is all going to be very easy), but it's also entirely possible that you will find that the ordering in a distributed environment is so unspecified, you'll prefer to do the merges by hand _anyway_ in the end.

So the first thing you should do is probably to just *try* adding that

	NEWS merge=union

line to your .gitattributes file, and see if it works for you. My personal guess is that you'll realize that you really prefer doing the trivial merges manually after all, but hey, maybe not. And as mentioned, you *can* create your very own merge strategy that knows about the particular rules of the files in question, but that gets more complex.

For example, the default 'union' merge will literally _duplicate_ identical that were added in both branches. So if you cherry-pick a commit so that it exists both in the branch you are merging _and_ the branch you are merging into, then any additions to the NEWS file will basically show up twice, and yet auto-merge "cleanly".

Which is very understandable, but is almost certainly not what you want.

			Linus

Linus Torvalds wrote:

Sounds like a good first-step. It's very unlikely that we're going to bother writing our own merge strategy for the NEWS file, so if union ends up being more trouble than its worth, we'll probably end sticking with manual merges.

Pieter also suggested (for some reason, I don't see the post on this list) the git-merge-changelog driver from Gnu Savannah. Unfortunately, the log format is a little different from ours (entries are sorted into BC-incompatible, features, bugfixes and internal changes), so the driver may not work (it's still worth a try, I imagine).

I'm slightly surprised no one suggested that I can the file, given that both Git and the Linux kernel don't have one.

I suppose that's why we have git reset --hard HEAD~. :-) I will certainly keep this gotcha in mind.

On Sat, 5 Jul 2008, Edward Z. Yang wrote:

Well, I personally think ChangeLog files are a total waste of time. You're much better off autogenerating those from the real logs, I think.

[ Although in all honesty, I also think we could improve on our reporting 
  tools, and have ways to perhaps highlight big or important changes some 
  way ]

But a NEWS file that actually talks about new features is a different thing. It can make lots of sense to maintain something like that, so I wouldn't suggest canning it if it works for you. I'm not convinced it would work for the kernel, but I suspect it can work really well for other projects.

Well, the real problem with a clean automatic merge is not that it can't be undone (or better yet - fixed: just edit the NEWS file and then do a "git commit --amend NEWS" to fix up the atomatic merge), but the fact that most of the time you'll simply never even notice.

IOW, when something merges cleanly (and the 'union' merge will basically always do so), the most common case is probably that people won't even _look_ at the end result - especially if it works fine most of the time.

That's why a trivial conflict can often be better than a silently clean merge: at least it forces people to spend a small amount of brainpower to look at the obvious fix.

But hey, give it a try. Maybe you'll like the union merge, together with occasional manual fixups. Or maybe you'll decide that a specialized merge strategy isn't that painful after all (or can find somebody who already went through the pain and wrote one you can use).

		Linus

On Sat, Jul 5, 2008 at 11:24 AM, Edward Z. Yang <edwardzyang@thewritingpot.com> wrote:

I believe it is better to put all this information directly to the commit message using some special tagging, so you can extract it automatically at the release time and generate the changelog file for users. You may edit the generated changelog and commit it directly before release.

Having one file changed on almost every commit is not a good idea, and not only because it will cause unnecessary conflicts but also it may considerable increase the size of the whole repository. By default, the delta compression has limit 50, which means that every 50 change of file will become its full copy. If the changelog file is changed very often and it is long, it may turn out that changelog alone takes as much space as the rest of the source tree.

Dmitry

Dmitry Potapov wrote:

That is certainly a good technical point, and I will certainly look into building a log parser after we wrap up our next release cycle.

P.S. Linus, we ended up manually merging the NEWS file; in some cases there were branch specific changes in the file which would have been completely inappropriate with a union merge. Thank you for the suggestion, however.