Kernel nightly snapshots..

I forget who it is that used to do the nightly snapshots for the BK kernels, but I _think_ it was David Woodhouse (every time I've blamed it on somebody, I've blamed the wrong person, so I'm probably off on this one too, but maybe I finally got it right).

I was wondering how to get that re-started.. It should be technically pretty easy, except I realized that my tree doesn't even have plain 2.6.11 in it. But I just fixed that in the tree, since I need such a baseline myself for my next release..

Anyway, I just pushed out a kernel tree that contains a tag of a _tree_ that points to the tree at the point of 2.6.11. I also had to teach fsck-cache about the fact that you can give it any kind of object to start your references at, and to make fsck-cache happy, you need to

	git-fsck-cache --unreachable HEAD v2.6.11

to tell it that the kernel tree now has an unconnected tree (described by the tag "v2.6.11-tree", and I made the appropriate entry for it in .git/refs/tags).

I also updated git-prune-script to not remove these kinds of things.

With this, it should be trivial to create snapshots with

	git-diff-tree -p v2.6.11 HEAD

or similar.

		Linus

It used to be Jeff Garzik, but David Woodhouse volunteered to take this over a few days ago, so yesterday I chowned those directories to him.

	-hpa

Linus Torvalds wrote:

On Wed, 2005-05-04 at 19:06 -0700, H. Peter Anvin wrote:

I still have a problem with tags....

hera /home/dwmw2/git/snapshot-2.6 $ cg-init /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &> ../asd hera /home/dwmw2/git/snapshot-2.6 $ cg-tag-ls v2.6.11 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c v2.6.11-tree 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c v2.6.12-rc2 9e734775f7c22d2f89943ad6c745571f1930105f v2.6.12-rc3 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a hera /home/dwmw2/git/snapshot-2.6 $ git-cat-file -t 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a .git/objects/03/97236d43e48e821cce5bbe6a80a1a56bb7cc3a: No such file or directory fatal: git-cat-file 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a: bad file

On Thu, 5 May 2005, David Woodhouse wrote:

Looks like cg uses git-http-pull instead of rsync, and doesn't download anything but the required objects.

In which case you probably don't have the v2.6.11 tree either, in fact, since it's not required to get a working copy of HEAD.

If you fetch the _whole_ object database (with rsync), you should get them.

		Linus

On Thu, 2005-05-05 at 07:44 -0700, Linus Torvalds wrote:

OK, I've changed my 'origin' to an rsync URL referring to the same place, to make sure I get tags correctly in future. 2.6.12-rc3-git1 is in the process of being built; if the attached script works and continues working when invoked from cron, we might even see nightly snapshots again as requested...

On Thu, 2005-05-05 at 16:10 +0100, David Woodhouse wrote:

Eep. That'll be a whole lot more useful if I do
	cg-diff -r $RELTREE:$CURTREE
instead of 
	cg-diff -r $CURTREE:$RELTREE
... but it should be OK now. This is the md5sum of the correct one:

21a0ee13c539e8b016682c1542bb9167  patch-2.6.12-rc3-git1.gz

On Thu, 5 May 2005, David Woodhouse wrote:

Btw, if you want to, you could simplify and optimize things heavily by doign some very specific git internal mucking around, and using the new "-m" flag I added to "git-diff-cache".

The "-m" flag says that a file that isn't checked out will always match the cache, which means that you can create your magic diffs that modify the Makefile with something like this:

	export SNAP_TAG_DIRECTORY=/pub/linux/kernel/v2.6/snapshots/tags
	export BASE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6
	export SHA1_FILE_DIRECTORY=$BASE_DIRECTORY/objects

	RELNAME=`ls $BASE_DIRECTORY/refs/tags | tail -1`
	RELCOMM=`cat $BASE_DIRECTORY/refs/tags/$RELNAME`
	CURCOMM=`cat $BASE_DIRECTORY/HEAD`

	SNAPNAME=`ls $SNAP_TAG_DIRECTORY | grep -- "$RELNAME-git" | tail -1`
	if [ "$SNAPNAME" ]; then
		SNAPCOMM=`cat $SNAP_TAG_DIRECTORY/$SNAPNAME`
		[ "$SNAPCOMM" == "$CURCOMM" ] && exit 0
		OLDGITNUM=`echo $SNAPNAME | sed s/^.*-git//`
		NEWGITNUM=`expr $OLDGITNUM + 1`
		CURNAME="$RELNAME-git$NEWGITNUM"
	else
		CURNAME="$RELNAME-git1"
	fi

	EXTRAVERSION=-`echo $CURNAME | cut -f2- -d-`

	rm -rf tmp-empty-tree
	mkdir -p tmp-empty-tree/.git
	( cd tmp-empty-tree
	  git-read-tree $CURCOMM
	  git-checkout-cache Makefile
	  perl -pi -e "s/EXTRAVERSION =.*/EXTRAVERSION = $EXTRAVERSION/" Makefile
	  git-diff-cache -m -p $RELCOMM ) | gzip -9 > patch-$CURNAME.gz

	echo CURCOMM > $SNAP_TAG_DIRECTORY/$CURNAME

but it's entirely untested, and only written in my mail editor, so..

Anyway, here's explaining what it does:

 - it creates a new "tmp-empty-tree" thing, and points the object database 
   to _my_ tree (which is unwritable for you, but you don't care, you just
   want to read it)

 - it picks up my last tag as the release, and your last git tag from your 
   private snapshot tag directory as the last git thing. Then it takes my 
   current HEAD as the new thing to diff.

 - we read that current tree into this empty tree, but we don't check any
   of it out except for the Makefile. So the "tmp-empty-tree" should 
   literally just have the files "Makefile" and ".git/index" in it.

 - we edit the Makefile, and then we do "git-diff-cache -m -p" between the 
   result and the old base.

This _should_ work. Modulo bugs. And it should be reasonably efficient. It just needs my current git tree as of five minutes ago..

		Linus

On Thu, 2005-05-05 at 09:40 -0700, Linus Torvalds wrote:

OK, this is in place now. I also made sure it was going to treat a real '2.6.12' release as newer than '2.6.12-rcX'. The selection of the tags and the output of the log are still chronological, which is of course wrong -- but it'll do for now.

As an added bonus I've also fixed the script to refrain from creating a -git1 snapshot even when there have been no commits since the last release. And deleted 2.6.12-rc4-git1.

David Woodhouse wrote:

Hi David,

Could you add that to 2.4 as well, too?

	-hpa

On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote:

Is there a 2.4 git tree?

David Woodhouse wrote:

I thought so, but now I can't find it.  Marcelo?

	-hpa

On Fri, May 06, 2005 at 02:50:17PM -0700, H. Peter Anvin wrote:

Not yet. I should be able to put a "beta" version next week.

David, can I bug you to add nightly snapshotting as soon as the v2.4 git tree is created ?

I hope there aren't many further changes in v2.4.x...

Hello,

do you have a timeframe? A friend of mine just asked me were to find the tree he wants to merge his jornada handheld pda stuff.

	Thomas

On Sun, 2005-05-22 at 13:50 +0200, Thomas Glanzmann wrote:

That should be done against the 2.6 kernel, and that tree is already available.

Hello,

it is a shame that 2.6 doesn't even boot on this machine[1]. And even if the patch doesn't do its way upstream its good to have something like git available to to vendortracking.

	Thomas

[1] http://bugzilla.kernel.org/show_bug.cgi?id=4415

On Sun, May 22, 2005 at 02:09:54PM +0200, Thomas Glanzmann wrote:

This bugzilla entry unfortunately gives me two problems:

1. I don't know what a 2.6.x-hh kernel is, or what modifications there
   are from the 2.6 mainline kernel.

2. I don't have visibility of this jornada list, nor what diagnosis
   has been done so far.

It's also worth pointing out that the patch there is for a 2.4 kernel, and therefore probably isn't suitable for 2.6 as is.

What I can say is that there's been very little flow of patches into mainline for Jornada 7xx - the last attempt was back in January 2002. Many of the problems from that time stemmed from me having a separate tree, which caused the expectation that I would handle integration of _anything_. Since the -rmk tree is no longer a public offering, and usable ARM support is merged into mainline, this expectation has been removed.

What this does mean, however, is that from my point of view, Jornada is a dead and unmaintained machine as far as 2.6 goes, and is ripe for removal from the kernel source - the only reason it builds today is because I've been tweaking it to keep it building. Whether it actually works or not is a completely different matter.

The way to stop it from being removed is for someone to step up and maintain the thing in mainline. How easy this would be is also something I have no view on, since I've no idea what's outstanding, nor what unmerged handhelds features Jornada 7xx depends on.

All in all, due to the above, I think submitting bugs for -hh kernels to the mainline bugzilla is a worthless exercise unless they can be proven to be relevant to mainline kernels.

Sorry.

On Thu, 5 May 2005, David Woodhouse wrote:

	RELNAME=`cg-tag-ls | grep -v 'git' | tail -n1 | cut -f1 | sed s/^v//`

this seems to assume that cg-tag-ls outputs things in the right order, which I'm not at all sure it does.

It looks like cg-tag-ls just does

	cd .git/refs/tags
	for tag in *; do
		...

which means that the ordering will depend on the shell ordering or globbing.

Bash does globbing alphabetically sorted (and documents that), and it's quite possible that all other shells do too. Even so, that doesn't actually mean that it would be sorted by release, since v2.6.2 will sort _after_ v2.6.12.

So this _should_ work for a while, but will eventually (before a -rc10 happens or we get to v2.6.100 ;^) need something better. That something better probably being to ignore any tags that don't point to commits, and then sorting by the date of the commit object.

		Linus

On Thu, 2005-05-05 at 08:46 -0700, Linus Torvalds wrote:

Enough with the dates already. I don't want to be doing snapshots against 2.6.11.9 just because that happened to get tagged more recently than 2.6.12-rc4. Time is an illusion; especially so when looking at branches which may not even have been merged.

The tag I want is the one which refers to a commit which is made by yourself and is accessible from the current HEAD in the lowest number of steps. And I don't _care_ if you tagged it 'Woozy Numbat'.