# Kernel nightly snapshots.. 18 messages from 2005-05-05 to 2005-05-22. Participants: Linus Torvalds, H. Peter Anvin, David Woodhouse, Marcelo Tosatti, Thomas Glanzmann, Russell King. Thread: https://gitlist.dev/t/482 ## Linus Torvalds, 2005-05-05 00:28 Subject: Kernel nightly snapshots.. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0505041639130.2328%40ppc970.osdl.org ``` I forget who it is that used to do the nightly snapshots for the BK kernels, but I _think_ it was David Woodhouse (every time I've blamed it on somebody, I've blamed the wrong person, so I'm probably off on this one too, but maybe I finally got it right). I was wondering how to get that re-started.. It should be technically pretty easy, except I realized that my tree doesn't even have plain 2.6.11 in it. But I just fixed that in the tree, since I need such a baseline myself for my next release.. Anyway, I just pushed out a kernel tree that contains a tag of a _tree_ that points to the tree at the point of 2.6.11. I also had to teach fsck-cache about the fact that you can give it any kind of object to start your references at, and to make fsck-cache happy, you need to git-fsck-cache --unreachable HEAD v2.6.11 to tell it that the kernel tree now has an unconnected tree (described by the tag "v2.6.11-tree", and I made the appropriate entry for it in .git/refs/tags). I also updated git-prune-script to not remove these kinds of things. With this, it should be trivial to create snapshots with git-diff-tree -p v2.6.11 HEAD or similar. Linus ``` ## H. Peter Anvin, 2005-05-05 02:06 Subject: Re: Kernel nightly snapshots.. Message-ID: <42797F9F.9030002@zytor.com> URL: https://gitlist.dev/e/42797F9F.9030002%40zytor.com In-Reply-To: ``` It used to be Jeff Garzik, but David Woodhouse volunteered to take this over a few days ago, so yesterday I chowned those directories to him. -hpa Linus Torvalds wrote: > I forget who it is that used to do the nightly snapshots for the BK > kernels, but I _think_ it was David Woodhouse (every time I've blamed it > on somebody, I've blamed the wrong person, so I'm probably off on this one > too, but maybe I finally got it right). > > I was wondering how to get that re-started.. It should be technically > pretty easy, except I realized that my tree doesn't even have plain 2.6.11 > in it. But I just fixed that in the tree, since I need such a baseline > myself for my next release.. > > Anyway, I just pushed out a kernel tree that contains a tag of a _tree_ > that points to the tree at the point of 2.6.11. I also had to teach > fsck-cache about the fact that you can give it any kind of object to start > your references at, and to make fsck-cache happy, you need to > > git-fsck-cache --unreachable HEAD v2.6.11 > > to tell it that the kernel tree now has an unconnected tree (described by > the tag "v2.6.11-tree", and I made the appropriate entry for it in > .git/refs/tags). > > I also updated git-prune-script to not remove these kinds of things. > > With this, it should be trivial to create snapshots with > > git-diff-tree -p v2.6.11 HEAD > > or similar. > > Linus ``` ## David Woodhouse, 2005-05-05 14:38 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115303933.16187.135.camel@hades.cambridge.redhat.com> URL: https://gitlist.dev/e/1115303933.16187.135.camel%40hades.cambridge.redhat.com In-Reply-To: <42797F9F.9030002@zytor.com> ``` On Wed, 2005-05-04 at 19:06 -0700, H. Peter Anvin wrote: > It used to be Jeff Garzik, but David Woodhouse volunteered to take > this over a few days ago, so yesterday I chowned those directories to > him. I still have a problem with tags.... hera /home/dwmw2/git/snapshot-2.6 $ cg-init /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &> ../asd hera /home/dwmw2/git/snapshot-2.6 $ cg-tag-ls v2.6.11 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c v2.6.11-tree 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c v2.6.12-rc2 9e734775f7c22d2f89943ad6c745571f1930105f v2.6.12-rc3 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a hera /home/dwmw2/git/snapshot-2.6 $ git-cat-file -t 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a .git/objects/03/97236d43e48e821cce5bbe6a80a1a56bb7cc3a: No such file or directory fatal: git-cat-file 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a: bad file -- dwmw2 ``` ## Linus Torvalds, 2005-05-05 14:44 Subject: Re: Kernel nightly snapshots.. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0505050742180.2328%40ppc970.osdl.org In-Reply-To: <1115303933.16187.135.camel@hades.cambridge.redhat.com> ``` On Thu, 5 May 2005, David Woodhouse wrote: > > hera /home/dwmw2/git/snapshot-2.6 $ cg-init /pub/scm/linux/kernel/git/torvalds/linux-2.6.git &> ../asd > hera /home/dwmw2/git/snapshot-2.6 $ cg-tag-ls > v2.6.11 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c > v2.6.11-tree 5dc01c595e6c6ec9ccda4f6f69c131c0dd945f8c > v2.6.12-rc2 9e734775f7c22d2f89943ad6c745571f1930105f > v2.6.12-rc3 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a > hera /home/dwmw2/git/snapshot-2.6 $ git-cat-file -t 0397236d43e48e821cce5bbe6a80a1a56bb7cc3a > .git/objects/03/97236d43e48e821cce5bbe6a80a1a56bb7cc3a: No such file or directory Looks like cg uses git-http-pull instead of rsync, and doesn't download anything but the required objects. In which case you probably don't have the v2.6.11 tree either, in fact, since it's not required to get a working copy of HEAD. If you fetch the _whole_ object database (with rsync), you should get them. Linus ``` ## David Woodhouse, 2005-05-05 15:10 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115305813.16187.143.camel@hades.cambridge.redhat.com> URL: https://gitlist.dev/e/1115305813.16187.143.camel%40hades.cambridge.redhat.com In-Reply-To: ``` On Thu, 2005-05-05 at 07:44 -0700, Linus Torvalds wrote: > If you fetch the _whole_ object database (with rsync), you should get > them. OK, I've changed my 'origin' to an rsync URL referring to the same place, to make sure I get tags correctly in future. 2.6.12-rc3-git1 is in the process of being built; if the attached script works and continues working when invoked from cron, we might even see nightly snapshots again as requested... -- dwmw2 ``` ## David Woodhouse, 2005-05-05 15:30 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115307033.16187.146.camel@hades.cambridge.redhat.com> URL: https://gitlist.dev/e/1115307033.16187.146.camel%40hades.cambridge.redhat.com In-Reply-To: <1115305813.16187.143.camel@hades.cambridge.redhat.com> ``` On Thu, 2005-05-05 at 16:10 +0100, David Woodhouse wrote: > 2.6.12-rc3-git1 is in the process of being built; if the attached > script works and continues working when invoked from cron, we might > even see nightly snapshots again as requested... Eep. That'll be a whole lot more useful if I do cg-diff -r $RELTREE:$CURTREE instead of cg-diff -r $CURTREE:$RELTREE ... but it should be OK now. This is the md5sum of the correct one: 21a0ee13c539e8b016682c1542bb9167 patch-2.6.12-rc3-git1.gz -- dwmw2 ``` ## Linus Torvalds, 2005-05-05 15:46 Subject: Re: Kernel nightly snapshots.. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0505050834550.2328%40ppc970.osdl.org In-Reply-To: <1115305813.16187.143.camel@hades.cambridge.redhat.com> ``` On Thu, 5 May 2005, David Woodhouse wrote: > > OK, I've changed my 'origin' to an rsync URL referring to the same > place, to make sure I get tags correctly in future. 2.6.12-rc3-git1 is > in the process of being built; if the attached script works and > continues working when invoked from cron, we might even see nightly > snapshots again as requested... RELNAME=`cg-tag-ls | grep -v 'git' | tail -n1 | cut -f1 | sed s/^v//` this seems to assume that cg-tag-ls outputs things in the right order, which I'm not at all sure it does. It looks like cg-tag-ls just does cd .git/refs/tags for tag in *; do ... which means that the ordering will depend on the shell ordering or globbing. Bash does globbing alphabetically sorted (and documents that), and it's quite possible that all other shells do too. Even so, that doesn't actually mean that it would be sorted by release, since v2.6.2 will sort _after_ v2.6.12. So this _should_ work for a while, but will eventually (before a -rc10 happens or we get to v2.6.100 ;^) need something better. That something better probably being to ignore any tags that don't point to commits, and then sorting by the date of the commit object. Linus ``` ## Linus Torvalds, 2005-05-05 16:40 Subject: Re: Kernel nightly snapshots.. Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.58.0505050846530.2328%40ppc970.osdl.org In-Reply-To: <1115307033.16187.146.camel@hades.cambridge.redhat.com> ``` On Thu, 5 May 2005, David Woodhouse wrote: > > Eep. That'll be a whole lot more useful if I do > cg-diff -r $RELTREE:$CURTREE > instead of > cg-diff -r $CURTREE:$RELTREE > ... but it should be OK now. This is the md5sum of the correct one: > > 21a0ee13c539e8b016682c1542bb9167 patch-2.6.12-rc3-git1.gz Btw, if you want to, you could simplify and optimize things heavily by doign some very specific git internal mucking around, and using the new "-m" flag I added to "git-diff-cache". The "-m" flag says that a file that isn't checked out will always match the cache, which means that you can create your magic diffs that modify the Makefile with something like this: export SNAP_TAG_DIRECTORY=/pub/linux/kernel/v2.6/snapshots/tags export BASE_DIRECTORY=/pub/scm/linux/kernel/git/torvalds/linux-2.6 export SHA1_FILE_DIRECTORY=$BASE_DIRECTORY/objects RELNAME=`ls $BASE_DIRECTORY/refs/tags | tail -1` RELCOMM=`cat $BASE_DIRECTORY/refs/tags/$RELNAME` CURCOMM=`cat $BASE_DIRECTORY/HEAD` SNAPNAME=`ls $SNAP_TAG_DIRECTORY | grep -- "$RELNAME-git" | tail -1` if [ "$SNAPNAME" ]; then SNAPCOMM=`cat $SNAP_TAG_DIRECTORY/$SNAPNAME` [ "$SNAPCOMM" == "$CURCOMM" ] && exit 0 OLDGITNUM=`echo $SNAPNAME | sed s/^.*-git//` NEWGITNUM=`expr $OLDGITNUM + 1` CURNAME="$RELNAME-git$NEWGITNUM" else CURNAME="$RELNAME-git1" fi EXTRAVERSION=-`echo $CURNAME | cut -f2- -d-` rm -rf tmp-empty-tree mkdir -p tmp-empty-tree/.git ( cd tmp-empty-tree git-read-tree $CURCOMM git-checkout-cache Makefile perl -pi -e "s/EXTRAVERSION =.*/EXTRAVERSION = $EXTRAVERSION/" Makefile git-diff-cache -m -p $RELCOMM ) | gzip -9 > patch-$CURNAME.gz echo CURCOMM > $SNAP_TAG_DIRECTORY/$CURNAME but it's entirely untested, and only written in my mail editor, so.. Anyway, here's explaining what it does: - it creates a new "tmp-empty-tree" thing, and points the object database to _my_ tree (which is unwritable for you, but you don't care, you just want to read it) - it picks up my last tag as the release, and your last git tag from your private snapshot tag directory as the last git thing. Then it takes my current HEAD as the new thing to diff. - we read that current tree into this empty tree, but we don't check any of it out except for the Makefile. So the "tmp-empty-tree" should literally just have the files "Makefile" and ".git/index" in it. - we edit the Makefile, and then we do "git-diff-cache -m -p" between the result and the old base. This _should_ work. Modulo bugs. And it should be reasonably efficient. It just needs my current git tree as of five minutes ago.. Linus ``` ## H. Peter Anvin, 2005-05-06 00:28 Subject: Re: Kernel nightly snapshots.. Message-ID: <427ABA45.3050803@zytor.com> URL: https://gitlist.dev/e/427ABA45.3050803%40zytor.com In-Reply-To: <1115307033.16187.146.camel@hades.cambridge.redhat.com> ``` David Woodhouse wrote: > On Thu, 2005-05-05 at 16:10 +0100, David Woodhouse wrote: > >>2.6.12-rc3-git1 is in the process of being built; if the attached >>script works and continues working when invoked from cron, we might >>even see nightly snapshots again as requested... > > Eep. That'll be a whole lot more useful if I do > cg-diff -r $RELTREE:$CURTREE > instead of > cg-diff -r $CURTREE:$RELTREE > ... but it should be OK now. This is the md5sum of the correct one: > > 21a0ee13c539e8b016682c1542bb9167 patch-2.6.12-rc3-git1.gz > Hi David, Could you add that to 2.4 as well, too? -hpa ``` ## David Woodhouse, 2005-05-06 07:29 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115364543.29495.24.camel@localhost.localdomain> URL: https://gitlist.dev/e/1115364543.29495.24.camel%40localhost.localdomain In-Reply-To: <427ABA45.3050803@zytor.com> ``` On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote: > Could you add that to 2.4 as well, too? Is there a 2.4 git tree? -- dwmw2 ``` ## David Woodhouse, 2005-05-06 07:42 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115365344.29495.37.camel@localhost.localdomain> URL: https://gitlist.dev/e/1115365344.29495.37.camel%40localhost.localdomain In-Reply-To: ``` On Thu, 2005-05-05 at 08:46 -0700, Linus Torvalds wrote: > So this _should_ work for a while, but will eventually (before a -rc10 > happens or we get to v2.6.100 ;^) need something better. That something > better probably being to ignore any tags that don't point to commits, and > then sorting by the date of the commit object. Enough with the dates already. I don't want to be doing snapshots against 2.6.11.9 just because that happened to get tagged more recently than 2.6.12-rc4. Time is an illusion; especially so when looking at branches which may not even have been merged. The tag I want is the one which refers to a commit which is made by yourself and is accessible from the current HEAD in the lowest number of steps. And I don't _care_ if you tagged it 'Woozy Numbat'. -- dwmw2 ``` ## H. Peter Anvin, 2005-05-06 21:50 Subject: Re: Kernel nightly snapshots.. Message-ID: <427BE699.60802@zytor.com> URL: https://gitlist.dev/e/427BE699.60802%40zytor.com In-Reply-To: <1115364543.29495.24.camel@localhost.localdomain> ``` David Woodhouse wrote: > On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote: > >>Could you add that to 2.4 as well, too? > > Is there a 2.4 git tree? > I thought so, but now I can't find it. Marcelo? -hpa ``` ## Marcelo Tosatti, 2005-05-06 21:51 Subject: Re: Kernel nightly snapshots.. Message-ID: <20050506215142.GA15357@logos.cnet> URL: https://gitlist.dev/e/20050506215142.GA15357%40logos.cnet In-Reply-To: <427BE699.60802@zytor.com> ``` On Fri, May 06, 2005 at 02:50:17PM -0700, H. Peter Anvin wrote: > David Woodhouse wrote: > >On Thu, 2005-05-05 at 17:28 -0700, H. Peter Anvin wrote: > > > >>Could you add that to 2.4 as well, too? > > > >Is there a 2.4 git tree? > > > > I thought so, but now I can't find it. Marcelo? Not yet. I should be able to put a "beta" version next week. David, can I bug you to add nightly snapshotting as soon as the v2.4 git tree is created ? I hope there aren't many further changes in v2.4.x... ``` ## David Woodhouse, 2005-05-07 12:39 Subject: Re: Kernel nightly snapshots.. Message-ID: <1115469570.29495.81.camel@localhost.localdomain> URL: https://gitlist.dev/e/1115469570.29495.81.camel%40localhost.localdomain In-Reply-To: ``` On Thu, 2005-05-05 at 09:40 -0700, Linus Torvalds wrote: > Btw, if you want to, you could simplify and optimize things heavily by > doign some very specific git internal mucking around, and using the new > "-m" flag I added to "git-diff-cache". OK, this is in place now. I also made sure it was going to treat a real '2.6.12' release as newer than '2.6.12-rcX'. The selection of the tags and the output of the log are still chronological, which is of course wrong -- but it'll do for now. As an added bonus I've also fixed the script to refrain from creating a -git1 snapshot even when there have been no commits since the last release. And deleted 2.6.12-rc4-git1. -- dwmw2 ``` ## Thomas Glanzmann, 2005-05-22 11:50 Subject: Re: Kernel nightly snapshots.. Message-ID: <20050522115038.GC15178@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20050522115038.GC15178%40cip.informatik.uni-erlangen.de In-Reply-To: <20050506215142.GA15357@logos.cnet> ``` Hello, > Not yet. I should be able to put a "beta" version > next week. do you have a timeframe? A friend of mine just asked me were to find the tree he wants to merge his jornada handheld pda stuff. Thomas ``` ## David Woodhouse, 2005-05-22 11:59 Subject: Re: Kernel nightly snapshots.. Message-ID: <1116763160.19183.15.camel@localhost.localdomain> URL: https://gitlist.dev/e/1116763160.19183.15.camel%40localhost.localdomain In-Reply-To: <20050522115038.GC15178@cip.informatik.uni-erlangen.de> ``` On Sun, 2005-05-22 at 13:50 +0200, Thomas Glanzmann wrote: > do you have a timeframe? A friend of mine just asked me were to find the > tree he wants to merge his jornada handheld pda stuff. That should be done against the 2.6 kernel, and that tree is already available. -- dwmw2 ``` ## Thomas Glanzmann, 2005-05-22 12:09 Subject: Re: Kernel nightly snapshots.. Message-ID: <20050522120954.GH15178@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20050522120954.GH15178%40cip.informatik.uni-erlangen.de In-Reply-To: <1116763160.19183.15.camel@localhost.localdomain> ``` Hello, > That should be done against the 2.6 kernel, and that tree is already > available. it is a shame that 2.6 doesn't even boot on this machine[1]. And even if the patch doesn't do its way upstream its good to have something like git available to to vendortracking. Thomas [1] http://bugzilla.kernel.org/show_bug.cgi?id=4415 ``` ## Russell King, 2005-05-22 17:01 Subject: Re: Kernel nightly snapshots.. Message-ID: <20050522180140.A21384@flint.arm.linux.org.uk> URL: https://gitlist.dev/e/20050522180140.A21384%40flint.arm.linux.org.uk In-Reply-To: <20050522120954.GH15178@cip.informatik.uni-erlangen.de> ``` On Sun, May 22, 2005 at 02:09:54PM +0200, Thomas Glanzmann wrote: > Hello, > > > That should be done against the 2.6 kernel, and that tree is already > > available. > > it is a shame that 2.6 doesn't even boot on this machine[1]. And even if > the patch doesn't do its way upstream its good to have something like > git available to to vendortracking. > > Thomas > > [1] http://bugzilla.kernel.org/show_bug.cgi?id=4415 This bugzilla entry unfortunately gives me two problems: 1. I don't know what a 2.6.x-hh kernel is, or what modifications there are from the 2.6 mainline kernel. 2. I don't have visibility of this jornada list, nor what diagnosis has been done so far. It's also worth pointing out that the patch there is for a 2.4 kernel, and therefore probably isn't suitable for 2.6 as is. What I can say is that there's been very little flow of patches into mainline for Jornada 7xx - the last attempt was back in January 2002. Many of the problems from that time stemmed from me having a separate tree, which caused the expectation that I would handle integration of _anything_. Since the -rmk tree is no longer a public offering, and usable ARM support is merged into mainline, this expectation has been removed. What this does mean, however, is that from my point of view, Jornada is a dead and unmaintained machine as far as 2.6 goes, and is ripe for removal from the kernel source - the only reason it builds today is because I've been tweaking it to keep it building. Whether it actually works or not is a completely different matter. The way to stop it from being removed is for someone to step up and maintain the thing in mainline. How easy this would be is also something I have no view on, since I've no idea what's outstanding, nor what unmerged handhelds features Jornada 7xx depends on. All in all, due to the above, I think submitting bugs for -hh kernels to the mainline bugzilla is a worthless exercise unless they can be proven to be relevant to mainline kernels. Sorry. -- Russell King ```