[RFC] git integrated bugtracking

  Hi, I'm currently trying to think about a bug tracking system that
would be tightly integrated with git, meaning that it would have to be
somehow decentralized.

  Though there is a few design issues I have, that block me from doing
first decisions about how to implement some kind of Proof of Concept. My
main problem is: should I put this bug tracking system in the repository
it tracks bugs for, or not.

  I mean, the immediate idea is to have some .bugs/ directories (or
alike). This has many good properties, e.g. for projects like the linux
kernel with its many subsystems or driver, it would make sense to have
per driver/subsystems/... bug packs, and move bugs from one pack to
another would be the way of assigning bugs to different modules.

  Also, a good thing is that when you "report" a bug, it gets commited
in the repository, and taints all the commit chilren, until you commit
the closing patch. This allow a release manager to rapidly _see_ if his
stable branch has this or this bug.

  OTOH it comes with many problems. First, and most obvious IMHO, it's
that it'll mean bugs will have to be pulled into the mainlines. Let's
take example with the linux repository, I'm not sure Linus would be
really keen on doing rounds of bugs pulls, not to mention it'll bloat
the repository somehow.

  The other problem I see is that at the time a bug gets reported, the
user knows it's found at a commit say 'X'. But it could in fact have
been generated at a commit Y, with this pattern:

  --o---o---Y---o---o---o---o---X---o---o--> master
                     \
                      o---o---o---o---o---o--> branch B

  Sadly, the bug report has been commited at 'X', hence it does taints
branch B. As "inserting" or "moving" 'X' commit before the branch is not
an option as it would rewrite history, that becomes also a major no-go
for in-tree collecting of bugs.

  Last of all, I'd also like to have a design where bugs pulls do not
create too much painful merges and conflicts. If we e.g. say that a bug
is a status file and a mbox of comments (makes sense to me), the mbox
can be merged easily (concatenate all mboxes, and purge duplicates), but
the status file is quite problematic on its own too.

  So here are the first ideas/problems/remarks I have with that. I'd
be thrilled to have your comments about those points.

PS: What I left over, is why I wanted such a tool. Programmers tends (or
    say I tend to, maybe I'm over-generalizing, but I seem to remember a
    thread on the lkml where Linus was basically saying the same) to
    hate bugzillas and such out-of-tree tool because they suck, and do
    not really fit in the programming cycle. I'd rather see a
    bugtracking system where the backend is in-tree, basically mboxes so
    that you can read them easily with your favourite MUA, as well as
    adding new comments in it the same way. It also accommodates with
    linux-like workflows where bugs usually are sent on the lkml, a bit
    like patches and pull requests are handled. That's the reasons why I
    came with this idea.

On Sun, Jun 03, 2007 at 01:48:44PM +0200, Pierre Habouzit wrote:

You may want to look at existing solutions, such as Bugs Everywhere[1], which has is modular wrt the backend SCM (supports Arch and Bazaar for now).

I'm not sure its storage format is extensible enough, but it at least shows some interesting ideas.

What about a requirement that .bugs/ is at the project toplevel ? We could then enforce that such an "assign to submodule" operation only assigns a bug to a real git submodule.

Maybe "commit annotations" would help here ? I have noticed a thread about a git-note tool, though did not open it yet - so I may be off-track here.

I still have mixed feelings towards the idea of implementing a brand new defect-tracking system, when there are already so many of them, with many features already. Eg., my initial opinion about Bugs Everywhere was that it was not complete enough to be used in many projects.

I tend to think it would be more productive to do any necessary changes in widely-used bug trackers (bugzilla, mantis, rt...), and work on glue tools, like scmbug[2].

[1] http://www.panoramicfeedback.com/opensource/ [2] http://www.mkgnu.net/?q=scmbug

(both projects listed in the wiki already - http://git.or.cz/gitwiki/InterfacesFrontendsAndToolsWishlist)

best regards, -- Yann.

On Sun, Jun 03, 2007 at 02:35:10PM +0200, Yann Dirson wrote:

  I looked at it, the idea is good, implementation quite wrong (too many
files, with too much crancky formats). But it's quite small, so if it's
the good way, it should be easy to adapt.

  I don't see why it's necessary, it's just random thoughts and quite
outside the scope of my concerns right now.

  Hmm, I'll try to search for it then.

  Well, why writing yet-another-scm like err git ? Because other were
doing things wrong. Bugzilla, or especially rt are not distributed,
hence completely inadequate for use with git, whatever clumsy plumbing
you do to make them work. And I know what I'm talking about, I wrote the
plumbing for the debian bug tracking system, I've looked at a lot of
them. And only bugs everywhere was near what looks the good way, because
when you're able to deal with your code like git allows you to, it makes
absolutely no sense not being able to deal with your bug database the
same way.

  And FWIW I'm just asking questions, nothing more, just because I don't
want to rush in yet-another-tool if it appears it will be broken.

Pierre Habouzit writes:

Mainly for that reason, I would suggest having it outside the code base's namespace: probably a different root in the same $GIT_DIR, but I can see people wanting to have a separate $GIT_DIR. If the database tracks bugs by what commit(s) introduce or expose the bug -- at least once that is known -- then you get nearly free tracking of which branches have the bug without having to check out largely redundant trees.

Michael Poole

On Sun, Jun 03, 2007 at 08:59:18AM -0400, Michael Poole wrote:

  Sure, but if it's completely out-of-tree, then cloning a repository
don't allow you to get the bug databases with it for free. I mean it'd
be great to have it somehow linked to the repository, but also I agree
that not everybody wants to clone the whole bugs databases. So maybe it
should just be in another shadow branch that annotates the devel ones.
Hmmm I definitely need to read the git-note thread...

On Sunday 03 June 2007, Pierre Habouzit wrote:

I guess I'm the one responsible for starting that git-note thread...

For the moment, I'm busy implementing some concepts that came out of that discussion (refactoring tag objects and building some infrastructure needed to support notes without the drawbacks present in my first version).

Hopefully I'll have a proof-of-concept ready before too long. In the meantime I'll be happy to answer questions you might have.

Regarding the notes themselves, I thought about possibly using them as a link between the repo and the bug tracker, with some glue code in between for making the connections. I haven't thought about integrating them more deeply into a bug tracker, but it might be worth thinking along those lines, especially for the kind of system you're proposing.

Now, back to hacking...

Have fun! :)

...Johan

On Sun, Jun 03, 2007 at 03:48:29PM +0200, Johan Herland wrote:

  Yeah, now that I read that thread, well yeah, I think notes are a hell
of a good concept for my ideas. I mean, a bug report would be basically
a collection of notes:
  * the bug has been found at this commit ;
  * the bug has been not-found at this commit ;
  * this commit is a fix for that bug ;
  * this commit enhance features for that wish and so on.
Some other bits are more followup comments and are disconnected to
commits, but could be attached to the "bug object" whatever would be
used for bugs, the whole concept is blurry anyways, we're just
discussing ideas :)

  Though, for a good bug tracking system, you need to be able to answer
some kind of questions fast enough:
  * list bugs that affect a given stage of the repository
    (tag/branch/commit/...) ;
  * be able to trace history of a given bug (yes, it's fairly obvious,
    but unlike many other bug tracking systems, for us it would not be
    contiguous, so it's not necessarily a O(1) operation) ;

  Other things nice to have is textual search, git-grep can help of
course, but that's an operation you do often with a bug tracking system,
and you expect it to be faster than git-grep is (though maybe an
optionnal index can be built around the notes for that purpose and not
be versionned ?).

  Another think that notes do not address are another operation we
usually do on bugs: merge (or duplicates). There is a think I hate in
bugzilla and love in debbugs, it's that duplicate bugs are closed in the
former, and merged in the latter. When two bugs are the same, their
history are often *both* valuable, and you really don't want to lose one
half, you want to merge them. And you also want the option to "unmerge"
them, but for that the better option is to have the ability to duplicate
a bug (aka debbugs cloning).

  Anyways it's just gossip, but maybe someone will have a brilliant
ideas, so I'm just throwing my thoughts into this mail :)

Pierre Habouzit <madcoder@debian.org> writes:

That's my feeling too. "Commiting" bug information in the tree is only half of a good idea. You want to be able to say, after the fact, "This commit had bug XYZ". OTOH, the idea (followed by bugs everywhere) that merging a branch would automatically close bugs fixed by this branch is a really cool thing.

The kind of information you're mentionning above can be a great starting point for "bisect". I can even imagine a kind of distributed bisect, where several users could give their "bad commits" for the same bug.

On Sun, Jun 03, 2007 at 05:44:58PM +0200, Matthieu Moy wrote:

  That would work with notes, as while merging you'll get the notes of
the commit in your branch, *and* the note about the fixing patch. So
there is no loss of "concept" here. In fact that was the thing that I
looked for. Notes are good. They just may not be enough to write an
in-git bugtracking tool, as a bug needs the "notes collection" concepts,
and maybe a few other.

  Heh yes. Sometimes it's more complex than that as bugs can come back
(regressions) or be the result of many commits (like the cases that suck
with bisect). For a complex bug it's more a set of [found..notfound[
intervals. Though this indeed is very valuable information, and the
distributed component thing *is* great.

On Sun, 3 Jun 2007, Pierre Habouzit wrote:

how would you identify bugs in such a way that they will match up when you merge different trees?

if you can manage to do this it sounds like a great idea. but I'm not seeing a good way to do it at the moment. the answer may be a combination of a number of factors.

1. bug number doesn't work well in a distributed environment

2. something based on indentifying the cause of the bug (commit id + file 
+ line????) will only work after you know the real cause of the bug

3. description is worthless, too many ways to describe things that have 
the same underlying cause

David Lang

On Sun, Jun 03, 2007 at 10:35:48AM -0700, david@lang.hm wrote:

  well, because they will be sha1's, a git object. And when it's a
duplicate, well, let's face it, not bugtracker helps *automatically*
tracking duplicates. The merge work is up to the QA people. Yeah,
bugtracker won't do all the tracking. In a way, that's good, that means
there is still place for humans in that world :)

  Sure, SCM revisions either. But git solved that once, I don't see why
the solution found at that time would be less of a solution now :)

  It does not work in a real world, where real user don't grok code.

  Sure, though it could help finding dupes. BUt in my experience what
helps most, is fine grained categorizing, because you end up with few
bugs for a given component, and "same" bugs end up actually being near
in the UI or query tool. Of course it let space for bugs that get
actually miscategorized. But hell, my experience is also that many bugs
are discovered as beeing fixed years after the fix anyway.

  I don't plan fixing all that stuff, it can't really be. I just would
like to create a tool that isn't as painful for the programmer as
bugzilla (or rt or ...) is, while it would still be as pleasant and easy
to stick a web UI for the users over it, hence not making the user
experience less pleasant.

  My experience with bugtracking is that the most efficient way to deal
it is to let the programmer in charge of the responsible module deal
with those bugs. What programmer aren't willing to do is the triaging,
and pulling the bugs off a distant database/UI/.. off something that
isn't in their usual workflow.

On Sun, 3 Jun 2007, Pierre Habouzit wrote:

git's tracking of revisions works becouse it's tracking the content, it doesn't care _how_ the content got that way, if it's the same it's the same and will have the same hash.

with bugs the reports aren't the same so you can use the sha1 to track a particular entry/tag/comment but not to identify the bug itself

fine grained categorization is something that takes place long after the bug is reported, users don't know how to correctly categorize bugs any more then they know what code caused the bug.

this only works if someone goes to the work to send the bugs to the right programmer. in many cases this is non-trivial.

David Lang

On Sun, Jun 03, 2007 at 12:07:44PM -0700, david@lang.hm wrote:

Do you see any problem with taking as bug id the hash of the first entry submitted ? Just like in a classical BTS, the bug id gets allocated at submussion time. Further informations then get associated with the initial report - here that would problably translate into the initial report being an annotation to a source commit, and further information being annotations to the initial annotation.

Is that a problem ?

This is he work of a QA team, and although it is not as common in the Free Software world as it should be, there are people willing to do the job. This discussion is about giving these people a better tool to help them focus on the tasks where a human being is really needed.

Best regards,

On Sun, Jun 03, 2007 at 05:19:21PM +0200, Pierre Habouzit wrote:

I can't wait for visualising the history of bugreports in gitk, along with duplicates/forks and merges :)

/me wishes all gossip would be as productive as this one :)

Best regards,

One thing that I just thought about is how such a system would play with eg. StGIT.

Precisely, if I stick a BTS note (found/not-found probably makes no sense here, so that'd mostly be fixes/enhances notes) to a given version of an StGIT patch, I probably want the BTS to be able to tell that the next version of the patch "probably has" the same note - if a given patch fixes a given bug, I infer the patch is primarily written as a fix for that bug, so later revisions of the patch (eg. cleanups) would also fix it. But maybe we'd want the patch author to explicitely make such an assertion, in case a cleanup would have broken the fix - both options could be useful in different cases.

Currently, old versions of an StGIT patch are only available through StGIT patchlogs, which are not ancestors of later versions of the same patch. That is, even the "subsequent versions inherit notes" policy would need extra StGIT-aware stuff to work.

This may be something to keep in mind here - but that issue could also be seen as belonging to the more general (and with ongoing work) aspect of git/StGIT interactions. Even better if the 2 ongoing reflexions cross-fertilisate :)

Best regards,

On Sun, Jun 03, 2007 at 10:04:31PM +0200, Yann Dirson wrote:

  I read the rest of the mail, but I don't know stgit at all, so I'm not
really able to tell how well it would integrate, especially since well,
it's still early.

  But for sure, I'm launching that brainstorming thread because I want
as much input as possible to know what are people needs, and add the one
that make sense to the todo list, the one that would be good to the
wishlist. FWIW playing well with usual git tools (like stgit, guilt, and
whatever other evolved "thing" in the git nebula) is definitely in the
TODO list.

Another aspect that I have not seen discussed yet is the handling of testcases that expose a bug, and which are usually integrated as non-reg tests afterwards.

If we're going the way of using git-note's, we can easily link the commit which adds the testcase to the non-reg testsuite.

But we could be interested in running the testcase against an arbitrary commit, thus allowing some sort of automated bisect (modulo applicability of the testcase to other commits, which may cause false negatives, but also false positives, and thus may need to be hand-driven or at least hand-reviewed).

For this, however, would it be sufficient to have the testcase committed somewhere in the history tree ? Our hypothetic automatic bisect tool could cherry-pick that commit and run the testcase, but at would place an arbitrary restriction of "one testcase per commit", which would probably a bad idea.

And having the annotation point to the blob does not seem a good idea either, at least because currently in git tools we have test scripts that excercise several testcases each.

Even an annotation to both a commit and a blob changed by that commit would not be sufficiently accurate, when several testcases are added to the same script in a single commit.

Another options would be to embed the testcase in an annotation: I can't really see that being used, since that would duplicate the testcase. That makes me wonder whether it would not be required to have testcases described in seperate files ?

Best regards,

On Tue, Jun 05, 2007 at 12:03:08AM +0200, Yann Dirson wrote:

  Well IMHO testcases are to be dealt with from the build sytem, and
versionned in the SCM. You may want to have some kind of helper doing
the bisect automatically for you (though, speaking of experience, it's
not always _that_ simple because of broken commits that do not build
e.g.) But I hardly see how it can be realted to a BTS anyways. I mean,
bugs will likely have test cases to exhibit the bug, but those should be
put in the test-case part of the repository. I mean well, maybe there is
fancy things to do, but I don't think it should be the role of a BTS at
all.

On Sun, 3 Jun 2007, Pierre Habouzit wrote:

Make it a separate (and independent) branch of the repository you track, and then you can do it - or not do it - as you want later.

See as an example the git "todo" branch, ie you can always look at what Junio may or may not be planning with a

	git show remotes/origin/todo:TODO

which just shows the "TODO" file in the "remotes/origin/todo" branch.

The same approach could be done for bug tracking: you *could* check out the bug-tracking branch separately (and you can create a repository that has _only_ that bug tracking branch, or _only_ the actual development branch in it), but you could also access it without even checking it out at all, and mix the two projects in the same repository.

I would suggest _not_ doing this kind of mixing. I think it might be appropriate for some cases, but I don't think it's appropriate in general. Partly because I don't think the people who change the bugs are at all necessarily at all the same people who actually do development.

IOW, bug-reporters obviously have to have write access to *some* bug-tracking thing, and I don't think you want to try to merge the bug-tracking together with the real development.

		Linus

On Sun, Jun 03, 2007 at 12:22:20PM -0700, Linus Torvalds wrote:

  Well I went that way, but we loose the quite cool "if I branch my
repository I branch the bugs coming with them too"-feature. And I'd be
sad to give that up. But maybe it's an error to want to use git to
encode that relation.

  [10 minutes of pondering later]

  Well yes, I think it's basically an error to encode the bug
dependencies in git branches and so on, because bug tracking is often
done asynchronously from devel. So you'll have to assign bug to parts of
the history that are already cold from a devel point of view.

  My conclusion is indeed that a separate branch is really the way to
go. That does not tells how I'll be able to answer the usual "what are
the current opened bugs in my branch" question fast, but well, one step
at a time.

  I agree, but I don't see how it's very different from you not trusting
the average kernel contributor and wanting your lieutenants to ack
patches first. My point is, the user-exposed part doesn't *has* to be
pushed in the mainline repository, it could be in ... the bugs
repository, from which developpers could pull some bug reports that are
worth. Or you can also have more layers if you have a QA team in charge
of the bugs. Possibilities are endless, and well, I don't pretend to be
the one that will teach you that :)

  Though I don't like 100% the idea of a .bugs repository. The reason
why I came up with that, is that (experience talking), as a
developper/maintainer/...whatever I have to get in touch with the
reporter. Most convenient way : mail. So I want to be able to use
`$MUA -f /path/to/bug/mbox` to answer, because this is the sole
operation that makes sense. Though I suppose it could be easy to make a
thin wrapper that does a local checkout calls $MUA, and commits the new
mail. So I'd easily give up on that, as I acknowledge I've no real
reasons to design it that way.

hoi :)

On Sun, Jun 03, 2007 at 10:16:32PM +0200, Pierre Habouzit wrote:

Just store the commit which introduced the bug (or where the bug was first found) and you will get that, too. You only have to check if this commit is reachable by a given branch to see if it is affected. When you fix the bug you store the commit id that fixed it and then you can check every branch if it points into bad..good.

You can also do this for released versions. If you have the bug database inside the repository you can't report any bugs for a released version, because it is, well already released.

Martin Waitz wrote:

Ok, that seems reasonable.

Now, how do you store updates to a bug? i.e. followup questions and answers?

As suggested in another message, I think that the bug id would have to be the hash of the initial report (incl whatever metadata is considered interesting).

I think that the individual reports might be stored by their id, perhaps via a fan out dir, like the .git/objects directory. Attachments can be stored by their hash in a separate directory structure. Modifications to a report would simply be appended to the original report. Simultaneous modifications could be dealt with by merging the two files, probably using a custom merge driver.

Obviously the format of a bug report and follow up message would need to be quite strictly defined and adhered to, for the merge driver to be able to do its work with the minimum/zero user interaction.

Closed bugs would be deleted from the filesystem, but would obviously be available via the history.

Indexes or categories could be implemented by means of symlinks/symrefs in a different set of "index directories". e.g.

/categories/drivers/deadbeef -> ../../bugs/de/adbeef /assignedto/joe@example.org/deadbeef -> ../../bugs/de/adbeef

or similar. These might not be strictly necessary, since all that information will be in the report anyway. Perhaps the indexes would be stored simply as cached data, and rebuilt if out of date.

The porcelain would then take care of presenting the bugs to the user in the preferred format, much like we have gitk, gitweb, qgit, tig, etc.

Does any of this make sense?

Rogan

On Sun, Jun 03, 2007 at 12:22:20PM -0700, Linus Torvalds wrote:

And since we have cheep branches, we can even have one BTS branch for each project branch.

But then, since we probably want git-merge to merge the BTS branch when present in the local repo, that would mean adding some support in the core porcelain - eg. a config option declaring some coupling between a branch and its bug-tracking pal, so git-merge can be taught to merge it too.

The same functionality could probably be obtained through more annotations (ie. record which subsystem the bug relates to, just like any other property of the bug).

Best regards,

On Sun, Jun 03, 2007 at 10:17:23PM +0200, Yann Dirson wrote:

  In fact, I don't think so. In my answer to Linus, I was almost writing
the same thing as you. In fact, no, we don't want a BTS branch per
project branch. We just want to be able to list commits where we found
the bug, and commits where we didn't found them. Lets call them good and
bad commits (yeah, ring a bell ;p). One of the "good" commit will be
better than any other good one, as it'll be the "fixing" commit. Or one
of the fixing commits, as you may have different ones if you just
cherry-pick just the patch in a stable branch.

  So what you need is a way to link a bug to git objects, namely
commits. Linking to files or subtrees also makes sense. In fact we need
to link bugs to well git objects (what a shock :D).

  And if you want to know what affects a given branch, well, you need to
list bugs whose affected objects are present in that given branch. Even
if you have one branch per project branch, as the bug branch chronology
will not be the same as the project's one, you'll still need to answer
the previous question the very same way. And I don't think it's be more
complicated if all project's branches (or may project's branches) are
dealt with or not.

  Well, annotations are certainly useful to create a [git object] ->
[bug #id] map. That way, listing bugs in a branch is "just" a matter of
listing still open bugs in the ancestry graph. Sadly, if I'm not
mistaken, this is at least a linear operation wrt the number of objects
in the branch (supposing that access to objects is O(1)).  IMHO that's
not good enough, but I'm not really in the implementation stage yet.

  FWIW I've begun to work on this (for real). I've called the tool
"grit". You can follow the developpement on:

  * gitweb: http://git.madism.org/?p=grit.git;a=summary
  * git:    git://git.madism.org/grit.git/

Cheers,

Pierre Habouzit wrote:

I have added info about it at the bottom of
  http://git.or.cz/gitwiki/InterfacesFrontendsAndTools

Feel free to correct and extend info.

On Sat, 9 Jun 2007, Pierre Habouzit wrote:

I've been working on a completely orthogonal portion of the bug-tracking problem (software to try to generate bug reports containing the information that a particular project would like for a particular sort of bug), and I've been thinking about how it would fit in with git-the-content-addressed-filesystem. I haven't actually implemented anything at all in this area, some you're ahead of me, but I have some ideas.

1) It's probably best to use some new types, rather than trees and 
commits. This gives you more flexibility to structure things in ways that 
exactly fit what's going on, which is one of the main reasons git is so 
good for version control. It also means that you can use inline strings 
for short answers (what's the name of the program that makes your kernel 
oops), and blobs for long answers (what's your lspci -vvv), and commits 
when appropriate (what commit did git-bisect find? what version resolves 
it?)

2) It's probably best to have the history be per-bug, with each revision 
being an update to that report, and have the complete database be a refs/ 
subdirectory. The complete list of bugs is going to become vast, and it's 
probably best to retain old bugs, at least in the databases at archival 
sites, so that you can get information on how often a bug turns out to be 
misuse of a particular API or something. This means that you really don't 
want each addition to be O(number of bugs).

3) I think it's worth separately representing "what problem somebody had" 
and "what was wrong with the program to cause problems" and linking these 
to each other. This will help in being able to at least represent multiple 
reports of the same bug, which is useful for finding patterns when the bug 
is non-trivial.

My idea of what the structure of the data is:

 - The project has essentially a troubleshooting procedure, written up 
   like a classic expert system (or like Kconfig). This takes the user 
   through a set of all the questions developers ever ask, with only the 
   appropriate ones visible (if you've got a build failure, it doesn't ask 
   for lspci output; it only asks for sysrq-t output if the system is 
   still sort of responding; etc).

 - The failure report is the set of all the questions the user answered 
   and the answers.

 - The hash of the initial failure report is the ID for the failure. 
   Revisions of the report (adding more information as people ask for 
   special things) retain the same ID.

 - After you generate a failure report, it searches for similar failures 
   and bugs in the main database. If it finds stuff, the user can try any 
   resolutions, and skip sending the report in. If there's nothing there 
   or there's still uncertainty as to what to do about the issue or the 
   resolution doesn't work for this case, the report is added to the 
   database.

 - It's up to developers to create bug records to pull together failure 
   reports, analysis of the situations, advice, and the ultimate 
   resolution. Failures get revised to link them to bugs so that people 
   with problems can find answers, and bugs list the failures they cause, 
   so that people working on something can find people to test.

 - Reports can be made on (1) failures that somebody would be able to test 
   resolutions to, but which aren't attached to any bugs; (2) bugs that 
   aren't resolved in a particular commit (which may be resolved in some 
   later or parallel commit, or have a patch). These are the things that a 
   release engineer would want to check on before releasing.

 - Reports can be made on clusters of unattached failures with similar 
   features. This would include unreproduceable failures, because they 
   might become fixable based on a large number of reports, or a test case 
   could be generated that makes them easier to trigger based on 
   distilling the common features of the rare failures.

	-Daniel
*This .sig left intentionally blank*

Hi,

On Sat, 9 Jun 2007, Daniel Barkalow wrote:

I fail to see why this has to be a new type. The flexibility in Git lies IMHO therein that it does _not_ have a plethora of objects. Rather, there are just 4 object types, which serve their purpose well, indeed. (I would even have argued that tag objects could have been simple commit objects, with an additional header "tag", but oh well.)

I suspect that you want to introduce a different object type to be able to implement a new algorithm. But I think that the appropriate data structure is still contained in the existing set of types in Git.

I don't think that this is a good solution:

performance-wise. Especially since we do not pack branch refs.

Side note: I recently kicked around the idea to actually keep the refs in the packed-refs file, and for updating refs do a lock-mmap-findref-replace-or-rewrite, where a rewrite only happens if we delete or insert a ref.

Then, we could even unify the info/refs and packed-refs, so that we don't hear bugreports about http transport not working every week or so.

Just an idea.

Ciao, Dscho

On 6/10/07, Pierre Habouzit <madcoder@debian.org> wrote:

Call me a fool, but writing a <new> bugtracker looks like a boil-the-oceans scheme.

Adding git & gitweb support to traq, bugzilla, mantis, gforge, etc is what is going to make the difference. Most of those have already the ability to "link" to one or more commits -- after the commits are done and in GIT.

So you can tell your bugtracker
 - which commit fixed it -- usually auto-linked if you include the
bugnumber in the commit message
 - which commit added the test -- auto linked as above
 - which commit introduced the bug -- if such thing exists and someone
digs it up

If the bugtracker can also auto-link things that look committish in text entered by users (someone might write "bisect sez that f345e is to blame"), with tooltips indicating in which heads those commits resides (like gitk does), then it's just gorgeous.

But I would _never_ try to describe all the possible relations in the schema -- existing trackers use a liberal mix of regexes and cache tables with some free form text fields for this kind of stuff.

And definitely, if you use git as an alibi to write a new bugtracker, don't use the "works only with git" as a feature. It should work with as many SCMs as possible.

OTOH, that's just me, I'm lazy and like to work on already-successful projects that are 99% there for my needs (and where I can add that 1%).

cheers,

m

"Martin Langhoff" <martin.langhoff@gmail.com> writes:

You are a brave person to say this (no sarcasm --- I wish I were, too).

On one hand, I very much applaud and appreciate your comment for injecting sanity to the discussion. On the other hand, if Linus had such an attitude, we might not be hacking on git right now.

After looking at the above existing alternatives, some brave soul might decide and say, "Hey, I can write something better in 2 weeks" ;-).

All of your examples are going from a single bug to commits, but from a release person's point of view, you are never interested in a single bug, just like a top-level maintainer is never interested in a single file. A release person would want to go in the reverse direction: from a commit range to a set of bugs. What bugs were fixed and what regressions were introduced during this release cycle. While embedded ticket numbers in commit log messages would certainly help, a change made to fix a particular bug may fix another as its side effect, and the develeoper who did the change may not know about the latter when the commit log message is written.

These are indeed very good points.

On 6/10/07, Junio C Hamano <gitster@pobox.com> wrote:

Thanks for the reply - sorry for being a bit of a stirrer ;-) Yes, I do appreciate the irony and I'd have never started git. If anyone really wants to do their own bugtracker... be like Linus not like me (a mere follower) ;-)

Definitely. But that takes a deeper look into the above alternatives, and probably a good perspective. I hope noone takes offence if I say that -- as a user of several bugtrackers over ~10 years -- I haven't yet read anything in this thread that is exciting.

And "it's closely integrated with git" can actually be a misfeature. Cool if it's what gets you going, but not enough for world domination ;-)

That's right. It's not that hard to correlate commits-in-the-changelog and ask the bugtracker to produce a "tracker changelog" showing all the bugs/tasks that link to the commits (with that liberal view of "links to" discussed before). Formatting of the tracker changelog _is_ tricky but that's due to the ambiguities in those relationships.

(In Moodle I use JIRA, a non-FOSS tracker, and it does exactly that. I think it has some notion of CVS branches, and where tags sit, so you can ask for a report of which bugs are fixed in the release you are preparing on branch X).

I agree it's useful, but I don't think it has any benefit having it in the SCM _at all_. Having them in the BT is a lot more flexible -- and the fact that git has stable commit IDs makes it easier to integrate in a BT; as the BT can spot that the commit fixing bug 123 is now merged into head ZZ as well as head YYY.

So the BT holds "external tags" to your commits. Lots of them. With a ton of data. And that's great -- in fact there's no stopping it -- the stability of SHA1s means that the whole internet holds tags to your repo. Every time a SHA1 (or output of git-describe) is mentioned in a mailing list, wiki or forum, there's a tag.

Now, to rule the world, BTs gain a lot more from being able to integrate with different SCMs, automated test systems (like tinderbox), MTAs (debbugs), wikis (traq), stats tracking for PHBs (bugzilla), etc. So loose coupling wins here, and git's SHA1s are great for that.

And at git's end we can get the smooth integration using/abusing that loose coupling strategy. So if git-log/gitk/qgit grow some hooks that can be used to perform a lookup... I can think of a perl script that gets events for each SHA1 that git-rev-list produces and returns one bug number, url and title (or more than one) that then git-log / gitk / qgit / annotate can use to further "decorate" the commit info in a similar style to what gitk is doing now with head names in each commit.

Would that fit your vision of a nicely integrated tracker?

martin (who's now thinking how to craft a proof of concept)

On Sun, Jun 10, 2007 at 06:59:13PM +1200, Martin Langhoff wrote:

  Sure, what if I like it anyway ?

  Sure, you can do that and still inherit from the many downsides of
those tools: central, needs another separate tool to work with, and a
tool that nowadays tends to eat 500Mb of your memory if you take the
most hype implementation of it (Yes I'm describing a web browser), that
is slow like hell (because yes, there is many clients at the same time
on the central server, that is overloaded like hell), and so on.

  You can like central web UIs, your choice. And I suppose if grit works
well, there will be one at some point.

  yeah, that is what bugtrackers already do. Though, that's of no use
for release managers. What is useful is what people call "milestones" in
trac e.g., meaning a set of bugs that has been flagged as needing to be
fixed. And what a release manager need is a tool to deal with this set
of bugs as a whole.

  That's the same argument that Linus has against per-file tracking.
Also atm when you e.g. backport a patch that fixes this or this bug,
you're no BTS helps you tagging the bug as fixed in that branch as well.
Not to mention that BTS I know do not make any use of the commits DAG.
And for a very good reason, there is no real DAG in many existing tools
(svn, cvs, ...).

  that's not up to the BTS tool to do that, it's way to high level. It's
up to the importing filters/hooks that will parse the associated ML, and
that would translate that to useful low level BTS commands.

  No it should not, because it can't. I want the distributed and
Bug status spanning-DAGS be a central feature. So that means that this
tool can only work on top of SCMs that support that. ttbomk git, hg,
_maybe_ bzr fit the description. I only know the former, but I really
plan to write the tool in a way that the underlying SCM does not matters
_too_ much. Maybe I'll fail. Honestly, I don't really care (yet ?).

  You're a lucky guy. All bug trackers I've used suck a way or another,
that impedes my workflow a lot. Let's cite a couple:
  * bugzilla: takes more from the -zilla than from the BTS side. It's
    huge, monstruous, slow (have ever used glibc's bugzilla ? it has
    maybe 5k bugs, it's slow like if it ran on an Atari ST), complex,
    the mail gateway suck hard, it's completely unusable for me. Believe
    me, I've packaged KDE for 2 years in Debian, now am in the glibc
    team. Every single day I have to work on this horrible tool is a
    PITA.

  * flyspray: I've been upstream for a short time. Visually nice, good
    to work with, UI is great. Integration totally suck, worthless.
    Can't use mails either, needs a Web Browser -> useless. The same
    holds for mantis, roundup and a lot of other friends.

  * debbugs: oh yeah there is a mail interface. So slow that you have to
    wait up to 15 minutes to see your command be taken into account. And
    when you have to deal with dozens of bugs (Yes I've done that on a
    regular basis) you _have_ sometimes to wait for the answer to come
    back (because you need an ID that will be in there e.g.) to continue
    your work. That is unacceptable, you pass most of your time waiting.
    Moreover sometimes you made an error in your commands, so you also
    need to parse the anwer ... one day later because "immediateley when
    you still remembered what this was about" is not an option.
    Unacceptable again. The plus: it uses mboxes, hence is worthwile in
    a hacking environment, it fits the workflow well.

  * trac: very very very nice tool. I mean it. We use it at work, where
    I have to suffer svn (through git-svn though). It's really nice (did
    I repeat myself ?). THough, it's on top of svn, and you can't use
    the Bugs informations from your repository. You can't say: I'm
    backporting that patch into that branch. Now what affects this
    branch please ? Trac can't answer that (and ttbomk now BTS really
    can anyway, except Debian's debbugs instance, and it's somehow
    limited). That is a question a release manager takes 80% of his time
    to ask. I hope grit can take back to the 0.01% of his time, which is
    still too much.

On Sun, Jun 10, 2007 at 08:55:21PM +1200, Martin Langhoff wrote:

  I trol^Wdiscuss everyday on debian's channel with friends that tell
that svn is the best tool ever, and that they would never ever use a
distributed SCM because it's too hard to understand. Your call.

  That's exactly why distributed rock. Because everyone could _really_
own _his_ bugtracker. This solves the same range of ACLish issues git
solves with respect to the code.

  Why wouldn't it exist a "public reporting interface"-branch ? that
would be the same purpose as the mainline ~linus/linux2.6.git tree. And
you can build/instanciate your beloved web UI on top of that, and people
would just have to pull from there. What a shock, it's easy !

  Oh, because the code never changes. Doh, how stupid I am :)
  No, really, you name your bug after the sha hash of the first report,
I think that's pretty obvious. That gives you a bug name. Then you ask
git for "what's the current sha for this bug in the tip of the BTS
branch", then you ask "so now what this new sha is pointing to in the
code". That's a small indirection that I suppose is bearable.

  It's not integrated with the workflow. And sorry, but git@vger (or
workse lkml@vger).. it does not work. Maybe for git it does because the
flow is still human manageable, and that it seems junio has enough time
for it. But for the kernel ? please. You should read Bastian Blank
frustration about regressions and nobody tracking them. Know why ?
because there is no tool that is well known and well integrated in the
workflow. There is a long rant against kernel.org's bugzilla, you should
read it as well. It's not really instructive (at least there wasn't
anything new for me in it, I was already bought to many arguments in
there). But you'll see the world isn't as rosa-lila you think it is.

On Sun, Jun 10, 2007 at 08:38:03PM +1200, Martin Langhoff wrote:

  I'm sure I could come up with something really better in say a
month... If I hadn't paid work to do too :)

  It's a misfeature for you because you read it as "non portable".
That's a fair point. And like said, it may be extended to SCM's with the
same set of features I need to build it. But let's be honnest, I don't
care about a BTS that uses the smallest common set of features of SCM's.
Reading this list, you should know it's almost a void set. No, I think a
good BTS should make an excellent use of high level features of the SCM.
The real problem is, there is maybe 2 or 3 SCMs that have this set of
strong and good features. Too bad for the other, I can't work with them,
they suck hard, and I don't see why I should support bad practices
anyways.

  (Yes I'm also a guy with strong opinions too, it's not really
restricted to Linus ;p)

  If you do that you loose:
  * fastness, and I don't want to work with debbugs anymore.

  * distribution: Meaning that for _one_ project everybody needs to use
    this central bugtracker. Whereas .oO(kernel) there is some projects
    where the subcomponents are dealt with from very different teams,
    very different way to release things, and they are interested with
    their bugs, and their bugs only. They would prefer a very fast
    interface to deal with their 1k bugs, and not worry about the 100k
    the rest of the project has.
    Branching bugs also makes sense you know ?

  You are the one saying it. I beg to differ.

  IMHO a BTS is a _low_ level tool. that's the road git took, I
sometimes describe the plumbing git commands as the "Assembler" of the
SCM world to friends when I talk to them about git. That's really the
best way to implement a thing: have a small small set of rock solid,
well designed tools, and build the others as porcelains with them.

  Testing is a high level tool. I don't need to support them, I need to
have exactly the low level querying rocket-fast query interfaces so that
integration scripts are at most 100 lines long.

  Honestly ? No, because that would be horribly slow (but I'd love to be
proven wrong).

Cheers,

On 6/10/07, Pierre Habouzit <madcoder@debian.org> wrote:

Be my guest. It might turn out to be a beautiful project, but has low chances of helping GIT-BTS integration in general.

Most usable BTSs work on lighter webbrowsers, and can be tuned to work reasonably. That's not a dead-end per se.

Bugtrackers are communication tools between developers and users. In many spaces, they are _teaching tools_, teaching the users about info developers need. That's why BTSs have explicit fields asking for important variables like OS, Arch, and version you are reporting a bug against. That's also why the BTS gains a lot from being web-based: extreme portability, reachability, zero-install for users.

Hmmm. Most BTSs have milestones, and the integration of the above with milestones is useful for release managers. How about the _rest_ of the BTS-using populace?

Making the BTS DAG-aware is overkill. The BTS can ask for every update on every branch "what commits does it bring into branch X?" and that's all you need. If you backport a patch and mention the original patch SHA1 the BTS can do its job.

And all of that can be done SCM-agnostic - except for the regex to spot a commitid.

But you lose portability. And gain... what that you can't do portably?

Ok - but BTSs are a compromise, something that must work for users, and developers.

Don't think I've seen politics over who owns the bugtracker ;-) but I _have_ seen politics over specific bugs (developers close unfixed bugs, flamefests ensue). I guess with a DBTS everyone can have their own status for the bug... but does that help the user? Or the developer?

Of course, you _can_ map the DBTS storage on git's storage. But git's storage has been designed to match the task. While I'm sure there are some good tricks to reuse, I don't think that it's a good fit.

What part would be slow?

cheers

martin

On Mon, Jun 11, 2007 at 11:14:03AM +1200, Martin Langhoff wrote:

  okay, my point is also that using yet another tool that is neither
integrated in my shell nor in my editor sucks. I use the shell to use
$SCM, and to perform tests of my work, so it's in my workflow. I use the
editor (well do I need to tell why ? :P), I just don't want to use a
browser where the two previous should be enough.

  Because it's git-based does not implies there won't be a web UI. And
it's not because there is a lot of options to the underlying process
that you need to show them all to the user.

  My experience is also that people never fill informations about OS,
Arch, version properly, because it's tiredsome. debbugs (through
reportbug) approach is excellent: assuming that the user reports the
error from the machine where he experienced it and taking all the
information automatically. _here_ is how it should work for the user,
e.g.: “please download and run that script, and paste the results in your
bug report pretty please ?”

  How many of those bts'es help the RMs to know if their
soon-to-be-stable branch fits the milestone ? milestones in any BTS I
know are global "yes the fix exists, in this revision (and this
information is not always here)". Sorry but it's worthless.

  Okay, who else uses a BTS ? developpers, as I'm a developper, I'll
write a tool that pleases me. Let's say developpers are taken care of.

  Users ? For a user, what should be made easy is reporting, and
interacting with whoever is dealing with the bug on the other end.
Advanced users will want to be able to track discussion about some bugs
they are interested into (meaning that it must be a way to crawl the
bugs database).

  So let's take the three points:
  * reporting: the tools to make reporting fast, efficient, _and_
    useful to the developpers - many many many bug reports suck, hard -
    are very often project dependant: a project in python e.g. won't
    care about the architecture usually (hence forcing it is stupid) but
    will probably mind the OS, python version, distribution, ... Another
    project in C will mind the architecture, but is knowingly working
    only on Linux so won't mind the OS, but would maybe mind the kernel
    version. etc...

    => What can I do ? provide a way to deal with some kind of
       property/values that can be customizeable at the BTS side and
       show some examples of reporting scripts. ttbomk only debbugs and
       reportbug knows how to do that. It does not prevents bad bug
       reports at all, but it makes many that would have been bad almost
       good, at least usable.

  * simplicity to follow up: there isn't 12987123 solutions here. mail,
    mail, mail, mail. A BTS should be able to track mail discussions one
    way or another. For pure-web BTS, there is still mail alerts to say
    "hey $someone has said $sth on your bug $bug". So the user already
    is in his MUA when he receives that, well, "reply"-button is the
    shortest way to answer, make it possible.

  * crawling the database: that one should not be done by the BTS. Every
    BTS on the planet doing that is wrong. There is plenty of good
    search engines (xapian, clucene, ...) just adapt one. And the most
    simple your storage backend is, the simplest it will be.

  What I'm saying is that maybe we don't call BTS the same object. To me
a BTS is a "thing" able to deal with large amounts of bugs in an
efficient way, and is able to deal with some basic things like bug
statuses, and some kind of queries to the database (not the full text
search ones). Then you have the Upper Layer Tools: guis, tracking
scripts, every bling you need. Note that Git is exactly designed this
way: low level tools, and then tools like git-mergetool, gitk, ... that
don't use _any_ git internal, but rather only use the underlying
commands. I want to build the core functions to deal with a BTS, so that
everybody is able to turn this into the BTS workflow he needs.

  Why ?

  or the BTS can do it for you and you would not need to feed him with
the sha directly.

  Sure, there is projects doing that out there, I believe they are
wrong. Portability is important. I mean, when you write a tool, that has
a purpose, yes you must try to make it available everywhere. That's why
there are people trying hard to make git build and work on windows.

  But when you write a tool to help _you_ work, there is now need to
make it work for every workflow in the planet, because that would mean
align yourself to the lowest common denominator of the workflows you can
work with. That's not an acceptable trade-off. Why would I have to
support cvs or svn ? I don't use them. I just don't care about them.
Wherever place I'll work in, I'll use git. Sorry but full genericity
(because it's what we are talking about, not portability) is an
impediment, not an enabler.

  I don't think it needs to be a compromise.

  That one is easy. Indeed, the big politics in bugtrackers are ...
severity-ping-pong, or close-wars. Good example of that is btw:
http://sourceware.org/bugzilla/show_bug.cgi?id=4509.

  Okay, what would we gain in a DBTS: developer would still be (sorry)
a perfect asshole with the user. That is a thing we cannot fix. Though,
the release manager will probably disagree with him. So this bug that
_he_ considers non existant will be closed in his repository, but still
remain open in the main one. Meaning that if another developer steps up,
he'll see this issue is not fixed. Else nobody will have any chance to
step up, ever.

  The reverse works equally well: if RM decides bugs have to be closed,
developer is able to reopen the bugs (or not merge) in his repository,
so that he can have a chance to fix them.

  Yes there is politics in BTSes, and yes it can help.

  The perl scripts. It would perceptibly slow down commits. And I don't
want that now that I finally have a fast SCM. I just don't want to turn
git into bzr.

On 6/11/07, Pierre Habouzit <madcoder@debian.org> wrote:

I've seen all those bugtracker-wars. But they never block a developer or fellow user from saying --hey, here's a patch. And with a DSCM, that clears things up quite quickly.

I don't understand how  "Else nobody will have any chance to step up, ever."

The model I was thinking of was of _not_ slowing down your commits ;-) but

 * Stick to a mostly centralised BTS that tracks a limited set of repos
 * When you push to the public repo, the BTS updates its bug status
 * on git-pull, update a (fast!) local cache of BTS data
 * on gitk use a similar technique to the "follows" line shown for
each commit to display bug info "inline"

cheers,

martin

On Sun, Jun 10, 2007 at 00:35:33 -0700, Junio C Hamano wrote:

It would be really useful to have a tool, that could link a bug report to a test case demonstrating it and reporting whenever output of that test case changes. This would make it much easier for developer to see which bugs he might have fixed when doing a refactoring.

It should probably report not just unexpected success, but also change to the
error output, because the test case does not have to be 100% correct. That is
if you have a test case that prints:
  testFoobar FAIL "foo" != "Bar"
and than starts to say:
  testFoobar FAIL "foo" != "Foo"
the problem is probably fixed, but the test still fails because of minor
error in the expected output string.

On Sun, 2007-06-10 at 03:50, Jan Hudec wrote:

I think the inverse is interesting too. That is, applying keywords to tests and then making the test database keywords searchable.

I used to work for a company that had a huge collection of test cases for its product. It would take days to run a validation run for some of the simplest of changes that were introduced during development.

Often what I wanted, and what was needed, was a simple sanity check that verified that the proposed change was not brain-dead from the start.

Over time, I found that certain tests proved to be good at detecting failures in particular portions of the code. Often, that test was not necessarily directly related to the concept it was _supposed_ to be designed to test, but incidentally was good at some _other_ concept too.

The developer who made the change, and ran the tests, then was able to state "Test t42 from suite Frotz is good at detecting changes to the <SomeModule>."

The trick is to now associate with that test the keyword "SomeModule" so that in the future, another developer could ask: "Say, I just modified <SomeModule>, are there any tests that are good at proving my changes sound?" A test driver could then focus some test cycles quickly.

Now, for git, it's not likely a problem to just run through all of its tests. At ${PriorCompany} it was essential to pare down the test suites to a manageable size, yet still have assurance that your coverage was reasonably good.

jdl

Hi,

This subject is quite interesting. I read that one of the main expected goal of integrating SCM and BT is to help release manager in its task.

In my point of view, we have to keep in mind that it's not because a commit solved a problem, that all the following commits will always solve the problem. Development ALWAYS suffers regression. The really way to avoid this is to have an organisation of code that allows automatic tests. So it needs something greater than the SCM: you have to be organized for this.

One interesting project to have a look for is aegis ( http://aegis.sourceforge.net/ ). It proposes a sort of SCM, that integrates process to ensure quality of code. One of them is that the /SCM/ will control the non regression before commiting.

I hope these informations will help defining how we can design a system that integrates SCM and BT in a distributed manner.

On Sun, Jun 10, 2007 at 18:59:13 +1200, Martin Langhoff wrote:

I don't know about any *distributed* bug tracker, which is the point here.

We have several distributed version control tools, but no other distributed tools for the other tasks in configuration management.

It would certainly be useful. But the problem with these bug trackers is, that they don't go all that well with how many hackers work. And the less they fit with distributed workflow.

 - The web interface is usually not a good match for the problem. Email
   interface is better in many respects, but it still does not cut it.
 - You can't really use the ability of version control to work disconnected,
   when you don't also have the bug information.
 - Distributed version control is designed to decrease the workload of the
   central maintainer(s) while keeping him in control. But with centralized
   bug tracking he still has a lot of work with that that the mob can't help
   him with. It helps if the bug tracker can close bugs based on something in
   commit-log, but you also need to see what it is that will be closed, which
   requires integrating the bug tracker into the version control.

The other thing is, that a good bug tracker also needs to be integrated with the *mailing list*. The major part of bug tracking is discussing those bugs and discussions usually take place on mailing-lists (and on irc -- integrating that as well would be nice too).

I've actually already seen some attempts at something like this project. The first one was for GNU Arch. Instead of using a bug-tracker, there was a branch where each bug had it's mbox with all the relevant data. This mbox was moved between directories depending on it's state. AFAIR there was no tool to maintain it, just a description of the workflow. (Tom Lord called it "the game". See eg. http://www.dasbistro.com/~sam/mail/tom-lords-game)

The other attempt is obviously the Bugs Everywhere thing.

A related thing is Bazaar's "Bundle Buggy". It tracks patch submission rather than bugs, but it is roughly what I meant by the "mailing list integration" above. It "reads" the mailing list, watching for anything labeled [PATCH] or [BUNDLE] with patch or bundle (bundle is bazaar's extended patch) attached. It also watches for replies with review (review outcome is stated with "-1" (veto), "-0", "+0" and "+1" (ok)) and with further patches/bundles (updated versions of the patch).

If it uses git as it's database, which it probably will, it will definitely require git. Now it should be possible to have the code in different version control, but it would be integration with things like git format-patch, that will really make it cool stuff -- and that won't work with other version control out of the box.

Yes. But for many people current bug tracking tools do NOT work 99%.

On 6/10/07, Jan Hudec <bulb@ucw.cz> wrote:

As an end user, I suspect I _don't_ want to have to report a bug in a distributed bug tracker ;-) In that space, the Malone guys (canonical) have their fingers on one of the most serious issues, and perhaps it's interesting to see what they've done there. It's really useful, even if I don't think I want to have to maintain it :-/

Bugtrackers are co-owned by developers, users and (where they exist) project managers.

I agree. I love/hate debbugs too.

A cache fixes the reading part - see my other post, and imagine being able to have a local sqlite cache of the BT key data indexed by referenced SHA1, showing up with your commits in gitk.

The write part is solved (in part) by committing to git the fix -- if you mentionm the bug ID, the central BT will pick it up when your commit appears in the branches/repos that the BT can see.

For "just adding a comment", the write part is solved by the "email" interface, like with debbugs.

And to provide a single place for users to report a problem and track its status.

Well - hmmm. Git's database is great at tracking _content identity_. But a bug's content identity (description+comments+status) changes all the time. I don't think it's naturally a good match.

Perhaps it makes sense to mix git's storage model with something else...?

Hmmm. I agree in that "does not work disconnected" is a big issue with web tools, but debbugs works disconnected, and is good. Git's bugtracker (git@vger) works disconnected too ;-) And googlegears might help the rest of us. Is there any other problem with current BTs?

cheers,

martin

On Sun, Jun 10, 2007 at 20:55:21 +1200, Martin Langhoff wrote:

As an end user, you would just post the bug report to a mailing list. The bug tracker would take care of adding it to the "master repository".

I don't think they even have a mail interface. You have to register to post a bug, just as with most of the crappy web-based bug trackers out there.

They have some nice integration with bazaar, but AFAIK the branch has to be mirrored on launchpad, to which the whole thing is really tied too much. Also I can find download link anywhere there, nor anywhere they'd state it's license (and from what I heard it is NOT open-source).

So while they might have some nice features, it's not that suitable for general public.

I agree that branches don't make that much sense in bug trackers. Except in rare cases like when the project is forked or such. The bug tracker is there so the people stay in sync. However I think trying to design a distributed bug tracker can bring useful insignts into the problems involved.

... there's a related issue of what the mail user agents can do. If my mail client was really good as personal task manager, mail-based bug trackers would work quite well. But the MUAs are not all that good.

Just like there is the "master" repository, there would be the "master" bug tracker.

You are right here. Git can be used to store the data bits, but they need to be glued together somehow (with tags or something). So we can as well store them in some other kind of database and it might be even better.

Well, there is no BT that would satisfy all the points above ;-). Debbugs has a good email interface, but it's a huge beast and tied with the Debian archive logic quite a lot. The rest -- except maybe gnats (I'll have to look at that -- it looks interesting) -- does not seem to have much sensible email support.

Following up on the note about MUAs above, one interesting idea to create
a bug tracker (that would not be git specific nor actually use git for
anything) would be to:
 - Create a mailing list bot, that would watch for bug reports either on
   dedicated address or by watching for [BUG] or something, and replies to
   them.
 - Export the bug database via read-only imap (or pop3 or webdav). There are
   already tools to create local cache for such protocols, which would
   resolve the disconnected issue and give good query speed, because the data
   would be local.
 - The server would make sure that all the messages applying to a particular
   issue are correctly linked with In-Reply-To: headers to form a single
   thread.
 - Status of the bug could be expressed with mailbox it is in (backwards
   compatible way) or some kind of tags (which would need to be supported by
   the mail client -- I really think it's about time for something like
   that).
 - Other metainformation about the bug could be added as a special message
   inserted at the begining of the thread, or added into the first message
 - Than support for todo management could be improved in some mail clients.
   It is not precondition for this kind of bug tracking to be useful and on
   the other hand would be useful on it's own.
 - It could even be done as another interface to an existing web-based BT,
   though I would prefer if the web interface would not have to be running
   then.

Jan Hudec <bulb@ucw.cz> writes:

Indeed, the bundle buggy is inspired from the "Bug goo" developed earlier for GNU Arch. This one was watching mails labeled with [BUG] on the mailing list.

I found the idea really good. Indeed, as a user, when I find a bug in a piece of software, I often hesitate between using the bugtracker (in which case my bug is often mostly ignored), or reporting it on the mailing list (in which case, a long discussion can follow, at the end of which everybody thinks that someone else will fix the bug, and no one actually does). With the bug goo, the bugtracker becomes the mailing list, with all the advantage of it (discussions easy to follow, threaded, ...), and a few additions, the biggest of which is status tracking, in particular, the ability to list all unsolved bugs.

Unfortunately, as most of the GNU Arch stuff, it started with good ideas, but was abandonned before it started being interesting :-\.

On Sun, Jun 10, 2007 at 06:59:13PM +1200, Martin Langhoff wrote:

  Sure, what if I like it anyway ?

  Sure, you can do that and still inherit from the many downsides of
those tools: central, needs another separate tool to work with, and a
tool that nowadays tends to eat 500Mb of your memory if you take the
most hype implementation of it (Yes I'm describing a web browser), that
is slow like hell (because yes, there is many clients at the same time
on the central server, that is overloaded like hell), and so on.

  You can like central web UIs, your choice. And I suppose if grit works
well, there will be one at some point.

  yeah, that is what bugtrackers already do. Though, that's of no use
for release managers. What is useful is what people call "milestones" in
trac e.g., meaning a set of bugs that has been flagged as needing to be
fixed. And what a release manager need is a tool to deal with this set
of bugs as a whole.

  That's the same argument that Linus has against per-file tracking.
Also atm when you e.g. backport a patch that fixes this or this bug,
you're no BTS helps you tagging the bug as fixed in that branch as well.
Not to mention that BTS I know do not make any use of the commits DAG.
And for a very good reason, there is no real DAG in many existing tools
(svn, cvs, ...).

  that's not up to the BTS tool to do that, it's way to high level. It's
up to the importing filters/hooks that will parse the associated ML, and
that would translate that to useful low level BTS commands.

  No it should not, because it can't. I want the distributed and
Bug status spanning-DAGS be a central feature. So that means that this
tool can only work on top of SCMs that support that. ttbomk git, hg,
_maybe_ bzr fit the description. I only know the former, but I really
plan to write the tool in a way that the underlying SCM does not matters
_too_ much. Maybe I'll fail. Honestly, I don't really care (yet ?).

  You're a lucky guy. All bug trackers I've used suck a way or another,
that impedes my workflow a lot. Let's cite a couple:
  * bugzilla: takes more from the -zilla than from the BTS side. It's
    huge, monstruous, slow (have ever used glibc's bugzilla ? it has
    maybe 5k bugs, it's slow like if it ran on an Atari ST), complex,
    the mail gateway suck hard, it's completely unusable for me. Believe
    me, I've packaged KDE for 2 years in Debian, now am in the glibc
    team. Every single day I have to work on this horrible tool is a
    PITA.

  * flyspray: I've been upstream for a short time. Visually nice, good
    to work with, UI is great. Integration totally suck, worthless.
    Can't use mails either, needs a Web Browser -> useless. The same
    holds for mantis, roundup and a lot of other friends.

  * debbugs: oh yeah there is a mail interface. So slow that you have to
    wait up to 15 minutes to see your command be taken into account. And
    when you have to deal with dozens of bugs (Yes I've done that on a
    regular basis) you _have_ sometimes to wait for the answer to come
    back (because you need an ID that will be in there e.g.) to continue
    your work. That is unacceptable, you pass most of your time waiting.
    Moreover sometimes you made an error in your commands, so you also
    need to parse the anwer ... one day later because "immediateley when
    you still remembered what this was about" is not an option.
    Unacceptable again. The plus: it uses mboxes, hence is worthwile in
    a hacking environment, it fits the workflow well.

  * trac: very very very nice tool. I mean it. We use it at work, where
    I have to suffer svn (through git-svn though). It's really nice (did
    I repeat myself ?). THough, it's on top of svn, and you can't use
    the Bugs informations from your repository. You can't say: I'm
    backporting that patch into that branch. Now what affects this
    branch please ? Trac can't answer that (and ttbomk now BTS really
    can anyway, except Debian's debbugs instance, and it's somehow
    limited). That is a question a release manager takes 80% of his time
    to ask. I hope grit can take back to the 0.01% of his time, which is
    still too much.

On Sun, Jun 10, 2007 at 08:55:21PM +1200, Martin Langhoff wrote:

  I trol^Wdiscuss everyday on debian's channel with friends that tell
that svn is the best tool ever, and that they would never ever use a
distributed SCM because it's too hard to understand. Your call.

  That's exactly why distributed rock. Because everyone could _really_
own _his_ bugtracker. This solves the same range of ACLish issues git
solves with respect to the code.

  Why wouldn't it exist a "public reporting interface"-branch ? that
would be the same purpose as the mainline ~linus/linux2.6.git tree. And
you can build/instanciate your beloved web UI on top of that, and people
would just have to pull from there. What a shock, it's easy !

  Oh, because the code never changes. Doh, how stupid I am :)
  No, really, you name your bug after the sha hash of the first report,
I think that's pretty obvious. That gives you a bug name. Then you ask
git for "what's the current sha for this bug in the tip of the BTS
branch", then you ask "so now what this new sha is pointing to in the
code". That's a small indirection that I suppose is bearable.

  It's not integrated with the workflow. And sorry, but git@vger (or
workse lkml@vger).. it does not work. Maybe for git it does because the
flow is still human manageable, and that it seems junio has enough time
for it. But for the kernel ? please. You should read Bastian Blank
frustration about regressions and nobody tracking them. Know why ?
because there is no tool that is well known and well integrated in the
workflow. There is a long rant against kernel.org's bugzilla, you should
read it as well. It's not really instructive (at least there wasn't
anything new for me in it, I was already bought to many arguments in
there). But you'll see the world isn't as rosa-lila you think it is.

On Sun, Jun 10, 2007 at 08:38:03PM +1200, Martin Langhoff wrote:

  I'm sure I could come up with something really better in say a
month... If I hadn't paid work to do too :)

  It's a misfeature for you because you read it as "non portable".
That's a fair point. And like said, it may be extended to SCM's with the
same set of features I need to build it. But let's be honnest, I don't
care about a BTS that uses the smallest common set of features of SCM's.
Reading this list, you should know it's almost a void set. No, I think a
good BTS should make an excellent use of high level features of the SCM.
The real problem is, there is maybe 2 or 3 SCMs that have this set of
strong and good features. Too bad for the other, I can't work with them,
they suck hard, and I don't see why I should support bad practices
anyways.

  (Yes I'm also a guy with strong opinions too, it's not really
restricted to Linus ;p)

  If you do that you loose:
  * fastness, and I don't want to work with debbugs anymore.

  * distribution: Meaning that for _one_ project everybody needs to use
    this central bugtracker. Whereas .oO(kernel) there is some projects
    where the subcomponents are dealt with from very different teams,
    very different way to release things, and they are interested with
    their bugs, and their bugs only. They would prefer a very fast
    interface to deal with their 1k bugs, and not worry about the 100k
    the rest of the project has.
    Branching bugs also makes sense you know ?

  You are the one saying it. I beg to differ.

  IMHO a BTS is a _low_ level tool. that's the road git took, I
sometimes describe the plumbing git commands as the "Assembler" of the
SCM world to friends when I talk to them about git. That's really the
best way to implement a thing: have a small small set of rock solid,
well designed tools, and build the others as porcelains with them.

  Testing is a high level tool. I don't need to support them, I need to
have exactly the low level querying rocket-fast query interfaces so that
integration scripts are at most 100 lines long.

  Honestly ? No, because that would be horribly slow (but I'd love to be
proven wrong).

Cheers,

On Sun, Jun 10, 2007 at 06:59:13PM +1200, Martin Langhoff wrote:

  Sure, what if I like it anyway ?

  Sure, you can do that and still inherit from the many downsides of
those tools: central, needs another separate tool to work with, and a
tool that nowadays tends to eat 500Mb of your memory if you take the
most hype implementation of it (Yes I'm describing a web browser), that
is slow like hell (because yes, there is many clients at the same time
on the central server, that is overloaded like hell), and so on.

  You can like central web UIs, your choice. And I suppose if grit works
well, there will be one at some point.

  yeah, that is what bugtrackers already do. Though, that's of no use
for release managers. What is useful is what people call "milestones" in
trac e.g., meaning a set of bugs that has been flagged as needing to be
fixed. And what a release manager need is a tool to deal with this set
of bugs as a whole.

  That's the same argument that Linus has against per-file tracking.
Also atm when you e.g. backport a patch that fixes this or this bug,
you're no BTS helps you tagging the bug as fixed in that branch as well.
Not to mention that BTS I know do not make any use of the commits DAG.
And for a very good reason, there is no real DAG in many existing tools
(svn, cvs, ...).

  that's not up to the BTS tool to do that, it's way to high level. It's
up to the importing filters/hooks that will parse the associated ML, and
that would translate that to useful low level BTS commands.

  No it should not, because it can't. I want the distributed and
Bug status spanning-DAGS be a central feature. So that means that this
tool can only work on top of SCMs that support that. ttbomk git, hg,
_maybe_ bzr fit the description. I only know the former, but I really
plan to write the tool in a way that the underlying SCM does not matters
_too_ much. Maybe I'll fail. Honestly, I don't really care (yet ?).

  You're a lucky guy. All bug trackers I've used suck a way or another,
that impedes my workflow a lot. Let's cite a couple:
  * bugzilla: takes more from the -zilla than from the BTS side. It's
    huge, monstruous, slow (have ever used glibc's bugzilla ? it has
    maybe 5k bugs, it's slow like if it ran on an Atari ST), complex,
    the mail gateway suck hard, it's completely unusable for me. Believe
    me, I've packaged KDE for 2 years in Debian, now am in the glibc
    team. Every single day I have to work on this horrible tool is a
    PITA.

  * flyspray: I've been upstream for a short time. Visually nice, good
    to work with, UI is great. Integration totally suck, worthless.
    Can't use mails either, needs a Web Browser -> useless. The same
    holds for mantis, roundup and a lot of other friends.

  * debbugs: oh yeah there is a mail interface. So slow that you have to
    wait up to 15 minutes to see your command be taken into account. And
    when you have to deal with dozens of bugs (Yes I've done that on a
    regular basis) you _have_ sometimes to wait for the answer to come
    back (because you need an ID that will be in there e.g.) to continue
    your work. That is unacceptable, you pass most of your time waiting.
    Moreover sometimes you made an error in your commands, so you also
    need to parse the anwer ... one day later because "immediateley when
    you still remembered what this was about" is not an option.
    Unacceptable again. The plus: it uses mboxes, hence is worthwile in
    a hacking environment, it fits the workflow well.

  * trac: very very very nice tool. I mean it. We use it at work, where
    I have to suffer svn (through git-svn though). It's really nice (did
    I repeat myself ?). THough, it's on top of svn, and you can't use
    the Bugs informations from your repository. You can't say: I'm
    backporting that patch into that branch. Now what affects this
    branch please ? Trac can't answer that (and ttbomk now BTS really
    can anyway, except Debian's debbugs instance, and it's somehow
    limited). That is a question a release manager takes 80% of his time
    to ask. I hope grit can take back to the 0.01% of his time, which is
    still too much.

On Sun, Jun 10, 2007 at 08:55:21PM +1200, Martin Langhoff wrote:

  I trol^Wdiscuss everyday on debian's channel with friends that tell
that svn is the best tool ever, and that they would never ever use a
distributed SCM because it's too hard to understand. Your call.

  That's exactly why distributed rock. Because everyone could _really_
own _his_ bugtracker. This solves the same range of ACLish issues git
solves with respect to the code.

  Why wouldn't it exist a "public reporting interface"-branch ? that
would be the same purpose as the mainline ~linus/linux2.6.git tree. And
you can build/instanciate your beloved web UI on top of that, and people
would just have to pull from there. What a shock, it's easy !

  Oh, because the code never changes. Doh, how stupid I am :)
  No, really, you name your bug after the sha hash of the first report,
I think that's pretty obvious. That gives you a bug name. Then you ask
git for "what's the current sha for this bug in the tip of the BTS
branch", then you ask "so now what this new sha is pointing to in the
code". That's a small indirection that I suppose is bearable.

  It's not integrated with the workflow. And sorry, but git@vger (or
workse lkml@vger).. it does not work. Maybe for git it does because the
flow is still human manageable, and that it seems junio has enough time
for it. But for the kernel ? please. You should read Bastian Blank
frustration about regressions and nobody tracking them. Know why ?
because there is no tool that is well known and well integrated in the
workflow. There is a long rant against kernel.org's bugzilla, you should
read it as well. It's not really instructive (at least there wasn't
anything new for me in it, I was already bought to many arguments in
there). But you'll see the world isn't as rosa-lila you think it is.

On Sun, Jun 10, 2007 at 08:38:03PM +1200, Martin Langhoff wrote:

  I'm sure I could come up with something really better in say a
month... If I hadn't paid work to do too :)

  It's a misfeature for you because you read it as "non portable".
That's a fair point. And like said, it may be extended to SCM's with the
same set of features I need to build it. But let's be honnest, I don't
care about a BTS that uses the smallest common set of features of SCM's.
Reading this list, you should know it's almost a void set. No, I think a
good BTS should make an excellent use of high level features of the SCM.
The real problem is, there is maybe 2 or 3 SCMs that have this set of
strong and good features. Too bad for the other, I can't work with them,
they suck hard, and I don't see why I should support bad practices
anyways.

  (Yes I'm also a guy with strong opinions too, it's not really
restricted to Linus ;p)

  If you do that you loose:
  * fastness, and I don't want to work with debbugs anymore.

  * distribution: Meaning that for _one_ project everybody needs to use
    this central bugtracker. Whereas .oO(kernel) there is some projects
    where the subcomponents are dealt with from very different teams,
    very different way to release things, and they are interested with
    their bugs, and their bugs only. They would prefer a very fast
    interface to deal with their 1k bugs, and not worry about the 100k
    the rest of the project has.
    Branching bugs also makes sense you know ?

  You are the one saying it. I beg to differ.

  IMHO a BTS is a _low_ level tool. that's the road git took, I
sometimes describe the plumbing git commands as the "Assembler" of the
SCM world to friends when I talk to them about git. That's really the
best way to implement a thing: have a small small set of rock solid,
well designed tools, and build the others as porcelains with them.

  Testing is a high level tool. I don't need to support them, I need to
have exactly the low level querying rocket-fast query interfaces so that
integration scripts are at most 100 lines long.

  Honestly ? No, because that would be horribly slow (but I'd love to be
proven wrong).

Cheers,

On Sun, Jun 10, 2007 at 06:59:13PM +1200, Martin Langhoff wrote:

  Sure, what if I like it anyway ?

  Sure, you can do that and still inherit from the many downsides of
those tools: central, needs another separate tool to work with, and a
tool that nowadays tends to eat 500Mb of your memory if you take the
most hype implementation of it (Yes I'm describing a web browser), that
is slow like hell (because yes, there is many clients at the same time
on the central server, that is overloaded like hell), and so on.

  You can like central web UIs, your choice. And I suppose if grit works
well, there will be one at some point.

  yeah, that is what bugtrackers already do. Though, that's of no use
for release managers. What is useful is what people call "milestones" in
trac e.g., meaning a set of bugs that has been flagged as needing to be
fixed. And what a release manager need is a tool to deal with this set
of bugs as a whole.

  That's the same argument that Linus has against per-file tracking.
Also atm when you e.g. backport a patch that fixes this or this bug,
you're no BTS helps you tagging the bug as fixed in that branch as well.
Not to mention that BTS I know do not make any use of the commits DAG.
And for a very good reason, there is no real DAG in many existing tools
(svn, cvs, ...).

  that's not up to the BTS tool to do that, it's way to high level. It's
up to the importing filters/hooks that will parse the associated ML, and
that would translate that to useful low level BTS commands.

  No it should not, because it can't. I want the distributed and
Bug status spanning-DAGS be a central feature. So that means that this
tool can only work on top of SCMs that support that. ttbomk git, hg,
_maybe_ bzr fit the description. I only know the former, but I really
plan to write the tool in a way that the underlying SCM does not matters
_too_ much. Maybe I'll fail. Honestly, I don't really care (yet ?).

  You're a lucky guy. All bug trackers I've used suck a way or another,
that impedes my workflow a lot. Let's cite a couple:
  * bugzilla: takes more from the -zilla than from the BTS side. It's
    huge, monstruous, slow (have ever used glibc's bugzilla ? it has
    maybe 5k bugs, it's slow like if it ran on an Atari ST), complex,
    the mail gateway suck hard, it's completely unusable for me. Believe
    me, I've packaged KDE for 2 years in Debian, now am in the glibc
    team. Every single day I have to work on this horrible tool is a
    PITA.

  * flyspray: I've been upstream for a short time. Visually nice, good
    to work with, UI is great. Integration totally suck, worthless.
    Can't use mails either, needs a Web Browser -> useless. The same
    holds for mantis, roundup and a lot of other friends.

  * debbugs: oh yeah there is a mail interface. So slow that you have to
    wait up to 15 minutes to see your command be taken into account. And
    when you have to deal with dozens of bugs (Yes I've done that on a
    regular basis) you _have_ sometimes to wait for the answer to come
    back (because you need an ID that will be in there e.g.) to continue
    your work. That is unacceptable, you pass most of your time waiting.
    Moreover sometimes you made an error in your commands, so you also
    need to parse the anwer ... one day later because "immediateley when
    you still remembered what this was about" is not an option.
    Unacceptable again. The plus: it uses mboxes, hence is worthwile in
    a hacking environment, it fits the workflow well.

  * trac: very very very nice tool. I mean it. We use it at work, where
    I have to suffer svn (through git-svn though). It's really nice (did
    I repeat myself ?). THough, it's on top of svn, and you can't use
    the Bugs informations from your repository. You can't say: I'm
    backporting that patch into that branch. Now what affects this
    branch please ? Trac can't answer that (and ttbomk now BTS really
    can anyway, except Debian's debbugs instance, and it's somehow
    limited). That is a question a release manager takes 80% of his time
    to ask. I hope grit can take back to the 0.01% of his time, which is
    still too much.

On Sun, Jun 10, 2007 at 08:55:21PM +1200, Martin Langhoff wrote:

  I trol^Wdiscuss everyday on debian's channel with friends that tell
that svn is the best tool ever, and that they would never ever use a
distributed SCM because it's too hard to understand. Your call.

  That's exactly why distributed rock. Because everyone could _really_
own _his_ bugtracker. This solves the same range of ACLish issues git
solves with respect to the code.

  Why wouldn't it exist a "public reporting interface"-branch ? that
would be the same purpose as the mainline ~linus/linux2.6.git tree. And
you can build/instanciate your beloved web UI on top of that, and people
would just have to pull from there. What a shock, it's easy !

  Oh, because the code never changes. Doh, how stupid I am :)
  No, really, you name your bug after the sha hash of the first report,
I think that's pretty obvious. That gives you a bug name. Then you ask
git for "what's the current sha for this bug in the tip of the BTS
branch", then you ask "so now what this new sha is pointing to in the
code". That's a small indirection that I suppose is bearable.

  It's not integrated with the workflow. And sorry, but git@vger (or
workse lkml@vger).. it does not work. Maybe for git it does because the
flow is still human manageable, and that it seems junio has enough time
for it. But for the kernel ? please. You should read Bastian Blank
frustration about regressions and nobody tracking them. Know why ?
because there is no tool that is well known and well integrated in the
workflow. There is a long rant against kernel.org's bugzilla, you should
read it as well. It's not really instructive (at least there wasn't
anything new for me in it, I was already bought to many arguments in
there). But you'll see the world isn't as rosa-lila you think it is.

On Sun, Jun 10, 2007 at 08:38:03PM +1200, Martin Langhoff wrote:

  I'm sure I could come up with something really better in say a
month... If I hadn't paid work to do too :)

  It's a misfeature for you because you read it as "non portable".
That's a fair point. And like said, it may be extended to SCM's with the
same set of features I need to build it. But let's be honnest, I don't
care about a BTS that uses the smallest common set of features of SCM's.
Reading this list, you should know it's almost a void set. No, I think a
good BTS should make an excellent use of high level features of the SCM.
The real problem is, there is maybe 2 or 3 SCMs that have this set of
strong and good features. Too bad for the other, I can't work with them,
they suck hard, and I don't see why I should support bad practices
anyways.

  (Yes I'm also a guy with strong opinions too, it's not really
restricted to Linus ;p)

  If you do that you loose:
  * fastness, and I don't want to work with debbugs anymore.

  * distribution: Meaning that for _one_ project everybody needs to use
    this central bugtracker. Whereas .oO(kernel) there is some projects
    where the subcomponents are dealt with from very different teams,
    very different way to release things, and they are interested with
    their bugs, and their bugs only. They would prefer a very fast
    interface to deal with their 1k bugs, and not worry about the 100k
    the rest of the project has.
    Branching bugs also makes sense you know ?

  You are the one saying it. I beg to differ.

  IMHO a BTS is a _low_ level tool. that's the road git took, I
sometimes describe the plumbing git commands as the "Assembler" of the
SCM world to friends when I talk to them about git. That's really the
best way to implement a thing: have a small small set of rock solid,
well designed tools, and build the others as porcelains with them.

  Testing is a high level tool. I don't need to support them, I need to
have exactly the low level querying rocket-fast query interfaces so that
integration scripts are at most 100 lines long.

  Honestly ? No, because that would be horribly slow (but I'd love to be
proved wrong).

Cheers,

Pierre Habouzit wrote:

One downside of using maildir (which I agree has several desirable properties), is that the filenames include characters that are illegal on various platforms (specifically ":" in Windows). This is quite unfortunate.

I think it would be very neat to be able to use the built-in mechanisms in mail clients for threading, etc, and handling attachments, sorting and filtering. Tracking things like who a bug is assigned to wouldn't work too well, but I imagine that many kernel developers have fine tuned their email clients to such a degree that it wouldn't be impossible :-)

I'm not familiar with how messages move around in {cur,new,tmp}, but I suspect that if we simply make a convention that all mails are created as "read", then we won't have clashes between developers holding some bugs as unread, etc while others have already read them.

Maybe.

Yes.

Yes, hooks might be the right approach for this.

Agreed.

Rogan