Pain points in Git's patch flow

Hi,

I'd like to introduce Raxel (cc-ed), who is starting an internship this June with the Git team at Google.

He'll be working on a bit of an experimental project: we want to take Patchwork[1], which in principle can be a helpful addition to a mailing list centric workflow[2], and improve it to be something that people in the Git open source project get day-to-day benefit from. Raxel's previous successes in making changes to tools to support a better user experience make me excited for the potential for this work.

Anyway, yesterday[3] Junio, Taylor, and Emily were discussing how to encourage more reviews:

 <gitster> this week, i'd be thinking about ways to get topics, that
           are not reviewed sufficiently, reviewed. I can act as the
           last-resort fallback reviewer, but that's not sufficient.
 <ttaylorr> gitster: I share your concern.
 <nasamuffin> gitster: yep, agree, on both counts

That reminded me that it would be useful preparation to collect descriptions of pain points we are having with our existing patch flow. For example:

- As a reviewer, I want to be able to easily find a series that needs
  review.  Using patchwork, I can see some recent patch series; or
  using a hierarchical threaded mail reader, I can find a neglected
  thread or one that seems to be likely to have an interesting
  discussion going on.  But without reading in detail, there is no
  easy way to see whether the series has reached a review, whether
  someone else intends to review it, and what the author believes its
  status to be.

- Relatedly, as a patch author or reviewer, I want to be able to
  easily tell whether a topic has been sufficiently reviewed.  Today,
  the signals for this are implicit: I have to judge consensus, or to
  check the Git repository for whether the patch has been merged, or
  to check the maintainer's latest "What's cooking in git.git"
  message.

- As a potential reviewer or interested user, I want to be able to
  follow all relevant discussion for a patch series, while also
  having the ability to stop following it if the discussion goes on
  too long and starts overwhelming my email inbox.  Today, I can join
  the discussion and then (1) it is hit-or-miss whether the patch
  author ccs me on later iterations of the patch and (2) there is no
  easy way without aggressive email filtering to stop watching it if
  I am cc-ed.

- After having diagnosed an issue to be due to a patch, I want to be
  able to easily find all relevant review discussion.  Today I can
  use the mailing list archive[4] or patchwork to find review
  discussion on the latest version of the series that patch was in,
  but tracing back to previous iterations of that same series can be
  non-trivial.  Moreover, if I'm interested in a particular puzzling
  line of code, finding which iteration introduced it can take a long
  time.

Those four are important in my everyday life.  Questions:

 1. What pain points in the patch flow for git.git are important to
    you?

 2. What tricks do you use to get by with those existing pain points?

 3. Do you think patchwork goes in a direction that is likely to help
    with these?

 4. What other tools would you like to see that could help?

Thanks, Jonathan

[1] http://jk.ozlabs.org/projects/patchwork/; you can see an instance for Git at https://patchwork.kernel.org/project/git/list/ [2] https://kernel-recipes.org/en/2016/talks/patches-carved-into-stone-tablets/, https://www.kernel.org/pub/software/scm/git/docs/gitworkflows.html#_patch_workflow [3] https://colabti.org/irclogger/irclogger_log/git-devel?date=2021-04-12#l40 [4] https://lore.kernel.org/git/

On 14/04/21 13.13, Jonathan Nieder wrote:

There is no lists of "beginner-friendly" issues that can be worked on by new contributors. They had to search this ML archive for bug report issues and determine themselves which are beginner-friendly.

No, unrelated to beginner-friendly issues above.

Some sort of bug tracker systems like Bugzilla (used by Linux kernel and many other projects) and Debbugs [1] (which is mail-centric and used by Debian for its BTS).

[1]: https://bugs.debian.org/debbugs-source/

Bagas Sanjaya <bagasdotme@gmail.com> writes:

Yeah, looking for "#leftoverbits" or "low-hanging" on the list archive is often cited as a way, and it does seem easy enough to do. You go to https://lore.kernel.org/git/, type "leftoverbits" or "low-hanging" in the text input and press SEARCH.

But that is only half of the story.

Anybody can throw random ideas and label them "#leftoverbits" or "low-hanging fruit", but some of these ideas might turn out to be ill-conceived or outright nonsense. Limiting search to the utterances by those with known good taste does help, but as a newbie, you do not know who these people with good taste are.

It might help to have a curated list of starter tasks, but I suspect that they tend to get depleted rather quickly---by definition the ones on the list are easy to do and there is nothing to stop an eager newbie from eating all of them in one sitting X-(.

So, I dunno. We seem to suffer from the same lack of good starter tasks before each GSoC begins.

Junio C Hamano <gitster@pobox.com> writes:

And long after sending the response, I realize that this has very little to do with "Git's patch flow" issue that you wanted to discuss. Helping newbies wet their toes may be a topic worth discussing, but that is not the focus of this thread.

Let me send a response to describe my pain points separately. Sorry for the noise.

Hi Bagas,

On Wed, Apr 14, 2021 at 02:22:51PM +0700, Bagas Sanjaya wrote:

An unofficial and semi-curated list of issues exists at [0]. I've seen many people new to git development pick up some beginner-friendly issues from there.

-Denton

[0]: https://github.com/gitgitgadget/git/issues

Jonathan Nieder <jrnieder@gmail.com> writes:

As the maintainer, I want to ensure that what I queue on 'seen' is kept reasonably up to date. Not picking up the latest every time a topic is rerolled is OK, but before declaring the topic will hit 'next' in a few days, it must be (1) the latest and (2) the greatest (meaning: what reviewers are happy with).

This requires a few features from any tracking system. It must be able to:

 - tell which round of patches are in 'seen'.

 - tell which e-mail messages are the newer round than what is
   queued, and which ones are the latest round.

 - tell which patch have been commented on, and been updated in
   response.

 - tell which patch have been positively accepted with an Ack or a
   Reviewed-by, and if a patch in a newer round is identical to an
   already accepted one (in which case, reviewers would not bother
   to send "this step still looks good to me").

The system I use for the first two points is to rely on the list archive and the mapping from each individual commit made out of a patch on the list (implemented as git notes, that records a blob with the Message-Id for each commit [*1*]). So

   $ git log --notes=amlog --no-merges --oneline master..$topic

would give a list of commits with the original Message-ID, and I can see which piece of e-mail each commit came from.

The third and fourth are maintained mostly manual, with me keeping notes in the draft of "What's cooking" report (which I send out from time to time). Having to notice, pick up and squeeze in Acked-by's and Reviewed-by's is quite painful and cumbersome, especially for a long series, and the buggy "git rebase -x" does not help, either [*2*].

If we run a patchwork instance for our project, the first two could be largely automated. Automation built around Patchwork should be able to, or at least should be able to help me to:

 - notice when a new round of an existing topic is posted.

 - fetch the "amlog" notes, together with a copy of daily 'seen', to
   see if a topic that is queued has an update, and notify me and
   others when the topic queued is stale [*3*].

 - tie a step in the latest round with a corresponding step in the
   previous round, and show Ack's and Reviewed-By's that are still
   valid [*4*].

[Footnotes]

*1* "git fetch https://github.com/gitster/git +refs/notes/amlog:refs/notes/amlog"
    should give you a copy of this database.  Then, for example you
    can ask where a commit came from:

    $ git show -s --notes=amlog format="%N%s" 61a7660516
    Message-Id: <pull.1001.git.git.1618254757074.gitgitgadget@gmail.com>
    reftable: document an alternate cleanup method on Windows

    Note that this is not a one-to-one mapping.  I may initially
    apply patches to an inappropriate base and push the integration
    result out that has it in 'seen', but I may realize that the
    series needs to be queued on a different commit and rebase the
    topic the next day.  Both commits before and after such a
    rebasing have come from the single piece of e-mail, so you can
    say "this commit came from this message", but it is impossible
    to expect a single answer to "which commit is the result of this
    message"---there will be multiple.

    Strictly speaking, when two rounds of the same topic had patches
    that were unchanged between the iterations in their earliest
    parts, two pieces of e-mail may convey the same patch, so in the
    ideal world, it might be more useful to record "this commit came
    from this and that messges, both of which record an identical
    patch".  I currently do not do so, though.

*2* It would be ideal if "rebase -i -x 'add-trailer -r peff@'" can
    be used to stop at each commit, run the 'add-trailer -r peff@'
    script that amends HEAD to add "Reviewed-by: peff@", and
    continue, while honoring the "notes.rewriteref" configuration
    variable (in my repository, set to "refs/notes/amlog").  That
    way, I can queue with "git am", at which time "amlog" gets
    populated to map each commit to the original message, find
    Reviewed-by: and do the above rebase, while carrying the message
    IDs to resulting commits.  Alas, "rebase -i -x" is buggy and
    loses the notes during this process (doing s/pick/reword/ and
    manually squeezing Reviewed-by: into the log message is a poor
    but workable workaround).
    cf. https://lore.kernel.org/git/xmqq8s6tcuxc.fsf@gitster.g/

*3* It is perfectly normal if a topic is left stale, if the newer
    iteration breaks integration.  So the stale notification going
    directly to a contributor from an automated system would not
    help very much, but it needs to come with the reason why it is
    kept out of 'seen', which must be supplied by human, not by an
    automation.

    If an automation around Patchwork can send, instead of "hey,
    here is an updated series available" notification, a mbox
    readily usable by "git am -s3c" to me, with Acked-by's and
    Reviewed-by's already incorporated, that might be ideal (these
    trailers may have to be filtered/curated to avoid spams,
    though).
 

*4* Judging a step in the latest round and the corresponding step in
    the previous round has not substantially changed may not be
    easily automatable, and carrying Ack's and Reviewed-by's forward
    would require human curator of the "patchwork" database.

Hi there,

I'm not a regular contributor but I have started to subscribe to the Git's Mailing List recently. So I thought it might be worth sharing my personal view on this.

After writting all the below, I do realize that I have written quite a rant, some of which I think some might consider to be off topic. For that, I do want to appologize before hand.

 Tue, Apr 13, 2021 at 11:13:26PM -0700, Jonathan Nieder wrote:

...

There are several points I want to highlight:

1. Issue about reading the Mailing List:

- Subscribing to Git's Mailing List is not trivial:
  It takes a lot of time to setup the email subscription.  I remember
  having to google through a few documents to get my subscription
  working.

- And even after having subscribed, I was bombarded with a set
  of spam emails that was sent to the mailing list address.  These spams
  range anywhere from absurd to disguising themselves as legitimate
  users trying to contact you about a new shiny tech product.

2. Issue about joining the conversation in the Maling List:

- Setting up email client to reply to the Mailing List was definitely
  not trivial.  It's not trivial to send a reply without subscribing to
  the ML(i.e. using a Header provided from one of the archive).
  The list does not accept HTML emails, which many clients
  use as default format.  Getting the formatting to work for line
  wrapping is also a challenge depends on the client that you use.

- It's a bit intimidating to ask 'trivial questions' about the patch and
  create 'noise' in the ML.

3. Isssue with archive:

- I don't find the ML archive trivial for new comers.  It took me a bit
  of time to realize: 'Oh if I scroll to bottom and find the "Thread 
  overview" then I can navigate a mailing thread a lot easier'.

- The lack of labeling / categorization that I can filter while browsing
  through the archive make the 'browse' experience to be quite
  unpleasant.  Search is one way to do it, but a new comers would not be
  knowledgable enough to craft search query to get the archive view just
  right.  Perhaps a way to provide a curate set of categories would be
  nice.

- Lost track of issues / discussion:
  A quick example would be me searching for Git's zstd support
  recently with 

  > https://lore.kernel.org/git/?q=zstandard 

  and got next to no relevant result.  However if I were to query

  > 'https://lore.kernel.org/git/?q=zstd'

  then a very relevant thread from Peff appeared.  I think this could be
  avoided if the search in ML archive do more than just matching exact
  text.

4. Lack of way to run test suite / CI:

  It would be nice if we can discuss patches while having CI result as
  part of the conversation.  Right now mostly I see that we have to
  manually running benchmarks/tests and share the paste the results.

  But for folks who don't have a dev environment ready at hand (new
  comers, during travel with only phone access), it would be nice to
  have a way to run tests without a dev environment.

  This was mostly solved in the context of works spent on Github's
  Action Workflow.  But if we are discussing about pure patch flow, this
  is a miss.

For (1):
- I had to invested a lot of time into setting up a set of Gmail search
  filter.  Move mails with topics that Im interested in into a special
  tag while the rest into archive.  Regularly check if anything
  interesting went to archive by accident.

For (2):
- I had to setup Mutt + Tmux to have a compatible experience sending
  replies like this one.

- All the patches I have submitted were through
  > https://github.com/gitgitgadget/git/pulls
  and it was not directly trivial to get permission to send email from a
  PR.

For (3):
- Spending time reading git blame / git log / commit message helps
  identifying the keywords I need to refine my search result in the ML
  archive.  This requires some commitments and is a barrier to entry for
  new comers.

- Using service like Github Search or SourceGraph helped a lot in term
  of navigating through the commit message / git blame.

For (4):
- I leverage both Github action and a patch that added Gitlab CI to run
  the test suite.

With all that said, I don't know if patchwork will solve the problems above. I do understand that the current patch workflow comes with a certain set of advantages, and adopting another tool will most likely be a trade-off.

Personally I have been spending more and more time reading through git.git via Sourcegraph Web UI and I would love for the search feature to be able to extend to be able to search in the Mailing List from relevant commit if possible. I have also tried both Github's Codespace and Microsoft's DevContainer to setup an opionated IDE with predefined tasks that help executing the test suite. I think these tools (or their competitors such as GitPod) are quite ideal to quickly onboard new contributors onto a history-rich codebase such as git.git.

Perhaps some configure a set of sane default, including editor extensions that would handle email config for first time users.

As for code review and issue tracking toolings, I don't think there are a perfect solution. Any solutions: Github PR, Gitlab MR, Gerrit, Phabricator would come with their own set of tradeoffs. I like the prospect of PatchWork gona improve the patch workflow though. Perhaps I will give it a try.

Thanks, Son Luong.

Son Luong Ngoc <sluongng@gmail.com> wrote:

Thanks for the feedback, some points below.

Note that subscription is totally optional.

Gmail's mail filters probably aren't very good, perhaps SpamAssassin or similar filters can be added locally to improve things for you.

Spam filtering is a complex topic and Google's monopolistic power probably doesn't inspire them to do better.

The spam (and phishing) problem would be worse if HTML mail were accepted. Obfuscation/misdirection techniques used by spammers and phishers aren't available in plain-text.

It's also more expensive to filter + archive HTML mail due to decoding and size overheads, which makes it more expensive for others to mirror/fork things.

I'm sorry you feel that way. I understand the Internet and its persistence (especially with mail archives :x) can have a chilling effect on people. I think the way to balance things is to allow/encourage anonymity or pseudonyms, but some folks here might disagree with me for copyright reasons. OTOH, don't ask, don't tell :)

(I am not speaking as a representative of the git project)

(I'm the maintainer of public-inbox, the archival software you seem to be referring to).

I'm not sure how to make "Thread overview" easier to find without cluttering the display near the top. Maybe I'll try aria labels in the Subject: link...

Perhaps TODO files/comments in the source tree are acceptable; or a regularly-posted mail similar to "What's cooking".

Having a centralized website/tracker would give too much power and influence to people/orgs who run the site. It would like either require network access or require learning more software to synchronize.

I'm planning to support Xapian synonyms for that, but haven't gotten around to making it configurable+reproducible by admins. Everything in public-inbox is designed to be reproducible+forkable.

Fwiw, the GCC Farm project gives ssh accounts for all free software contributors, not just gcc hackers: https://cfarm.tetaneutral.net Perhaps there's other similar services, too.

Slow down and enjoy travel :) There's very little in free software urgent enough to require constant attention. Email is well-suited for asynchronous work, and nobody should expect instant replies. The always-on nature of the modern Internet and smartphones increases stress and dangerous situations; so I hope free software hackers aren't contributing to that.

Fwiw, git-send-email works for non-patch mails, too. I don't want a monoculture around mutt or any particular clients, either. (I've never used tmux and don't see why it's necessary, here).

Anyways, thanks again for the feedback.

On Mon, Apr 19, 2021 at 02:57:54AM +0000, Eric Wong wrote:

FWIW, I find the disconnected, e-mail based workflow using a command-line interface to be *ideal* for working while travelling on an airplane. I'm mostly disconnected from the internet, because the airplane wifi is so slow that you *really* don't want to use a web-based interface, but I can use offlineimap to sync my e-mail onto my laptop, and using the command-line interface and the lack of distractions is great since you really can't surf the web on the gogoonline's pathetically slow 'net access.

This also means I have an excuse to work on open source projects which are using e-mail and off-line git, as opposed to $WORK which mandates the use of gerrit. :-)

(All of the above applies pre-pandemic, of course. I've been working from home and not travelling for the past year+, sigh.)

     	      	  	     	     	  - Ted

P.S. Also, while working on the road, I find that web-based interfaces are much more tolerable when I'm at my desk with a 40" screen. When I'm using a 13" laptop screen, I much prefer CLI interfaces. YMMV, of course.

On Mon, Apr 19 2021, Eric Wong wrote:

I'd say the bare-bones style of it is probably jarring to most users today. I had to check if the site even had any CSS at all.

I.e. I think a more intuitive UI to users today would probably be some collapsible side-bar on the left of the screen, which would have a threaded view. The "Archives are clonable" would probably belong in some "help" tab in such a UI.

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

The plan is to support read-only JMAP, so it's a stable API that users can build their own displays on top of (of course, NNTP and IMAP support already exists).

I can't make drastic UI changes such as a sidebar without breaking things for users who like the current UI. I only know about GNOME3 and Digg because they made drastic UI changes that angered their existing userbase.

The current UI is designed to for a terminal with w3m|lynx since it's the lowest common denominator. Graphics drivers/stacks seem to be most frequently broken thing on GNU/Linux systems, so it's important users can find patches/configs/help easily with a text-only browser in order to get graphics working.

Eric Wong <e@80x24.org> wrote:

I think I made [thread overview] easier-to-find without adding more clutter:

	https://public-inbox.org/meta/20210428065522.12795-1-e@80x24.org/

Not sure about title attribute or aria labels (my version of w3m doesn't support that, yet). Anyways, an an example of it deployed:

	https://public-inbox.org/git/20210419025754.GA26065@dcvr/

On 14-Apr-2021, at 11:43, Jonathan Nieder <jrnieder@gmail.com> wrote:

This is a great initiative!

While I do not have anything new to add in terms of pain points, I just wanted to let you know that this is definitely something that would have eased the process of bringing in a new contributor like me.

As a new contributor (and also someone new to the patch flow) I would have especially liked the second and fourth point addressed. When I was preparing my GSoC proposal, I wanted to gather the status of a previous contributor's work and even though searching the mailing list helped, it was hard to immediately know what were the status of the patches, and which changes got introduced in which version of the patch series.

Also with my first patch series that I sent to the mailing list, I initially felt unsure about what the status of my patch was after a few people discussed over it. The 'implicit signals' is something that was not immediately obvious to me, and only after reading other interactions in the mailing list did I start getting a hold of how I should interpret the responses, and what my next action should be.

In order to learn about a previous patch series and what was added, I used git blame on the relevant part of the codebase, and tried to search the commit message in the mailing list archive. From there on it was just opening a ton of tabs in order to see how the patches developed over time.

The limitation with this trick is it will work only if the patch actually landed in the codebase. A part of building my proposal required me to read a patch that did not get merged, and I had to just aggressively search the mailing list and hope I managed to catch everything I wanted.

I have noticed that a patchwork instance for this mailing list already exists[1] so I decided to try it out. It definitely addresses the problem of explicitly identifying the status of a patch. I also liked that I could search for the previous contributor that I spoke of and sort his contributions by date. If I knew this existed, I would have saved a lot of time.

But as you also mentioned, it does not yet help me locate an older version of a particular patch, and let me observe how it developed over time. So that would definitely be a welcome addition.

As Bagas mentioned in the thread, it seems to lend itself well to identify beginner-friendly tasks. I did not personally have too much difficulty with those thanks to the GitHub issue tracker and the Git documentation, but if new contributors are anyway going to refer to patchwork to study previous patches and learn from it, it might be helpful to keep beginner issues accessible there as well. Even a generic labelling system to help categorise issues will do (a downside being that the labels will have to be maintained and managed too).

Some small nits:
- The searching capability was not super obvious to me. My
  eyes naturally scan for a search box or search icon, it
  took me a few minutes of fiddling to realise that
  'show patches with' is a link that opens all the search
  filters.
- It would be nice (though probably out of scope) to allow
  me to do a code search in the patches.

(...I don't really have an opinion on this)

[1] https://patchwork.kernel.org/project/git/list/

Jonathan Nieder <jrnieder@gmail.com> writes:

So here is a real-life example.

Let's say somebody is looking at a "gentle ping" [*1*]

znh> The patch seems to have fallen into the crack. zhn> Jeff and Junio, willing to help?

How would we figure out what happened to the patch today without visiting patchwork would be:

 1. Visit the message at lore.kernel.org/git/ [*1*]

 2. Notice that it is a response to a message, and click the link to
    be taken to [*2*]

 3. Notice that nobody commented on the patch.

 4. Type "f:zhening ref-filter" to the search box and search, with
    suspicion that this was an updated version of something.

 5. Click one of them in the result [*3*]

 6. This time, we can tell that this seemed to have had two earlier
    iterations, and after reading the discussion through, the last
    one changed the course in a major way.  Not just a new helper
    introduced in the earlier rounds has gone away, but an existing
    helper got removed.

 7. All comments in the discussion for the earlier two rounds can be
    read as supporting the new direction the latest round takes.

 8. The fact remains that even if the direction has been endorsed
    (see 7. above) nobody took a look at the implementation for the
    latest round.

 9. Make the final verdict.

I use my newsreader to do pretty much the equivalent of the above without hitting https://lore.kernel.org/git/ but the above is written to use the web interface, in order to make it reproducible more easily by anybody on the list.

Now, how can patchwork improve the above reviewer experience, out of the box and possibly with new helpe rools around it?

I can see #3 would immediately become obvious, and I hope #4-#5 would become unnecessary.

Anything else?

At steps #6 and #7, there is human judgment involved that may not be automatable, but would there be some mechanism to make it easy to help these steps if the user visits patchwork (instead of staying in my newsreader or web interface to the lore archive)?

I am of course not expecting to automate step #9 ;-) It would be nice though.

Thanks.

[References]

*1* https://lore.kernel.org/git/CAOLTT8Tis5Yjg8UR0c-i0BnqiFQvLXvDgxUQJ-WcP6jjQPu9cQ@mail.gmail.com/

*2* https://lore.kernel.org/git/pull.928.git.1617975348494.gitgitgadget@gmail.com/

*3* https://lore.kernel.org/git/pull.927.v2.git.1617809209164.gitgitgadget@gmail.com/

Junio C Hamano <gitster@pobox.com> writes:

Also, it would be ideal if it is made easy for willing reviewers with excess bandwidth to preemptively find and review patches that need reviewing. I think your original write-up upthread covered this use case sufficiently.

Thanks.

Haaaa, everyone, I am this example :) Sorry I haven't checked these mails cc to me for a long time.

Junio C Hamano <gitster@pobox.com> 于2021年4月17日周六 上午3:50写道：

Here are my thoughts:

For the reviewers like Junio, after missing a new patch iteration, need to
review the past history to find the correct patch and related comments
from other reviewers. Just like I once read a github blog saying that
"patch" is also a special object in git. I would like to have a "new" tool
 which can link multiple related patches and comments.

1. Coder need Reviewers' help.
2. This new tool will obtained multiple different patches contents automatically
or coder provided those pathes versions links.
3. This tool will analyze the differences between multiple patches
versions, get all
the reviewers comments and coder comments related to the "patch stream",
organize it into "patch graph".
4. The tool will notify the reviewer(by email or something else) and
show the links
and patch graph or patch range-diff. It can visualize the entire patch process,
 It’s best that comments from different people can be displayed on one page.

In order to be more accurate, I made a picture [*1*].

Using this new tool, reviewers can choose to see or not see the range-diff and diff in multiple different patch versions, Instead of the range-diff automatically sent by GGG. When my second patch processing was greatly changed from the previous one, I have to rebuild a new branch and create a new PR, this is my pain point.

Thanks! -- ZheNing Hu

*1* https://github.com/adlternative/git/blob/pic/git-patch-pain-point-solve-idea.png

On 2021-04-14 08:13, Jonathan Nieder wrote:

Thanks for bringing up these questions in a dedicated format. I'll take this as an opportunity to share my thoughts on this topic, which have accompanied me for quite a while.

Well, it's email-based. As a result it's error prone to things like formatting / quoting issues, putting the right people it CC, etc.

I have always wondered why Git core development does not start to make use of the Git ecosystem that we have by now, esp. in the form of review tools / platforms like GitHub (via pull-requests), GitLab (via merge-requests), or Gerrit (via patches). From these, Gerrit would IMO be the best fit for Git, due to its capability to cope well with rebase-workflows. Those tools avoid things like formatting / quoting issues completely, and shift the responsibility of assigning reviewers from the contributor to the tool, where people can subscribe to code changes or code ownership can be defined and automatically taken into account.

Sure, I get that that the contribution workflow to Git core has historically grown, but what concerns me is that the efforts to "bridge" the contribution workflow to the "modern world" seem to go into the wrong direction: Tools like submitgit [1], gitgitgadget [2] and now patchwork [3] were created / are considered for use to allow the legacy email path workflow to remain, but also allow more "GUI minded" people to contribute. While this has worked quite well for some time, and esp. gitgitgadget [2] seems to haven gotten popular, I wonder whether it's now the time to "swap the default", and make a patch / contribution tool with a GUI the standard, and bridge the legacy workflow by using / creating tooling that makes it convenient to use those modern tools from the CLI, instead of the opposite.

None. I simply have stopped contributing to Git core, to be frank.

No. To me, this is yet another effort that tries to come up with a work-around instead of fixing the root cause: It tries to lift the limitations of an email-based contribution workflow instead of getting rid of the email-based contribution workflow altogether.

Currently, only Gerrit [4] comes to my mind, as a complete substitute for the email-based contribution workflow.

[1] https://github.com/rtyley/submitgit [2] https://github.com/gitgitgadget/gitgitgadget [3] http://jk.ozlabs.org/projects/patchwork [4] https://www.gerritcodereview.com

On Sun, Apr 18 2021, Sebastian Schuberth wrote:

And thank you for participating in the discussion. I think it's especially valuable to get a viewpoint like yours, i.e. someone who (per this E-Mail below) gave up in frustration with the current development flow.

The below isn't meant as a retort, but to hopefully clarify things a bit.

I think it's important not to conflate tooling issues with social issues. It's not that we e.g. couldn't whip up a quick script to round-robin randomly assign reviewers on the basis of topics Junio has picked up, which is basically the function of some of these "open a MR end get on the review train" tools.

Rather it's that it's a volunteer project and people work on what they're interested in.

So maybe having assigned reviewers would help move things along. But I wonder if it wouldn't also lead down the rut of PRs/MRs languishing for months, because the reviewers just want to spend their time in some other way.

I.e. the design of many of these tools in this regard assumes you have a workforce, not the cat-herding problem of volunteers working on whatever strikes their fancy.

I think characterizing E-Mail as a "legacy" workflow isn't accurate. All of these proposed alternatives involve moving away from something that's a distributed system today (E-Mail infrastructure, local clients), to what's essentially some website run by a centralized entity, in some cases proprietary.

Even in cases where the tool itself isn't proprietary (e.g. GitLab instead of GitHub) using GitHub/GitLab/Gerrit/Atlassian Bitbucket etc. means having some centralized infrastructure somewhere holding a bunch of data only the operator of that infrastructure can realistically access.

So really basic things that are comparatively trivial with E-Mail (e.g. "I think the search sucks, try another client") run up against a brick wall with those tools.

And to e.g. as one good example to use (as is the common convention on this list) git-range-diff to display a diff to the "last rebased revision" would mean some long feature cycle in those tools, if they're even interested in implementing such a thing at all.

Because we're E-Mail based that's just something some people started using (well, it was called "tbdiff" then), some others picked it up etc.

Which is not to say that one can't argue that on balance using those tools isn't better overall, I'm just responding to the characterization of E-Mail based development as "legacy", or something those tools supersede. I think it's better to think of them as orthagonal ways to reach similar aims.

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Agreed

Some further clarifications on my part below.

<snip some of Ævar's excellent clarifications>

Thanks all for bringing this up. I should add the mail archives at lore.kernel.org are backed by public-inbox, thus all mail and code are completely reproducible by anyone. It even targets old, slow, legacy hardware and tries to minimize bandwidth to benefit the economically-disadvantaged.

Forking is the checks-and-balances system of free software to prevent any central entity from becoming too powerful (remember: power corrupts). DVCS (e.g. git) makes forking easier, public-inbox uses git to make text communications history reproducible (and therefore, forkable). My end goal is completely forkable communities without any central arbiters.

Email has problems, of course. Big players are constantly introducing more complexity to squeeze out smaller players. And most mail servers require DNS through ICANN, an organization that tried to extort .org users and hence likely to attempt further abuses of power in the future. Again, power corrupts.

I'm working on some local tooling based on public-inbox ;) (of course, totally optional)

And public-inbox will support JMAP in coming months, so it'll be a standardized API and hopefully compatible with a wider variety of clients and frontends. This should help users who prefer other other layouts.

Anyways, I do what I can to keep hardware and bandwidth requirements low for folks who: a) can't afford to keep up with Moore's law b) won't accept mystery firmware blobs in modern HW I got into free software because HW was constantly obsoleted by proprietary software; and I'm sad so much "modern" free software has followed that path...

The modern web is largely unusable with HW I first tried git with in 2005. Despite being technically free software, the size of "modern" browsers makes it impractical for people on slow HW/connections to actually exercise software freedom. IMHO, any HW that worked well in 2005 when git started ought to work well for git and anything associated with it today.

On Sun, Apr 18, 2021 at 10:54 PM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

To be fair, Git's contribution flow isn't the only reason why I chose to stop contributing. Another reason is the very lengthy and tedious discussions that too often spark from rather small changes.

Also, I wouldn't say I "gave up in frustration". It was a mostly unemotional decision on which of the many OSS projects I contribute to my rare spare time is spent best.

I'm not sure I agree completely here, because some tools make it easier to overcome social issues than others.

Exactly. That's why I believe tooling should allow people to subscribe to changes in code areas they're interested in, rather than a contributor having to know which subsystem maintainer to put in CC (e.g. for gitk changes). At least at the time when I contributed it was sometimes hard to move things forward if you didn't reach out to the right people.

Having default assignees for reviews / code owners / however you want to call it does not mean that only these people should review, or that something cannot be merged without their review. It just makes it more clear who's opinion would be the best to get, and who should execute a "word of command" if things do not move forward.

E.g. GitHub makes the distinction between "reviewers" for a PR and "assignees" for a PR, and the former can be configured from a CODEOWNERS file. In projects I contribute to on GitHub, "reviewers" are used as an optional list of named reviewers, i.e. these people are explicitly invited for a review. There's no obligation to review, though. On the other hand, if there are additional "assignees", these people are explicitly asked for a review. Assignees can also be assigned only at a later stage of the review, to "settle" a discussion.

The cat-herd of volunteers would neither be "reviewers" nor "assignees", but they would just browse the list or open PRs can jump it where they want to.

I admit it was a deliberately provocative choice of words, well knowing it's not reflecting the current state, to underline how I'm feeling about the workflow. E-mail is great. Also plain text e-mail is great (I've configured all my client to only send plain text), but please, not for sending around code patches.

If you send around code patches by mail instead of directly working on Git repos plus some UI, that feels to me like serializing a data class instance to JSON, printing the JSON string to paper, taking that sheet of paper to another PC with a scanner, using OCR to scan it into a JSON string, and then deserialize it again to a new data class instance, when you could have just a REST API to push the data from on PC to the other.

That's a good point, I admit I haven't thought of that. Probably because I also don't care much. So *does* it really matter? What exactly concerns you about a "centralized entity"? Is it the technical aspect of a single point of failure, or the political / social aspect of being dependent on someone you do not want to get influenced by? I guess it's a bit of both.

While these concerns could probably be addressed somewhat e.g. by multiple independently operated Gerrit servers that are kept in sync, I was curious and quickly search for more fitting "truly decentralized" solutions, and came across radicle [1]. Just FYI.

Not necessarily. As many of these tools have (REST) APIs, also different API clients exist that you could try.

AFAIK Gerrit can already do that.

On Mon, Apr 19, 2021 at 7:54 AM Sebastian Schuberth <sschuberth@gmail.com> wrote:

Forgot to add the link:

[1] https://radicle.xyz/

On Mon, Apr 19 2021, Sebastian Schuberth wrote:

Indeed, to clarify I'm not dismissing that. E.g. is an easier UX or other "prodding" going to result in better collaboration? Maybe. I'm just pointing out that it may not mostly/entirely be a tooling issue.

CODEOWNERS and default assignment etc. is something that tends to over-assign things, which is fine for a workforce you're paying, but it's important to realize that the practice in the git project is to put the onus on the submitter of the change to manually find who they should CC.

Because it's not a problem you can really solve automatically without the a trade-off of feeding potentially uninterested parties a bunch of patches they're not interested in, which could be another thing that makes them give up contributing.

So for example I've got a lot of code in git I consider that I "own" in the sense that I'm responsible for creating it, have been involved in past design discussions about it etc.

But very little of that cleanly maps to a file as the CODEOWNERS workflow expects. E.g. there's parts of grep.c that I'd definitely like to be CC'd on, but others not. Even a "git blame" of the specific lines you're touching isn't always what you want.

Having used CODEOWNERS in a corporate setting I think it's most useful for e.g. when you have a monorepo with different subdirectories that do (at least mostly) map o different teams or peope, think drivers/usb/* or arch/s390/*.

That's not inherent with the E-Mail workflow, e.g. Linus on the LKML also pulls from remotes.

It does ensure that e.g. if someone submits patches and then deletes their GitHub account the patches are still on the ML.

To begin with if we'd have used the bugtracker solution from the beginning we'd probably be talking about moving away from Bugzilla now. I.e. using those things means your data becomes entangled with the their opinionated data models.

That's interesting, but I haven't looked into that tool. Browsing their documentation earlier many of the links were 404s.

API use that usually (always?) requires an account/EULA with some entity holding the data, and as a practical concern getting all the data is usually some huge number of API requests.

Sure, FWIW the point was that you needed Gerrit to implement that, and to suggest what if they weren't interested. Would you need to maintain a forked Gerrit?

Not to say that's a dealbreaker, just trying to bridge the understanding of why some people prefer the E-Mail workflow.

Anyway, as before don't take any of the above as arguing, FWIW I wouldn't mind using one of these websites overall if it helped development velocity in the project.

Ultimately those things are up to Junio though, which these discussions always come down to.

I just wanted to help bridge the gap between the distributed E-Mail v.s centralized website flow.

On Mon, Apr 19, 2021 at 10:26 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Yeah, I was vaguely aware of this. To me, the question is why "also"? Why not *only* pull from remotes? What's the feature gap email patches try to close?

Ah, so it's basically just about a backup? That could also be solved differently by forking / syncing Git repos.

Indeed, it's an art to choose the right tool at the time, and to ensure you're not running into some "vendor-lock-in" if data export is made too hard. And aligning on someone's "opinionated data model" is not necessarily a bad thing, as standardization can also help interoperability and to smoothen workflows.

I'm not sure how relevant that concern really is, but in any cause it would be irrelevant for a self-hosted solution.

Sorry, I can't follow that. Why would you need to maintain a fork of Gerrit if Gerrit already has the feature you're looking for? Is it a hypothetical question about what to do if Gerrit would not have the feature yet?

I appreciate that open mindset of yours here.

Maybe, instead of jumping into something like an email vs Gerrit discussion, what would help is to get back one step and gather the abstract requirements. Then, with a fresh and unbiased mind, look at all the tools and infrastructure out there that are able to fulfill the needs, and then make a choice.

On Mon, Apr 19, 2021 at 09:23:14PM +0200, Sebastian Schuberth wrote:

Linus mostly pulls from git trees. The e-mail workflow tends to be used by maintainers, who are reviewing submissions from their contributors. People submitting changes relating to ext4 know to send it to the linux-ext4 mailing list; people who are submitting changes to the xfs file system send it to linux-xfs, etc.

The primary reason why the kernel uses mailing lists is because code reviews are fundamentally *discussions*, and people are used to using inboxes. Sure, you can have a gerrit server send e-mail notifications about code reviews, but then you have to reply by going to the gerrit server (and gerrit really doesn't work well on slow network link such as those found on airplanes and cruise ships). I'd say that most maintainers simply find e-mail reviews to simply be more *convenient* than using gerrit. And over time, we've used other tools to track metadata over the status of a patch, such as patchwork, which are optional.

I'll note that the kernel folks have done this, starting with a 2019 Kernel Summit talk at the Linux Plumbers Conference in Lisbon. A description of the follow-up discussions from that talk can be found here:

	https://lwn.net/Articles/803619/

There was a collection of requirements on a thread on the newly created workflows@vger.kernel.org mailing list. This has led to a number of proposals to make improvements to git, public-inbox, patchwork, the kernel.org infrastructures, etc., some of which were funded by the Linux Foundation last year.

Konstantin Ryabitsev has been driving a large amount of that work, and one of the things that has come out of that is b4. (Yes, that's a Star Trek reference... https://memory-alpha.fandom.com/wiki/B-4)

  https://people.kernel.org/monsieuricon/introducing-b4-and-patch-attestation

Obviously, this isn't intended to be a solution for everyone, and I'm sure there are many projects that are happy forcing developers to use, say, Gerrit, which might be a better solution for them.

However, there are a number of core kernel developers who are super-allergic to solutions which force users to use web interfaces. So solutions that have a combination of CLI's as well as web interface is probably going to be the right approach. Things like pwclient and b4 are exciting starting points for improved kernel workflows.

Of course, we've gone a bit farther afield from the original question which is what should git's development workflows should be. Given that git is using some of the kernel.org infrastructures, certainly some of the kernel workflow tools are options for the git development community to consider.

One of the advantages of the kernel workflows model is that we don't force users to use github or gitlab or gerrit, without having to make a global decision for the entire community. For example, if some developers want to start using b4 to download patch series for git, they could start doing that today.

Cheers,

						- Ted

On Tue, Apr 20, 2021 at 12:34 AM Theodore Ts'o <tytso@mit.edu> wrote:

[...]

That still sounds to me as if people are stuck to what they know. Maintainers are "used to using inboxes'', and that's *why* they find e-mail reviews to be convenient.

Of course, there's basically nothing wrong with sticking to a flow that works. But I understood the start of this discussion as a sign that you guys acknowledge that something does *not* work. At least not when it comes to attracting new contributors.

I'm reading a lot about "maintainers" and "kernel developers" here. But what I believe is important to accept is that Git is not only about kernel development anymore. While I'm well aware of Git's history, there are by far more people using Git than there are kernel developers, and also by lines of code (or whatever "stupid" metric you want to choose) the kernel is not the biggest project maintained in Git. Maybe not even the most important one, but that's highly subjective anyway. So asked in a heretic way, why should the opinion of a kernel developer count more than the opinion of, say, an Eclipse Foundation developer when it comes to Git workflow questions?

To me, that means if you want to make contributions to Git more attractive to the Git community beyond the kernel, you need to stop making incremental improvements to existing tools and start thinking out of the box by looking at the tools that are most popular in that "other side" of the community.

And, please don't take anything I've written as a try to talk you into anything. But as the lead of a team who's day job it is to contribute to various Open Source projects, I believe to have a good feeling about what the pain points of developers are to start contributing. I'm just trying to foster some appreciation for the thinking of the "other side".

On Tue, Apr 20, 2021 at 08:30:57AM +0200, Sebastian Schuberth wrote:

I think it should be up to each development community to decide what workflows makes sense for that community.

The kernel community has been taking requirements for what works well for *that* community, and we've found companies who are willing to fund improvements in the tools that we use (which include git, public-inbox, patchwork, etc.) so that it can meet our needs.

Our approach is that we don't want to *force* everyone to switch to some web interface. Instead, instead of either-or, we're looking for some kind both/and, so we don't have to insult people who are *extremely* productive with an e-mail workflow by calling them dinosaurs, and force them to use a web interface which would make them much less productive, on the hope that maybe we would get some more new contributors. (And at least for the kernel, we're blessed by the fact that there is no shortage of new contributors; so our goals are to make *everyone* more productive, and not have an attitude of "you shall use gerrit/github and everyone else can go suck wind".)

Git is going to have to decide what development workflows will work well for its development community. Historically, since git was originally authored by Linus Torvalds, it's not surprising that there is a bias towards an e-mail workflow. Indeed, git commands like "git send-email" and "git apply-mbox" are there from the very beginning because it was *designed* to work well with the kernel workflow.

I'm not sure that's what was meant by the question of pain points in Git's patch flow. Your perspective is certainly a valid one, and it's certainly easier to choose sides and make an opinioned decision which disenfranches "one side" of the community in favor of the "other side" of the community.

It's not the approach that was adopted by the folks who are working on improving the Kernel development workflows. The git development community will need what approach makes sense for it.

Cheers,

						- Ted

Sebastian Schuberth wrote:

I'm not stuck with what I know.

The reason why I use the email workflow is not because "I'm stuck" with it, it's because I've tried every approach out there, and they are *all* inferior.

You tell me of an approach and I will tell you all the ways in which it's inferior to email.

If some people want to use Gerrit, and/or Patchwork on top of email, that's fine. You can use inferior approaches if you want, just don't force the rest of us to stop using the superior approach.

Cheers.

On Mon, Apr 19 2021, Sebastian Schuberth wrote:

I just mentioned that as an example, FWIW I think Linus mainly uses that method for pulling "for-linus" tags from lieutenants, but that those "forks" in turn are developed over E-Mail (e.g. on the linux-usb list).

So they're supporting a flow the Git ML doesn't, of e.g. needing hundreds of patches from a subsystem maintainer to bring that subsystem up-to-date for a release.

Aside from that:

 1. Once you have patches sent to the list for review / commenting you
    might as well use them to apply the changes, why also require a
    repository to pull from?

    I may just not be understanding what you're suggesting here.

 2. One thing you may have missed is that patches != sending someone a
    repo URL to "pull" / run "log" etc. on because:

    2.1: You can provide different context (-U<n> -W) and diff algorithm
         for patches, and submitters sometimes do on this list to make
         the job of reviewers easier. I.e. sometimes I'll want to
         manually extend the context to some relevant code related to
         what's being changed.

    2.2: There's also the convention of a free-form (e.g. extra
         commentary, a reply to another E-Mail, whatever) after the
         "---" in the patch.

Yes and no, getting kicked off the service due to e.g. being from Iran would probably be a non-issue (as was the case with GitHub until recently).

I've still done various searching/for-looping over the ML archive in a way that would if translated to API requests against some remote service probably be considered a DoS, so being distributed helps there.

My example of range-diff upthread in <87fszn48lh.fsf@evledraar.gmail.com> which started this discussion wasn't to make a point about range-diff per-se, but just mention it offhand as a "feature" that in an E-Mail based flow is simply a matter of someone including a thing like that in their cover letter and sending a patch.

So for that example the specifics of range-diff being a part of Gerrit now don't matter that much, the point is that the next thing lik range-diff first has to be made part of such a service, it can't easily emerge "bottom-up".

Of course there's disadvantages to that too, I'm just pointing out that the free-form of text also has advantages that shouldn't be overlooked.

The abstract requirement comes down to one thing: Whatever Junio decides.

For a bit of context on the current discussion:

There have been versions of this discussion in-person in various "git merge" developer meets over the years, Junio doesn't attend most of those (I believe the last one was the one in April 2016 in NYC, but I missed a couple since then, so don't take my word for it).

So those discussions re-hashing of the various pain points that have been brought up over the years (and again in this thread), but have had to tip-toe around the fact that most things can't make radical forward progress (as is being proposed by some here) unless Junio is on board with them, without the benefit of being able to get feedback from him in real-time.

So the things that have come out of them are things like submitGit (later GitGitGadget), which are very useful, but still something implemented inside the narrow confines of fitting into the confines of the existing E-Mail/ML-based workflow.

My reading of [1] and [2] (including some "between the lines", so I may be wrong) is that Junio's very much interested in using his current workflow as the primary "source of truth", and that any (issue) tracking system will need to be shimmied on top of that.

Which I think means that any such "system on top" is going to have to be bespoke and inevitably drift from the real "source of truth" which is the mailing list, What's Cooking E-Mails etc.

As opposed to say, imagining some "light" in-between state where we don't use a bugtracker, or have any discussion on some centralized website, but would (or rather, Junio would) commit to creating one-line tracking issues for what's now the topics in "next" and "seen" branches. I.e. just the smaller step of answering "what's the status of my topic" via a some bug tracker, nevermind using it for anything else.

Which at least for some contributors like myself means that I'm uninterested in using any such system myself, not because I would mind using it in theory, but as long as it's not the "source of truth" I don't see the point. I'd still need to scour the ML for anything it may have missed.

1. https://lore.kernel.org/git/xmqqv98orsj5.fsf@gitster.g/
2. https://lore.kernel.org/git/xmqqfszqko0k.fsf@gitster.g/

Sebastian Schuberth <sschuberth@gmail.com> wrote:

I guess some things aren't for everybody. When I started git-svn, I never expected git to be the right tool for others. I figured most folks could just continue using SVN since they seem to like centralized things or at least have some sort of "authority" to look to.

I'm largely uninvolved with git nowadays since I'm reasonably satisfied with how it works; that and I prefer scripting languages rather than ahead-of-time languages.

Fwiw, any public-inbox endpoint with Xapian search enabled lets you request an Atom feed via "x=A" query parameter.

To watch a particular filename, the "dfn:" prefix may be used. The prefixes supported for a particular instance are documented in <https://public-inbox.org/git/_/text/help/>, and you can watch multiple files by combining with "OR".

https://public-inbox.org/git/?q=dfn:cache.h+OR+dfn:git-send-email.perl&x=A

You can also POST to get a gzipped mboxrd file:

curl -d '' \
  'https://public-inbox.org/git/?q=dfn:cache.h+OR+dfn:git-send-email.perl&x=m'

Yes, both for me. The political/social aspect is the main reason I'm involved with DVCS (and a large part of why I'm involved with free software in general).

I don't think any sort of radicle "flag day" or tool mandate is going to fly. I seem to recall at least one prominent Linux kernel hacker doesn't even use git; though I'm not sure if that's still the case.

Despite being a DVCS user even pre-git, I'm actually pessimistic about decentralization protocols that either:

1) rely on planet-destroying proof-of-work schemes

2) will need to reinvent the spam filtering techniques
   of email once they hit critical mass

Email is already well-established with a good amount of small players, and plain-text is relatively inexpensive. So it seems best to build off the only halfway-decentralized thing we have in wide use, rather than trying to start from scratch.

On Mon, Apr 19, 2021 at 9:36 PM Eric Wong <e@80x24.org> wrote:

True, since quite a while I'm also at a point where I'm satisfied with how Git (for Windows) works, so I also ceased to see the need to contribute. That's indeed another reason I forgot to mention.

Thanks for pointing out these interesting features, I wasn't aware of them.

Like you said in the beginning, I guess some things aren't for everybody.

While I can understand that conservative approach for a community around a tool as important as Git, I still fear that only ever sticking to technology that is already in wide use will hinder to look over the rim of the tea cup.

On Mon, Apr 19, 2021 at 09:49:46PM +0200, Sebastian Schuberth wrote:

Eric is being modest. There are very cool things brewing in public-inbox, like ability to create saved searches and follow threads you're interested in. E.g. you should be able to define something like "whenever someone mentions my favourite file, function name, or term, copy the entire thread into my inbox and continuously update it with new messages."

I'm hoping that this will help turn the concept of mailing lists on their head -- instead of subscribing to a list, folks will instead subscribe to closely relevant saved searches across any number of remote and local sources.

I view email as merely one way of exchanging RFC2822-formatted messages. There are others and RFC2822 is robust enough to serve as a good standard base that allows both free-form and structured content, including mixed.

-K

On 19.04.2021 18:00, Konstantin Ryabitsev wrote:

+1 on RFC2822 as universal message format. It is simple, easy to understand, trivial to manipulate in any programming language and widely supported. Standard file formats, along with standard protocols, both without "proprietary extensinos" is the key to maintaining decentralization and avoiding siloing of data and users.

Cheers! Dave

On Mon, Apr 19, 2021 at 07:54:37AM +0200, Sebastian Schuberth wrote:

It's all of the above, and really should not be discounted. Let's take what Russian government is doing lately as an example. In its effort to control social dissent, Russian censorship organization RosKomNadzor (RKN) has taken steps to deliberately break internet operation -- in a very ham-fisted way. Just a month ago they tried to "slow down" Twitter by blocking DNS queries for any domains containing the substring "t.co" -- which, hey, broke gihubusercontent.com among many other sites. There's every reason to believe that this won't be the only time they do something idiotic like that, so as a result it is increasingly difficult for Russian contributors to justify participating in projects that are hosted on GitHub -- one day they may not be able to reach it reliably (or at all).

(If you think the answer to that would be "just use a VPN", it's one of those recommendations that are easy to make for someone not worried about their ISP reporting "sketchy encrypted traffic" to "the authorities.")

Patches sent via email remain immune to this. Even if vger falls over, it's merely a list service -- there are alternative ways of transmitting RFC2822 messages that don't involve a central host (such as via a NNTP gateway, publishing a public-inbox "feed", etc). Email remains one of the few protocols that are designed ground-up to be decentralized and I'm afraid that we are again finding ourselves in a world where this is increasingly relevant.

I know Radicle folks -- I was on their technical board. A lot of what they have implemented is very similar to my initial thoughts expressed in https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains

I have high hopes for the project, but it's not ready to take on the world until they implement code collaboration aspects (issue tracking, change requests, etc). It's going to be tough and I really hope they succeed.

-K

On Monday, April 19, 2021 2:49:21 PM MST Konstantin Ryabitsev wrote:

It can be other things too.   

For instance a corporation that for a variety of reasons has an urgent need to restrict internet traffic. Email will usually get through, but web site traffic may not.

While Github is unlikely to get taken offline, other sites that hold data may not be so lucky. Think bankrupcy or other issues.

Email traffic allows for routing around such issues.

On 19.04.2021 17:49, Konstantin Ryabitsev wrote:

I agree with Konstantin on this one. To this day, email is still the most decentralized and "user sovereign" system on the internet. The standardization of protocols and file formats is not perfect but it is "complete" in the sense that it meets all of the requirements for decentralized software development.

Think about it like this. Right now, I could use an IMAP client to download all of my emails from GMail, store them in mbox files, then use the IMAP client to upload the email to Fastmail or SDF.org or some other email provider. Or better yet, I can install local tools for working with my email. The fact that email providers/tools are largely interchangeable and replacable--despite Google/Yahoo/Microsoft's best efforts--gives maximum power to users.

Like I said, I totally agree with Konstantin and I think the vision he described in his post on developer sigchains is what I've always wanted as an open source developer. It is common to hear the argument that centralized systems are more convient and easier to use and the more decentralized a sysetem, the harder it gets to use. I suspect that is only a half-truth because I don't think we've achieved full decentralization which is what Konstantin touches on in his post too. Full decentralization will bring automatic maintainence of p2p connections and synchronization. Things will "just work".

I know I'm veering off topic a bit here but decentralization has been the focus of all of my learning, research and work for more than a decade now. Email is critical for maintaining decentralized development capabilities.

Cheers! Dave

On 08/05/21 09.08, dwh@linuxprogrammer.org wrote:

Well, I use Thunderbird because it supports Gmail accounts out of the box. But I wish I could use Mutt or similar, alas Gmail requires that I need to enable 2FA and s/<google account password>/<specific app password>/ in order to access Gmail via Mutt. I currently steer clear from 2FA, because once upon a time in 2018 I screwed up (locked-out) from my older account, which IMO mission-critical., for I couldn't pass all possible verification methods for that. So I created new account one in about Día de Kartini (Kartini's day).

And yes, Google/Yahoo/MS have webmail interface for their mail services (Gmail/YahooMail/Outlook), but for purposes for sending to vger.kernel.org, these above are rubbish because they genereated HTML emails, and vger hate HTML emails.

Sebastian Schuberth wrote:

I completely agree with this assessment.

In the spectrum from all code is allowed (0) to only perfect code is allowed (100) I'd say the git project is around 95. It's good in the sense that user expectation rarely breaks, but on the other hand not much progress happens.

Personally I would turn the dial of perfectedness down to 90, or even 80.

It's because of this focus on perfection that discussions get tedious, and thus perfect becomes the enemy of good.

But since the current maintership is never going to change that focus, I think a fork of git is necessary.

Cheers.

Hi all,

Greetings Raxel! Myself and Stephen F are patchwork maintainers so we'll be reviewing and merging any proposals you have for patchwork. We try to be a welcoming place. We're also both extremely busy so (unfortunately) you will probably need to ping me if I forget to respond.

Patchwork does have the A/R/T/F (Acked-by:/Reviewed-by:/Tested-by:/Fixes:) column, but this doesn't have a good way to capture something that falls short of a full review. There's also the patch states (e.g. Changes Requested), but that requires either the author or a maintainer to change the status via the web interface or with an API client.

Yeah, cross-series linking is something we've been interested in for a long time. We have a little bit of the infrastructure already but there's a long way to go.

One of the real challenges for us has been figuring out how to reliably link iterations. It's one thing if iterations are sent in-reply-to the early version, but at least for the kernel that's not The Way Things Are Done. There's lot of common things people do (split series, rename series, add/drop patches) that makes reliable linking very challenging. And traditionally Patchwork has tried to Do It Right rather than go for a probabilistic approach. (We get a lot of email complaints if we get things wrong.)

(Having said that, I'd certainly be open to considering any attempts to automatically link series, even if only probabilistic, so long as they err by missing things rather than err by linking things that are unrelated.)

One thing that's come up as another possible option is free-form tags. I think there's some old series on the list from Veronika that attempts this. That'd allow Someone to 'tag' a patch or a series on patchwork. For Veronika's use case this was for complex CI - things like "ready-for-real-hw-tests" that shouldn't happen until someone has cast an eye over the patches.

One big challenge for patchwork development (which catches us out regularly) is the odd interaction between patchwork-the-project and patchwork-the-deployments. There are 2 large deployments: patchwork.ozlabs.org and patchwork.kernel.org, and they all have multiple-GB databases. This means that performance and db load matters, but is really hard to test locally. A good example is writing efficient migrations - historically we have struggled with this.

Anyway, good luck with the dozens of messages and strong opinions!

Kind regards, Daniel

On 2021-04-14 at 06:13:26, Jonathan Nieder wrote:

I realize I'm a bit late here, but I've been thinking about this some and wanted to chime in.

I have trouble finding all the spots where people have given me review feedback. I have patch mails and responses to those mails go to a particular folder, but I still often find that I'm not quite sure if I've gotten every piece of feedback in a review. Sometimes, embarrassingly, I don't, and then I have to send another reroll. Regardless, this makes rerolling a series much slower as I have to comb my mail multiple times.

I find I'm often unsure what to put in the cover letter for a v2 or subsequent series. Clearly people don't want the same thing as v1, but I rarely have useful information other than a summary of changes.

I have tooling to automatically generate the proper range for range-diffs in cover letters, but that tooling requires some sort of manual timestamp, which means I need to go search for my previous series to find the date and generate the range diff, or if I'm in a rush, I just have to omit it. This can take some time, having to guess what I named the cover letter the last time and search for it in a mailbox with a 6-digit quantity of mails[0].

In general, I have trouble keeping track of the patch mails I've sent. I do definitely need to refer to them later, but I don't generally keep them around on my system since they tend to duplicate my repository, so I end up needing to find them in my mailbox, which as mentioned, is slow and error prone.

I find that the git-contacts script is often not helpful to find reviewers. When I send out a series, it often suggests Peff and Junio. While both of them are very capable, they are also not capable of reviewing every series, and in many cases I know full well that one or the other is not going to be able to give me a good review (for lack of familiarity with the SHA-256 work, due to having many other things to review and to do in life, or for other reasons). It also, unfortunately, suggests me as a reviewer for many things, which while flattering, reflects the fact that I've touched a lot of code and not that I have a deep understanding of most of the codebase, which I do not. For areas where I do have relevant insight, such as the signature code, I'm often not chosen.

I realize a lot of these are not intrinsic to our workflow and can be solved with tooling, but because I haven't built that tooling, they're pain points that I experience in our workflow.

I've built some tooling around this, including mail filtering, aliases, and scripting, but it doesn't seem like enough. I know others have built really great tooling for themselves, but by the time I notice these pain points, it's usually the evening and I don't have time to build tooling and get things sent out as well. I also don't especially enjoy building tooling here.

The friction here makes me less likely to send out patches and much slower to reroll patches than I'd otherwise be. And I feel like that means that I practically can only ever send out a series when I have more time on the weekend, and as a result, I worry that my patches hold up others in the tree much more often than I'd like. It also makes contributing to Git less fun, which is important since overwhelmingly my patches are sent on my own time.

I don't think I know enough about it to say. If it can more clearly track review feedback and help keep track of patch emails, I think it would be a major improvement, for me at least.

I think we definitely need a bug tracker. We extremely frequently lose bugs and feature requests on the list and people aren't very likely to search the list. If we could use the same one as someone else, such as the kernel, that would be ideal, because it means people are more likely to already have an account and therefore the friction to report a bug is lower. Alternatively, we could use something like debbugs which is controllable entirely by email and therefore requires no accounts (but does require someone to occasionally prune reported spam).

I know full well why we don't use a forge-based model and I'm not recommending that, but I do want to point out that forges solve all of my pain points, and I do have a much quicker turnaround time on patches when I'm using a forge. So ideally we'd have some standard or recommended tooling, whether built by us or by others (e.g., an open source project for patch workflows), that addresses these pain points so that everyone doesn't have to build their own and turnaround time can be improved.

I have seen replies downthread that some developers really are reticent to use more common tooling, like web interfaces. While I do want to keep our project as accessible as possible to as many people as possible, I worry that by catering to folks who don't want to adopt this tooling, we are drastically reducing the number of possible contributors of all sorts (code authors, documentation writers, bug reporters) by not doing so and worsening our own experience in many ways. I do think we should adopt modern tooling (e.g., web interfaces) provided that it is usable for people with accessibility needs, even if that makes some people unhappy.

[0] I don't, as a general rule, delete emails to this list or otherwise.

On Mon, Apr 26, 2021 at 02:04:32AM +0000, brian m. carlson wrote:

A quick and easy feature request to this (which I have had as well) would be implementing a sendmail.fcc config, which if set, would append any mail messages sent by git send-email to the Unix mbox file specified by sendmail.fcc.

Once you have the message id of any patch mail that you've sent...

This becomes pretty easy to solve using existing tooling. For people who like web interfaces:

    https://lore.kernel.org/r/<message-id>

(This works today because git@vger.kernel.org is archived by the lore.kernel.org public-inbox archive.)

Or for those who like CLI's and/or text-based mail readers such as mutt or pine:

   b4 mbox -o /tmp <message-id>

This will dump the full mail thread (given any any message-id in that mail thread) to /tmp/<messaige-id>.mbx in Unix mbox format, again relying on lore.kernel.org. I've found this to be especially handy if I've been cc'ed part-way through a mail thread, or if I was only cc'ed on a single patch and I want to see the full patch series for context.

Cheers,

						- Ted

On Mon, Apr 26 2021, brian m. carlson wrote:

I'm not disagreeing, just replying to point out that I think for your suggestion & others having as much of a split as possible between "what" and "how" would, I think, be useful in moving things along.

A web interface is a "how", but it's also implicitly a "what" in the way that most people think about it in this context.

I.e. it's not like we couldn't have a bug tracker now using the ML, you'd send a patch, Junio would pick up the report and we'd drop it into bug/some-description.md (with some handwaiving for formatting, merge conflicts etc.). We'd remember bug reports, feature requests etc. forever, and patches could atomically change/close/remove those as they fix/change/implement them.

The point I'm getting at is that the "what" we're also implicitly discussing is the developer community shouldering the burden of keeping such a tracker and the information within it up-to-date.

A web-based interface that worked like our mailing list does now would be one that, say, deleted your bug 30 days of inactivity (or otherwise made it as "archived" as something in the ML lore).

I couldn't find a reference to it now, but as I recall (and maybe I wrote some) there's been some prominent defenses of this model of development in the past.

I.e. it puts the onus on reporters to make sure their issue is being addressed, if nobody cares to pick it up it probably wasn't that important, and we shouldn't so lightly assume the fixed cost of adding that one-off report to an ever-growing list of reports we'd need to continually look at / curate / keep up to date etc.

But none of that's an argument I'm looking to get into right now, or really have much of a firm stance on. I just wanted to point out that it's a clear case where a "what" is being conflated with a "how".

I.e. we're implicitly not only talking about how something gets done, but a big change in what gets done.

I had a similar comment upthread (or in a side-thread) about how much of the suggestions of making use of the various reviewer/approver PR/MR/whatever tools in the wild seem to simply assume a move away from the long-time model where there's effectively only one approver/merge master/committer (i.e. Junio). That's an entirely defensible argument, but another case of making a "how" and "what" argument at the same time.

Maybe the people proposing both a "how" and a "what" will have an easier time if those are untangled, and we change one thing at a time.

"brian m. carlson" <sandals@crustytoothpaste.net> wrote:

Along the lines of what Ted said about Fcc, I've always Bcc-ed myself on every message I send to verify deliverability and check/train my spam filter.

What search tool do you use? mairix can handle the 6-digit quantity of the git list fairly well. The following finds all threads with "sandals" in From/To/Cc:

	mairix -t a:sandals d:YYYYMMDD-YYYYMMDD

and dumps it to whatever Maildir/mbox/IMAP "mfolder" you've configured. (prefixes in public-inbox such as "a:", "d:" and "s:" are stolen from mairix; though mairix ranges use "-" and public-inbox uses ".." due to Xapian).

I've also heard good things about notmuch, but I archive old mail to gzipped mboxrd right now[1], and that only supports Maildir... I learned to use Xapian by reading code in notmuch.

[1] Fwiw, I'm also working on an AGPL Perl5 storage+search CLI
    that scales to 7/8-digit mail collections.  It's not ready
    for prime-time, yet, but getting there...  (Assuming it
    doesn't set my SSD on fire, first :x)

On 2021-04-28 at 07:59:27, Eric Wong wrote:

I simply use mutt to read my mailbox and search. Nothing fancy. I should point out that it's not local; it's on a Dovecot IMAP server located on a VPS in New York City.

Eric Wong wrote:

For what it's worth I've been using notmuch for several years now and it's much superior to anything I've seen. I can find all kinds of emails instantly and using my prefered text-editor (vim).

But yeah, the only problem is getting the mail in the right format in my machine in the first place. But once I have it, it's blazingly fast to deal with email.

brian m. carlson wrote:

I wrote a tool to deal precisely with that: git-send-series[1]. All the metadata in the patch series is stored in a text file (version, subject, cc, cover letter, etc.).

So when you want to send the next version of your series you just do `git send-series` and it deals with all that tedious stuff automatically. You just need to update the cover letter.

It also keeps track of the previous versions of your series in order to automatically generate the range-diff.

I have my mailbox stored in my machine syncronized with isync[2], and indexed with notmuch[3]. I can view all mails I've ever sent instantly with a simple search:

  nmm tag:git tag:sent subject:PATCH

git-contacts is a subpar rewrite of my original script: git-related[4].

Using git-contacts on your last merged patch I get this:

  Cornelius Weig <cornelius.weig@tngtech.com>
  Jeff King <peff@peff.net>
  Junio C Hamano <gitster@pobox.com>
  Johannes Schindelin <Johannes.Schindelin@gmx.de>

However, with git-related I get this:

  Junio C Hamano <gitster@pobox.com> (signer: 62%, author: 37%)
  Johannes Schindelin <Johannes.Schindelin@gmx.de> (author: 37%)
  Jeff King <peff@peff.net> (reviewer: 12%, author: 12%)
  Cornelius Weig <cornelius.weig@tngtech.com> (author: 12%)

Which is much more useful.

However, you actually have options to catch more changes:

  % git related --min-percent=5 --since=10-years-ago 75555676ad -1
  Junio C Hamano <gitster@pobox.com> (signer: 80%, author: 20%)
  Jeff King <peff@peff.net> (reviewer: 10%, author: 30%)
  Johannes Schindelin <Johannes.Schindelin@gmx.de> (author: 30%)
  Patrick Steinhardt <ps@pks.im> (author: 10%)
  Cornelius Weig <cornelius.weig@tngtech.com> (author: 10%)

This may not solve your paticular complaint, but it's clearly superior.

Cheers.

[1] https://github.com/felipec/git-send-series [2] https://isync.sourceforge.io/ [3] https://notmuchmail.org/ [4] https://github.com/felipec/git-related

Jonathan Nieder wrote:

Raxel, I would consider looking at nmbug[1]. It is a tool developed by the notmuch team in order to deal with patches both in command line and web interfaces.

As you can see from the discussion that ensured from the original mail, the git community does care about the command line interface a great deal.

My only real pain point is that sometimes a patch series is sent as a subthread of another patch series, which makes it difficult for me to mentally separate the two.

There is nothing I can do. This is a culture thing that hopefully will change.

No.

nmbug[1], but again, it depends on how it's actually used.

Cheers.

[1] https://notmuchmail.org/nmbug/