threads / rfc / 58216

RFC patchgitweb: improve title shortening heuristics

Subject: [RFC PATCH] gitweb: improve title shortening heuristics

## tl;dr

7 messages between Jul 24, 2022 and Jul 28, 2022. Diffs are folded; open one to read it.

replies: 6people: 3as markdown or json

Julien Rouhaud· Jul 24, 2022, 06:12 UTC · lore
Hi,

First of all, this is my first time on this ML so apologies in advance if I missed anything in the patch submission guidelines.

We got some report recently that the commit short title on the postgres gitweb instance was sometimes being mangled (1). After a bit of digging, it appears to be due to some long time heuristics to remove some uninteresting parts of a commit message (see 198066916a8 from August 2005). In our case, it removed any occurrence of "master." in the commit message even if the message contains "postmaster.c" rather than a cname (or something that looks like it), leading to the commit message:

Remove postmaster.c's reset_shared() wrapper function.
being displayed as:
Remove postc's reset_shared() wrapper function.

It's probably some corner case for which there's barely any complaint, so it doesn't look worthwhile to spend too much effort on it. It also seems impossible to make the current approach entirely bullet proof, but if we simply make sure that the prefix is preceded by at least one whitespace and isn't followed by another one we could avoid almost all of the incorrect matches (and all of them as far as postgres is concerned). Would that be an acceptable compromise? If yes, I'm attaching a patch that does that (and also adds git:// and https:// to the list of trimmed protocols while at it).

Otherwise, would it be acceptable to disable the whole block (the "remove leading stuff of merges to make the interesting part visible") with some new configuration option?

Cheers, Julien.

[1] https://www.postgresql.org/message-id/flat/4025723.1658013974%40sss.pgh.pa.us
From ed46dcd2796b9af6ba3f73d46a3141a88964ed11 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v1] gitweb: improve title_short shortening heuristics

In order to shorten the title, some common domain prefixes can be detected and removed. However, the current regex matches those prefix anywhere in the title which makes it likely to remove it where it's not intended.

To make that case less likely, make sure that the prefix is preceded by at least one whitespace and isn't followed by another whitespace.

While at it, also add git:// and https:// to the list of detected and trimmed protocols.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---
 gitweb/gitweb.perl | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to gitweb/gitweb.perl +2 −2
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..18dd0b93fb 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3565,10 +3565,10 @@ sub parse_commit_text {
 				$title =~ s/^Automatic //;
 				$title =~ s/^merge (of|with) /Merge ... /i;
 				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
+					$title =~ s/(git|http|https|rsync):\/\///;
 				}
 				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
+					$title =~ s/\s+(master|www|rsync)\.([^\s])/ \2/;
 				}
 				if (length($title) > 50) {
 					$title =~ s/kernel.org:?//;
-- 
2.37.0
Junio C Hamano· Jul 25, 2022, 01:30 UTC · re: Julien Rouhaud · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

Julien Rouhaud <rjuju123@gmail.com> writes:
> Otherwise, would it be acceptable to disable the whole block (the "remove
> leading stuff of merges to make the interesting part visible") with some new
> configuration option?

I personally find that "shortening" logic way too specific to the kernel project hosted at kernel.org and would be inappropriate to use it anywhere else.

	if (length($title) > 50) {
		$title =~ s/^Automatic //;
		$title =~ s/^merge (of|with) /Merge ... /i;
		if (length($title) > 50) {
			$title =~ s/(http|rsync):\/\///;
		}
		if (length($title) > 50) {
			$title =~ s/(master|www|rsync)\.//;
		}
		if (length($title) > 50) {
			$title =~ s/kernel.org:?//;
		}
		if (length($title) > 50) {
			$title =~ s/\/pub\/scm//;
		}
	}
	$co{'title_short'} = chop_str($title, 50, 5);

Of course, http:// and rsync:// are way outdated (https://, ssh:// and git:// are probably reasonable). Equally outdated is to merge branches from master.kernel.org, www.kernel.org, or rsync.kernel.org (many merges are recorded as pulling from git://git.kernel.org/ or https://git.kernel.org/ these days).

The above code is all from 198066916a8 (Kay Sievers 2005-08-07), so it is very much understandable that its shortening heuristics no longer match reality.

Even worse, I somehow thought that kernel.org no longer uses gitweb but uses something else. So I suspect that nobody sheds tears if we remove the whole block unconditionally. In fact, it would make the world a better place.

Julien Rouhaud· Jul 25, 2022, 02:12 UTC · re: Junio C Hamano · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

On Sun, Jul 24, 2022 at 06:30:44PM -0700, Junio C Hamano wrote:
Show 27 quoted lines
> Julien Rouhaud <rjuju123@gmail.com> writes:
>
> > Otherwise, would it be acceptable to disable the whole block (the "remove
> > leading stuff of merges to make the interesting part visible") with some new
> > configuration option?
>
> I personally find that "shortening" logic way too specific to the
> kernel project hosted at kernel.org and would be inappropriate to
> use it anywhere else.
>
> 	if (length($title) > 50) {
> 		$title =~ s/^Automatic //;
> 		$title =~ s/^merge (of|with) /Merge ... /i;
> 		if (length($title) > 50) {
> 			$title =~ s/(http|rsync):\/\///;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/(master|www|rsync)\.//;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/kernel.org:?//;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/\/pub\/scm//;
> 		}
> 	}
> 	$co{'title_short'} = chop_str($title, 50, 5);

That's probably true, although some parts (like the protocol) seemed general enough to me to be worth considering for general use case, and some of the really specific parts seems so specific that they shouldn't really matter if used outside kernel.org.

Show 11 quoted lines
>
> Of course, http:// and rsync:// are way outdated (https://, ssh://
> and git:// are probably reasonable).  Equally outdated is to merge
> branches from master.kernel.org, www.kernel.org, or rsync.kernel.org
> (many merges are recorded as pulling from git://git.kernel.org/ or
> https://git.kernel.org/ these days).
>
> Even worse, I somehow thought that kernel.org no longer uses gitweb
> but uses something else.  So I suspect that nobody sheds tears if we
> remove the whole block unconditionally.  In fact, it would make the
> world a better place.

Well, I'm obviously totally fine with getting rid of it. FWIW, even the protocol part wouldn't change anything for the instance hosted by postgres.

I'm attaching an updated v2 patch that removes all that logic and just keeps the title_short chopped at 50 chars.

From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics

Those heuristics are way outdated and too general to be useful outside of kernel.org. Since kernel.org doesn't use gitweb anymore and at least one project complained about incorrect behavior, entirely remove them.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---
 gitweb/gitweb.perl | 17 -----------------
 1 file changed, 17 deletions(-)
Show changes to gitweb/gitweb.perl +0 −17
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..e66eb3d9ba 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3560,23 +3560,6 @@ sub parse_commit_text {
 		$title =~ s/^    //;
 		if ($title ne "") {
 			$co{'title'} = chop_str($title, 80, 5);
-			# remove leading stuff of merges to make the interesting part visible
-			if (length($title) > 50) {
-				$title =~ s/^Automatic //;
-				$title =~ s/^merge (of|with) /Merge ... /i;
-				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
-				}
-				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/kernel.org:?//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/\/pub\/scm//;
-				}
-			}
 			$co{'title_short'} = chop_str($title, 50, 5);
 			last;
 		}
-- 
2.37.0
Ævar Arnfjörð Bjarmason· Jul 25, 2022, 05:54 UTC · re: Julien Rouhaud · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

On Mon, Jul 25 2022, Julien Rouhaud wrote:
Show 50 quoted lines
> On Sun, Jul 24, 2022 at 06:30:44PM -0700, Junio C Hamano wrote:
>> Julien Rouhaud <rjuju123@gmail.com> writes:
>>
>> > Otherwise, would it be acceptable to disable the whole block (the "remove
>> > leading stuff of merges to make the interesting part visible") with some new
>> > configuration option?
>>
>> I personally find that "shortening" logic way too specific to the
>> kernel project hosted at kernel.org and would be inappropriate to
>> use it anywhere else.
>>
>> 	if (length($title) > 50) {
>> 		$title =~ s/^Automatic //;
>> 		$title =~ s/^merge (of|with) /Merge ... /i;
>> 		if (length($title) > 50) {
>> 			$title =~ s/(http|rsync):\/\///;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/(master|www|rsync)\.//;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/kernel.org:?//;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/\/pub\/scm//;
>> 		}
>> 	}
>> 	$co{'title_short'} = chop_str($title, 50, 5);
>
> That's probably true, although some parts (like the protocol) seemed general
> enough to me to be worth considering for general use case, and some of the
> really specific parts seems so specific that they shouldn't really matter if
> used outside kernel.org.
>>
>> Of course, http:// and rsync:// are way outdated (https://, ssh://
>> and git:// are probably reasonable).  Equally outdated is to merge
>> branches from master.kernel.org, www.kernel.org, or rsync.kernel.org
>> (many merges are recorded as pulling from git://git.kernel.org/ or
>> https://git.kernel.org/ these days).
>>
>> Even worse, I somehow thought that kernel.org no longer uses gitweb
>> but uses something else.  So I suspect that nobody sheds tears if we
>> remove the whole block unconditionally.  In fact, it would make the
>> world a better place.
>
> Well, I'm obviously totally fine with getting rid of it.  FWIW, even the
> protocol part wouldn't change anything for the instance hosted by postgres.
>
> I'm attaching an updated v2 patch that removes all that logic and just keeps
> the title_short chopped at 50 chars.

Please "inline" your patches, see "Sending your patches" in Documentation/SubmittingPatches (I.e. send it with git-send-email, or similar). I see this as:

> [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...
In my MUA. But manually extracting it and quoting it:
Show 27 quoted lines
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 1835487ab2..e66eb3d9ba 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3560,23 +3560,6 @@ sub parse_commit_text {
>  		$title =~ s/^    //;
>  		if ($title ne "") {
>  			$co{'title'} = chop_str($title, 80, 5);
> -			# remove leading stuff of merges to make the interesting part visible
> -			if (length($title) > 50) {
> -				$title =~ s/^Automatic //;
> -				$title =~ s/^merge (of|with) /Merge ... /i;
> -				if (length($title) > 50) {
> -					$title =~ s/(http|rsync):\/\///;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/(master|www|rsync)\.//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/kernel.org:?//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/\/pub\/scm//;
> -				}
> -			}
>  			$co{'title_short'} = chop_str($title, 50, 5);
>  			last;
This looks good to me, The one thing I'd add is that we're just running:
	 git rev-list --parents --header  --max-count=1 HEAD

And parsing that, but if we're truncating things perhaps we should just run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])" syntax or similar.

That's obviously a follow-up, but if anyone's interested in deleting even more code here...

Julien Rouhaud· Jul 26, 2022, 13:59 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

Hi,
On Mon, Jul 25, 2022 at 07:54:34AM +0200, Ævar Arnfjörð Bjarmason wrote:
Show 6 quoted lines
>
> Please "inline" your patches, see "Sending your patches" in
> Documentation/SubmittingPatches (I.e. send it with git-send-email, or
> similar). I see this as:
>
> > [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...

Ah I'm sorry about that. For the archives sake (and to make sure that I correctly configured my MUA), I'm reattaching the v2 inline.

Show 10 quoted lines
> This looks good to me, The one thing I'd add is that we're just running:
>
> 	 git rev-list --parents --header  --max-count=1 HEAD
>
> And parsing that, but if we're truncating things perhaps we should just
> run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])"
> syntax or similar.
>
> That's obviously a follow-up, but if anyone's interested in deleting
> even more code here...

Note that I didn't try to do anything about that. I have the feeling that gitweb isn't wildly used nowadays, especially for bigger projects, so it didn't seems worth spending too much efforts.

From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics

Those heuristics are way outdated and too general to be useful outside of kernel.org. Since kernel.org doesn't use gitweb anymore and at least one project complained about incorrect behavior, entirely remove them.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---
Notes:
    Patch re-submitted inline rather than in attachment, sorry about the
    previous attachments.
 gitweb/gitweb.perl | 17 -----------------
 1 file changed, 17 deletions(-)
Show changes to gitweb/gitweb.perl +0 −18
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..e66eb3d9ba 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3560,23 +3560,6 @@ sub parse_commit_text {
 		$title =~ s/^    //;
 		if ($title ne "") {
 			$co{'title'} = chop_str($title, 80, 5);
-			# remove leading stuff of merges to make the interesting part visible
-			if (length($title) > 50) {
-				$title =~ s/^Automatic //;
-				$title =~ s/^merge (of|with) /Merge ... /i;
-				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
-				}
-				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/kernel.org:?//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/\/pub\/scm//;
-				}
-			}
 			$co{'title_short'} = chop_str($title, 50, 5);
 			last;
 		}
--
2.37.0
Junio C Hamano· Jul 27, 2022, 06:31 UTC · re: Julien Rouhaud · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

Julien Rouhaud <rjuju123@gmail.com> writes:
Show 12 quoted lines
> Hi,
>
> On Mon, Jul 25, 2022 at 07:54:34AM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>> Please "inline" your patches, see "Sending your patches" in
>> Documentation/SubmittingPatches (I.e. send it with git-send-email, or
>> similar). I see this as:
>>
>> > [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...
>
> Ah I'm sorry about that.  For the archives sake (and to make sure that I
> correctly configured my MUA), I'm reattaching the v2 inline.

Thanks. It is not for the archives' sake; I do not take (or notice in the first place) patches in attachments, simply because rarely anybody reviews attachments.

Show 16 quoted lines
>> This looks good to me, The one thing I'd add is that we're just running:
>>
>> 	 git rev-list --parents --header  --max-count=1 HEAD
>>
>> And parsing that, but if we're truncating things perhaps we should just
>> run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])"
>> syntax or similar.
>>
>> That's obviously a follow-up, but if anyone's interested in deleting
>> even more code here...
>
> Note that I didn't try to do anything about that.  I have the feeling that
> gitweb isn't wildly used nowadays, especially for bigger projects, so it didn't
> seems worth spending too much efforts.
>
> From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001

When you are sending the patch in for real, after making a non-patch discussion (all of the above), you can write a "scissors" line

----- >8 -----

right here, immediately before your "From:" and "Subject:". What it does is to tell "git am" on the receiving end to ignore everything it saw up to that point and pretend the message starts there.

You do not want to include "From <object name>" line that is used as an internal separator in the mailbox format. You do not want to include the "Date:" in-body header, either. "From:" and "Subject:" are used often because they are frequently different from the e-mail (e.g. in the message I am responding to, the sender is rjuju123@gmail, not julien.rouhaud@free.fr, and we want the latter on the From: line because that is what appears on the Signed-off-by: line).

Show 13 quoted lines
> From: Julien Rouhaud <julien.rouhaud@free.fr>
> Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics
>
> Those heuristics are way outdated and too general to be useful outside of
> kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
> project complained about incorrect behavior, entirely remove them.
>
> Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
> ---
>
> Notes:
>     Patch re-submitted inline rather than in attachment, sorry about the
>     previous attachments.
Thanks.  Will queue.
Show 33 quoted lines
>  gitweb/gitweb.perl | 17 -----------------
>  1 file changed, 17 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 1835487ab2..e66eb3d9ba 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3560,23 +3560,6 @@ sub parse_commit_text {
>  		$title =~ s/^    //;
>  		if ($title ne "") {
>  			$co{'title'} = chop_str($title, 80, 5);
> -			# remove leading stuff of merges to make the interesting part visible
> -			if (length($title) > 50) {
> -				$title =~ s/^Automatic //;
> -				$title =~ s/^merge (of|with) /Merge ... /i;
> -				if (length($title) > 50) {
> -					$title =~ s/(http|rsync):\/\///;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/(master|www|rsync)\.//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/kernel.org:?//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/\/pub\/scm//;
> -				}
> -			}
>  			$co{'title_short'} = chop_str($title, 50, 5);
>  			last;
>  		}
> --
> 2.37.0
Julien Rouhaud· Jul 28, 2022, 01:30 UTC · re: Junio C Hamano · lore

Re: [RFC PATCH] gitweb: improve title shortening heuristics

Hi,
On Tue, Jul 26, 2022 at 11:31:32PM -0700, Junio C Hamano wrote:
Show 17 quoted lines
>
> When you are sending the patch in for real, after making a non-patch
> discussion (all of the above), you can write a "scissors" line
>
> ----- >8 -----
>
> right here, immediately before your "From:" and "Subject:".  What it
> does is to tell "git am" on the receiving end to ignore everything
> it saw up to that point and pretend the message starts there.
>
> You do not want to include "From <object name>" line that is used as
> an internal separator in the mailbox format.  You do not want to
> include the "Date:" in-body header, either.  "From:" and "Subject:"
> are used often because they are frequently different from the e-mail
> (e.g. in the message I am responding to, the sender is rjuju123@gmail,
> not julien.rouhaud@free.fr, and we want the latter on the From: line
> because that is what appears on the Signed-off-by: line).
Ok, got it.
Show 15 quoted lines
> > From: Julien Rouhaud <julien.rouhaud@free.fr>
> > Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics
> >
> > Those heuristics are way outdated and too general to be useful outside of
> > kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
> > project complained about incorrect behavior, entirely remove them.
> >
> > Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
> > ---
> >
> > Notes:
> >     Patch re-submitted inline rather than in attachment, sorry about the
> >     previous attachments.
>
> Thanks.  Will queue.
Thanks!

← back to recent threads