threads / patch / 19287

patchgit-send-email: parse all messages in mbox

Subject: [PATCH] git-send-email: parse all messages in mbox

## tl;dr

5 messages between May 11, 2009 and May 14, 2009. Diffs are folded; open one to read it.

replies: 4people: 3as markdown or json

Vitaly Mayatskikh· May 11, 2009, 13:46 UTC · lore

Currently git-send-email sends all mbox in one email. This seems to be wrong, because mbox can contain several messages. For example, `git format-patch --stdout' forms mbox file with all patches in it.

This patch allows git send-email to send several messages from one mbox separately.

Signed-off-by: Vitaly Mayatskikh <v.mayatskih@gmail.com>
---
 git-send-email.perl |  278 ++++++++++++++++++++++++++-------------------------
 1 files changed, 141 insertions(+), 137 deletions(-)
Show changes to git-send-email.perl +141 −137
diff --git a/git-send-email.perl b/git-send-email.perl
index cccbf45..614fcbd 100755
--- a/git-send-email.perl
+++ b/git-send-email.perl
@@ -984,168 +984,172 @@ $message_num = 0;
 foreach my $t (@files) {
 	open(F,"<",$t) or die "can't open file $t";
 
-	my $author = undef;
-	my $author_encoding;
-	my $has_content_type;
-	my $body_encoding;
-	@cc = ();
-	@xh = ();
-	my $input_format = undef;
-	my @header = ();
-	$message = "";
-	$message_num++;
-	# First unfold multiline header fields
-	while(<F>) {
-		last if /^\s*$/;
-		if (/^\s+\S/ and @header) {
-			chomp($header[$#header]);
-			s/^\s+/ /;
-			$header[$#header] .= $_;
-	    } else {
-			push(@header, $_);
-		}
-	}
-	# Now parse the header
-	foreach(@header) {
-		if (/^From /) {
-			$input_format = 'mbox';
-			next;
-		}
-		chomp;
-		if (!defined $input_format && /^[-A-Za-z]+:\s/) {
-			$input_format = 'mbox';
+	while (1) {
+		my $author = undef;
+		my $author_encoding;
+		my $has_content_type;
+		my $body_encoding;
+		@cc = ();
+		@xh = ();
+		my $input_format = undef;
+		my @header = ();
+		$message = "";
+		$message_num++;
+		# First unfold multiline header fields
+		while(<F>) {
+			last if /^\s*$/;
+			if (/^\s+\S/ and @header) {
+				chomp($header[$#header]);
+				s/^\s+/ /;
+				$header[$#header] .= $_;
+			} else {
+				push(@header, $_);
+			}
 		}
-
-		if (defined $input_format && $input_format eq 'mbox') {
-			if (/^Subject:\s+(.*)$/) {
-				$subject = $1;
+		# Now parse the header
+		foreach(@header) {
+			if (/^From /) {
+				$input_format = 'mbox';
+				next;
 			}
-			elsif (/^From:\s+(.*)$/) {
-				($author, $author_encoding) = unquote_rfc2047($1);
-				next if $suppress_cc{'author'};
-				next if $suppress_cc{'self'} and $author eq $sender;
-				printf("(mbox) Adding cc: %s from line '%s'\n",
-					$1, $_) unless $quiet;
-				push @cc, $1;
+			chomp;
+			if (!defined $input_format && /^[-A-Za-z]+:\s/) {
+				$input_format = 'mbox';
 			}
-			elsif (/^Cc:\s+(.*)$/) {
-				foreach my $addr (parse_address_line($1)) {
-					if (unquote_rfc2047($addr) eq $sender) {
-						next if ($suppress_cc{'self'});
-					} else {
-						next if ($suppress_cc{'cc'});
-					}
+
+			if (defined $input_format && $input_format eq 'mbox') {
+				if (/^Subject:\s+(.*)$/) {
+					$subject = $1;
+				}
+				elsif (/^From:\s+(.*)$/) {
+					($author, $author_encoding) = unquote_rfc2047($1);
+					next if $suppress_cc{'author'};
+					next if $suppress_cc{'self'} and $author eq $sender;
 					printf("(mbox) Adding cc: %s from line '%s'\n",
-						$addr, $_) unless $quiet;
-					push @cc, $addr;
+					       $1, $_) unless $quiet;
+					push @cc, $1;
 				}
-			}
-			elsif (/^Content-type:/i) {
-				$has_content_type = 1;
-				if (/charset="?([^ "]+)/) {
-					$body_encoding = $1;
+				elsif (/^Cc:\s+(.*)$/) {
+					foreach my $addr (parse_address_line($1)) {
+						if (unquote_rfc2047($addr) eq $sender) {
+							next if ($suppress_cc{'self'});
+						} else {
+							next if ($suppress_cc{'cc'});
+						}
+						printf("(mbox) Adding cc: %s from line '%s'\n",
+						       $addr, $_) unless $quiet;
+						push @cc, $addr;
+					}
+				}
+				elsif (/^Content-type:/i) {
+					$has_content_type = 1;
+					if (/charset="?([^ "]+)/) {
+						$body_encoding = $1;
+					}
+					push @xh, $_;
+				}
+				elsif (/^Message-Id: (.*)/i) {
+					$message_id = $1;
+				}
+				elsif (!/^Date:\s/ && /^[-A-Za-z]+:\s+\S/) {
+					push @xh, $_;
 				}
-				push @xh, $_;
-			}
-			elsif (/^Message-Id: (.*)/i) {
-				$message_id = $1;
-			}
-			elsif (!/^Date:\s/ && /^[-A-Za-z]+:\s+\S/) {
-				push @xh, $_;
-			}
 
-		} else {
-			# In the traditional
-			# "send lots of email" format,
-			# line 1 = cc
-			# line 2 = subject
-			# So let's support that, too.
-			$input_format = 'lots';
-			if (@cc == 0 && !$suppress_cc{'cc'}) {
-				printf("(non-mbox) Adding cc: %s from line '%s'\n",
-					$_, $_) unless $quiet;
-				push @cc, $_;
-			} elsif (!defined $subject) {
-				$subject = $_;
+			} else {
+				# In the traditional
+				# "send lots of email" format,
+				# line 1 = cc
+				# line 2 = subject
+				# So let's support that, too.
+				$input_format = 'lots';
+				if (@cc == 0 && !$suppress_cc{'cc'}) {
+					printf("(non-mbox) Adding cc: %s from line '%s'\n",
+					       $_, $_) unless $quiet;
+					push @cc, $_;
+				} elsif (!defined $subject) {
+					$subject = $_;
+				}
 			}
 		}
-	}
-	# Now parse the message body
-	while(<F>) {
-		$message .=  $_;
-		if (/^(Signed-off-by|Cc): (.*)$/i) {
-			chomp;
-			my ($what, $c) = ($1, $2);
-			chomp $c;
-			if ($c eq $sender) {
-				next if ($suppress_cc{'self'});
-			} else {
-				next if $suppress_cc{'sob'} and $what =~ /Signed-off-by/i;
-				next if $suppress_cc{'bodycc'} and $what =~ /Cc/i;
+		# Now parse the message body
+		while(<F>) {
+			last if /^From /; # Next message in file
+			$message .=  $_;
+			if (/^(Signed-off-by|Cc): (.*)$/i) {
+				chomp;
+				my ($what, $c) = ($1, $2);
+				chomp $c;
+				if ($c eq $sender) {
+					next if ($suppress_cc{'self'});
+				} else {
+					next if $suppress_cc{'sob'} and $what =~ /Signed-off-by/i;
+					next if $suppress_cc{'bodycc'} and $what =~ /Cc/i;
+				}
+				push @cc, $c;
+				printf("(body) Adding cc: %s from line '%s'\n",
+				       $c, $_) unless $quiet;
 			}
-			push @cc, $c;
-			printf("(body) Adding cc: %s from line '%s'\n",
-				$c, $_) unless $quiet;
 		}
-	}
-	close F;
 
-	if (defined $cc_cmd && !$suppress_cc{'cccmd'}) {
-		open(F, "$cc_cmd $t |")
-			or die "(cc-cmd) Could not execute '$cc_cmd'";
-		while(<F>) {
-			my $c = $_;
-			$c =~ s/^\s*//g;
-			$c =~ s/\n$//g;
-			next if ($c eq $sender and $suppress_from);
-			push @cc, $c;
-			printf("(cc-cmd) Adding cc: %s from: '%s'\n",
-				$c, $cc_cmd) unless $quiet;
+		if (defined $cc_cmd && !$suppress_cc{'cccmd'}) {
+			open(CC_CMD, "$cc_cmd $t |")
+			    or die "(cc-cmd) Could not execute '$cc_cmd'";
+			while(<CC_CMD>) {
+				my $c = $_;
+				$c =~ s/^\s*//g;
+				$c =~ s/\n$//g;
+				next if ($c eq $sender and $suppress_from);
+				push @cc, $c;
+				printf("(cc-cmd) Adding cc: %s from: '%s'\n",
+				       $c, $cc_cmd) unless $quiet;
+			}
+			close CC_CMD
+			    or die "(cc-cmd) failed to close pipe to '$cc_cmd'";
 		}
-		close F
-			or die "(cc-cmd) failed to close pipe to '$cc_cmd'";
-	}
 
-	if (defined $author and $author ne $sender) {
-		$message = "From: $author\n\n$message";
-		if (defined $author_encoding) {
-			if ($has_content_type) {
-				if ($body_encoding eq $author_encoding) {
-					# ok, we already have the right encoding
+		if (defined $author and $author ne $sender) {
+			$message = "From: $author\n\n$message";
+			if (defined $author_encoding) {
+				if ($has_content_type) {
+					if ($body_encoding eq $author_encoding) {
+						# ok, we already have the right encoding
+					}
+					else {
+						# uh oh, we should re-encode
+					}
 				}
 				else {
-					# uh oh, we should re-encode
+					push @xh,
+					'MIME-Version: 1.0',
+					"Content-Type: text/plain; charset=$author_encoding",
+					'Content-Transfer-Encoding: 8bit';
 				}
 			}
-			else {
-				push @xh,
-				  'MIME-Version: 1.0',
-				  "Content-Type: text/plain; charset=$author_encoding",
-				  'Content-Transfer-Encoding: 8bit';
-			}
 		}
-	}
 
-	$needs_confirm = (
-		$confirm eq "always" or
-		($confirm =~ /^(?:auto|cc)$/ && @cc) or
-		($confirm =~ /^(?:auto|compose)$/ && $compose && $message_num == 1));
-	$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && @cc);
+		$needs_confirm = (
+			$confirm eq "always" or
+			($confirm =~ /^(?:auto|cc)$/ && @cc) or
+			($confirm =~ /^(?:auto|compose)$/ && $compose && $message_num == 1));
+		$needs_confirm = "inform" if ($needs_confirm && $confirm_unconfigured && @cc);
 
-	@cc = (@initial_cc, @cc);
+		@cc = (@initial_cc, @cc);
 
-	send_message();
+		send_message();
 
-	# set up for the next message
-	if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
-		$reply_to = $message_id;
-		if (length $references > 0) {
-			$references .= "\n $message_id";
-		} else {
-			$references = "$message_id";
+		# set up for the next message
+		if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
+			$reply_to = $message_id;
+			if (length $references > 0) {
+				$references .= "\n $message_id";
+			} else {
+				$references = "$message_id";
+			}
 		}
+		$message_id = undef;
+		last unless (<F>);
 	}
-	$message_id = undef;
+	close F;
 }
 
 cleanup_compose_files();
-- 
wbr, Vitaly
Junio C Hamano· May 12, 2009, 23:27 UTC · re: Vitaly Mayatskikh · lore

Re: [PATCH] git-send-email: parse all messages in mbox

Vitaly Mayatskikh <v.mayatskih@gmail.com> writes:
Show 6 quoted lines
> Currently git-send-email sends all mbox in one email. This seems to be wrong,
> because mbox can contain several messages. For example,
> `git format-patch --stdout' forms mbox file with all patches in it.
>
> This patch allows git send-email to send several messages from one mbox
> separately.

I suspect nobody would comment on it because with reindentation this was extremely painful to review.

I _think_ the gist of your change is that (1) here you stop slurping the body when you see a line that begins with "From "...

> +		# Now parse the message body
> +		while(<F>) {
> +			last if /^From /; # Next message in file

... and (2) the "last unless (<F>)" at the end of the loop where you attempt to see if there is any more lines to be read from the stream (and if so go back and continue from the header parsing).

Show 21 quoted lines
> -	# set up for the next message
> -	if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
> -		$reply_to = $message_id;
> -		if (length $references > 0) {
> -			$references .= "\n $message_id";
> -		} else {
> -			$references = "$message_id";
> +		# set up for the next message
> +		if ($chain_reply_to || !defined $reply_to || length($reply_to) == 0) {
> +			$reply_to = $message_id;
> +			if (length $references > 0) {
> +				$references .= "\n $message_id";
> +			} else {
> +				$references = "$message_id";
> +			}
>  		}
> +		$message_id = undef;
> +		last unless (<F>);
>  	}
> -	$message_id = undef;
> +	close F;

But I think the code structure is wrong. When you said "last if /^From /" earlier, you have already read that line (and you do not unread it), and here with "unless (<F>)" you are reading yet another line (one line after the UNIX-From line) and discarding it. The next round loses the real RFC2822 header line you discarded with this unless (<F>), and begins from the next line, and would break "First unfold multiline header fields" logic among other things.

But this is only from reviewing a patch with reindentation noise so I might have missed some subtle issues.

Can you make this into two patches for easier review? One to split out the existing loop for a single input stream into a helper function without changing any behaviour (i.e. the loop reads everything to the end), and then as a follow-up patch introduce "when we see a UNIX-From line we are at the beginning of the next message so return early" logic to the helper?

IOW, after the two-patch series, the current main-loop may look something like:

	my $unread_line = undef;
	while (1) {
		$unread_line = handle_one_stream(\*F, $unread_line);
                last if (!defined $unread_line);
	}
        close(\*F);
and your new handle_one_stream() sub will look something like:
	sub handle_one_stream {
		my ($fh, $last_line) = @_;
		local ($_);
		my $author = undef;
                my $author_encoding;
                my $has_content_type;
                my $body_encoding;
                @cc = ();
                @xh = ();
                my $input_format = undef;
                my @header = ();
                $message = "";
                $message_num++;
                # First unfold multiline header fields
                while (1) {
			if (defined $last_line) {
				$_ = $last_line;
                                $last_line = undef;
			} else {
                        	$_ = <F>;
			}
                        ...
		}
                # Now parse the header
                ...
                # Now parse the message body
                $last_line = undef;
                while (1) {
                	$_ = <F>;
			if (/^From /) {
				# This is the beginning of the
                                # next message; unread it.
				$last_line = $_;
                                last;
                        }
                        $message .= $_;
			...
		}
		return $last_line;
	}
Vitaly Mayatskikh· May 14, 2009, 09:47 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-send-email: parse all messages in mbox

At Tue, 12 May 2009 16:27:21 -0700, Junio C Hamano wrote:

Thanks for your review Junio! Once I'll found a free minute, I'll refresh my almost forgotten Perl knowledge and try to deuglify the patch.

-- 
wbr, Vitaly
Jay Soffian· May 14, 2009, 14:48 UTC · re: Vitaly Mayatskikh · lore

Re: [PATCH] git-send-email: parse all messages in mbox

On Mon, May 11, 2009 at 9:46 AM, Vitaly Mayatskikh <v.mayatskih@gmail.com> wrote:

> Currently git-send-email sends all mbox in one email. This seems to be wrong,
> because mbox can contain several messages. For example,
> `git format-patch --stdout' forms mbox file with all patches in it.
Just out of curiosity, what is the motivation for this patch?

By default, format-patch generates a file per commit, which is what send-email currently expects. You can also specify one or more commits to send-email directly, in which case it runs format-patch on your behalf. And, send-email cannot handle messages on stdin, so "git format-patch | git send-email" is not a valid workflow, even after this patch.

So the only way I can see this being useful is if you're doing something like:

$ git format-patch --stdout > mbox $ git send-email mbox

Or you can combine those in bash with:
$ git send-email <(git format-patch --stdout)

But, why? The only way this patch would make any sense to me is if it _also_ extended send-email to read its messages from stdin.

j.
Vitaly Mayatskikh· May 14, 2009, 18:41 UTC · re: Jay Soffian · lore

Re: [PATCH] git-send-email: parse all messages in mbox

At Thu, 14 May 2009 10:48:33 -0400, Jay Soffian wrote:
Show 26 quoted lines
> On Mon, May 11, 2009 at 9:46 AM, Vitaly Mayatskikh
> <v.mayatskih@gmail.com> wrote:
> > Currently git-send-email sends all mbox in one email. This seems to be wrong,
> > because mbox can contain several messages. For example,
> > `git format-patch --stdout' forms mbox file with all patches in it.
> 
> Just out of curiosity, what is the motivation for this patch?
> 
> By default, format-patch generates a file per commit, which is what
> send-email currently expects. You can also specify one or more commits
> to send-email directly, in which case it runs format-patch on your
> behalf. And, send-email cannot handle messages on stdin, so "git
> format-patch | git send-email" is not a valid workflow, even after
> this patch.
> 
> So the only way I can see this being useful is if you're doing something like:
> 
> $ git format-patch --stdout > mbox
> $ git send-email mbox
> 
> Or you can combine those in bash with:
> 
> $ git send-email <(git format-patch --stdout)
> 
> But, why? The only way this patch would make any sense to me is if it
> _also_ extended send-email to read its messages from stdin.

Because I want to edit cover letter, make --dry-run, etc. And it's more handy to me to have one entity (file) then a dozen of separate patches.

-- 
wbr, Vitaly

← back to recent threads