threads / patch / 61641

patch, 2 partscat-file related doc and test

Subject: [PATCH 0/2] cat-file related doc and test

## tl;dr

16 messages between Jun 17, 2024 and Jun 24, 2024. Diffs are folded; open one to read it.

replies: 15people: 4as markdown or json

Eric Wong· Jun 17, 2024, 10:43 UTC · lore

I was working on reducing syscalls required for cat-file --batch (and readers) in a different patch, but noticed a bug in my yet-to-be-published patch wasn't detected in our test suite, only via 3rd-party Perl code.

Then I noticed Git.pm documentation was wrong..., so fixes are in reverse order.

Eric Wong (2):
  Git.pm: use array in command_bidi_pipe example
  t9700: ensure cat-file info isn't buffered by default
 perl/Git.pm     |  4 ++--
 t/t9700/test.pl | 14 ++++++++++++++
 2 files changed, 16 insertions(+), 2 deletions(-)
Eric Wong· Jun 17, 2024, 10:43 UTC · re: Eric Wong · lore

[PATCH 1/2] Git.pm: use array in command_bidi_pipe example

command_bidi_pipe takes the git command and optional arguments as an array, not a string. Make sure the documentation example is usable code.

Signed-off-by: Eric Wong <e@80x24.org>
---
 perl/Git.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to perl/Git.pm +2 −2
diff --git a/perl/Git.pm b/perl/Git.pm
index 03bf570bf4..aebfe0c6e0 100644
--- a/perl/Git.pm
+++ b/perl/Git.pm
@@ -418,7 +418,7 @@ sub command_bidi_pipe {
 and it is the fourth value returned by C<command_bidi_pipe()>.  The call idiom
 is:
 
-	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe('cat-file --batch-check');
+	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
 	print $out "000000000\n";
 	while (<$in>) { ... }
 	$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
@@ -431,7 +431,7 @@ sub command_bidi_pipe {
 calling this function.  This may be useful in a query-response type of
 commands where caller first writes a query and later reads response, eg:
 
-	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe('cat-file --batch-check');
+	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
 	print $out "000000000\n";
 	close $out;
 	while (<$in>) { ... }
Junio C Hamano· Jun 17, 2024, 20:33 UTC · re: Eric Wong · lore

Re: [PATCH 1/2] Git.pm: use array in command_bidi_pipe example

Eric Wong <e@80x24.org> writes:
> command_bidi_pipe takes the git command and optional arguments as an
> array, not a string.  Make sure the documentation example is usable
> code.
Makes sense.
Show 28 quoted lines
>
> Signed-off-by: Eric Wong <e@80x24.org>
> ---
>  perl/Git.pm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/perl/Git.pm b/perl/Git.pm
> index 03bf570bf4..aebfe0c6e0 100644
> --- a/perl/Git.pm
> +++ b/perl/Git.pm
> @@ -418,7 +418,7 @@ sub command_bidi_pipe {
>  and it is the fourth value returned by C<command_bidi_pipe()>.  The call idiom
>  is:
>  
> -	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe('cat-file --batch-check');
> +	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
>  	print $out "000000000\n";
>  	while (<$in>) { ... }
>  	$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
> @@ -431,7 +431,7 @@ sub command_bidi_pipe {
>  calling this function.  This may be useful in a query-response type of
>  commands where caller first writes a query and later reads response, eg:
>  
> -	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe('cat-file --batch-check');
> +	my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
>  	print $out "000000000\n";
>  	close $out;
>  	while (<$in>) { ... }
Eric Wong· Jun 17, 2024, 10:43 UTC · re: Eric Wong · lore

[PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

While working on buffering changes to `git cat-file' in a separate patch, I inadvertently made the output of --batch-check and the `info' command of --batch-command buffered by default.

Buffering by default breaks some 3rd-party Perl scripts using cat-file, but this breakage was not detected anywhere in our test suite. The easiest place to test this behavior is with Git.pm, since (AFAIK) other equivalent way to test this behavior from Bourne shell and/or awk would require racy sleeps, non-portable FIFOs or tedious C code.

Signed-off-by: Eric Wong <e@80x24.org>
---
 t/t9700/test.pl | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
Show changes to t/t9700/test.pl +14 −0
diff --git a/t/t9700/test.pl b/t/t9700/test.pl
index d8e85482ab..94a2e2c09d 100755
--- a/t/t9700/test.pl
+++ b/t/t9700/test.pl
@@ -154,6 +154,20 @@ sub adjust_dirsep {
 		     "abc\"\\ \x07\x08\x09\x0a\x0b\x0c\x0d\x01 ",
 		     'unquote escape sequences');
 
+# ensure --batch-check is unbuffered by default
+my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
+print $out $file1hash, "\n" or die $!;
+my $info = <$in>;
+is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-check';
+$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
+
+# ditto with `info' with --batch-command
+($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-command));
+print $out 'info ', $file1hash, "\n" or die $!;
+$info = <$in>;
+is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-command=info';
+$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
+
 printf "1..%d\n", Test::More->builder->current_test;
 
 my $is_passing = eval { Test::More->is_passing };
Junio C Hamano· Jun 17, 2024, 20:50 UTC · re: Eric Wong · lore

Re: [PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

Eric Wong <e@80x24.org> writes:
Show 6 quoted lines
> Buffering by default breaks some 3rd-party Perl scripts using
> cat-file, but this breakage was not detected anywhere in our
> test suite.  The easiest place to test this behavior is with
> Git.pm, since (AFAIK) other equivalent way to test this behavior
> from Bourne shell and/or awk would require racy sleeps,
> non-portable FIFOs or tedious C code.

Yes, using Perl is a good substitute for writing it in C in this case. I however question the choice to use t9700/test.pl here, which is clearly stated that its purpose is to "test perl interface which is Git.pm", and added tests are not testing anything in Git.pm at all.

Using t9700/test.pl only because it happens to use "perl -MTest::More" sounds a bit eh, suboptimal.

It seems that there are Perl snippets in other tests (including t1006 that is specifically about cat-file). How involved would it be to implement these new tests without modifying unrelated test scripts?

Show 30 quoted lines
> Signed-off-by: Eric Wong <e@80x24.org>
> ---
>  t/t9700/test.pl | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/t/t9700/test.pl b/t/t9700/test.pl
> index d8e85482ab..94a2e2c09d 100755
> --- a/t/t9700/test.pl
> +++ b/t/t9700/test.pl
> @@ -154,6 +154,20 @@ sub adjust_dirsep {
>  		     "abc\"\\ \x07\x08\x09\x0a\x0b\x0c\x0d\x01 ",
>  		     'unquote escape sequences');
>  
> +# ensure --batch-check is unbuffered by default
> +my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
> +print $out $file1hash, "\n" or die $!;
> +my $info = <$in>;
> +is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-check';
> +$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
> +
> +# ditto with `info' with --batch-command
> +($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-command));
> +print $out 'info ', $file1hash, "\n" or die $!;
> +$info = <$in>;
> +is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-command=info';
> +$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
> +
>  printf "1..%d\n", Test::More->builder->current_test;
>  
>  my $is_passing = eval { Test::More->is_passing };
Eric Wong· Jun 18, 2024, 21:30 UTC · re: Junio C Hamano · lore

[PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

Junio C Hamano <gitster@pobox.com> wrote:
Show 17 quoted lines
> Eric Wong <e@80x24.org> writes:
> 
> > Buffering by default breaks some 3rd-party Perl scripts using
> > cat-file, but this breakage was not detected anywhere in our
> > test suite.  The easiest place to test this behavior is with
> > Git.pm, since (AFAIK) other equivalent way to test this behavior
> > from Bourne shell and/or awk would require racy sleeps,
> > non-portable FIFOs or tedious C code.
> 
> Yes, using Perl is a good substitute for writing it in C in this
> case.  I however question the choice to use t9700/test.pl here,
> which is clearly stated that its purpose is to "test perl interface
> which is Git.pm", and added tests are not testing anything in Git.pm
> at all.
> 
> Using t9700/test.pl only because it happens to use "perl -MTest::More"
> sounds a bit eh, suboptimal.

*shrug* I figure Test::More is common enough since it's part of the Perl standard library; but I consider Perl a better scripting language than sh by far and wish our whole test suite were Perl :>

> It seems that there are Perl snippets in other tests (including
> t1006 that is specifically about cat-file).  How involved would it
> be to implement these new tests without modifying unrelated test
> scripts?
> >  t/t9700/test.pl | 14 ++++++++++++++
> >  1 file changed, 14 insertions(+)

More code than that. At least IPC::Open2 takes care of the nasty portability bits, but getting the Perl quoting nested properly inside sh was confusing :x

v2: moved test to t1006 to avoid Test::More,
    add select timeout in case a buffering bug does get introduced,
    updated commit message and clarified the bug it's supposed
    to guard against
    (I initially tried stdio buffering, but moved away from it for the
    patch I'm testing...)
----8<----
Subject: [PATCH] t1006: ensure cat-file info isn't buffered by default

While working on buffering changes to `git cat-file' in a separate patch, I inadvertently made the output of --batch-check and the `info' command of --batch-command buffered as if opt->buffer_output is turned on by default.

Buffering by default breaks some 3rd-party Perl scripts using cat-file, but this breakage was not detected anywhere in our test suite. Add a small Perl snippet to test this problem since (AFAIK) other equivalent ways to test this behavior from Bourne shell and/or awk would require racy sleeps, non-portable FIFOs or tedious C code.

Signed-off-by: Eric Wong <e@80x24.org>
---
 t/t1006-cat-file.sh | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
Show changes to t/t1006-cat-file.sh +30 −0
diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh
index e12b221972..ff9bf213aa 100755
--- a/t/t1006-cat-file.sh
+++ b/t/t1006-cat-file.sh
@@ -1294,4 +1294,34 @@ test_expect_success 'batch-command flush without --buffer' '
 	grep "^fatal:.*flush is only for --buffer mode.*" err
 '
 
+script='
+use warnings;
+use strict;
+use IPC::Open2;
+my ($opt, $oid, $expect, @pfx) = @ARGV;
+my @cmd = (qw(git cat-file), $opt);
+my $pid = open2(my $out, my $in, @cmd) or die "open2: @cmd";
+print $in @pfx, $oid, "\n" or die "print $!";
+my $rvec = "";
+vec($rvec, fileno($out), 1) = 1;
+select($rvec, undef, undef, 30) or die "no response to `@pfx $oid` from @cmd";
+my $info = <$out>;
+chop($info) eq "\n" or die "no LF";
+$info eq $expect or die "`$info` != `$expect`";
+close $in or die "close in $!";
+close $out or die "close out $!";
+waitpid $pid, 0;
+$? == 0 or die "\$?=$?";
+'
+
+expect="$hello_oid blob $hello_size"
+
+test_expect_success PERL '--batch-check is unbuffered by default' '
+	perl -e "$script" -- --batch-check $hello_oid "$expect"
+'
+
+test_expect_success PERL '--batch-command info is unbuffered by default' '
+	perl -e "$script" -- --batch-command $hello_oid "$expect" "info "
+'
+
 test_done
Junio C Hamano· Jun 18, 2024, 23:37 UTC · re: Eric Wong · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

Eric Wong <e@80x24.org> writes:
Show 12 quoted lines
>> Yes, using Perl is a good substitute for writing it in C in this
>> case.  I however question the choice to use t9700/test.pl here,
>> which is clearly stated that its purpose is to "test perl interface
>> which is Git.pm", and added tests are not testing anything in Git.pm
>> at all.
>> 
>> Using t9700/test.pl only because it happens to use "perl -MTest::More"
>> sounds a bit eh, suboptimal.
>
> *shrug*  I figure Test::More is common enough since it's part of
> the Perl standard library; but I consider Perl a better scripting
> language than sh by far and wish our whole test suite were Perl :>

Oh, I think we (actually the author of t9700) considers it common enough that we have PERL_TEST_MORE prerequisite to allow us to write tests, assuming that it is available, and let us easily skip where it is not available. So I do not think I mind the dependency on Test::More at all. Moving the tests to t1006 and rewriting the tests not to use Test::More are two separate and unrelated things, and if you are more comfortable with Test::More (and more importantly if it is natural to write Perl based tests using Test::More), it is not necessary to switch away from it.

Eric Wong· Jun 19, 2024, 17:56 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

Junio C Hamano <gitster@pobox.com> wrote:
Show 24 quoted lines
> Eric Wong <e@80x24.org> writes:
> 
> >> Yes, using Perl is a good substitute for writing it in C in this
> >> case.  I however question the choice to use t9700/test.pl here,
> >> which is clearly stated that its purpose is to "test perl interface
> >> which is Git.pm", and added tests are not testing anything in Git.pm
> >> at all.
> >> 
> >> Using t9700/test.pl only because it happens to use "perl -MTest::More"
> >> sounds a bit eh, suboptimal.
> >
> > *shrug*  I figure Test::More is common enough since it's part of
> > the Perl standard library; but I consider Perl a better scripting
> > language than sh by far and wish our whole test suite were Perl :>
> 
> Oh, I think we (actually the author of t9700) considers it common
> enough that we have PERL_TEST_MORE prerequisite to allow us to write
> tests, assuming that it is available, and let us easily skip where
> it is not available.  So I do not think I mind the dependency on
> Test::More at all.  Moving the tests to t1006 and rewriting the
> tests not to use Test::More are two separate and unrelated things,
> and if you are more comfortable with Test::More (and more
> importantly if it is natural to write Perl based tests using
> Test::More), it is not necessary to switch away from it.

OK, fair enough. Given t1006 is mostly sh, I prefer keeping v2 as-is because the Test::More->builder munging of test numbers in t9700/test.pl is nasty too and I wouldn't enjoy duplicating those bits in a hypothetical t1006/test.pl, either.

It would be nice to have first class support for Test::More in our suite so we could just have t/t0006-cat-file.t and t/t9700-perl-git.t implemented in Perl without sh at all, but that's a separate discussion.

Junio C Hamano· Jun 20, 2024, 17:28 UTC · re: Eric Wong · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

Eric Wong <e@80x24.org> writes:
> OK, fair enough.  Given t1006 is mostly sh, I prefer keeping v2
> as-is because the Test::More->builder munging of test numbers in
> t9700/test.pl is nasty too and I wouldn't enjoy duplicating
> those bits in a hypothetical t1006/test.pl, either.
That is quite sensible.
Jeff King· Jun 21, 2024, 07:16 UTC · re: Eric Wong · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

On Tue, Jun 18, 2024 at 09:30:41PM +0000, Eric Wong wrote:
Show 25 quoted lines
> +script='
> +use warnings;
> +use strict;
> +use IPC::Open2;
> +my ($opt, $oid, $expect, @pfx) = @ARGV;
> +my @cmd = (qw(git cat-file), $opt);
> +my $pid = open2(my $out, my $in, @cmd) or die "open2: @cmd";
> +print $in @pfx, $oid, "\n" or die "print $!";
> +my $rvec = "";
> +vec($rvec, fileno($out), 1) = 1;
> +select($rvec, undef, undef, 30) or die "no response to `@pfx $oid` from @cmd";
> +my $info = <$out>;
> +chop($info) eq "\n" or die "no LF";
> +$info eq $expect or die "`$info` != `$expect`";
> +close $in or die "close in $!";
> +close $out or die "close out $!";
> +waitpid $pid, 0;
> +$? == 0 or die "\$?=$?";
> +'
> +
> +expect="$hello_oid blob $hello_size"
> +
> +test_expect_success PERL '--batch-check is unbuffered by default' '
> +	perl -e "$script" -- --batch-check $hello_oid "$expect"
> +'

We often use "perl -e" for one-liners, etc, but this is pretty big. Maybe:

  cat >foo.pl <<-\EOF
  ...
  EOF
  perl foo.pl -- ...

would be more readable? To be clear I don't think there's anything incorrect about your usage, but it would match the style of our suite a bit better.

Likewise, it would be usual in our suite for the helper to do the minimum that needs to be in perl, and use our normal functions for things like comparing output (rather than taking its own "expect" argument).

So maybe:
Show changes to t/t1006-cat-file.sh +29 −1
diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh
index e12b221972..929d7a7579 100755
--- a/t/t1006-cat-file.sh
+++ b/t/t1006-cat-file.sh
@@ -1294,4 +1294,33 @@ test_expect_success 'batch-command flush without --buffer' '
 	grep "^fatal:.*flush is only for --buffer mode.*" err
 '
 
+# Copy a single line from stdin to the program specified
+# by @ARGV, and then wait for a response _without_ closing
+# the pipe.
+cat >run-and-wait.pl <<-\EOF
+use IPC::Open2;
+open2(my $out, my $in, @ARGV) or die "open2: @ARGV";
+print $in scalar(<STDIN>) or die "print $!";
+
+my $rvec = "";
+vec($rvec, fileno($out), 1) = 1;
+select($rvec, undef, undef, 30) or die "no response after 30 seconds";
+
+print scalar(<$out>);
+EOF
+
+test_expect_success PERL '--batch-check is unbuffered by default' '
+	echo "$hello_oid" |
+	perl run-and-wait.pl git cat-file --batch-check >out &&
+	echo "$hello_oid blob $hello_size" >expect &&
+	test_cmp expect out
+'
+
+test_expect_success PERL '--batch-command info is unbuffered by default' '
+	echo "info $hello_oid" |
+	perl run-and-wait.pl git cat-file --batch-command >out &&
+	echo "$hello_oid blob $hello_size" >expect &&
+	test_cmp expect out
+'
+
 test_done

I went for brevity above. Notably missing are:

  - the use of strict/warnings. I think we've shied away from these in
    the test suite because we want to run on any version of perl. In my
    experience most strict/warnings output is actually telling you about
    obvious garbage, but not always. IIRC perl got more strict about
    "()" around lists in some contexts a few years back, and code which
    used to be OK started generating warnings. OTOH, those warnings were
    probably a sign of problems-to-come, anyway. Without "FATAL",
    though, I think "use warnings" is not doing much good (nobody is
    ever going to see its output if the test isn't failing).

  - I dropped the close/waitpid. I guess maybe it is valuable to confirm
    that cat-file did not barf, but IMHO the important thing here is
    testing that it produced the single line of output we expected.

-Peff
Eric Wong· Jun 21, 2024, 20:00 UTC · re: Jeff King · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

Jeff King <peff@peff.net> wrote:
> On Tue, Jun 18, 2024 at 09:30:41PM +0000, Eric Wong wrote:
> 
> > +script='
<snip>
Show 17 quoted lines
> > +expect="$hello_oid blob $hello_size"
> > +
> > +test_expect_success PERL '--batch-check is unbuffered by default' '
> > +	perl -e "$script" -- --batch-check $hello_oid "$expect"
> > +'
> 
> We often use "perl -e" for one-liners, etc, but this is pretty big.
> Maybe:
> 
>   cat >foo.pl <<-\EOF
>   ...
>   EOF
>   perl foo.pl -- ...
> 
> would be more readable? To be clear I don't think there's anything
> incorrect about your usage, but it would match the style of our suite a
> bit better.

*shrug* It doesn't save the nested quoting/expansion confusion; but it's Junio's call. I don't think a v3 is worth the effort.

> Likewise, it would be usual in our suite for the helper to do the
> minimum that needs to be in perl, and use our normal functions for
> things like comparing output (rather than taking its own "expect"
> argument).
<snip>
Show 5 quoted lines
> +test_expect_success PERL '--batch-check is unbuffered by default' '
> +	echo "$hello_oid" |
> +	perl run-and-wait.pl git cat-file --batch-check >out &&
> +	echo "$hello_oid blob $hello_size" >expect &&
> +	test_cmp expect out

I prefer to avoid process spawning overhead from test_cmp; but that's a small drop in a big bucket.

Show 11 quoted lines
> I went for brevity above. Notably missing are:
> 
>   - the use of strict/warnings. I think we've shied away from these in
>     the test suite because we want to run on any version of perl. In my
>     experience most strict/warnings output is actually telling you about
>     obvious garbage, but not always. IIRC perl got more strict about
>     "()" around lists in some contexts a few years back, and code which
>     used to be OK started generating warnings. OTOH, those warnings were
>     probably a sign of problems-to-come, anyway. Without "FATAL",
>     though, I think "use warnings" is not doing much good (nobody is
>     ever going to see its output if the test isn't failing).

It may make problems easier to find if there are failures, so I think the potential benefits outweight any downsides.

>   - I dropped the close/waitpid. I guess maybe it is valuable to confirm
>     that cat-file did not barf, but IMHO the important thing here is
>     testing that it produced the single line of output we expected.

I've found some unexpected bugs through excessive error checking in the past, so much preferred to keep them.

Jeff King· Jun 24, 2024, 15:19 UTC · re: Eric Wong · lore

Re: [PATCH v2 2/2] t1006: ensure cat-file info isn't buffered by default

On Fri, Jun 21, 2024 at 08:00:02PM +0000, Eric Wong wrote:
Show 14 quoted lines
> > We often use "perl -e" for one-liners, etc, but this is pretty big.
> > Maybe:
> > 
> >   cat >foo.pl <<-\EOF
> >   ...
> >   EOF
> >   perl foo.pl -- ...
> > 
> > would be more readable? To be clear I don't think there's anything
> > incorrect about your usage, but it would match the style of our suite a
> > bit better.
> 
> *shrug*  It doesn't save the nested quoting/expansion confusion;
> but it's Junio's call.  I don't think a v3 is worth the effort.

It does allow you to use single quotes in the script, though I think you managed without it.

Show 8 quoted lines
> > +test_expect_success PERL '--batch-check is unbuffered by default' '
> > +	echo "$hello_oid" |
> > +	perl run-and-wait.pl git cat-file --batch-check >out &&
> > +	echo "$hello_oid blob $hello_size" >expect &&
> > +	test_cmp expect out
> 
> I prefer to avoid process spawning overhead from test_cmp;
> but that's a small drop in a big bucket.

If we care about that, I'd rather see us make test_cmp zero-process with a shell helper than come up with ad-hoc solutions. I've tried to measure something like that before, but couldn't come up with any conclusive improvements (my findings were mostly that running Git itself accounts for most of the process overhead).

-Peff
Junio C Hamano· Jun 17, 2024, 23:08 UTC · re: Eric Wong · lore

Re: [PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

Eric Wong <e@80x24.org> writes:
> While working on buffering changes to `git cat-file' in a
> separate patch, I inadvertently made the output of --batch-check
> and the `info' command of --batch-command buffered by default.

Here "buffered" means "as if opt->buffer_output is turned on", which in turn means "the output goes through stdio"? Just making sure that my understanding of what the breakage was is in line with what you wanted to convey.

Thanks.
Phillip Wood· Jun 19, 2024, 09:08 UTC · re: Eric Wong · lore

Re: [PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

Hi Eric
On 17/06/2024 11:43, Eric Wong wrote:
> +# ensure --batch-check is unbuffered by default
> +my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
> +print $out $file1hash, "\n" or die $!;

It's been a while since I did any perl scripting and I'm not clear whether $out is buffered or not and if it is whether it is guaranteed to be flushed when we print "\n". It might be worth adding a explicit flush so it is clear that any deadlocks come from cat-file and not our test code.

> +my $info = <$in>;

Is there an easy way to add a timeout to this read so that the failure mode isn't "the test hangs without printing anything"? I'm not sure that failure mode is easy to diagnose from our CI output as it is hard to tell which test caused the CI to timeout and it takes ages for the CI to time out.

Best Wishes
Phillip
Show 14 quoted lines
> +is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-check';
> +$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
> +
> +# ditto with `info' with --batch-command
> +($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-command));
> +print $out 'info ', $file1hash, "\n" or die $!;
> +$info = <$in>;
> +is $info, "$file1hash blob 15\n", 'command_bidi_pipe w/ --batch-command=info';
> +$r->command_close_bidi_pipe($pid, $in, $out, $ctx);
> +
>   printf "1..%d\n", Test::More->builder->current_test;
>   
>   my $is_passing = eval { Test::More->is_passing };
> 
Eric Wong· Jun 19, 2024, 18:08 UTC · re: Phillip Wood · lore

Re: [PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

Phillip Wood <phillip.wood123@gmail.com> wrote:
Show 11 quoted lines
> Hi Eric
> 
> On 17/06/2024 11:43, Eric Wong wrote:
> > +# ensure --batch-check is unbuffered by default
> > +my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
> > +print $out $file1hash, "\n" or die $!;
> 
> It's been a while since I did any perl scripting and I'm not clear whether
> $out is buffered or not and if it is whether it is guaranteed to be flushed
> when we print "\n". It might be worth adding a explicit flush so it is clear
> that any deadlocks come from cat-file and not our test code.

Pipes and sockets created by Perl are always unbuffered since 5.8, at least. If they were buffered, Git.pm users (including git-svn) wouldn't have worked at all.

Show 6 quoted lines
> > +my $info = <$in>;
> 
> Is there an easy way to add a timeout to this read so that the failure mode
> isn't "the test hangs without printing anything"? I'm not sure that failure
> mode is easy to diagnose from our CI output as it is hard to tell which test
> caused the CI to timeout and it takes ages for the CI to time out.
Yeah, select() has been added in v2.
Phillip Wood· Jun 21, 2024, 13:03 UTC · re: Eric Wong · lore

Re: [PATCH 2/2] t9700: ensure cat-file info isn't buffered by default

On 19/06/2024 19:08, Eric Wong wrote:
Show 16 quoted lines
> Phillip Wood <phillip.wood123@gmail.com> wrote:
>> Hi Eric
>>
>> On 17/06/2024 11:43, Eric Wong wrote:
>>> +# ensure --batch-check is unbuffered by default
>>> +my ($pid, $in, $out, $ctx) = $r->command_bidi_pipe(qw(cat-file --batch-check));
>>> +print $out $file1hash, "\n" or die $!;
>>
>> It's been a while since I did any perl scripting and I'm not clear whether
>> $out is buffered or not and if it is whether it is guaranteed to be flushed
>> when we print "\n". It might be worth adding a explicit flush so it is clear
>> that any deadlocks come from cat-file and not our test code.
> 
> Pipes and sockets created by Perl are always unbuffered since
> 5.8, at least.  If they were buffered, Git.pm users (including
> git-svn) wouldn't have worked at all.
Thanks for clarifying that
Show 8 quoted lines
>>> +my $info = <$in>;
>>
>> Is there an easy way to add a timeout to this read so that the failure mode
>> isn't "the test hangs without printing anything"? I'm not sure that failure
>> mode is easy to diagnose from our CI output as it is hard to tell which test
>> caused the CI to timeout and it takes ages for the CI to time out.
> 
> Yeah, select() has been added in v2.
That's much nicer.
Thanks
Phillip

← back to recent threads