# [PATCH 0/3] GPG running out of pipes fixes 13 messages from 2013-01-31 to 2013-01-31. Participants: Stephen Boyd, Jeff King, Junio C Hamano, Jonathan Nieder. Thread: https://gitlist.dev/t/32787 ## Stephen Boyd, 2013-01-31 02:01 Subject: [PATCH 0/3] GPG running out of pipes fixes Message-ID: <1359597666-10108-1-git-send-email-sboyd@codeaurora.org> URL: https://gitlist.dev/e/1359597666-10108-1-git-send-email-sboyd%40codeaurora.org ``` While running --show-signatures on the linux kernel I noticed that after a while git failed with an error message indicating it had run out of file descriptors. The first patch fixes this problem, and the next two are randmom bits since I was in the area. Stephen Boyd (3): gpg: Close stderr once finished with it in verify_signed_buffer() run-command: Be more informative about what failed gpg: Allow translation of more error messages gpg-interface.c | 8 +++++--- run-command.c | 8 ++++++-- 2 files changed, 11 insertions(+), 5 deletions(-) -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation ``` ## Stephen Boyd, 2013-01-31 02:01 Subject: [PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer() Message-ID: <1359597666-10108-2-git-send-email-sboyd@codeaurora.org> URL: https://gitlist.dev/e/1359597666-10108-2-git-send-email-sboyd%40codeaurora.org In-Reply-To: <1359597666-10108-1-git-send-email-sboyd@codeaurora.org> ``` Failing to close the stderr pipe in verify_signed_buffer() causes git to run out of file descriptors if there are many calls to verify_signed_buffer(). An easy way to trigger this is to run git log --show-signature --merges | grep "key" on the linux kernel git repo. Eventually it will fail with error: cannot create pipe for gpg: Too many open files error: could not run gpg. Close the stderr pipe so that this can't happen. Signed-off-by: Stephen Boyd --- gpg-interface.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/gpg-interface.c b/gpg-interface.c index 0863c61..2c0bed3 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -133,6 +133,8 @@ int verify_signed_buffer(const char *payload, size_t payload_size, if (gpg_output) strbuf_read(gpg_output, gpg.err, 0); ret = finish_command(&gpg); + if (gpg_output) + close(gpg.err); unlink_or_warn(path); -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation ``` ## Stephen Boyd, 2013-01-31 02:01 Subject: [PATCH 2/3] run-command: Be more informative about what failed Message-ID: <1359597666-10108-3-git-send-email-sboyd@codeaurora.org> URL: https://gitlist.dev/e/1359597666-10108-3-git-send-email-sboyd%40codeaurora.org In-Reply-To: <1359597666-10108-1-git-send-email-sboyd@codeaurora.org> ``` While debugging an error with verify_signed_buffer() the error messages from run-command weren't very useful: error: cannot create pipe for gpg: Too many open files error: could not run gpg. because they didn't indicate *which* pipe couldn't be created. Print which pipe failed to be created in the error message so we can more easily debug similar problems in the future. For example, the above error now prints: error: cannot create stderr pipe for gpg: Too many open files error: could not run gpg. Signed-off-by: Stephen Boyd --- run-command.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/run-command.c b/run-command.c index 12d4ddb..016dd05 100644 --- a/run-command.c +++ b/run-command.c @@ -274,6 +274,7 @@ int start_command(struct child_process *cmd) int need_in, need_out, need_err; int fdin[2], fdout[2], fderr[2]; int failed_errno = failed_errno; + char *str; /* * In case of errors we must keep the promise to close FDs @@ -286,6 +287,7 @@ int start_command(struct child_process *cmd) failed_errno = errno; if (cmd->out > 0) close(cmd->out); + str = "stdin"; goto fail_pipe; } cmd->in = fdin[1]; @@ -301,6 +303,7 @@ int start_command(struct child_process *cmd) close_pair(fdin); else if (cmd->in) close(cmd->in); + str = "stdout"; goto fail_pipe; } cmd->out = fdout[0]; @@ -318,9 +321,10 @@ int start_command(struct child_process *cmd) close_pair(fdout); else if (cmd->out) close(cmd->out); + str = "stderr"; fail_pipe: - error("cannot create pipe for %s: %s", - cmd->argv[0], strerror(failed_errno)); + error("cannot create %s pipe for %s: %s", + str, cmd->argv[0], strerror(failed_errno)); errno = failed_errno; return -1; } -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation ``` ## Stephen Boyd, 2013-01-31 02:01 Subject: [PATCH 3/3] gpg: Allow translation of more error messages Message-ID: <1359597666-10108-4-git-send-email-sboyd@codeaurora.org> URL: https://gitlist.dev/e/1359597666-10108-4-git-send-email-sboyd%40codeaurora.org In-Reply-To: <1359597666-10108-1-git-send-email-sboyd@codeaurora.org> ``` Mark these strings for translation so that error messages are printed in the user's language of choice. Signed-off-by: Stephen Boyd --- gpg-interface.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/gpg-interface.c b/gpg-interface.c index 2c0bed3..474c037 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -109,10 +109,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size, args_gpg[0] = gpg_program; fd = git_mkstemp(path, PATH_MAX, ".git_vtag_tmpXXXXXX"); if (fd < 0) - return error("could not create temporary file '%s': %s", + return error(_("could not create temporary file '%s': %s"), path, strerror(errno)); if (write_in_full(fd, signature, signature_size) < 0) - return error("failed writing detached signature to '%s': %s", + return error(_("failed writing detached signature to '%s': %s"), path, strerror(errno)); close(fd); @@ -124,7 +124,7 @@ int verify_signed_buffer(const char *payload, size_t payload_size, args_gpg[2] = path; if (start_command(&gpg)) { unlink(path); - return error("could not run gpg."); + return error(_("could not run gpg.")); } write_in_full(gpg.in, payload, payload_size); -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation ``` ## Jeff King, 2013-01-31 05:50 Subject: Re: [PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer() Message-ID: <20130131055053.GA11912@sigill.intra.peff.net> URL: https://gitlist.dev/e/20130131055053.GA11912%40sigill.intra.peff.net In-Reply-To: <1359597666-10108-2-git-send-email-sboyd@codeaurora.org> ``` On Wed, Jan 30, 2013 at 06:01:04PM -0800, Stephen Boyd wrote: > Failing to close the stderr pipe in verify_signed_buffer() causes > git to run out of file descriptors if there are many calls to > verify_signed_buffer(). An easy way to trigger this is to run > > git log --show-signature --merges | grep "key" > > on the linux kernel git repo. Eventually it will fail with > > error: cannot create pipe for gpg: Too many open files > error: could not run gpg. > > Close the stderr pipe so that this can't happen. I was able to easily reproduce the bug and verify your fix here. > diff --git a/gpg-interface.c b/gpg-interface.c > index 0863c61..2c0bed3 100644 > --- a/gpg-interface.c > +++ b/gpg-interface.c > @@ -133,6 +133,8 @@ int verify_signed_buffer(const char *payload, size_t payload_size, > if (gpg_output) > strbuf_read(gpg_output, gpg.err, 0); > ret = finish_command(&gpg); > + if (gpg_output) > + close(gpg.err); The strbuf_read above will read to EOF, so it should be equivalent (and IMHO slightly more readable) to do: diff --git a/gpg-interface.c b/gpg-interface.c index 0863c61..5f142f6 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size, write_in_full(gpg.in, payload, payload_size); close(gpg.in); - if (gpg_output) + if (gpg_output) { strbuf_read(gpg_output, gpg.err, 0); + close(gpg.err); + } ret = finish_command(&gpg); unlink_or_warn(path); But that is a minor nit; either way, the patch looks good to me. -Peff ``` ## Junio C Hamano, 2013-01-31 16:24 Subject: Re: [PATCH 2/3] run-command: Be more informative about what failed Message-ID: <7vfw1hiami.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7vfw1hiami.fsf%40alter.siamese.dyndns.org In-Reply-To: <1359597666-10108-3-git-send-email-sboyd@codeaurora.org> ``` Stephen Boyd writes: > While debugging an error with verify_signed_buffer() the error > messages from run-command weren't very useful: > > error: cannot create pipe for gpg: Too many open files > error: could not run gpg. > > because they didn't indicate *which* pipe couldn't be created. For the message emitted here with your update (or without for that matter) to be useful, it has to hold that there is a single leaker, that leaker fails in this codepath, and that there is nobody else involved. Otherwise, you may be able to tell that one caller could not create its stdin, but the reason it couldn't may be because somebody else consumed all the available file descriptors. I am not opposed to this change per-se, but I am not sure that saying "stdin" etc. makes the message more useful for the purpose of debugging. > For example, the above error now prints: > > error: cannot create stderr pipe for gpg: Too many open files > error: could not run gpg. I'd prefer to see these names spelled out (e.g. "standard error") in any case. Thanks. > Signed-off-by: Stephen Boyd > --- > run-command.c | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff --git a/run-command.c b/run-command.c > index 12d4ddb..016dd05 100644 > --- a/run-command.c > +++ b/run-command.c > @@ -274,6 +274,7 @@ int start_command(struct child_process *cmd) > int need_in, need_out, need_err; > int fdin[2], fdout[2], fderr[2]; > int failed_errno = failed_errno; > + char *str; > > /* > * In case of errors we must keep the promise to close FDs > @@ -286,6 +287,7 @@ int start_command(struct child_process *cmd) > failed_errno = errno; > if (cmd->out > 0) > close(cmd->out); > + str = "stdin"; > goto fail_pipe; > } > cmd->in = fdin[1]; > @@ -301,6 +303,7 @@ int start_command(struct child_process *cmd) > close_pair(fdin); > else if (cmd->in) > close(cmd->in); > + str = "stdout"; > goto fail_pipe; > } > cmd->out = fdout[0]; > @@ -318,9 +321,10 @@ int start_command(struct child_process *cmd) > close_pair(fdout); > else if (cmd->out) > close(cmd->out); > + str = "stderr"; > fail_pipe: > - error("cannot create pipe for %s: %s", > - cmd->argv[0], strerror(failed_errno)); > + error("cannot create %s pipe for %s: %s", > + str, cmd->argv[0], strerror(failed_errno)); > errno = failed_errno; > return -1; > } ``` ## Stephen Boyd, 2013-01-31 18:05 Subject: Re: [PATCH 2/3] run-command: Be more informative about what failed Message-ID: <510AB255.40302@codeaurora.org> URL: https://gitlist.dev/e/510AB255.40302%40codeaurora.org In-Reply-To: <7vfw1hiami.fsf@alter.siamese.dyndns.org> ``` On 01/31/13 08:24, Junio C Hamano wrote: > Stephen Boyd writes: > >> While debugging an error with verify_signed_buffer() the error >> messages from run-command weren't very useful: >> >> error: cannot create pipe for gpg: Too many open files >> error: could not run gpg. >> >> because they didn't indicate *which* pipe couldn't be created. > For the message emitted here with your update (or without for that > matter) to be useful, it has to hold that there is a single leaker, > that leaker fails in this codepath, and that there is nobody else > involved. Otherwise, you may be able to tell that one caller could > not create its stdin, but the reason it couldn't may be because > somebody else consumed all the available file descriptors. > > I am not opposed to this change per-se, but I am not sure that > saying "stdin" etc. makes the message more useful for the purpose of > debugging. It helped me avoid firing up gdb, but if you don't see much use feel free to ignore this patch. > >> For example, the above error now prints: >> >> error: cannot create stderr pipe for gpg: Too many open files >> error: could not run gpg. > I'd prefer to see these names spelled out (e.g. "standard error") > in any case. Sure, I can do that. -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ``` ## Stephen Boyd, 2013-01-31 18:06 Subject: Re: [PATCH 1/3] gpg: Close stderr once finished with it in verify_signed_buffer() Message-ID: <510AB2AA.4000400@codeaurora.org> URL: https://gitlist.dev/e/510AB2AA.4000400%40codeaurora.org In-Reply-To: <20130131055053.GA11912@sigill.intra.peff.net> ``` On 01/30/13 21:50, Jeff King wrote: > > The strbuf_read above will read to EOF, so it should be equivalent (and > IMHO slightly more readable) to do: > > diff --git a/gpg-interface.c b/gpg-interface.c > index 0863c61..5f142f6 100644 > --- a/gpg-interface.c > +++ b/gpg-interface.c > @@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size, > write_in_full(gpg.in, payload, payload_size); > close(gpg.in); > > - if (gpg_output) > + if (gpg_output) { > strbuf_read(gpg_output, gpg.err, 0); > + close(gpg.err); > + } > ret = finish_command(&gpg); > > unlink_or_warn(path); > > But that is a minor nit; either way, the patch looks good to me. Looks better. I'll resend with this. -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation ``` ## Stephen Boyd, 2013-01-31 18:18 Subject: [PATCHv2 1/3] gpg: Close stderr once finished with it in verify_signed_buffer() Message-ID: <1359656320-4434-1-git-send-email-sboyd@codeaurora.org> URL: https://gitlist.dev/e/1359656320-4434-1-git-send-email-sboyd%40codeaurora.org In-Reply-To: <20130131055053.GA11912@sigill.intra.peff.net> ``` Failing to close the stderr pipe in verify_signed_buffer() causes git to run out of file descriptors if there are many calls to verify_signed_buffer(). An easy way to trigger this is to run git log --show-signature --merges | grep "key" on the linux kernel git repo. Eventually it will fail with error: cannot create pipe for gpg: Too many open files error: could not run gpg. Close the stderr pipe so that this can't happen. Suggested-by: Jeff King Signed-off-by: Stephen Boyd --- gpg-interface.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gpg-interface.c b/gpg-interface.c index 0863c61..5f142f6 100644 --- a/gpg-interface.c +++ b/gpg-interface.c @@ -130,8 +130,10 @@ int verify_signed_buffer(const char *payload, size_t payload_size, write_in_full(gpg.in, payload, payload_size); close(gpg.in); - if (gpg_output) + if (gpg_output) { strbuf_read(gpg_output, gpg.err, 0); + close(gpg.err); + } ret = finish_command(&gpg); unlink_or_warn(path); -- The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, hosted by The Linux Foundation ``` ## Jonathan Nieder, 2013-01-31 18:20 Subject: Re: [PATCH 3/3] gpg: Allow translation of more error messages Message-ID: <20130131182052.GA27340@google.com> URL: https://gitlist.dev/e/20130131182052.GA27340%40google.com In-Reply-To: <1359597666-10108-4-git-send-email-sboyd@codeaurora.org> ``` Stephen Boyd wrote: > Mark these strings for translation so that error messages are > printed in the user's language of choice. Yeah. Other error messages in this file are already translated, so it's oddly inconsistent. Assuming this passes tests with GETTEXT_POISON=YesPlease, Reviewed-by: Jonathan Nieder ``` ## Jeff King, 2013-01-31 22:35 Subject: Re: [PATCH 2/3] run-command: Be more informative about what failed Message-ID: <20130131223559.GC21729@sigill.intra.peff.net> URL: https://gitlist.dev/e/20130131223559.GC21729%40sigill.intra.peff.net In-Reply-To: <7vfw1hiami.fsf@alter.siamese.dyndns.org> ``` On Thu, Jan 31, 2013 at 08:24:21AM -0800, Junio C Hamano wrote: > Stephen Boyd writes: > > > While debugging an error with verify_signed_buffer() the error > > messages from run-command weren't very useful: > > > > error: cannot create pipe for gpg: Too many open files > > error: could not run gpg. > > > > because they didn't indicate *which* pipe couldn't be created. > > For the message emitted here with your update (or without for that > matter) to be useful, it has to hold that there is a single leaker, > that leaker fails in this codepath, and that there is nobody else > involved. Otherwise, you may be able to tell that one caller could > not create its stdin, but the reason it couldn't may be because > somebody else consumed all the available file descriptors. > > I am not opposed to this change per-se, but I am not sure that > saying "stdin" etc. makes the message more useful for the purpose of > debugging. Yeah, I had the same feeling. All that failed is pipe(), which does not have anything to do with what we are going to use the pipe for. So it gives some context, perhaps, but does not necessarily tell us anything useful. But it is not much code, and sometimes it is surprising what information can be helpful when debugging, so like you, I am not opposed, just doubtful. -Peff ``` ## Jeff King, 2013-01-31 22:37 Subject: Re: [PATCHv2 1/3] gpg: Close stderr once finished with it in verify_signed_buffer() Message-ID: <20130131223710.GD21729@sigill.intra.peff.net> URL: https://gitlist.dev/e/20130131223710.GD21729%40sigill.intra.peff.net In-Reply-To: <1359656320-4434-1-git-send-email-sboyd@codeaurora.org> ``` On Thu, Jan 31, 2013 at 10:18:40AM -0800, Stephen Boyd wrote: > Failing to close the stderr pipe in verify_signed_buffer() causes > git to run out of file descriptors if there are many calls to > verify_signed_buffer(). An easy way to trigger this is to run > > git log --show-signature --merges | grep "key" > > on the linux kernel git repo. Eventually it will fail with > > error: cannot create pipe for gpg: Too many open files > error: could not run gpg. > > Close the stderr pipe so that this can't happen. > > Suggested-by: Jeff King > Signed-off-by: Stephen Boyd Thanks, looks good to me (obviously :) ). The rest of the series looks fine, too, with the caveat I mentioned on 2/3. Thanks for fixing this. -Peff ``` ## Junio C Hamano, 2013-01-31 23:57 Subject: Re: [PATCH 2/3] run-command: Be more informative about what failed Message-ID: <7v8v78ewhx.fsf@alter.siamese.dyndns.org> URL: https://gitlist.dev/e/7v8v78ewhx.fsf%40alter.siamese.dyndns.org In-Reply-To: <20130131223559.GC21729@sigill.intra.peff.net> ``` Jeff King writes: > But it is not much code, and sometimes it is surprising what information > can be helpful when debugging, so like you, I am not opposed, just > doubtful. Yes, exactly my feeling. Perhaps I should just amend the 'stdin' and friends away without asking Stephen to reroll. In the other two I did not see any issues. I've queued all three of them including this one but as separate topics. Thanks. ```