threads / rfc / 47628

RFC patch, 2 partsCookie redaction during GIT_TRACE_CURL

Subject: [RFC PATCH 0/2] Cookie redaction during GIT_TRACE_CURL

## tl;dr

5 messages between Jan 18, 2018 and Jan 18, 2018. Diffs are folded; open one to read it.

replies: 4people: 2as markdown or json

Jonathan Tan· Jan 18, 2018, 00:34 UTC · lore

Sometimes authentication information is sent over HTTP through cookies, but when using GIT_TRACE_CURL, that information appears in logs. There are some HTTP headers already redacted ("Authorization:" and "Proxy-Authorization:") - the first patch extends such redaction to a user-specified list.

I've also included another patch to allow omission of data transmission information from being logged when using GIT_TRACE_CURL. This reduces the information logged to that similar to GIT_CURL_VERBOSE. (As for why not use GIT_CURL_VERBOSE instead - that is because GIT_CURL_VERBOSE does not perform any redaction, merely using Curl's default logging mechanism.)

The patches are ready for merging, but I marked this as "RFC" just in case there is a better way to accomplish this.

Jonathan Tan (2):
  http: support cookie redaction when tracing
  http: support omitting data from traces
 http.c                      | 82 ++++++++++++++++++++++++++++++++++++++++-----
 t/t5551-http-fetch-smart.sh | 24 +++++++++++++
 2 files changed, 98 insertions(+), 8 deletions(-)
-- 
2.16.0.rc1.238.g530d649a79-goog
Jonathan Tan· Jan 18, 2018, 00:34 UTC · re: Jonathan Tan · lore

[RFC PATCH 2/2] http: support omitting data from traces

GIT_TRACE_CURL provides a way to debug what is being sent and received over HTTP, with automatic redaction of sensitive information. But it also logs data transmissions, which significantly increases the log file size, sometimes unnecessarily. Add an option "GIT_TRACE_CURL_NO_DATA" to allow the user to omit such data transmissions.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 http.c                      | 27 +++++++++++++++++++--------
 t/t5551-http-fetch-smart.sh | 12 ++++++++++++
 2 files changed, 31 insertions(+), 8 deletions(-)
Show changes to 2 files +31 −8

http.c, t/t5551-http-fetch-smart.sh

diff --git a/http.c b/http.c
index 088cf70bf..32a823895 100644
--- a/http.c
+++ b/http.c
@@ -16,6 +16,7 @@
 #include "string-list.h"
 
 static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
+static int trace_curl_data = 1;
 static struct string_list cookies_to_redact = STRING_LIST_INIT_DUP;
 #if LIBCURL_VERSION_NUM >= 0x070a08
 long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER;
@@ -695,24 +696,32 @@ static int curl_trace(CURL *handle, curl_infotype type, char *data, size_t size,
 		curl_dump_header(text, (unsigned char *)data, size, DO_FILTER);
 		break;
 	case CURLINFO_DATA_OUT:
-		text = "=> Send data";
-		curl_dump_data(text, (unsigned char *)data, size);
+		if (trace_curl_data) {
+			text = "=> Send data";
+			curl_dump_data(text, (unsigned char *)data, size);
+		}
 		break;
 	case CURLINFO_SSL_DATA_OUT:
-		text = "=> Send SSL data";
-		curl_dump_data(text, (unsigned char *)data, size);
+		if (trace_curl_data) {
+			text = "=> Send SSL data";
+			curl_dump_data(text, (unsigned char *)data, size);
+		}
 		break;
 	case CURLINFO_HEADER_IN:
 		text = "<= Recv header";
 		curl_dump_header(text, (unsigned char *)data, size, NO_FILTER);
 		break;
 	case CURLINFO_DATA_IN:
-		text = "<= Recv data";
-		curl_dump_data(text, (unsigned char *)data, size);
+		if (trace_curl_data) {
+			text = "<= Recv data";
+			curl_dump_data(text, (unsigned char *)data, size);
+		}
 		break;
 	case CURLINFO_SSL_DATA_IN:
-		text = "<= Recv SSL data";
-		curl_dump_data(text, (unsigned char *)data, size);
+		if (trace_curl_data) {
+			text = "<= Recv SSL data";
+			curl_dump_data(text, (unsigned char *)data, size);
+		}
 		break;
 
 	default:		/* we ignore unknown types by default */
@@ -857,6 +866,8 @@ static CURL *get_curl_handle(void)
 	if (getenv("GIT_CURL_VERBOSE"))
 		curl_easy_setopt(result, CURLOPT_VERBOSE, 1L);
 	setup_curl_trace(result);
+	if (getenv("GIT_TRACE_CURL_NO_DATA"))
+		trace_curl_data = 0;
 	if (getenv("GIT_REDACT_COOKIES")) {
 		string_list_split(&cookies_to_redact,
 				  getenv("GIT_REDACT_COOKIES"), ',', -1);
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index 88bd9c094..0c62d00af 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -376,5 +376,17 @@ test_expect_success 'GIT_REDACT_COOKIES redacts cookies' '
 	! grep "Cookie:.*Secret=Bar" err
 '
 
+test_expect_success 'GIT_TRACE_CURL_NO_DATA prevents data from being traced' '
+	rm -rf clone &&
+	GIT_TRACE_CURL=true \
+		git clone $HTTPD_URL/smart/repo.git clone 2>err &&
+	grep "=> Send data" err &&
+
+	rm -rf clone &&
+	GIT_TRACE_CURL=true GIT_TRACE_CURL_NO_DATA=1 \
+		git clone $HTTPD_URL/smart/repo.git clone 2>err &&
+	! grep "=> Send data" err
+'
+
 stop_httpd
 test_done
-- 
2.16.0.rc1.238.g530d649a79-goog
Eric Sunshine· Jan 18, 2018, 18:25 UTC · re: Jonathan Tan · lore

Re: [RFC PATCH 2/2] http: support omitting data from traces

On Wed, Jan 17, 2018 at 7:34 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
Show 11 quoted lines
> GIT_TRACE_CURL provides a way to debug what is being sent and received
> over HTTP, with automatic redaction of sensitive information. But it
> also logs data transmissions, which significantly increases the log file
> size, sometimes unnecessarily. Add an option "GIT_TRACE_CURL_NO_DATA" to
> allow the user to omit such data transmissions.
>
> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
> ---
>  http.c                      | 27 +++++++++++++++++++--------
>  t/t5551-http-fetch-smart.sh | 12 ++++++++++++
>  2 files changed, 31 insertions(+), 8 deletions(-)
Please document GIT_TRACE_CURL_NO_DATA in Documentation/git.txt.
Jonathan Tan· Jan 18, 2018, 00:34 UTC · re: Jonathan Tan · lore

[RFC PATCH 1/2] http: support cookie redaction when tracing

When using GIT_TRACE_CURL, Git already redacts the "Authorization:" and "Proxy-Authorization:" HTTP headers. Extend this redaction to a user-specified list of cookies, specified through the "GIT_REDACT_COOKIES" environment variable.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 http.c                      | 55 +++++++++++++++++++++++++++++++++++++++++++++
 t/t5551-http-fetch-smart.sh | 12 ++++++++++
 2 files changed, 67 insertions(+)
Show changes to 2 files +67 −0

http.c, t/t5551-http-fetch-smart.sh

diff --git a/http.c b/http.c
index 597771271..088cf70bf 100644
--- a/http.c
+++ b/http.c
@@ -13,8 +13,10 @@
 #include "transport.h"
 #include "packfile.h"
 #include "protocol.h"
+#include "string-list.h"
 
 static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
+static struct string_list cookies_to_redact = STRING_LIST_INIT_DUP;
 #if LIBCURL_VERSION_NUM >= 0x070a08
 long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER;
 #else
@@ -575,6 +577,54 @@ static void redact_sensitive_header(struct strbuf *header)
 		/* Everything else is opaque and possibly sensitive */
 		strbuf_setlen(header,  sensitive_header - header->buf);
 		strbuf_addstr(header, " <redacted>");
+	} else if (cookies_to_redact.nr &&
+		   skip_prefix(header->buf, "Cookie:", &sensitive_header)) {
+		struct strbuf redacted_header = STRBUF_INIT;
+		char *cookie;
+
+		while (isspace(*sensitive_header))
+			sensitive_header++;
+
+		/*
+		 * The contents of header starting from sensitive_header will
+		 * subsequently be overridden, so it is fine to mutate this
+		 * string (hence the assignment to "char *").
+		 */
+		cookie = (char *) sensitive_header;
+
+		while (cookie) {
+			char *equals;
+			char *semicolon = strstr(cookie, "; ");
+			if (semicolon)
+				*semicolon = 0;
+			equals = strchrnul(cookie, '=');
+			if (!equals) {
+				/* invalid cookie, just append and continue */
+				strbuf_addstr(&redacted_header, cookie);
+				continue;
+			}
+			*equals = 0; /* temporarily set to NUL for lookup */
+			if (string_list_lookup(&cookies_to_redact, cookie)) {
+				strbuf_addstr(&redacted_header, cookie);
+				strbuf_addstr(&redacted_header, "=<redacted>");
+			} else {
+				*equals = '=';
+				strbuf_addstr(&redacted_header, cookie);
+			}
+			if (semicolon) {
+				/*
+				 * There are more cookies. (Or, for some
+				 * reason, the input string ends in "; ".)
+				 */
+				strbuf_addstr(&redacted_header, "; ");
+				cookie = semicolon + strlen("; ");
+			} else {
+				cookie = NULL;
+			}
+		}
+
+		strbuf_setlen(header, sensitive_header - header->buf);
+		strbuf_addbuf(header, &redacted_header);
 	}
 }
 
@@ -807,6 +857,11 @@ static CURL *get_curl_handle(void)
 	if (getenv("GIT_CURL_VERBOSE"))
 		curl_easy_setopt(result, CURLOPT_VERBOSE, 1L);
 	setup_curl_trace(result);
+	if (getenv("GIT_REDACT_COOKIES")) {
+		string_list_split(&cookies_to_redact,
+				  getenv("GIT_REDACT_COOKIES"), ',', -1);
+		string_list_sort(&cookies_to_redact);
+	}
 
 	curl_easy_setopt(result, CURLOPT_USERAGENT,
 		user_agent ? user_agent : git_user_agent());
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index a51b7e20d..88bd9c094 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -364,5 +364,17 @@ test_expect_success 'custom http headers' '
 		submodule update sub
 '
 
+test_expect_success 'GIT_REDACT_COOKIES redacts cookies' '
+	rm -rf clone &&
+	echo "Set-Cookie: NotSecret=Foo" >cookies &&
+	echo "Set-Cookie: Secret=Bar" >>cookies &&
+	GIT_TRACE_CURL=true GIT_REDACT_COOKIES=Secret,AnotherSecret \
+		git -c "http.cookieFile=$(pwd)/cookies" clone \
+		$HTTPD_URL/smart/repo.git clone 2>err &&
+	grep "Cookie:.*NotSecret=Foo" err &&
+	grep "Cookie:.*Secret=<redacted>" err &&
+	! grep "Cookie:.*Secret=Bar" err
+'
+
 stop_httpd
 test_done
-- 
2.16.0.rc1.238.g530d649a79-goog
Eric Sunshine· Jan 18, 2018, 18:25 UTC · re: Jonathan Tan · lore

Re: [RFC PATCH 1/2] http: support cookie redaction when tracing

On Wed, Jan 17, 2018 at 7:34 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
Show 10 quoted lines
> When using GIT_TRACE_CURL, Git already redacts the "Authorization:" and
> "Proxy-Authorization:" HTTP headers. Extend this redaction to a
> user-specified list of cookies, specified through the
> "GIT_REDACT_COOKIES" environment variable.
>
> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
> ---
>  http.c                      | 55 +++++++++++++++++++++++++++++++++++++++++++++
>  t/t5551-http-fetch-smart.sh | 12 ++++++++++
>  2 files changed, 67 insertions(+)
Please document GIT_REDACT_COOKIES in Documentation/git.txt.
Show 13 quoted lines
> diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
> @@ -364,5 +364,17 @@ test_expect_success 'custom http headers' '
> +test_expect_success 'GIT_REDACT_COOKIES redacts cookies' '
> +       rm -rf clone &&
> +       echo "Set-Cookie: NotSecret=Foo" >cookies &&
> +       echo "Set-Cookie: Secret=Bar" >>cookies &&
> +       GIT_TRACE_CURL=true GIT_REDACT_COOKIES=Secret,AnotherSecret \
> +               git -c "http.cookieFile=$(pwd)/cookies" clone \
> +               $HTTPD_URL/smart/repo.git clone 2>err &&
> +       grep "Cookie:.*NotSecret=Foo" err &&
> +       grep "Cookie:.*Secret=<redacted>" err &&
> +       ! grep "Cookie:.*Secret=Bar" err
> +'

The looseness of these grep expressions (/Cookie:.*Secret/ also matches "Cookie: NotSecret", for instance) requires extra concentration on the part of the reader to see that you do indeed cover all cases. I wonder, therefore, if it would be better to tighten them to instead match the exact string.

Also, after reading the implementation, I had expected to see testing of the "Cookie: foo=bar; cow=moo" case, as well as the handled corner cases, such as missing missing "=" and missing value after "=".

← back to recent threads