{"thread":{"id":"47641","subject":"[PATCH v2 2/2] http: support omitting data from traces","startedAt":"2018-01-19T00:28:30Z","lastAt":"2018-01-19T00:28:36Z","messageCount":3,"participants":["Jonathan Tan"],"isPatch":true,"patchVersion":2,"patchTotal":2},"messages":[{"id":"336867","messageId":"42366886098655a6bf712f55c6c0246e31218f6f.1516321355.git.jonathantanmy@google.com","threadId":"47641","inReplyTo":"cover.1516321355.git.jonathantanmy@google.com","subject":"[PATCH v2 2/2] http: support omitting data from traces","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-19T00:28:02Z","receivedAt":"2018-01-19T00:28:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"GIT_TRACE_CURL provides a way to debug what is being sent and received\nover HTTP, with automatic redaction of sensitive information. But it\nalso logs data transmissions, which significantly increases the log file\nsize, sometimes unnecessarily. Add an option \"GIT_TRACE_CURL_NO_DATA\" to\nallow the user to omit such data transmissions.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/git.txt       |  4 ++++\n http.c                      | 27 +++++++++++++++++++--------\n t/t5551-http-fetch-smart.sh | 12 ++++++++++++\n 3 files changed, 35 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 5446d2143..8163b5796 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -646,6 +646,10 @@ of clones and fetches.\n \tvariable.\n \tSee `GIT_TRACE` for available trace output options.\n \n+`GIT_TRACE_CURL_NO_DATA`::\n+\tWhen a curl trace is enabled (see `GIT_TRACE_CURL` above), do not dump\n+\tdata (that is, only dump info lines and headers).\n+\n `GIT_REDACT_COOKIES`::\n \tThis can be set to a comma-separated list of strings. When a curl trace\n \tis enabled (see `GIT_TRACE_CURL` above), whenever a \"Cookies:\" header\ndiff --git a/http.c b/http.c\nindex 088cf70bf..32a823895 100644\n--- a/http.c\n+++ b/http.c\n@@ -16,6 +16,7 @@\n #include \"string-list.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n+static int trace_curl_data = 1;\n static struct string_list cookies_to_redact = STRING_LIST_INIT_DUP;\n #if LIBCURL_VERSION_NUM >= 0x070a08\n long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER;\n@@ -695,24 +696,32 @@ static int curl_trace(CURL *handle, curl_infotype type, char *data, size_t size,\n \t\tcurl_dump_header(text, (unsigned char *)data, size, DO_FILTER);\n \t\tbreak;\n \tcase CURLINFO_DATA_OUT:\n-\t\ttext = \"=> Send data\";\n-\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tif (trace_curl_data) {\n+\t\t\ttext = \"=> Send data\";\n+\t\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\t}\n \t\tbreak;\n \tcase CURLINFO_SSL_DATA_OUT:\n-\t\ttext = \"=> Send SSL data\";\n-\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tif (trace_curl_data) {\n+\t\t\ttext = \"=> Send SSL data\";\n+\t\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\t}\n \t\tbreak;\n \tcase CURLINFO_HEADER_IN:\n \t\ttext = \"<= Recv header\";\n \t\tcurl_dump_header(text, (unsigned char *)data, size, NO_FILTER);\n \t\tbreak;\n \tcase CURLINFO_DATA_IN:\n-\t\ttext = \"<= Recv data\";\n-\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tif (trace_curl_data) {\n+\t\t\ttext = \"<= Recv data\";\n+\t\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\t}\n \t\tbreak;\n \tcase CURLINFO_SSL_DATA_IN:\n-\t\ttext = \"<= Recv SSL data\";\n-\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tif (trace_curl_data) {\n+\t\t\ttext = \"<= Recv SSL data\";\n+\t\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\t}\n \t\tbreak;\n \n \tdefault:\t\t/* we ignore unknown types by default */\n@@ -857,6 +866,8 @@ static CURL *get_curl_handle(void)\n \tif (getenv(\"GIT_CURL_VERBOSE\"))\n \t\tcurl_easy_setopt(result, CURLOPT_VERBOSE, 1L);\n \tsetup_curl_trace(result);\n+\tif (getenv(\"GIT_TRACE_CURL_NO_DATA\"))\n+\t\ttrace_curl_data = 0;\n \tif (getenv(\"GIT_REDACT_COOKIES\")) {\n \t\tstring_list_split(&cookies_to_redact,\n \t\t\t\t  getenv(\"GIT_REDACT_COOKIES\"), ',', -1);\ndiff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh\nindex 21a5ce860..f5721b4a5 100755\n--- a/t/t5551-http-fetch-smart.sh\n+++ b/t/t5551-http-fetch-smart.sh\n@@ -385,5 +385,17 @@ test_expect_success 'GIT_REDACT_COOKIES handles empty values' '\n \tgrep \"Cookie:.*Foo=<redacted>\" err\n '\n \n+test_expect_success 'GIT_TRACE_CURL_NO_DATA prevents data from being traced' '\n+\trm -rf clone &&\n+\tGIT_TRACE_CURL=true \\\n+\t\tgit clone $HTTPD_URL/smart/repo.git clone 2>err &&\n+\tgrep \"=> Send data\" err &&\n+\n+\trm -rf clone &&\n+\tGIT_TRACE_CURL=true GIT_TRACE_CURL_NO_DATA=1 \\\n+\t\tgit clone $HTTPD_URL/smart/repo.git clone 2>err &&\n+\t! grep \"=> Send data\" err\n+'\n+\n stop_httpd\n test_done\n-- \n2.16.0.rc2.37.ge0d575025.dirty\n\n"},{"id":"336868","messageId":"701e28d97f8450ef75a7e78b6c7ecfc7a825031a.1516321355.git.jonathantanmy@google.com","threadId":"47641","inReplyTo":"cover.1516321355.git.jonathantanmy@google.com","subject":"[PATCH v2 1/2] http: support cookie redaction when tracing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-19T00:28:01Z","receivedAt":"2018-01-19T00:28:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When using GIT_TRACE_CURL, Git already redacts the \"Authorization:\" and\n\"Proxy-Authorization:\" HTTP headers. Extend this redaction to a\nuser-specified list of cookies, specified through the\n\"GIT_REDACT_COOKIES\" environment variable.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/git.txt       |  6 +++++\n http.c                      | 55 +++++++++++++++++++++++++++++++++++++++++++++\n t/t5551-http-fetch-smart.sh | 21 +++++++++++++++++\n 3 files changed, 82 insertions(+)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 3f4161a79..5446d2143 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -646,6 +646,12 @@ of clones and fetches.\n \tvariable.\n \tSee `GIT_TRACE` for available trace output options.\n \n+`GIT_REDACT_COOKIES`::\n+\tThis can be set to a comma-separated list of strings. When a curl trace\n+\tis enabled (see `GIT_TRACE_CURL` above), whenever a \"Cookies:\" header\n+\tsent by the client is dumped, values of cookies whose key is in that\n+\tlist (case-sensitive) are redacted.\n+\n `GIT_LITERAL_PATHSPECS`::\n \tSetting this variable to `1` will cause Git to treat all\n \tpathspecs literally, rather than as glob patterns. For example,\ndiff --git a/http.c b/http.c\nindex 597771271..088cf70bf 100644\n--- a/http.c\n+++ b/http.c\n@@ -13,8 +13,10 @@\n #include \"transport.h\"\n #include \"packfile.h\"\n #include \"protocol.h\"\n+#include \"string-list.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n+static struct string_list cookies_to_redact = STRING_LIST_INIT_DUP;\n #if LIBCURL_VERSION_NUM >= 0x070a08\n long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER;\n #else\n@@ -575,6 +577,54 @@ static void redact_sensitive_header(struct strbuf *header)\n \t\t/* Everything else is opaque and possibly sensitive */\n \t\tstrbuf_setlen(header,  sensitive_header - header->buf);\n \t\tstrbuf_addstr(header, \" <redacted>\");\n+\t} else if (cookies_to_redact.nr &&\n+\t\t   skip_prefix(header->buf, \"Cookie:\", &sensitive_header)) {\n+\t\tstruct strbuf redacted_header = STRBUF_INIT;\n+\t\tchar *cookie;\n+\n+\t\twhile (isspace(*sensitive_header))\n+\t\t\tsensitive_header++;\n+\n+\t\t/*\n+\t\t * The contents of header starting from sensitive_header will\n+\t\t * subsequently be overridden, so it is fine to mutate this\n+\t\t * string (hence the assignment to \"char *\").\n+\t\t */\n+\t\tcookie = (char *) sensitive_header;\n+\n+\t\twhile (cookie) {\n+\t\t\tchar *equals;\n+\t\t\tchar *semicolon = strstr(cookie, \"; \");\n+\t\t\tif (semicolon)\n+\t\t\t\t*semicolon = 0;\n+\t\t\tequals = strchrnul(cookie, '=');\n+\t\t\tif (!equals) {\n+\t\t\t\t/* invalid cookie, just append and continue */\n+\t\t\t\tstrbuf_addstr(&redacted_header, cookie);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\t*equals = 0; /* temporarily set to NUL for lookup */\n+\t\t\tif (string_list_lookup(&cookies_to_redact, cookie)) {\n+\t\t\t\tstrbuf_addstr(&redacted_header, cookie);\n+\t\t\t\tstrbuf_addstr(&redacted_header, \"=<redacted>\");\n+\t\t\t} else {\n+\t\t\t\t*equals = '=';\n+\t\t\t\tstrbuf_addstr(&redacted_header, cookie);\n+\t\t\t}\n+\t\t\tif (semicolon) {\n+\t\t\t\t/*\n+\t\t\t\t * There are more cookies. (Or, for some\n+\t\t\t\t * reason, the input string ends in \"; \".)\n+\t\t\t\t */\n+\t\t\t\tstrbuf_addstr(&redacted_header, \"; \");\n+\t\t\t\tcookie = semicolon + strlen(\"; \");\n+\t\t\t} else {\n+\t\t\t\tcookie = NULL;\n+\t\t\t}\n+\t\t}\n+\n+\t\tstrbuf_setlen(header, sensitive_header - header->buf);\n+\t\tstrbuf_addbuf(header, &redacted_header);\n \t}\n }\n \n@@ -807,6 +857,11 @@ static CURL *get_curl_handle(void)\n \tif (getenv(\"GIT_CURL_VERBOSE\"))\n \t\tcurl_easy_setopt(result, CURLOPT_VERBOSE, 1L);\n \tsetup_curl_trace(result);\n+\tif (getenv(\"GIT_REDACT_COOKIES\")) {\n+\t\tstring_list_split(&cookies_to_redact,\n+\t\t\t\t  getenv(\"GIT_REDACT_COOKIES\"), ',', -1);\n+\t\tstring_list_sort(&cookies_to_redact);\n+\t}\n \n \tcurl_easy_setopt(result, CURLOPT_USERAGENT,\n \t\tuser_agent ? user_agent : git_user_agent());\ndiff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh\nindex a51b7e20d..21a5ce860 100755\n--- a/t/t5551-http-fetch-smart.sh\n+++ b/t/t5551-http-fetch-smart.sh\n@@ -364,5 +364,26 @@ test_expect_success 'custom http headers' '\n \t\tsubmodule update sub\n '\n \n+test_expect_success 'GIT_REDACT_COOKIES redacts cookies' '\n+\trm -rf clone &&\n+\techo \"Set-Cookie: Foo=1\" >cookies &&\n+\techo \"Set-Cookie: Bar=2\" >>cookies &&\n+\tGIT_TRACE_CURL=true GIT_REDACT_COOKIES=Bar,Baz \\\n+\t\tgit -c \"http.cookieFile=$(pwd)/cookies\" clone \\\n+\t\t$HTTPD_URL/smart/repo.git clone 2>err &&\n+\tgrep \"Cookie:.*Foo=1\" err &&\n+\tgrep \"Cookie:.*Bar=<redacted>\" err &&\n+\t! grep \"Cookie:.*Bar=2\" err\n+'\n+\n+test_expect_success 'GIT_REDACT_COOKIES handles empty values' '\n+\trm -rf clone &&\n+\techo \"Set-Cookie: Foo=\" >cookies &&\n+\tGIT_TRACE_CURL=true GIT_REDACT_COOKIES=Foo \\\n+\t\tgit -c \"http.cookieFile=$(pwd)/cookies\" clone \\\n+\t\t$HTTPD_URL/smart/repo.git clone 2>err &&\n+\tgrep \"Cookie:.*Foo=<redacted>\" err\n+'\n+\n stop_httpd\n test_done\n-- \n2.16.0.rc2.37.ge0d575025.dirty\n\n"},{"id":"336869","messageId":"cover.1516321355.git.jonathantanmy@google.com","threadId":"47641","inReplyTo":"cover.1516235197.git.jonathantanmy@google.com","subject":"[PATCH v2 0/2] Cookie redaction during GIT_TRACE_CURL","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-01-19T00:28:00Z","receivedAt":"2018-01-19T00:28:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Eric. Changes in v2:\n - documented all environment variables introduced\n - made test more clear by ensuring that no cookie keys are suffixes or\n   prefixes of others\n - tested empty value\n\nAs far as I can tell, it does not seem possible that Git generates a\ncookie with no equals sign (like \"Secure\" or \"HttpOnly\" described in RFC\n6265). When I try to craft a cookie file containing that (using\n\"Set-Cookie: Foo=; HttpOnly\", for example), the no-equals-sign cookie\njust disappears.\n\nJonathan Tan (2):\n  http: support cookie redaction when tracing\n  http: support omitting data from traces\n\n Documentation/git.txt       | 10 ++++++\n http.c                      | 82 ++++++++++++++++++++++++++++++++++++++++-----\n t/t5551-http-fetch-smart.sh | 33 ++++++++++++++++++\n 3 files changed, 117 insertions(+), 8 deletions(-)\n\n-- \n2.16.0.rc2.37.ge0d575025.dirty\n\n"}]}