{"thread":{"id":"42418","subject":"[PATCH v7 0/2] Implement the GIT_TRACE_CURL environment variable","startedAt":"2016-05-23T13:44:01Z","lastAt":"2016-05-25T00:50:30Z","messageCount":4,"participants":["Elia Pinto","Junio C Hamano"],"isPatch":true,"patchVersion":7,"patchTotal":2},"messages":[{"id":"287242","messageId":"20160523134403.43880-1-gitter.spiros@gmail.com","threadId":"42418","inReplyTo":null,"subject":"[PATCH v7 0/2] Implement the GIT_TRACE_CURL environment variable","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2016-05-23T13:44:01Z","receivedAt":"2016-05-23T13:44:01Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"This is the seventh version but in reality is the complete rewriting of the patches discussed here\n(here called V1)\n\n$gmane/290520\n$gmane/290521\n\n*Changes from V6\n($gmane/295158)\n- updated with junio suggestions ($gmane/295207)\n * http.c: add redact_sensitive_header() helper\n * http.c: renamed header_list and ptr_list to\n headers and header\n * http.c: reduce width in curl_dump_data\n * http.c: curl_dump_data. cut strbuf_addch at a sensible place\n * http.c: drop the nopriv_header variable: add an enum instead\n\n\n*Changes from V5\n($gmane/293236)\n\n- don't export curl_trace anymore. Define it static\n- fix a minor cleanup (style) in setup_curl_trace\n- and, finally, i rewrote completely curl_dump, separating it\ninto two functions (one for http header and one for the http data),\nhoping for a coherent implementation with curl --ascii-trace output\nbut easier to read than the previous implementation that\nused a hard-to-read double-loop.\n\n \n*Changes from V4\n($gmane/292867)\n\n- add a better abstraction with the routine setup_curl_trace\n- curl_dump : drop the noex parameter, define nopriv boolean as int\n- use decimal constant where appropiate\n- fix multi-line comment\n- redo the authorization header skip with a replace of possible sensitive data.\n We prefer to print only:\n 09:00:53.238330 http.c:534 => Send header: Authorization: \n intested of\n 09:00:53.238330 http.c:534 => Send header: Authorization: Basic(o other scheme) \n as it was done in the original proposed suggestion by Jeff King.\n This is because i think it's better not to print even the authorization scheme.\n We add also the (previously missing) proxy-authorization case\n- curl_dump: fix strbuf memory leak\n\nas suggested by Jeff King\n($gmane/292891)\n($gmane/292892)\n\nIn this series i keep the original curl_dump parsing code, even though it is\nobjectively difficult to read. This is because the same code is used internally by curl\nto do \"ascii-trace\" and is also reported in the libcurl code examples and test.\nI think this may make maintenance of code easier in the future (libcurl\nnew dev, new features and so on)\n\nOf course if the maintainer (or other) believes it is really necessary\nto rewrite the above code to accept the patches i will do.\n\n*Changes from V3\n($gmane/292040)\n\n- add missing static to curl_dump\n- reorder the patch order\n- tried to fix all (but i am not sure) the problems reported by Julio ($gmane/292055)\n- * squash the documentation with the http.c commit.\n * in the trace prefix each line to indicate it is about sending a header, and drop the\n initial hex count\n * auto-censor Authorization headers by default\n\n as suggested by Jeff King ($gmane/292074)\n\n*Changes from V2\n($gmane/291868)\n\n- fix garbage comment in http.c (i am very sorry for that)\n- add final '.' to the commit message for imap-send.c and to other commit messages\n- typofix double ; in http.c\n- merge the nice cleanup and code refactoring of Ramsay Jones (Thank you very much !!)\n- squash the previous commit 2/4\n\n*Changes from V1\n\n- introduced GIT_TRACE_CURL variable with its documentation\n- changed the name of the temporary variable \"i\" in \"w\" in the helper routine\n- used the c escape sequences instead of the hex equivalent\n- dropped the previous GIT_DEBUG_CURL env var\n- curl_dump and curl_trace factored out to a shared implementation\nin http.c\n\n\n\nElia Pinto (2):\n http.c: implement the GIT_TRACE_CURL environment variable\n imap-send.c: introduce the GIT_TRACE_CURL enviroment variable\n\n Documentation/git.txt | 8 ++++\n http.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++-\n http.h | 2 +\n imap-send.c | 1 +\n 4 files changed, 134 insertions(+), 2 deletions(-)\n\n-- \n2.8.3.614.g2da74fb.dirty\n"},{"id":"287243","messageId":"20160523134403.43880-2-gitter.spiros@gmail.com","threadId":"42418","inReplyTo":"20160523134403.43880-1-gitter.spiros@gmail.com","subject":"[PATCH v7 1/2] http.c: implement the GIT_TRACE_CURL environment variable","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2016-05-23T13:44:02Z","receivedAt":"2016-05-23T13:44:02Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Implement the GIT_TRACE_CURL environment variable to allow a\ngreater degree of detail of GIT_CURL_VERBOSE, in particular\nthe complete transport header and all the data payload exchanged.\nIt might be useful if a particular situation could require a more\nthorough debugging analysis. Document the new GIT_TRACE_CURL\nenvironment variable.\n\nHelped-by: Torsten Bögershausen \nHelped-by: Ramsay Jones \nHelped-by: Junio C Hamano \nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Elia Pinto \n---\n Documentation/git.txt | 8 ++++\n http.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++-\n http.h | 2 +\n 3 files changed, 133 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex dd6dbf7..a46a356 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -1077,6 +1077,14 @@ of clones and fetches.\n \tcloning of shallow repositories.\n \tSee 'GIT_TRACE' for available trace output options.\n \n+'GIT_TRACE_CURL'::\n+\tEnables a curl full trace dump of all incoming and outgoing data,\n+\tincluding descriptive information, of the git transport protocol.\n+\tThis is similar to doing curl --trace-ascii on the command line.\n+\tThis option overrides setting the GIT_CURL_VERBOSE environment\n+\tvariable.\n+\tSee 'GIT_TRACE' for available trace output options.\n+\n 'GIT_LITERAL_PATHSPECS'::\n \tSetting this variable to `1` will cause Git to treat all\n \tpathspecs literally, rather than as glob patterns. For example,\ndiff --git a/http.c b/http.c\nindex df6dd01..d661408 100644\n--- a/http.c\n+++ b/http.c\n@@ -11,6 +11,7 @@\n #include \"gettext.h\"\n #include \"transport.h\"\n \n+static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n #if LIBCURL_VERSION_NUM >= 0x070a08\n long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER;\n #else\n@@ -477,6 +478,126 @@ static void set_curl_keepalive(CURL *c)\n }\n #endif\n \n+static void redact_sensitive_header(struct strbuf *header)\n+{\n+\tconst char *sensitive_header;\n+\n+\tif (skip_prefix(header->buf, \"Authorization:\", &sensitive_header) ||\n+\t skip_prefix(header->buf, \"Proxy-Authorization:\", &sensitive_header)) {\n+\t\t/* The first token is the type, which is OK to log */\n+\t\twhile (isspace(*sensitive_header))\n+\t\t\tsensitive_header++;\n+\t\twhile (*sensitive_header && !isspace(*sensitive_header))\n+\t\t\tsensitive_header++;\n+\t\t/* Everything else is opaque and possibly sensitive */\n+\t\tstrbuf_setlen(header, sensitive_header - header->buf);\n+\t\tstrbuf_addstr(header, \" \");\n+\t}\n+}\n+\n+static void curl_dump_header(const char *text, unsigned char *ptr, size_t size, int hide_sensitive_header)\n+{\n+\tstruct strbuf out = STRBUF_INIT;\n+\tconst char *sensitive_header;\n+\tstruct strbuf **headers, **header;\n+\n+\tstrbuf_addf(&out, \"%s, %10.10ld bytes (0x%8.8lx)\\n\",\n+\t\ttext, (long)size, (long)size);\n+\ttrace_strbuf(&trace_curl, &out);\n+\tstrbuf_reset(&out);\n+\tstrbuf_add(&out, ptr, size);\n+\theaders = strbuf_split_max(&out, '\\n', 0);\n+\n+\tfor (header = headers; *header; header++) {\n+\t\tif (hide_sensitive_header)\n+\t\t\tredact_sensitive_header(*header);\n+\t\tstrbuf_insert((*header), 0, text, strlen(text));\n+\t\tstrbuf_insert((*header), strlen(text), \": \", 2);\n+\t\tstrbuf_rtrim((*header));\n+\t\tstrbuf_addch((*header), '\\n');\n+\t\ttrace_strbuf(&trace_curl, (*header));\n+\t}\n+\tstrbuf_list_free(headers);\n+\tstrbuf_release(&out);\n+}\n+\n+static void curl_dump_data(const char *text, unsigned char *ptr, size_t size)\n+{\n+\tsize_t i;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tunsigned int width = 60;\n+\n+\tstrbuf_addf(&out, \"%s, %10.10ld bytes (0x%8.8lx)\\n\",\n+\t\ttext, (long)size, (long)size);\n+\ttrace_strbuf(&trace_curl, &out);\n+\n+\tfor (i = 0; i < size; i += width) {\n+\t\tsize_t w;\n+\n+\t\tstrbuf_reset(&out);\n+\t\tstrbuf_addf(&out, \"%s: \", text);\n+\t\tfor (w = 0; (w < width) && (i + w < size); w++) {\n+\t\t\tunsigned char ch = ptr[i + w];\n+\n+\t\t\tstrbuf_addch(&out,\n+\t\t\t\t (ch >= 0x20) && (ch < 0x80)\n+\t\t\t\t ? ch : '.');\n+\t\t}\n+\t\tstrbuf_addch(&out, '\\n');\n+\t\ttrace_strbuf(&trace_curl, &out);\n+\t}\n+\tstrbuf_release(&out);\n+}\n+\n+static int curl_trace(CURL *handle, curl_infotype type, char *data, size_t size, void *userp)\n+{\n+\tconst char *text;\n+\tenum { NO_FILTER = 0, DO_FILTER = 1 };\n+\n+\tswitch (type) {\n+\tcase CURLINFO_TEXT:\n+\t\ttrace_printf_key(&trace_curl, \"== Info: %s\", data);\n+\tdefault:\t\t/* we ignore unknown types by default */\n+\t\treturn 0;\n+\n+\tcase CURLINFO_HEADER_OUT:\n+\t\ttext = \"=> Send header\";\n+\t\tcurl_dump_header(text, (unsigned char *)data, size, DO_FILTER);\n+\t\tbreak;\n+\tcase CURLINFO_DATA_OUT:\n+\t\ttext = \"=> Send data\";\n+\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tbreak;\n+\tcase CURLINFO_SSL_DATA_OUT:\n+\t\ttext = \"=> Send SSL data\";\n+\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tbreak;\n+\tcase CURLINFO_HEADER_IN:\n+\t\ttext = \"<= Recv header\";\n+\t\tcurl_dump_header(text, (unsigned char *)data, size, NO_FILTER);\n+\t\tbreak;\n+\tcase CURLINFO_DATA_IN:\n+\t\ttext = \"<= Recv data\";\n+\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tbreak;\n+\tcase CURLINFO_SSL_DATA_IN:\n+\t\ttext = \"<= Recv SSL data\";\n+\t\tcurl_dump_data(text, (unsigned char *)data, size);\n+\t\tbreak;\n+\t}\n+\treturn 0;\n+}\n+\n+void setup_curl_trace(CURL *handle)\n+{\n+\tif (!trace_want(&trace_curl))\n+\t\treturn;\n+\tcurl_easy_setopt(handle, CURLOPT_VERBOSE, 1L);\n+\tcurl_easy_setopt(handle, CURLOPT_DEBUGFUNCTION, curl_trace);\n+\tcurl_easy_setopt(handle, CURLOPT_DEBUGDATA, NULL);\n+}\n+\n+\n static CURL *get_curl_handle(void)\n {\n \tCURL *result = curl_easy_init();\n@@ -575,9 +696,9 @@ static CURL *get_curl_handle(void)\n \t\twarning(\"protocol restrictions not applied to curl redirects because\\n\"\n \t\t\t\"your curl version is too old (>= 7.19.4)\");\n #endif\n-\n \tif (getenv(\"GIT_CURL_VERBOSE\"))\n-\t\tcurl_easy_setopt(result, CURLOPT_VERBOSE, 1);\n+\t\tcurl_easy_setopt(result, CURLOPT_VERBOSE, 1L);\n+\tsetup_curl_trace(result);\n \n \tcurl_easy_setopt(result, CURLOPT_USERAGENT,\n \t\tuser_agent ? user_agent : git_user_agent());\ndiff --git a/http.h b/http.h\nindex 36f558b..5ab9d9c 100644\n--- a/http.h\n+++ b/http.h\n@@ -225,4 +225,6 @@ extern int finish_http_object_request(struct http_object_request *freq);\n extern void abort_http_object_request(struct http_object_request *freq);\n extern void release_http_object_request(struct http_object_request *freq);\n \n+/* setup routine for curl_easy_setopt CURLOPT_DEBUGFUNCTION */\n+void setup_curl_trace(CURL *handle);\n #endif /* HTTP_H */\n-- \n2.8.3.614.g2da74fb.dirty\n"},{"id":"287245","messageId":"20160523134403.43880-3-gitter.spiros@gmail.com","threadId":"42418","inReplyTo":"20160523134403.43880-1-gitter.spiros@gmail.com","subject":"[PATCH v7 2/2] imap-send.c: introduce the GIT_TRACE_CURL enviroment variable","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2016-05-23T13:44:03Z","receivedAt":"2016-05-23T13:44:03Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Permit the use of the GIT_TRACE_CURL environment variable calling\nthe setup_curl_trace http.c helper routine.\n\nHelped-by: Torsten Bögershausen \nHelped-by: Ramsay Jones \nHelped-by: Junio C Hamano \nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Elia Pinto \n---\n imap-send.c | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/imap-send.c b/imap-send.c\nindex 938c691..50377c5 100644\n--- a/imap-send.c\n+++ b/imap-send.c\n@@ -1443,6 +1443,7 @@ static CURL *setup_curl(struct imap_server_conf *srvc)\n \n \tif (0 < verbosity || getenv(\"GIT_CURL_VERBOSE\"))\n \t\tcurl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);\n+\tsetup_curl_trace(curl);\n \n \treturn curl;\n }\n-- \n2.8.3.614.g2da74fb.dirty\n"},{"id":"287468","messageId":"CAPc5daXZU2oDHJnuwvaCO_C8VRFPNLiT_qei2NoDgHRFKVnZ=w@mail.gmail.com","threadId":"42418","inReplyTo":"20160523134403.43880-2-gitter.spiros@gmail.com","subject":"Re: [PATCH v7 1/2] http.c: implement the GIT_TRACE_CURL environment variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-05-25T00:50:30Z","receivedAt":"2016-05-25T00:50:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Mon, May 23, 2016 at 6:44 AM, Elia Pinto wrote:\n> +static void curl_dump_header(const char *text, unsigned char *ptr, size_t size, int hide_sensitive_header)\n> +{\n> + struct strbuf out = STRBUF_INIT;\n> + const char *sensitive_header;\n\nThis variable now has become unused, so I'll remove it while queuing.\n\nThanks.\n\n> + struct strbuf **headers, **header;\n> +\n> + strbuf_addf(&out, \"%s, %10.10ld bytes (0x%8.8lx)\\n\",\n> + text, (long)size, (long)size);\n> + trace_strbuf(&trace_curl, &out);\n> + strbuf_reset(&out);\n> + strbuf_add(&out, ptr, size);\n> + headers = strbuf_split_max(&out, '\\n', 0);\n> +\n> + for (header = headers; *header; header++) {\n> + if (hide_sensitive_header)\n> + redact_sensitive_header(*header);\n> + strbuf_insert((*header), 0, text, strlen(text));\n> + strbuf_insert((*header), strlen(text), \": \", 2);\n> + strbuf_rtrim((*header));\n> + strbuf_addch((*header), '\\n');\n> + trace_strbuf(&trace_curl, (*header));\n> + }\n> + strbuf_list_free(headers);\n> + strbuf_release(&out);\n> +}\n"}]}