{"thread":{"id":"57795","subject":"[PATCH] Added Curl Option to Override Request Method","startedAt":"2022-04-26T11:01:40Z","lastAt":"2022-04-27T00:49:12Z","messageCount":5,"participants":["Drew Green via GitGitGadget","brian m. carlson","Drew Green"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"454424","messageId":"pull.1225.git.1650970894143.gitgitgadget@gmail.com","threadId":"57795","inReplyTo":null,"subject":"[PATCH] Added Curl Option to Override Request Method","fromName":"Drew Green via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-26T11:01:34Z","receivedAt":"2022-04-26T11:01:40Z","isPatch":true,"sender":{"key":"agreenbhm@gmail.com","avatar":"https://gravatar.com/avatar/b7897d50cf93cae85b5e7f0a306b11f6a8f805592e4323c690a970d6b4ec63a2?d=mp&s=160"},"body":"From: agreenbhm \n\nAdded support for environment variable \"CURLOPT_CUSTOMREQUEST\"\nto allow setting the curl option to override the default request\nmethod used by HTTP Git operations. Primary reason for this is to\nallow support for cloning repositories where only GET requests\nare allowed but not POSTs. When cloning a repo first a GET is\nmade to the server and then a POST is made to the \"git-upload-pack\"\nendpoint. In some corporate environments with strong controls\nonly GET requests are allowed to known repository hosts (such\nas GitHub) to prevent data leakage by sending data. Using this\nnew environmental variable, a user can set\n\"CURLOPT_CUSTOMREQUEST=GET\" which will change the second request\nfrom a POST to a GET, bypassing web proxy restrictions on the type\nof requests allowed. Tested with GitHub, changing the request\nfrom POST to GET still results in the expected behavior of the\nrepo successfully being cloned.\n\nSigned-off-by: agreenbhm \n---\n Added Curl Option to Override Request Method\n \n Added support for environment variable \"CURLOPT_CUSTOMREQUEST\" to allow\n setting the curl option to override the default request method used by\n HTTP Git operations. Primary reason for this is to allow support for\n cloning repositories where only GET requests are allowed but not POSTs.\n When cloning a repo first a GET is made to the server and then a POST is\n made to the \"git-upload-pack\" endpoint. In some corporate environments\n with strong controls only GET requests are allowed to known repository\n hosts (such as GitHub) to prevent data leakage by sending data. Using\n this new environmental variable, a user can set\n \"CURLOPT_CUSTOMREQUEST=GET\" which will change the second request from a\n POST to a GET, bypassing web proxy restrictions on the type of requests\n allowed. Tested with GitHub, changing the request from POST to GET still\n results in the expected behavior of the repo successfully being cloned.\n \n Signed-off-by: agreenbhm agreenbhm@gmail.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1225%2Fagreenbhm%2Fmaster-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1225/agreenbhm/master-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1225\n\n http.c | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/http.c b/http.c\nindex 229da4d1488..2ca13b848a5 100644\n--- a/http.c\n+++ b/http.c\n@@ -1235,6 +1235,8 @@ struct active_request_slot *get_active_slot(void)\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);\n \tif (http_auth.password || curl_empty_auth_enabled())\n \t\tinit_curl_http_auth(slot->curl);\n+\tif(getenv(\"CURLOPT_CUSTOMREQUEST\"))\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, getenv(\"CURLOPT_CUSTOMREQUEST\"));\n \n \treturn slot;\n }\n\nbase-commit: 6cd33dceed60949e2dbc32e3f0f5e67c4c882e1e\n-- \ngitgitgadget\n"},{"id":"454447","messageId":"pull.1225.v2.git.1650988659280.gitgitgadget@gmail.com","threadId":"57795","inReplyTo":"pull.1225.git.1650970894143.gitgitgadget@gmail.com","subject":"[PATCH v2] Added Curl Option to Override Request Method v2","fromName":"Drew Green via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-04-26T15:57:39Z","receivedAt":"2022-04-26T15:57:46Z","isPatch":true,"sender":{"key":"agreenbhm@gmail.com","avatar":"https://gravatar.com/avatar/b7897d50cf93cae85b5e7f0a306b11f6a8f805592e4323c690a970d6b4ec63a2?d=mp&s=160"},"body":"From: agreenbhm \n\nAdded support for environment variable \"CURLOPT_CUSTOMREQUEST\"\nand config option \"http.customrequest\" to allow setting the Curl\noption to override the default request method used by HTTP Git\noperations. Primary reason for this is to allow support for\ncloning repositories where only GET requests\nare allowed by a local web proxy but not POSTs. When cloning\na repo first a GET is made to the server and then a\nPOST is made to the \"git-upload-pack\" endpoint. In some\ncorporate environments with strong controls\nonly GET requests are allowed to known repository hosts (such\nas GitHub) through a web proxy to prevent data leakage. Using this\nnew setting, a user can set the \"CURLOPT_CUSTOMREQUEST=GET\" env at runtime\nor \"http.customrequest = GET\" in their config file which will\nchange the second request from a POST to a GET, bypassing\nweb proxy restrictions on the type of requests allowed.\nTested with GitHub, changing the request from POST to GET still\nresults in the expected behavior of the repo successfully being cloned.\n\nThis is v2 of this patch, which refactored the placement of the env\nand added the ability to set the config file option.\n\nSigned-off-by: agreenbhm \n---\n Added Curl Option to Override Request Method\n \n Added support for environment variable \"CURLOPT_CUSTOMREQUEST\" to allow\n setting the curl option to override the default request method used by\n HTTP Git operations. Primary reason for this is to allow support for\n cloning repositories where only GET requests are allowed but not POSTs.\n When cloning a repo first a GET is made to the server and then a POST is\n made to the \"git-upload-pack\" endpoint. In some corporate environments\n with strong controls only GET requests are allowed to known repository\n hosts (such as GitHub) to prevent data leakage by sending data. Using\n this new environmental variable, a user can set\n \"CURLOPT_CUSTOMREQUEST=GET\" which will change the second request from a\n POST to a GET, bypassing web proxy restrictions on the type of requests\n allowed. Tested with GitHub, changing the request from POST to GET still\n results in the expected behavior of the repo successfully being cloned.\n \n Signed-off-by: agreenbhm agreenbhm@gmail.com\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1225%2Fagreenbhm%2Fmaster-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1225/agreenbhm/master-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1225\n\nRange-diff vs v1:\n\n 1: 8bf14c61c2a < -: ----------- Added Curl Option to Override Request Method\n -: ----------- > 1: 8734bf28344 Added Curl Option to Override Request Method v2\n\n\n http.c | 7 ++++++-\n 1 file changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/http.c b/http.c\nindex 229da4d1488..eaf269fc5a7 100644\n--- a/http.c\n+++ b/http.c\n@@ -75,6 +75,7 @@ static const char *http_proxy_ssl_key;\n static const char *http_proxy_ssl_ca_info;\n static struct credential proxy_cert_auth = CREDENTIAL_INIT;\n static int proxy_ssl_cert_password_required;\n+static const char *http_custom_request;\n \n static struct {\n \tconst char *name;\n@@ -403,6 +404,9 @@ static int http_options(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif(!strcmp(\"http.customrequest\", var))\n+\t\treturn git_config_string(&http_custom_request, var, value);\n+\n \t/* Fall back on the default ones */\n \treturn git_default_config(var, value, cb);\n }\n@@ -1099,6 +1103,7 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \t\t starts_with(url, \"https://\"))\n \t\t\tssl_cert_password_required = 1;\n \t}\n+\tset_from_env(&http_custom_request, \"CURLOPT_CUSTOMREQUEST\");\n \n \tcurl_default = get_curl_handle();\n }\n@@ -1212,7 +1217,7 @@ struct active_request_slot *get_active_slot(void)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, http_custom_request);\n \tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, NULL);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, NULL);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, NULL);\n\nbase-commit: 6cd33dceed60949e2dbc32e3f0f5e67c4c882e1e\n-- \ngitgitgadget\n"},{"id":"454477","messageId":"YmhkCAbmNE8A7Wqy@camp.crustytoothpaste.net","threadId":"57795","inReplyTo":"pull.1225.v2.git.1650988659280.gitgitgadget@gmail.com","subject":"Re: [PATCH v2] Added Curl Option to Override Request Method v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2022-04-26T21:28:40Z","receivedAt":"2022-04-26T21:28:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2022-04-26 at 15:57:39, Drew Green via GitGitGadget wrote:\n> From: agreenbhm \n> \n> Added support for environment variable \"CURLOPT_CUSTOMREQUEST\"\n> and config option \"http.customrequest\" to allow setting the Curl\n> option to override the default request method used by HTTP Git\n> operations. Primary reason for this is to allow support for\n> cloning repositories where only GET requests\n> are allowed by a local web proxy but not POSTs. When cloning\n> a repo first a GET is made to the server and then a\n> POST is made to the \"git-upload-pack\" endpoint. In some\n> corporate environments with strong controls\n> only GET requests are allowed to known repository hosts (such\n> as GitHub) through a web proxy to prevent data leakage. Using this\n> new setting, a user can set the \"CURLOPT_CUSTOMREQUEST=GET\" env at runtime\n> or \"http.customrequest = GET\" in their config file which will\n> change the second request from a POST to a GET, bypassing\n> web proxy restrictions on the type of requests allowed.\n> Tested with GitHub, changing the request from POST to GET still\n> results in the expected behavior of the repo successfully being cloned.\n\nI don't think this is a good idea. It may happen that GitHub or other\nservers happen to accept a GET request here, but that is a bug and\nshould be fixed. It is definitely not something we should depend on or\nrely on, and it isn't a documented part of the protocol.\n\nIf your corporate environment doesn't allow POST requests, you may wish\nto use SSH for Git operations instead, or you may need to explain to\nyour company why you cannot do your job with their proxy in place.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"454480","messageId":"CANDN8xGh31E3VwgRUb6YgFuswiyYjzJwK-tyvmo2c6KY+pOtJA@mail.gmail.com","threadId":"57795","inReplyTo":"YmhkCAbmNE8A7Wqy@camp.crustytoothpaste.net","subject":"Re: [PATCH v2] Added Curl Option to Override Request Method v2","fromName":"Drew Green","fromEmail":"agreenbhm@gmail.com","sentAt":"2022-04-26T23:34:32Z","receivedAt":"2022-04-26T23:34:49Z","isPatch":true,"sender":{"key":"agreenbhm@gmail.com","avatar":"https://gravatar.com/avatar/b7897d50cf93cae85b5e7f0a306b11f6a8f805592e4323c690a970d6b4ec63a2?d=mp&s=160"},"body":"On Tue, Apr 26, 2022 at 7:30 PM brian m. carlson\n wrote:\n>\n> On 2022-04-26 at 15:57:39, Drew Green via GitGitGadget wrote:\n> > From: agreenbhm \n> >\n> > Added support for environment variable \"CURLOPT_CUSTOMREQUEST\"\n> > and config option \"http.customrequest\" to allow setting the Curl\n> > option to override the default request method used by HTTP Git\n> > operations. Primary reason for this is to allow support for\n> > cloning repositories where only GET requests\n> > are allowed by a local web proxy but not POSTs. When cloning\n> > a repo first a GET is made to the server and then a\n> > POST is made to the \"git-upload-pack\" endpoint. In some\n> > corporate environments with strong controls\n> > only GET requests are allowed to known repository hosts (such\n> > as GitHub) through a web proxy to prevent data leakage. Using this\n> > new setting, a user can set the \"CURLOPT_CUSTOMREQUEST=GET\" env at runtime\n> > or \"http.customrequest = GET\" in their config file which will\n> > change the second request from a POST to a GET, bypassing\n> > web proxy restrictions on the type of requests allowed.\n> > Tested with GitHub, changing the request from POST to GET still\n> > results in the expected behavior of the repo successfully being cloned.\n>\n> I don't think this is a good idea. It may happen that GitHub or other\n> servers happen to accept a GET request here, but that is a bug and\n> should be fixed. It is definitely not something we should depend on or\n> rely on, and it isn't a documented part of the protocol.\n>\n> If your corporate environment doesn't allow POST requests, you may wish\n> to use SSH for Git operations instead, or you may need to explain to\n> your company why you cannot do your job with their proxy in place.\n> --\n> brian m. carlson (he/him or they/them)\n> Toronto, Ontario, CA\n\nBrian - I understand what you're saying, however I don't think adding\nthis feature is detrimental in any way. It is simply leveraging a\nfeature of curl without any promises of resulting behavior. Why not\nallow users to take advantage of a library feature if it can help?\n-- \nDrew Green\nwww.drewgreen.net\nPGP: 17BDDD7E\n"},{"id":"454484","messageId":"YmiS/U34WJG/qRoT@camp.crustytoothpaste.net","threadId":"57795","inReplyTo":"CANDN8xGh31E3VwgRUb6YgFuswiyYjzJwK-tyvmo2c6KY+pOtJA@mail.gmail.com","subject":"Re: [PATCH v2] Added Curl Option to Override Request Method v2","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2022-04-27T00:49:01Z","receivedAt":"2022-04-27T00:49:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2022-04-26 at 23:34:32, Drew Green wrote:\n> Brian - I understand what you're saying, however I don't think adding\n> this feature is detrimental in any way. It is simply leveraging a\n> feature of curl without any promises of resulting behavior. Why not\n> allow users to take advantage of a library feature if it can help?\n\nBecause the protocol is defined to use POST. We might well define GET\nto mean something different in the future. HTTP verbs are not\ninterchangeable, and GET requests can be subject to caching while POST\nrequests typically cannot. Similarly, GET requests are idempotent,\nwhile POST requests are not, and what you're proposing would allow\ncreating non-idempotent GET requests, thereby violating the HTTP\nspecification.\n\nAlso, people will expect it to work and complain when it doesn't. People\nwill complain that GitHub no longer supports it if we fix the bug[0]\nbecause Git has an option to use GET. People will also expect this\nsupport in Git LFS as well even though it cannot possibly work there,\nand be angry that we won't support it.\n\nI'm very much opposed to this. There are all sorts of broken proxies\nwhich break Git in various ways, and this is just another situation in\nwhich they do that. The solution is to use SSH, if that's available, or\nfix the proxy. A proxy that doesn't handle POST properly doesn't even\nspeak HTTP/1.1 properly, and that was defined in 1999. There's really\nno excuse to have such software on the Internet today. Such a proxy\nwouldn't even be usable for a web browser.\n\n[0] It is my intention to fix this bug in GitHub's Git service in short\norder.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"}]}