{"thread":{"id":"52195","subject":"[PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","startedAt":"2019-11-05T21:59:23Z","lastAt":"2019-11-08T13:44:35Z","messageCount":17,"participants":["Johannes Schindelin via GitGitGadget","Jeff King","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"385489","messageId":"d47a2aa5949a5dd3a10b89d9a77ebb89af6ba57e.1572991158.git.gitgitgadget@gmail.com","threadId":"52195","inReplyTo":"pull.453.git.1572991158.gitgitgadget@gmail.com","subject":"[PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-05T21:59:18Z","receivedAt":"2019-11-05T21:59:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\nask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\nimplicitly a different allocator than the system one.\n\nWhich means that all of cURL's allocations and releases now _need_ to\nuse that allocator.\n\nHowever, the `http_options()` function used `slist_append()` to add any\nconfigured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\nand `http_cleanup()` would release them _afterwards_, i.e. in the\npresence of custom allocators, cURL would attempt to use the wrong\nallocator to release the memory.\n\nLet's fix this by moving the initialization _before_ the\n`http_options()` function is called.\n\nSigned-off-by: Johannes Schindelin \n---\n http.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 27aa0a3192..13f50ba158 100644\n--- a/http.c\n+++ b/http.c\n@@ -1062,6 +1062,9 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tchar *normalized_url;\n \tstruct urlmatch_config config = { STRING_LIST_INIT_DUP };\n \n+\tif (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)\n+\t\tdie(\"curl_global_init failed\");\n+\n \tconfig.section = \"http\";\n \tconfig.key = NULL;\n \tconfig.collect_fn = http_options;\n@@ -1101,9 +1104,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \t}\n #endif\n \n-\tif (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)\n-\t\tdie(\"curl_global_init failed\");\n-\n \thttp_proactive_auth = proactive_auth;\n \n \tif (remote && remote->http_proxy)\n-- \ngitgitgadget\n"},{"id":"385490","messageId":"pull.453.git.1572991158.gitgitgadget@gmail.com","threadId":"52195","inReplyTo":null,"subject":"[PATCH 0/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-05T21:59:17Z","receivedAt":"2019-11-05T21:59:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This is one of those bugs that can cost entire days of work.\n\nThe symptom of this bug is that git -c http.extraheader push ... will fail\nfrequently in Git for Windows v2.24.0, but not always. When it fails,\nthough, it will cause a segmentation fault in git remote-https while calling \nhttp_cleanup() and leave no visual indication of that problem, the only\nindication of a problem will be the exit code of the caller (in this\ninstance, git push will fail with exit code 1).\n\nIn my tests during the pre-release period, I pushed many a time, it probably\nfailed a lot, in the way indicated above, and due to the absence of any\nerror message, I failed to realize that there was a problem in the first\nplace.\n\nFun side note: this bug haunted me for the best part of yesterday, when I\ntried to get Git for Windows v2.24.0 out the door. Large parts of Git for\nWindows' release management are scripted, and that script failed, claiming\nto have been unsuccessful in pushing the tag v2.24.0.windows.1, just after\nprinting a message to the extent that the tag is already up to date (except\nin the first attempt, when it reported to have been successful in pushing\nthe tag). My attempts to fix the release script were doomed to fail because\nthe root cause was not a bug in the script, but the bug fixed in this patch.\n\nJohannes Schindelin (1):\n remote-curl: unbreak http.extraHeader with custom allocators\n\n http.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\n\nbase-commit: 93b980e58f5624ee4e3b2dc0d0babaa97ef66d19\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-453%2Fdscho%2Ffix-curl-xmalloc-regression-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-453/dscho/fix-curl-xmalloc-regression-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/453\n-- \ngitgitgadget\n"},{"id":"385527","messageId":"20191106041644.GE4307@sigill.intra.peff.net","threadId":"52195","inReplyTo":"d47a2aa5949a5dd3a10b89d9a77ebb89af6ba57e.1572991158.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-06T04:16:44Z","receivedAt":"2019-11-06T04:16:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 05, 2019 at 09:59:18PM +0000, Johannes Schindelin via GitGitGadget wrote:\n\n> From: Johannes Schindelin \n> \n> In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\n> ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\n> implicitly a different allocator than the system one.\n> \n> Which means that all of cURL's allocations and releases now _need_ to\n> use that allocator.\n> \n> However, the `http_options()` function used `slist_append()` to add any\n> configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\n> and `http_cleanup()` would release them _afterwards_, i.e. in the\n> presence of custom allocators, cURL would attempt to use the wrong\n> allocator to release the memory.\n> \n> Let's fix this by moving the initialization _before_ the\n> `http_options()` function is called.\n\nNicely explained.\n\nAnother option would be to separate our config mechanism from curl\nentirely by putting the list of headers into a string_list, and then\ntransforming it later into a curl_slist. I don't think that really buys\nus much, though. This is all inside http.c, so it's fairly contained.\nIt's not like other random parts of Git are using curl's slist before\ncalling http_init().\n\nI did briefly grep around for other slist users, but they're all what\nyou'd expect: code in http-push.c and remote-curl.c creating header\nlists while working with an active http request (so well after\nhttp_init() has been called).\n\n> ---\n> http.c | 6 +++---\n> 1 file changed, 3 insertions(+), 3 deletions(-)\n\nThe patch itself looks good.\n\n-Peff\n"},{"id":"385542","messageId":"nycvar.QRO.7.76.6.1911061009230.46@tvgsbejvaqbjf.bet","threadId":"52195","inReplyTo":"20191106041644.GE4307@sigill.intra.peff.net","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-06T09:14:37Z","receivedAt":"2019-11-06T09:15:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Tue, 5 Nov 2019, Jeff King wrote:\n\n> On Tue, Nov 05, 2019 at 09:59:18PM +0000, Johannes Schindelin via GitGitGadget wrote:\n>\n> > From: Johannes Schindelin \n> >\n> > In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\n> > ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\n> > implicitly a different allocator than the system one.\n> >\n> > Which means that all of cURL's allocations and releases now _need_ to\n> > use that allocator.\n> >\n> > However, the `http_options()` function used `slist_append()` to add any\n> > configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\n> > and `http_cleanup()` would release them _afterwards_, i.e. in the\n> > presence of custom allocators, cURL would attempt to use the wrong\n> > allocator to release the memory.\n> >\n> > Let's fix this by moving the initialization _before_ the\n> > `http_options()` function is called.\n>\n> Nicely explained.\n>\n> Another option would be to separate our config mechanism from curl\n> entirely by putting the list of headers into a string_list, and then\n> transforming it later into a curl_slist. I don't think that really buys\n> us much, though.\n\nAlas, it _does_ buy us a lot, as I *just* found out (can you imagine how\nglad I am not to have rushed out another Git for Windows release?). It\nbuys us one less bug: my patch introduces the bug where\n`http.sslbackend` can no longer be used, because `curl_global_sslset()`\nneeds to be called _before_ `curl_global_init()`, but my patch breaks\nthat because we _need_ to parse the config before we can ask cURL for a\nspecific backend.\n\n> This is all inside http.c, so it's fairly contained. It's not like\n> other random parts of Git are using curl's slist before calling\n> http_init().\n\nIndeed. We cannot use cURL's slist anywhere outside of the\ncURL-dependent code because we want to keep `NO_CURL=Yep` working.\n\n> I did briefly grep around for other slist users, but they're all what\n> you'd expect: code in http-push.c and remote-curl.c creating header\n> lists while working with an active http request (so well after\n> http_init() has been called).\n\nIndeed, I came to the same conclusion that Carlo's patch only broke\nsupport for `http.extraheaders` (and only with custom allocators),\nnothing else.\n\nI will change the patch to avoid using `slist` early and send another\niteration.\n\nThanks,\nDscho\n\n> > ---\n> > http.c | 6 +++---\n> > 1 file changed, 3 insertions(+), 3 deletions(-)\n>\n> The patch itself looks good.\n>\n> -Peff\n>\n"},{"id":"385552","messageId":"xmqqsgn1tjup.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"nycvar.QRO.7.76.6.1911061009230.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-06T09:49:50Z","receivedAt":"2019-11-06T09:49:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> On Tue, 5 Nov 2019, Jeff King wrote:\n>\n>> ... transforming it later into a curl_slist. I don't think that really buys\n>> us much, though.\n>\n> Alas, it _does_ buy us a lot, as I *just* found out (can you imagine how\n> glad I am not to have rushed out another Git for Windows release?).\n\nThanks both, and especially thanks Dscho for both for a fix and for\nrestraining yourself from the urge to pull trigger too soon ;-)\n\n> I will change the patch to avoid using `slist` early and send another\n> iteration.\n\nWill look forward to seeing it.\n\nThanks.\n"},{"id":"385576","messageId":"3168ba2c9eadcf0cd7e4f2533c9306b5d2c627d0.1573034695.git.gitgitgadget@gmail.com","threadId":"52195","inReplyTo":"pull.453.v2.git.1573034695.gitgitgadget@gmail.com","subject":"[PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-06T10:04:55Z","receivedAt":"2019-11-06T10:05:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\nask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\nimplicitly a different allocator than the system one.\n\nWhich means that all of cURL's allocations and releases now _need_ to\nuse that allocator.\n\nHowever, the `http_options()` function used `slist_append()` to add any\nconfigured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\nand `http_cleanup()` would release them _afterwards_, i.e. in the\npresence of custom allocators, cURL would attempt to use the wrong\nallocator to release the memory.\n\nA naïve attempt at fixing this would move the call to\n`curl_global_init()` _before_ the config is parsed (i.e. before that\ncall to `slist_append()`).\n\nHowever, that does work, as we _also_ parse the config setting\n`http.sslbackend` and if found, call `curl_global_sslset()` which *must*\nbe called before `curl_global_init()`, for details see:\nhttps://curl.haxx.se/libcurl/c/curl_global_sslset.html\n\nSo let's instead make the config parsing entirely independent from\ncURL's data structures. Incidentally, this deletes two more lines than\nit introduces, which is nice.\n\nSigned-off-by: Johannes Schindelin \n---\n http.c | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 27aa0a3192..82f493c7fd 100644\n--- a/http.c\n+++ b/http.c\n@@ -150,7 +150,7 @@ static unsigned long empty_auth_useless =\n \n static struct curl_slist *pragma_header;\n static struct curl_slist *no_pragma_header;\n-static struct curl_slist *extra_http_headers;\n+static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;\n \n static struct active_request_slot *active_queue_head;\n \n@@ -414,11 +414,9 @@ static int http_options(const char *var, const char *value, void *cb)\n \t\tif (!value) {\n \t\t\treturn config_error_nonbool(var);\n \t\t} else if (!*value) {\n-\t\t\tcurl_slist_free_all(extra_http_headers);\n-\t\t\textra_http_headers = NULL;\n+\t\t\tstring_list_clear(&extra_http_headers, 0);\n \t\t} else {\n-\t\t\textra_http_headers =\n-\t\t\t\tcurl_slist_append(extra_http_headers, value);\n+\t\t\tstring_list_append(&extra_http_headers, value);\n \t\t}\n \t\treturn 0;\n \t}\n@@ -1199,8 +1197,7 @@ void http_cleanup(void)\n #endif\n \tcurl_global_cleanup();\n \n-\tcurl_slist_free_all(extra_http_headers);\n-\textra_http_headers = NULL;\n+\tstring_list_clear(&extra_http_headers, 0);\n \n \tcurl_slist_free_all(pragma_header);\n \tpragma_header = NULL;\n@@ -1624,10 +1621,11 @@ int run_one_slot(struct active_request_slot *slot,\n \n struct curl_slist *http_copy_default_headers(void)\n {\n-\tstruct curl_slist *headers = NULL, *h;\n+\tstruct curl_slist *headers = NULL;\n+\tconst struct string_list_item *item;\n \n-\tfor (h = extra_http_headers; h; h = h->next)\n-\t\theaders = curl_slist_append(headers, h->data);\n+\tfor_each_string_list_item(item, &extra_http_headers)\n+\t\theaders = curl_slist_append(headers, item->string);\n \n \treturn headers;\n }\n-- \ngitgitgadget\n"},{"id":"385577","messageId":"pull.453.v2.git.1573034695.gitgitgadget@gmail.com","threadId":"52195","inReplyTo":"pull.453.git.1572991158.gitgitgadget@gmail.com","subject":"[PATCH v2 0/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-11-06T10:04:54Z","receivedAt":"2019-11-06T10:05:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"This is one of those bugs that can cost entire days of work.\n\nThe symptom of this bug is that git -c http.extraheader push ... will fail\nfrequently in Git for Windows v2.24.0, but not always. When it fails,\nthough, it will cause a segmentation fault in git remote-https while calling \nhttp_cleanup() and leave no visual indication of that problem, the only\nindication of a problem will be the exit code of the caller (in this\ninstance, git push will fail with exit code 1).\n\nIn my tests during the pre-release period, I pushed many a time, it probably\nfailed a lot, in the way indicated above, and due to the absence of any\nerror message, I failed to realize that there was a problem in the first\nplace.\n\nFun side note: this bug haunted me for the best part of this past Monday,\nwhen I tried to get Git for Windows v2.24.0 out the door. Large parts of Git\nfor Windows' release management are scripted, and that script failed,\nclaiming to have been unsuccessful in pushing the tag v2.24.0.windows.1,\njust after printing a message to the extent that the tag is already up to\ndate (except in the first attempt, when it reported to have been successful\nin pushing the tag). My attempts to fix the release script were doomed to\nfail because the root cause was not a bug in the script, but the bug fixed\nin this patch.\n\nChanges since v1:\n\n * The patch was completely redone: instead of moving the call to \n curl_global_init() (which would have broken support for http.sslbackend),\n this iteration instead replaces the usage of cURL's slist in the config\n handling by using Git's own string_list.\n\nJohannes Schindelin (1):\n remote-curl: unbreak http.extraHeader with custom allocators\n\n http.c | 18 ++++++++----------\n 1 file changed, 8 insertions(+), 10 deletions(-)\n\n\nbase-commit: 93b980e58f5624ee4e3b2dc0d0babaa97ef66d19\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-453%2Fdscho%2Ffix-curl-xmalloc-regression-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-453/dscho/fix-curl-xmalloc-regression-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/453\n\nRange-diff vs v1:\n\n 1: d47a2aa594 ! 1: 3168ba2c9e remote-curl: unbreak http.extraHeader with custom allocators\n @@ -15,8 +15,18 @@\n presence of custom allocators, cURL would attempt to use the wrong\n allocator to release the memory.\n \n - Let's fix this by moving the initialization _before_ the\n - `http_options()` function is called.\n + A naïve attempt at fixing this would move the call to\n + `curl_global_init()` _before_ the config is parsed (i.e. before that\n + call to `slist_append()`).\n +\n + However, that does work, as we _also_ parse the config setting\n + `http.sslbackend` and if found, call `curl_global_sslset()` which *must*\n + be called before `curl_global_init()`, for details see:\n + https://curl.haxx.se/libcurl/c/curl_global_sslset.html\n +\n + So let's instead make the config parsing entirely independent from\n + cURL's data structures. Incidentally, this deletes two more lines than\n + it introduces, which is nice.\n \n Signed-off-by: Johannes Schindelin \n \n @@ -24,22 +34,50 @@\n --- a/http.c\n +++ b/http.c\n @@\n - \tchar *normalized_url;\n - \tstruct urlmatch_config config = { STRING_LIST_INIT_DUP };\n - \n -+\tif (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)\n -+\t\tdie(\"curl_global_init failed\");\n -+\n - \tconfig.section = \"http\";\n - \tconfig.key = NULL;\n - \tconfig.collect_fn = http_options;\n + \n + static struct curl_slist *pragma_header;\n + static struct curl_slist *no_pragma_header;\n +-static struct curl_slist *extra_http_headers;\n ++static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;\n + \n + static struct active_request_slot *active_queue_head;\n + \n @@\n + \t\tif (!value) {\n + \t\t\treturn config_error_nonbool(var);\n + \t\t} else if (!*value) {\n +-\t\t\tcurl_slist_free_all(extra_http_headers);\n +-\t\t\textra_http_headers = NULL;\n ++\t\t\tstring_list_clear(&extra_http_headers, 0);\n + \t\t} else {\n +-\t\t\textra_http_headers =\n +-\t\t\t\tcurl_slist_append(extra_http_headers, value);\n ++\t\t\tstring_list_append(&extra_http_headers, value);\n + \t\t}\n + \t\treturn 0;\n \t}\n +@@\n #endif\n + \tcurl_global_cleanup();\n + \n +-\tcurl_slist_free_all(extra_http_headers);\n +-\textra_http_headers = NULL;\n ++\tstring_list_clear(&extra_http_headers, 0);\n + \n + \tcurl_slist_free_all(pragma_header);\n + \tpragma_header = NULL;\n +@@\n + \n + struct curl_slist *http_copy_default_headers(void)\n + {\n +-\tstruct curl_slist *headers = NULL, *h;\n ++\tstruct curl_slist *headers = NULL;\n ++\tconst struct string_list_item *item;\n \n --\tif (curl_global_init(CURL_GLOBAL_ALL) != CURLE_OK)\n --\t\tdie(\"curl_global_init failed\");\n --\n - \thttp_proactive_auth = proactive_auth;\n +-\tfor (h = extra_http_headers; h; h = h->next)\n +-\t\theaders = curl_slist_append(headers, h->data);\n ++\tfor_each_string_list_item(item, &extra_http_headers)\n ++\t\theaders = curl_slist_append(headers, item->string);\n \n - \tif (remote && remote->http_proxy)\n + \treturn headers;\n + }\n\n-- \ngitgitgadget\n"},{"id":"385583","messageId":"20191106112901.GA25966@sigill.intra.peff.net","threadId":"52195","inReplyTo":"3168ba2c9eadcf0cd7e4f2533c9306b5d2c627d0.1573034695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-11-06T11:29:01Z","receivedAt":"2019-11-06T11:29:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 06, 2019 at 10:04:55AM +0000, Johannes Schindelin via GitGitGadget wrote:\n\n> A naïve attempt at fixing this would move the call to\n> `curl_global_init()` _before_ the config is parsed (i.e. before that\n> call to `slist_append()`).\n> \n> However, that does work, as we _also_ parse the config setting\n> `http.sslbackend` and if found, call `curl_global_sslset()` which *must*\n> be called before `curl_global_init()`, for details see:\n> https://curl.haxx.se/libcurl/c/curl_global_sslset.html\n\nYikes, good catch. It didn't even occur to me that there might be curl\nthings we had to do _before_ calling curl_global_init().\n\n> So let's instead make the config parsing entirely independent from\n> cURL's data structures. Incidentally, this deletes two more lines than\n> it introduces, which is nice.\n\nYes, I actually find the resulting code easier to read. I had feared\nhaving to add an extra step to initialize the slist, but it's all\nhandled quite neatly in http_copy_default_headers().\n\n> http.c | 18 ++++++++----------\n> 1 file changed, 8 insertions(+), 10 deletions(-)\n\nThe patch itself looks good to me.\n\n-Peff\n"},{"id":"385587","messageId":"xmqq7e4dtd9n.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"3168ba2c9eadcf0cd7e4f2533c9306b5d2c627d0.1573034695.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-06T12:12:04Z","receivedAt":"2019-11-06T12:12:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\n> ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\n> implicitly a different allocator than the system one.\n>\n> Which means that all of cURL's allocations and releases now _need_ to\n> use that allocator.\n>\n> However, the `http_options()` function used `slist_append()` to add any\n> configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\n> and `http_cleanup()` would release them _afterwards_, i.e. in the\n> presence of custom allocators, cURL would attempt to use the wrong\n> allocator to release the memory.\n\ns/allocator/de&/; perhaps, even though it is clear enough from the\ncontext, so it is probably OK as is.\n\n> A naïve attempt at fixing this would move the call to\n> `curl_global_init()` _before_ the config is parsed (i.e. before that\n> call to `slist_append()`).\n>\n> However, that does work, as we _also_ parse the config setting\n\ns/does work/does not work/; presumably?\n\n> `http.sslbackend` and if found, call `curl_global_sslset()` which *must*\n> be called before `curl_global_init()`, for details see:\n> https://curl.haxx.se/libcurl/c/curl_global_sslset.html\n>\n> So let's instead make the config parsing entirely independent from\n> cURL's data structures. Incidentally, this deletes two more lines than\n> it introduces, which is nice.\n\nYeah, string_list_clear() is more concise than curl_slist_free_all(),\nand we have already been copying one list to another anyway, so we\nlucked out ;-)\n\nThe patch looked good to me, too.\n"},{"id":"385588","messageId":"xmqq36f1td4e.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"pull.453.v2.git.1573034695.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-06T12:15:13Z","receivedAt":"2019-11-06T12:15:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> Johannes Schindelin (1):\n> remote-curl: unbreak http.extraHeader with custom allocators\n>\n> http.c | 18 ++++++++----------\n> 1 file changed, 8 insertions(+), 10 deletions(-)\n>\n> base-commit: 93b980e58f5624ee4e3b2dc0d0babaa97ef66d19\n\nThis fixes by immediately building on top of the patch that caused\nthe issue, so I should do the same so that any integration branch\n(either mine or somebody else's) that has the breakage can merge the\nfix in. Good.\n\nThanks.\n"},{"id":"385617","messageId":"nycvar.QRO.7.76.6.1911062031480.46@tvgsbejvaqbjf.bet","threadId":"52195","inReplyTo":"xmqq7e4dtd9n.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-06T19:34:51Z","receivedAt":"2019-11-06T19:35:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 6 Nov 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n>\n> > From: Johannes Schindelin \n> >\n> > In 93b980e58f5 (http: use xmalloc with cURL, 2019-08-15), we started to\n> > ask cURL to use `xmalloc()`, and if compiled with nedmalloc, that means\n> > implicitly a different allocator than the system one.\n> >\n> > Which means that all of cURL's allocations and releases now _need_ to\n> > use that allocator.\n> >\n> > However, the `http_options()` function used `slist_append()` to add any\n> > configured extra HTTP header(s) _before_ asking cURL to use `xmalloc()`,\n> > and `http_cleanup()` would release them _afterwards_, i.e. in the\n> > presence of custom allocators, cURL would attempt to use the wrong\n> > allocator to release the memory.\n>\n> s/allocator/de&/; perhaps, even though it is clear enough from the\n> context, so it is probably OK as is.\n>\n> > A naïve attempt at fixing this would move the call to\n> > `curl_global_init()` _before_ the config is parsed (i.e. before that\n> > call to `slist_append()`).\n> >\n> > However, that does work, as we _also_ parse the config setting\n>\n> s/does work/does not work/; presumably?\n\nIndeed. Could I ask you to fix up locally, or do you want me to send a\nnew revision of the patch?\n\nCiao,\nDscho\n\n>\n> > `http.sslbackend` and if found, call `curl_global_sslset()` which *must*\n> > be called before `curl_global_init()`, for details see:\n> > https://curl.haxx.se/libcurl/c/curl_global_sslset.html\n> >\n> > So let's instead make the config parsing entirely independent from\n> > cURL's data structures. Incidentally, this deletes two more lines than\n> > it introduces, which is nice.\n>\n> Yeah, string_list_clear() is more concise than curl_slist_free_all(),\n> and we have already been copying one list to another anyway, so we\n> lucked out ;-)\n>\n> The patch looked good to me, too.\n>\n"},{"id":"385618","messageId":"nycvar.QRO.7.76.6.1911062029230.46@tvgsbejvaqbjf.bet","threadId":"52195","inReplyTo":"xmqqsgn1tjup.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-06T19:38:46Z","receivedAt":"2019-11-06T19:39:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 6 Nov 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > On Tue, 5 Nov 2019, Jeff King wrote:\n> >\n> >> ... transforming it later into a curl_slist. I don't think that really buys\n> >> us much, though.\n> >\n> > Alas, it _does_ buy us a lot, as I *just* found out (can you imagine how\n> > glad I am not to have rushed out another Git for Windows release?).\n>\n> Thanks both, and especially thanks Dscho for both for a fix and for\n> restraining yourself from the urge to pull trigger too soon ;-)\n\nWell, I _will_ trigger the pipeline publishing Git for Windows\nv2.24.0(2) in a moment, with this fix, but more importantly with a fix\nfor the installer that tries to install some Cygwin-style symlinks,\nwhich was broken in non-US locales (most notably in a Japanese setup;\napparently there _still_ are developers who did not yet move away from a\nJapanese locale... at least on Windows).\n\nCiao,\nDscho\n\n>\n> > I will change the patch to avoid using `slist` early and send another\n> > iteration.\n>\n> Will look forward to seeing it.\n>\n> Thanks.\n>\n"},{"id":"385657","messageId":"xmqq1ruks0rz.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"nycvar.QRO.7.76.6.1911062031480.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-07T05:39:28Z","receivedAt":"2019-11-07T05:39:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> On Wed, 6 Nov 2019, Junio C Hamano wrote:\n>\n>> > presence of custom allocators, cURL would attempt to use the wrong\n>> > allocator to release the memory.\n>>\n>> s/allocator/de&/; perhaps, even though it is clear enough from the\n>> context, so it is probably OK as is.\n>>\n>> > A naïve attempt at fixing this would move the call to\n>> > `curl_global_init()` _before_ the config is parsed (i.e. before that\n>> > call to `slist_append()`).\n>> >\n>> > However, that does work, as we _also_ parse the config setting\n>>\n>> s/does work/does not work/; presumably?\n>\n> Indeed. Could I ask you to fix up locally, or do you want me to send a\n> new revision of the patch?\n\nYou can certainly tell me to locally tweak, but you'd need to be\nmore specific when some of my suggestions were \"perhaps\" (aka \"you\ncould do it this way, which may be better, but I do not care too\ndeeply---I am OK with whichever you chooose after you (re)think\nabout it, but I am not choosing for you\").\n\nFor now, I'll do the \"does not work\" thing only.\n\nThanks.\n"},{"id":"385700","messageId":"nycvar.QRO.7.76.6.1911071338320.46@tvgsbejvaqbjf.bet","threadId":"52195","inReplyTo":"xmqq1ruks0rz.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-07T12:40:36Z","receivedAt":"2019-11-07T12:41:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 7 Nov 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > On Wed, 6 Nov 2019, Junio C Hamano wrote:\n> >\n> >> > presence of custom allocators, cURL would attempt to use the wrong\n> >> > allocator to release the memory.\n> >>\n> >> s/allocator/de&/; perhaps, even though it is clear enough from the\n> >> context, so it is probably OK as is.\n> >>\n> >> > A naïve attempt at fixing this would move the call to\n> >> > `curl_global_init()` _before_ the config is parsed (i.e. before that\n> >> > call to `slist_append()`).\n> >> >\n> >> > However, that does work, as we _also_ parse the config setting\n> >>\n> >> s/does work/does not work/; presumably?\n> >\n> > Indeed. Could I ask you to fix up locally, or do you want me to send a\n> > new revision of the patch?\n>\n> You can certainly tell me to locally tweak, but you'd need to be\n> more specific when some of my suggestions were \"perhaps\" (aka \"you\n> could do it this way, which may be better, but I do not care too\n> deeply---I am OK with whichever you chooose after you (re)think\n> about it, but I am not choosing for you\").\n\nRight, I should have been more specific.\n\n> For now, I'll do the \"does not work\" thing only.\n\nThanks!\n\nFWIW I would like to stick with \"custom allocator\" because even\nreleasing the memory is the duty of an \"allocator\", even if that\nallocator happens to \"deallocate\" at that point what it has allocated\nbefore.\n\nCiao,\nDscho\n"},{"id":"385748","messageId":"xmqqy2wrnk6o.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"nycvar.QRO.7.76.6.1911071338320.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-08T03:03:43Z","receivedAt":"2019-11-08T03:03:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> FWIW I would like to stick with \"custom allocator\" because even\n> releasing the memory is the duty of an \"allocator\", even if that\n> allocator happens to \"deallocate\" at that point what it has allocated\n> before.\n\nMy preference is to think of the set of functions we feed\ncurl-global-init-mem as a (allocator, deallocator, reallocator, ...)\ntuple.\n\nBut you are calling that set itself as an allocator that has service\nfunctions like (alloc, free, realloc), and as long as that (i.e. how\nyou defined the word \"allocator\") is clear to the readers that is\nperfectly OK. \n\nTo me as one reader, it was not clear and that was where my comment\ncame from.\n"},{"id":"385764","messageId":"xmqqimnun4v5.fsf@gitster-ct.c.googlers.com","threadId":"52195","inReplyTo":"nycvar.QRO.7.76.6.1911062029230.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-11-08T08:34:38Z","receivedAt":"2019-11-08T08:34:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> ... which was broken in non-US locales (most notably in a Japanese setup;\n> apparently there _still_ are developers who did not yet move away from a\n> Japanese locale... at least on Windows).\n\nI think the above was a reference to a tangential comment I made in\na response to Peff.\n\nOh, I do not expect Japanese Windows users would \"move away from\"\nJapanese locale, ever.\n\nI do however know that an app that supports only legacy encoding is\nfrowned upon these days. They still use and will keep using\nJapanese menus and messages, and they still write their documents in\nJapanese and not in US English.\n\nBut they store their Japenese documents encoded in UTF-8 on their\nsystem that is in Japanese locale.\n\nThere are two models of gadgets I am interested in getting, between\nwhich one of them that is slightly older supports UTF-8 and MS-Kanji\n(aka Shift-JIS) while the latest model only supports MS-Kanji. The\nlist price of them are comparable (actually, the latest one lists a\nbit more), but the latest model is deeply discounted while the other\none with UTF-8 is not as much. At shopping sites, user reviews\noften mention \"I've migrated my text to UTF-8 already and going back\nto Shift JIS in this year is too cumbersome, so I'll skip this\nlatest model\".\n\nI am hoping a software update might happen, and will pull the\ntrigger once the latest one starts supporting UTF-8 ;-)\n"},{"id":"385783","messageId":"nycvar.QRO.7.76.6.1911081443500.46@tvgsbejvaqbjf.bet","threadId":"52195","inReplyTo":"xmqqimnun4v5.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] remote-curl: unbreak http.extraHeader with custom allocators","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-11-08T13:44:06Z","receivedAt":"2019-11-08T13:44:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Fri, 8 Nov 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > ... which was broken in non-US locales (most notably in a Japanese setup;\n> > apparently there _still_ are developers who did not yet move away from a\n> > Japanese locale... at least on Windows).\n>\n> I think the above was a reference to a tangential comment I made in\n> a response to Peff.\n>\n> Oh, I do not expect Japanese Windows users would \"move away from\"\n> Japanese locale, ever.\n>\n> I do however know that an app that supports only legacy encoding is\n> frowned upon these days. They still use and will keep using\n> Japanese menus and messages, and they still write their documents in\n> Japanese and not in US English.\n>\n> But they store their Japenese documents encoded in UTF-8 on their\n> system that is in Japanese locale.\n>\n> There are two models of gadgets I am interested in getting, between\n> which one of them that is slightly older supports UTF-8 and MS-Kanji\n> (aka Shift-JIS) while the latest model only supports MS-Kanji. The\n> list price of them are comparable (actually, the latest one lists a\n> bit more), but the latest model is deeply discounted while the other\n> one with UTF-8 is not as much. At shopping sites, user reviews\n> often mention \"I've migrated my text to UTF-8 already and going back\n> to Shift JIS in this year is too cumbersome, so I'll skip this\n> latest model\".\n>\n> I am hoping a software update might happen, and will pull the\n> trigger once the latest one starts supporting UTF-8 ;-)\n\nThank you for that interesting note.\n\nCiao,\nDscho\n"}]}