{"thread":{"id":"58425","subject":"[PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","startedAt":"2022-09-13T19:26:05Z","lastAt":"2023-03-28T18:55:25Z","messageCount":223,"participants":["Matthew John Cheetham via GitGitGadget","Derrick Stolee","Lessley Dennington","Matthew John Cheetham","M Hickford","Jeff Hostetler","Glen Choo","Junio C Hamano","Victoria Dye","Ævar Arnfjörð Bjarmason","Jeff King","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"462984","messageId":"pull.1352.git.1663097156.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":null,"subject":"[PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:48Z","receivedAt":"2022-09-13T19:26:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Hello! I have an RFC to update the existing credential helper design in\norder to allow for some new scenarios, and future evolution of auth methods\nthat Git hosts may wish to provide. I outline the background, summary of\nchanges and some challenges below. I also attach a series of patches to\nillustrate the design proposal.\n\nOne missing element from the patches are extensive tests of the new\nbehaviour. It appears existing tests focus either on the credential helper\nprotocol/format, or rely on testing basic authentication only via an Apache\nwebserver. In order to have a full end to end test coverage of these new\nfeatures it make be that we need a more comprehensive test bed to mock these\nmore nuanced authentication methods. I lean on the experts on the list for\nadvice here.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [1]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [2], and Negotiate (RFC 2478)\n[3]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [4]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [5], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [6]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Leverage newest identity standards, enhancements, and threat\n mitigations - all without updating Git.\n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[n]=value, where n\n is a zero based index of the values.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[n].\n\n 3. Teach Git to specify authentication schemes other than Basic in\n subsequent HTTP requests based on credential helper responses.\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [7] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [8] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [8] authority, a credential helper can use OpenID Connect's\nDiscovery [9] and Dynamic Client Registration [9] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[0]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[1]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [11]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture flexibility\n==================\n\nBy allowing the credential helpers decide the best authentication scheme, we\ncan allow the remote Git server to both offer new schemes (or remove old\nones) that enlightened credential helpers could take immediate advantage of,\nand to use credentials that are much more tightly scoped and bound to the\nspecific request.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[0]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \n\n\n\nShould Git not control the set of authentication schemes?\n=========================================================\n\nOne concern that the reader may have regarding these changes is in allowing\nhelpers to select the authentication mechanism to use, it may be possible\nthat a weaker form of authentication is used.\n\nTake for example a Git remote server that responds with the following\nauthentication schemes:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Negotiate ...\nWWW-Authenticate: Basic ...\n\n\nToday Git (and libcurl) prefer to Negotiate over Basic authentication [12].\nIf a helper responded with authtype=basic Git would now be using a \"less\nsecure\" mechanism.\n\nThe reason we still propose the credential helper decide on the\nauthentication scheme is that Git is not the best placed entity to decide\nwhat type of authentication should be used for a particular request (see\nDesign Principle 3).\n\nOAuth Bearer tokens are often bundled in Basic Authorization headers [13],\nbut given that the tokens are/can be short-lived and have a highly scoped\nset of permissions, this solution could be argued as being more secure than\nsomething like NTLM [14]. Similarly, the user may wish to be consulted on\nselecting a particular user account, or directly selecting an authentication\nmechanism for a request that otherwise they would not be able to use.\n\nAlso, as new authentication protocols appear Git does not need to be\nmodified or updated for the user to take advantage of them; the credential\nhelpers take on the responsibility of learning and selecting the \"best\"\noption.\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [1] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [2] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [3] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [4] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [5] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [6] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [7] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [8] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [9] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [10] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [11] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n * [12] libcurl http.c pickoneauth Function\n https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416\n\n * [13] Git Credential Manager GitHub Host Provider (using PAT as password)\n https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157\n\n * [14] NT LAN Manager (NTLM) Authentication Protocol\n https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4\n\nMatthew John Cheetham (8):\n wincred: ignore unknown lines (do not die)\n netrc: ignore unknown lines (do not die)\n osxkeychain: clarify that we ignore unknown lines\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n http: store all request headers on active_request_slot\n http: move proactive auth to first slot creation\n http: set specific auth scheme depending on credential\n\n Documentation/git-credential.txt | 18 ++\n .../netrc/git-credential-netrc.perl | 5 +-\n .../osxkeychain/git-credential-osxkeychain.c | 5 +\n .../wincred/git-credential-wincred.c | 7 +-\n credential.c | 18 ++\n credential.h | 11 +\n git-curl-compat.h | 7 +\n http-push.c | 103 ++++-----\n http-walker.c | 2 +-\n http.c | 199 +++++++++++++-----\n http.h | 4 +-\n remote-curl.c | 36 ++--\n t/lib-httpd/apache.conf | 13 ++\n t/t5551-http-fetch-smart.sh | 46 ++++\n 14 files changed, 335 insertions(+), 139 deletions(-)\n\n\nbase-commit: dd3f6c4cae7e3b15ce984dce8593ff7569650e24\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n-- \ngitgitgadget\n"},{"id":"462985","messageId":"6426f9c3954866b3fd9259d1a58d2c41dc42e17f.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 1/8] wincred: ignore unknown lines (do not die)","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:49Z","receivedAt":"2022-09-13T19:26:07Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIt is the expectation that credential helpers be liberal in what they\naccept and conservative in what they return, to allow for future growth\nand evolution of the protocol/interaction.\n\nAll of the other helpers (store, cache, osxkeychain, libsecret,\ngnome-keyring) except `netrc` currently ignore any credential lines\nthat are not recognised, whereas the Windows helper (wincred) instead\ndies.\n\nFix the discrepancy and ignore unknown lines in the wincred helper.\n\nSigned-off-by: Matthew John Cheetham \n---\n contrib/credential/wincred/git-credential-wincred.c | 7 +++++--\n 1 file changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/contrib/credential/wincred/git-credential-wincred.c b/contrib/credential/wincred/git-credential-wincred.c\nindex 5091048f9c6..ead6e267c78 100644\n--- a/contrib/credential/wincred/git-credential-wincred.c\n+++ b/contrib/credential/wincred/git-credential-wincred.c\n@@ -278,8 +278,11 @@ static void read_credential(void)\n \t\t\twusername = utf8_to_utf16_dup(v);\n \t\t} else if (!strcmp(buf, \"password\"))\n \t\t\tpassword = utf8_to_utf16_dup(v);\n-\t\telse\n-\t\t\tdie(\"unrecognized input\");\n+\t\t/*\n+\t\t * Ignore other lines; we don't know what they mean, but\n+\t\t * this future-proofs us when later versions of git do\n+\t\t * learn new lines, and the helpers are updated to match.\n+\t\t */\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"462986","messageId":"ae5c1bfc092e98b810757e752efd7cfde48a3809.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 2/8] netrc: ignore unknown lines (do not die)","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:50Z","receivedAt":"2022-09-13T19:26:09Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nContrary to the documentation on credential helpers, as well as the help\ntext for git-credential-netrc itself, this helper will `die` when\npresented with an unknown property/attribute/token.\n\nCorrect the behaviour here by skipping and ignoring any tokens that are\nunknown. This means all helpers in the tree are consistent and ignore\nany unknown credential properties/attributes.\n\nSigned-off-by: Matthew John Cheetham \n---\n contrib/credential/netrc/git-credential-netrc.perl | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/credential/netrc/git-credential-netrc.perl b/contrib/credential/netrc/git-credential-netrc.perl\nindex bc57cc65884..9fb998ae090 100755\n--- a/contrib/credential/netrc/git-credential-netrc.perl\n+++ b/contrib/credential/netrc/git-credential-netrc.perl\n@@ -356,7 +356,10 @@ sub read_credential_data_from_stdin {\n \t\tnext unless m/^([^=]+)=(.+)/;\n \n \t\tmy ($token, $value) = ($1, $2);\n-\t\tdie \"Unknown search token $token\" unless exists $q{$token};\n+\n+\t\t# skip any unknown tokens\n+\t\tnext unless exists $q{$token};\n+\n \t\t$q{$token} = $value;\n \t\tlog_debug(\"We were given search token $token and value $value\");\n \t}\n-- \ngitgitgadget\n\n"},{"id":"462987","messageId":"2ece562a5952b5752de5040b17ec36076164c72f.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 3/8] osxkeychain: clarify that we ignore unknown lines","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:51Z","receivedAt":"2022-09-13T19:26:12Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nLike in all the other credential helpers, the osxkeychain helper\nignores unknown credential lines.\n\nAdd a comment (a la the other helpers) to make it clear and explicit\nthat this is the desired behaviour.\n\nSigned-off-by: Matthew John Cheetham \n---\n contrib/credential/osxkeychain/git-credential-osxkeychain.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/contrib/credential/osxkeychain/git-credential-osxkeychain.c b/contrib/credential/osxkeychain/git-credential-osxkeychain.c\nindex bf77748d602..e29cc28779d 100644\n--- a/contrib/credential/osxkeychain/git-credential-osxkeychain.c\n+++ b/contrib/credential/osxkeychain/git-credential-osxkeychain.c\n@@ -159,6 +159,11 @@ static void read_credential(void)\n \t\t\tusername = xstrdup(v);\n \t\telse if (!strcmp(buf, \"password\"))\n \t\t\tpassword = xstrdup(v);\n+\t\t/*\n+\t\t * Ignore other lines; we don't know what they mean, but\n+\t\t * this future-proofs us when later versions of git do\n+\t\t * learn new lines, and the helpers are updated to match.\n+\t\t */\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"462988","messageId":"936545004b8b46cbe24d8069cfd95ae5b5f98593.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 5/8] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:53Z","receivedAt":"2022-09-13T19:26:16Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[n]` properties where `n` is a\nzero-indexed number, reflecting the order the WWW-Authenticate headers\nappeared in the HTTP response.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 9 +++++++\n credential.c | 12 +++++++++\n t/lib-httpd/apache.conf | 13 +++++++++\n t/t5551-http-fetch-smart.sh | 46 ++++++++++++++++++++++++++++++++\n 4 files changed, 80 insertions(+)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex f18673017f5..7d4a788c63d 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -151,6 +151,15 @@ Git understands the following attributes:\n \twere read (e.g., `url=https://example.com` would behave as if\n \t`protocol=https` and `host=example.com` had been provided). This\n \tcan help callers avoid parsing URLs themselves.\n+\n+`wwwauth[n]`::\n+\n+\tWhen an HTTP response is received that includes one or more\n+\t'WWW-Authenticate' authentication headers, these can be passed to Git\n+\t(and subsequent credential helpers) with these attributes.\n+\tEach 'WWW-Authenticate' header value should be passed as a separate\n+\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n+\tappear in the HTTP response.\n +\n Note that specifying a protocol is mandatory and if the URL\n doesn't specify a hostname (e.g., \"cert:///path/to/file\") the\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..4ad40323fc7 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tint i = 0;\n+\tfor (; i < vec->nr; i++) {\n+\t\tconst char *full_key = xstrfmt(\"%s[%d]\", key, i);\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t\tfree((void*)full_key);\n+\t}\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 497b9b9d927..fe118d76f98 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -235,6 +235,19 @@ SSLEngine On\n \tRequire valid-user\n \n \n+# Advertise two additional auth methods above \"Basic\".\n+# Neither of them actually work but serve test cases showing these\n+# additional auth headers are consumed correctly.\n+\n+\tAuthType Basic\n+\tAuthName \"git-auth\"\n+\tAuthUserFile passwd\n+\tRequire valid-user\n+\tSetEnvIf Authorization \"^\\S+\" authz\n+\tHeader always add WWW-Authenticate \"Bearer authority=https://login.example.com\" env=!authz\n+\tHeader always add WWW-Authenticate \"FooAuth foo=bar baz=1\" env=!authz\n+\n+\n RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]\n RewriteCond %{REQUEST_URI} /git-receive-pack$\n RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]\ndiff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh\nindex 6a38294a476..c99d8e253df 100755\n--- a/t/t5551-http-fetch-smart.sh\n+++ b/t/t5551-http-fetch-smart.sh\n@@ -564,6 +564,52 @@ test_expect_success 'http auth forgets bogus credentials' '\n \texpect_askpass both user@host\n '\n \n+test_expect_success 'http auth sends www-auth headers to credential helper' '\n+\twrite_script git-credential-tee <<-\\EOF &&\n+\t\tcmd=$1\n+\t\tteefile=credential-$cmd\n+\t\tif [ -f \"$teefile\" ]; then\n+\t\t\trm $teefile\n+\t\tfi\n+\t\t(\n+\t\t\twhile read line;\n+\t\t\tdo\n+\t\t\t\tif [ -z \"$line\" ]; then\n+\t\t\t\t\texit 0\n+\t\t\t\tfi\n+\t\t\t\techo \"$line\" >> $teefile\n+\t\t\t\techo $line\n+\t\t\tdone\n+\t\t) | git credential-store $cmd\n+\tEOF\n+\n+\tcat >expected-get <<-EOF &&\n+\tprotocol=http\n+\thost=127.0.0.1:5551\n+\twwwauth[0]=Bearer authority=https://login.example.com\n+\twwwauth[1]=FooAuth foo=bar baz=1\n+\twwwauth[2]=Basic realm=\"git-auth\"\n+\tEOF\n+\n+\tcat >expected-store <<-EOF &&\n+\tprotocol=http\n+\thost=127.0.0.1:5551\n+\tusername=user@host\n+\tpassword=pass@host\n+\tEOF\n+\n+\trm -f .git-credentials &&\n+\ttest_config credential.helper tee &&\n+\tset_askpass user@host pass@host &&\n+\t(\n+\t\tPATH=\"$PWD:$PATH\" &&\n+\t\tgit ls-remote \"$HTTPD_URL/auth-wwwauth/smart/repo.git\"\n+\t) &&\n+\texpect_askpass both user@host &&\n+\ttest_cmp expected-get credential-get &&\n+\ttest_cmp expected-store credential-store\n+'\n+\n test_expect_success 'client falls back from v2 to v0 to match server' '\n \tGIT_TRACE_PACKET=$PWD/trace \\\n \tGIT_TEST_PROTOCOL_VERSION=2 \\\n-- \ngitgitgadget\n\n"},{"id":"462989","messageId":"78e66d56605cfb1c7000edf329ac16c05a5d69b0.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 4/8] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:52Z","receivedAt":"2022-09-13T19:26:20Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 10 +++++++\n http.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 88 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6a9d4e3de07 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,14 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +139,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex 5d0502f51fd..091321af98e 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,81 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tconst char *z = NULL;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val)) val++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\tconst char **v = values->v + values->nr - 1;\n+\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n+\n+\t\tfree((void*)*v);\n+\t\t*v = append;\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (skip_iprefix(buf.buf, \"http/\", &z))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1829,6 +1904,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"462990","messageId":"cae7180bc37663e0499fd15fe36b39e70b046d35.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 7/8] http: move proactive auth to first slot creation","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:55Z","receivedAt":"2022-09-13T19:26:22Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRather than proactively seek credentials to authenticate a request at\n`http_init()` time, do it when the first `active_request_slot` is\ncreated.\n\nBecause credential helpers may modify the headers used for a request we\ncan only auth when a slot is created (when we can first start to gather\nrequest headers).\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 20 +++++++++++---------\n 1 file changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 42616f746b1..8e107ff19b8 100644\n--- a/http.c\n+++ b/http.c\n@@ -514,18 +514,18 @@ static int curl_empty_auth_enabled(void)\n \treturn 0;\n }\n \n-static void init_curl_http_auth(CURL *result)\n+static void init_curl_http_auth(struct active_request_slot *slot)\n {\n \tif (!http_auth.username || !*http_auth.username) {\n \t\tif (curl_empty_auth_enabled())\n-\t\t\tcurl_easy_setopt(result, CURLOPT_USERPWD, \":\");\n+\t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n \t}\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n }\n \n /* *var must be free-able */\n@@ -900,9 +900,6 @@ static CURL *get_curl_handle(void)\n #endif\n \t}\n \n-\tif (http_proactive_auth)\n-\t\tinit_curl_http_auth(result);\n-\n \tif (getenv(\"GIT_SSL_VERSION\"))\n \t\tssl_version = getenv(\"GIT_SSL_VERSION\");\n \tif (ssl_version && *ssl_version) {\n@@ -1259,6 +1256,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n \n+\tint proactive_auth = 0;\n \tint num_transfers;\n \n \t/* Wait for a slot to open up if the queue is full */\n@@ -1281,6 +1279,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n \t\t\tactive_queue_head = newslot;\n+\n+\t\t\t/* Auth first slot if asked for proactive auth */\n+\t\t\tproactive_auth = http_proactive_auth;\n \t\t} else {\n \t\t\twhile (slot->next != NULL)\n \t\t\t\tslot = slot->next;\n@@ -1335,8 +1336,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);\n-\tif (http_auth.password || curl_empty_auth_enabled())\n-\t\tinit_curl_http_auth(slot->curl);\n+\n+\tif (http_auth.password || curl_empty_auth_enabled() || proactive_auth)\n+\t\tinit_curl_http_auth(slot);\n \n \treturn slot;\n }\n-- \ngitgitgadget\n\n"},{"id":"462991","messageId":"7f827067f55d596284eb2ad764e59d402c75be18.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 8/8] http: set specific auth scheme depending on credential","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:56Z","receivedAt":"2022-09-13T19:26:24Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a new credential field `authtype` that can be used by\ncredential helpers to indicate the type of the credential or\nauthentication mechanism to use for a request.\n\nModify http.c to now specify the correct authentication scheme or\ncredential type when authenticating the curl handle. If the new\n`authtype` field in the credential structure is `NULL` or \"Basic\" then\nuse the existing username/password options. If the field is \"Bearer\"\nthen use the OAuth bearer token curl option. Otherwise, the `authtype`\nfield is the authentication scheme and the `password` field is the\nraw, unencoded value.\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 9 +++++++++\n credential.c | 5 +++++\n credential.h | 1 +\n git-curl-compat.h | 7 +++++++\n http.c | 24 +++++++++++++++++++++---\n 5 files changed, 43 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex 7d4a788c63d..3b6ef6f4906 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -152,6 +152,15 @@ Git understands the following attributes:\n \t`protocol=https` and `host=example.com` had been provided). This\n \tcan help callers avoid parsing URLs themselves.\n \n+`authtype`::\n+\n+\tIndicates the type of authentication scheme used. If this is not\n+\tpresent the default is \"Basic\".\n+\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n+\tIf an unknown value is provided, this is taken as the authentication\n+\tscheme for the `Authorization` header, and the `password` field is\n+\tused as the raw unencoded authorization parameters of the same header.\n+\n `wwwauth[n]`::\n \n \tWhen an HTTP response is received that includes one or more\ndiff --git a/credential.c b/credential.c\nindex 4ad40323fc7..9d4a0f3fd51 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)\n \tfree(c->path);\n \tfree(c->username);\n \tfree(c->password);\n+\tfree(c->authtype);\n \tstring_list_clear(&c->helpers, 0);\n \tstrvec_clear(&c->wwwauth_headers);\n \n@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)\n \t\t} else if (!strcmp(key, \"path\")) {\n \t\t\tfree(c->path);\n \t\t\tc->path = xstrdup(value);\n+\t\t} else if (!strcmp(key, \"authtype\")) {\n+\t\t\tfree(c->authtype);\n+\t\t\tc->authtype = xstrdup(value);\n \t\t} else if (!strcmp(key, \"url\")) {\n \t\t\tcredential_from_url(c, value);\n \t\t} else if (!strcmp(key, \"quit\")) {\n@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_item(fp, \"authtype\", c->authtype, 0);\n \tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \ndiff --git a/credential.h b/credential.h\nindex 6a9d4e3de07..a6572aacf1d 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -135,6 +135,7 @@ struct credential {\n \tchar *protocol;\n \tchar *host;\n \tchar *path;\n+\tchar *authtype;\n };\n \n #define CREDENTIAL_INIT { \\\ndiff --git a/git-curl-compat.h b/git-curl-compat.h\nindex 56a83b6bbd8..74732500a9f 100644\n--- a/git-curl-compat.h\n+++ b/git-curl-compat.h\n@@ -126,4 +126,11 @@\n #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS\n #endif\n \n+/**\n+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.\n+ */\n+#if LIBCURL_VERSION_NUM >= 0x073D00\n+#define GIT_CURL_HAVE_CURLAUTH_BEARER\n+#endif\n+\n #endif\ndiff --git a/http.c b/http.c\nindex 8e107ff19b8..d8913b2c641 100644\n--- a/http.c\n+++ b/http.c\n@@ -516,7 +516,8 @@ static int curl_empty_auth_enabled(void)\n \n static void init_curl_http_auth(struct active_request_slot *slot)\n {\n-\tif (!http_auth.username || !*http_auth.username) {\n+\tif (!http_auth.authtype &&\n+\t\t(!http_auth.username || !*http_auth.username)) {\n \t\tif (curl_empty_auth_enabled())\n \t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n@@ -524,8 +525,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n+\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n+\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n+\t\t\thttp_auth.username);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n+\t\t\thttp_auth.password);\n+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n+\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n+\t\t\thttp_auth.password);\n+#endif\n+\t} else {\n+\t\tstruct strbuf auth = STRBUF_INIT;\n+\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n+\t\t\thttp_auth.authtype, http_auth.password);\n+\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n+\t\tstrbuf_release(&auth);\n+\t}\n }\n \n /* *var must be free-able */\n-- \ngitgitgadget\n"},{"id":"462992","messageId":"20843e2051eeab71c5b7555f3e10383484e34b0e.1663097156.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH 6/8] http: store all request headers on active_request_slot","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-09-13T19:25:54Z","receivedAt":"2022-09-13T19:26:25Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nOnce a list of headers has been set on the curl handle, it is not\npossible to recover that `struct curl_slist` instance to add or modify\nheaders.\n\nIn future commits we will want to modify the set of request headers in\nresponse to an authentication challenge/401 response from the server,\nwith information provided by a credential helper.\n\nThere are a number of different places where curl is used for an HTTP\nrequest, and they do not have a common handling of request headers.\nHowever, given that they all do call the `start_active_slot()` function,\neither directly or indirectly via `run_slot()` or `run_one_slot()`, we\nuse this as the point to set the `CURLOPT_HTTPHEADER` option just\nbefore the request is made.\n\nWe collect all request headers in a `struct curl_slist` on the\n`struct active_request_slot` that is obtained from a call to\n`get_active_slot(int)`. This function now takes a single argument to\ndefine if the initial set of headers on the slot should include the\n\"Pragma: no-cache\" header, along with all extra headers specified via\n`http.extraHeader` config values.\n\nThe active request slot obtained from `get_active_slot(int)` will always\ncontain a fresh set of default headers and any headers set in previous\nusages of this slot will be freed.\n\nSigned-off-by: Matthew John Cheetham \n---\n http-push.c | 103 ++++++++++++++++++++++----------------------------\n http-walker.c | 2 +-\n http.c | 82 ++++++++++++++++++----------------------\n http.h | 4 +-\n remote-curl.c | 36 +++++++++---------\n 5 files changed, 101 insertions(+), 126 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 5f4340a36e6..2b40959b376 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,\n \tcurl_easy_setopt(curl, CURLOPT_UPLOAD, 1);\n }\n \n-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)\n+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,\n+\tstruct remote_lock *lock, enum dav_header_flag options)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n \tif (options & DAV_HEADER_IF) {\n \t\tstrbuf_addf(&buf, \"If: (<%s>)\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_LOCK) {\n \t\tstrbuf_addf(&buf, \"Lock-Token: <%s>\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_TIMEOUT) {\n \t\tstrbuf_addf(&buf, \"Timeout: Second-%ld\", lock->timeout);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tstrbuf_release(&buf);\n \n-\treturn dav_headers;\n+\treturn headers;\n }\n \n static void finish_request(struct transfer_request *request);\n@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)\n \n \trequest->url = get_remote_object_url(repo->url, hex, 1);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MKCOL);\n@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)\n \tstrbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);\n \trequest->url = strbuf_detach(&buf, NULL);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http(slot->curl, request->url, DAV_PUT,\n@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)\n static void start_move(struct transfer_request *request)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MOVE);\n-\tdav_headers = curl_slist_append(dav_headers, request->dest);\n-\tdav_headers = curl_slist_append(dav_headers, \"Overwrite: T\");\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n+\tslot->headers = curl_slist_append(slot->headers, request->dest);\n+\tslot->headers = curl_slist_append(slot->headers, \"Overwrite: T\");\n \n \tif (start_active_slot(slot)) {\n \t\trequest->slot = slot;\n@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n \tlock->refreshing = 1;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_LOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)\n \t}\n \n \tlock->refreshing = 0;\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn rc;\n }\n@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tchar *ep;\n \tchar timeout_header[25];\n \tstruct remote_lock *lock = NULL;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tchar *escaped;\n \n@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \twhile (ep) {\n \t\tchar saved_character = ep[1];\n \t\tep[1] = '\\0';\n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n \t\tcurl_setup_http_get(slot->curl, url, DAV_MKCOL);\n \t\tif (start_active_slot(slot)) {\n@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tstrbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);\n \tfree(escaped);\n \n+\tslot = get_active_slot(0);\n+\tslot->results = &results;\n+\n \txsnprintf(timeout_header, sizeof(timeout_header), \"Timeout: Second-%ld\", timeout);\n-\tdav_headers = curl_slist_append(dav_headers, timeout_header);\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n+\tslot->headers = curl_slist_append(slot->headers, timeout_header);\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n \n-\tslot = get_active_slot();\n-\tslot->results = &results;\n \tcurl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tCALLOC_ARRAY(lock, 1);\n@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \t\tfprintf(stderr, \"Unable to start LOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n \n@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct remote_lock *prev = repo->locks;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_LOCK);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)\n \t\tfprintf(stderr, \"Unable to start UNLOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n-\n \tif (repo->locks == lock) {\n \t\trepo->locks = lock->next;\n \t} else {\n@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tstruct remote_ls_ctx ls;\n \n@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,\n \n \tstrbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 1\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 1\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,\n \tfree(url);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n }\n \n static void get_remote_object_list(unsigned char parent)\n@@ -1199,7 +1189,6 @@ static int locking_available(void)\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tint lock_flags = 0;\n \tchar *escaped;\n@@ -1208,14 +1197,14 @@ static int locking_available(void)\n \tstrbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);\n \tfree(escaped);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 0\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 0\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, repo->url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1257,7 +1246,6 @@ static int locking_available(void)\n \n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn lock_flags;\n }\n@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers;\n-\n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n \tstrbuf_addf(&out_buffer.buf, \"%s\\n\", oid_to_hex(oid));\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF);\n+\n \tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t&out_buffer, fwrite_null);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)\n \tstruct buffer buffer = { STRBUF_INIT, 0 };\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \n \tremote_ls(\"refs/\", (PROCESS_FILES | RECURSIVE),\n \t\t add_remote_info_ref, &buffer.buf);\n \tif (!aborted) {\n-\t\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n+\t\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\t\tDAV_HEADER_IF);\n+\n \t\tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t\t&buffer, fwrite_null);\n-\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \t\tif (start_active_slot(slot)) {\n \t\t\trun_active_slot(slot);\n@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)\n \tif (dry_run)\n \t\treturn 0;\n \turl = xstrfmt(\"%s%s\", repo->url, remote_ref->name);\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n \tcurl_setup_http_get(slot->curl, url, DAV_DELETE);\n \tif (start_active_slot(slot)) {\ndiff --git a/http-walker.c b/http-walker.c\nindex b8f0f98ae14..8747de2fcdb 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)\n \t * Use a callback to process the result, since another request\n \t * may fail and need to have alternates loaded before continuing\n \t */\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_alternates_response;\n \talt_req.walker = walker;\n \tslot->callback_data = &alt_req;\ndiff --git a/http.c b/http.c\nindex 091321af98e..42616f746b1 100644\n--- a/http.c\n+++ b/http.c\n@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =\n \t| CURLAUTH_DIGEST_IE\n \t| CURLAUTH_DIGEST;\n \n-static struct curl_slist *pragma_header;\n-static struct curl_slist *no_pragma_header;\n static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;\n \n static struct curl_slist *host_resolutions;\n@@ -1132,11 +1130,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n-\tpragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma: no-cache\");\n-\tno_pragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma:\");\n-\n \t{\n \t\tchar *http_max_requests = getenv(\"GIT_HTTP_MAX_REQUESTS\");\n \t\tif (http_max_requests)\n@@ -1198,6 +1191,8 @@ void http_cleanup(void)\n \n \twhile (slot != NULL) {\n \t\tstruct active_request_slot *next = slot->next;\n+\t\tif (slot->headers)\n+\t\t\tcurl_slist_free_all(slot->headers);\n \t\tif (slot->curl) {\n \t\t\txmulti_remove_handle(slot);\n \t\t\tcurl_easy_cleanup(slot->curl);\n@@ -1214,12 +1209,6 @@ void http_cleanup(void)\n \n \tstring_list_clear(&extra_http_headers, 0);\n \n-\tcurl_slist_free_all(pragma_header);\n-\tpragma_header = NULL;\n-\n-\tcurl_slist_free_all(no_pragma_header);\n-\tno_pragma_header = NULL;\n-\n \tcurl_slist_free_all(host_resolutions);\n \thost_resolutions = NULL;\n \n@@ -1254,7 +1243,18 @@ void http_cleanup(void)\n \tFREE_AND_NULL(cached_accept_language);\n }\n \n-struct active_request_slot *get_active_slot(void)\n+static struct curl_slist *http_copy_default_headers(void)\n+{\n+\tstruct curl_slist *headers = NULL;\n+\tconst struct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, &extra_http_headers)\n+\t\theaders = curl_slist_append(headers, item->string);\n+\n+\treturn headers;\n+}\n+\n+struct active_request_slot *get_active_slot(int no_pragma_header)\n {\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n@@ -1276,6 +1276,7 @@ struct active_request_slot *get_active_slot(void)\n \t\tnewslot->curl = NULL;\n \t\tnewslot->in_use = 0;\n \t\tnewslot->next = NULL;\n+\t\tnewslot->headers = NULL;\n \n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n@@ -1293,6 +1294,15 @@ struct active_request_slot *get_active_slot(void)\n \t\tcurl_session_count++;\n \t}\n \n+\tif (slot->headers)\n+\t\tcurl_slist_free_all(slot->headers);\n+\n+\tslot->headers = http_copy_default_headers();\n+\n+\tif (!no_pragma_header)\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Pragma: no-cache\");\n+\n \tactive_requests++;\n \tslot->in_use = 1;\n \tslot->results = NULL;\n@@ -1302,7 +1312,6 @@ struct active_request_slot *get_active_slot(void)\n \tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);\n \tif (curl_save_cookies)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);\n \tcurl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n \tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);\n@@ -1334,9 +1343,12 @@ struct active_request_slot *get_active_slot(void)\n \n int start_active_slot(struct active_request_slot *slot)\n {\n-\tCURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\tCURLMcode curlm_result;\n \tint num_transfers;\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);\n+\tcurlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\n \tif (curlm_result != CURLM_OK &&\n \t curlm_result != CURLM_CALL_MULTI_PERFORM) {\n \t\twarning(\"curl_multi_add_handle failed: %s\",\n@@ -1651,17 +1663,6 @@ int run_one_slot(struct active_request_slot *slot,\n \treturn handle_curl_result(results);\n }\n \n-struct curl_slist *http_copy_default_headers(void)\n-{\n-\tstruct curl_slist *headers = NULL;\n-\tconst struct string_list_item *item;\n-\n-\tfor_each_string_list_item(item, &extra_http_headers)\n-\t\theaders = curl_slist_append(headers, item->string);\n-\n-\treturn headers;\n-}\n-\n static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)\n {\n \tchar *ptr;\n@@ -1879,12 +1880,11 @@ static int http_request(const char *url,\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tint no_cache = options && options->no_cache;\n \tconst char *accept_language;\n \tint ret;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(!no_cache);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);\n \n \tif (!result) {\n@@ -1909,27 +1909,23 @@ static int http_request(const char *url,\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-\t\theaders = curl_slist_append(headers, accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\taccept_language);\n \n-\tstrbuf_addstr(&buf, \"Pragma:\");\n-\tif (options && options->no_cache)\n-\t\tstrbuf_addstr(&buf, \" no-cache\");\n \tif (options && options->initial_request &&\n \t http_follow_config == HTTP_FOLLOW_INITIAL)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);\n \n-\theaders = curl_slist_append(headers, buf.buf);\n-\n \t/* Add additional headers here */\n \tif (options && options->extra_headers) {\n \t\tconst struct string_list_item *item;\n \t\tfor_each_string_list_item(item, options->extra_headers) {\n-\t\t\theaders = curl_slist_append(headers, item->string);\n+\t\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\titem->string);\n \t\t}\n \t}\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);\n \n@@ -1947,9 +1943,6 @@ static int http_request(const char *url,\n \t\tcurlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,\n \t\t\t\toptions->effective_url);\n \n-\tcurl_slist_free_all(headers);\n-\tstrbuf_release(&buf);\n-\n \treturn ret;\n }\n \n@@ -2310,12 +2303,10 @@ struct http_pack_request *new_direct_http_pack_request(\n \t\tgoto abort;\n \t}\n \n-\tpreq->slot = get_active_slot();\n+\tpreq->slot = get_active_slot(1);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);\n-\tcurl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,\n-\t\tno_pragma_header);\n \n \t/*\n \t * If there is data present from a previous transfer attempt,\n@@ -2480,14 +2471,13 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \t\t}\n \t}\n \n-\tfreq->slot = get_active_slot();\n+\tfreq->slot = get_active_slot(1);\n \n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \n \t/*\n \t * If we have successfully processed data from a previous fetch\ndiff --git a/http.h b/http.h\nindex 3c94c479100..a304cc408b2 100644\n--- a/http.h\n+++ b/http.h\n@@ -22,6 +22,7 @@ struct slot_results {\n struct active_request_slot {\n \tCURL *curl;\n \tint in_use;\n+\tstruct curl_slist *headers;\n \tCURLcode curl_result;\n \tlong http_code;\n \tint *finished;\n@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);\n curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);\n \n /* Slot lifecycle functions */\n-struct active_request_slot *get_active_slot(void);\n+struct active_request_slot *get_active_slot(int no_pragma_header);\n int start_active_slot(struct active_request_slot *slot);\n void run_active_slot(struct active_request_slot *slot);\n void finish_all_active_slots(void);\n@@ -64,7 +65,6 @@ void step_active_slots(void);\n void http_init(struct remote *remote, const char *url,\n \t int proactive_auth);\n void http_cleanup(void);\n-struct curl_slist *http_copy_default_headers(void);\n \n extern long int git_curl_ipresolve;\n extern int active_requests;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 72dfb8fb86a..edbd4504beb 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,\n static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tstruct strbuf buf = STRBUF_INIT;\n \tint err;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, \"0000\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);\n \n \terr = run_slot(slot, results);\n \n-\tcurl_slist_free_all(headers);\n \tstrbuf_release(&buf);\n \treturn err;\n }\n@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)\n static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tint use_gzip = rpc->gzip_request;\n \tchar *gzip_body = NULL;\n \tsize_t gzip_size = 0;\n@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece\n \t\t\tneeds_100_continue = 1;\n \t}\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n-\theaders = curl_slist_append(headers, needs_100_continue ?\n+retry:\n+\tslot = get_active_slot(0);\n+\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, needs_100_continue ?\n \t\t\"Expect: 100-continue\" : \"Expect:\");\n \n \t/* Add Accept-Language header */\n \tif (rpc->hdr_accept_language)\n-\t\theaders = curl_slist_append(headers, rpc->hdr_accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->hdr_accept_language);\n \n \t/* Add the extra Git-Protocol header */\n \tif (rpc->protocol_header)\n-\t\theaders = curl_slist_append(headers, rpc->protocol_header);\n-\n-retry:\n-\tslot = get_active_slot();\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->protocol_header);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -955,7 +953,8 @@ retry:\n \t\t/* The request body is large and the size cannot be predicted.\n \t\t * We must use chunked encoding to send it.\n \t\t */\n-\t\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Transfer-Encoding: chunked\");\n \t\trpc->initial_buffer = 1;\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);\n@@ -1002,7 +1001,8 @@ retry:\n \n \t\tgzip_size = stream.total_out;\n \n-\t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Content-Encoding: gzip\");\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));\n \n@@ -1025,7 +1025,6 @@ retry:\n \t\t}\n \t}\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n \trpc_in_data.rpc = rpc;\n \trpc_in_data.slot = slot;\n@@ -1055,7 +1054,6 @@ retry:\n \tif (stateless_connect)\n \t\tpacket_response_end(rpc->in);\n \n-\tcurl_slist_free_all(headers);\n \tfree(gzip_body);\n \treturn err;\n }\n-- \ngitgitgadget\n\n"},{"id":"463185","messageId":"452acd3b-14bd-a3c8-df58-39b5e9edfd0b@github.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:08:21Z","receivedAt":"2022-09-19T16:08:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> Hello! I have an RFC to update the existing credential helper design in\n> order to allow for some new scenarios, and future evolution of auth methods\n> that Git hosts may wish to provide. I outline the background, summary of\n> changes and some challenges below. I also attach a series of patches to\n> illustrate the design proposal.\n\nIt's unfortunate that we didn't get to talk about this during the\ncontributor summit, but it is super-technical and worth looking closely\nat all the details. \n\n> One missing element from the patches are extensive tests of the new\n> behaviour. It appears existing tests focus either on the credential helper\n> protocol/format, or rely on testing basic authentication only via an Apache\n> webserver. In order to have a full end to end test coverage of these new\n> features it make be that we need a more comprehensive test bed to mock these\n> more nuanced authentication methods. I lean on the experts on the list for\n> advice here.\n\nThe microsoft/git fork has a feature (the GVFS Protocol) that requires a\ncustom HTTP server as a test helper. We might need a similar test helper\nto return these WWW-Authenticate headers and check the full request list\nfrom Git matches the spec. Doing that while also executing the proper Git\ncommands to serve the HTTP bodies is hopefully not too large. It might be\nnice to adapt such a helper to replace the need for a full Apache install\nin our test suite, but that's an independent concern from this RFC.\n\n> Limitations\n> ===========\n> \n> Because this credential model was built mostly for password based\n> authentication systems, it's somewhat limited. In particular:\n> \n> 1. To generate valid credentials, additional information about the request\n> (or indeed the requestee and their device) may be required. For example,\n> OAuth is based around scopes. A scope, like \"git.read\", might be\n> required to read data from the remote. However, the remote cannot tell\n> the credential helper what scope is required for this request.\n> \n> 2. This system is not fully extensible. Each time a new type of\n> authentication (like OAuth Bearer) is invented, Git needs updates before\n> credential helpers can take advantage of it (or leverage a new\n> capability in libcurl).\n> \n> \n> Goals\n> =====\n> \n> * As a user with multiple federated cloud identities:\n\nI'm not sure if you mentioned it anywhere else, but this is specifically\nfor cases where a user might have multiple identities _on the same host\nby DNS name_. The credential.useHttpPath config option might seem like it\ncould help here, but the credential helper might pick the wrong identity\nthat is the most-recent login. Either this workflow will require the user\nto re-login with every new URL or the fetches/clones will fail when the\nguess is wrong and the user would need to learn how to log into that other\nidentity.\n\nPlease correct me if I'm wrong about any of this, but the details of your\ngoals make it clear that the workflow will be greatly improved:\n\n> * Reach out to a remote and have my credential helper automatically\n> prompt me for the correct identity.\n> * Leverage existing authentication systems built-in to many operating\n> systems and devices to boost security and reduce reliance on passwords.\n> \n> * As a Git host and/or cloud identity provider:\n> \n> * Leverage newest identity standards, enhancements, and threat\n> mitigations - all without updating Git.\n> * Enforce security policies (like requiring two-factor authentication)\n> dynamically.\n> * Allow integration with third party standard based identity providers in\n> enterprises allowing customers to have a single plane of control for\n> critical identities with access to source code.\n\nI had a question with this part of your proposal:\n\n> Because the extra information forms an ordered list, and the existing\n> credential helper I/O format only provides for simple key=value pairs,\n> we introduce a new convention for transmitting an ordered list of\n> values. Key names that are suffixed with a C-style array syntax should\n> have values considered to form an order list, i.e. key[n]=value, where n\n> is a zero based index of the values.\n> \n> For the WWW-Authenticate header values we opt to use the key wwwauth[n].\n...\n> Git sends over standard input:\n> \n> protocol=https\n> host=example.com\n> wwwauth[0]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\n> wwwauth[1]=Basic realm=\"login.example\"\n\nThe important part here is that we provide a way to specify a multi-valued\nkey as opposed to a \"last one wins\" key, right?\n\nUsing empty braces (wwwauth[]) would suffice to indicate this, right? That\nallows us to not care about the values inside the braces. The biggest\nissues I see with a value in the braces are:\n\n1. What if it isn't an integer?\n2. What if we are missing a value?\n3. What if they come out of order?\n\nWithout a value inside, then the order in which they appear provides\nimplicit indices in their multi-valued list.\n\nOther than that, I support this idea and will start looking at the code\nnow.\n\nThanks,\n-Stolee\n"},{"id":"463186","messageId":"ead6293c-8906-04c9-6566-e47efb624385@github.com","threadId":"58425","inReplyTo":"2ece562a5952b5752de5040b17ec36076164c72f.1663097156.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] osxkeychain: clarify that we ignore unknown lines","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:12:20Z","receivedAt":"2022-09-19T16:12:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Like in all the other credential helpers, the osxkeychain helper\n> ignores unknown credential lines.\n> \n> Add a comment (a la the other helpers) to make it clear and explicit\n> that this is the desired behaviour.\n\nI recommend that these first three patches be submitted for full\nreview and merging, since they seem important independent of this\nRFC.\n\nThanks,\n-Stolee\n"},{"id":"463187","messageId":"9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com","threadId":"58425","inReplyTo":"78e66d56605cfb1c7000edf329ac16c05a5d69b0.1663097156.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:21:10Z","receivedAt":"2022-09-19T16:21:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n\n> +\t/**\n> +\t * A `strvec` of WWW-Authenticate header values. Each string\n> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n> +\t * in the order they were received in the response.\n> +\t */\n> +\tstruct strvec wwwauth_headers;\n\nI like this careful documentation.\n\n> +\tunsigned header_is_last_match:1;\n\nBut then this member is unclear how it is attached. It could use its\nown \"for internal use\" comment if we don't want to describe it in full\ndetail here.\n\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tconst char *val;\n> +\tconst char *z = NULL;\n> +\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\tstrbuf_add(&buf, ptr, size);\n> +\n> +\t/* Strip the CRLF that should be present at the end of each field */\n\nIs it really a CRLF? Or just an LF?\n\n> +\tstrbuf_trim_trailing_newline(&buf);\n\nThankfully, this will trim an LF _or_ CR/LF pair, so either way would be fine.\n\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n> +\t\twhile (isspace(*val)) val++;\n\nBreak the \"val++;\" to its own line:\n\n\t\twhile (isspace(*val))\n\t\t\tval++;\n\nWhile we are here, do we need to be careful about the end of the string at\nthis point? Is it possible that the server will send all spaces up until the\nmaximum header size (as mentioned in the message)?\n\n> +\n> +\t\tstrvec_push(values, val);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t */\n> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n> +\t\tconst char **v = values->v + values->nr - 1;\n\nI suppose we expect leading spaces as critical to this header, right?\n\n> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n\nWe might have better luck using a strbuf, initializing it with the expected\nsize and using strbuf_add() to append the strings. Maybe I'm just prematurely\noptimizing, though.\n\n> +\n> +\t\tfree((void*)*v);\n> +\t\t*v = append;\n> +\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/* This is the start of a new header we don't care about */\n> +\thttp_auth.header_is_last_match = 0;\n> +\n> +\t/*\n> +\t * If this is a HTTP status line and not a header field, this signals\n> +\t * a different HTTP response. libcurl writes all the output of all\n> +\t * response headers of all responses, including redirects.\n> +\t * We only care about the last HTTP request response's headers so clear\n> +\t * the existing array.\n> +\t */\n> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n> +\t\tstrvec_clear(values);\n> +\n> +exit:\n> +\tstrbuf_release(&buf);\n> +\treturn size;\n> +}\n> +\n> size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n> {\n> \treturn nmemb;\n> @@ -1829,6 +1904,8 @@ static int http_request(const char *url,\n> \t\t\t\t\t fwrite_buffer);\n> \t}\n> \n> +\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n\nNice integration point!\n\nThanks,\n-Stolee\n"},{"id":"463191","messageId":"69cbe403-48a2-ac5c-5743-5b7cae5ae523@github.com","threadId":"58425","inReplyTo":"936545004b8b46cbe24d8069cfd95ae5b5f98593.1663097156.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/8] credential: add WWW-Authenticate header to cred requests","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:33:33Z","receivedAt":"2022-09-19T16:33:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n\n> In this case we send multiple `wwwauth[n]` properties where `n` is a\n> zero-indexed number, reflecting the order the WWW-Authenticate headers\n> appeared in the HTTP response.\n> @@ -151,6 +151,15 @@ Git understands the following attributes:\n> \twere read (e.g., `url=https://example.com` would behave as if\n> \t`protocol=https` and `host=example.com` had been provided). This\n> \tcan help callers avoid parsing URLs themselves.\n> +\n> +`wwwauth[n]`::\n> +\n> +\tWhen an HTTP response is received that includes one or more\n> +\t'WWW-Authenticate' authentication headers, these can be passed to Git\n> +\t(and subsequent credential helpers) with these attributes.\n> +\tEach 'WWW-Authenticate' header value should be passed as a separate\n> +\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n> +\tappear in the HTTP response.\n> +\n> Note that specifying a protocol is mandatory and if the URL\n> doesn't specify a hostname (e.g., \"cert:///path/to/file\") the\n\nThis \"+\" means that this paragraph should be connected to the previous\none, so it seems that you've inserted your new value in the middle of\nthe `url` key. You'll want to move yours to be after those two connected\nparagraphs. Your diff hunk should look like this:\n\n--- >8 ---\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex f18673017f..127ae29be3 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -160,6 +160,15 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[n]`::\n+\n+\tWhen an HTTP response is received that includes one or more\n+\t'WWW-Authenticate' authentication headers, these can be passed to Git\n+\t(and subsequent credential helpers) with these attributes.\n+\tEach 'WWW-Authenticate' header value should be passed as a separate\n+\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n+\tappear in the HTTP response.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n\n\n--- >8 ---\n\n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 497b9b9d927..fe118d76f98 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -235,6 +235,19 @@ SSLEngine On\n> \tRequire valid-user\n> \n> \n> +# Advertise two additional auth methods above \"Basic\".\n> +# Neither of them actually work but serve test cases showing these\n> +# additional auth headers are consumed correctly.\n> +\n> +\tAuthType Basic\n> +\tAuthName \"git-auth\"\n> +\tAuthUserFile passwd\n> +\tRequire valid-user\n> +\tSetEnvIf Authorization \"^\\S+\" authz\n> +\tHeader always add WWW-Authenticate \"Bearer authority=https://login.example.com\" env=!authz\n> +\tHeader always add WWW-Authenticate \"FooAuth foo=bar baz=1\" env=!authz\n> +\n> +\n\nThis is cool that you've figured out how to make our Apache tests\nadd these headers! Maybe we won't need that extra test helper like\nI thought (unless we want to confirm the second request sends the\nright information).\n\n> +test_expect_success 'http auth sends www-auth headers to credential helper' '\n> +\twrite_script git-credential-tee <<-\\EOF &&\n> +\t\tcmd=$1\n> +\t\tteefile=credential-$cmd\n> +\t\tif [ -f \"$teefile\" ]; then\n\nI think we prefer using \"test\" over the braces (and linebreak\nbefore then) like this:\n\n\t\tif test -n \"$teefile\"\n\t\tthen\n\n> +\t\t\trm $teefile\n> +\t\tfi\n\nAlternatively, you could always run \"rm -f $teefile\" for\nsimplicity.\n\n> +\t\t(\n> +\t\t\twhile read line;\n> +\t\t\tdo\n> +\t\t\t\tif [ -z \"$line\" ]; then\n> +\t\t\t\t\texit 0\n> +\t\t\t\tfi\n> +\t\t\t\techo \"$line\" >> $teefile\n> +\t\t\t\techo $line\n> +\t\t\tdone\n> +\t\t) | git credential-store $cmd\n\nSince I'm not sure, I'll ask the question: do we need the sub-shell\nhere, or could we pipe directly off of the \"done\"? Like this:\n\n\t\twhile read line;\n\t\tdo\n\t\t\tif [ -z \"$line\" ]; then\n\t\t\t\texit 0\n\t\t\tfi\n\t\t\techo \"$line\" >> $teefile\n\t\t\techo $line\n\t\tdone | git credential-store $cmd\n\n> +\tEOF\n\n\n> +\tcat >expected-get <<-EOF &&\n> +\tprotocol=http\n> +\thost=127.0.0.1:5551\n> +\twwwauth[0]=Bearer authority=https://login.example.com\n> +\twwwauth[1]=FooAuth foo=bar baz=1\n> +\twwwauth[2]=Basic realm=\"git-auth\"\n> +\tEOF\n> +\n> +\tcat >expected-store <<-EOF &&\n> +\tprotocol=http\n> +\thost=127.0.0.1:5551\n> +\tusername=user@host\n> +\tpassword=pass@host\n> +\tEOF\n> +\n> +\trm -f .git-credentials &&\n> +\ttest_config credential.helper tee &&\n> +\tset_askpass user@host pass@host &&\n> +\t(\n> +\t\tPATH=\"$PWD:$PATH\" &&\n> +\t\tgit ls-remote \"$HTTPD_URL/auth-wwwauth/smart/repo.git\"\n> +\t) &&\n> +\texpect_askpass both user@host &&\n> +\ttest_cmp expected-get credential-get &&\n> +\ttest_cmp expected-store credential-store\n\nElegant check for both calls.\n\nThanks,\n-Stolee\n"},{"id":"463195","messageId":"0f792d92-62a9-7a4e-787f-7aa39262149d@github.com","threadId":"58425","inReplyTo":"7f827067f55d596284eb2ad764e59d402c75be18.1663097156.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 8/8] http: set specific auth scheme depending on credential","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:42:12Z","receivedAt":"2022-09-19T16:42:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Introduce a new credential field `authtype` that can be used by\n> credential helpers to indicate the type of the credential or\n> authentication mechanism to use for a request.\n> \n> Modify http.c to now specify the correct authentication scheme or\n> credential type when authenticating the curl handle. If the new\n> `authtype` field in the credential structure is `NULL` or \"Basic\" then\n> use the existing username/password options. If the field is \"Bearer\"\n> then use the OAuth bearer token curl option. Otherwise, the `authtype`\n> field is the authentication scheme and the `password` field is the\n> raw, unencoded value.\n\n\n> @@ -524,8 +525,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n> \n> \tcredential_fill(&http_auth);\n> \n> -\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n> -\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n> +\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n> +\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n> +\t\t\thttp_auth.username);\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n> +\t\t\thttp_auth.password);\n> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n> +\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n> +\t\t\thttp_auth.password);\n> +#endif\n> +\t} else {\n> +\t\tstruct strbuf auth = STRBUF_INIT;\n> +\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n> +\t\t\thttp_auth.authtype, http_auth.password);\n> +\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n> +\t\tstrbuf_release(&auth);\n> +\t}\n> }\n\nIt would be good to have a test here, and the only way I can think\nto add it would be to modify one of the test credential helpers to\nindicate that OAuth is being used.\n\nThe test would somehow need to be careful about the curl version,\nthough, and I'm not sure if we have prior work for writing prereqs\nbased on the linked curl version.\n\nThanks,\n-Stolee\n"},{"id":"463197","messageId":"710c8868-800a-7140-0b47-74784b1e2b1d@github.com","threadId":"58425","inReplyTo":"452acd3b-14bd-a3c8-df58-39b5e9edfd0b@github.com","subject":"Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-19T16:44:57Z","receivedAt":"2022-09-19T16:45:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/19/2022 12:08 PM, Derrick Stolee wrote:\n> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n\n>> protocol=https\n>> host=example.com\n>> wwwauth[0]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\n>> wwwauth[1]=Basic realm=\"login.example\"\n> \n> The important part here is that we provide a way to specify a multi-valued\n> key as opposed to a \"last one wins\" key, right?\n> \n> Using empty braces (wwwauth[]) would suffice to indicate this, right? That\n> allows us to not care about the values inside the braces. The biggest\n> issues I see with a value in the braces are:\n> \n> 1. What if it isn't an integer?\n> 2. What if we are missing a value?\n> 3. What if they come out of order?\n> \n> Without a value inside, then the order in which they appear provides\n> implicit indices in their multi-valued list.\n\nAfter looking at the code, it would not be difficult at all to make this\nchange in-place for these patches. But I won't push too hard if there is\nsome reason to keep the index values.\n \n> Other than that, I support this idea and will start looking at the code\n> now.\n\nI took a look and provided feedback as I could. Patches 6 and 7 eluded\nme only because I'm so unfamiliar with the http.c code and don't have\ntime to learn it today.\n\nI mentioned that patches 1-3 could easily be picked up as a topic while\nthe rest of the series is considered carefully.\n\nI tried to add some mentions of testing, but you've already tested more\nthan I expected, by adding the headers to the Apache output.\n\nThanks,\n-Stolee\n\n"},{"id":"463241","messageId":"04a08e67-9043-ad15-c9aa-4ecf4ddaf92a@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","fromName":"Lessley Dennington","fromEmail":"lessleydennington@gmail.com","sentAt":"2022-09-19T23:36:26Z","receivedAt":"2022-09-19T23:36:34Z","isPatch":true,"sender":{"key":"lessleydennington@gmail.com","avatar":"https://avatars.githubusercontent.com/u/11321782?v=4"},"body":"This is a really exciting idea! Based on your patches, it seems to be a\ngreat opportunity to add extensibility and flexibility to the credential\nhelper model without huge disruptions to the codebase. Well done!\n\nOn 9/13/22 12:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> 3. Teach Git to specify authentication schemes other than Basic in\n> subsequent HTTP requests based on credential helper responses.\n> \nThis!! Yes!!\n> \n> ...\n> wwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n> \nI think sending the fields individually (as you describe in this doc and\nimplement in your patches) is the right call. In my opinion, it's more\nlegible, consistent with the remote response, and aligns with your goal of\nminimizing authentication-related actions in Git.\n\nBest,\n\nLessley\n"},{"id":"463401","messageId":"AS8PR03MB8689B42AC5CAD6D4002C3C2EC04F9@AS8PR03MB8689.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"452acd3b-14bd-a3c8-df58-39b5e9edfd0b@github.com","subject":"Re: [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-09-21T22:19:37Z","receivedAt":"2022-09-21T22:19:59Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-09-19 09:08, Derrick Stolee wrote:\n> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n>> Hello! I have an RFC to update the existing credential helper design in\n>> order to allow for some new scenarios, and future evolution of auth methods\n>> that Git hosts may wish to provide. I outline the background, summary of\n>> changes and some challenges below. I also attach a series of patches to\n>> illustrate the design proposal.\n> \n> It's unfortunate that we didn't get to talk about this during the\n> contributor summit, but it is super-technical and worth looking closely\n> at all the details. \n> \n>> One missing element from the patches are extensive tests of the new\n>> behaviour. It appears existing tests focus either on the credential helper\n>> protocol/format, or rely on testing basic authentication only via an Apache\n>> webserver. In order to have a full end to end test coverage of these new\n>> features it make be that we need a more comprehensive test bed to mock these\n>> more nuanced authentication methods. I lean on the experts on the list for\n>> advice here.\n> \n> The microsoft/git fork has a feature (the GVFS Protocol) that requires a\n> custom HTTP server as a test helper. We might need a similar test helper\n> to return these WWW-Authenticate headers and check the full request list\n> from Git matches the spec. Doing that while also executing the proper Git\n> commands to serve the HTTP bodies is hopefully not too large. It might be\n> nice to adapt such a helper to replace the need for a full Apache install\n> in our test suite, but that's an independent concern from this RFC.\n\nThat's a good reference and possible solution to the testing question, and\ndefinitely something I can look at adding. I just wanted another pair of\neyes and thoughts on any other options that I may have been missing in the\nexisting testing repertoire, before embarking on writing such a test helper.\n\n>> Limitations\n>> ===========\n>>\n>> Because this credential model was built mostly for password based\n>> authentication systems, it's somewhat limited. In particular:\n>>\n>> 1. To generate valid credentials, additional information about the request\n>> (or indeed the requestee and their device) may be required. For example,\n>> OAuth is based around scopes. A scope, like \"git.read\", might be\n>> required to read data from the remote. However, the remote cannot tell\n>> the credential helper what scope is required for this request.\n>>\n>> 2. This system is not fully extensible. Each time a new type of\n>> authentication (like OAuth Bearer) is invented, Git needs updates before\n>> credential helpers can take advantage of it (or leverage a new\n>> capability in libcurl).\n>>\n>>\n>> Goals\n>> =====\n>>\n>> * As a user with multiple federated cloud identities:\n> \n> I'm not sure if you mentioned it anywhere else, but this is specifically\n> for cases where a user might have multiple identities _on the same host\n> by DNS name_. The credential.useHttpPath config option might seem like it\n> could help here, but the credential helper might pick the wrong identity\n> that is the most-recent login. Either this workflow will require the user\n> to re-login with every new URL or the fetches/clones will fail when the\n> guess is wrong and the user would need to learn how to log into that other\n> identity.\n> \n> Please correct me if I'm wrong about any of this, but the details of your\n> goals make it clear that the workflow will be greatly improved:\n\nSuch a scenario where multiple identities may be available for the same DNS\nhostname would indeed be improved (with an appropriately enlightened\ncredential helper of course). As you mentioned, credential.useHttpPath can\nalso be used to workaround such a situation, but that just creates another\nproblem in that users need to provide the same set of credentials for each\nrepository with a full remote URL path that use the same identity.\n\nBy providing information about the auth challenge (including parameters\nlike authority or realm if present) would allow credential helpers select\nor filter known identities and credentials automatically, avoiding user\ninput.\n\n>> * Reach out to a remote and have my credential helper automatically\n>> prompt me for the correct identity.\n>> * Leverage existing authentication systems built-in to many operating\n>> systems and devices to boost security and reduce reliance on passwords.\n>>\n>> * As a Git host and/or cloud identity provider:\n>> \n>> * Leverage newest identity standards, enhancements, and threat\n>> mitigations - all without updating Git.\n>> * Enforce security policies (like requiring two-factor authentication)\n>> dynamically.\n>> * Allow integration with third party standard based identity providers in\n>> enterprises allowing customers to have a single plane of control for\n>> critical identities with access to source code.\n> \n> I had a question with this part of your proposal:\n> \n>> Because the extra information forms an ordered list, and the existing\n>> credential helper I/O format only provides for simple key=value pairs,\n>> we introduce a new convention for transmitting an ordered list of\n>> values. Key names that are suffixed with a C-style array syntax should\n>> have values considered to form an order list, i.e. key[n]=value, where n\n>> is a zero based index of the values.\n>> \n>> For the WWW-Authenticate header values we opt to use the key wwwauth[n].\n> ...\n>> Git sends over standard input:\n>>\n>> protocol=https\n>> host=example.com\n>> wwwauth[0]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\n>> wwwauth[1]=Basic realm=\"login.example\"\n> \n> The important part here is that we provide a way to specify a multi-valued\n> key as opposed to a \"last one wins\" key, right?\n> \n> Using empty braces (wwwauth[]) would suffice to indicate this, right? That\n> allows us to not care about the values inside the braces. The biggest\n> issues I see with a value in the braces are:\n> \n> 1. What if it isn't an integer?\n> 2. What if we are missing a value?\n> 3. What if they come out of order?\n> \n> Without a value inside, then the order in which they appear provides\n> implicit indices in their multi-valued list.\n> \n> Other than that, I support this idea and will start looking at the code\n> now.\n\nThere are two important things this extension to the I/O format provides:\n1) multi-valued keys, and 2) ordering to the multiple values.\n\nYou are correct that dropping the integer index still means we still meet\nrequirement 1, and implicitly meet requirement 2. In this proposal I was\njust being explicit in the ordering - it's not something I'm overly\nattached to however, and may indeed make parsing or identifiying these\nmulti-valued keys easier on the credential helper side of things.\n\n> Thanks,\n> -Stolee\n"},{"id":"463402","messageId":"AS8PR03MB86899B51BD151490CC16283FC04F9@AS8PR03MB8689.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"69cbe403-48a2-ac5c-5743-5b7cae5ae523@github.com","subject":"Re: [PATCH 5/8] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-09-21T22:20:51Z","receivedAt":"2022-09-21T22:21:07Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-09-19 09:33, Derrick Stolee wrote:\n> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n> \n>> In this case we send multiple `wwwauth[n]` properties where `n` is a\n>> zero-indexed number, reflecting the order the WWW-Authenticate headers\n>> appeared in the HTTP response.\n>> @@ -151,6 +151,15 @@ Git understands the following attributes:\n>> \twere read (e.g., `url=https://example.com` would behave as if\n>> \t`protocol=https` and `host=example.com` had been provided). This\n>> \tcan help callers avoid parsing URLs themselves.\n>> +\n>> +`wwwauth[n]`::\n>> +\n>> +\tWhen an HTTP response is received that includes one or more\n>> +\t'WWW-Authenticate' authentication headers, these can be passed to Git\n>> +\t(and subsequent credential helpers) with these attributes.\n>> +\tEach 'WWW-Authenticate' header value should be passed as a separate\n>> +\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n>> +\tappear in the HTTP response.\n>> +\n>> Note that specifying a protocol is mandatory and if the URL\n>> doesn't specify a hostname (e.g., \"cert:///path/to/file\") the\n> \n> This \"+\" means that this paragraph should be connected to the previous\n> one, so it seems that you've inserted your new value in the middle of\n> the `url` key. You'll want to move yours to be after those two connected\n> paragraphs. Your diff hunk should look like this:\n> \n> --- >8 ---\n> \n> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\n> index f18673017f..127ae29be3 100644\n> --- a/Documentation/git-credential.txt\n> +++ b/Documentation/git-credential.txt\n> @@ -160,6 +160,15 @@ empty string.\n> Components which are missing from the URL (e.g., there is no\n> username in the example above) will be left unset.\n> \n> +`wwwauth[n]`::\n> +\n> +\tWhen an HTTP response is received that includes one or more\n> +\t'WWW-Authenticate' authentication headers, these can be passed to Git\n> +\t(and subsequent credential helpers) with these attributes.\n> +\tEach 'WWW-Authenticate' header value should be passed as a separate\n> +\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n> +\tappear in the HTTP response.\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> \n> \n> --- >8 ---\n\nThanks for catching!\n\n>> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n>> index 497b9b9d927..fe118d76f98 100644\n>> --- a/t/lib-httpd/apache.conf\n>> +++ b/t/lib-httpd/apache.conf\n>> @@ -235,6 +235,19 @@ SSLEngine On\n>> \tRequire valid-user\n>> \n>> \n>> +# Advertise two additional auth methods above \"Basic\".\n>> +# Neither of them actually work but serve test cases showing these\n>> +# additional auth headers are consumed correctly.\n>> +\n>> +\tAuthType Basic\n>> +\tAuthName \"git-auth\"\n>> +\tAuthUserFile passwd\n>> +\tRequire valid-user\n>> +\tSetEnvIf Authorization \"^\\S+\" authz\n>> +\tHeader always add WWW-Authenticate \"Bearer authority=https://login.example.com\" env=!authz\n>> +\tHeader always add WWW-Authenticate \"FooAuth foo=bar baz=1\" env=!authz\n>> +\n>> +\n> \n> This is cool that you've figured out how to make our Apache tests\n> add these headers! Maybe we won't need that extra test helper like\n> I thought (unless we want to confirm the second request sends the\n> right information).\n\nThis will exercise the new header parsing and passing the info to the helper\nbut will indeed not test the response. I feel like a test helper would be\nbeneficial still.. what I've done here doesn't feel 100% clean or complete.\n\n>> +test_expect_success 'http auth sends www-auth headers to credential helper' '\n>> +\twrite_script git-credential-tee <<-\\EOF &&\n>> +\t\tcmd=$1\n>> +\t\tteefile=credential-$cmd\n>> +\t\tif [ -f \"$teefile\" ]; then\n> \n> I think we prefer using \"test\" over the braces (and linebreak\n> before then) like this:\n> \n> \t\tif test -n \"$teefile\"\n> \t\tthen\n> \n>> +\t\t\trm $teefile\n>> +\t\tfi\n> \n> Alternatively, you could always run \"rm -f $teefile\" for\n> simplicity.\nI like simple :-)\n\n>> +\t\t(\n>> +\t\t\twhile read line;\n>> +\t\t\tdo\n>> +\t\t\t\tif [ -z \"$line\" ]; then\n>> +\t\t\t\t\texit 0\n>> +\t\t\t\tfi\n>> +\t\t\t\techo \"$line\" >> $teefile\n>> +\t\t\t\techo $line\n>> +\t\t\tdone\n>> +\t\t) | git credential-store $cmd\n> \n> Since I'm not sure, I'll ask the question: do we need the sub-shell\n> here, or could we pipe directly off of the \"done\"? Like this:\n> \n> \t\twhile read line;\n> \t\tdo\n> \t\t\tif [ -z \"$line\" ]; then\n> \t\t\t\texit 0\n> \t\t\tfi\n> \t\t\techo \"$line\" >> $teefile\n> \t\t\techo $line\n> \t\tdone | git credential-store $cmd\n\nThat we can.. I will update in next iteration.\n\n>> +\tEOF\n> \n> \n>> +\tcat >expected-get <<-EOF &&\n>> +\tprotocol=http\n>> +\thost=127.0.0.1:5551\n>> +\twwwauth[0]=Bearer authority=https://login.example.com\n>> +\twwwauth[1]=FooAuth foo=bar baz=1\n>> +\twwwauth[2]=Basic realm=\"git-auth\"\n>> +\tEOF\n>> +\n>> +\tcat >expected-store <<-EOF &&\n>> +\tprotocol=http\n>> +\thost=127.0.0.1:5551\n>> +\tusername=user@host\n>> +\tpassword=pass@host\n>> +\tEOF\n>> +\n>> +\trm -f .git-credentials &&\n>> +\ttest_config credential.helper tee &&\n>> +\tset_askpass user@host pass@host &&\n>> +\t(\n>> +\t\tPATH=\"$PWD:$PATH\" &&\n>> +\t\tgit ls-remote \"$HTTPD_URL/auth-wwwauth/smart/repo.git\"\n>> +\t) &&\n>> +\texpect_askpass both user@host &&\n>> +\ttest_cmp expected-get credential-get &&\n>> +\ttest_cmp expected-store credential-store\n> \n> Elegant check for both calls.\n> \n> Thanks,\n> -Stolee\n\nThanks,\nMatthew\n"},{"id":"463403","messageId":"AS8PR03MB8689A10E534EC69BB71AEA56C04F9@AS8PR03MB8689.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com","subject":"Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-09-21T22:24:31Z","receivedAt":"2022-09-21T22:24:45Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-09-19 09:21, Derrick Stolee wrote:\n> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> +\t/**\n>> +\t * A `strvec` of WWW-Authenticate header values. Each string\n>> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n>> +\t * in the order they were received in the response.\n>> +\t */\n>> +\tstruct strvec wwwauth_headers;\n> \n> I like this careful documentation.\n> \n>> +\tunsigned header_is_last_match:1;\n> \n> But then this member is unclear how it is attached. It could use its\n> own \"for internal use\" comment if we don't want to describe it in full\n> detail here.\n\nA fair point. I will update in a future iteration.\n\n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> +{\n>> +\tsize_t size = eltsize * nmemb;\n>> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\tconst char *val;\n>> +\tconst char *z = NULL;\n>> +\n>> +\t/*\n>> +\t * Header lines may not come NULL-terminated from libcurl so we must\n>> +\t * limit all scans to the maximum length of the header line, or leverage\n>> +\t * strbufs for all operations.\n>> +\t *\n>> +\t * In addition, it is possible that header values can be split over\n>> +\t * multiple lines as per RFC 2616 (even though this has since been\n>> +\t * deprecated in RFC 7230). A continuation header field value is\n>> +\t * identified as starting with a space or horizontal tab.\n>> +\t *\n>> +\t * The formal definition of a header field as given in RFC 2616 is:\n>> +\t *\n>> +\t * message-header = field-name \":\" [ field-value ]\n>> +\t * field-name = token\n>> +\t * field-value = *( field-content | LWS )\n>> +\t * field-content = > +\t * and consisting of either *TEXT or combinations\n>> +\t * of token, separators, and quoted-string>\n>> +\t */\n>> +\n>> +\tstrbuf_add(&buf, ptr, size);\n>> +\n>> +\t/* Strip the CRLF that should be present at the end of each field */\n> \n> Is it really a CRLF? Or just an LF?\n\nIt is indeed an CRLF, agnostic of platform. HTTP defines CRLF as the\nend-of-line marker for all entities other than the body.\n\nSee RFC 2616 section 2.2: https://www.rfc-editor.org/rfc/rfc2616#section-2.2\n\n>> +\tstrbuf_trim_trailing_newline(&buf);\n> \n> Thankfully, this will trim an LF _or_ CR/LF pair, so either way would be fine.\n> \n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n>> +\t\twhile (isspace(*val)) val++;\n> \n> Break the \"val++;\" to its own line:\n> \n> \t\twhile (isspace(*val))\n> \t\t\tval++;\n\nSure! Sorry I missed this one.\n\n> While we are here, do we need to be careful about the end of the string at\n> this point? Is it possible that the server will send all spaces up until the\n> maximum header size (as mentioned in the message)?\n> \n>> +\n>> +\t\tstrvec_push(values, val);\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * This line could be a continuation of the previously matched header\n>> +\t * field. If this is the case then we should append this value to the\n>> +\t * end of the previously consumed value.\n>> +\t */\n>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>> +\t\tconst char **v = values->v + values->nr - 1;\n> \n> I suppose we expect leading spaces as critical to this header, right?\n\nLeading (and trailing) spaces are not part of the header value.\n\nFrom RFC 2616 section 2.2 regarding header field values:\n\n\"All linear white space, including folding, has the same semantics as SP.\nA recipient MAY replace any linear white space with a single SP before\ninterpreting the field value or forwarding the message downstream.\"\n\n>> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n> \n> We might have better luck using a strbuf, initializing it with the expected\n> size and using strbuf_add() to append the strings. Maybe I'm just prematurely\n> optimizing, though.\n\nThis code path is used to re-join/fold a header value continuation, which is\npretty rare in the wild (if at all with modern web servers).\n\n>> +\n>> +\t\tfree((void*)*v);\n>> +\t\t*v = append;\n>> +\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/* This is the start of a new header we don't care about */\n>> +\thttp_auth.header_is_last_match = 0;\n>> +\n>> +\t/*\n>> +\t * If this is a HTTP status line and not a header field, this signals\n>> +\t * a different HTTP response. libcurl writes all the output of all\n>> +\t * response headers of all responses, including redirects.\n>> +\t * We only care about the last HTTP request response's headers so clear\n>> +\t * the existing array.\n>> +\t */\n>> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n>> +\t\tstrvec_clear(values);\n>> +\n>> +exit:\n>> +\tstrbuf_release(&buf);\n>> +\treturn size;\n>> +}\n>> +\n>> size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n>> {\n>> \treturn nmemb;\n>> @@ -1829,6 +1904,8 @@ static int http_request(const char *url,\n>> \t\t\t\t\t fwrite_buffer);\n>> \t}\n>> \n>> +\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n> \n> Nice integration point!\n> \n> Thanks,\n> -Stolee\n\nThanks,\nMatthew\n"},{"id":"463405","messageId":"AS8PR03MB86897FAC3E1E4F03D4420644C04F9@AS8PR03MB8689.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"ead6293c-8906-04c9-6566-e47efb624385@github.com","subject":"Re: [PATCH 3/8] osxkeychain: clarify that we ignore unknown lines","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-09-21T22:48:21Z","receivedAt":"2022-09-21T22:48:41Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-09-19 09:12, Derrick Stolee wrote:\n> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Like in all the other credential helpers, the osxkeychain helper\n>> ignores unknown credential lines.\n>>\n>> Add a comment (a la the other helpers) to make it clear and explicit\n>> that this is the desired behaviour.\n> \n> I recommend that these first three patches be submitted for full\n> review and merging, since they seem important independent of this\n> RFC.\n> \n> Thanks,\n> -Stolee\n\nThat's a fair point. I will submit these independently.\n\nThanks,\nMatthew\n"},{"id":"463630","messageId":"de9bd893-a986-20e6-226e-913d0b6930cf@github.com","threadId":"58425","inReplyTo":"AS8PR03MB8689A10E534EC69BB71AEA56C04F9@AS8PR03MB8689.eurprd03.prod.outlook.com","subject":"Re: [PATCH 4/8] http: read HTTP WWW-Authenticate response headers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-09-26T14:13:18Z","receivedAt":"2022-09-26T15:31:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/21/2022 6:24 PM, Matthew John Cheetham wrote:\n> On 2022-09-19 09:21, Derrick Stolee wrote:\n>> On 9/13/2022 3:25 PM, Matthew John Cheetham via GitGitGadget wrote:\n\n>>> +\n>>> +\t\tstrvec_push(values, val);\n>>> +\t\thttp_auth.header_is_last_match = 1;\n>>> +\t\tgoto exit;\n>>> +\t}\n>>> +\n>>> +\t/*\n>>> +\t * This line could be a continuation of the previously matched header\n>>> +\t * field. If this is the case then we should append this value to the\n>>> +\t * end of the previously consumed value.\n>>> +\t */\n>>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>>> +\t\tconst char **v = values->v + values->nr - 1;\n>>\n>> I suppose we expect leading spaces as critical to this header, right?\n> \n> Leading (and trailing) spaces are not part of the header value.\n> \n> From RFC 2616 section 2.2 regarding header field values:\n> \n> \"All linear white space, including folding, has the same semantics as SP.\n> A recipient MAY replace any linear white space with a single SP before\n> interpreting the field value or forwarding the message downstream.\"\n> \n>>> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n>>\n>> We might have better luck using a strbuf, initializing it with the expected\n>> size and using strbuf_add() to append the strings. Maybe I'm just prematurely\n>> optimizing, though.\n> \n> This code path is used to re-join/fold a header value continuation, which is\n> pretty rare in the wild (if at all with modern web servers).\n\nI think the point is that I noticed that you removed the leading whitespace\nin a header's first line, but additional whitespace after this first space\nwill be included in the concatenated content of the header value.\n\nAs long as that is the intention, then I'm happy here.\n\nThanks,\n-Stolee\n"},{"id":"465487","messageId":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.git.1663097156.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:07:57Z","receivedAt":"2022-10-21T17:08:13Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server based\nheavily on git-daemon, with simple authentication configurable by command\nline args.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Leverage newest identity standards, enhancements, and threat\n mitigations - all without updating Git.\n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n 3. Teach Git to specify authentication schemes other than Basic in\n subsequent HTTP requests based on credential helper responses.\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture flexibility\n==================\n\nBy allowing the credential helpers decide the best authentication scheme, we\ncan allow the remote Git server to both offer new schemes (or remove old\nones) that enlightened credential helpers could take immediate advantage of,\nand to use credentials that are much more tightly scoped and bound to the\nspecific request.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \n\n\n\nShould Git not control the set of authentication schemes?\n=========================================================\n\nOne concern that the reader may have regarding these changes is in allowing\nhelpers to select the authentication mechanism to use, it may be possible\nthat a weaker form of authentication is used.\n\nTake for example a Git remote server that responds with the following\nauthentication schemes:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Negotiate ...\nWWW-Authenticate: Basic ...\n\n\nToday Git (and libcurl) prefer to Negotiate over Basic authentication [13].\nIf a helper responded with authtype=basic Git would now be using a \"less\nsecure\" mechanism.\n\nThe reason we still propose the credential helper decide on the\nauthentication scheme is that Git is not the best placed entity to decide\nwhat type of authentication should be used for a particular request (see\nDesign Principle 3).\n\nOAuth Bearer tokens are often bundled in Basic Authorization headers [14],\nbut given that the tokens are/can be short-lived and have a highly scoped\nset of permissions, this solution could be argued as being more secure than\nsomething like NTLM [15]. Similarly, the user may wish to be consulted on\nselecting a particular user account, or directly selecting an authentication\nmechanism for a request that otherwise they would not be able to use.\n\nAlso, as new authentication protocols appear Git does not need to be\nmodified or updated for the user to take advantage of them; the credential\nhelpers take on the responsibility of learning and selecting the \"best\"\noption.\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n * [13] libcurl http.c pickoneauth Function\n https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416\n\n * [14] Git Credential Manager GitHub Host Provider (using PAT as password)\n https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157\n\n * [15] NT LAN Manager (NTLM) Authentication Protocol\n https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\nMatthew John Cheetham (6):\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n http: store all request headers on active_request_slot\n http: move proactive auth to first slot creation\n http: set specific auth scheme depending on credential\n t5556-http-auth: add test for HTTP auth hdr logic\n\n Documentation/git-credential.txt | 18 +\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n credential.c | 18 +\n credential.h | 16 +\n git-curl-compat.h | 10 +\n http-push.c | 103 +-\n http-walker.c | 2 +-\n http.c | 200 +++-\n http.h | 4 +-\n remote-curl.c | 36 +-\n t/helper/.gitignore | 1 +\n t/helper/test-credential-helper-replay.sh | 14 +\n t/helper/test-http-server.c | 1134 +++++++++++++++++++++\n t/t5556-http-auth.sh | 260 +++++\n 15 files changed, 1695 insertions(+), 136 deletions(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n create mode 100644 t/helper/test-http-server.c\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: 9c32cfb49c60fa8173b9666db02efe3b45a8522f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v1:\n\n 1: 6426f9c3954 < -: ----------- wincred: ignore unknown lines (do not die)\n 2: ae5c1bfc092 < -: ----------- netrc: ignore unknown lines (do not die)\n 3: 2ece562a595 < -: ----------- osxkeychain: clarify that we ignore unknown lines\n 4: 78e66d56605 ! 1: f297c78f60a http: read HTTP WWW-Authenticate response headers\n @@ credential.h: struct credential {\n +\t * in the order they were received in the response.\n +\t */\n +\tstruct strvec wwwauth_headers;\n ++\n ++\t/**\n ++\t * Internal use only. Used to keep track of split header fields\n ++\t * in order to fold multiple lines into one value.\n ++\t */\n +\tunsigned header_is_last_match:1;\n +\n \tunsigned approved:1,\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\n +\t/* Start of a new WWW-Authenticate header */\n +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n -+\t\twhile (isspace(*val)) val++;\n ++\t\twhile (isspace(*val))\n ++\t\t\tval++;\n +\n +\t\tstrvec_push(values, val);\n +\t\thttp_auth.header_is_last_match = 1;\n 5: 936545004b8 ! 2: 0838d992744 credential: add WWW-Authenticate header to cred requests\n @@ Commit message\n Signed-off-by: Matthew John Cheetham \n \n ## Documentation/git-credential.txt ##\n -@@ Documentation/git-credential.txt: Git understands the following attributes:\n - \twere read (e.g., `url=https://example.com` would behave as if\n - \t`protocol=https` and `host=example.com` had been provided). This\n - \tcan help callers avoid parsing URLs themselves.\n -+\n -+`wwwauth[n]`::\n +@@ Documentation/git-credential.txt: empty string.\n + Components which are missing from the URL (e.g., there is no\n + username in the example above) will be left unset.\n + \n ++`wwwauth[]`::\n +\n +\tWhen an HTTP response is received that includes one or more\n +\t'WWW-Authenticate' authentication headers, these can be passed to Git\n +\t(and subsequent credential helpers) with these attributes.\n +\tEach 'WWW-Authenticate' header value should be passed as a separate\n -+\tattribute 'wwwauth[n]' where 'n' is the zero-indexed order the headers\n -+\tappear in the HTTP response.\n - +\n - Note that specifying a protocol is mandatory and if the URL\n - doesn't specify a hostname (e.g., \"cert:///path/to/file\") the\n ++\tattribute 'wwwauth[]' where the order of the attributes is the same\n ++\tas they appear in the HTTP response.\n ++\n + GIT\n + ---\n + Part of the linkgit:git[1] suite\n \n ## credential.c ##\n @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,\n @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const\n +\t\t\t\t const struct strvec *vec)\n +{\n +\tint i = 0;\n ++\tconst char *full_key = xstrfmt(\"%s[]\", key);\n +\tfor (; i < vec->nr; i++) {\n -+\t\tconst char *full_key = xstrfmt(\"%s[%d]\", key, i);\n +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n -+\t\tfree((void*)full_key);\n +\t}\n ++\tfree((void*)full_key);\n +}\n +\n void credential_write(const struct credential *c, FILE *fp)\n @@ credential.c: void credential_write(const struct credential *c, FILE *fp)\n }\n \n static int run_credential_helper(struct credential *c,\n -\n - ## t/lib-httpd/apache.conf ##\n -@@ t/lib-httpd/apache.conf: SSLEngine On\n - \tRequire valid-user\n - \n - \n -+# Advertise two additional auth methods above \"Basic\".\n -+# Neither of them actually work but serve test cases showing these\n -+# additional auth headers are consumed correctly.\n -+\n -+\tAuthType Basic\n -+\tAuthName \"git-auth\"\n -+\tAuthUserFile passwd\n -+\tRequire valid-user\n -+\tSetEnvIf Authorization \"^\\S+\" authz\n -+\tHeader always add WWW-Authenticate \"Bearer authority=https://login.example.com\" env=!authz\n -+\tHeader always add WWW-Authenticate \"FooAuth foo=bar baz=1\" env=!authz\n -+\n -+\n - RewriteCond %{QUERY_STRING} service=git-receive-pack [OR]\n - RewriteCond %{REQUEST_URI} /git-receive-pack$\n - RewriteRule ^/half-auth-complete/ - [E=AUTHREQUIRED:yes]\n -\n - ## t/t5551-http-fetch-smart.sh ##\n -@@ t/t5551-http-fetch-smart.sh: test_expect_success 'http auth forgets bogus credentials' '\n - \texpect_askpass both user@host\n - '\n - \n -+test_expect_success 'http auth sends www-auth headers to credential helper' '\n -+\twrite_script git-credential-tee <<-\\EOF &&\n -+\t\tcmd=$1\n -+\t\tteefile=credential-$cmd\n -+\t\tif [ -f \"$teefile\" ]; then\n -+\t\t\trm $teefile\n -+\t\tfi\n -+\t\t(\n -+\t\t\twhile read line;\n -+\t\t\tdo\n -+\t\t\t\tif [ -z \"$line\" ]; then\n -+\t\t\t\t\texit 0\n -+\t\t\t\tfi\n -+\t\t\t\techo \"$line\" >> $teefile\n -+\t\t\t\techo $line\n -+\t\t\tdone\n -+\t\t) | git credential-store $cmd\n -+\tEOF\n -+\n -+\tcat >expected-get <<-EOF &&\n -+\tprotocol=http\n -+\thost=127.0.0.1:5551\n -+\twwwauth[0]=Bearer authority=https://login.example.com\n -+\twwwauth[1]=FooAuth foo=bar baz=1\n -+\twwwauth[2]=Basic realm=\"git-auth\"\n -+\tEOF\n -+\n -+\tcat >expected-store <<-EOF &&\n -+\tprotocol=http\n -+\thost=127.0.0.1:5551\n -+\tusername=user@host\n -+\tpassword=pass@host\n -+\tEOF\n -+\n -+\trm -f .git-credentials &&\n -+\ttest_config credential.helper tee &&\n -+\tset_askpass user@host pass@host &&\n -+\t(\n -+\t\tPATH=\"$PWD:$PATH\" &&\n -+\t\tgit ls-remote \"$HTTPD_URL/auth-wwwauth/smart/repo.git\"\n -+\t) &&\n -+\texpect_askpass both user@host &&\n -+\ttest_cmp expected-get credential-get &&\n -+\ttest_cmp expected-store credential-store\n -+'\n -+\n - test_expect_success 'client falls back from v2 to v0 to match server' '\n - \tGIT_TRACE_PACKET=$PWD/trace \\\n - \tGIT_TEST_PROTOCOL_VERSION=2 \\\n 6: 20843e2051e = 3: c62fef65f46 http: store all request headers on active_request_slot\n 7: cae7180bc37 = 4: a790c01f9f2 http: move proactive auth to first slot creation\n 8: 7f827067f55 ! 5: b0b7cd7ee5e http: set specific auth scheme depending on credential\n @@ Commit message\n Signed-off-by: Matthew John Cheetham \n \n ## Documentation/git-credential.txt ##\n -@@ Documentation/git-credential.txt: Git understands the following attributes:\n - \t`protocol=https` and `host=example.com` had been provided). This\n - \tcan help callers avoid parsing URLs themselves.\n +@@ Documentation/git-credential.txt: username in the example above) will be left unset.\n + \tattribute 'wwwauth[]' where the order of the attributes is the same\n + \tas they appear in the HTTP response.\n \n +`authtype`::\n +\n @@ Documentation/git-credential.txt: Git understands the following attributes:\n +\tscheme for the `Authorization` header, and the `password` field is\n +\tused as the raw unencoded authorization parameters of the same header.\n +\n - `wwwauth[n]`::\n - \n - \tWhen an HTTP response is received that includes one or more\n + GIT\n + ---\n + Part of the linkgit:git[1] suite\n \n ## credential.c ##\n @@ credential.c: void credential_clear(struct credential *c)\n @@ git-curl-compat.h\n \n +/**\n + * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.\n ++ * However, only 7.69.0 fixes a bug where Bearer headers were not\n ++ * actually sent with reused connections on subsequent transfers\n ++ * (curl/curl@dea17b519dc1).\n + */\n -+#if LIBCURL_VERSION_NUM >= 0x073D00\n ++#if LIBCURL_VERSION_NUM >= 0x074500\n +#define GIT_CURL_HAVE_CURLAUTH_BEARER\n +#endif\n +\n -: ----------- > 6: f3f13ed8c82 t5556-http-auth: add test for HTTP auth hdr logic\n\n-- \ngitgitgadget\n"},{"id":"465488","messageId":"f297c78f60a6996c2d2e5397b05efa6b94fd2ae0.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:07:58Z","receivedAt":"2022-10-21T17:08:15Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++++++\n http.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 94 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex 5d0502f51fd..03d43d352e7 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tconst char *z = NULL;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\tconst char **v = values->v + values->nr - 1;\n+\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n+\n+\t\tfree((void*)*v);\n+\t\t*v = append;\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (skip_iprefix(buf.buf, \"http/\", &z))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1829,6 +1905,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"465489","messageId":"0838d992744a4b06523be6df0edb046ebba033ee.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:07:59Z","receivedAt":"2022-10-21T17:08:17Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[n]` properties where `n` is a\nzero-indexed number, reflecting the order the WWW-Authenticate headers\nappeared in the HTTP response.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 9 +++++++++\n credential.c | 12 ++++++++++++\n 2 files changed, 21 insertions(+)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex f18673017f5..0ff3cbc25b9 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -160,6 +160,15 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received that includes one or more\n+\t'WWW-Authenticate' authentication headers, these can be passed to Git\n+\t(and subsequent credential helpers) with these attributes.\n+\tEach 'WWW-Authenticate' header value should be passed as a separate\n+\tattribute 'wwwauth[]' where the order of the attributes is the same\n+\tas they appear in the HTTP response.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..8a3ad6c0ae2 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tint i = 0;\n+\tconst char *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree((void*)full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\n-- \ngitgitgadget\n\n"},{"id":"465490","messageId":"c62fef65f46867557934f95dd2acb9ec96365ea3.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] http: store all request headers on active_request_slot","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:08:00Z","receivedAt":"2022-10-21T17:08:29Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nOnce a list of headers has been set on the curl handle, it is not\npossible to recover that `struct curl_slist` instance to add or modify\nheaders.\n\nIn future commits we will want to modify the set of request headers in\nresponse to an authentication challenge/401 response from the server,\nwith information provided by a credential helper.\n\nThere are a number of different places where curl is used for an HTTP\nrequest, and they do not have a common handling of request headers.\nHowever, given that they all do call the `start_active_slot()` function,\neither directly or indirectly via `run_slot()` or `run_one_slot()`, we\nuse this as the point to set the `CURLOPT_HTTPHEADER` option just\nbefore the request is made.\n\nWe collect all request headers in a `struct curl_slist` on the\n`struct active_request_slot` that is obtained from a call to\n`get_active_slot(int)`. This function now takes a single argument to\ndefine if the initial set of headers on the slot should include the\n\"Pragma: no-cache\" header, along with all extra headers specified via\n`http.extraHeader` config values.\n\nThe active request slot obtained from `get_active_slot(int)` will always\ncontain a fresh set of default headers and any headers set in previous\nusages of this slot will be freed.\n\nSigned-off-by: Matthew John Cheetham \n---\n http-push.c | 103 ++++++++++++++++++++++----------------------------\n http-walker.c | 2 +-\n http.c | 82 ++++++++++++++++++----------------------\n http.h | 4 +-\n remote-curl.c | 36 +++++++++---------\n 5 files changed, 101 insertions(+), 126 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 5f4340a36e6..2b40959b376 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,\n \tcurl_easy_setopt(curl, CURLOPT_UPLOAD, 1);\n }\n \n-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)\n+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,\n+\tstruct remote_lock *lock, enum dav_header_flag options)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n \tif (options & DAV_HEADER_IF) {\n \t\tstrbuf_addf(&buf, \"If: (<%s>)\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_LOCK) {\n \t\tstrbuf_addf(&buf, \"Lock-Token: <%s>\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_TIMEOUT) {\n \t\tstrbuf_addf(&buf, \"Timeout: Second-%ld\", lock->timeout);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tstrbuf_release(&buf);\n \n-\treturn dav_headers;\n+\treturn headers;\n }\n \n static void finish_request(struct transfer_request *request);\n@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)\n \n \trequest->url = get_remote_object_url(repo->url, hex, 1);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MKCOL);\n@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)\n \tstrbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);\n \trequest->url = strbuf_detach(&buf, NULL);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http(slot->curl, request->url, DAV_PUT,\n@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)\n static void start_move(struct transfer_request *request)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MOVE);\n-\tdav_headers = curl_slist_append(dav_headers, request->dest);\n-\tdav_headers = curl_slist_append(dav_headers, \"Overwrite: T\");\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n+\tslot->headers = curl_slist_append(slot->headers, request->dest);\n+\tslot->headers = curl_slist_append(slot->headers, \"Overwrite: T\");\n \n \tif (start_active_slot(slot)) {\n \t\trequest->slot = slot;\n@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n \tlock->refreshing = 1;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_LOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)\n \t}\n \n \tlock->refreshing = 0;\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn rc;\n }\n@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tchar *ep;\n \tchar timeout_header[25];\n \tstruct remote_lock *lock = NULL;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tchar *escaped;\n \n@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \twhile (ep) {\n \t\tchar saved_character = ep[1];\n \t\tep[1] = '\\0';\n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n \t\tcurl_setup_http_get(slot->curl, url, DAV_MKCOL);\n \t\tif (start_active_slot(slot)) {\n@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tstrbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);\n \tfree(escaped);\n \n+\tslot = get_active_slot(0);\n+\tslot->results = &results;\n+\n \txsnprintf(timeout_header, sizeof(timeout_header), \"Timeout: Second-%ld\", timeout);\n-\tdav_headers = curl_slist_append(dav_headers, timeout_header);\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n+\tslot->headers = curl_slist_append(slot->headers, timeout_header);\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n \n-\tslot = get_active_slot();\n-\tslot->results = &results;\n \tcurl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tCALLOC_ARRAY(lock, 1);\n@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \t\tfprintf(stderr, \"Unable to start LOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n \n@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct remote_lock *prev = repo->locks;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_LOCK);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)\n \t\tfprintf(stderr, \"Unable to start UNLOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n-\n \tif (repo->locks == lock) {\n \t\trepo->locks = lock->next;\n \t} else {\n@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tstruct remote_ls_ctx ls;\n \n@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,\n \n \tstrbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 1\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 1\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,\n \tfree(url);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n }\n \n static void get_remote_object_list(unsigned char parent)\n@@ -1199,7 +1189,6 @@ static int locking_available(void)\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tint lock_flags = 0;\n \tchar *escaped;\n@@ -1208,14 +1197,14 @@ static int locking_available(void)\n \tstrbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);\n \tfree(escaped);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 0\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 0\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, repo->url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1257,7 +1246,6 @@ static int locking_available(void)\n \n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn lock_flags;\n }\n@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers;\n-\n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n \tstrbuf_addf(&out_buffer.buf, \"%s\\n\", oid_to_hex(oid));\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF);\n+\n \tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t&out_buffer, fwrite_null);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)\n \tstruct buffer buffer = { STRBUF_INIT, 0 };\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \n \tremote_ls(\"refs/\", (PROCESS_FILES | RECURSIVE),\n \t\t add_remote_info_ref, &buffer.buf);\n \tif (!aborted) {\n-\t\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n+\t\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\t\tDAV_HEADER_IF);\n+\n \t\tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t\t&buffer, fwrite_null);\n-\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \t\tif (start_active_slot(slot)) {\n \t\t\trun_active_slot(slot);\n@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)\n \tif (dry_run)\n \t\treturn 0;\n \turl = xstrfmt(\"%s%s\", repo->url, remote_ref->name);\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n \tcurl_setup_http_get(slot->curl, url, DAV_DELETE);\n \tif (start_active_slot(slot)) {\ndiff --git a/http-walker.c b/http-walker.c\nindex b8f0f98ae14..8747de2fcdb 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)\n \t * Use a callback to process the result, since another request\n \t * may fail and need to have alternates loaded before continuing\n \t */\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_alternates_response;\n \talt_req.walker = walker;\n \tslot->callback_data = &alt_req;\ndiff --git a/http.c b/http.c\nindex 03d43d352e7..f2ebb17c8c4 100644\n--- a/http.c\n+++ b/http.c\n@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =\n \t| CURLAUTH_DIGEST_IE\n \t| CURLAUTH_DIGEST;\n \n-static struct curl_slist *pragma_header;\n-static struct curl_slist *no_pragma_header;\n static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;\n \n static struct curl_slist *host_resolutions;\n@@ -1133,11 +1131,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n-\tpragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma: no-cache\");\n-\tno_pragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma:\");\n-\n \t{\n \t\tchar *http_max_requests = getenv(\"GIT_HTTP_MAX_REQUESTS\");\n \t\tif (http_max_requests)\n@@ -1199,6 +1192,8 @@ void http_cleanup(void)\n \n \twhile (slot != NULL) {\n \t\tstruct active_request_slot *next = slot->next;\n+\t\tif (slot->headers)\n+\t\t\tcurl_slist_free_all(slot->headers);\n \t\tif (slot->curl) {\n \t\t\txmulti_remove_handle(slot);\n \t\t\tcurl_easy_cleanup(slot->curl);\n@@ -1215,12 +1210,6 @@ void http_cleanup(void)\n \n \tstring_list_clear(&extra_http_headers, 0);\n \n-\tcurl_slist_free_all(pragma_header);\n-\tpragma_header = NULL;\n-\n-\tcurl_slist_free_all(no_pragma_header);\n-\tno_pragma_header = NULL;\n-\n \tcurl_slist_free_all(host_resolutions);\n \thost_resolutions = NULL;\n \n@@ -1255,7 +1244,18 @@ void http_cleanup(void)\n \tFREE_AND_NULL(cached_accept_language);\n }\n \n-struct active_request_slot *get_active_slot(void)\n+static struct curl_slist *http_copy_default_headers(void)\n+{\n+\tstruct curl_slist *headers = NULL;\n+\tconst struct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, &extra_http_headers)\n+\t\theaders = curl_slist_append(headers, item->string);\n+\n+\treturn headers;\n+}\n+\n+struct active_request_slot *get_active_slot(int no_pragma_header)\n {\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n@@ -1277,6 +1277,7 @@ struct active_request_slot *get_active_slot(void)\n \t\tnewslot->curl = NULL;\n \t\tnewslot->in_use = 0;\n \t\tnewslot->next = NULL;\n+\t\tnewslot->headers = NULL;\n \n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n@@ -1294,6 +1295,15 @@ struct active_request_slot *get_active_slot(void)\n \t\tcurl_session_count++;\n \t}\n \n+\tif (slot->headers)\n+\t\tcurl_slist_free_all(slot->headers);\n+\n+\tslot->headers = http_copy_default_headers();\n+\n+\tif (!no_pragma_header)\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Pragma: no-cache\");\n+\n \tactive_requests++;\n \tslot->in_use = 1;\n \tslot->results = NULL;\n@@ -1303,7 +1313,6 @@ struct active_request_slot *get_active_slot(void)\n \tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);\n \tif (curl_save_cookies)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);\n \tcurl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n \tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);\n@@ -1335,9 +1344,12 @@ struct active_request_slot *get_active_slot(void)\n \n int start_active_slot(struct active_request_slot *slot)\n {\n-\tCURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\tCURLMcode curlm_result;\n \tint num_transfers;\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);\n+\tcurlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\n \tif (curlm_result != CURLM_OK &&\n \t curlm_result != CURLM_CALL_MULTI_PERFORM) {\n \t\twarning(\"curl_multi_add_handle failed: %s\",\n@@ -1652,17 +1664,6 @@ int run_one_slot(struct active_request_slot *slot,\n \treturn handle_curl_result(results);\n }\n \n-struct curl_slist *http_copy_default_headers(void)\n-{\n-\tstruct curl_slist *headers = NULL;\n-\tconst struct string_list_item *item;\n-\n-\tfor_each_string_list_item(item, &extra_http_headers)\n-\t\theaders = curl_slist_append(headers, item->string);\n-\n-\treturn headers;\n-}\n-\n static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)\n {\n \tchar *ptr;\n@@ -1880,12 +1881,11 @@ static int http_request(const char *url,\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tint no_cache = options && options->no_cache;\n \tconst char *accept_language;\n \tint ret;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(!no_cache);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);\n \n \tif (!result) {\n@@ -1910,27 +1910,23 @@ static int http_request(const char *url,\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-\t\theaders = curl_slist_append(headers, accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\taccept_language);\n \n-\tstrbuf_addstr(&buf, \"Pragma:\");\n-\tif (options && options->no_cache)\n-\t\tstrbuf_addstr(&buf, \" no-cache\");\n \tif (options && options->initial_request &&\n \t http_follow_config == HTTP_FOLLOW_INITIAL)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);\n \n-\theaders = curl_slist_append(headers, buf.buf);\n-\n \t/* Add additional headers here */\n \tif (options && options->extra_headers) {\n \t\tconst struct string_list_item *item;\n \t\tfor_each_string_list_item(item, options->extra_headers) {\n-\t\t\theaders = curl_slist_append(headers, item->string);\n+\t\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\titem->string);\n \t\t}\n \t}\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);\n \n@@ -1948,9 +1944,6 @@ static int http_request(const char *url,\n \t\tcurlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,\n \t\t\t\toptions->effective_url);\n \n-\tcurl_slist_free_all(headers);\n-\tstrbuf_release(&buf);\n-\n \treturn ret;\n }\n \n@@ -2311,12 +2304,10 @@ struct http_pack_request *new_direct_http_pack_request(\n \t\tgoto abort;\n \t}\n \n-\tpreq->slot = get_active_slot();\n+\tpreq->slot = get_active_slot(1);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);\n-\tcurl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,\n-\t\tno_pragma_header);\n \n \t/*\n \t * If there is data present from a previous transfer attempt,\n@@ -2481,14 +2472,13 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \t\t}\n \t}\n \n-\tfreq->slot = get_active_slot();\n+\tfreq->slot = get_active_slot(1);\n \n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \n \t/*\n \t * If we have successfully processed data from a previous fetch\ndiff --git a/http.h b/http.h\nindex 3c94c479100..a304cc408b2 100644\n--- a/http.h\n+++ b/http.h\n@@ -22,6 +22,7 @@ struct slot_results {\n struct active_request_slot {\n \tCURL *curl;\n \tint in_use;\n+\tstruct curl_slist *headers;\n \tCURLcode curl_result;\n \tlong http_code;\n \tint *finished;\n@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);\n curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);\n \n /* Slot lifecycle functions */\n-struct active_request_slot *get_active_slot(void);\n+struct active_request_slot *get_active_slot(int no_pragma_header);\n int start_active_slot(struct active_request_slot *slot);\n void run_active_slot(struct active_request_slot *slot);\n void finish_all_active_slots(void);\n@@ -64,7 +65,6 @@ void step_active_slots(void);\n void http_init(struct remote *remote, const char *url,\n \t int proactive_auth);\n void http_cleanup(void);\n-struct curl_slist *http_copy_default_headers(void);\n \n extern long int git_curl_ipresolve;\n extern int active_requests;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 72dfb8fb86a..edbd4504beb 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,\n static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tstruct strbuf buf = STRBUF_INIT;\n \tint err;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, \"0000\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);\n \n \terr = run_slot(slot, results);\n \n-\tcurl_slist_free_all(headers);\n \tstrbuf_release(&buf);\n \treturn err;\n }\n@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)\n static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tint use_gzip = rpc->gzip_request;\n \tchar *gzip_body = NULL;\n \tsize_t gzip_size = 0;\n@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece\n \t\t\tneeds_100_continue = 1;\n \t}\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n-\theaders = curl_slist_append(headers, needs_100_continue ?\n+retry:\n+\tslot = get_active_slot(0);\n+\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, needs_100_continue ?\n \t\t\"Expect: 100-continue\" : \"Expect:\");\n \n \t/* Add Accept-Language header */\n \tif (rpc->hdr_accept_language)\n-\t\theaders = curl_slist_append(headers, rpc->hdr_accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->hdr_accept_language);\n \n \t/* Add the extra Git-Protocol header */\n \tif (rpc->protocol_header)\n-\t\theaders = curl_slist_append(headers, rpc->protocol_header);\n-\n-retry:\n-\tslot = get_active_slot();\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->protocol_header);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -955,7 +953,8 @@ retry:\n \t\t/* The request body is large and the size cannot be predicted.\n \t\t * We must use chunked encoding to send it.\n \t\t */\n-\t\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Transfer-Encoding: chunked\");\n \t\trpc->initial_buffer = 1;\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);\n@@ -1002,7 +1001,8 @@ retry:\n \n \t\tgzip_size = stream.total_out;\n \n-\t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Content-Encoding: gzip\");\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));\n \n@@ -1025,7 +1025,6 @@ retry:\n \t\t}\n \t}\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n \trpc_in_data.rpc = rpc;\n \trpc_in_data.slot = slot;\n@@ -1055,7 +1054,6 @@ retry:\n \tif (stateless_connect)\n \t\tpacket_response_end(rpc->in);\n \n-\tcurl_slist_free_all(headers);\n \tfree(gzip_body);\n \treturn err;\n }\n-- \ngitgitgadget\n\n"},{"id":"465491","messageId":"a790c01f9f279bba227a8a27077e54b95fb991f7.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] http: move proactive auth to first slot creation","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:08:01Z","receivedAt":"2022-10-21T17:08:36Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRather than proactively seek credentials to authenticate a request at\n`http_init()` time, do it when the first `active_request_slot` is\ncreated.\n\nBecause credential helpers may modify the headers used for a request we\ncan only auth when a slot is created (when we can first start to gather\nrequest headers).\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 20 +++++++++++---------\n 1 file changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex f2ebb17c8c4..17b47195d22 100644\n--- a/http.c\n+++ b/http.c\n@@ -515,18 +515,18 @@ static int curl_empty_auth_enabled(void)\n \treturn 0;\n }\n \n-static void init_curl_http_auth(CURL *result)\n+static void init_curl_http_auth(struct active_request_slot *slot)\n {\n \tif (!http_auth.username || !*http_auth.username) {\n \t\tif (curl_empty_auth_enabled())\n-\t\t\tcurl_easy_setopt(result, CURLOPT_USERPWD, \":\");\n+\t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n \t}\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n }\n \n /* *var must be free-able */\n@@ -901,9 +901,6 @@ static CURL *get_curl_handle(void)\n #endif\n \t}\n \n-\tif (http_proactive_auth)\n-\t\tinit_curl_http_auth(result);\n-\n \tif (getenv(\"GIT_SSL_VERSION\"))\n \t\tssl_version = getenv(\"GIT_SSL_VERSION\");\n \tif (ssl_version && *ssl_version) {\n@@ -1260,6 +1257,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n \n+\tint proactive_auth = 0;\n \tint num_transfers;\n \n \t/* Wait for a slot to open up if the queue is full */\n@@ -1282,6 +1280,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n \t\t\tactive_queue_head = newslot;\n+\n+\t\t\t/* Auth first slot if asked for proactive auth */\n+\t\t\tproactive_auth = http_proactive_auth;\n \t\t} else {\n \t\t\twhile (slot->next != NULL)\n \t\t\t\tslot = slot->next;\n@@ -1336,8 +1337,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);\n-\tif (http_auth.password || curl_empty_auth_enabled())\n-\t\tinit_curl_http_auth(slot->curl);\n+\n+\tif (http_auth.password || curl_empty_auth_enabled() || proactive_auth)\n+\t\tinit_curl_http_auth(slot);\n \n \treturn slot;\n }\n-- \ngitgitgadget\n\n"},{"id":"465492","messageId":"b0b7cd7ee5e4914dba99ac93cc4800f25df9cb9a.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] http: set specific auth scheme depending on credential","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:08:02Z","receivedAt":"2022-10-21T17:08:48Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a new credential field `authtype` that can be used by\ncredential helpers to indicate the type of the credential or\nauthentication mechanism to use for a request.\n\nModify http.c to now specify the correct authentication scheme or\ncredential type when authenticating the curl handle. If the new\n`authtype` field in the credential structure is `NULL` or \"Basic\" then\nuse the existing username/password options. If the field is \"Bearer\"\nthen use the OAuth bearer token curl option. Otherwise, the `authtype`\nfield is the authentication scheme and the `password` field is the\nraw, unencoded value.\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 9 +++++++++\n credential.c | 5 +++++\n credential.h | 1 +\n git-curl-compat.h | 10 ++++++++++\n http.c | 24 +++++++++++++++++++++---\n 5 files changed, 46 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex 0ff3cbc25b9..82ade09b5e9 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -169,6 +169,15 @@ username in the example above) will be left unset.\n \tattribute 'wwwauth[]' where the order of the attributes is the same\n \tas they appear in the HTTP response.\n \n+`authtype`::\n+\n+\tIndicates the type of authentication scheme used. If this is not\n+\tpresent the default is \"Basic\".\n+\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n+\tIf an unknown value is provided, this is taken as the authentication\n+\tscheme for the `Authorization` header, and the `password` field is\n+\tused as the raw unencoded authorization parameters of the same header.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/credential.c b/credential.c\nindex 8a3ad6c0ae2..a556f9f375a 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)\n \tfree(c->path);\n \tfree(c->username);\n \tfree(c->password);\n+\tfree(c->authtype);\n \tstring_list_clear(&c->helpers, 0);\n \tstrvec_clear(&c->wwwauth_headers);\n \n@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)\n \t\t} else if (!strcmp(key, \"path\")) {\n \t\t\tfree(c->path);\n \t\t\tc->path = xstrdup(value);\n+\t\t} else if (!strcmp(key, \"authtype\")) {\n+\t\t\tfree(c->authtype);\n+\t\t\tc->authtype = xstrdup(value);\n \t\t} else if (!strcmp(key, \"url\")) {\n \t\t\tcredential_from_url(c, value);\n \t\t} else if (!strcmp(key, \"quit\")) {\n@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_item(fp, \"authtype\", c->authtype, 0);\n \tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \ndiff --git a/credential.h b/credential.h\nindex 6f2e5bc610b..8d580b054d0 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -140,6 +140,7 @@ struct credential {\n \tchar *protocol;\n \tchar *host;\n \tchar *path;\n+\tchar *authtype;\n };\n \n #define CREDENTIAL_INIT { \\\ndiff --git a/git-curl-compat.h b/git-curl-compat.h\nindex 56a83b6bbd8..839049f6dfe 100644\n--- a/git-curl-compat.h\n+++ b/git-curl-compat.h\n@@ -126,4 +126,14 @@\n #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS\n #endif\n \n+/**\n+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.\n+ * However, only 7.69.0 fixes a bug where Bearer headers were not\n+ * actually sent with reused connections on subsequent transfers\n+ * (curl/curl@dea17b519dc1).\n+ */\n+#if LIBCURL_VERSION_NUM >= 0x074500\n+#define GIT_CURL_HAVE_CURLAUTH_BEARER\n+#endif\n+\n #endif\ndiff --git a/http.c b/http.c\nindex 17b47195d22..ac620bcbf0c 100644\n--- a/http.c\n+++ b/http.c\n@@ -517,7 +517,8 @@ static int curl_empty_auth_enabled(void)\n \n static void init_curl_http_auth(struct active_request_slot *slot)\n {\n-\tif (!http_auth.username || !*http_auth.username) {\n+\tif (!http_auth.authtype &&\n+\t\t(!http_auth.username || !*http_auth.username)) {\n \t\tif (curl_empty_auth_enabled())\n \t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n@@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n+\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n+\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n+\t\t\thttp_auth.username);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n+\t\t\thttp_auth.password);\n+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n+\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n+\t\t\thttp_auth.password);\n+#endif\n+\t} else {\n+\t\tstruct strbuf auth = STRBUF_INIT;\n+\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n+\t\t\thttp_auth.authtype, http_auth.password);\n+\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n+\t\tstrbuf_release(&auth);\n+\t}\n }\n \n /* *var must be free-able */\n-- \ngitgitgadget\n\n"},{"id":"465493","messageId":"f3f13ed8c8238f396163dd0e6a3d6c948c2b879b.1666372083.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-21T17:08:03Z","receivedAt":"2022-10-21T17:08:50Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a series of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers can respond\nto requests that contain WWW-Authenticate information with the ability\nto select the response Authenticate header scheme.\n\nIntroduce a mini HTTP server helper that provides a frontend for the\ngit-http-backend, with support for arbitrary authentication schemes.\nThe test-http-server is based heavily on the git-daemon, and forwards\nall successfully authenticated requests to the http-backend.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n t/helper/.gitignore | 1 +\n t/helper/test-credential-helper-replay.sh | 14 +\n t/helper/test-http-server.c | 1134 +++++++++++++++++++++\n t/t5556-http-auth.sh | 260 +++++\n 6 files changed, 1424 insertions(+)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n create mode 100644 t/helper/test-http-server.c\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/Makefile b/Makefile\nindex d93ad956e58..39b130f711d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1500,6 +1500,8 @@ else\n \tendif\n \tBASIC_CFLAGS += $(CURL_CFLAGS)\n \n+\tTEST_PROGRAMS_NEED_X += test-http-server\n+\n \tREMOTE_CURL_PRIMARY = git-remote-http$X\n \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 787738e6fa3..45251695ce0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -989,6 +989,19 @@ set(wrapper_scripts\n set(wrapper_test_scripts\n \ttest-fake-ssh test-tool)\n \n+if(CURL_FOUND)\n+ list(APPEND wrapper_test_scripts test-http-server)\n+\n+ add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+ target_link_libraries(test-http-server common-main)\n+\n+ if(MSVC)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+ endif()\n+endif()\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..1a94ab6eed5 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+test-http-server\ndiff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\nnew file mode 100755\nindex 00000000000..03e5e63dad6\n--- /dev/null\n+++ b/t/helper/test-credential-helper-replay.sh\n@@ -0,0 +1,14 @@\n+cmd=$1\n+teefile=$cmd-actual.cred\n+catfile=$cmd-response.cred\n+rm -f $teefile\n+while read line;\n+do\n+\tif test -z \"$line\"; then\n+\t\tbreak;\n+\tfi\n+\techo \"$line\" >> $teefile\n+done\n+if test \"$cmd\" = \"get\"; then\n+\tcat $catfile\n+fi\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..92139c04c90\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,1134 @@\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+\" [--anonymous-allowed]\\n\"\n+\" [--auth=[:] [--auth-token=:]]*\\n\"\n+;\n+\n+/* Timeout, and initial timeout */\n+static unsigned int timeout;\n+static unsigned int init_timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+static void set_keep_alive(int sockfd)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+//////////////////////////////////////////////////////////////////\n+// The code in this section is used by \"worker\" instances to service\n+// a single connection from a client. The worker talks to the client\n+// on 0 and 1.\n+//////////////////////////////////////////////////////////////////\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\t/*\n+\t * Various errors while processing the request and/or the response.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_IO_ERROR = 1<<0,\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 1<<1,\n+\n+\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n+};\n+\n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tssize_t content_length;\n+};\n+\n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = -1 \\\n+\t}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n+static enum worker_result send_http_error(\n+\tint fd,\n+\tint http_code, const char *http_code_name,\n+\tint retry_after_seconds, struct string_list *response_headers,\n+\tenum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n+\t\t http_code, http_code_name);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf (&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_OK | WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* store common request headers separately */\n+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\treq->content_length = strtol(hv, &hp, 10);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->content_length >= 0)\n+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req, const char *user)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\tif (write(1, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n+\t\t\treq->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n+\t\t\t\treq->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n+\t\t\t\treq->content_type);\n+\tif (req->content_length >= 0)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n+\t\t\t\t(intmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(1);\n+\tclose(0);\n+\treturn !!res;\n+}\n+\n+enum auth_result {\n+\tAUTH_UNKNOWN = 0,\n+\tAUTH_DENY = 1,\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tconst char *scheme;\n+\tconst char *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+\n+static struct auth_module *get_auth_module(struct strbuf *scheme)\n+{\n+\tint i;\n+\tstruct auth_module *mod;\n+\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme->buf))\n+\t\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static void add_auth_module(struct auth_module *mod)\n+{\n+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\tauth_modules[auth_modules_nr++] = mod;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/* ask all auth modules to validate the request */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (!split[0] || !split[1]) continue;\n+\n+\t\t\t// trim trailing space ' '\n+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(split[0]);\n+\t\t\tif (mod) {\n+\n+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\tgoto done;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tif (result != AUTH_UNKNOWN)\n+\t\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (allow_anonymous)\n+\t\t\tbreak;\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n+\t}\n+\n+\tstrbuf_list_free(split);\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n+\tif (is_git_request(req))\n+\t\treturn do__git(req, user);\n+\n+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_OK | WR_HANGUP);\n+}\n+\n+static enum worker_result worker(void)\n+{\n+\tstruct req req = REQ__INIT;\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0);\n+\n+\twhile (1) {\n+\t\treq__release(&req);\n+\n+\t\talarm(init_timeout ? init_timeout : timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\n+\t\twr = dispatch(&req);\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(0);\n+\tclose(1);\n+\n+\treturn !!(wr & WR_IO_ERROR);\n+}\n+\n+//////////////////////////////////////////////////////////////////\n+// This section contains the listener and child-process management\n+// code used by the primary instance to accept incoming connections\n+// and dispatch them to async child process \"worker\" instances.\n+//////////////////////////////////////////////////////////////////\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+\t\t const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+} *firstborn;\n+\n+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tnewborn = xcalloc(1, sizeof(*newborn));\n+\tlive_children++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+/*\n+ * This gets called if the number of connections grows\n+ * past \"max_connections\".\n+ *\n+ * We kill the newest connection from a duplicate IP.\n+ */\n+static void kill_some_child(void)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+static void check_dead_children(void)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tconst char *dead = \"\";\n+\t\t\tif (status)\n+\t\t\t\tdead = \" (with error)\";\n+\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\tlive_children--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child();\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children();\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\tif (!reuseaddr)\n+\t\treturn 0;\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd);\n+\n+\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children();\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+//////////////////////////////////////////////////////////////////\n+// This section is executed by both the primary instance and all\n+// worker instances. So, yes, each child-process re-parses the\n+// command line argument and re-discovers how it should behave.\n+//////////////////////////////////////////////////////////////////\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\tstruct auth_module *mod = NULL;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n+\t\t\tinit_timeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n+\t\t\tallow_anonymous = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t// trim trailing ':'\n+\t\t\tif (p[1])\n+\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tif (get_auth_module(p[0])) {\n+\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tmod = xmalloc(sizeof(struct auth_module));\n+\t\t\tmod->scheme = xstrdup(p[0]->buf);\n+\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n+\t\t\tmod->tokens = xmalloc(sizeof(struct string_list));\n+\t\t\tstring_list_init_dup(mod->tokens);\n+\n+\t\t\tadd_auth_module(mod);\n+\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (!p[1]) {\n+\t\t\t\terror(\"missing token value '%s'\\n\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t// trim trailing ':'\n+\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(p[0]);\n+\t\t\tif (!mod) {\n+\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..43f1791a0fe\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,260 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+\n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$(pwd)\"/repo\n+\n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$(pwd)\"/pid-file.pid\n+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n+\t&& export CREDENTIAL_HELPER\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n+per_test_cleanup () {\n+\tstop_http_server &&\n+\trm -f OUT.* &&\n+\trm -f *.cred\n+}\n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server --allow-anonymous &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=bearer:secret-token &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=basic:$USERPASS64 &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tauthtype=basic\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tauthtype=basic\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=foobar:alg=test\\ widget=1 \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=foobar:SECRET-FOOBAR-VALUE &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=SECRET-FOOBAR-VALUE\n+\tauthtype=foobar\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=SECRET-FOOBAR-VALUE\n+\tauthtype=foobar\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=bearer:secret-token &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >erase-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-token\n+\tauthtype=bearer\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp erase-expected.cred erase-actual.cred\n+'\n+\n+test_done\n-- \ngitgitgadget\n"},{"id":"465678","messageId":"20221025022623.5449-1-mirth.hickford@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"git-credential.txt","fromName":"M Hickford","fromEmail":"mirth.hickford@gmail.com","sentAt":"2022-10-25T02:26:23Z","receivedAt":"2022-10-25T02:26:39Z","isPatch":false,"sender":{"key":"mirth.hickford@gmail.com","avatar":"https://avatars.githubusercontent.com/u/105314?v=4"},"body":"Reading git-credential.txt, I'm not quite clear:\n\n1. Are the new wwwauth[] and authtype attributes populated by Git and passed to helpers? Or vice versa?\n2. Should a storage helper store these attributes? If so, must the values be treated as confidential?\n"},{"id":"465731","messageId":"AS2PR03MB9815DEC9CE65268675A3A68BC0319@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"20221025022623.5449-1-mirth.hickford@gmail.com","subject":"Re: git-credential.txt","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-10-25T20:49:56Z","receivedAt":"2022-10-25T20:50:11Z","isPatch":false,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-10-24 19:26, M Hickford wrote:\n> Reading git-credential.txt, I'm not quite clear:\n> \n> 1. Are the new wwwauth[] and authtype attributes populated by Git and passed to helpers? Or vice versa?\n\nThe wwwauth[] attribute is from Git -> helper, and the authtype attribute is\nfrom helper -> Git. I can update the doc to make this more explicit.\n\n> 2. Should a storage helper store these attributes? If so, must the values be treated as confidential?\n\nGood question. A simple credential helper may wish to inspect these headers only\nto differentiate the different authentication schemes available (basic, bearer,\netc) and return a credential of the correct/available type (and include an\n`authtype` attribute in the response).\n\nHowever it's unlikely such a helper would need to store the wwwauth[] values\nas verbatim unless it can directly understand the parameters of the challenges.\nThe addition of this attribute is for credential helpers to gain more context\nabout the auth challenge from the remote.\n\nFor example, a helper may receive a bearer challenge including minimum required\nOAuth scopes and an authentication authority:\n\nwwwauth[]=Bearer authority=login.example.com/oauth scopes=\"code_rw userinfo_read\"\n\nUsing these extra parameters the helper can try and locate an existing stored\ncredential that satisfies the request.\n\nSuch an enlightened helper would need to query stored credentials looking for\nmatching metadata including the authority, and a bearer token that has at least\nthe minimum required scopes (but could have a superset).\n\nThanks,\nMatthew\n"},{"id":"465949","messageId":"8593dd49-4d95-ed4f-b414-8170efc138d4@github.com","threadId":"58425","inReplyTo":"f3f13ed8c8238f396163dd0e6a3d6c948c2b879b.1666372083.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-28T15:08:28Z","receivedAt":"2022-10-28T15:08:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/21/22 1:08 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n\n> @@ -1500,6 +1500,8 @@ else\n> \tendif\n> \tBASIC_CFLAGS += $(CURL_CFLAGS)\n> \n> +\tTEST_PROGRAMS_NEED_X += test-http-server\n> +\n> \tREMOTE_CURL_PRIMARY = git-remote-http$X\n> \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n> \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\n\nThis hunk is in the \"else\" block of \"ifdef NO_CURL\",\nso this makes sense for why TEST_PROGRAMS_NEED_X is\naugmented here, away from other instances.\n\n> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\n> index 787738e6fa3..45251695ce0 100644\n> --- a/contrib/buildsystems/CMakeLists.txt\n> +++ b/contrib/buildsystems/CMakeLists.txt\n> @@ -989,6 +989,19 @@ set(wrapper_scripts\n> set(wrapper_test_scripts\n> \ttest-fake-ssh test-tool)\n> \n> +if(CURL_FOUND)\n> + list(APPEND wrapper_test_scripts test-http-server)\n> +\n> + add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n> + target_link_libraries(test-http-server common-main)\n> +\n> + if(MSVC)\n> + set_target_properties(test-http-server\n> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n> + set_target_properties(test-http-server\n> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n> + endif()\n> +endif()\n\nAnd this file has the pattern of many \"if(CURL_FOUND)\"\nblocks with isolated purposes, so it makes sense to\nhave this be an isolated change instead of grouped with\na different case.\n\n> diff --git a/t/helper/.gitignore b/t/helper/.gitignore\n> index 8c2ddcce95f..1a94ab6eed5 100644\n> --- a/t/helper/.gitignore\n> +++ b/t/helper/.gitignore\n> @@ -1,2 +1,3 @@\n> /test-tool\n> /test-fake-ssh\n> +test-http-server\n\nShould this start with a \"/\" like the other entries?\n\n> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\n> new file mode 100755\n> index 00000000000..03e5e63dad6\n> --- /dev/null\n> +++ b/t/helper/test-credential-helper-replay.sh\n> @@ -0,0 +1,14 @@\n> +cmd=$1\n> +teefile=$cmd-actual.cred\n> +catfile=$cmd-response.cred\n> +rm -f $teefile\n> +while read line;\n> +do\n> +\tif test -z \"$line\"; then\n> +\t\tbreak;\n> +\tfi\n> +\techo \"$line\" >> $teefile\n> +done\n> +if test \"$cmd\" = \"get\"; then\n> +\tcat $catfile\n> +fi\n\nShould this be a helper method within another script, such\nas t/lib-credential.sh or t/lib-httpd.sh? The read over\nstdin will still work, as in this example:\n\nread_chunk() {\n\twhile read line; do\n\t\tcase \"$line\" in\n\t\t--) break ;;\n\t\t*) echo \"$line\" ;;\n\t\tesac\n\tdone\n}\n\n> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n\n> @@ -0,0 +1,1134 @@\n> +#include \"config.h\"\n> +#include \"run-command.h\"\n> +#include \"strbuf.h\"\n> +#include \"string-list.h\"\n> +#include \"trace2.h\"\n> +#include \"version.h\"\n> +#include \"dir.h\"\n> +#include \"date.h\"\n> +\n> +#define TR2_CAT \"test-http-server\"\n> +\n> +static const char *pid_file;\n> +static int verbose;\n> +static int reuseaddr;\n> +\n> +static const char test_http_auth_usage[] =\n> +\"http-server [--verbose]\\n\"\n> +\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n> +\" [--reuseaddr] [--pid-file=]\\n\"\n> +\" [--listen=]* [--port=]\\n\"\n> +\" [--anonymous-allowed]\\n\"\n> +\" [--auth=[:] [--auth-token=:]]*\\n\"\n> +;\n\nThese are a lot of options to implement all at once. They are probably\nsimple enough, but depending on the implementation and tests, it might\nbe helpful to split this patch into smaller ones that introduce these\noptions along with the tests that exercise each. That will help\nverify that they are being tested properly instead of needing to track\nback and forth across the patch for each one.\n\n> +\n> +/* Timeout, and initial timeout */\n> +static unsigned int timeout;\n> +static unsigned int init_timeout;\n> +\n> +static void logreport(const char *label, const char *err, va_list params)\n> +{\n> +\tstruct strbuf msg = STRBUF_INIT;\n> +\n> +\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n> +\tstrbuf_vaddf(&msg, err, params);\n> +\tstrbuf_addch(&msg, '\\n');\n> +\n> +\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n> +\tfflush(stderr);\n> +\n> +\tstrbuf_release(&msg);\n> +}\n> +\n> +__attribute__((format (printf, 1, 2)))\n> +static void logerror(const char *err, ...)\n> +{\n> +\tva_list params;\n> +\tva_start(params, err);\n> +\tlogreport(\"error\", err, params);\n> +\tva_end(params);\n> +}\n> +\n> +__attribute__((format (printf, 1, 2)))\n> +static void loginfo(const char *err, ...)\n> +{\n> +\tva_list params;\n> +\tif (!verbose)\n> +\t\treturn;\n> +\tva_start(params, err);\n> +\tlogreport(\"info\", err, params);\n> +\tva_end(params);\n> +}\n\nI wonder how much of this we need or is just a nice thing. I would\nerr on the side of making things as simple as possible, but being\nable to debug this test server may be important based on your\nexperience.\n\n> +static void set_keep_alive(int sockfd)\n> +{\n> +\tint ka = 1;\n> +\n> +\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n> +\t\tif (errno != ENOTSOCK)\n> +\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n> +\t\t\t\tstrerror(errno));\n> +\t}\n> +}\n> +\n> +//////////////////////////////////////////////////////////////////\n> +// The code in this section is used by \"worker\" instances to service\n> +// a single connection from a client. The worker talks to the client\n> +// on 0 and 1.\n> +//////////////////////////////////////////////////////////////////\n\nUse /* */ style comments. You can repeat the asterisks to get a\nsimilar visual block.\n\n> +\n> +enum worker_result {\n> +\t/*\n> +\t * Operation successful.\n> +\t * Caller *might* keep the socket open and allow keep-alive.\n> +\t */\n> +\tWR_OK = 0,\n> +\t/*\n> +\t * Various errors while processing the request and/or the response.\n> +\t * Close the socket and clean up.\n> +\t * Exit child-process with non-zero status.\n> +\t */\n> +\tWR_IO_ERROR = 1<<0,\n> +\t/*\n> +\t * Close the socket and clean up. Does not imply an error.\n> +\t */\n> +\tWR_HANGUP = 1<<1,\n\nnit: add a whitespace line between an item and the next\nitem's comment.\n\n> +\n> +\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n> +};\n\n(I read, but have no comments on the http-server boilerplate.)\n\n> +\n> +enum auth_result {\n> +\tAUTH_UNKNOWN = 0,\n> +\tAUTH_DENY = 1,\n> +\tAUTH_ALLOW = 2,\n> +};\n> +\n> +struct auth_module {\n> +\tconst char *scheme;\n> +\tconst char *challenge_params;\n\nLater, I notice that you set challenge_params using an\nxstrdup() so this shouldn't be const and you should\nfree it in any freeing code.\n\n> +\tstruct string_list *tokens;\n> +};\n> +\n> +static int allow_anonymous;\n> +static struct auth_module **auth_modules = NULL;\n> +static size_t auth_modules_nr = 0;\n> +static size_t auth_modules_alloc = 0;\n\nSo, we are setting up a number of potential auth modules,\neach of which has a scheme to match a request to the module,\nand a list of tokens that would be considered worthy of the\nAUTH_ALLOW result. Otherwise, if the scheme matches but no\ntoken matches, we get AUTH_DENY. Finally, if no scheme matches\nwe get AUTH_UNKNOWN.\n\nThis concept might be worth a comment here around the data\nstructures before we get into how that is implemented.\n\n> +static struct auth_module *get_auth_module(struct strbuf *scheme)\n> +{\n> +\tint i;\n> +\tstruct auth_module *mod;\n> +\tfor (i = 0; i < auth_modules_nr; i++) {\n> +\t\tmod = auth_modules[i];\n> +\t\tif (!strcasecmp(mod->scheme, scheme->buf))\n> +\t\t\treturn mod;\n> +\t}\n> +\n> +\treturn NULL;\n> +}\n\nMatching the input scheme against the list of modules.\n\nOnly complaint: there is no reason that 'scheme' needs t\nbe a strbuf, but could be a 'const char *' here.\n\n> +static void add_auth_module(struct auth_module *mod)\n> +{\n> +\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n> +\tauth_modules[auth_modules_nr++] = mod;\n> +}\n\nnit: this could be located earlier, next to the list\ndefinition, or delayed until it is needed. That would\nallow get_auth_module() to be closer to its first use.\n\n> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n> +{\n> +\tenum auth_result result = AUTH_UNKNOWN;\n> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n> +\tstruct auth_module *mod;\n> +\n> +\tstruct string_list_item *hdr;\n> +\tstruct string_list_item *token;\n> +\tconst char *v;\n> +\tstruct strbuf **split = NULL;\n> +\tint i;\n> +\tchar *challenge;\n> +\n> +\t/* ask all auth modules to validate the request */\n> +\tfor_each_string_list_item(hdr, &req->header_list) {\n> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n> +\t\t\tif (!split[0] || !split[1]) continue;\n\nFor each valid request header...\n\n> +\t\t\t// trim trailing space ' '\n> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n> +\n> +\t\t\tmod = get_auth_module(split[0]);\n> +\t\t\tif (mod) {\n\n...get an appropriate module, if it exists...\n\n> +\n> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n> +\t\t\t\t\t\tgoto done;\n> +\t\t\t\t\t}\n> +\t\t\t\t}\n> +\n> +\t\t\t\tif (result != AUTH_UNKNOWN)\n> +\t\t\t\t\tgoto done;\n\n...and report if we find a valid token.\n\nHere, it seems I was wrong in my expectation of AUTH_DENY:\nif a matching module exists but no token exists in that\nmodule, then we keep searching other modules. \n\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +done:\n> +\tswitch (result) {\n> +\tcase AUTH_ALLOW:\n> +\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n> +\t\t*user = \"VALID_TEST_USER\";\n> +\t\t*wr = WR_OK;\n> +\t\tbreak;\n> +\n> +\tcase AUTH_DENY:\n> +\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n> +\t\t/* fall-through */\n\nI'm not sure that I see a case where this is possible. Maybe\nwe should have a 'result = AUTH_DENY' at the start of the\n\"if (mod)\" block, followed by a 'goto done' in all cases\ninstead of \"if (result != AUTH_UNKNOWN)\"?\n\n> +\tcase AUTH_UNKNOWN:\n> +\t\tif (allow_anonymous)\n> +\t\t\tbreak;\n\nIf we do not require auth, then we want to continue if there\nis no matching authentication.\n\n> +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n> +\t\t\tmod = auth_modules[i];\n> +\t\t\tif (mod->challenge_params)\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n> +\t\t\t\t\t\t mod->scheme,\n> +\t\t\t\t\t\t mod->challenge_params);\n> +\t\t\telse\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n> +\t\t\t\t\t\t mod->scheme);\n> +\t\t\tstring_list_append(&hdrs, challenge);\n> +\t\t}\n> +\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n\nHowever, here is the critical piece about how servers will\nstart to act with the new WWW-Authenticate header usage in\nthe Git credential helper interface. This will be critical\nin the testing for Git to retry the credential helper while\npassing these authentications schemes from the installed\nmodules.\n\n> +\t}\n> +\n> +\tstrbuf_list_free(split);\n> +\tstring_list_clear(&hdrs, 0);\n> +\n> +\treturn result == AUTH_ALLOW ||\n> +\t (result == AUTH_UNKNOWN && allow_anonymous);\n\nDid it work? Or did it not need to work? I'm interested to\ninvestigate the case that the client sent an authentication\nheader that matches a module but doesn't match any tokens,\nbut we allow anonymous access, anyway. Is that a 400? Or\nis that a 401?\n\n> +static enum worker_result dispatch(struct req *req)\n> +{\n> +\tenum worker_result wr = WR_OK;\n> +\tconst char *user = NULL;\n> +\n> +\tif (!is_authed(req, &user, &wr))\n> +\t\treturn wr;\n\nIf we are not authed, send the 401 response.\n\n> +\tif (is_git_request(req))\n> +\t\treturn do__git(req, user);\n\nIf we are authed, then pass through to the Git response.\n\n> +\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> +\t\t\t WR_OK | WR_HANGUP);\n\nIf the Git request fails, we don't care. This is a test.\nJust pass a 500-level error and the client will barf,\nletting us know that something went wrong.\n\n> +static void kill_some_child(void)\n\n> +static void check_dead_children(void)\n\nThese technically sound methods have unfortunate names.\nUsing something like \"connection\" over \"child\" might\nalleviate some of the horror. (I initially wanted to\nsuggest \"subprocess\" but you compare live_children to\nmax_connections in the next method, so connection seemed\nappropriate.)\n\n> +static struct strvec cld_argv = STRVEC_INIT;\n> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n> +{\n> +\tstruct child_process cld = CHILD_PROCESS_INIT;\n> +\n> +\tif (max_connections && live_children >= max_connections) {\n> +\t\tkill_some_child();\n> +\t\tsleep(1); /* give it some time to die */\n> +\t\tcheck_dead_children();\n> +\t\tif (live_children >= max_connections) {\n> +\t\t\tclose(incoming);\n> +\t\t\tlogerror(\"Too many children, dropping connection\");\n> +\t\t\treturn;\n> +\t\t}\n> +\t}\n\nDo we anticipate exercising concurrent requests in our\ntests? Perhaps it's not worth putting a cap on the\nconnection count so we can keep the test helpers simple.\n\n> +\tif (addr->sa_family == AF_INET) {\n> +\t\tchar buf[128] = \"\";\n> +\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n> +\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n> +\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n> +\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n> +\t\t\t\t ntohs(sin_addr->sin_port));\n> +#ifndef NO_IPV6\n> +\t} else if (addr->sa_family == AF_INET6) {\n> +\t\tchar buf[128] = \"\";\n> +\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n> +\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n> +\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n> +\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n> +\t\t\t\t ntohs(sin6_addr->sin6_port));\n> +#endif\n> +\t}\n> +\n> +\tstrvec_pushv(&cld.args, cld_argv.v);\n> +\tcld.in = incoming;\n> +\tcld.out = dup(incoming);\n> +\n> +\tif (cld.out < 0)\n> +\t\tlogerror(\"could not dup() `incoming`\");\n> +\telse if (start_command(&cld))\n> +\t\tlogerror(\"unable to fork\");\n> +\telse\n> +\t\tadd_child(&cld, addr, addrlen);\n> +}\n> +\n\nI scanned the socket creation code, but my eyes were\nglazing over. I'm definitely in the camp of \"if it works,\nthat's enough for our tests.\" If we start to rely on this\ntest harness in more places, we can improve any shortcomings\nas they arise.\n\n> +//////////////////////////////////////////////////////////////////\n> +// This section is executed by both the primary instance and all\n> +// worker instances. So, yes, each child-process re-parses the\n> +// command line argument and re-discovers how it should behave.\n> +//////////////////////////////////////////////////////////////////\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tint listen_port = 0;\n> +\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n> +\tint worker_mode = 0;\n> +\tint i;\n> +\tstruct auth_module *mod = NULL;\n> +\n> +\ttrace2_cmd_name(\"test-http-server\");\n> +\tsetup_git_directory_gently(NULL);\n> +\n> +\tfor (i = 1; i < argc; i++) {\n> +\t\tconst char *arg = argv[i];\n> +\t\tconst char *v;\n> +\n> +\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n> +\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n> +\t\t\tchar *end;\n> +\t\t\tunsigned long n;\n> +\t\t\tn = strtoul(v, &end, 0);\n> +\t\t\tif (*v && !*end) {\n> +\t\t\t\tlisten_port = n;\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t}\n> +\t\tif (!strcmp(arg, \"--worker\")) {\n> +\t\t\tworker_mode = 1;\n> +\t\t\ttrace2_cmd_mode(\"worker\");\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (!strcmp(arg, \"--verbose\")) {\n> +\t\t\tverbose = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n> +\t\t\ttimeout = atoi(v);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n> +\t\t\tinit_timeout = atoi(v);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n> +\t\t\tmax_connections = atoi(v);\n> +\t\t\tif (max_connections < 0)\n> +\t\t\t\tmax_connections = 0; /* unlimited */\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n> +\t\t\treuseaddr = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n> +\t\t\tpid_file = v;\n> +\t\t\tcontinue;\n> +\t\t}\n\nok, most of these arguments are actually about the per-connection\nsubprocesses.\n\n> +\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n> +\t\t\tallow_anonymous = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n\nHere is how we choose to allo anonymous access.\n\n> +\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n> +\n> +\t\t\tif (!p[0]) {\n> +\t\t\t\terror(\"invalid argument '%s'\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\t// trim trailing ':'\n> +\t\t\tif (p[1])\n> +\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n> +\n> +\t\t\tif (get_auth_module(p[0])) {\n> +\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tmod = xmalloc(sizeof(struct auth_module));\n> +\t\t\tmod->scheme = xstrdup(p[0]->buf);\n> +\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n\nHere, you xstrdup() into a 'const char *', but you are really\npassing ownership so it shouldn't be conts.\n\n> +\t\t\tmod->tokens = xmalloc(sizeof(struct string_list));\n\nnit: this could also be \"CALLOC_ARRAY(mod->tokens, 1);\"\n\n> +\t\t\tstring_list_init_dup(mod->tokens);\n> +\n> +\t\t\tadd_auth_module(mod);\n> +\n> +\t\t\tstrbuf_list_free(p);\n> +\t\t\tcontinue;\n\nOk, we gain the auth schemes from the command line.\n\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n> +\t\t\tif (!p[0]) {\n> +\t\t\t\terror(\"invalid argument '%s'\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tif (!p[1]) {\n> +\t\t\t\terror(\"missing token value '%s'\\n\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\t// trim trailing ':'\n\nUse /* */ (Aside: I'm surprised we don't have a build option in\nDEVELOPER=1 that catches the use of these comments.)\n\n> +\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n> +\n> +\t\t\tmod = get_auth_module(p[0]);\n> +\t\t\tif (!mod) {\n> +\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n> +\t\t\tstrbuf_list_free(p);\n> +\t\t\tcontinue;\n> +\t\t}\n\nAnd the token lists. It is important that the scheme is added\nbefore any token is added.\n\n> +\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n> +\t\tusage(test_http_auth_usage);\n> +\t}\n> +\n> +\t/* avoid splitting a message in the middle */\n> +\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n> +\n> +\tif (listen_port == 0)\n> +\t\tlisten_port = DEFAULT_GIT_PORT;\n> +\n> +\t/*\n> +\t * If no --listen= args are given, the setup_named_sock()\n> +\t * code will use receive a NULL address and set INADDR_ANY.\n> +\t * This exposes both internal and external interfaces on the\n> +\t * port.\n> +\t *\n> +\t * Disallow that and default to the internal-use-only loopback\n> +\t * address.\n> +\t */\n> +\tif (!listen_addr.nr)\n> +\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n> +\n> +\t/*\n> +\t * worker_mode is set in our own child process instances\n> +\t * (that are bound to a connected socket from a client).\n> +\t */\n> +\tif (worker_mode)\n> +\t\treturn worker();\n> +\n> +\t/*\n> +\t * `cld_argv` is a bit of a clever hack. The top-level instance\n> +\t * of test-http-server does the normal bind/listen/accept stuff.\n> +\t * For each incoming socket, the top-level process spawns\n> +\t * a child instance of test-http-server *WITH* the additional\n> +\t * `--worker` argument. This causes the child to set `worker_mode`\n> +\t * and immediately call `worker()` using the connected socket (and\n> +\t * without the usual need for fork() or threads).\n> +\t *\n> +\t * The magic here is made possible because `cld_argv` is static\n> +\t * and handle() (called by service_loop()) knows about it.\n> +\t */\n> +\tstrvec_push(&cld_argv, argv[0]);\n> +\tstrvec_push(&cld_argv, \"--worker\");\n> +\tfor (i = 1; i < argc; ++i)\n> +\t\tstrvec_push(&cld_argv, argv[i]);\n> +\n> +\t/*\n> +\t * Setup primary instance to listen for connections.\n> +\t */\n> +\treturn serve(&listen_addr, listen_port);\n> +}\n\nAnd complete the thing with some boilerplate.\n\nThis was a lot to read, and the interesting bits are all mixed in\nwith the http server code, which is less interesting to what we\nare trying to accomplish. It would be beneficial to split this\ninto one or two patches before we actually introduce the tests.\n\nThe most important thing that I think would be helpful is to\nisolate all the authentication behavior into its own patch so\nwe can see how those connections from the command-line arguments\naffect the behavior of the server responses.\n\nI think ideally we would have the following split:\n\n 1. All server boilerblate. All requests 500 not-implemented.\n\n 2. Add Git fall-through with no authentication. Add the tests\n that are intended to allow anonymous auth.\n\n 3. Add authentication data structures read from command-line,\n but not processed at all in the logic.\n\n 4. Act on the authentication data structures to alter the\n requests. Add the tests that use these authentication\n schemes.\n\nI could easily see a case for combining 1&2 as well as 3&4,\nfor slightly larger but more completely-testable changes at\nevery step.\n\nFrom what I read, I don't think there is much to change in\nthe end result of the code, but it definitely was hard to read\nthe important things when surrounded by many lines of\nboilerplate.\n\n> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n\nI'm going to pause here and come back to the test script in\na separate reply.\n\nThanks,\n-Stolee\n"},{"id":"465965","messageId":"08de9e0e-af08-2bc3-6693-7bf2b798f20a@jeffhostetler.com","threadId":"58425","inReplyTo":"0838d992744a4b06523be6df0edb046ebba033ee.1666372083.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-28T18:22:15Z","receivedAt":"2022-10-28T18:22:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/21/22 1:07 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Add the value of the WWW-Authenticate response header to credential\n> requests. Credential helpers that understand and support HTTP\n> authentication and authorization can use this standard header (RFC 2616\n> Section 14.47 [1]) to generate valid credentials.\n> \n> WWW-Authenticate headers can contain information pertaining to the\n> authority, authentication mechanism, or extra parameters/scopes that are\n> required.\n> \n> The current I/O format for credential helpers only allows for unique\n> names for properties/attributes, so in order to transmit multiple header\n> values (with a specific order) we introduce a new convention whereby a\n> C-style array syntax is used in the property name to denote multiple\n> ordered values for the same property.\n> \n> In this case we send multiple `wwwauth[n]` properties where `n` is a\n > zero-indexed number, reflecting the order the WWW-Authenticate headers\n > appeared in the HTTP response.\n\nHere (and maybe in the cover letter) you mention `wwwauth[n]` and `n`...\n> +`wwwauth[]`::\n> +\n> +\tWhen an HTTP response is received that includes one or more\n> +\t'WWW-Authenticate' authentication headers, these can be passed to Git\n> +\t(and subsequent credential helpers) with these attributes.\n> +\tEach 'WWW-Authenticate' header value should be passed as a separate\n> +\tattribute 'wwwauth[]' where the order of the attributes is the same\n> +\tas they appear in the HTTP response.\n\n...but here you don't include the `n`.\n\n[...]\n> +static void credential_write_strvec(FILE *fp, const char *key,\n> +\t\t\t\t const struct strvec *vec)\n> +{\n> +\tint i = 0;\n> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n\n...nor here.\n\nJeff\n"},{"id":"465966","messageId":"d61d8881-ce58-de02-2c3b-e3cc812e316a@jeffhostetler.com","threadId":"58425","inReplyTo":"8593dd49-4d95-ed4f-b414-8170efc138d4@github.com","subject":"Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-28T19:14:00Z","receivedAt":"2022-10-28T19:14:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/28/22 11:08 AM, Derrick Stolee wrote:\n> }\n> \n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> \n>> @@ -0,0 +1,1134 @@\n>> +#include \"config.h\"\n>> +#include \"run-command.h\"\n>> +#include \"strbuf.h\"\n>> +#include \"string-list.h\"\n>> +#include \"trace2.h\"\n>> +#include \"version.h\"\n>> +#include \"dir.h\"\n>> +#include \"date.h\"\n>> +\n>> +#define TR2_CAT \"test-http-server\"\n>> +\n>> +static const char *pid_file;\n>> +static int verbose;\n>> +static int reuseaddr;\n>> +\n>> +static const char test_http_auth_usage[] =\n>> +\"http-server [--verbose]\\n\"\n>> +\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n>> +\" [--reuseaddr] [--pid-file=]\\n\"\n>> +\" [--listen=]* [--port=]\\n\"\n>> +\" [--anonymous-allowed]\\n\"\n>> +\" [--auth=[:] [--auth-token=:]]*\\n\"\n>> +;\n> \n> These are a lot of options to implement all at once. They are probably\n> simple enough, but depending on the implementation and tests, it might\n> be helpful to split this patch into smaller ones that introduce these\n> options along with the tests that exercise each. That will help\n> verify that they are being tested properly instead of needing to track\n> back and forth across the patch for each one.\n\nhow many of these options were inherited from test-gvfs-protocol or\nfrom upstream git-daemon? If most came from git-daemon, it's probably\neasier to see that this was a cut-n-paste from it if it comes over in\none commit, since all of the OPT_ processing, usage(), and static global\nstate vars will come over together I would think -- rather than to build\nup the arg parsing bit by bit. More on this in a minute...\n\n\n>> +\n>> +/* Timeout, and initial timeout */\n>> +static unsigned int timeout;\n>> +static unsigned int init_timeout;\n>> +\n>> +static void logreport(const char *label, const char *err, va_list params)\n>> +{\n>> +\tstruct strbuf msg = STRBUF_INIT;\n>> +\n>> +\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n>> +\tstrbuf_vaddf(&msg, err, params);\n>> +\tstrbuf_addch(&msg, '\\n');\n>> +\n>> +\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n>> +\tfflush(stderr);\n>> +\n>> +\tstrbuf_release(&msg);\n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void logerror(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"error\", err, params);\n>> +\tva_end(params);\n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void loginfo(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tif (!verbose)\n>> +\t\treturn;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"info\", err, params);\n>> +\tva_end(params);\n>> +}\n\n...Maybe it would be easier to see/diff this large new test server\nif we copied `daemon.c` into this source file in 1 commit and then\nconverted it to what you have now in 1 commit -- so that only new\ncode shows up here. For example, all of the above logreport, logerror,\nand loginfo routines would show up as new in the copy commit, but not\nin the edit commit. However, that may lead to too much noise when\nyou actually get into the meat of the auth changes, maybe.\n\n\n> I wonder how much of this we need or is just a nice thing. I would\n> err on the side of making things as simple as possible, but being\n> able to debug this test server may be important based on your\n> experience.\n\ni'd vote to keep it.\n\n[...]\n>> +static void kill_some_child(void)\n> \n>> +static void check_dead_children(void)\n> \n> These technically sound methods have unfortunate names.\n> Using something like \"connection\" over \"child\" might\n> alleviate some of the horror. (I initially wanted to\n> suggest \"subprocess\" but you compare live_children to\n> max_connections in the next method, so connection seemed\n> appropriate.)\n\nThese names were inherited from `daemon.c` IIRC. I wouldn't change\nthem since it'll just introduce noise when diffing. Especially,\nif we do the copy commit first.\n\n\n[...]\n>> +static struct strvec cld_argv = STRVEC_INIT;\n>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>> +{\n>> +\tstruct child_process cld = CHILD_PROCESS_INIT;\n>> +\n>> +\tif (max_connections && live_children >= max_connections) {\n>> +\t\tkill_some_child();\n>> +\t\tsleep(1); /* give it some time to die */\n>> +\t\tcheck_dead_children();\n>> +\t\tif (live_children >= max_connections) {\n>> +\t\t\tclose(incoming);\n>> +\t\t\tlogerror(\"Too many children, dropping connection\");\n>> +\t\t\treturn;\n>> +\t\t}\n>> +\t}\n> \n> Do we anticipate exercising concurrent requests in our\n> tests? Perhaps it's not worth putting a cap on the\n> connection count so we can keep the test helpers simple.\n\nagain, this code was inherited from `daemon.c`, so we could leave it.\n\n[...]\n>> +\t\t\tmod = xmalloc(sizeof(struct auth_module));\n>> +\t\t\tmod->scheme = xstrdup(p[0]->buf);\n>> +\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n> \n> Here, you xstrdup() into a 'const char *', but you are really\n> passing ownership so it shouldn't be conts.\n\nThere is a strbuf_detach() that will let you steal the buffer from the\nstrbuf if that would help.\n\n\n[...]\n> This was a lot to read, and the interesting bits are all mixed in\n> with the http server code, which is less interesting to what we\n> are trying to accomplish. It would be beneficial to split this\n> into one or two patches before we actually introduce the tests.\n\nagreed. it is big, but it does make sense. perhaps doing the\ncopy daemon.c commit and then see how this commit diffs from it\nwould make it more manageable. (not sure, but worth a try.)\n\n[...]\n> From what I read, I don't think there is much to change in\n> the end result of the code, but it definitely was hard to read\n> the important things when surrounded by many lines of\n> boilerplate.\n\nagreed. i think the end result is good.\n\nThanks\nJeff\n\n\n"},{"id":"466267","messageId":"AS2PR03MB9815C34D7DB3C96CF1D0CC54C0369@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"08de9e0e-af08-2bc3-6693-7bf2b798f20a@jeffhostetler.com","subject":"Re: [PATCH v2 2/6] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-11-01T23:07:55Z","receivedAt":"2022-11-01T23:10:52Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-10-28 11:22, Jeff Hostetler wrote:\n> On 10/21/22 1:07 PM, Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Add the value of the WWW-Authenticate response header to credential\n>> requests. Credential helpers that understand and support HTTP\n>> authentication and authorization can use this standard header (RFC 2616\n>> Section 14.47 [1]) to generate valid credentials.\n>>\n>> WWW-Authenticate headers can contain information pertaining to the\n>> authority, authentication mechanism, or extra parameters/scopes that are\n>> required.\n>>\n>> The current I/O format for credential helpers only allows for unique\n>> names for properties/attributes, so in order to transmit multiple header\n>> values (with a specific order) we introduce a new convention whereby a\n>> C-style array syntax is used in the property name to denote multiple\n>> ordered values for the same property.\n>>\n>> In this case we send multiple `wwwauth[n]` properties where `n` is a\n>> zero-indexed number, reflecting the order the WWW-Authenticate headers\n>> appeared in the HTTP response.\n> \n> Here (and maybe in the cover letter) you mention `wwwauth[n]` and `n`...\n>> +`wwwauth[]`::\n>> +\n>> + When an HTTP response is received that includes one or more\n>> + 'WWW-Authenticate' authentication headers, these can be passed to Git\n>> + (and subsequent credential helpers) with these attributes.\n>> + Each 'WWW-Authenticate' header value should be passed as a separate\n>> + attribute 'wwwauth[]' where the order of the attributes is the same\n>> + as they appear in the HTTP response.\n> \n> ...but here you don't include the `n`.\n> \n> [...]\n>> +static void credential_write_strvec(FILE *fp, const char *key,\n>> + const struct strvec *vec)\n>> +{\n>> + int i = 0;\n>> + const char *full_key = xstrfmt(\"%s[]\", key);\n> \n> ...nor here.\n> \nAh. This is an oversight in my v2 rebasing! Will fix in v3.\n\nThanks,\nMatthew\n"},{"id":"466268","messageId":"AS2PR03MB981549CCF945BF26DD212BDBC0369@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"d61d8881-ce58-de02-2c3b-e3cc812e316a@jeffhostetler.com","subject":"Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-11-01T23:14:36Z","receivedAt":"2022-11-01T23:14:57Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-10-28 12:14, Jeff Hostetler wrote:\n>\n>\n> On 10/28/22 11:08 AM, Derrick Stolee wrote:\n>> }\n>>\n>>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n>>\n>>> @@ -0,0 +1,1134 @@\n>>> +#include \"config.h\"\n>>> +#include \"run-command.h\"\n>>> +#include \"strbuf.h\"\n>>> +#include \"string-list.h\"\n>>> +#include \"trace2.h\"\n>>> +#include \"version.h\"\n>>> +#include \"dir.h\"\n>>> +#include \"date.h\"\n>>> +\n>>> +#define TR2_CAT \"test-http-server\"\n>>> +\n>>> +static const char *pid_file;\n>>> +static int verbose;\n>>> +static int reuseaddr;\n>>> +\n>>> +static const char test_http_auth_usage[] =\n>>> +\"http-server [--verbose]\\n\"\n>>> +\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n>>> +\" [--reuseaddr] [--pid-file=]\\n\"\n>>> +\" [--listen=]* [--port=]\\n\"\n>>> +\" [--anonymous-allowed]\\n\"\n>>> +\" [--auth=[:] [--auth-token=:]]*\\n\"\n>>> +;\n>>\n>> These are a lot of options to implement all at once. They are probably\n>> simple enough, but depending on the implementation and tests, it might\n>> be helpful to split this patch into smaller ones that introduce these\n>> options along with the tests that exercise each. That will help\n>> verify that they are being tested properly instead of needing to track\n>> back and forth across the patch for each one.\n>\n> how many of these options were inherited from test-gvfs-protocol or\n> from upstream git-daemon? If most came from git-daemon, it's probably\n> easier to see that this was a cut-n-paste from it if it comes over in\n> one commit, since all of the OPT_ processing, usage(), and static global\n> state vars will come over together I would think -- rather than to build\n> up the arg parsing bit by bit. More on this in a minute...\n>\n\nOnly --anonymous-allowed, --auth and --auth-token are added over git-daemon.\n\n>\n>>> +\n>>> +/* Timeout, and initial timeout */\n>>> +static unsigned int timeout;\n>>> +static unsigned int init_timeout;\n>>> +\n>>> +static void logreport(const char *label, const char *err, va_list params)\n>>> +{\n>>> + struct strbuf msg = STRBUF_INIT;\n>>> +\n>>> + strbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n>>> + strbuf_vaddf(&msg, err, params);\n>>> + strbuf_addch(&msg, '\\n');\n>>> +\n>>> + fwrite(msg.buf, sizeof(char), msg.len, stderr);\n>>> + fflush(stderr);\n>>> +\n>>> + strbuf_release(&msg);\n>>> +}\n>>> +\n>>> +__attribute__((format (printf, 1, 2)))\n>>> +static void logerror(const char *err, ...)\n>>> +{\n>>> + va_list params;\n>>> + va_start(params, err);\n>>> + logreport(\"error\", err, params);\n>>> + va_end(params);\n>>> +}\n>>> +\n>>> +__attribute__((format (printf, 1, 2)))\n>>> +static void loginfo(const char *err, ...)\n>>> +{\n>>> + va_list params;\n>>> + if (!verbose)\n>>> + return;\n>>> + va_start(params, err);\n>>> + logreport(\"info\", err, params);\n>>> + va_end(params);\n>>> +}\n>\n> ...Maybe it would be easier to see/diff this large new test server\n> if we copied `daemon.c` into this source file in 1 commit and then\n> converted it to what you have now in 1 commit -- so that only new\n> code shows up here. For example, all of the above logreport, logerror,\n> and loginfo routines would show up as new in the copy commit, but not\n> in the edit commit. However, that may lead to too much noise when\n> you actually get into the meat of the auth changes, maybe.\n\nI take from git-daemon and the test-gvfs-protocol helper from microsoft/git\nfork, but then also delete lots of not required pieces too just as much as\nI have added. Copying git-daemon.c, to then delete, and then add feels like\nlots of noise.\n\n>> I wonder how much of this we need or is just a nice thing. I would\n>> err on the side of making things as simple as possible, but being\n>> able to debug this test server may be important based on your\n>> experience.\n>\n> i'd vote to keep it.\n>\n> [...]\n>>> +static void kill_some_child(void)\n>>\n>>> +static void check_dead_children(void)\n>>\n>> These technically sound methods have unfortunate names.\n>> Using something like \"connection\" over \"child\" might\n>> alleviate some of the horror. (I initially wanted to\n>> suggest \"subprocess\" but you compare live_children to\n>> max_connections in the next method, so connection seemed\n>> appropriate.)\n>\n> These names were inherited from `daemon.c` IIRC. I wouldn't change\n> them since it'll just introduce noise when diffing. Especially,\n> if we do the copy commit first.\n\nIndeed. These functions are untouched from daemon.c. I do plan to split\nthis mega-patch up however in to a single 'add the boilerplate' based on\ngit-daemon patch, then add the extra pieces like HTTP request parsing and\nthe auth pieces in a v3.\n\n> [...]\n>>> +static struct strvec cld_argv = STRVEC_INIT;\n>>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>>> +{\n>>> + struct child_process cld = CHILD_PROCESS_INIT;\n>>> +\n>>> + if (max_connections && live_children >= max_connections) {\n>>> + kill_some_child();\n>>> + sleep(1); /* give it some time to die */\n>>> + check_dead_children();\n>>> + if (live_children >= max_connections) {\n>>> + close(incoming);\n>>> + logerror(\"Too many children, dropping connection\");\n>>> + return;\n>>> + }\n>>> + }\n>>\n>> Do we anticipate exercising concurrent requests in our\n>> tests? Perhaps it's not worth putting a cap on the\n>> connection count so we can keep the test helpers simple.\n>\n> again, this code was inherited from `daemon.c`, so we could leave it.\n>\n> [...]\n>>> + mod = xmalloc(sizeof(struct auth_module));\n>>> + mod->scheme = xstrdup(p[0]->buf);\n>>> + mod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n>>\n>> Here, you xstrdup() into a 'const char *', but you are really\n>> passing ownership so it shouldn't be conts.\n>\n> There is a strbuf_detach() that will let you steal the buffer from the\n> strbuf if that would help.\n\nWill update in v3 to drop the const.\n\n> [...]\n>> This was a lot to read, and the interesting bits are all mixed in\n>> with the http server code, which is less interesting to what we\n>> are trying to accomplish. It would be beneficial to split this\n>> into one or two patches before we actually introduce the tests.\n>\n> agreed. it is big, but it does make sense. perhaps doing the\n> copy daemon.c commit and then see how this commit diffs from it\n> would make it more manageable. (not sure, but worth a try.)\n>\n> [...]\n>> From what I read, I don't think there is much to change in\n>> the end result of the code, but it definitely was hard to read\n>> the important things when surrounded by many lines of\n>> boilerplate.\n>\n> agreed. i think the end result is good.\n>\n> Thanks\n> Jeff\n>\n>\n"},{"id":"466270","messageId":"AS2PR03MB9815B1AA9C780D650BD88FA2C0369@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"8593dd49-4d95-ed4f-b414-8170efc138d4@github.com","subject":"Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-11-01T23:59:12Z","receivedAt":"2022-11-01T23:59:27Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-10-28 08:08, Derrick Stolee wrote:\n> On 10/21/22 1:08 PM, Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n> \n>> @@ -1500,6 +1500,8 @@ else\n>> \tendif\n>> \tBASIC_CFLAGS += $(CURL_CFLAGS)\n>> \n>> +\tTEST_PROGRAMS_NEED_X += test-http-server\n>> +\n>> \tREMOTE_CURL_PRIMARY = git-remote-http$X\n>> \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n>> \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\n> \n> This hunk is in the \"else\" block of \"ifdef NO_CURL\",\n> so this makes sense for why TEST_PROGRAMS_NEED_X is\n> augmented here, away from other instances.\n> \n>> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\n>> index 787738e6fa3..45251695ce0 100644\n>> --- a/contrib/buildsystems/CMakeLists.txt\n>> +++ b/contrib/buildsystems/CMakeLists.txt\n>> @@ -989,6 +989,19 @@ set(wrapper_scripts\n>> set(wrapper_test_scripts\n>> \ttest-fake-ssh test-tool)\n>> \n>> +if(CURL_FOUND)\n>> + list(APPEND wrapper_test_scripts test-http-server)\n>> +\n>> + add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n>> + target_link_libraries(test-http-server common-main)\n>> +\n>> + if(MSVC)\n>> + set_target_properties(test-http-server\n>> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n>> + set_target_properties(test-http-server\n>> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n>> + endif()\n>> +endif()\n> \n> And this file has the pattern of many \"if(CURL_FOUND)\"\n> blocks with isolated purposes, so it makes sense to\n> have this be an isolated change instead of grouped with\n> a different case.\n> \n>> diff --git a/t/helper/.gitignore b/t/helper/.gitignore\n>> index 8c2ddcce95f..1a94ab6eed5 100644\n>> --- a/t/helper/.gitignore\n>> +++ b/t/helper/.gitignore\n>> @@ -1,2 +1,3 @@\n>> /test-tool\n>> /test-fake-ssh\n>> +test-http-server\n> \n> Should this start with a \"/\" like the other entries?\n\nThat it probably should! Will update.\n\n>> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\n>> new file mode 100755\n>> index 00000000000..03e5e63dad6\n>> --- /dev/null\n>> +++ b/t/helper/test-credential-helper-replay.sh\n>> @@ -0,0 +1,14 @@\n>> +cmd=$1\n>> +teefile=$cmd-actual.cred\n>> +catfile=$cmd-response.cred\n>> +rm -f $teefile\n>> +while read line;\n>> +do\n>> +\tif test -z \"$line\"; then\n>> +\t\tbreak;\n>> +\tfi\n>> +\techo \"$line\" >> $teefile\n>> +done\n>> +if test \"$cmd\" = \"get\"; then\n>> +\tcat $catfile\n>> +fi\n> \n> Should this be a helper method within another script, such\n> as t/lib-credential.sh or t/lib-httpd.sh? The read over\n> stdin will still work, as in this example:\n> \n> read_chunk() {\n> \twhile read line; do\n> \t\tcase \"$line\" in\n> \t\t--) break ;;\n> \t\t*) echo \"$line\" ;;\n> \t\tesac\n> \tdone\n> }\n\nThis script file is used as a credential helper that is invoked by Git.\nWe specify that Git should use this credential helper in the tests using\nthe -c option:\n\n CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n\t && export CREDENTIAL_HELPER\n..\n git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n\n\nWould extracting a read_chunk() function to one of the lib-* test scripts\nbe worth it given we already need another entry script anyway?\n\nWhat other scripts would be calling read_chunk()?\n\n\n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> \n>> @@ -0,0 +1,1134 @@\n>> +#include \"config.h\"\n>> +#include \"run-command.h\"\n>> +#include \"strbuf.h\"\n>> +#include \"string-list.h\"\n>> +#include \"trace2.h\"\n>> +#include \"version.h\"\n>> +#include \"dir.h\"\n>> +#include \"date.h\"\n>> +\n>> +#define TR2_CAT \"test-http-server\"\n>> +\n>> +static const char *pid_file;\n>> +static int verbose;\n>> +static int reuseaddr;\n>> +\n>> +static const char test_http_auth_usage[] =\n>> +\"http-server [--verbose]\\n\"\n>> +\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n>> +\" [--reuseaddr] [--pid-file=]\\n\"\n>> +\" [--listen=]* [--port=]\\n\"\n>> +\" [--anonymous-allowed]\\n\"\n>> +\" [--auth=[:] [--auth-token=:]]*\\n\"\n>> +;\n> \n> These are a lot of options to implement all at once. They are probably\n> simple enough, but depending on the implementation and tests, it might\n> be helpful to split this patch into smaller ones that introduce these\n> options along with the tests that exercise each. That will help\n> verify that they are being tested properly instead of needing to track\n> back and forth across the patch for each one.\n\nI plan to split this patch in to several in a v3.\n\n>> +\n>> +/* Timeout, and initial timeout */\n>> +static unsigned int timeout;\n>> +static unsigned int init_timeout;\n>> +\n>> +static void logreport(const char *label, const char *err, va_list params)\n>> +{\n>> +\tstruct strbuf msg = STRBUF_INIT;\n>> +\n>> +\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n>> +\tstrbuf_vaddf(&msg, err, params);\n>> +\tstrbuf_addch(&msg, '\\n');\n>> +\n>> +\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n>> +\tfflush(stderr);\n>> +\n>> +\tstrbuf_release(&msg);\n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void logerror(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"error\", err, params);\n>> +\tva_end(params);\n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void loginfo(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tif (!verbose)\n>> +\t\treturn;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"info\", err, params);\n>> +\tva_end(params);\n>> +}\n> \n> I wonder how much of this we need or is just a nice thing. I would\n> err on the side of making things as simple as possible, but being\n> able to debug this test server may be important based on your\n> experience.\n\nThese are useful to debug failures. Plus they also come from my copy\nfrom daemon.c, so didn't want to touch/delete too much from that\nstarting point.\n\n>> +static void set_keep_alive(int sockfd)\n>> +{\n>> +\tint ka = 1;\n>> +\n>> +\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n>> +\t\tif (errno != ENOTSOCK)\n>> +\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n>> +\t\t\t\tstrerror(errno));\n>> +\t}\n>> +}\n>> +\n>> +//////////////////////////////////////////////////////////////////\n>> +// The code in this section is used by \"worker\" instances to service\n>> +// a single connection from a client. The worker talks to the client\n>> +// on 0 and 1.\n>> +//////////////////////////////////////////////////////////////////\n> \n> Use /* */ style comments. You can repeat the asterisks to get a\n> similar visual block.\n\nYep!\n\n>> +\n>> +enum worker_result {\n>> +\t/*\n>> +\t * Operation successful.\n>> +\t * Caller *might* keep the socket open and allow keep-alive.\n>> +\t */\n>> +\tWR_OK = 0,\n>> +\t/*\n>> +\t * Various errors while processing the request and/or the response.\n>> +\t * Close the socket and clean up.\n>> +\t * Exit child-process with non-zero status.\n>> +\t */\n>> +\tWR_IO_ERROR = 1<<0,\n>> +\t/*\n>> +\t * Close the socket and clean up. Does not imply an error.\n>> +\t */\n>> +\tWR_HANGUP = 1<<1,\n> \n> nit: add a whitespace line between an item and the next\n> item's comment.\n\nSure\n\n>> +\n>> +\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n>> +};\n> \n> (I read, but have no comments on the http-server boilerplate.)\n> \n>> +\n>> +enum auth_result {\n>> +\tAUTH_UNKNOWN = 0,\n>> +\tAUTH_DENY = 1,\n>> +\tAUTH_ALLOW = 2,\n>> +};\n>> +\n>> +struct auth_module {\n>> +\tconst char *scheme;\n>> +\tconst char *challenge_params;\n> \n> Later, I notice that you set challenge_params using an\n> xstrdup() so this shouldn't be const and you should\n> free it in any freeing code.\n\nOne question on this suggestion.. where would be appropriate to\nfree said char*? We need them for the lifetime of the process,\nand they never grown in number beyond initial allocation from\nparsing command line args.\n\nI could move to stack alloc these in `cmd_main` and instead pass\na pointer to the `auth_modules` and count down through every\nserve/handle etc function, rather than rely on them being global?\n\nThoughts or preferences?\n\n>> +\tstruct string_list *tokens;\n>> +};\n>> +\n>> +static int allow_anonymous;\n>> +static struct auth_module **auth_modules = NULL;\n>> +static size_t auth_modules_nr = 0;\n>> +static size_t auth_modules_alloc = 0;\n> \n> So, we are setting up a number of potential auth modules,\n> each of which has a scheme to match a request to the module,\n> and a list of tokens that would be considered worthy of the\n> AUTH_ALLOW result. Otherwise, if the scheme matches but no\n> token matches, we get AUTH_DENY. Finally, if no scheme matches\n> we get AUTH_UNKNOWN.\n> \n> This concept might be worth a comment here around the data\n> structures before we get into how that is implemented.\n> \n>> +static struct auth_module *get_auth_module(struct strbuf *scheme)\n>> +{\n>> +\tint i;\n>> +\tstruct auth_module *mod;\n>> +\tfor (i = 0; i < auth_modules_nr; i++) {\n>> +\t\tmod = auth_modules[i];\n>> +\t\tif (!strcasecmp(mod->scheme, scheme->buf))\n>> +\t\t\treturn mod;\n>> +\t}\n>> +\n>> +\treturn NULL;\n>> +}\n> \n> Matching the input scheme against the list of modules.\n> \n> Only complaint: there is no reason that 'scheme' needs t\n> be a strbuf, but could be a 'const char *' here.\n\nTrue.\n\n>> +static void add_auth_module(struct auth_module *mod)\n>> +{\n>> +\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n>> +\tauth_modules[auth_modules_nr++] = mod;\n>> +}\n> \n> nit: this could be located earlier, next to the list\n> definition, or delayed until it is needed. That would\n> allow get_auth_module() to be closer to its first use.\n\nNot sure I follow.. are you saying I should move `add_auth_module`\nto earlier in the file?\n\n>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n>> +{\n>> +\tenum auth_result result = AUTH_UNKNOWN;\n>> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n>> +\tstruct auth_module *mod;\n>> +\n>> +\tstruct string_list_item *hdr;\n>> +\tstruct string_list_item *token;\n>> +\tconst char *v;\n>> +\tstruct strbuf **split = NULL;\n>> +\tint i;\n>> +\tchar *challenge;\n>> +\n>> +\t/* ask all auth modules to validate the request */\n>> +\tfor_each_string_list_item(hdr, &req->header_list) {\n>> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n>> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n>> +\t\t\tif (!split[0] || !split[1]) continue;\n> \n> For each valid request header...\n> \n>> +\t\t\t// trim trailing space ' '\n>> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n>> +\n>> +\t\t\tmod = get_auth_module(split[0]);\n>> +\t\t\tif (mod) {\n> \n> ...get an appropriate module, if it exists...\n> \n>> +\n>> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n>> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n>> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n>> +\t\t\t\t\t\tgoto done;\n>> +\t\t\t\t\t}\n>> +\t\t\t\t}\n>> +\n>> +\t\t\t\tif (result != AUTH_UNKNOWN)\n>> +\t\t\t\t\tgoto done;\n> \n> ...and report if we find a valid token.\n> \n> Here, it seems I was wrong in my expectation of AUTH_DENY:\n> if a matching module exists but no token exists in that\n> module, then we keep searching other modules. \n\nAUTH_DENY denies a request immediately and stops searching other modules.\nAUTH_ALLOW approves the request and stops looking at other modules.\nAUTH_UNKNOWN means this module didn't match or 'decide' to reject, so keep\nlooking/asking other modules.\n\nAfter reading you review, I think it may be better to change this to\nmore closely match your expectations (and how typical servers behave):\n\nReturn AUTH_ALLOW if we find a matching valid token for the module.\nIf we match a module and do NOT find a token, then return AUTH_DENY.\nOtherwise return AUTH_UNKNOWN - this means the user provided some auth\nmechanism we don't understand, or no auth at all.\n\n>> +\t\t\t}\n>> +\t\t}\n>> +\t}\n>> +\n>> +done:\n>> +\tswitch (result) {\n>> +\tcase AUTH_ALLOW:\n>> +\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n>> +\t\t*user = \"VALID_TEST_USER\";\n>> +\t\t*wr = WR_OK;\n>> +\t\tbreak;\n>> +\n>> +\tcase AUTH_DENY:\n>> +\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n>> +\t\t/* fall-through */\n> \n> I'm not sure that I see a case where this is possible. Maybe\n> we should have a 'result = AUTH_DENY' at the start of the\n> \"if (mod)\" block, followed by a 'goto done' in all cases\n> instead of \"if (result != AUTH_UNKNOWN)\"?\n\nIn this version, you're correct.. AUTH_DENY is never returned.\nThis tri-state response from an auth module is an oversight from an earlier\nlocal version - sorry for the confusion here, and thanks for catching!\nI will update in a v3 to match sane expectations.\n\n>> +\tcase AUTH_UNKNOWN:\n>> +\t\tif (allow_anonymous)\n>> +\t\t\tbreak;\n> \n> If we do not require auth, then we want to continue if there\n> is no matching authentication.\n> \n>> +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n>> +\t\t\tmod = auth_modules[i];\n>> +\t\t\tif (mod->challenge_params)\n>> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n>> +\t\t\t\t\t\t mod->scheme,\n>> +\t\t\t\t\t\t mod->challenge_params);\n>> +\t\t\telse\n>> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n>> +\t\t\t\t\t\t mod->scheme);\n>> +\t\t\tstring_list_append(&hdrs, challenge);\n>> +\t\t}\n>> +\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n> \n> However, here is the critical piece about how servers will\n> start to act with the new WWW-Authenticate header usage in\n> the Git credential helper interface. This will be critical\n> in the testing for Git to retry the credential helper while\n> passing these authentications schemes from the installed\n> modules.\n> \n>> +\t}\n>> +\n>> +\tstrbuf_list_free(split);\n>> +\tstring_list_clear(&hdrs, 0);\n>> +\n>> +\treturn result == AUTH_ALLOW ||\n>> +\t (result == AUTH_UNKNOWN && allow_anonymous);\n> \n> Did it work? Or did it not need to work? I'm interested to\n> investigate the case that the client sent an authentication\n> header that matches a module but doesn't match any tokens,\n> but we allow anonymous access, anyway. Is that a 400? Or\n> is that a 401?\n\nIt should probably be a 401 as the credentials are understood, but\nare just 'bad'.\n\n>> +static enum worker_result dispatch(struct req *req)\n>> +{\n>> +\tenum worker_result wr = WR_OK;\n>> +\tconst char *user = NULL;\n>> +\n>> +\tif (!is_authed(req, &user, &wr))\n>> +\t\treturn wr;\n> \n> If we are not authed, send the 401 response.\n> \n>> +\tif (is_git_request(req))\n>> +\t\treturn do__git(req, user);\n> \n> If we are authed, then pass through to the Git response.\n> \n>> +\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n>> +\t\t\t WR_OK | WR_HANGUP);\n> \n> If the Git request fails, we don't care. This is a test.\n> Just pass a 500-level error and the client will barf,\n> letting us know that something went wrong.\n\nCorrect assessment!\n\n>> +static void kill_some_child(void)\n> \n>> +static void check_dead_children(void)\n> \n> These technically sound methods have unfortunate names.\n> Using something like \"connection\" over \"child\" might\n> alleviate some of the horror. (I initially wanted to\n> suggest \"subprocess\" but you compare live_children to\n> max_connections in the next method, so connection seemed\n> appropriate.)\n\nThese are copied exactly from git-daemon, so I'd rather\navoid the churn in renaming things.\n\n>> +static struct strvec cld_argv = STRVEC_INIT;\n>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>> +{\n>> +\tstruct child_process cld = CHILD_PROCESS_INIT;\n>> +\n>> +\tif (max_connections && live_children >= max_connections) {\n>> +\t\tkill_some_child();\n>> +\t\tsleep(1); /* give it some time to die */\n>> +\t\tcheck_dead_children();\n>> +\t\tif (live_children >= max_connections) {\n>> +\t\t\tclose(incoming);\n>> +\t\t\tlogerror(\"Too many children, dropping connection\");\n>> +\t\t\treturn;\n>> +\t\t}\n>> +\t}\n> \n> Do we anticipate exercising concurrent requests in our\n> tests? Perhaps it's not worth putting a cap on the\n> connection count so we can keep the test helpers simple.\n\nProbably not, but again.. 100% of the boilerplate here came from\nthe prior art in daemon.c, so didn't want to touch any of it!\nI'm happy to start deleting things however if needed?\n\n>> +\tif (addr->sa_family == AF_INET) {\n>> +\t\tchar buf[128] = \"\";\n>> +\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n>> +\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n>> +\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n>> +\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n>> +\t\t\t\t ntohs(sin_addr->sin_port));\n>> +#ifndef NO_IPV6\n>> +\t} else if (addr->sa_family == AF_INET6) {\n>> +\t\tchar buf[128] = \"\";\n>> +\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n>> +\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n>> +\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n>> +\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n>> +\t\t\t\t ntohs(sin6_addr->sin6_port));\n>> +#endif\n>> +\t}\n>> +\n>> +\tstrvec_pushv(&cld.args, cld_argv.v);\n>> +\tcld.in = incoming;\n>> +\tcld.out = dup(incoming);\n>> +\n>> +\tif (cld.out < 0)\n>> +\t\tlogerror(\"could not dup() `incoming`\");\n>> +\telse if (start_command(&cld))\n>> +\t\tlogerror(\"unable to fork\");\n>> +\telse\n>> +\t\tadd_child(&cld, addr, addrlen);\n>> +}\n>> +\n> \n> I scanned the socket creation code, but my eyes were\n> glazing over. I'm definitely in the camp of \"if it works,\n> that's enough for our tests.\" If we start to rely on this\n> test harness in more places, we can improve any shortcomings\n> as they arise.\n> \n>> +//////////////////////////////////////////////////////////////////\n>> +// This section is executed by both the primary instance and all\n>> +// worker instances. So, yes, each child-process re-parses the\n>> +// command line argument and re-discovers how it should behave.\n>> +//////////////////////////////////////////////////////////////////\n>> +\n>> +int cmd_main(int argc, const char **argv)\n>> +{\n>> +\tint listen_port = 0;\n>> +\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n>> +\tint worker_mode = 0;\n>> +\tint i;\n>> +\tstruct auth_module *mod = NULL;\n>> +\n>> +\ttrace2_cmd_name(\"test-http-server\");\n>> +\tsetup_git_directory_gently(NULL);\n>> +\n>> +\tfor (i = 1; i < argc; i++) {\n>> +\t\tconst char *arg = argv[i];\n>> +\t\tconst char *v;\n>> +\n>> +\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n>> +\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n>> +\t\t\tchar *end;\n>> +\t\t\tunsigned long n;\n>> +\t\t\tn = strtoul(v, &end, 0);\n>> +\t\t\tif (*v && !*end) {\n>> +\t\t\t\tlisten_port = n;\n>> +\t\t\t\tcontinue;\n>> +\t\t\t}\n>> +\t\t}\n>> +\t\tif (!strcmp(arg, \"--worker\")) {\n>> +\t\t\tworker_mode = 1;\n>> +\t\t\ttrace2_cmd_mode(\"worker\");\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (!strcmp(arg, \"--verbose\")) {\n>> +\t\t\tverbose = 1;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n>> +\t\t\ttimeout = atoi(v);\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n>> +\t\t\tinit_timeout = atoi(v);\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n>> +\t\t\tmax_connections = atoi(v);\n>> +\t\t\tif (max_connections < 0)\n>> +\t\t\t\tmax_connections = 0; /* unlimited */\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n>> +\t\t\treuseaddr = 1;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n>> +\t\t\tpid_file = v;\n>> +\t\t\tcontinue;\n>> +\t\t}\n> \n> ok, most of these arguments are actually about the per-connection\n> subprocesses.\n> \n>> +\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n>> +\t\t\tallow_anonymous = 1;\n>> +\t\t\tcontinue;\n>> +\t\t}\n> \n> Here is how we choose to allo anonymous access.\n> \n>> +\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n>> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n>> +\n>> +\t\t\tif (!p[0]) {\n>> +\t\t\t\terror(\"invalid argument '%s'\", v);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\t// trim trailing ':'\n>> +\t\t\tif (p[1])\n>> +\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n>> +\n>> +\t\t\tif (get_auth_module(p[0])) {\n>> +\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\tmod = xmalloc(sizeof(struct auth_module));\n>> +\t\t\tmod->scheme = xstrdup(p[0]->buf);\n>> +\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n> \n> Here, you xstrdup() into a 'const char *', but you are really\n> passing ownership so it shouldn't be conts.\nOk\n\n> \n>> +\t\t\tmod->tokens = xmalloc(sizeof(struct string_list));\n> \n> nit: this could also be \"CALLOC_ARRAY(mod->tokens, 1);\"\nSure!\n>> +\t\t\tstring_list_init_dup(mod->tokens);\n>> +\n>> +\t\t\tadd_auth_module(mod);\n>> +\n>> +\t\t\tstrbuf_list_free(p);\n>> +\t\t\tcontinue;\n> \n> Ok, we gain the auth schemes from the command line.\n> \n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n>> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n>> +\t\t\tif (!p[0]) {\n>> +\t\t\t\terror(\"invalid argument '%s'\", v);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (!p[1]) {\n>> +\t\t\t\terror(\"missing token value '%s'\\n\", v);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\t// trim trailing ':'\n> \n> Use /* */ (Aside: I'm surprised we don't have a build option in\n> DEVELOPER=1 that catches the use of these comments.)\nMe too! Appologies here.\n>> +\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n>> +\n>> +\t\t\tmod = get_auth_module(p[0]);\n>> +\t\t\tif (!mod) {\n>> +\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n>> +\t\t\tstrbuf_list_free(p);\n>> +\t\t\tcontinue;\n>> +\t\t}\n> \n> And the token lists. It is important that the scheme is added\n> before any token is added.\n> \n>> +\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n>> +\t\tusage(test_http_auth_usage);\n>> +\t}\n>> +\n>> +\t/* avoid splitting a message in the middle */\n>> +\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n>> +\n>> +\tif (listen_port == 0)\n>> +\t\tlisten_port = DEFAULT_GIT_PORT;\n>> +\n>> +\t/*\n>> +\t * If no --listen= args are given, the setup_named_sock()\n>> +\t * code will use receive a NULL address and set INADDR_ANY.\n>> +\t * This exposes both internal and external interfaces on the\n>> +\t * port.\n>> +\t *\n>> +\t * Disallow that and default to the internal-use-only loopback\n>> +\t * address.\n>> +\t */\n>> +\tif (!listen_addr.nr)\n>> +\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n>> +\n>> +\t/*\n>> +\t * worker_mode is set in our own child process instances\n>> +\t * (that are bound to a connected socket from a client).\n>> +\t */\n>> +\tif (worker_mode)\n>> +\t\treturn worker();\n>> +\n>> +\t/*\n>> +\t * `cld_argv` is a bit of a clever hack. The top-level instance\n>> +\t * of test-http-server does the normal bind/listen/accept stuff.\n>> +\t * For each incoming socket, the top-level process spawns\n>> +\t * a child instance of test-http-server *WITH* the additional\n>> +\t * `--worker` argument. This causes the child to set `worker_mode`\n>> +\t * and immediately call `worker()` using the connected socket (and\n>> +\t * without the usual need for fork() or threads).\n>> +\t *\n>> +\t * The magic here is made possible because `cld_argv` is static\n>> +\t * and handle() (called by service_loop()) knows about it.\n>> +\t */\n>> +\tstrvec_push(&cld_argv, argv[0]);\n>> +\tstrvec_push(&cld_argv, \"--worker\");\n>> +\tfor (i = 1; i < argc; ++i)\n>> +\t\tstrvec_push(&cld_argv, argv[i]);\n>> +\n>> +\t/*\n>> +\t * Setup primary instance to listen for connections.\n>> +\t */\n>> +\treturn serve(&listen_addr, listen_port);\n>> +}\n> \n> And complete the thing with some boilerplate.\n> \n> This was a lot to read, and the interesting bits are all mixed in\n> with the http server code, which is less interesting to what we\n> are trying to accomplish. It would be beneficial to split this\n> into one or two patches before we actually introduce the tests.\n> \n> The most important thing that I think would be helpful is to\n> isolate all the authentication behavior into its own patch so\n> we can see how those connections from the command-line arguments\n> affect the behavior of the server responses.\n> \n> I think ideally we would have the following split:\n> \n> 1. All server boilerblate. All requests 500 not-implemented.\n> \n> 2. Add Git fall-through with no authentication. Add the tests\n> that are intended to allow anonymous auth.\n> \n> 3. Add authentication data structures read from command-line,\n> but not processed at all in the logic.\n> \n> 4. Act on the authentication data structures to alter the\n> requests. Add the tests that use these authentication\n> schemes.\n> \n> I could easily see a case for combining 1&2 as well as 3&4,\n> for slightly larger but more completely-testable changes at\n> every step.\nI agree, and my appologies for not splitting these out.\nI'll follow up with a split that should make more sense.\n> From what I read, I don't think there is much to change in\n> the end result of the code, but it definitely was hard to read\n> the important things when surrounded by many lines of\n> boilerplate.\n> \n>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n> \n> I'm going to pause here and come back to the test script in\n> a separate reply.\n> \n> Thanks,\n> -Stolee\nThanks,\nMatthew\n"},{"id":"466314","messageId":"5666c8cc-158b-dfc7-a35f-d39d8f53ea54@github.com","threadId":"58425","inReplyTo":"AS2PR03MB981549CCF945BF26DD212BDBC0369@AS2PR03MB9815.eurprd03.prod.outlook.com","subject":"Re: [PATCH v2 6/6] t5556-http-auth: add test for HTTP auth hdr logic","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-11-02T14:38:17Z","receivedAt":"2022-11-02T14:38:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/1/22 7:14 PM, Matthew John Cheetham wrote:\n> On 2022-10-28 12:14, Jeff Hostetler wrote:\n>> On 10/28/22 11:08 AM, Derrick Stolee wrote:\n\n>>>> +static void kill_some_child(void)\n>>>\n>>>> +static void check_dead_children(void)\n>>>\n>>> These technically sound methods have unfortunate names.\n>>> Using something like \"connection\" over \"child\" might\n>>> alleviate some of the horror. (I initially wanted to\n>>> suggest \"subprocess\" but you compare live_children to\n>>> max_connections in the next method, so connection seemed\n>>> appropriate.)\n>>\n>> These names were inherited from `daemon.c` IIRC. I wouldn't change\n>> them since it'll just introduce noise when diffing. Especially,\n>> if we do the copy commit first.\n> \n> Indeed. These functions are untouched from daemon.c. I do plan to split\n> this mega-patch up however in to a single 'add the boilerplate' based on\n> git-daemon patch, then add the extra pieces like HTTP request parsing and\n> the auth pieces in a v3.\n\nIf these are copied from daemon.c, it may be worth trying\nto lib-ify these data structures and code so they can be\nshared across the two places. That can also come up as a\ncleanup later, too.\n\nFor now, don't bother changing the names since they exist\nsomewhere else.\n \n>> [...]\n>>>> +static struct strvec cld_argv = STRVEC_INIT;\n>>>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>>>> +{\n>>>> + struct child_process cld = CHILD_PROCESS_INIT;\n>>>> +\n>>>> + if (max_connections && live_children >= max_connections) {\n>>>> + kill_some_child();\n>>>> + sleep(1); /* give it some time to die */\n>>>> + check_dead_children();\n>>>> + if (live_children >= max_connections) {\n>>>> + close(incoming);\n>>>> + logerror(\"Too many children, dropping connection\");\n>>>> + return;\n>>>> + }\n>>>> + }\n>>>\n>>> Do we anticipate exercising concurrent requests in our\n>>> tests? Perhaps it's not worth putting a cap on the\n>>> connection count so we can keep the test helpers simple.\n>>\n>> again, this code was inherited from `daemon.c`, so we could leave it.\n\nI wonder how much could be extracted from daemon.c using a\ncopy into a 'daemon-lib.c' with methods defined in 'daemon-lib.h'\nthen consumed from this file instead. Not sure it's worth the\nchurn to daemon.c, though.\n\nThanks,\n-Stolee\n"},{"id":"466337","messageId":"e45e23406a5e1609a36375acf9cb36ac6efc2dd6.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 02/11] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:20Z","receivedAt":"2022-11-02T22:09:42Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 17 ++++++++++++++++-\n credential.c | 12 ++++++++++++\n 2 files changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex f18673017f5..791a57dddfb 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,15 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n+\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n+\tattribute 'wwwauth[]', where the order of the attributes is the same as\n+\tthey appear in the HTTP response.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..8a3ad6c0ae2 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tint i = 0;\n+\tconst char *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree((void*)full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\n-- \ngitgitgadget\n\n"},{"id":"466338","messageId":"f297c78f60a6996c2d2e5397b05efa6b94fd2ae0.1667426969.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 01/11] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:19Z","receivedAt":"2022-11-02T22:09:45Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++++++\n http.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 94 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex 5d0502f51fd..03d43d352e7 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tconst char *z = NULL;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\tconst char **v = values->v + values->nr - 1;\n+\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n+\n+\t\tfree((void*)*v);\n+\t\t*v = append;\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (skip_iprefix(buf.buf, \"http/\", &z))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1829,6 +1905,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"466340","messageId":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v2.git.1666372083.gitgitgadget@gmail.com","subject":"[PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:18Z","receivedAt":"2022-11-02T22:09:46Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server based\nheavily on git-daemon, with simple authentication configurable by command\nline args.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Leverage newest identity standards, enhancements, and threat\n mitigations - all without updating Git.\n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n 3. Teach Git to specify authentication schemes other than Basic in\n subsequent HTTP requests based on credential helper responses.\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture flexibility\n==================\n\nBy allowing the credential helpers decide the best authentication scheme, we\ncan allow the remote Git server to both offer new schemes (or remove old\nones) that enlightened credential helpers could take immediate advantage of,\nand to use credentials that are much more tightly scoped and bound to the\nspecific request.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \n\n\n\nShould Git not control the set of authentication schemes?\n=========================================================\n\nOne concern that the reader may have regarding these changes is in allowing\nhelpers to select the authentication mechanism to use, it may be possible\nthat a weaker form of authentication is used.\n\nTake for example a Git remote server that responds with the following\nauthentication schemes:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Negotiate ...\nWWW-Authenticate: Basic ...\n\n\nToday Git (and libcurl) prefer to Negotiate over Basic authentication [13].\nIf a helper responded with authtype=basic Git would now be using a \"less\nsecure\" mechanism.\n\nThe reason we still propose the credential helper decide on the\nauthentication scheme is that Git is not the best placed entity to decide\nwhat type of authentication should be used for a particular request (see\nDesign Principle 3).\n\nOAuth Bearer tokens are often bundled in Basic Authorization headers [14],\nbut given that the tokens are/can be short-lived and have a highly scoped\nset of permissions, this solution could be argued as being more secure than\nsomething like NTLM [15]. Similarly, the user may wish to be consulted on\nselecting a particular user account, or directly selecting an authentication\nmechanism for a request that otherwise they would not be able to use.\n\nAlso, as new authentication protocols appear Git does not need to be\nmodified or updated for the user to take advantage of them; the credential\nhelpers take on the responsibility of learning and selecting the \"best\"\noption.\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n * [13] libcurl http.c pickoneauth Function\n https://github.com/curl/curl/blob/c495dcd02e885fc3f35164b1c3c5f72fa4b60c46/lib/http.c#L381-L416\n\n * [14] Git Credential Manager GitHub Host Provider (using PAT as password)\n https://github.com/GitCredentialManager/git-credential-manager/blob/f77b766f6875b90251249f2aa1702b921309cf00/src/shared/GitHub/GitHubHostProvider.cs#L157\n\n * [15] NT LAN Manager (NTLM) Authentication Protocol\n https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-nlmp/b38c36ed-2804-4868-a9ff-8dd3182128e4\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\nMatthew John Cheetham (11):\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n http: store all request headers on active_request_slot\n http: move proactive auth to first slot creation\n http: set specific auth scheme depending on credential\n test-http-server: add stub HTTP server test helper\n test-http-server: add HTTP error response function\n test-http-server: add HTTP request parsing\n test-http-server: pass Git requests to http-backend\n test-http-server: add simple authentication\n t5556: add HTTP authentication tests\n\n Documentation/git-credential.txt | 29 +-\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n credential.c | 18 +\n credential.h | 16 +\n git-curl-compat.h | 10 +\n http-push.c | 103 +-\n http-walker.c | 2 +-\n http.c | 200 +++-\n http.h | 4 +-\n remote-curl.c | 36 +-\n t/helper/.gitignore | 1 +\n t/helper/test-credential-helper-replay.sh | 14 +\n t/helper/test-http-server.c | 1146 +++++++++++++++++++++\n t/t5556-http-auth.sh | 260 +++++\n 15 files changed, 1717 insertions(+), 137 deletions(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n create mode 100644 t/helper/test-http-server.c\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: 9c32cfb49c60fa8173b9666db02efe3b45a8522f\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v2:\n\n 1: f297c78f60a = 1: f297c78f60a http: read HTTP WWW-Authenticate response headers\n 2: 0838d992744 ! 2: e45e23406a5 credential: add WWW-Authenticate header to cred requests\n @@ Commit message\n C-style array syntax is used in the property name to denote multiple\n ordered values for the same property.\n \n - In this case we send multiple `wwwauth[n]` properties where `n` is a\n - zero-indexed number, reflecting the order the WWW-Authenticate headers\n - appeared in the HTTP response.\n + In this case we send multiple `wwwauth[]` properties where the order\n + that the repeated attributes appear in the conversation reflects the\n + order that the WWW-Authenticate headers appeared in the HTTP response.\n \n [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n \n Signed-off-by: Matthew John Cheetham \n \n ## Documentation/git-credential.txt ##\n +@@ Documentation/git-credential.txt: separated by an `=` (equals) sign, followed by a newline.\n + The key may contain any bytes except `=`, newline, or NUL. The value may\n + contain any bytes except newline or NUL.\n + \n +-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n ++Attributes with keys that end with C-style array brackets `[]` can have\n ++multiple values. Each instance of a multi-valued attribute forms an\n ++ordered list of values - the order of the repeated attributes defines\n ++the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n ++acts to clear any previous entries and reset the list.\n ++\n ++In all cases, all bytes are treated as-is (i.e., there is no quoting,\n + and one cannot transmit a value with newline or NUL in it). The list of\n + attributes is terminated by a blank line or end-of-file.\n + \n @@ Documentation/git-credential.txt: empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n +`wwwauth[]`::\n +\n -+\tWhen an HTTP response is received that includes one or more\n -+\t'WWW-Authenticate' authentication headers, these can be passed to Git\n -+\t(and subsequent credential helpers) with these attributes.\n -+\tEach 'WWW-Authenticate' header value should be passed as a separate\n -+\tattribute 'wwwauth[]' where the order of the attributes is the same\n -+\tas they appear in the HTTP response.\n ++\tWhen an HTTP response is received by Git that includes one or more\n ++\t'WWW-Authenticate' authentication headers, these will be passed by Git\n ++\tto credential helpers.\n ++\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n ++\tattribute 'wwwauth[]', where the order of the attributes is the same as\n ++\tthey appear in the HTTP response.\n +\n GIT\n ---\n 3: c62fef65f46 = 3: 65ac638b8a0 http: store all request headers on active_request_slot\n 4: a790c01f9f2 = 4: 4d75ca29cc5 http: move proactive auth to first slot creation\n 5: b0b7cd7ee5e ! 5: 2f38427aa8d http: set specific auth scheme depending on credential\n @@ Commit message\n \n ## Documentation/git-credential.txt ##\n @@ Documentation/git-credential.txt: username in the example above) will be left unset.\n - \tattribute 'wwwauth[]' where the order of the attributes is the same\n - \tas they appear in the HTTP response.\n + \tattribute 'wwwauth[]', where the order of the attributes is the same as\n + \tthey appear in the HTTP response.\n \n +`authtype`::\n +\n -+\tIndicates the type of authentication scheme used. If this is not\n -+\tpresent the default is \"Basic\".\n ++\tIndicates the type of authentication scheme that should be used by Git.\n ++\tCredential helpers may reply to a request from Git with this attribute,\n ++\tsuch that subsequent authenticated requests include the correct\n ++\t`Authorization` header.\n ++\tIf this attribute is not present, the default value is \"Basic\".\n +\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n +\tIf an unknown value is provided, this is taken as the authentication\n +\tscheme for the `Authorization` header, and the `password` field is\n 6: f3f13ed8c82 ! 6: 4947e81546a t5556-http-auth: add test for HTTP auth hdr logic\n @@ Metadata\n Author: Matthew John Cheetham \n \n ## Commit message ##\n - t5556-http-auth: add test for HTTP auth hdr logic\n + test-http-server: add stub HTTP server test helper\n \n - Add a series of tests to exercise the HTTP authentication header parsing\n - and the interop with credential helpers. Credential helpers can respond\n - to requests that contain WWW-Authenticate information with the ability\n - to select the response Authenticate header scheme.\n + Introduce a mini HTTP server helper that in the future will be enhanced\n + to provide a frontend for the git-http-backend, with support for\n + arbitrary authentication schemes.\n \n - Introduce a mini HTTP server helper that provides a frontend for the\n - git-http-backend, with support for arbitrary authentication schemes.\n - The test-http-server is based heavily on the git-daemon, and forwards\n - all successfully authenticated requests to the http-backend.\n + Right now, test-http-server is a pared-down copy of the git-daemon that\n + always returns a 501 Not Implemented response to all callers.\n \n Signed-off-by: Matthew John Cheetham \n \n @@ t/helper/.gitignore\n @@\n /test-tool\n /test-fake-ssh\n -+test-http-server\n -\n - ## t/helper/test-credential-helper-replay.sh (new) ##\n -@@\n -+cmd=$1\n -+teefile=$cmd-actual.cred\n -+catfile=$cmd-response.cred\n -+rm -f $teefile\n -+while read line;\n -+do\n -+\tif test -z \"$line\"; then\n -+\t\tbreak;\n -+\tfi\n -+\techo \"$line\" >> $teefile\n -+done\n -+if test \"$cmd\" = \"get\"; then\n -+\tcat $catfile\n -+fi\n ++/test-http-server\n \n ## t/helper/test-http-server.c (new) ##\n @@\n @@ t/helper/test-http-server.c (new)\n +\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n +\" [--reuseaddr] [--pid-file=]\\n\"\n +\" [--listen=]* [--port=]\\n\"\n -+\" [--anonymous-allowed]\\n\"\n -+\" [--auth=[:] [--auth-token=:]]*\\n\"\n +;\n +\n +/* Timeout, and initial timeout */\n @@ t/helper/test-http-server.c (new)\n +\t}\n +}\n +\n -+//////////////////////////////////////////////////////////////////\n -+// The code in this section is used by \"worker\" instances to service\n -+// a single connection from a client. The worker talks to the client\n -+// on 0 and 1.\n -+//////////////////////////////////////////////////////////////////\n ++/*\n ++ * The code in this section is used by \"worker\" instances to service\n ++ * a single connection from a client. The worker talks to the client\n ++ * on 0 and 1.\n ++ */\n +\n +enum worker_result {\n +\t/*\n @@ t/helper/test-http-server.c (new)\n +\t * Caller *might* keep the socket open and allow keep-alive.\n +\t */\n +\tWR_OK = 0,\n ++\n +\t/*\n +\t * Various errors while processing the request and/or the response.\n +\t * Close the socket and clean up.\n +\t * Exit child-process with non-zero status.\n +\t */\n +\tWR_IO_ERROR = 1<<0,\n ++\n +\t/*\n +\t * Close the socket and clean up. Does not imply an error.\n +\t */\n @@ t/helper/test-http-server.c (new)\n +\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n +};\n +\n -+/*\n -+ * Fields from a parsed HTTP request.\n -+ */\n -+struct req {\n -+\tstruct strbuf start_line;\n -+\n -+\tconst char *method;\n -+\tconst char *http_version;\n -+\n -+\tstruct strbuf uri_path;\n -+\tstruct strbuf query_args;\n -+\n -+\tstruct string_list header_list;\n -+\tconst char *content_type;\n -+\tssize_t content_length;\n -+};\n -+\n -+#define REQ__INIT { \\\n -+\t.start_line = STRBUF_INIT, \\\n -+\t.uri_path = STRBUF_INIT, \\\n -+\t.query_args = STRBUF_INIT, \\\n -+\t.header_list = STRING_LIST_INIT_NODUP, \\\n -+\t.content_type = NULL, \\\n -+\t.content_length = -1 \\\n -+\t}\n -+\n -+static void req__release(struct req *req)\n -+{\n -+\tstrbuf_release(&req->start_line);\n -+\n -+\tstrbuf_release(&req->uri_path);\n -+\tstrbuf_release(&req->query_args);\n -+\n -+\tstring_list_clear(&req->header_list, 0);\n -+}\n -+\n -+static enum worker_result send_http_error(\n -+\tint fd,\n -+\tint http_code, const char *http_code_name,\n -+\tint retry_after_seconds, struct string_list *response_headers,\n -+\tenum worker_result wr_in)\n -+{\n -+\tstruct strbuf response_header = STRBUF_INIT;\n -+\tstruct strbuf response_content = STRBUF_INIT;\n -+\tstruct string_list_item *h;\n -+\tenum worker_result wr;\n -+\n -+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n -+\t\t http_code, http_code_name);\n -+\tif (retry_after_seconds > 0)\n -+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n -+\t\t\t retry_after_seconds);\n -+\n -+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n -+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n -+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n -+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n -+\tif (retry_after_seconds > 0)\n -+\t\tstrbuf_addf (&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n -+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n -+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n -+\tif (response_headers)\n -+\t\tfor_each_string_list_item(h, response_headers)\n -+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n -+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n -+\n -+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n -+\t\tlogerror(\"unable to write response header\");\n -+\t\twr = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n -+\t\tlogerror(\"unable to write response content body\");\n -+\t\twr = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\twr = wr_in;\n -+\n -+done:\n -+\tstrbuf_release(&response_header);\n -+\tstrbuf_release(&response_content);\n -+\n -+\treturn wr;\n -+}\n -+\n -+/*\n -+ * Read the HTTP request up to the start of the optional message-body.\n -+ * We do this byte-by-byte because we have keep-alive turned on and\n -+ * cannot rely on an EOF.\n -+ *\n -+ * https://tools.ietf.org/html/rfc7230\n -+ *\n -+ * We cannot call die() here because our caller needs to properly\n -+ * respond to the client and/or close the socket before this\n -+ * child exits so that the client doesn't get a connection reset\n -+ * by peer error.\n -+ */\n -+static enum worker_result req__read(struct req *req, int fd)\n -+{\n -+\tstruct strbuf h = STRBUF_INIT;\n -+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n -+\tint nr_start_line_fields;\n -+\tconst char *uri_target;\n -+\tconst char *query;\n -+\tchar *hp;\n -+\tconst char *hv;\n -+\n -+\tenum worker_result result = WR_OK;\n -+\n -+\t/*\n -+\t * Read line 0 of the request and split it into component parts:\n -+\t *\n -+\t * SP SP CRLF\n -+\t *\n -+\t */\n -+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n -+\t\tresult = WR_OK | WR_HANGUP;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tstrbuf_trim_trailing_newline(&req->start_line);\n -+\n -+\tnr_start_line_fields = string_list_split(&start_line_fields,\n -+\t\t\t\t\t\t req->start_line.buf,\n -+\t\t\t\t\t\t ' ', -1);\n -+\tif (nr_start_line_fields != 3) {\n -+\t\tlogerror(\"could not parse request start-line '%s'\",\n -+\t\t\t req->start_line.buf);\n -+\t\tresult = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\treq->method = xstrdup(start_line_fields.items[0].string);\n -+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n -+\n -+\turi_target = start_line_fields.items[1].string;\n -+\n -+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n -+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n -+\t\t\t req->http_version);\n -+\t\tresult = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tquery = strchr(uri_target, '?');\n -+\n -+\tif (query) {\n -+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n -+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n -+\t\tstrbuf_addstr(&req->query_args, query + 1);\n -+\t} else {\n -+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n -+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n -+\t}\n -+\n -+\t/*\n -+\t * Read the set of HTTP headers into a string-list.\n -+\t */\n -+\twhile (1) {\n -+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n -+\t\t\tgoto done;\n -+\t\tstrbuf_trim_trailing_newline(&h);\n -+\n -+\t\tif (!h.len)\n -+\t\t\tgoto done; /* a blank line ends the header */\n -+\n -+\t\thp = strbuf_detach(&h, NULL);\n -+\t\tstring_list_append(&req->header_list, hp);\n -+\n -+\t\t/* store common request headers separately */\n -+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n -+\t\t\treq->content_type = hv;\n -+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n -+\t\t\treq->content_length = strtol(hv, &hp, 10);\n -+\t\t}\n -+\t}\n -+\n -+\t/*\n -+\t * We do not attempt to read the , if it exists.\n -+\t * We let our caller read/chunk it in as appropriate.\n -+\t */\n -+\n -+done:\n -+\tstring_list_clear(&start_line_fields, 0);\n -+\n -+\t/*\n -+\t * This is useful for debugging the request, but very noisy.\n -+\t */\n -+\tif (trace2_is_enabled()) {\n -+\t\tstruct string_list_item *item;\n -+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n -+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n -+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n -+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n -+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n -+\t\tif (req->content_length >= 0)\n -+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n -+\t\tif (req->content_type)\n -+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n -+\t\tfor_each_string_list_item(item, &req->header_list)\n -+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n -+\t}\n -+\n -+\treturn result;\n -+}\n -+\n -+static int is_git_request(struct req *req)\n -+{\n -+\tstatic regex_t *smart_http_regex;\n -+\tstatic int initialized;\n -+\n -+\tif (!initialized) {\n -+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n -+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n -+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n -+\t\t\t REG_EXTENDED)) {\n -+\t\t\twarning(\"could not compile smart HTTP regex\");\n -+\t\t\tsmart_http_regex = NULL;\n -+\t\t}\n -+\t\tinitialized = 1;\n -+\t}\n -+\n -+\treturn smart_http_regex &&\n -+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n -+}\n -+\n -+static enum worker_result do__git(struct req *req, const char *user)\n -+{\n -+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n -+\tstruct child_process cp = CHILD_PROCESS_INIT;\n -+\tint res;\n -+\n -+\tif (write(1, ok, strlen(ok)) < 0)\n -+\t\treturn error(_(\"could not send '%s'\"), ok);\n -+\n -+\tif (user)\n -+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n -+\n -+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n -+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n -+\t\t\treq->uri_path.buf);\n -+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n -+\tif (req->query_args.len)\n -+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n -+\t\t\t\treq->query_args.buf);\n -+\tif (req->content_type)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n -+\t\t\t\treq->content_type);\n -+\tif (req->content_length >= 0)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n -+\t\t\t\t(intmax_t)req->content_length);\n -+\tcp.git_cmd = 1;\n -+\tstrvec_push(&cp.args, \"http-backend\");\n -+\tres = run_command(&cp);\n -+\tclose(1);\n -+\tclose(0);\n -+\treturn !!res;\n -+}\n -+\n -+enum auth_result {\n -+\tAUTH_UNKNOWN = 0,\n -+\tAUTH_DENY = 1,\n -+\tAUTH_ALLOW = 2,\n -+};\n -+\n -+struct auth_module {\n -+\tconst char *scheme;\n -+\tconst char *challenge_params;\n -+\tstruct string_list *tokens;\n -+};\n -+\n -+static int allow_anonymous;\n -+static struct auth_module **auth_modules = NULL;\n -+static size_t auth_modules_nr = 0;\n -+static size_t auth_modules_alloc = 0;\n -+\n -+static struct auth_module *get_auth_module(struct strbuf *scheme)\n -+{\n -+\tint i;\n -+\tstruct auth_module *mod;\n -+\tfor (i = 0; i < auth_modules_nr; i++) {\n -+\t\tmod = auth_modules[i];\n -+\t\tif (!strcasecmp(mod->scheme, scheme->buf))\n -+\t\t\treturn mod;\n -+\t}\n -+\n -+\treturn NULL;\n -+}\n -+\n -+static void add_auth_module(struct auth_module *mod)\n -+{\n -+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n -+\tauth_modules[auth_modules_nr++] = mod;\n -+}\n -+\n -+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n -+{\n -+\tenum auth_result result = AUTH_UNKNOWN;\n -+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n -+\tstruct auth_module *mod;\n -+\n -+\tstruct string_list_item *hdr;\n -+\tstruct string_list_item *token;\n -+\tconst char *v;\n -+\tstruct strbuf **split = NULL;\n -+\tint i;\n -+\tchar *challenge;\n -+\n -+\t/* ask all auth modules to validate the request */\n -+\tfor_each_string_list_item(hdr, &req->header_list) {\n -+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n -+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n -+\t\t\tif (!split[0] || !split[1]) continue;\n -+\n -+\t\t\t// trim trailing space ' '\n -+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n -+\n -+\t\t\tmod = get_auth_module(split[0]);\n -+\t\t\tif (mod) {\n -+\n -+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n -+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n -+\t\t\t\t\t\tresult = AUTH_ALLOW;\n -+\t\t\t\t\t\tgoto done;\n -+\t\t\t\t\t}\n -+\t\t\t\t}\n -+\n -+\t\t\t\tif (result != AUTH_UNKNOWN)\n -+\t\t\t\t\tgoto done;\n -+\t\t\t}\n -+\t\t}\n -+\t}\n -+\n -+done:\n -+\tswitch (result) {\n -+\tcase AUTH_ALLOW:\n -+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n -+\t\t*user = \"VALID_TEST_USER\";\n -+\t\t*wr = WR_OK;\n -+\t\tbreak;\n -+\n -+\tcase AUTH_DENY:\n -+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n -+\t\t/* fall-through */\n -+\n -+\tcase AUTH_UNKNOWN:\n -+\t\tif (allow_anonymous)\n -+\t\t\tbreak;\n -+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n -+\t\t\tmod = auth_modules[i];\n -+\t\t\tif (mod->challenge_params)\n -+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n -+\t\t\t\t\t\t mod->scheme,\n -+\t\t\t\t\t\t mod->challenge_params);\n -+\t\t\telse\n -+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n -+\t\t\t\t\t\t mod->scheme);\n -+\t\t\tstring_list_append(&hdrs, challenge);\n -+\t\t}\n -+\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n -+\t}\n -+\n -+\tstrbuf_list_free(split);\n -+\tstring_list_clear(&hdrs, 0);\n -+\n -+\treturn result == AUTH_ALLOW ||\n -+\t (result == AUTH_UNKNOWN && allow_anonymous);\n -+}\n -+\n -+static enum worker_result dispatch(struct req *req)\n -+{\n -+\tenum worker_result wr = WR_OK;\n -+\tconst char *user = NULL;\n -+\n -+\tif (!is_authed(req, &user, &wr))\n -+\t\treturn wr;\n -+\n -+\tif (is_git_request(req))\n -+\t\treturn do__git(req, user);\n -+\n -+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n -+\t\t\t WR_OK | WR_HANGUP);\n -+}\n -+\n +static enum worker_result worker(void)\n +{\n -+\tstruct req req = REQ__INIT;\n ++\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n +\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n +\tchar *client_port = getenv(\"REMOTE_PORT\");\n +\tenum worker_result wr = WR_OK;\n @@ t/helper/test-http-server.c (new)\n +\tset_keep_alive(0);\n +\n +\twhile (1) {\n -+\t\treq__release(&req);\n -+\n -+\t\talarm(init_timeout ? init_timeout : timeout);\n -+\t\twr = req__read(&req, 0);\n -+\t\talarm(0);\n -+\n -+\t\tif (wr & WR_STOP_THE_MUSIC)\n -+\t\t\tbreak;\n ++\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n ++\t\t\tlogerror(\"unable to write response\");\n ++\t\t\twr = WR_IO_ERROR;\n ++\t\t}\n +\n -+\t\twr = dispatch(&req);\n +\t\tif (wr & WR_STOP_THE_MUSIC)\n +\t\t\tbreak;\n +\t}\n @@ t/helper/test-http-server.c (new)\n +\treturn !!(wr & WR_IO_ERROR);\n +}\n +\n -+//////////////////////////////////////////////////////////////////\n -+// This section contains the listener and child-process management\n -+// code used by the primary instance to accept incoming connections\n -+// and dispatch them to async child process \"worker\" instances.\n -+//////////////////////////////////////////////////////////////////\n ++/*\n ++ * This section contains the listener and child-process management\n ++ * code used by the primary instance to accept incoming connections\n ++ * and dispatch them to async child process \"worker\" instances.\n ++ */\n +\n +static int addrcmp(const struct sockaddr_storage *s1,\n +\t\t const struct sockaddr_storage *s2)\n @@ t/helper/test-http-server.c (new)\n +\n +\tset_keep_alive(sockfd);\n +\n -+\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n ++\tif (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {\n +\t\tlogerror(\"Could not bind to %s: %s\",\n +\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n +\t\t\t strerror(errno));\n @@ t/helper/test-http-server.c (new)\n +\treturn service_loop(&socklist);\n +}\n +\n -+//////////////////////////////////////////////////////////////////\n -+// This section is executed by both the primary instance and all\n -+// worker instances. So, yes, each child-process re-parses the\n -+// command line argument and re-discovers how it should behave.\n -+//////////////////////////////////////////////////////////////////\n ++/*\n ++ * This section is executed by both the primary instance and all\n ++ * worker instances. So, yes, each child-process re-parses the\n ++ * command line argument and re-discovers how it should behave.\n ++ */\n +\n +int cmd_main(int argc, const char **argv)\n +{\n @@ t/helper/test-http-server.c (new)\n +\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n +\tint worker_mode = 0;\n +\tint i;\n -+\tstruct auth_module *mod = NULL;\n +\n +\ttrace2_cmd_name(\"test-http-server\");\n +\tsetup_git_directory_gently(NULL);\n @@ t/helper/test-http-server.c (new)\n +\t\t\tpid_file = v;\n +\t\t\tcontinue;\n +\t\t}\n -+\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n -+\t\t\tallow_anonymous = 1;\n -+\t\t\tcontinue;\n -+\t\t}\n -+\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n -+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n -+\n -+\t\t\tif (!p[0]) {\n -+\t\t\t\terror(\"invalid argument '%s'\", v);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\t// trim trailing ':'\n -+\t\t\tif (p[1])\n -+\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n -+\n -+\t\t\tif (get_auth_module(p[0])) {\n -+\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\tmod = xmalloc(sizeof(struct auth_module));\n -+\t\t\tmod->scheme = xstrdup(p[0]->buf);\n -+\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n -+\t\t\tmod->tokens = xmalloc(sizeof(struct string_list));\n -+\t\t\tstring_list_init_dup(mod->tokens);\n -+\n -+\t\t\tadd_auth_module(mod);\n -+\n -+\t\t\tstrbuf_list_free(p);\n -+\t\t\tcontinue;\n -+\t\t}\n -+\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n -+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n -+\t\t\tif (!p[0]) {\n -+\t\t\t\terror(\"invalid argument '%s'\", v);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\tif (!p[1]) {\n -+\t\t\t\terror(\"missing token value '%s'\\n\", v);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\t// trim trailing ':'\n -+\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n -+\n -+\t\t\tmod = get_auth_module(p[0]);\n -+\t\t\tif (!mod) {\n -+\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n -+\t\t\tstrbuf_list_free(p);\n -+\t\t\tcontinue;\n -+\t\t}\n +\n +\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n +\t\tusage(test_http_auth_usage);\n @@ t/helper/test-http-server.c (new)\n +\t */\n +\treturn serve(&listen_addr, listen_port);\n +}\n -\n - ## t/t5556-http-auth.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description='test http auth header and credential helper interop'\n -+\n -+. ./test-lib.sh\n -+\n -+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n -+\n -+# Setup a repository\n -+#\n -+REPO_DIR=\"$(pwd)\"/repo\n -+\n -+# Setup some lookback URLs where test-http-server will be listening.\n -+# We will spawn it directly inside the repo directory, so we avoid\n -+# any need to configure directory mappings etc - we only serve this\n -+# repository from the root '/' of the server.\n -+#\n -+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n -+ORIGIN_URL=http://$HOST_PORT/\n -+\n -+# The pid-file is created by test-http-server when it starts.\n -+# The server will shutdown if/when we delete it (this is easier than\n -+# killing it by PID).\n -+#\n -+PID_FILE=\"$(pwd)\"/pid-file.pid\n -+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n -+\n -+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n -+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n -+\t&& export CREDENTIAL_HELPER\n -+\n -+test_expect_success 'setup repos' '\n -+\ttest_create_repo \"$REPO_DIR\" &&\n -+\tgit -C \"$REPO_DIR\" branch -M main\n -+'\n -+\n -+stop_http_server () {\n -+\tif ! test -f \"$PID_FILE\"\n -+\tthen\n -+\t\treturn 0\n -+\tfi\n -+\t#\n -+\t# The server will shutdown automatically when we delete the pid-file.\n -+\t#\n -+\trm -f \"$PID_FILE\"\n -+\t#\n -+\t# Give it a few seconds to shutdown (mainly to completely release the\n -+\t# port before the next test start another instance and it attempts to\n -+\t# bind to it).\n -+\t#\n -+\tfor k in 0 1 2 3 4\n -+\tdo\n -+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n -+\t\tthen\n -+\t\t\treturn 0\n -+\t\tfi\n -+\t\tsleep 1\n -+\tdone\n -+\n -+\techo \"stop_http_server: timeout waiting for server shutdown\"\n -+\treturn 1\n -+}\n -+\n -+start_http_server () {\n -+\t#\n -+\t# Launch our server into the background in repo_dir.\n -+\t#\n -+\t(\n -+\t\tcd \"$REPO_DIR\"\n -+\t\ttest-http-server --verbose \\\n -+\t\t\t--listen=127.0.0.1 \\\n -+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n -+\t\t\t--reuseaddr \\\n -+\t\t\t--pid-file=\"$PID_FILE\" \\\n -+\t\t\t\"$@\" \\\n -+\t\t\t2>\"$SERVER_LOG\" &\n -+\t)\n -+\t#\n -+\t# Give it a few seconds to get started.\n -+\t#\n -+\tfor k in 0 1 2 3 4\n -+\tdo\n -+\t\tif test -f \"$PID_FILE\"\n -+\t\tthen\n -+\t\t\treturn 0\n -+\t\tfi\n -+\t\tsleep 1\n -+\tdone\n -+\n -+\techo \"start_http_server: timeout waiting for server startup\"\n -+\treturn 1\n -+}\n -+\n -+per_test_cleanup () {\n -+\tstop_http_server &&\n -+\trm -f OUT.* &&\n -+\trm -f *.cred\n -+}\n -+\n -+test_expect_success 'http auth anonymous no challenge' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server --allow-anonymous &&\n -+\n -+\t# Attempt to read from a protected repository\n -+\tgit ls-remote $ORIGIN_URL\n -+'\n -+\n -+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=bearer:secret-token &&\n -+\n -+\tcat >get-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >store-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-token\n -+\tauthtype=bearer\n -+\tEOF\n -+\n -+\tcat >get-response.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-token\n -+\tauthtype=bearer\n -+\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n -+'\n -+\n -+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=basic:$USERPASS64 &&\n -+\n -+\tcat >get-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >store-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n -+\tauthtype=basic\n -+\tEOF\n -+\n -+\tcat >get-response.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n -+\tauthtype=basic\n -+\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n -+'\n -+\n -+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server \\\n -+\t\t--auth=foobar:alg=test\\ widget=1 \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=foobar:SECRET-FOOBAR-VALUE &&\n -+\n -+\tcat >get-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=foobar alg=test widget=1\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >store-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=SECRET-FOOBAR-VALUE\n -+\tauthtype=foobar\n -+\tEOF\n -+\n -+\tcat >get-response.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=SECRET-FOOBAR-VALUE\n -+\tauthtype=foobar\n -+\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n -+'\n -+\n -+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=bearer:secret-token &&\n -+\n -+\tcat >get-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >erase-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=invalid-token\n -+\tauthtype=bearer\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >get-response.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=invalid-token\n -+\tauthtype=bearer\n -+\tEOF\n -+\n -+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp erase-expected.cred erase-actual.cred\n -+'\n -+\n -+test_done\n -: ----------- > 7: 93bdf1d7060 test-http-server: add HTTP error response function\n -: ----------- > 8: b3e9156755f test-http-server: add HTTP request parsing\n -: ----------- > 9: 5fb248c074a test-http-server: pass Git requests to http-backend\n -: ----------- > 10: 192f09b9de4 test-http-server: add simple authentication\n -: ----------- > 11: b64d2f2c473 t5556: add HTTP authentication tests\n\n-- \ngitgitgadget\n"},{"id":"466339","messageId":"4d75ca29cc5506031c003998b3d27709d0e541b9.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 04/11] http: move proactive auth to first slot creation","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:22Z","receivedAt":"2022-11-02T22:09:47Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRather than proactively seek credentials to authenticate a request at\n`http_init()` time, do it when the first `active_request_slot` is\ncreated.\n\nBecause credential helpers may modify the headers used for a request we\ncan only auth when a slot is created (when we can first start to gather\nrequest headers).\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 20 +++++++++++---------\n 1 file changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex f2ebb17c8c4..17b47195d22 100644\n--- a/http.c\n+++ b/http.c\n@@ -515,18 +515,18 @@ static int curl_empty_auth_enabled(void)\n \treturn 0;\n }\n \n-static void init_curl_http_auth(CURL *result)\n+static void init_curl_http_auth(struct active_request_slot *slot)\n {\n \tif (!http_auth.username || !*http_auth.username) {\n \t\tif (curl_empty_auth_enabled())\n-\t\t\tcurl_easy_setopt(result, CURLOPT_USERPWD, \":\");\n+\t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n \t}\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(result, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(result, CURLOPT_PASSWORD, http_auth.password);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n }\n \n /* *var must be free-able */\n@@ -901,9 +901,6 @@ static CURL *get_curl_handle(void)\n #endif\n \t}\n \n-\tif (http_proactive_auth)\n-\t\tinit_curl_http_auth(result);\n-\n \tif (getenv(\"GIT_SSL_VERSION\"))\n \t\tssl_version = getenv(\"GIT_SSL_VERSION\");\n \tif (ssl_version && *ssl_version) {\n@@ -1260,6 +1257,7 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n \n+\tint proactive_auth = 0;\n \tint num_transfers;\n \n \t/* Wait for a slot to open up if the queue is full */\n@@ -1282,6 +1280,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n \t\t\tactive_queue_head = newslot;\n+\n+\t\t\t/* Auth first slot if asked for proactive auth */\n+\t\t\tproactive_auth = http_proactive_auth;\n \t\t} else {\n \t\t\twhile (slot->next != NULL)\n \t\t\t\tslot = slot->next;\n@@ -1336,8 +1337,9 @@ struct active_request_slot *get_active_slot(int no_pragma_header)\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);\n-\tif (http_auth.password || curl_empty_auth_enabled())\n-\t\tinit_curl_http_auth(slot->curl);\n+\n+\tif (http_auth.password || curl_empty_auth_enabled() || proactive_auth)\n+\t\tinit_curl_http_auth(slot);\n \n \treturn slot;\n }\n-- \ngitgitgadget\n\n"},{"id":"466341","messageId":"2f38427aa8db188060d153d8ece9503e1b604e91.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 05/11] http: set specific auth scheme depending on credential","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:23Z","receivedAt":"2022-11-02T22:09:49Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a new credential field `authtype` that can be used by\ncredential helpers to indicate the type of the credential or\nauthentication mechanism to use for a request.\n\nModify http.c to now specify the correct authentication scheme or\ncredential type when authenticating the curl handle. If the new\n`authtype` field in the credential structure is `NULL` or \"Basic\" then\nuse the existing username/password options. If the field is \"Bearer\"\nthen use the OAuth bearer token curl option. Otherwise, the `authtype`\nfield is the authentication scheme and the `password` field is the\nraw, unencoded value.\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 12 ++++++++++++\n credential.c | 5 +++++\n credential.h | 1 +\n git-curl-compat.h | 10 ++++++++++\n http.c | 24 +++++++++++++++++++++---\n 5 files changed, 49 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex 791a57dddfb..9069bfb2d50 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -175,6 +175,18 @@ username in the example above) will be left unset.\n \tattribute 'wwwauth[]', where the order of the attributes is the same as\n \tthey appear in the HTTP response.\n \n+`authtype`::\n+\n+\tIndicates the type of authentication scheme that should be used by Git.\n+\tCredential helpers may reply to a request from Git with this attribute,\n+\tsuch that subsequent authenticated requests include the correct\n+\t`Authorization` header.\n+\tIf this attribute is not present, the default value is \"Basic\".\n+\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n+\tIf an unknown value is provided, this is taken as the authentication\n+\tscheme for the `Authorization` header, and the `password` field is\n+\tused as the raw unencoded authorization parameters of the same header.\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/credential.c b/credential.c\nindex 8a3ad6c0ae2..a556f9f375a 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -21,6 +21,7 @@ void credential_clear(struct credential *c)\n \tfree(c->path);\n \tfree(c->username);\n \tfree(c->password);\n+\tfree(c->authtype);\n \tstring_list_clear(&c->helpers, 0);\n \tstrvec_clear(&c->wwwauth_headers);\n \n@@ -235,6 +236,9 @@ int credential_read(struct credential *c, FILE *fp)\n \t\t} else if (!strcmp(key, \"path\")) {\n \t\t\tfree(c->path);\n \t\t\tc->path = xstrdup(value);\n+\t\t} else if (!strcmp(key, \"authtype\")) {\n+\t\t\tfree(c->authtype);\n+\t\t\tc->authtype = xstrdup(value);\n \t\t} else if (!strcmp(key, \"url\")) {\n \t\t\tcredential_from_url(c, value);\n \t\t} else if (!strcmp(key, \"quit\")) {\n@@ -281,6 +285,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_item(fp, \"authtype\", c->authtype, 0);\n \tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \ndiff --git a/credential.h b/credential.h\nindex 6f2e5bc610b..8d580b054d0 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -140,6 +140,7 @@ struct credential {\n \tchar *protocol;\n \tchar *host;\n \tchar *path;\n+\tchar *authtype;\n };\n \n #define CREDENTIAL_INIT { \\\ndiff --git a/git-curl-compat.h b/git-curl-compat.h\nindex 56a83b6bbd8..839049f6dfe 100644\n--- a/git-curl-compat.h\n+++ b/git-curl-compat.h\n@@ -126,4 +126,14 @@\n #define GIT_CURL_HAVE_CURLSSLSET_NO_BACKENDS\n #endif\n \n+/**\n+ * CURLAUTH_BEARER was added in 7.61.0, released in July 2018.\n+ * However, only 7.69.0 fixes a bug where Bearer headers were not\n+ * actually sent with reused connections on subsequent transfers\n+ * (curl/curl@dea17b519dc1).\n+ */\n+#if LIBCURL_VERSION_NUM >= 0x074500\n+#define GIT_CURL_HAVE_CURLAUTH_BEARER\n+#endif\n+\n #endif\ndiff --git a/http.c b/http.c\nindex 17b47195d22..ac620bcbf0c 100644\n--- a/http.c\n+++ b/http.c\n@@ -517,7 +517,8 @@ static int curl_empty_auth_enabled(void)\n \n static void init_curl_http_auth(struct active_request_slot *slot)\n {\n-\tif (!http_auth.username || !*http_auth.username) {\n+\tif (!http_auth.authtype &&\n+\t\t(!http_auth.username || !*http_auth.username)) {\n \t\tif (curl_empty_auth_enabled())\n \t\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERPWD, \":\");\n \t\treturn;\n@@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n \n \tcredential_fill(&http_auth);\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n+\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n+\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n+\t\t\thttp_auth.username);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n+\t\t\thttp_auth.password);\n+#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n+\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n+\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n+\t\t\thttp_auth.password);\n+#endif\n+\t} else {\n+\t\tstruct strbuf auth = STRBUF_INIT;\n+\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n+\t\t\thttp_auth.authtype, http_auth.password);\n+\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n+\t\tstrbuf_release(&auth);\n+\t}\n }\n \n /* *var must be free-able */\n-- \ngitgitgadget\n\n"},{"id":"466342","messageId":"65ac638b8a077c04687d9cf3d33c7480024034ea.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 03/11] http: store all request headers on active_request_slot","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:21Z","receivedAt":"2022-11-02T22:09:53Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nOnce a list of headers has been set on the curl handle, it is not\npossible to recover that `struct curl_slist` instance to add or modify\nheaders.\n\nIn future commits we will want to modify the set of request headers in\nresponse to an authentication challenge/401 response from the server,\nwith information provided by a credential helper.\n\nThere are a number of different places where curl is used for an HTTP\nrequest, and they do not have a common handling of request headers.\nHowever, given that they all do call the `start_active_slot()` function,\neither directly or indirectly via `run_slot()` or `run_one_slot()`, we\nuse this as the point to set the `CURLOPT_HTTPHEADER` option just\nbefore the request is made.\n\nWe collect all request headers in a `struct curl_slist` on the\n`struct active_request_slot` that is obtained from a call to\n`get_active_slot(int)`. This function now takes a single argument to\ndefine if the initial set of headers on the slot should include the\n\"Pragma: no-cache\" header, along with all extra headers specified via\n`http.extraHeader` config values.\n\nThe active request slot obtained from `get_active_slot(int)` will always\ncontain a fresh set of default headers and any headers set in previous\nusages of this slot will be freed.\n\nSigned-off-by: Matthew John Cheetham \n---\n http-push.c | 103 ++++++++++++++++++++++----------------------------\n http-walker.c | 2 +-\n http.c | 82 ++++++++++++++++++----------------------\n http.h | 4 +-\n remote-curl.c | 36 +++++++++---------\n 5 files changed, 101 insertions(+), 126 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 5f4340a36e6..2b40959b376 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -211,29 +211,29 @@ static void curl_setup_http(CURL *curl, const char *url,\n \tcurl_easy_setopt(curl, CURLOPT_UPLOAD, 1);\n }\n \n-static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum dav_header_flag options)\n+static struct curl_slist *append_dav_token_headers(struct curl_slist *headers,\n+\tstruct remote_lock *lock, enum dav_header_flag options)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n \tif (options & DAV_HEADER_IF) {\n \t\tstrbuf_addf(&buf, \"If: (<%s>)\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_LOCK) {\n \t\tstrbuf_addf(&buf, \"Lock-Token: <%s>\", lock->token);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tif (options & DAV_HEADER_TIMEOUT) {\n \t\tstrbuf_addf(&buf, \"Timeout: Second-%ld\", lock->timeout);\n-\t\tdav_headers = curl_slist_append(dav_headers, buf.buf);\n+\t\theaders = curl_slist_append(headers, buf.buf);\n \t\tstrbuf_reset(&buf);\n \t}\n \tstrbuf_release(&buf);\n \n-\treturn dav_headers;\n+\treturn headers;\n }\n \n static void finish_request(struct transfer_request *request);\n@@ -281,7 +281,7 @@ static void start_mkcol(struct transfer_request *request)\n \n \trequest->url = get_remote_object_url(repo->url, hex, 1);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MKCOL);\n@@ -399,7 +399,7 @@ static void start_put(struct transfer_request *request)\n \tstrbuf_add(&buf, request->lock->tmpfile_suffix, the_hash_algo->hexsz + 1);\n \trequest->url = strbuf_detach(&buf, NULL);\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http(slot->curl, request->url, DAV_PUT,\n@@ -417,15 +417,13 @@ static void start_put(struct transfer_request *request)\n static void start_move(struct transfer_request *request)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \tcurl_setup_http_get(slot->curl, request->url, DAV_MOVE);\n-\tdav_headers = curl_slist_append(dav_headers, request->dest);\n-\tdav_headers = curl_slist_append(dav_headers, \"Overwrite: T\");\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n+\tslot->headers = curl_slist_append(slot->headers, request->dest);\n+\tslot->headers = curl_slist_append(slot->headers, \"Overwrite: T\");\n \n \tif (start_active_slot(slot)) {\n \t\trequest->slot = slot;\n@@ -440,17 +438,16 @@ static int refresh_lock(struct remote_lock *lock)\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n \tlock->refreshing = 1;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF | DAV_HEADER_TIMEOUT);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_LOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -464,7 +461,6 @@ static int refresh_lock(struct remote_lock *lock)\n \t}\n \n \tlock->refreshing = 0;\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn rc;\n }\n@@ -838,7 +834,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tchar *ep;\n \tchar timeout_header[25];\n \tstruct remote_lock *lock = NULL;\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tchar *escaped;\n \n@@ -849,7 +844,7 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \twhile (ep) {\n \t\tchar saved_character = ep[1];\n \t\tep[1] = '\\0';\n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n \t\tcurl_setup_http_get(slot->curl, url, DAV_MKCOL);\n \t\tif (start_active_slot(slot)) {\n@@ -875,14 +870,15 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \tstrbuf_addf(&out_buffer.buf, LOCK_REQUEST, escaped);\n \tfree(escaped);\n \n+\tslot = get_active_slot(0);\n+\tslot->results = &results;\n+\n \txsnprintf(timeout_header, sizeof(timeout_header), \"Timeout: Second-%ld\", timeout);\n-\tdav_headers = curl_slist_append(dav_headers, timeout_header);\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n+\tslot->headers = curl_slist_append(slot->headers, timeout_header);\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n \n-\tslot = get_active_slot();\n-\tslot->results = &results;\n \tcurl_setup_http(slot->curl, url, DAV_LOCK, &out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tCALLOC_ARRAY(lock, 1);\n@@ -921,7 +917,6 @@ static struct remote_lock *lock_remote(const char *path, long timeout)\n \t\tfprintf(stderr, \"Unable to start LOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n \n@@ -945,15 +940,14 @@ static int unlock_remote(struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct remote_lock *prev = repo->locks;\n-\tstruct curl_slist *dav_headers;\n \tint rc = 0;\n \n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_LOCK);\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_LOCK);\n+\n \tcurl_setup_http_get(slot->curl, lock->url, DAV_UNLOCK);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -966,8 +960,6 @@ static int unlock_remote(struct remote_lock *lock)\n \t\tfprintf(stderr, \"Unable to start UNLOCK request\\n\");\n \t}\n \n-\tcurl_slist_free_all(dav_headers);\n-\n \tif (repo->locks == lock) {\n \t\trepo->locks = lock->next;\n \t} else {\n@@ -1121,7 +1113,6 @@ static void remote_ls(const char *path, int flags,\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tstruct remote_ls_ctx ls;\n \n@@ -1134,14 +1125,14 @@ static void remote_ls(const char *path, int flags,\n \n \tstrbuf_addstr(&out_buffer.buf, PROPFIND_ALL_REQUEST);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 1\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 1\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1177,7 +1168,6 @@ static void remote_ls(const char *path, int flags,\n \tfree(url);\n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n }\n \n static void get_remote_object_list(unsigned char parent)\n@@ -1199,7 +1189,6 @@ static int locking_available(void)\n \tstruct slot_results results;\n \tstruct strbuf in_buffer = STRBUF_INIT;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers = http_copy_default_headers();\n \tstruct xml_ctx ctx;\n \tint lock_flags = 0;\n \tchar *escaped;\n@@ -1208,14 +1197,14 @@ static int locking_available(void)\n \tstrbuf_addf(&out_buffer.buf, PROPFIND_SUPPORTEDLOCK_REQUEST, escaped);\n \tfree(escaped);\n \n-\tdav_headers = curl_slist_append(dav_headers, \"Depth: 0\");\n-\tdav_headers = curl_slist_append(dav_headers, \"Content-Type: text/xml\");\n-\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = curl_slist_append(slot->headers, \"Depth: 0\");\n+\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\"Content-Type: text/xml\");\n+\n \tcurl_setup_http(slot->curl, repo->url, DAV_PROPFIND,\n \t\t\t&out_buffer, fwrite_buffer);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &in_buffer);\n \n \tif (start_active_slot(slot)) {\n@@ -1257,7 +1246,6 @@ static int locking_available(void)\n \n \tstrbuf_release(&out_buffer.buf);\n \tstrbuf_release(&in_buffer);\n-\tcurl_slist_free_all(dav_headers);\n \n \treturn lock_flags;\n }\n@@ -1374,17 +1362,16 @@ static int update_remote(const struct object_id *oid, struct remote_lock *lock)\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n \tstruct buffer out_buffer = { STRBUF_INIT, 0 };\n-\tstruct curl_slist *dav_headers;\n-\n-\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n \tstrbuf_addf(&out_buffer.buf, \"%s\\n\", oid_to_hex(oid));\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n+\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\tDAV_HEADER_IF);\n+\n \tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t&out_buffer, fwrite_null);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n@@ -1486,18 +1473,18 @@ static void update_remote_info_refs(struct remote_lock *lock)\n \tstruct buffer buffer = { STRBUF_INIT, 0 };\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *dav_headers;\n \n \tremote_ls(\"refs/\", (PROCESS_FILES | RECURSIVE),\n \t\t add_remote_info_ref, &buffer.buf);\n \tif (!aborted) {\n-\t\tdav_headers = get_dav_token_headers(lock, DAV_HEADER_IF);\n \n-\t\tslot = get_active_slot();\n+\t\tslot = get_active_slot(0);\n \t\tslot->results = &results;\n+\t\tslot->headers = append_dav_token_headers(slot->headers, lock,\n+\t\t\tDAV_HEADER_IF);\n+\n \t\tcurl_setup_http(slot->curl, lock->url, DAV_PUT,\n \t\t\t\t&buffer, fwrite_null);\n-\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);\n \n \t\tif (start_active_slot(slot)) {\n \t\t\trun_active_slot(slot);\n@@ -1652,7 +1639,7 @@ static int delete_remote_branch(const char *pattern, int force)\n \tif (dry_run)\n \t\treturn 0;\n \turl = xstrfmt(\"%s%s\", repo->url, remote_ref->name);\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->results = &results;\n \tcurl_setup_http_get(slot->curl, url, DAV_DELETE);\n \tif (start_active_slot(slot)) {\ndiff --git a/http-walker.c b/http-walker.c\nindex b8f0f98ae14..8747de2fcdb 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -373,7 +373,7 @@ static void fetch_alternates(struct walker *walker, const char *base)\n \t * Use a callback to process the result, since another request\n \t * may fail and need to have alternates loaded before continuing\n \t */\n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \tslot->callback_func = process_alternates_response;\n \talt_req.walker = walker;\n \tslot->callback_data = &alt_req;\ndiff --git a/http.c b/http.c\nindex 03d43d352e7..f2ebb17c8c4 100644\n--- a/http.c\n+++ b/http.c\n@@ -124,8 +124,6 @@ static unsigned long empty_auth_useless =\n \t| CURLAUTH_DIGEST_IE\n \t| CURLAUTH_DIGEST;\n \n-static struct curl_slist *pragma_header;\n-static struct curl_slist *no_pragma_header;\n static struct string_list extra_http_headers = STRING_LIST_INIT_DUP;\n \n static struct curl_slist *host_resolutions;\n@@ -1133,11 +1131,6 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n-\tpragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma: no-cache\");\n-\tno_pragma_header = curl_slist_append(http_copy_default_headers(),\n-\t\t\"Pragma:\");\n-\n \t{\n \t\tchar *http_max_requests = getenv(\"GIT_HTTP_MAX_REQUESTS\");\n \t\tif (http_max_requests)\n@@ -1199,6 +1192,8 @@ void http_cleanup(void)\n \n \twhile (slot != NULL) {\n \t\tstruct active_request_slot *next = slot->next;\n+\t\tif (slot->headers)\n+\t\t\tcurl_slist_free_all(slot->headers);\n \t\tif (slot->curl) {\n \t\t\txmulti_remove_handle(slot);\n \t\t\tcurl_easy_cleanup(slot->curl);\n@@ -1215,12 +1210,6 @@ void http_cleanup(void)\n \n \tstring_list_clear(&extra_http_headers, 0);\n \n-\tcurl_slist_free_all(pragma_header);\n-\tpragma_header = NULL;\n-\n-\tcurl_slist_free_all(no_pragma_header);\n-\tno_pragma_header = NULL;\n-\n \tcurl_slist_free_all(host_resolutions);\n \thost_resolutions = NULL;\n \n@@ -1255,7 +1244,18 @@ void http_cleanup(void)\n \tFREE_AND_NULL(cached_accept_language);\n }\n \n-struct active_request_slot *get_active_slot(void)\n+static struct curl_slist *http_copy_default_headers(void)\n+{\n+\tstruct curl_slist *headers = NULL;\n+\tconst struct string_list_item *item;\n+\n+\tfor_each_string_list_item(item, &extra_http_headers)\n+\t\theaders = curl_slist_append(headers, item->string);\n+\n+\treturn headers;\n+}\n+\n+struct active_request_slot *get_active_slot(int no_pragma_header)\n {\n \tstruct active_request_slot *slot = active_queue_head;\n \tstruct active_request_slot *newslot;\n@@ -1277,6 +1277,7 @@ struct active_request_slot *get_active_slot(void)\n \t\tnewslot->curl = NULL;\n \t\tnewslot->in_use = 0;\n \t\tnewslot->next = NULL;\n+\t\tnewslot->headers = NULL;\n \n \t\tslot = active_queue_head;\n \t\tif (!slot) {\n@@ -1294,6 +1295,15 @@ struct active_request_slot *get_active_slot(void)\n \t\tcurl_session_count++;\n \t}\n \n+\tif (slot->headers)\n+\t\tcurl_slist_free_all(slot->headers);\n+\n+\tslot->headers = http_copy_default_headers();\n+\n+\tif (!no_pragma_header)\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Pragma: no-cache\");\n+\n \tactive_requests++;\n \tslot->in_use = 1;\n \tslot->results = NULL;\n@@ -1303,7 +1313,6 @@ struct active_request_slot *get_active_slot(void)\n \tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEFILE, curl_cookie_file);\n \tif (curl_save_cookies)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_COOKIEJAR, curl_cookie_file);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);\n \tcurl_easy_setopt(slot->curl, CURLOPT_RESOLVE, host_resolutions);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);\n \tcurl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, NULL);\n@@ -1335,9 +1344,12 @@ struct active_request_slot *get_active_slot(void)\n \n int start_active_slot(struct active_request_slot *slot)\n {\n-\tCURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\tCURLMcode curlm_result;\n \tint num_transfers;\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, slot->headers);\n+\tcurlm_result = curl_multi_add_handle(curlm, slot->curl);\n+\n \tif (curlm_result != CURLM_OK &&\n \t curlm_result != CURLM_CALL_MULTI_PERFORM) {\n \t\twarning(\"curl_multi_add_handle failed: %s\",\n@@ -1652,17 +1664,6 @@ int run_one_slot(struct active_request_slot *slot,\n \treturn handle_curl_result(results);\n }\n \n-struct curl_slist *http_copy_default_headers(void)\n-{\n-\tstruct curl_slist *headers = NULL;\n-\tconst struct string_list_item *item;\n-\n-\tfor_each_string_list_item(item, &extra_http_headers)\n-\t\theaders = curl_slist_append(headers, item->string);\n-\n-\treturn headers;\n-}\n-\n static CURLcode curlinfo_strbuf(CURL *curl, CURLINFO info, struct strbuf *buf)\n {\n \tchar *ptr;\n@@ -1880,12 +1881,11 @@ static int http_request(const char *url,\n {\n \tstruct active_request_slot *slot;\n \tstruct slot_results results;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n-\tstruct strbuf buf = STRBUF_INIT;\n+\tint no_cache = options && options->no_cache;\n \tconst char *accept_language;\n \tint ret;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(!no_cache);\n \tcurl_easy_setopt(slot->curl, CURLOPT_HTTPGET, 1);\n \n \tif (!result) {\n@@ -1910,27 +1910,23 @@ static int http_request(const char *url,\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-\t\theaders = curl_slist_append(headers, accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\taccept_language);\n \n-\tstrbuf_addstr(&buf, \"Pragma:\");\n-\tif (options && options->no_cache)\n-\t\tstrbuf_addstr(&buf, \" no-cache\");\n \tif (options && options->initial_request &&\n \t http_follow_config == HTTP_FOLLOW_INITIAL)\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1);\n \n-\theaders = curl_slist_append(headers, buf.buf);\n-\n \t/* Add additional headers here */\n \tif (options && options->extra_headers) {\n \t\tconst struct string_list_item *item;\n \t\tfor_each_string_list_item(item, options->extra_headers) {\n-\t\t\theaders = curl_slist_append(headers, item->string);\n+\t\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\titem->string);\n \t\t}\n \t}\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, \"\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0);\n \n@@ -1948,9 +1944,6 @@ static int http_request(const char *url,\n \t\tcurlinfo_strbuf(slot->curl, CURLINFO_EFFECTIVE_URL,\n \t\t\t\toptions->effective_url);\n \n-\tcurl_slist_free_all(headers);\n-\tstrbuf_release(&buf);\n-\n \treturn ret;\n }\n \n@@ -2311,12 +2304,10 @@ struct http_pack_request *new_direct_http_pack_request(\n \t\tgoto abort;\n \t}\n \n-\tpreq->slot = get_active_slot();\n+\tpreq->slot = get_active_slot(1);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEDATA, preq->packfile);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(preq->slot->curl, CURLOPT_URL, preq->url);\n-\tcurl_easy_setopt(preq->slot->curl, CURLOPT_HTTPHEADER,\n-\t\tno_pragma_header);\n \n \t/*\n \t * If there is data present from a previous transfer attempt,\n@@ -2481,14 +2472,13 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \t\t}\n \t}\n \n-\tfreq->slot = get_active_slot();\n+\tfreq->slot = get_active_slot(1);\n \n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEDATA, freq);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_FAILONERROR, 0);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n \tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \n \t/*\n \t * If we have successfully processed data from a previous fetch\ndiff --git a/http.h b/http.h\nindex 3c94c479100..a304cc408b2 100644\n--- a/http.h\n+++ b/http.h\n@@ -22,6 +22,7 @@ struct slot_results {\n struct active_request_slot {\n \tCURL *curl;\n \tint in_use;\n+\tstruct curl_slist *headers;\n \tCURLcode curl_result;\n \tlong http_code;\n \tint *finished;\n@@ -43,7 +44,7 @@ size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);\n curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp);\n \n /* Slot lifecycle functions */\n-struct active_request_slot *get_active_slot(void);\n+struct active_request_slot *get_active_slot(int no_pragma_header);\n int start_active_slot(struct active_request_slot *slot);\n void run_active_slot(struct active_request_slot *slot);\n void finish_all_active_slots(void);\n@@ -64,7 +65,6 @@ void step_active_slots(void);\n void http_init(struct remote *remote, const char *url,\n \t int proactive_auth);\n void http_cleanup(void);\n-struct curl_slist *http_copy_default_headers(void);\n \n extern long int git_curl_ipresolve;\n extern int active_requests;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 72dfb8fb86a..edbd4504beb 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -847,14 +847,13 @@ static int run_slot(struct active_request_slot *slot,\n static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tstruct strbuf buf = STRBUF_INIT;\n \tint err;\n \n-\tslot = get_active_slot();\n+\tslot = get_active_slot(0);\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -862,13 +861,11 @@ static int probe_rpc(struct rpc_state *rpc, struct slot_results *results)\n \tcurl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, \"0000\");\n \tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);\n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEDATA, &buf);\n \n \terr = run_slot(slot, results);\n \n-\tcurl_slist_free_all(headers);\n \tstrbuf_release(&buf);\n \treturn err;\n }\n@@ -888,7 +885,6 @@ static curl_off_t xcurl_off_t(size_t len)\n static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_received)\n {\n \tstruct active_request_slot *slot;\n-\tstruct curl_slist *headers = http_copy_default_headers();\n \tint use_gzip = rpc->gzip_request;\n \tchar *gzip_body = NULL;\n \tsize_t gzip_size = 0;\n@@ -930,21 +926,23 @@ static int post_rpc(struct rpc_state *rpc, int stateless_connect, int flush_rece\n \t\t\tneeds_100_continue = 1;\n \t}\n \n-\theaders = curl_slist_append(headers, rpc->hdr_content_type);\n-\theaders = curl_slist_append(headers, rpc->hdr_accept);\n-\theaders = curl_slist_append(headers, needs_100_continue ?\n+retry:\n+\tslot = get_active_slot(0);\n+\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_content_type);\n+\tslot->headers = curl_slist_append(slot->headers, rpc->hdr_accept);\n+\tslot->headers = curl_slist_append(slot->headers, needs_100_continue ?\n \t\t\"Expect: 100-continue\" : \"Expect:\");\n \n \t/* Add Accept-Language header */\n \tif (rpc->hdr_accept_language)\n-\t\theaders = curl_slist_append(headers, rpc->hdr_accept_language);\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->hdr_accept_language);\n \n \t/* Add the extra Git-Protocol header */\n \tif (rpc->protocol_header)\n-\t\theaders = curl_slist_append(headers, rpc->protocol_header);\n-\n-retry:\n-\tslot = get_active_slot();\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\trpc->protocol_header);\n \n \tcurl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);\n \tcurl_easy_setopt(slot->curl, CURLOPT_POST, 1);\n@@ -955,7 +953,8 @@ retry:\n \t\t/* The request body is large and the size cannot be predicted.\n \t\t * We must use chunked encoding to send it.\n \t\t */\n-\t\theaders = curl_slist_append(headers, \"Transfer-Encoding: chunked\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Transfer-Encoding: chunked\");\n \t\trpc->initial_buffer = 1;\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);\n@@ -1002,7 +1001,8 @@ retry:\n \n \t\tgzip_size = stream.total_out;\n \n-\t\theaders = curl_slist_append(headers, \"Content-Encoding: gzip\");\n+\t\tslot->headers = curl_slist_append(slot->headers,\n+\t\t\t\"Content-Encoding: gzip\");\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);\n \t\tcurl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE_LARGE, xcurl_off_t(gzip_size));\n \n@@ -1025,7 +1025,6 @@ retry:\n \t\t}\n \t}\n \n-\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);\n \tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, rpc_in);\n \trpc_in_data.rpc = rpc;\n \trpc_in_data.slot = slot;\n@@ -1055,7 +1054,6 @@ retry:\n \tif (stateless_connect)\n \t\tpacket_response_end(rpc->in);\n \n-\tcurl_slist_free_all(headers);\n \tfree(gzip_body);\n \treturn err;\n }\n-- \ngitgitgadget\n\n"},{"id":"466343","messageId":"4947e81546a51883365d0087ce616b6b77e24a63.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 06/11] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:24Z","receivedAt":"2022-11-02T22:09:55Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a mini HTTP server helper that in the future will be enhanced\nto provide a frontend for the git-http-backend, with support for\narbitrary authentication schemes.\n\nRight now, test-http-server is a pared-down copy of the git-daemon that\nalways returns a 501 Not Implemented response to all callers.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 685 ++++++++++++++++++++++++++++\n 4 files changed, 701 insertions(+)\n create mode 100644 t/helper/test-http-server.c\n\ndiff --git a/Makefile b/Makefile\nindex d93ad956e58..39b130f711d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1500,6 +1500,8 @@ else\n \tendif\n \tBASIC_CFLAGS += $(CURL_CFLAGS)\n \n+\tTEST_PROGRAMS_NEED_X += test-http-server\n+\n \tREMOTE_CURL_PRIMARY = git-remote-http$X\n \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 787738e6fa3..45251695ce0 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -989,6 +989,19 @@ set(wrapper_scripts\n set(wrapper_test_scripts\n \ttest-fake-ssh test-tool)\n \n+if(CURL_FOUND)\n+ list(APPEND wrapper_test_scripts test-http-server)\n+\n+ add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+ target_link_libraries(test-http-server common-main)\n+\n+ if(MSVC)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+ endif()\n+endif()\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..9aa9c752997 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+/test-http-server\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..18f1f741305\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,685 @@\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+;\n+\n+/* Timeout, and initial timeout */\n+static unsigned int timeout;\n+static unsigned int init_timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+static void set_keep_alive(int sockfd)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+/*\n+ * The code in this section is used by \"worker\" instances to service\n+ * a single connection from a client. The worker talks to the client\n+ * on 0 and 1.\n+ */\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\n+\t/*\n+\t * Various errors while processing the request and/or the response.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_IO_ERROR = 1<<0,\n+\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 1<<1,\n+\n+\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n+};\n+\n+static enum worker_result worker(void)\n+{\n+\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0);\n+\n+\twhile (1) {\n+\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n+\t\t\tlogerror(\"unable to write response\");\n+\t\t\twr = WR_IO_ERROR;\n+\t\t}\n+\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(0);\n+\tclose(1);\n+\n+\treturn !!(wr & WR_IO_ERROR);\n+}\n+\n+/*\n+ * This section contains the listener and child-process management\n+ * code used by the primary instance to accept incoming connections\n+ * and dispatch them to async child process \"worker\" instances.\n+ */\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+\t\t const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+} *firstborn;\n+\n+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tnewborn = xcalloc(1, sizeof(*newborn));\n+\tlive_children++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+/*\n+ * This gets called if the number of connections grows\n+ * past \"max_connections\".\n+ *\n+ * We kill the newest connection from a duplicate IP.\n+ */\n+static void kill_some_child(void)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+static void check_dead_children(void)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tconst char *dead = \"\";\n+\t\t\tif (status)\n+\t\t\t\tdead = \" (with error)\";\n+\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\tlive_children--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child();\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children();\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\tif (!reuseaddr)\n+\t\treturn 0;\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd);\n+\n+\tif (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children();\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+/*\n+ * This section is executed by both the primary instance and all\n+ * worker instances. So, yes, each child-process re-parses the\n+ * command line argument and re-discovers how it should behave.\n+ */\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n+\t\t\tinit_timeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\n-- \ngitgitgadget\n\n"},{"id":"466344","messageId":"93bdf1d7060301d3794f83a927fe72e09274e8ab.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 07/11] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:25Z","receivedAt":"2022-11-02T22:09:57Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a function to the test-http-server test helper to write more\nfull and valid HTTP error responses, including all the standard response\nheaders like `Server` and `Date`.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 59 +++++++++++++++++++++++++++++++++----\n 1 file changed, 53 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 18f1f741305..53508639714 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -97,9 +97,59 @@ enum worker_result {\n \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n };\n \n+static enum worker_result send_http_error(\n+\tint fd,\n+\tint http_code, const char *http_code_name,\n+\tint retry_after_seconds, struct string_list *response_headers,\n+\tenum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n+\t\t http_code, http_code_name);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n static enum worker_result worker(void)\n {\n-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -110,11 +160,8 @@ static enum worker_result worker(void)\n \tset_keep_alive(0);\n \n \twhile (1) {\n-\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n-\t\t\tlogerror(\"unable to write response\");\n-\t\t\twr = WR_IO_ERROR;\n-\t\t}\n-\n+\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n+\t\t\tWR_OK | WR_HANGUP);\n \t\tif (wr & WR_STOP_THE_MUSIC)\n \t\t\tbreak;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"466345","messageId":"b3e9156755fa4d4d1b83b6b6e1816ca54e40218a.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 08/11] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:26Z","receivedAt":"2022-11-02T22:10:01Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd ability to parse HTTP requests to the test-http-server test helper.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 176 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 174 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 53508639714..7bde678e264 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -97,6 +97,42 @@ enum worker_result {\n \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n };\n \n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tssize_t content_length;\n+};\n+\n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = -1 \\\n+\t}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n static enum worker_result send_http_error(\n \tint fd,\n \tint http_code, const char *http_code_name,\n@@ -148,8 +184,136 @@ done:\n \treturn wr;\n }\n \n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_OK | WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* store common request headers separately */\n+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\treq->content_length = strtol(hv, &hp, 10);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->content_length >= 0)\n+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_OK | WR_HANGUP);\n+}\n+\n static enum worker_result worker(void)\n {\n+\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -160,8 +324,16 @@ static enum worker_result worker(void)\n \tset_keep_alive(0);\n \n \twhile (1) {\n-\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n-\t\t\tWR_OK | WR_HANGUP);\n+\t\treq__release(&req);\n+\n+\t\talarm(init_timeout ? init_timeout : timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\n+\t\twr = dispatch(&req);\n \t\tif (wr & WR_STOP_THE_MUSIC)\n \t\t\tbreak;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"466346","messageId":"5fb248c074acff1552874d98b28f746e1e43eac5.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 09/11] test-http-server: pass Git requests to http-backend","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:27Z","receivedAt":"2022-11-02T22:10:03Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nTeach the test-http-sever test helper to forward Git requests to the\n`git-http-backend`.\n\nIntroduce a new test script t5556-http-auth.sh that spins up the test\nHTTP server and attempts an `ls-remote` on the served repository,\nwithout any authentication.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 56 +++++++++++++++++++\n t/t5556-http-auth.sh | 105 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 161 insertions(+)\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 7bde678e264..9f1d6b58067 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -305,8 +305,64 @@ done:\n \treturn result;\n }\n \n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req, const char *user)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\tif (write(1, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n+\t\t\treq->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n+\t\t\t\treq->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n+\t\t\t\treq->content_type);\n+\tif (req->content_length >= 0)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n+\t\t\t\t(intmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(1);\n+\tclose(0);\n+\treturn !!res;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tif (is_git_request(req))\n+\t\treturn do__git(req, NULL);\n+\n \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n }\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..78da151f122\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,105 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+\n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$(pwd)\"/repo\n+\n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$(pwd)\"/pid-file.pid\n+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n+per_test_cleanup () {\n+\tstop_http_server &&\n+\trm -f OUT.*\n+}\n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server --allow-anonymous &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"466347","messageId":"192f09b9de4d1b93348a4d0cd3b35fd0ea9ba60d.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 10/11] test-http-server: add simple authentication","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:28Z","receivedAt":"2022-11-02T22:10:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd simple authentication to the test-http-server test helper.\nAuthentication schemes and sets of valid tokens can be specified via\ncommand-line arguments. Incoming requests are compared against the set\nof valid schemes and tokens and only approved if a matching token is\nfound, or if no auth was provided and anonymous auth is enabled.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 188 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 187 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 9f1d6b58067..9a458743d13 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -18,6 +18,8 @@ static const char test_http_auth_usage[] =\n \" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n+\" [--anonymous-allowed]\\n\"\n+\" [--auth=[:] [--auth-token=:]]*\\n\"\n ;\n \n /* Timeout, and initial timeout */\n@@ -358,10 +360,136 @@ static enum worker_result do__git(struct req *req, const char *user)\n \treturn !!res;\n }\n \n+enum auth_result {\n+\t/* No auth module matches the request. */\n+\tAUTH_UNKNOWN = 0,\n+\n+\t/* Auth module denied the request. */\n+\tAUTH_DENY = 1,\n+\n+\t/* Auth module successfully validated the request. */\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tchar *scheme;\n+\tchar *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+\n+static struct auth_module *get_auth_module(const char *scheme)\n+{\n+\tint i;\n+\tstruct auth_module *mod;\n+\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme))\n+\t\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static void add_auth_module(struct auth_module *mod)\n+{\n+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\tauth_modules[auth_modules_nr++] = mod;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/*\n+\t * Check all auth modules and try to validate the request.\n+\t * The first module that matches a valid token approves the request.\n+\t * If no module is found, or if there is no valid token, then 401 error.\n+\t * Otherwise, only permit the request if anonymous auth is enabled.\n+\t */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (!split[0] || !split[1]) continue;\n+\n+\t\t\t/* trim trailing space ' ' */\n+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(split[0]->buf);\n+\t\t\tif (mod) {\n+\t\t\t\tresult = AUTH_DENY;\n+\n+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (result != AUTH_DENY && allow_anonymous)\n+\t\t\tbreak;\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n+\t}\n+\n+\tstrbuf_list_free(split);\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n \tif (is_git_request(req))\n-\t\treturn do__git(req, NULL);\n+\t\treturn do__git(req, user);\n \n \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n@@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)\n \tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n \tint worker_mode = 0;\n \tint i;\n+\tstruct auth_module *mod = NULL;\n \n \ttrace2_cmd_name(\"test-http-server\");\n \tsetup_git_directory_gently(NULL);\n@@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n+\t\t\tallow_anonymous = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t/* trim trailing ':' */\n+\t\t\tif (p[1])\n+\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tif (get_auth_module(p[0]->buf)) {\n+\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tmod = xmalloc(sizeof(struct auth_module));\n+\t\t\tmod->scheme = xstrdup(p[0]->buf);\n+\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n+\t\t\tCALLOC_ARRAY(mod->tokens, 1);\n+\t\t\tstring_list_init_dup(mod->tokens);\n+\n+\t\t\tadd_auth_module(mod);\n+\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (!p[1]) {\n+\t\t\t\terror(\"missing token value '%s'\\n\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t/* trim trailing ':' */\n+\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(p[0]->buf);\n+\t\t\tif (!mod) {\n+\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\n-- \ngitgitgadget\n\n"},{"id":"466348","messageId":"b64d2f2c473c0b55f79930d0394b1fa8d81af7f3.1667426970.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v3 11/11] t5556: add HTTP authentication tests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-11-02T22:09:29Z","receivedAt":"2022-11-02T22:10:12Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a series of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers can respond\nto requests that contain WWW-Authenticate information with the ability\nto select the response Authenticate header scheme.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-credential-helper-replay.sh | 14 ++\n t/t5556-http-auth.sh | 157 +++++++++++++++++++++-\n 2 files changed, 170 insertions(+), 1 deletion(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n\ndiff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\nnew file mode 100755\nindex 00000000000..03e5e63dad6\n--- /dev/null\n+++ b/t/helper/test-credential-helper-replay.sh\n@@ -0,0 +1,14 @@\n+cmd=$1\n+teefile=$cmd-actual.cred\n+catfile=$cmd-response.cred\n+rm -f $teefile\n+while read line;\n+do\n+\tif test -z \"$line\"; then\n+\t\tbreak;\n+\tfi\n+\techo \"$line\" >> $teefile\n+done\n+if test \"$cmd\" = \"get\"; then\n+\tcat $catfile\n+fi\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 78da151f122..43f1791a0fe 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -26,6 +26,8 @@ PID_FILE=\"$(pwd)\"/pid-file.pid\n SERVER_LOG=\"$(pwd)\"/OUT.server.log\n \n PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n+\t&& export CREDENTIAL_HELPER\n \n test_expect_success 'setup repos' '\n \ttest_create_repo \"$REPO_DIR\" &&\n@@ -91,7 +93,8 @@ start_http_server () {\n \n per_test_cleanup () {\n \tstop_http_server &&\n-\trm -f OUT.*\n+\trm -f OUT.* &&\n+\trm -f *.cred\n }\n \n test_expect_success 'http auth anonymous no challenge' '\n@@ -102,4 +105,156 @@ test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=bearer:secret-token &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=basic:$USERPASS64 &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tauthtype=basic\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tauthtype=basic\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=foobar:alg=test\\ widget=1 \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=foobar:SECRET-FOOBAR-VALUE &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=SECRET-FOOBAR-VALUE\n+\tauthtype=foobar\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=SECRET-FOOBAR-VALUE\n+\tauthtype=foobar\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=bearer:secret-token &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >erase-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-token\n+\tauthtype=bearer\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-token\n+\tauthtype=bearer\n+\tEOF\n+\n+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp erase-expected.cred erase-actual.cred\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"466447","messageId":"CAGJzqs=o2O+W=Uecu+TJ0Nuw7FsehocKu4Dyko2iKdz5HyiKrA@mail.gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"M Hickford","fromEmail":"mirth.hickford@gmail.com","sentAt":"2022-11-03T19:00:37Z","receivedAt":"2022-11-03T19:01:22Z","isPatch":true,"sender":{"key":"mirth.hickford@gmail.com","avatar":"https://avatars.githubusercontent.com/u/105314?v=4"},"body":"On Wed, 2 Nov 2022 at 22:09, Matthew John Cheetham via GitGitGadget\n wrote:\n>\n> `authtype`::\n>\n> Indicates the type of authentication scheme that should be used by Git.\n> Credential helpers may reply to a request from Git with this attribute,\n> such that subsequent authenticated requests include the correct\n> `Authorization` header.\n> If this attribute is not present, the default value is \"Basic\".\n> Known values include \"Basic\", \"Digest\", and \"Bearer\".\n> If an unknown value is provided, this is taken as the authentication\n> scheme for the `Authorization` header, and the `password` field is\n> used as the raw unencoded authorization parameters of the same header.\n\nDo you have an example using authtype=Digest? Would the helper\npopulate the password field with the user's verbatim password or the\nDigest challenge response? Put another way, is the Digest\nchallenge-response logic in Git (libcurl) or the helper?\n\nhttps://www.rfc-editor.org/rfc/rfc7616#section-3.4\n"},{"id":"466717","messageId":"19724e50-cb49-8c74-6807-93ac70a8adba@github.com","threadId":"58425","inReplyTo":"4947e81546a51883365d0087ce616b6b77e24a63.1667426970.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 06/11] test-http-server: add stub HTTP server test helper","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-11-07T19:19:03Z","receivedAt":"2022-11-07T19:19:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/22 6:09 PM, Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Introduce a mini HTTP server helper that in the future will be enhanced\n> to provide a frontend for the git-http-backend, with support for\n> arbitrary authentication schemes.\n> \n> Right now, test-http-server is a pared-down copy of the git-daemon that\n> always returns a 501 Not Implemented response to all callers.\n\nThanks for splitting this out. I ran a diff between daemon.c and\nthis version of t/helper/test-http-server.c. Most of the diff was\nfunctionality removed from daemon.c, and the small bits that were\nnew to this file are either comments detailing how the helper\nworks or custom bits related to the test environment (like the\npid file). It was much easier to validate that these changes made\nsense.\n\nLooking good.\n\nThanks,\n-Stolee\n"},{"id":"466718","messageId":"b6068839-f4aa-3f79-192c-07e3bdfb6afb@github.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-11-07T19:23:36Z","receivedAt":"2022-11-07T19:23:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/2/22 6:09 PM, Matthew John Cheetham via GitGitGadget wrote:\n> Following from my original RFC submission [0], this submission is considered\n> ready for full review. This patch series is now based on top of current\n> master (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\n> separately submitted patches [1] to fix up the other credential helpers'\n> behaviour.\n\n> Updates in v3\n> =============\n> \n> * Split final patch that added the test-http-server in to several, easier\n> to review patches.\n> \n> * Updated wording in git-credential.txt to clarify which side of the\n> credential helper protocol is sending/receiving the new wwwauth and\n> authtype attributes.\n\nYou also updated some commit messages based on v2 feedback. Thanks!\n\nThe commit splitting you did in this version is greatly appreciated.\nI found this version to be in good shape. It's a solid foundation to\nbuild upon (if any future work is necessary).\n\nThanks,\n-Stolee\n"},{"id":"467001","messageId":"kl6l5yfn4tj7.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-11-09T23:06:36Z","receivedAt":"2022-11-09T23:06:47Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Hi Matthew!\n\nWe covered this series in Review Club. As usual, participants will send\ntheir own feedback on this thread, but you may also find the meeting\nnotes handy:\n\n https://docs.google.com/document/d/14L8BAumGTpsXpjDY8VzZ4rRtpAjuGrFSRqn3stCuS_w/edit?pli=1#\n\n\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> Background\n> ==========\n>\n> [...]\n>\n> Limitations\n> ===========\n>\n> [...]\n>\n> Goals\n> =====\n>\n> [...]\n>\n> Design Principles\n> =================\n>\n> [...]\n\nThanks for the well-written cover letter! I suspect that not many folks\nare familiar with the history and workings of credential helpers, the\ncurrent state of auth and how credential helper limitations create\nchallenges for auth.\n\nI've learned a lot reading this, and it makes the motivations of this\nseries clear :)\n\n> Proposed Changes\n> ================\n>\n> 1. Teach Git to read HTTP response headers, specifically the standard\n> WWW-Authenticate (RFC 7235 Section 4.1) headers.\n>\n> 2. Teach Git to include extra information about HTTP responses that require\n> authentication when calling credential helpers. Specifically the\n> WWW-Authenticate header information.\n> \n> Because the extra information forms an ordered list, and the existing\n> credential helper I/O format only provides for simple key=value pairs,\n> we introduce a new convention for transmitting an ordered list of\n> values. Key names that are suffixed with a C-style array syntax should\n> have values considered to form an order list, i.e. key[]=value, where\n> the order of the key=value pairs in the stream specifies the order.\n> \n> For the WWW-Authenticate header values we opt to use the key wwwauth[].\n>\n> 3. Teach Git to specify authentication schemes other than Basic in\n> subsequent HTTP requests based on credential helper responses.\n>\n\nFrom a reading of this section + the subject line, it's not immediately\nobvious that 3. also requires extending the credential helper protocol\nto include the \"authtype\" field. IMO it's significant enough to warrant\nan explicit call-out.\n"},{"id":"467006","messageId":"kl6l35ar4szz.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58425","inReplyTo":"65ac638b8a077c04687d9cf3d33c7480024034ea.1667426970.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/11] http: store all request headers on active_request_slot","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-11-09T23:18:08Z","receivedAt":"2022-11-09T23:18:13Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> Once a list of headers has been set on the curl handle, it is not\n> possible to recover that `struct curl_slist` instance to add or modify\n> headers.\n>\n> In future commits we will want to modify the set of request headers in\n> response to an authentication challenge/401 response from the server,\n> with information provided by a credential helper.\n>\n> There are a number of different places where curl is used for an HTTP\n> request, and they do not have a common handling of request headers.\n> However, given that they all do call the `start_active_slot()` function,\n> either directly or indirectly via `run_slot()` or `run_one_slot()`, we\n> use this as the point to set the `CURLOPT_HTTPHEADER` option just\n> before the request is made.\n>\n> We collect all request headers in a `struct curl_slist` on the\n> `struct active_request_slot` that is obtained from a call to\n> `get_active_slot(int)`. This function now takes a single argument to\n> define if the initial set of headers on the slot should include the\n> \"Pragma: no-cache\" header, along with all extra headers specified via\n> `http.extraHeader` config values.\n\nI admit that I'm not that familiar with the http subsystem, so I'll\nfocus on the style.\n\nIf I'm reading this patch correctly, there are two related, but distinct\nchanges:\n\n- store and modify the headers on the slot\n- change how headers are initialized and remove now-unncessary libcurl\n calls that set headers\n\nBoth are simple, but given the number of LoCs changed, I found it quite\ndifficult to track which LoCs were part of which work. Could this be\nbroken up into two patches instead, i.e.:\n\n- store headers on the slot without changing how they are initialized\n- add extra header initialization logic to get_active_slot() and remove\n the unnecessary libcurl calls\n\n"},{"id":"467007","messageId":"kl6lzgcz3ddq.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58425","inReplyTo":"2f38427aa8db188060d153d8ece9503e1b604e91.1667426970.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 05/11] http: set specific auth scheme depending on credential","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-11-09T23:40:49Z","receivedAt":"2022-11-09T23:41:14Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> From: Matthew John Cheetham \n>\n> Introduce a new credential field `authtype` that can be used by\n> credential helpers to indicate the type of the credential or\n> authentication mechanism to use for a request.\n>\n> Modify http.c to now specify the correct authentication scheme or\n> credential type when authenticating the curl handle. If the new\n> `authtype` field in the credential structure is `NULL` or \"Basic\" then\n> use the existing username/password options. If the field is \"Bearer\"\n> then use the OAuth bearer token curl option. Otherwise, the `authtype`\n> field is the authentication scheme and the `password` field is the\n> raw, unencoded value.\n>\n> Signed-off-by: Matthew John Cheetham \n> ---\n> Documentation/git-credential.txt | 12 ++++++++++++\n> credential.c | 5 +++++\n> credential.h | 1 +\n> git-curl-compat.h | 10 ++++++++++\n> http.c | 24 +++++++++++++++++++++---\n> 5 files changed, 49 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\n> index 791a57dddfb..9069bfb2d50 100644\n> --- a/Documentation/git-credential.txt\n> +++ b/Documentation/git-credential.txt\n> @@ -175,6 +175,18 @@ username in the example above) will be left unset.\n> \tattribute 'wwwauth[]', where the order of the attributes is the same as\n> \tthey appear in the HTTP response.\n> \n> +`authtype`::\n> +\n> +\tIndicates the type of authentication scheme that should be used by Git.\n> +\tCredential helpers may reply to a request from Git with this attribute,\n> +\tsuch that subsequent authenticated requests include the correct\n> +\t`Authorization` header.\n> +\tIf this attribute is not present, the default value is \"Basic\".\n> +\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n> +\tIf an unknown value is provided, this is taken as the authentication\n> +\tscheme for the `Authorization` header, and the `password` field is\n> +\tused as the raw unencoded authorization parameters of the same header.\n> +\n\n[...]\n\n> @@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n> \n> \tcredential_fill(&http_auth);\n> \n> -\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n> -\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n> +\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n> +\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n> +\t\t\thttp_auth.username);\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n> +\t\t\thttp_auth.password);\n> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n> +\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n> +\t\t\thttp_auth.password);\n> +#endif\n> +\t} else {\n> +\t\tstruct strbuf auth = STRBUF_INIT;\n> +\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n> +\t\t\thttp_auth.authtype, http_auth.password);\n> +\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n> +\t\tstrbuf_release(&auth);\n> +\t}\n\nAs expected, a \"Bearer\" authtype doesn't require passing a username to\ncurl, but as you noted in the cover letter, credential helpers were\ndesigned with username-password authentication in mind, which raises the\nquestion of what a credential helper should do with \"Bearer\"\ncredentials.\n\ne.g. it is not clear to me where the \"username\" comes from in the tests, e.g.\n\n +test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n +\t# base64(\"alice:secret-passwd\")\n +\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n +\texport USERPASS64 &&\n +\n +\tstart_http_server \\\n +\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n +\t\t--auth=basic:realm=\\\"example.com\\\" \\\n +\t\t--auth-token=basic:$USERPASS64 &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n +\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n +\tcat >store-expected.cred <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tauthtype=basic\n +\tEOF\n +\n +\tcat >get-response.cred <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tauthtype=basic\n +\tEOF\n +\n +\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n +\n +\ttest_cmp get-expected.cred get-actual.cred &&\n +\ttest_cmp store-expected.cred store-actual.cred\n +'\n\nI'm not sure how we plan to handle this. Some approaches I can see are:\n\n- We require that credential helpers set a reasonable value for\n \"username\". Presumably most credential helpers generating bearer\n tokens have some idea of user identity, so this might be reasonable,\n though it is wasteful, since we never use it in a meaningul way, e.g.\n I don't think Git asks the credential helper for \"username=alice\" and\n the credential helper decides to return the 'alice' credential instead\n of the 'bob' credential (but I could be mistaken).\n\n- We require that credential helpers set _some_ value for \"username\",\n even if it is bogus. If so, we should communicate this explicitly.\n\n- It is okay for \"username\" to be missing. This seems like the most\n elegant approach for credential helpers. I'm not sure if we're there\n yet with this series, e.g. http.c::handle_curl_result() reads:\n\n else if (results->http_code == 401) {\n if (http_auth.username && http_auth.password) {\n credential_reject(&http_auth);\n return HTTP_NOAUTH;\n\n which seems to assume both a username _and_ password. If the username\n is missing, we presumably don't send \"erase\", which might be a problem\n for revoked access tokens (though presumably not an issue for OIDC id\n tokens).\n"},{"id":"468065","messageId":"xmqq5yez76ye.fsf@gitster.g","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-11-28T09:40:25Z","receivedAt":"2022-11-28T09:40:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> Testing these new additions, I introduce a new test helper test-http-server\n> that acts as a frontend to git-http-backend; a mini HTTP server based\n> heavily on git-daemon, with simple authentication configurable by command\n> line args.\n\nI did not try to figure out the reason but the topic with its tests\nseem to break in 'seen' the linux-cmake-ctest CI job.\n\n https://github.com/git/git/actions/runs/3562942886/jobs/5985179202\n\nbut the same test does not break under usual \"make test\".\n\nCan people who are interested in the cmake-ctest stuff take a look?\n\nIt is tempting to eject the ab/cmake-nix-and-ci topic that is\nalready in 'next', under the theory that what that topic does to the\ntests \"works\" for some tests but not for others, and this topic is\nan unfortunate collateral damage whose tests weren't something the\nother topic did not support well. If the cmake-ctest stuff is in\nsuch a shape, then it may have been a bit premature to merge it\ndown.\n\nThanks.\n"},{"id":"468932","messageId":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v3.git.1667426969.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:15Z","receivedAt":"2022-12-12T21:36:35Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server based\nheavily on git-daemon, with simple authentication configurable by command\nline args.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\nMatthew John Cheetham (8):\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n test-http-server: add stub HTTP server test helper\n test-http-server: add HTTP error response function\n test-http-server: add HTTP request parsing\n test-http-server: pass Git requests to http-backend\n test-http-server: add simple authentication\n t5556: add HTTP authentication tests\n\n Documentation/git-credential.txt | 18 +-\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n credential.c | 13 +\n credential.h | 15 +\n http.c | 78 ++\n t/helper/.gitignore | 1 +\n t/helper/test-credential-helper-replay.sh | 14 +\n t/helper/test-http-server.c | 1146 +++++++++++++++++++++\n t/t5556-http-auth.sh | 223 ++++\n 10 files changed, 1522 insertions(+), 1 deletion(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n create mode 100644 t/helper/test-http-server.c\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v3:\n\n 1: f297c78f60a = 1: b5b56ccd941 http: read HTTP WWW-Authenticate response headers\n 2: e45e23406a5 ! 2: d02875dda7c credential: add WWW-Authenticate header to cred requests\n @@ Documentation/git-credential.txt: empty string.\n +\tto credential helpers.\n +\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n +\tattribute 'wwwauth[]', where the order of the attributes is the same as\n -+\tthey appear in the HTTP response.\n ++\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n ++\tto pass additional information to credential helpers.\n +\n + Unrecognised attributes are silently discarded.\n + \n GIT\n - ---\n - Part of the linkgit:git[1] suite\n \n ## credential.c ##\n @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,\n 3: 65ac638b8a0 < -: ----------- http: store all request headers on active_request_slot\n 4: 4d75ca29cc5 < -: ----------- http: move proactive auth to first slot creation\n 5: 2f38427aa8d < -: ----------- http: set specific auth scheme depending on credential\n 6: 4947e81546a = 3: 07a1845ea56 test-http-server: add stub HTTP server test helper\n 7: 93bdf1d7060 = 4: 98dd286db7c test-http-server: add HTTP error response function\n 8: b3e9156755f = 5: 5c4e36e23ee test-http-server: add HTTP request parsing\n 9: 5fb248c074a = 6: 0a0f4fd10c8 test-http-server: pass Git requests to http-backend\n 10: 192f09b9de4 = 7: 794256754c1 test-http-server: add simple authentication\n 11: b64d2f2c473 ! 8: 8ecf6383522 t5556: add HTTP authentication tests\n @@ Commit message\n t5556: add HTTP authentication tests\n \n Add a series of tests to exercise the HTTP authentication header parsing\n - and the interop with credential helpers. Credential helpers can respond\n - to requests that contain WWW-Authenticate information with the ability\n - to select the response Authenticate header scheme.\n + and the interop with credential helpers. Credential helpers will receive\n + WWW-Authenticate information in credential requests.\n \n Signed-off-by: Matthew John Cheetham \n \n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n -+test_expect_success 'http auth www-auth headers to credential helper bearer valid' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=bearer:secret-token &&\n -+\n -+\tcat >get-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+\n -+\tcat >store-expected.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-token\n -+\tauthtype=bearer\n -+\tEOF\n -+\n -+\tcat >get-response.cred <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-token\n -+\tauthtype=bearer\n -+\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n -+'\n -+\n +test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n +\t# base64(\"alice:secret-passwd\")\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\texport USERPASS64 &&\n +\n +\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n +\t\t--auth=basic:realm=\\\"example.com\\\" \\\n +\t\t--auth-token=basic:$USERPASS64 &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n +\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n -+\tauthtype=basic\n +\tEOF\n +\n +\tcat >get-response.cred <<-EOF &&\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n -+\tauthtype=basic\n +\tEOF\n +\n +\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper custom scheme' '\n ++test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n ++\t# base64(\"alice:secret-passwd\")\n ++\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n ++\texport USERPASS64 &&\n ++\n +\tstart_http_server \\\n +\t\t--auth=foobar:alg=test\\ widget=1 \\\n +\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n +\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=foobar:SECRET-FOOBAR-VALUE &&\n ++\t\t--auth-token=basic:$USERPASS64 &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n -+\tpassword=SECRET-FOOBAR-VALUE\n -+\tauthtype=foobar\n ++\tpassword=secret-passwd\n +\tEOF\n +\n +\tcat >get-response.cred <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n -+\tpassword=SECRET-FOOBAR-VALUE\n -+\tauthtype=foobar\n ++\tpassword=secret-passwd\n +\tEOF\n +\n +\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\n +test_expect_success 'http auth www-auth headers to credential helper invalid' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n ++\t# base64(\"alice:secret-passwd\")\n ++\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n ++\texport USERPASS64 &&\n +\tstart_http_server \\\n +\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n +\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=bearer:secret-token &&\n ++\t\t--auth-token=basic:$USERPASS64 &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n -+\tpassword=invalid-token\n -+\tauthtype=bearer\n ++\tpassword=invalid-passwd\n +\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n +\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n -+\tpassword=invalid-token\n -+\tauthtype=bearer\n ++\tpassword=invalid-passwd\n +\tEOF\n +\n +\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n\n-- \ngitgitgadget\n"},{"id":"468933","messageId":"b5b56ccd9419353a4bf5bc9d751a711af07d2197.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:16Z","receivedAt":"2022-12-12T21:36:36Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++++++\n http.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 94 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..c4e9cd73e14 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tconst char *z = NULL;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\tconst char **v = values->v + values->nr - 1;\n+\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n+\n+\t\tfree((void*)*v);\n+\t\t*v = append;\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (skip_iprefix(buf.buf, \"http/\", &z))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1940,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"468934","messageId":"d02875dda7c0939a0de59a47fa9eb3a73ebd29a4.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:17Z","receivedAt":"2022-12-12T21:36:39Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 18 +++++++++++++++++-\n credential.c | 12 ++++++++++++\n 2 files changed, 29 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..bf0de0e9408 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,16 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n+\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n+\tattribute 'wwwauth[]', where the order of the attributes is the same as\n+\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n+\tto pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..8a3ad6c0ae2 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tint i = 0;\n+\tconst char *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree((void*)full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\n-- \ngitgitgadget\n\n"},{"id":"468935","messageId":"98dd286db7c95b6401167c4a9b5e2336843d2629.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 4/8] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:19Z","receivedAt":"2022-12-12T21:36:41Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a function to the test-http-server test helper to write more\nfull and valid HTTP error responses, including all the standard response\nheaders like `Server` and `Date`.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 59 +++++++++++++++++++++++++++++++++----\n 1 file changed, 53 insertions(+), 6 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 18f1f741305..53508639714 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -97,9 +97,59 @@ enum worker_result {\n \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n };\n \n+static enum worker_result send_http_error(\n+\tint fd,\n+\tint http_code, const char *http_code_name,\n+\tint retry_after_seconds, struct string_list *response_headers,\n+\tenum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n+\t\t http_code, http_code_name);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n static enum worker_result worker(void)\n {\n-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -110,11 +160,8 @@ static enum worker_result worker(void)\n \tset_keep_alive(0);\n \n \twhile (1) {\n-\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n-\t\t\tlogerror(\"unable to write response\");\n-\t\t\twr = WR_IO_ERROR;\n-\t\t}\n-\n+\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n+\t\t\tWR_OK | WR_HANGUP);\n \t\tif (wr & WR_STOP_THE_MUSIC)\n \t\t\tbreak;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"468936","messageId":"07a1845ea5693fc8d3716e7f97e65d467f34a40e.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:18Z","receivedAt":"2022-12-12T21:36:44Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a mini HTTP server helper that in the future will be enhanced\nto provide a frontend for the git-http-backend, with support for\narbitrary authentication schemes.\n\nRight now, test-http-server is a pared-down copy of the git-daemon that\nalways returns a 501 Not Implemented response to all callers.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 13 +\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 685 ++++++++++++++++++++++++++++\n 4 files changed, 701 insertions(+)\n create mode 100644 t/helper/test-http-server.c\n\ndiff --git a/Makefile b/Makefile\nindex b258fdbed86..1eb795bbfd4 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1611,6 +1611,8 @@ else\n \tendif\n \tBASIC_CFLAGS += $(CURL_CFLAGS)\n \n+\tTEST_PROGRAMS_NEED_X += test-http-server\n+\n \tREMOTE_CURL_PRIMARY = git-remote-http$X\n \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 2f6e0197ffa..e9b9bfbb437 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -989,6 +989,19 @@ set(wrapper_scripts\n set(wrapper_test_scripts\n \ttest-fake-ssh test-tool)\n \n+if(CURL_FOUND)\n+ list(APPEND wrapper_test_scripts test-http-server)\n+\n+ add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+ target_link_libraries(test-http-server common-main)\n+\n+ if(MSVC)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+ set_target_properties(test-http-server\n+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+ endif()\n+endif()\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..9aa9c752997 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+/test-http-server\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..18f1f741305\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,685 @@\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+;\n+\n+/* Timeout, and initial timeout */\n+static unsigned int timeout;\n+static unsigned int init_timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+static void set_keep_alive(int sockfd)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+/*\n+ * The code in this section is used by \"worker\" instances to service\n+ * a single connection from a client. The worker talks to the client\n+ * on 0 and 1.\n+ */\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\n+\t/*\n+\t * Various errors while processing the request and/or the response.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_IO_ERROR = 1<<0,\n+\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 1<<1,\n+\n+\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n+};\n+\n+static enum worker_result worker(void)\n+{\n+\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0);\n+\n+\twhile (1) {\n+\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n+\t\t\tlogerror(\"unable to write response\");\n+\t\t\twr = WR_IO_ERROR;\n+\t\t}\n+\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(0);\n+\tclose(1);\n+\n+\treturn !!(wr & WR_IO_ERROR);\n+}\n+\n+/*\n+ * This section contains the listener and child-process management\n+ * code used by the primary instance to accept incoming connections\n+ * and dispatch them to async child process \"worker\" instances.\n+ */\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+\t\t const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+} *firstborn;\n+\n+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tnewborn = xcalloc(1, sizeof(*newborn));\n+\tlive_children++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+/*\n+ * This gets called if the number of connections grows\n+ * past \"max_connections\".\n+ *\n+ * We kill the newest connection from a duplicate IP.\n+ */\n+static void kill_some_child(void)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+static void check_dead_children(void)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tconst char *dead = \"\";\n+\t\t\tif (status)\n+\t\t\t\tdead = \" (with error)\";\n+\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\tlive_children--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child();\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children();\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\tif (!reuseaddr)\n+\t\treturn 0;\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd);\n+\n+\tif (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children();\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+/*\n+ * This section is executed by both the primary instance and all\n+ * worker instances. So, yes, each child-process re-parses the\n+ * command line argument and re-discovers how it should behave.\n+ */\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n+\t\t\tinit_timeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\n-- \ngitgitgadget\n\n"},{"id":"468937","messageId":"5c4e36e23eecbb7841078939a982b7150e2f4ab8.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:20Z","receivedAt":"2022-12-12T21:36:46Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd ability to parse HTTP requests to the test-http-server test helper.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 176 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 174 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 53508639714..7bde678e264 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -97,6 +97,42 @@ enum worker_result {\n \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n };\n \n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tssize_t content_length;\n+};\n+\n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = -1 \\\n+\t}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n static enum worker_result send_http_error(\n \tint fd,\n \tint http_code, const char *http_code_name,\n@@ -148,8 +184,136 @@ done:\n \treturn wr;\n }\n \n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_OK | WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* store common request headers separately */\n+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\treq->content_length = strtol(hv, &hp, 10);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->content_length >= 0)\n+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_OK | WR_HANGUP);\n+}\n+\n static enum worker_result worker(void)\n {\n+\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -160,8 +324,16 @@ static enum worker_result worker(void)\n \tset_keep_alive(0);\n \n \twhile (1) {\n-\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n-\t\t\tWR_OK | WR_HANGUP);\n+\t\treq__release(&req);\n+\n+\t\talarm(init_timeout ? init_timeout : timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr & WR_STOP_THE_MUSIC)\n+\t\t\tbreak;\n+\n+\t\twr = dispatch(&req);\n \t\tif (wr & WR_STOP_THE_MUSIC)\n \t\t\tbreak;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"468938","messageId":"0a0f4fd10c8b29f327c35dadc7b17881f22b253a.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 6/8] test-http-server: pass Git requests to http-backend","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:21Z","receivedAt":"2022-12-12T21:36:48Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nTeach the test-http-sever test helper to forward Git requests to the\n`git-http-backend`.\n\nIntroduce a new test script t5556-http-auth.sh that spins up the test\nHTTP server and attempts an `ls-remote` on the served repository,\nwithout any authentication.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 56 +++++++++++++++++++\n t/t5556-http-auth.sh | 105 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 161 insertions(+)\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 7bde678e264..9f1d6b58067 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -305,8 +305,64 @@ done:\n \treturn result;\n }\n \n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req, const char *user)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\tif (write(1, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n+\t\t\treq->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n+\t\t\t\treq->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n+\t\t\t\treq->content_type);\n+\tif (req->content_length >= 0)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n+\t\t\t\t(intmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(1);\n+\tclose(0);\n+\treturn !!res;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tif (is_git_request(req))\n+\t\treturn do__git(req, NULL);\n+\n \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n }\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..78da151f122\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,105 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+\n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$(pwd)\"/repo\n+\n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$(pwd)\"/pid-file.pid\n+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n+per_test_cleanup () {\n+\tstop_http_server &&\n+\trm -f OUT.*\n+}\n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server --allow-anonymous &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"468939","messageId":"794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] test-http-server: add simple authentication","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:22Z","receivedAt":"2022-12-12T21:37:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd simple authentication to the test-http-server test helper.\nAuthentication schemes and sets of valid tokens can be specified via\ncommand-line arguments. Incoming requests are compared against the set\nof valid schemes and tokens and only approved if a matching token is\nfound, or if no auth was provided and anonymous auth is enabled.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 188 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 187 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 9f1d6b58067..9a458743d13 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -18,6 +18,8 @@ static const char test_http_auth_usage[] =\n \" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n+\" [--anonymous-allowed]\\n\"\n+\" [--auth=[:] [--auth-token=:]]*\\n\"\n ;\n \n /* Timeout, and initial timeout */\n@@ -358,10 +360,136 @@ static enum worker_result do__git(struct req *req, const char *user)\n \treturn !!res;\n }\n \n+enum auth_result {\n+\t/* No auth module matches the request. */\n+\tAUTH_UNKNOWN = 0,\n+\n+\t/* Auth module denied the request. */\n+\tAUTH_DENY = 1,\n+\n+\t/* Auth module successfully validated the request. */\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tchar *scheme;\n+\tchar *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+\n+static struct auth_module *get_auth_module(const char *scheme)\n+{\n+\tint i;\n+\tstruct auth_module *mod;\n+\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme))\n+\t\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static void add_auth_module(struct auth_module *mod)\n+{\n+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\tauth_modules[auth_modules_nr++] = mod;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/*\n+\t * Check all auth modules and try to validate the request.\n+\t * The first module that matches a valid token approves the request.\n+\t * If no module is found, or if there is no valid token, then 401 error.\n+\t * Otherwise, only permit the request if anonymous auth is enabled.\n+\t */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (!split[0] || !split[1]) continue;\n+\n+\t\t\t/* trim trailing space ' ' */\n+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(split[0]->buf);\n+\t\t\tif (mod) {\n+\t\t\t\tresult = AUTH_DENY;\n+\n+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (result != AUTH_DENY && allow_anonymous)\n+\t\t\tbreak;\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n+\t}\n+\n+\tstrbuf_list_free(split);\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n \tif (is_git_request(req))\n-\t\treturn do__git(req, NULL);\n+\t\treturn do__git(req, user);\n \n \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n@@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)\n \tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n \tint worker_mode = 0;\n \tint i;\n+\tstruct auth_module *mod = NULL;\n \n \ttrace2_cmd_name(\"test-http-server\");\n \tsetup_git_directory_gently(NULL);\n@@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n+\t\t\tallow_anonymous = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t/* trim trailing ':' */\n+\t\t\tif (p[1])\n+\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tif (get_auth_module(p[0]->buf)) {\n+\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tmod = xmalloc(sizeof(struct auth_module));\n+\t\t\tmod->scheme = xstrdup(p[0]->buf);\n+\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n+\t\t\tCALLOC_ARRAY(mod->tokens, 1);\n+\t\t\tstring_list_init_dup(mod->tokens);\n+\n+\t\t\tadd_auth_module(mod);\n+\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n+\t\t\tif (!p[0]) {\n+\t\t\t\terror(\"invalid argument '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (!p[1]) {\n+\t\t\t\terror(\"missing token value '%s'\\n\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\t/* trim trailing ':' */\n+\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(p[0]->buf);\n+\t\t\tif (!mod) {\n+\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n+\t\t\tstrbuf_list_free(p);\n+\t\t\tcontinue;\n+\t\t}\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\n-- \ngitgitgadget\n\n"},{"id":"468940","messageId":"8ecf63835229676677e3f7e33f634eb5d3a568b7.1670880984.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] t5556: add HTTP authentication tests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-12-12T21:36:23Z","receivedAt":"2022-12-12T21:37:11Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a series of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-credential-helper-replay.sh | 14 +++\n t/t5556-http-auth.sh | 120 +++++++++++++++++++++-\n 2 files changed, 133 insertions(+), 1 deletion(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n\ndiff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\nnew file mode 100755\nindex 00000000000..03e5e63dad6\n--- /dev/null\n+++ b/t/helper/test-credential-helper-replay.sh\n@@ -0,0 +1,14 @@\n+cmd=$1\n+teefile=$cmd-actual.cred\n+catfile=$cmd-response.cred\n+rm -f $teefile\n+while read line;\n+do\n+\tif test -z \"$line\"; then\n+\t\tbreak;\n+\tfi\n+\techo \"$line\" >> $teefile\n+done\n+if test \"$cmd\" = \"get\"; then\n+\tcat $catfile\n+fi\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 78da151f122..541fa32bd77 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -26,6 +26,8 @@ PID_FILE=\"$(pwd)\"/pid-file.pid\n SERVER_LOG=\"$(pwd)\"/OUT.server.log\n \n PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n+\t&& export CREDENTIAL_HELPER\n \n test_expect_success 'setup repos' '\n \ttest_create_repo \"$REPO_DIR\" &&\n@@ -91,7 +93,8 @@ start_http_server () {\n \n per_test_cleanup () {\n \tstop_http_server &&\n-\trm -f OUT.*\n+\trm -f OUT.* &&\n+\trm -f *.cred\n }\n \n test_expect_success 'http auth anonymous no challenge' '\n@@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tstart_http_server \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=basic:$USERPASS64 &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tstart_http_server \\\n+\t\t--auth=foobar:alg=test\\ widget=1 \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=basic:$USERPASS64 &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\tstart_http_server \\\n+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n+\t\t--auth-token=basic:$USERPASS64 &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >erase-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\tEOF\n+\n+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp erase-expected.cred erase-actual.cred\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"468942","messageId":"AS2PR03MB981510046E40F6E943A9B981C0E29@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"kl6lzgcz3ddq.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 05/11] http: set specific auth scheme depending on credential","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-12-12T21:53:10Z","receivedAt":"2022-12-12T21:54:31Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-11-09 15:40, Glen Choo wrote:\n> \"Matthew John Cheetham via GitGitGadget\" \n> writes:\n> \n>> From: Matthew John Cheetham \n>>\n>> Introduce a new credential field `authtype` that can be used by\n>> credential helpers to indicate the type of the credential or\n>> authentication mechanism to use for a request.\n>>\n>> Modify http.c to now specify the correct authentication scheme or\n>> credential type when authenticating the curl handle. If the new\n>> `authtype` field in the credential structure is `NULL` or \"Basic\" then\n>> use the existing username/password options. If the field is \"Bearer\"\n>> then use the OAuth bearer token curl option. Otherwise, the `authtype`\n>> field is the authentication scheme and the `password` field is the\n>> raw, unencoded value.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> Documentation/git-credential.txt | 12 ++++++++++++\n>> credential.c | 5 +++++\n>> credential.h | 1 +\n>> git-curl-compat.h | 10 ++++++++++\n>> http.c | 24 +++++++++++++++++++++---\n>> 5 files changed, 49 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\n>> index 791a57dddfb..9069bfb2d50 100644\n>> --- a/Documentation/git-credential.txt\n>> +++ b/Documentation/git-credential.txt\n>> @@ -175,6 +175,18 @@ username in the example above) will be left unset.\n>> \tattribute 'wwwauth[]', where the order of the attributes is the same as\n>> \tthey appear in the HTTP response.\n>> \n>> +`authtype`::\n>> +\n>> +\tIndicates the type of authentication scheme that should be used by Git.\n>> +\tCredential helpers may reply to a request from Git with this attribute,\n>> +\tsuch that subsequent authenticated requests include the correct\n>> +\t`Authorization` header.\n>> +\tIf this attribute is not present, the default value is \"Basic\".\n>> +\tKnown values include \"Basic\", \"Digest\", and \"Bearer\".\n>> +\tIf an unknown value is provided, this is taken as the authentication\n>> +\tscheme for the `Authorization` header, and the `password` field is\n>> +\tused as the raw unencoded authorization parameters of the same header.\n>> +\n> \n> [...]\n> \n>> @@ -525,8 +526,25 @@ static void init_curl_http_auth(struct active_request_slot *slot)\n>> \n>> \tcredential_fill(&http_auth);\n>> \n>> -\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME, http_auth.username);\n>> -\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD, http_auth.password);\n>> +\tif (!http_auth.authtype || !strcasecmp(http_auth.authtype, \"basic\")\n>> +\t\t\t\t|| !strcasecmp(http_auth.authtype, \"digest\")) {\n>> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_USERNAME,\n>> +\t\t\thttp_auth.username);\n>> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_PASSWORD,\n>> +\t\t\thttp_auth.password);\n>> +#ifdef GIT_CURL_HAVE_CURLAUTH_BEARER\n>> +\t} else if (!strcasecmp(http_auth.authtype, \"bearer\")) {\n>> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BEARER);\n>> +\t\tcurl_easy_setopt(slot->curl, CURLOPT_XOAUTH2_BEARER,\n>> +\t\t\thttp_auth.password);\n>> +#endif\n>> +\t} else {\n>> +\t\tstruct strbuf auth = STRBUF_INIT;\n>> +\t\tstrbuf_addf(&auth, \"Authorization: %s %s\",\n>> +\t\t\thttp_auth.authtype, http_auth.password);\n>> +\t\tslot->headers = curl_slist_append(slot->headers, auth.buf);\n>> +\t\tstrbuf_release(&auth);\n>> +\t}\n> \n> As expected, a \"Bearer\" authtype doesn't require passing a username to\n> curl, but as you noted in the cover letter, credential helpers were\n> designed with username-password authentication in mind, which raises the\n> question of what a credential helper should do with \"Bearer\"\n> credentials.\n> \n> e.g. it is not clear to me where the \"username\" comes from in the tests, e.g.\n> \n> +test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> +\t# base64(\"alice:secret-passwd\")\n> +\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n> +\texport USERPASS64 &&\n> +\n> +\tstart_http_server \\\n> +\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n> +\t\t--auth=basic:realm=\\\"example.com\\\" \\\n> +\t\t--auth-token=basic:$USERPASS64 &&\n> +\n> +\tcat >get-expected.cred <<-EOF &&\n> +\tprotocol=http\n> +\thost=$HOST_PORT\n> +\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n> +\twwwauth[]=basic realm=\"example.com\"\n> +\tEOF\n> +\n> +\tcat >store-expected.cred <<-EOF &&\n> +\tprotocol=http\n> +\thost=$HOST_PORT\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tauthtype=basic\n> +\tEOF\n> +\n> +\tcat >get-response.cred <<-EOF &&\n> +\tprotocol=http\n> +\thost=$HOST_PORT\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tauthtype=basic\n> +\tEOF\n> +\n> +\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n> +\n> +\ttest_cmp get-expected.cred get-actual.cred &&\n> +\ttest_cmp store-expected.cred store-actual.cred\n> +'\n> \n> I'm not sure how we plan to handle this. Some approaches I can see are:\n> \n> - We require that credential helpers set a reasonable value for\n> \"username\". Presumably most credential helpers generating bearer\n> tokens have some idea of user identity, so this might be reasonable,\n> though it is wasteful, since we never use it in a meaningul way, e.g.\n> I don't think Git asks the credential helper for \"username=alice\" and\n> the credential helper decides to return the 'alice' credential instead\n> of the 'bob' credential (but I could be mistaken).\n> \n> - We require that credential helpers set _some_ value for \"username\",\n> even if it is bogus. If so, we should communicate this explicitly.\n> \n> - It is okay for \"username\" to be missing. This seems like the most\n> elegant approach for credential helpers. I'm not sure if we're there\n> yet with this series, e.g. http.c::handle_curl_result() reads:\n> \n> else if (results->http_code == 401) {\n> if (http_auth.username && http_auth.password) {\n> credential_reject(&http_auth);\n> return HTTP_NOAUTH;\n> \n> which seems to assume both a username _and_ password. If the username\n> is missing, we presumably don't send \"erase\", which might be a problem\n> for revoked access tokens (though presumably not an issue for OIDC id\n> tokens).\nYou are correct here that a missing username here may cause some unexpected\nissues, and there should be more test coverage here.\n\nMy recent v4 iteration has actually dropped the `authtype` patches here,\nand I'll pick these back up along with these concerns in a future series.\nSplitting this in to a future series is probably a good idea as I feel\nthere's going to need to be several cleanup patches adjacent to the core\nnew-feature patch, so I wouldn't want to polute this series :)\n\nThanks!\nMatthew\n\n"},{"id":"468950","messageId":"AS2PR03MB9815D67BEFBE5E48B83A5406C0E29@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"kl6l5yfn4tj7.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-12-12T22:03:05Z","receivedAt":"2022-12-12T22:03:29Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-11-09 15:06, Glen Choo wrote:\n>> Proposed Changes\n>> ================\n>>\n>> 1. Teach Git to read HTTP response headers, specifically the standard\n>> WWW-Authenticate (RFC 7235 Section 4.1) headers.\n>>\n>> 2. Teach Git to include extra information about HTTP responses that require\n>> authentication when calling credential helpers. Specifically the\n>> WWW-Authenticate header information.\n>> \n>> Because the extra information forms an ordered list, and the existing\n>> credential helper I/O format only provides for simple key=value pairs,\n>> we introduce a new convention for transmitting an ordered list of\n>> values. Key names that are suffixed with a C-style array syntax should\n>> have values considered to form an order list, i.e. key[]=value, where\n>> the order of the key=value pairs in the stream specifies the order.\n>> \n>> For the WWW-Authenticate header values we opt to use the key wwwauth[].\n>>\n>> 3. Teach Git to specify authentication schemes other than Basic in\n>> subsequent HTTP requests based on credential helper responses.\n>>\n> \n> From a reading of this section + the subject line, it's not immediately\n> obvious that 3. also requires extending the credential helper protocol\n> to include the \"authtype\" field. IMO it's significant enough to warrant\n> an explicit call-out.\nAfter some consideration I've decided to split out #3 here to a future patch\nseries. Parts 1 and 2 surround Git to credential helper contextual information\nwhich is still useful in it's own right. Part 3 should really be expanded here\nto better cover and explain the reverse helper-to-Git direction, whereby\nhelpers can modify Git's response headers to the remote.\n\nWith 1+2 most of the benefits of having an enlightened helper understand the\nauth challenge, and intelligently select identities is still possible. Remotes\njust need to continue to extract tokens from the basic Authorization header as\nthey do today until then.\n\n\nThanks,\nMatthew\n"},{"id":"468951","messageId":"AS2PR03MB9815205F73EFE1D5DBF17510C0E29@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"CAGJzqs=o2O+W=Uecu+TJ0Nuw7FsehocKu4Dyko2iKdz5HyiKrA@mail.gmail.com","subject":"Re: [PATCH v3 00/11] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2022-12-12T22:07:44Z","receivedAt":"2022-12-12T22:07:59Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-11-03 12:00, M Hickford wrote:\n> On Wed, 2 Nov 2022 at 22:09, Matthew John Cheetham via GitGitGadget\n> wrote:\n>>\n>> `authtype`::\n>>\n>> Indicates the type of authentication scheme that should be used by Git.\n>> Credential helpers may reply to a request from Git with this attribute,\n>> such that subsequent authenticated requests include the correct\n>> `Authorization` header.\n>> If this attribute is not present, the default value is \"Basic\".\n>> Known values include \"Basic\", \"Digest\", and \"Bearer\".\n>> If an unknown value is provided, this is taken as the authentication\n>> scheme for the `Authorization` header, and the `password` field is\n>> used as the raw unencoded authorization parameters of the same header.\n> \n> Do you have an example using authtype=Digest? Would the helper\n> populate the password field with the user's verbatim password or the\n> Digest challenge response? Put another way, is the Digest\n> challenge-response logic in Git (libcurl) or the helper?\n> \n> https://www.rfc-editor.org/rfc/rfc7616#section-3.4\nDigest should be handled by libcurl, but you've spotted that I missed\nconfiguring libcurl here to select digest over basic for a returned\nusername and password.\n\nYou may have noticed I've dropped these `authtype`/response config\npatches from the latest iteration (v4) as I intend to expand this part\nin a separate future series. I'll be sure to specifically test and handle\ndigest here! Thanks for spotting :)\n\nThanks,\nMatthew\n"},{"id":"469049","messageId":"c255896d-637d-f7b0-8698-10a2112852c1@github.com","threadId":"58425","inReplyTo":"b5b56ccd9419353a4bf5bc9d751a711af07d2197.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:15:06Z","receivedAt":"2022-12-14T23:15:15Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tconst char *val;\n> +\tconst char *z = NULL;\n> +\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\tstrbuf_add(&buf, ptr, size);\n> +\n> +\t/* Strip the CRLF that should be present at the end of each field */\n> +\tstrbuf_trim_trailing_newline(&buf);\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n> +\t\twhile (isspace(*val))\n> +\t\t\tval++;\n\nPer the RFC [1]: \n\n> The field value MAY be preceded by any amount of LWS, though a single SP\n> is preferred.\n\nAnd LWS (linear whitespace) is defined as:\n\n> CRLF = CR LF \n> LWS = [CRLF] 1*( SP | HT )\n\nand 'isspace()' includes CR, LF, SP, and HT [2]. \n\nLooks good!\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4-2\n[2] https://linux.die.net/man/3/isspace\n\n> +\n> +\t\tstrvec_push(values, val);\n\nI had the same question about \"what happens with an empty 'val' here?\" as\nStolee did earlier [3], but I *think* the \"zero length\" (i.e., single null\nterminator) will be copied successfully. It's probably worth testing that\nexplicitly, though (I see you set up tests in later patches - ideally a \n\"www-authenticate:\" line could be tested there).\n\n[3] https://lore.kernel.org/git/9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com/\n\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t */\n> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n> +\t\tconst char **v = values->v + values->nr - 1;\n> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n\nIn this case (where the line is a continuation of a 'www-authenticate'\nheader), it looks like the code here expects *exactly* one LWS at the start\nof the line ('isspace(*buf.buf)' requiring at least one space to append the\nheader, 'ptr + 1' skipping no more than one). But, according to the RFC, it\ncould be more than one:\n\n> Header fields can be extended over multiple lines by preceding each extra\n> line with at least one SP or HT.\n\nSo I think 'buf.buf' might need to have all preceding spaces removed, like\nyou did in the \"Start of a new WWW-Authenticate header\" block.\n\nAlso, if you're copying 'ptr' into 'buf' to avoid issues from a missing null\nterminator, wouldn't you want to use 'buf.buf' (instead of 'ptr') in\n'xstrfmt()'?\n\n> +\n> +\t\tfree((void*)*v);\n> +\t\t*v = append;\n\nI was about to suggest (optionally) rewriting this to use 'strvec_pop()' and\n'strvec_push_nodup()':\n\n\tstrvec_pop(values); \n\tstrvec_push_nodup(values, append);\n\nto maybe make this a bit easier to follow, but unfortunately\n'strvec_push_nodup()' isn't available outside of 'strvec.c'. If you did want\nto use 'strvec' functions, you could remove the 'static' from\n'strvec_push_nodup()' and add it to 'strvec.h' it in a later reroll, but I\ndon't consider that change \"blocking\" or even important enough to warrant\nits own reroll. \n\n> +\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/* This is the start of a new header we don't care about */\n> +\thttp_auth.header_is_last_match = 0;\n> +\n> +\t/*\n> +\t * If this is a HTTP status line and not a header field, this signals\n> +\t * a different HTTP response. libcurl writes all the output of all\n> +\t * response headers of all responses, including redirects.\n> +\t * We only care about the last HTTP request response's headers so clear\n> +\t * the existing array.\n> +\t */\n> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n> +\t\tstrvec_clear(values);\n\nThe comments describing the intended behavior (as well as the commit\nmessage) are clear and explain the somewhat esoteric (at least to my\nuntrained eye ;) ) code. Thanks!\n\n> +\n> +exit:\n> +\tstrbuf_release(&buf);\n> +\treturn size;\n> +}\n> +\n> size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n> {\n> \treturn nmemb;\n> @@ -1864,6 +1940,8 @@ static int http_request(const char *url,\n> \t\t\t\t\t fwrite_buffer);\n> \t}\n> \n> +\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n> +\n> \taccept_language = http_get_accept_language_header();\n> \n> \tif (accept_language)\n\n"},{"id":"469050","messageId":"35352a10-3fc0-365f-11af-b821d114c409@github.com","threadId":"58425","inReplyTo":"d02875dda7c0939a0de59a47fa9eb3a73ebd29a4.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:15:23Z","receivedAt":"2022-12-14T23:15:35Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Add the value of the WWW-Authenticate response header to credential\n> requests. Credential helpers that understand and support HTTP\n> authentication and authorization can use this standard header (RFC 2616\n> Section 14.47 [1]) to generate valid credentials.\n> \n> WWW-Authenticate headers can contain information pertaining to the\n> authority, authentication mechanism, or extra parameters/scopes that are\n> required.\n> \n> The current I/O format for credential helpers only allows for unique\n> names for properties/attributes, so in order to transmit multiple header\n> values (with a specific order) we introduce a new convention whereby a\n> C-style array syntax is used in the property name to denote multiple\n> ordered values for the same property.\n> \n> In this case we send multiple `wwwauth[]` properties where the order\n> that the repeated attributes appear in the conversation reflects the\n> order that the WWW-Authenticate headers appeared in the HTTP response.\n> \n> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\n...\n\n> +Attributes with keys that end with C-style array brackets `[]` can have\n> +multiple values. Each instance of a multi-valued attribute forms an\n> +ordered list of values - the order of the repeated attributes defines\n> +the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n> +acts to clear any previous entries and reset the list.\n> +\n\nThe commit message & documentation changes (here and the 'www-auth[]'\ndefinition below) are concise, easy-to-understand explanations of what\nyou're doing here with the 'www-authenticate' header values.\n\n> \n> @@ -160,6 +166,16 @@ empty string.\n> Components which are missing from the URL (e.g., there is no\n> username in the example above) will be left unset.\n> \n> +`wwwauth[]`::\n> +\n> +\tWhen an HTTP response is received by Git that includes one or more\n> +\t'WWW-Authenticate' authentication headers, these will be passed by Git\n> +\tto credential helpers.\n> +\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n> +\tattribute 'wwwauth[]', where the order of the attributes is the same as\n> +\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n> +\tto pass additional information to credential helpers.\n\nnit: if you're trying to get a paragraph break between \"...to credential\nhelpers.\" and \"Each 'WWW-Authenticate' header value\", you need to add an\nexplicit break:\n\n-------- 8< --------\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex bf0de0e940..50759153ef 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -171,10 +171,11 @@ username in the example above) will be left unset.\n \tWhen an HTTP response is received by Git that includes one or more\n \t'WWW-Authenticate' authentication headers, these will be passed by Git\n \tto credential helpers.\n-\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n-\tattribute 'wwwauth[]', where the order of the attributes is the same as\n-\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n-\tto pass additional information to credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n \n Unrecognised attributes are silently discarded.\n \n-------- >8 --------\n\nYou can test to see how the docs look by running 'make doc' from the\nrepository root and looking at the generated 'git-credential.html' (note\nthat, if you've installed Git dependencies with Homebrew, you might need to\nspecify 'XML_CATALOG_FILES=$(brew --prefix)/etc/xml/catalog' to get it to\nwork).\n\n> +\n> Unrecognised attributes are silently discarded.\n> \n> GIT\n> diff --git a/credential.c b/credential.c\n> index 897b4679333..8a3ad6c0ae2 100644\n> --- a/credential.c\n> +++ b/credential.c\n> @@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n> \tfprintf(fp, \"%s=%s\\n\", key, value);\n> }\n> \n> +static void credential_write_strvec(FILE *fp, const char *key,\n> +\t\t\t\t const struct strvec *vec)\n> +{\n> +\tint i = 0;\n> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n> +\tfor (; i < vec->nr; i++) {\n> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n> +\t}\n> +\tfree((void*)full_key);\n> +}\n> +\n> void credential_write(const struct credential *c, FILE *fp)\n> {\n> \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n> @@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n> \tcredential_write_item(fp, \"path\", c->path, 0);\n> \tcredential_write_item(fp, \"username\", c->username, 0);\n> \tcredential_write_item(fp, \"password\", c->password, 0);\n> +\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n\nThis implementation looks good to me.\n\n> }\n> \n> static int run_credential_helper(struct credential *c,\n\n"},{"id":"469051","messageId":"752da6b2-9c75-0f68-e507-cca02bf918ca@github.com","threadId":"58425","inReplyTo":"07a1845ea5693fc8d3716e7f97e65d467f34a40e.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/8] test-http-server: add stub HTTP server test helper","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:16:44Z","receivedAt":"2022-12-14T23:17:02Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Introduce a mini HTTP server helper that in the future will be enhanced\n> to provide a frontend for the git-http-backend, with support for\n> arbitrary authentication schemes.\n\nI really like this approach, particularly because it opens up the\npossibility of writing more fine-grained tests in other contexts (e.g.,\ntesting how a bundle-uri client handles different kinds of erroneous server\nresponses by intercepting and customizing those responses).\n\n> \n> Right now, test-http-server is a pared-down copy of the git-daemon that\n> always returns a 501 Not Implemented response to all callers.\n> \n> Signed-off-by: Matthew John Cheetham \n> ---\n> Makefile | 2 +\n> contrib/buildsystems/CMakeLists.txt | 13 +\n> t/helper/.gitignore | 1 +\n> t/helper/test-http-server.c | 685 ++++++++++++++++++++++++++++\n> 4 files changed, 701 insertions(+)\n> create mode 100644 t/helper/test-http-server.c\n> \n> diff --git a/Makefile b/Makefile\n> index b258fdbed86..1eb795bbfd4 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1611,6 +1611,8 @@ else\n> \tendif\n> \tBASIC_CFLAGS += $(CURL_CFLAGS)\n> \n> +\tTEST_PROGRAMS_NEED_X += test-http-server\n\nThis works because all usage of 'TEST_PROGRAMS_NEED_X' are either lazily\nevaluated (in the case of 'TEST_PROGRAMS') or are assigned later in the\n'Makefile' than the addition here (in the case of 'test_bindir_programs'). \n\nOn a related note, I think it would be helpful to mention 'test-http-server'\nin the \"=== Optional library: libcurl ===\" section of the documentation at\nthe top of the Makefile, to clarify that it (like 'git-http-fetch' and\n'git-http-push') are not built.\n\n> +\n> \tREMOTE_CURL_PRIMARY = git-remote-http$X\n> \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n> \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\n> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\n> index 2f6e0197ffa..e9b9bfbb437 100644\n> --- a/contrib/buildsystems/CMakeLists.txt\n> +++ b/contrib/buildsystems/CMakeLists.txt\n> @@ -989,6 +989,19 @@ set(wrapper_scripts\n> set(wrapper_test_scripts\n> \ttest-fake-ssh test-tool)\n> \n> +if(CURL_FOUND)\n> + list(APPEND wrapper_test_scripts test-http-server)\n> +\n> + add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n> + target_link_libraries(test-http-server common-main)\n> +\n> + if(MSVC)\n> + set_target_properties(test-http-server\n> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n> + set_target_properties(test-http-server\n> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n> + endif()\n> +endif()\n> \n> foreach(script ${wrapper_scripts})\n> \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\n> diff --git a/t/helper/.gitignore b/t/helper/.gitignore\n> index 8c2ddcce95f..9aa9c752997 100644\n> --- a/t/helper/.gitignore\n> +++ b/t/helper/.gitignore\n> @@ -1,2 +1,3 @@\n> /test-tool\n> /test-fake-ssh\n> +/test-http-server\n> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> new file mode 100644\n> index 00000000000..18f1f741305\n> --- /dev/null\n> +++ b/t/helper/test-http-server.c\n\nA lot of the functions in this file are modified versions of ones in\n'daemon.c'. It would help reviewers/future readers to mention that in the\ncommit message. \n\nMy comments are mostly going to be around the similarities/differences from\n'daemon.c', hopefully to understand how 'test-http-server' is meant to be\nused.\n\n> +static void logreport(const char *label, const char *err, va_list params)\n> +{\n> +\tstruct strbuf msg = STRBUF_INIT;\n> +\n> +\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n> +\tstrbuf_vaddf(&msg, err, params);\n> +\tstrbuf_addch(&msg, '\\n');\n> +\n> +\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n> +\tfflush(stderr);\n> +\n> +\tstrbuf_release(&msg);\n\nThis looks like the 'LOG_DESTINATION_STDERR' case of 'logreport()' in\n'daemon.c', but adds a \"label\" to represent the priority. Makes sense; these\nlogs will be helpful to have in stderr when running tests, and the priority\nwill be captured as well.\n\n> +}\n> +\n> +__attribute__((format (printf, 1, 2)))\n> +static void logerror(const char *err, ...)\n> +{\n> +\tva_list params;\n> +\tva_start(params, err);\n> +\tlogreport(\"error\", err, params);\n> +\tva_end(params);\n> +}\n> +\n> +__attribute__((format (printf, 1, 2)))\n> +static void loginfo(const char *err, ...)\n> +{\n> +\tva_list params;\n> +\tif (!verbose)\n> +\t\treturn;\n> +\tva_start(params, err);\n> +\tlogreport(\"info\", err, params);\n> +\tva_end(params);\n> +}\n\nThese two functions replace the \"priority\" int with the \"label\" string, but\notherwise capture the same information.\n\n> +\n> +static void set_keep_alive(int sockfd)\n\nThis function is identical to its 'daemon.c' counterpart; its usage in\n'test-http-server.c' doesn't indicate any need to differ.\n\n> +\n> +/*\n> + * The code in this section is used by \"worker\" instances to service\n> + * a single connection from a client. The worker talks to the client\n> + * on 0 and 1.\n> + */\n> +\n> +enum worker_result {\n> +\t/*\n> +\t * Operation successful.\n> +\t * Caller *might* keep the socket open and allow keep-alive.\n> +\t */\n> +\tWR_OK = 0,\n> +\n> +\t/*\n> +\t * Various errors while processing the request and/or the response.\n> +\t * Close the socket and clean up.\n> +\t * Exit child-process with non-zero status.\n> +\t */\n> +\tWR_IO_ERROR = 1<<0,\n> +\n> +\t/*\n> +\t * Close the socket and clean up. Does not imply an error.\n> +\t */\n> +\tWR_HANGUP = 1<<1,\n> +\n> +\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n\nAs much as I love the name, I'm not sure having this value defined makes\nmuch sense as its own \"state\". AFAICT, 'WR_IO_ERROR' means \"error AND exit\",\nbut 'WR_HANGUP' just means \"exit\", so the latter is a superset of the\nformer. Even if you interpret 'WR_HANGUP' as \"*no* error and exit\", that\nmakes it and 'WR_IO_ERROR' mutually exclusive, so the \"combined\" state\ndoesn't represent anything \"real\".\n\n> +};\n> +\n> +static enum worker_result worker(void)\n> +{\n> +\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n\nHere's the hardcoded 501 error, as mentioned in the commit message.\n\n> +\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n> +\tchar *client_port = getenv(\"REMOTE_PORT\");\n> +\tenum worker_result wr = WR_OK;\n> +\n> +\tif (client_addr)\n> +\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n> +\n> +\tset_keep_alive(0);\n> +\n> +\twhile (1) {\n> +\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n> +\t\t\tlogerror(\"unable to write response\");\n> +\t\t\twr = WR_IO_ERROR;\n> +\t\t}\n\nThis tries to write the response out to stdout (optional nit: you could use\n'STDOUT_FILENO' instead of '1' to make this clearer), and sets 'WR_IO_ERROR'\nif it fails. \n\n> +\n> +\t\tif (wr & WR_STOP_THE_MUSIC)\n> +\t\t\tbreak;\n\nThis will trigger if 'wr' is 'WR_HANGUP' *or* 'WR_IO_ERROR'. Is that\nintentional? If it is, I think 'wr != 'WR_OK' might make that more obvious?\n\n> +\t}\n> +\n> +\tclose(0);\n> +\tclose(1);\n> +\n> +\treturn !!(wr & WR_IO_ERROR);\n\nThen finish by closing out 'stdin' and 'stdout', and returning '0' for \"no\nerror\", '1' for \"error\".\n\n> +}\n> +\n> +/*\n> + * This section contains the listener and child-process management\n> + * code used by the primary instance to accept incoming connections\n> + * and dispatch them to async child process \"worker\" instances.\n> + */\n> +\n> +static int addrcmp(const struct sockaddr_storage *s1,\n\n\nIdentical to 'daemon.c'.\n\n> +static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n> +{\n> +\tstruct child *newborn, **cradle;\n> +\n> +\tnewborn = xcalloc(1, sizeof(*newborn));\n> +\tlive_children++;\n> +\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n> +\tmemcpy(&newborn->address, addr, addrlen);\n> +\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n> +\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n> +\t\t\tbreak;\n> +\tnewborn->next = *cradle;\n> +\t*cradle = newborn;\n> +}\n\nThis is mostly the same as 'daemon.c', but uses 'xcalloc()' instead of\n'CALLOC_ARRAY()'. The latter is an alias for the former, so this is fine.\n\n> +static void kill_some_child(void)\n\n...\n\n> +static void check_dead_children(void)\nBoth of these are identical to 'daemon.c'.\n\n> +\n> +static struct strvec cld_argv = STRVEC_INIT;\n> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n\nThis matches 'daemon.c' except for the addition of:\n\n> +\tif (cld.out < 0)\n> +\t\tlogerror(\"could not dup() `incoming`\");\n\nThe extra context provided by this message could be helpful in debugging. If\nnothing else, it doesn't hurt.\n\n> +\telse if (start_command(&cld))\n> +\t\tlogerror(\"unable to fork\");\n> +\telse\n> +\t\tadd_child(&cld, addr, addrlen);\n> +}\n> +\n> +static void child_handler(int signo)\n\n...\n\n> +static int set_reuse_addr(int sockfd)\n\n...\n\n> +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n\n...\n\n> +#ifndef NO_IPV6\n> +\n> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n...\n\n> +#else /* NO_IPV6 */\n> +\n> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n\nAll of these functions match 'daemon.c' (save for some whitespace fixups).\n\n> +\n> +static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n> +{\n> +\tif (!listen_addr->nr)\n> +\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n\nThis is the only difference in this function from 'daemon.c' (there, the\nfirst arg is 'NULL', which ends up mapping to 'INADDR_ANY'). Why the change\nin default?\n\n> +\telse {\n> +\t\tint i, socknum;\n> +\t\tfor (i = 0; i < listen_addr->nr; i++) {\n> +\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n> +\t\t\t\t\t\t listen_port, socklist);\n> +\n> +\t\t\tif (socknum == 0)\n> +\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n> +\t\t\t\t\t listen_addr->items[i].string, listen_port);\n> +\t\t}\n> +\t}\n> +}\n> +\n> +static int service_loop(struct socketlist *socklist)\n\nThis function differs from 'daemon.c' by using removal of the 'pid_file' to\nforce a graceful shutdown of the server.\n\n> +{\n> +\tstruct pollfd *pfd;\n> +\tint i;\n> +\n> +\tCALLOC_ARRAY(pfd, socklist->nr);\n> +\n> +\tfor (i = 0; i < socklist->nr; i++) {\n> +\t\tpfd[i].fd = socklist->list[i];\n> +\t\tpfd[i].events = POLLIN;\n> +\t}\n> +\n> +\tsignal(SIGCHLD, child_handler);\n> +\n> +\tfor (;;) {\n> +\t\tint i;\n> +\t\tint nr_ready;\n> +\t\tint timeout = (pid_file ? 100 : -1);\n> +\n> +\t\tcheck_dead_children();\n> +\n> +\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n\nSetting a timeout here (if 'pid_file' is present) allows us to operate in a\nmode where the removal of a 'pid_file' indicates that the server should shut\ndown.\n\n> +\t\tif (nr_ready < 0) {\n\n'nr_ready < 0' indicates an error [1]; handle the same way as 'daemon.c'.\n\n[1] https://man7.org/linux/man-pages/man2/poll.2.html\n\n> +\t\t\tif (errno != EINTR) {\n> +\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n> +\t\t\t\t strerror(errno));\n> +\t\t\t\tsleep(1);\n> +\t\t\t}\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\telse if (nr_ready == 0) {\n\n'nr_ready == 0' indicates a polling timeout (see [1] above)...\n\n> +\t\t\t/*\n> +\t\t\t * If we have a pid_file, then we watch it.\n> +\t\t\t * If someone deletes it, we shutdown the service.\n> +\t\t\t * The shell scripts in the test suite will use this.\n> +\t\t\t */\n> +\t\t\tif (!pid_file || file_exists(pid_file))\n> +\t\t\t\tcontinue;\n> +\t\t\tgoto shutdown;\n\n...and that timeout exists so that we can check whether the 'pid_file' still\nexists and, if so, shut down gracefully.\n\n> +\t\t}\n> +\n\nOtherwise, 'nr_ready > 1', so handle the polled events.\n\n> +\t\tfor (i = 0; i < socklist->nr; i++) {\n> +\t\t\tif (pfd[i].revents & POLLIN) {\n> +\t\t\t\tunion {\n> +\t\t\t\t\tstruct sockaddr sa;\n> +\t\t\t\t\tstruct sockaddr_in sai;\n> +#ifndef NO_IPV6\n> +\t\t\t\t\tstruct sockaddr_in6 sai6;\n> +#endif\n> +\t\t\t\t} ss;\n> +\t\t\t\tsocklen_t sslen = sizeof(ss);\n> +\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n> +\t\t\t\tif (incoming < 0) {\n> +\t\t\t\t\tswitch (errno) {\n> +\t\t\t\t\tcase EAGAIN:\n> +\t\t\t\t\tcase EINTR:\n> +\t\t\t\t\tcase ECONNABORTED:\n> +\t\t\t\t\t\tcontinue;\n> +\t\t\t\t\tdefault:\n> +\t\t\t\t\t\tdie_errno(\"accept returned\");\n> +\t\t\t\t\t}\n> +\t\t\t\t}\n> +\t\t\t\thandle(incoming, &ss.sa, sslen);\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +shutdown:\n> +\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n> +\tfor (i = 0; i < socklist->nr; i++)\n> +\t\tclose(socklist->list[i]);\n> +\n> +\treturn 0;\n\nThis addition logs the shutdown and closes out sockets. Looks good!\n\n> +}\n> +\n> +static int serve(struct string_list *listen_addr, int listen_port)\n> +{\n> +\tstruct socketlist socklist = { NULL, 0, 0 };\n> +\n> +\tsocksetup(listen_addr, listen_port, &socklist);\n> +\tif (socklist.nr == 0)\n> +\t\tdie(\"unable to allocate any listen sockets on port %u\",\n> +\t\t listen_port);\n> +\n> +\tloginfo(\"Ready to rumble\");\n\nI thought this was a leftover debug printout, but it turns out that\n'serve()' in 'daemon.c' has the same message. :) \n\n> +\n> +\t/*\n> +\t * Wait to create the pid-file until we've setup the sockets\n> +\t * and are open for business.\n> +\t */\n> +\tif (pid_file)\n> +\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n> +\n> +\treturn service_loop(&socklist);\n> +}\n> +\n> +/*\n> + * This section is executed by both the primary instance and all\n> + * worker instances. So, yes, each child-process re-parses the\n> + * command line argument and re-discovers how it should behave.\n> + */\n> +\n> +int cmd_main(int argc, const char **argv)\n> +{\n> +\tint listen_port = 0;\n> +\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n> +\tint worker_mode = 0;\n> +\tint i;\n> +\n> +\ttrace2_cmd_name(\"test-http-server\");\n> +\tsetup_git_directory_gently(NULL);\n\nSince this isn't part of 'test-tool', it needs to do its own trace2 setup,\nbut it seems to be missing some of the relevant function calls. Could you\ninclude 'trace2_cmd_list_config()' and 'trace2_cmd_list_env_vars()' as well? \n\n> +\n> +\tfor (i = 1; i < argc; i++) {\n\nCan this loop be replaced with 'parse_options()' and the appropriate 'struct\noption[]'? Newer test helpers ('test-bundle-uri', 'test-cache-tree',\n'test-getcwd') have been using it, and it generally seems much easier to\nwork with/more flexible than a custom 'if()' block (handling option\nnegation, interpreting both '--option=' and '--option value' syntax\netc.).\n\nThat said, it looks this was mostly pulled from 'daemon.c' (which might\npredate 'parse_options()'), so I'd also understand if you want to keep it as\nsimilar to that as possible. Up to you!\n\n> +\t/* avoid splitting a message in the middle */\n> +\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n> +\n> +\tif (listen_port == 0)\n> +\t\tlisten_port = DEFAULT_GIT_PORT;\n> +\n> +\t/*\n> +\t * If no --listen= args are given, the setup_named_sock()\n> +\t * code will use receive a NULL address and set INADDR_ANY.\n> +\t * This exposes both internal and external interfaces on the\n> +\t * port.\n> +\t *\n> +\t * Disallow that and default to the internal-use-only loopback\n> +\t * address.\n> +\t */\n> +\tif (!listen_addr.nr)\n> +\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n> +\n> +\t/*\n> +\t * worker_mode is set in our own child process instances\n> +\t * (that are bound to a connected socket from a client).\n> +\t */\n> +\tif (worker_mode)\n> +\t\treturn worker();\n> +\n> +\t/*\n> +\t * `cld_argv` is a bit of a clever hack. The top-level instance\n> +\t * of test-http-server does the normal bind/listen/accept stuff.\n> +\t * For each incoming socket, the top-level process spawns\n> +\t * a child instance of test-http-server *WITH* the additional\n> +\t * `--worker` argument. This causes the child to set `worker_mode`\n> +\t * and immediately call `worker()` using the connected socket (and\n> +\t * without the usual need for fork() or threads).\n> +\t *\n> +\t * The magic here is made possible because `cld_argv` is static\n> +\t * and handle() (called by service_loop()) knows about it.\n> +\t */\n> +\tstrvec_push(&cld_argv, argv[0]);\n> +\tstrvec_push(&cld_argv, \"--worker\");\n> +\tfor (i = 1; i < argc; ++i)\n> +\t\tstrvec_push(&cld_argv, argv[i]);\n> +\n> +\t/*\n> +\t * Setup primary instance to listen for connections.\n> +\t */\n> +\treturn serve(&listen_addr, listen_port);\n\nThe rest of the function is \"new\", but is well-documented and appears to\nwork as intended.\n\n> +}\n\nOne last note/suggestion - while a lot of the functions in\n'test-http-server.c' are modified from those in 'daemon.c', there are a fair\nnumber of identical functions as well. Would it be possible to libify some\nof 'daemon.c's functions (mainly by creating a 'daemon.h' and making the\nfunctions non-static) so that they don't need to be copied?\n\n"},{"id":"469052","messageId":"7b7d1059-cecf-744d-6927-b41963b9e5a8@github.com","threadId":"58425","inReplyTo":"98dd286db7c95b6401167c4a9b5e2336843d2629.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/8] test-http-server: add HTTP error response function","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:17:10Z","receivedAt":"2022-12-14T23:17:26Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> +static enum worker_result send_http_error(\n> +\tint fd,\n> +\tint http_code, const char *http_code_name,\n> +\tint retry_after_seconds, struct string_list *response_headers,\n> +\tenum worker_result wr_in)\n> +{\n> +\tstruct strbuf response_header = STRBUF_INIT;\n> +\tstruct strbuf response_content = STRBUF_INIT;\n> +\tstruct string_list_item *h;\n> +\tenum worker_result wr;\n> +\n> +\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n> +\t\t http_code, http_code_name);\n> +\tif (retry_after_seconds > 0)\n> +\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n> +\t\t\t retry_after_seconds);\n> +\n> +\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n> +\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n> +\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n> +\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n> +\tif (retry_after_seconds > 0)\n> +\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n> +\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n> +\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n> +\tif (response_headers)\n> +\t\tfor_each_string_list_item(h, response_headers)\n> +\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n> +\tstrbuf_addstr(&response_header, \"\\r\\n\");\n> +\n> +\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n> +\t\tlogerror(\"unable to write response header\");\n> +\t\twr = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n> +\t\tlogerror(\"unable to write response content body\");\n> +\t\twr = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\twr = wr_in;\n\nBy setting this here, if there's a 'goto done' added sometime in the future\nthat doesn't explicitly set 'wr' first, it'll trigger a compiler error.\nThat's good for a case like this, where we don't want to assume a \"default\"\nfor 'wr' before handling it.\n\n> +\n> +done:\n> +\tstrbuf_release(&response_header);\n> +\tstrbuf_release(&response_content);\n> +\n> +\treturn wr;\n> +}\n> +\n> static enum worker_result worker(void)\n> {\n> -\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n> \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n> \tchar *client_port = getenv(\"REMOTE_PORT\");\n> \tenum worker_result wr = WR_OK;\n> @@ -110,11 +160,8 @@ static enum worker_result worker(void)\n> \tset_keep_alive(0);\n> \n> \twhile (1) {\n> -\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n> -\t\t\tlogerror(\"unable to write response\");\n> -\t\t\twr = WR_IO_ERROR;\n> -\t\t}\n> -\n> +\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> +\t\t\tWR_OK | WR_HANGUP);\n\nThis is a nice incremental improvement on the original hardcoded response.\n\n> \t\tif (wr & WR_STOP_THE_MUSIC)\n> \t\t\tbreak;\n> \t}\n\n"},{"id":"469053","messageId":"e957d4f4-fa94-7a68-f378-38e6ed131244@github.com","threadId":"58425","inReplyTo":"5c4e36e23eecbb7841078939a982b7150e2f4ab8.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 5/8] test-http-server: add HTTP request parsing","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:18:08Z","receivedAt":"2022-12-14T23:18:33Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> +/*\n> + * Read the HTTP request up to the start of the optional message-body.\n> + * We do this byte-by-byte because we have keep-alive turned on and\n> + * cannot rely on an EOF.\n> + *\n> + * https://tools.ietf.org/html/rfc7230\n> + *\n> + * We cannot call die() here because our caller needs to properly\n> + * respond to the client and/or close the socket before this\n> + * child exits so that the client doesn't get a connection reset\n> + * by peer error.\n> + */\n> +static enum worker_result req__read(struct req *req, int fd)\n> +{\n> +\tstruct strbuf h = STRBUF_INIT;\n> +\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n> +\tint nr_start_line_fields;\n> +\tconst char *uri_target;\n> +\tconst char *query;\n> +\tchar *hp;\n> +\tconst char *hv;\n> +\n> +\tenum worker_result result = WR_OK;\n> +\n> +\t/*\n> +\t * Read line 0 of the request and split it into component parts:\n> +\t *\n> +\t * SP SP CRLF\n> +\t *\n> +\t */\n> +\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n> +\t\tresult = WR_OK | WR_HANGUP;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tstrbuf_trim_trailing_newline(&req->start_line);\n> +\n> +\tnr_start_line_fields = string_list_split(&start_line_fields,\n> +\t\t\t\t\t\t req->start_line.buf,\n> +\t\t\t\t\t\t ' ', -1);\n> +\tif (nr_start_line_fields != 3) {\n> +\t\tlogerror(\"could not parse request start-line '%s'\",\n> +\t\t\t req->start_line.buf);\n> +\t\tresult = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\treq->method = xstrdup(start_line_fields.items[0].string);\n> +\treq->http_version = xstrdup(start_line_fields.items[2].string);\n> +\n> +\turi_target = start_line_fields.items[1].string;\n> +\n> +\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n> +\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n> +\t\t\t req->http_version);\n> +\t\tresult = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tquery = strchr(uri_target, '?');\n> +\n> +\tif (query) {\n> +\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n> +\t\tstrbuf_addstr(&req->query_args, query + 1);\n> +\t} else {\n> +\t\tstrbuf_addstr(&req->uri_path, uri_target);\n> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n> +\t}\n\nThis \"line 0\" parsing looks good, and aligns with the RFC you linked\n(specifically section 3.1.1 [1]).\n\n[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1\n\n> +\n> +\t/*\n> +\t * Read the set of HTTP headers into a string-list.\n> +\t */\n> +\twhile (1) {\n> +\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n> +\t\t\tgoto done;\n> +\t\tstrbuf_trim_trailing_newline(&h);\n> +\n> +\t\tif (!h.len)\n> +\t\t\tgoto done; /* a blank line ends the header */\n> +\n> +\t\thp = strbuf_detach(&h, NULL);\n> +\t\tstring_list_append(&req->header_list, hp);\n> +\n> +\t\t/* store common request headers separately */\n> +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n> +\t\t\treq->content_type = hv;\n> +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n> +\t\t\treq->content_length = strtol(hv, &hp, 10);\n> +\t\t}\n\nThe \"separately\" is somewhat confusing - you unconditionally add 'hp' to\n'req->header_list', so the \"Content-Type\" and \"Content-Length\" headers are\nincluded there as well. If that's the desired behavior, a comment like \"Also\nstore common headers as 'req' fields\" might be clearer.\n\n> +\t}\n> +\n> +\t/*\n> +\t * We do not attempt to read the , if it exists.\n> +\t * We let our caller read/chunk it in as appropriate.\n> +\t */\n> +\n> +done:\n> +\tstring_list_clear(&start_line_fields, 0);\n> +\n> +\t/*\n> +\t * This is useful for debugging the request, but very noisy.\n> +\t */\n> +\tif (trace2_is_enabled()) {\n\n'trace2_printf()' is gated internally by 'trace2_enabled' anyway, so I don't\nthink this 'if()' is necessary. You could add a 'DEBUG_HTTP_SERVER'\npreprocessor directive (like 'DEBUG_CACHE_TREE' in 'cache-tree.c') if you\nwanted to prevent these printouts unless a developer sets it to '1'.\n\n> +\t\tstruct string_list_item *item;\n> +\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n> +\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n> +\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n> +\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n> +\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n> +\t\tif (req->content_length >= 0)\n> +\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n> +\t\tif (req->content_type)\n> +\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n> +\t\tfor_each_string_list_item(item, &req->header_list)\n> +\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n> +\t}\n> +\n> +\treturn result;\n> +}\n> +\n> +static enum worker_result dispatch(struct req *req)\n> +{\n> +\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> +\t\t\t WR_OK | WR_HANGUP);\n\nAlthough the request is now being read & parsed, the response creation code\nis still a hardcoded \"Not Implemented\". This means that the now-parsed 'req'\nis be temporarily unused, but I think that's reasonable (since it allows for\nbreaking up the implementation of 'test-http-server' into multiple, less\noverwhelming patches).\n\n> +}\n> +\n> static enum worker_result worker(void)\n> {\n> +\tstruct req req = REQ__INIT;\n> \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n> \tchar *client_port = getenv(\"REMOTE_PORT\");\n> \tenum worker_result wr = WR_OK;\n> @@ -160,8 +324,16 @@ static enum worker_result worker(void)\n> \tset_keep_alive(0);\n> \n> \twhile (1) {\n> -\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> -\t\t\tWR_OK | WR_HANGUP);\n> +\t\treq__release(&req);\n> +\n> +\t\talarm(init_timeout ? init_timeout : timeout);\n> +\t\twr = req__read(&req, 0);\n> +\t\talarm(0);\n\nI know 'init_timeout' and 'timeout' were pulled from 'daemon.c', but what's\nthe difference between them/why do they both exist? It looks like\n'init_timeout' just acts as a permanent override to the value of 'timeout'.\n\n> +\n> +\t\tif (wr & WR_STOP_THE_MUSIC)\n> +\t\t\tbreak;\n> +\n> +\t\twr = dispatch(&req);\n> \t\tif (wr & WR_STOP_THE_MUSIC)\n> \t\t\tbreak;\n> \t}\n\n"},{"id":"469054","messageId":"f99c381c-1d30-7c95-6158-cecd5321dafd@github.com","threadId":"58425","inReplyTo":"0a0f4fd10c8b29f327c35dadc7b17881f22b253a.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:20:14Z","receivedAt":"2022-12-14T23:20:22Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Teach the test-http-sever test helper to forward Git requests to the\n> `git-http-backend`.\n> \n> Introduce a new test script t5556-http-auth.sh that spins up the test\n> HTTP server and attempts an `ls-remote` on the served repository,\n> without any authentication.\n> \n> Signed-off-by: Matthew John Cheetham \n> ---\n> t/helper/test-http-server.c | 56 +++++++++++++++++++\n> t/t5556-http-auth.sh | 105 ++++++++++++++++++++++++++++++++++++\n> 2 files changed, 161 insertions(+)\n> create mode 100755 t/t5556-http-auth.sh\n> \n> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> index 7bde678e264..9f1d6b58067 100644\n> --- a/t/helper/test-http-server.c\n> +++ b/t/helper/test-http-server.c\n> @@ -305,8 +305,64 @@ done:\n> \treturn result;\n> }\n> \n> +static int is_git_request(struct req *req)\n> +{\n> +\tstatic regex_t *smart_http_regex;\n> +\tstatic int initialized;\n> +\n> +\tif (!initialized) {\n> +\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n> +\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n> +\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n> +\t\t\t REG_EXTENDED)) {\n\nCould you explain the reasoning behind this regex (e.g., in a comment)? What\nsorts of valid/invalid requests does it represent? Is that the full set of\nrequests that are \"valid\" to Git, or is it a test-specific subset?\n\n> +\t\t\twarning(\"could not compile smart HTTP regex\");\n> +\t\t\tsmart_http_regex = NULL;\n> +\t\t}\n> +\t\tinitialized = 1;\n> +\t}\n> +\n> +\treturn smart_http_regex &&\n> +\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n> +}\n> +\n> +static enum worker_result do__git(struct req *req, const char *user)\n> +{\n> +\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n> +\tint res;\n> +\n> +\tif (write(1, ok, strlen(ok)) < 0)\n> +\t\treturn error(_(\"could not send '%s'\"), ok);\n\nIs it correct to hardcode the response status to '200 OK'? Even when\n'http-backend' exits with an error?\n\n> +\n> +\tif (user)\n> +\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n\nI'm guessing that 'user' isn't used until a later patch? I think it might be\nbetter to not introduce that arg at all until it's needed (it'll put the\nusage of 'user' in context with how its value is determined), rather than\nhardcode it to 'NULL' for now.\n\n> +\n> +\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n> +\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n> +\t\t\treq->uri_path.buf);\n> +\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n> +\tif (req->query_args.len)\n> +\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n> +\t\t\t\treq->query_args.buf);\n> +\tif (req->content_type)\n> +\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n> +\t\t\t\treq->content_type);\n> +\tif (req->content_length >= 0)\n> +\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n> +\t\t\t\t(intmax_t)req->content_length);\n> +\tcp.git_cmd = 1;\n> +\tstrvec_push(&cp.args, \"http-backend\");\n> +\tres = run_command(&cp);\n\nI'm not super familiar with 'http-backend' but as long as it 1) uses the\ncontent passed into the environment to parse the request, and 2) writes the\nresponse to stdout, I think this is right.\n\n> +\tclose(1);\n> +\tclose(0);\n> +\treturn !!res;\n> +}\n> +\n> static enum worker_result dispatch(struct req *req)\n> {\n> +\tif (is_git_request(req))\n> +\t\treturn do__git(req, NULL);\n> +\n> \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> \t\t\t WR_OK | WR_HANGUP);\n> }\n> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n> new file mode 100755\n> index 00000000000..78da151f122\n> --- /dev/null\n> +++ b/t/t5556-http-auth.sh\n> @@ -0,0 +1,105 @@\n> +#!/bin/sh\n> +\n> +test_description='test http auth header and credential helper interop'\n> +\n> +. ./test-lib.sh\n> +\n> +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n> +\n> +# Setup a repository\n> +#\n> +REPO_DIR=\"$(pwd)\"/repo\n\nnit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with\nwhat I see in other tests. \n\nAlso, if you're creating a repo in its own subdirectory ('repo'), you can\nset 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating\na repo at the root level of the test output dir - it can help avoid\npotential weird/unexpected behavior as a result of being in a repo inside of\nanother repo.\n\n> +\n> +# Setup some lookback URLs where test-http-server will be listening.\n> +# We will spawn it directly inside the repo directory, so we avoid\n> +# any need to configure directory mappings etc - we only serve this\n> +# repository from the root '/' of the server.\n> +#\n> +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n> +ORIGIN_URL=http://$HOST_PORT/\n> +\n> +# The pid-file is created by test-http-server when it starts.\n> +# The server will shutdown if/when we delete it (this is easier than\n> +# killing it by PID).\n> +#\n> +PID_FILE=\"$(pwd)\"/pid-file.pid\n> +SERVER_LOG=\"$(pwd)\"/OUT.server.log\n> +\n> +PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n> +\n> +test_expect_success 'setup repos' '\n> +\ttest_create_repo \"$REPO_DIR\" &&\n> +\tgit -C \"$REPO_DIR\" branch -M main\n> +'\n> +\n> +stop_http_server () {\n> +\tif ! test -f \"$PID_FILE\"\n> +\tthen\n> +\t\treturn 0\n> +\tfi\n> +\t#\n> +\t# The server will shutdown automatically when we delete the pid-file.\n> +\t#\n> +\trm -f \"$PID_FILE\"\n> +\t#\n> +\t# Give it a few seconds to shutdown (mainly to completely release the\n> +\t# port before the next test start another instance and it attempts to\n> +\t# bind to it).\n> +\t#\n> +\tfor k in 0 1 2 3 4\n> +\tdo\n> +\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n> +\t\tthen\n> +\t\t\treturn 0\n> +\t\tfi\n> +\t\tsleep 1\n> +\tdone\n> +\n> +\techo \"stop_http_server: timeout waiting for server shutdown\"\n> +\treturn 1\n> +}\n> +\n> +start_http_server () {\n> +\t#\n> +\t# Launch our server into the background in repo_dir.\n> +\t#\n> +\t(\n> +\t\tcd \"$REPO_DIR\"\n> +\t\ttest-http-server --verbose \\\n> +\t\t\t--listen=127.0.0.1 \\\n> +\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n> +\t\t\t--reuseaddr \\\n> +\t\t\t--pid-file=\"$PID_FILE\" \\\n> +\t\t\t\"$@\" \\\n> +\t\t\t2>\"$SERVER_LOG\" &\n> +\t)\n> +\t#\n> +\t# Give it a few seconds to get started.\n> +\t#\n> +\tfor k in 0 1 2 3 4\n> +\tdo\n> +\t\tif test -f \"$PID_FILE\"\n> +\t\tthen\n> +\t\t\treturn 0\n> +\t\tfi\n> +\t\tsleep 1\n> +\tdone\n> +\n> +\techo \"start_http_server: timeout waiting for server startup\"\n> +\treturn 1\n> +}\n\nThese start/stop functions look good to me!\n\n> +\n> +per_test_cleanup () {\n> +\tstop_http_server &&\n> +\trm -f OUT.*\n> +}\n> +\n> +test_expect_success 'http auth anonymous no challenge' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> +\tstart_http_server --allow-anonymous &&\n\nThe '--allow-anonymous' option isn't added until patch 7 [1], so the test\nwill fail in this patch. I think the easiest way to solve that is to remove\nit here (although I think it's fine to leave the title \"anonymous no\nchallenge\", though), then add it in patch 7. \n\n[1] https://lore.kernel.org/git/794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com/\n\n> +\n> +\t# Attempt to read from a protected repository\n> +\tgit ls-remote $ORIGIN_URL\n> +'\n> +\n> +test_done\n\n"},{"id":"469055","messageId":"2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com","threadId":"58425","inReplyTo":"794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 7/8] test-http-server: add simple authentication","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:23:24Z","receivedAt":"2022-12-14T23:26:11Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n> +{\n> +\tenum auth_result result = AUTH_UNKNOWN;\n> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n> +\tstruct auth_module *mod;\n> +\n> +\tstruct string_list_item *hdr;\n> +\tstruct string_list_item *token;\n> +\tconst char *v;\n> +\tstruct strbuf **split = NULL;\n> +\tint i;\n> +\tchar *challenge;\n> +\n> +\t/*\n> +\t * Check all auth modules and try to validate the request.\n> +\t * The first module that matches a valid token approves the request.\n> +\t * If no module is found, or if there is no valid token, then 401 error.\n> +\t * Otherwise, only permit the request if anonymous auth is enabled.\n> +\t */\n> +\tfor_each_string_list_item(hdr, &req->header_list) {\n> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n\nIs only one \"Authorization:\" header allowed? If so, adding a 'break;' at the\nend of this if-statement would make that clearer. If not, what's the\nexpected allow/deny behavior if e.g. one header is ALLOW'd by one auth\nmodule, and another header is DENY'd by a different auth module?\n\n> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n> +\t\t\tif (!split[0] || !split[1]) continue;\n> +\n> +\t\t\t/* trim trailing space ' ' */\n> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n> +\n> +\t\t\tmod = get_auth_module(split[0]->buf);\n> +\t\t\tif (mod) {\n> +\t\t\t\tresult = AUTH_DENY;\n> +\n> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n> +\t\t\t\t\t\tbreak;\n> +\t\t\t\t\t}\n> +\t\t\t\t}\n> +\n> +\t\t\t\tgoto done;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> +done:\n> +\tswitch (result) {\n> +\tcase AUTH_ALLOW:\n> +\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n> +\t\t*user = \"VALID_TEST_USER\";\n> +\t\t*wr = WR_OK;\n> +\t\tbreak;\n> +\n> +\tcase AUTH_DENY:\n> +\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n> +\t\t/* fall-through */\n> +\n> +\tcase AUTH_UNKNOWN:\n> +\t\tif (result != AUTH_DENY && allow_anonymous)\n> +\t\t\tbreak;\n\nI think this just needs to be 'if (allow_anonymous)' - we already know\n'result' is 'AUTH_UNKNOWN' once we reach this block.\n\n> +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n> +\t\t\tmod = auth_modules[i];\n> +\t\t\tif (mod->challenge_params)\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n> +\t\t\t\t\t\t mod->scheme,\n> +\t\t\t\t\t\t mod->challenge_params);\n> +\t\t\telse\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n> +\t\t\t\t\t\t mod->scheme);\n> +\t\t\tstring_list_append(&hdrs, challenge);\n> +\t\t}\n> +\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n> +\t}\n> +\n> +\tstrbuf_list_free(split);\n> +\tstring_list_clear(&hdrs, 0);\n> +\n> +\treturn result == AUTH_ALLOW ||\n> +\t (result == AUTH_UNKNOWN && allow_anonymous);\n\nSo if a user is explicitly denied, even with 'allow_anonymous', this fails?\nIs there a test case that uses that behavior and/or is that standard auth\nbehavior? Otherwise, it'd be simpler to skip the 'is_authed()' check (in\n'dispatch()') altogether if 'allow_anonymous' is enabled.\n\n> +}\n> +\n> static enum worker_result dispatch(struct req *req)\n> {\n> +\tenum worker_result wr = WR_OK;\n> +\tconst char *user = NULL;\n> +\n> +\tif (!is_authed(req, &user, &wr))\n> +\t\treturn wr;\n> +\n> \tif (is_git_request(req))\n> -\t\treturn do__git(req, NULL);\n> +\t\treturn do__git(req, user);\n> \n> \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> \t\t\t WR_OK | WR_HANGUP);\n> @@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)\n> \tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n> \tint worker_mode = 0;\n> \tint i;\n> +\tstruct auth_module *mod = NULL;\n> \n> \ttrace2_cmd_name(\"test-http-server\");\n> \tsetup_git_directory_gently(NULL);\n> @@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)\n> \t\t\tpid_file = v;\n> \t\t\tcontinue;\n> \t\t}\n> +\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n> +\t\t\tallow_anonymous = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n...\n\n> +\t\t}\n> +\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n> +\t\t\tif (!p[0]) {\n> +\t\t\t\terror(\"invalid argument '%s'\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tif (!p[1]) {\n> +\t\t\t\terror(\"missing token value '%s'\\n\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\t/* trim trailing ':' */\n> +\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n> +\n> +\t\t\tmod = get_auth_module(p[0]->buf);\n> +\t\t\tif (!mod) {\n> +\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n\nDoes this mean that '--auth' needs to be specified before '--auth-token' to\navoid the \"auth scheme not defined\" error? If so, this could be made less\nfragile by just setting the string value of the arg in this 'if()' block,\nthen processing the value after the option-parsing loop.\n\n> +\n> +\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n> +\t\t\tstrbuf_list_free(p);\n> +\t\t\tcontinue;\n> +\t\t}\n> \n> \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n> \t\tusage(test_http_auth_usage);\n\nI think a test (in this patch) showing how the auth headers are handled by\nthis HTTP server would be really helpful in demonstrating/exercising the\nintended behavior. \n\n"},{"id":"469057","messageId":"1dc44716-2550-47de-e666-9972b102905d@github.com","threadId":"58425","inReplyTo":"8ecf63835229676677e3f7e33f634eb5d3a568b7.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 8/8] t5556: add HTTP authentication tests","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2022-12-14T23:48:08Z","receivedAt":"2022-12-14T23:48:17Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Add a series of tests to exercise the HTTP authentication header parsing\n> and the interop with credential helpers. Credential helpers will receive\n> WWW-Authenticate information in credential requests.\n\nA general comment about this series - the way you have the patches organized\nmeans that the \"feature\" content you're trying to integrate (the first two\npatches) is contextually separated from these tests. For people that\nlearn/understand code via examples in tests, this makes it really difficult\nto understand what's going on. To avoid that, I think you could rearrange\nthe patches pretty easily:\n\n1. test-http-server: add stub HTTP server test helper (prev. patch 3)\n - t5556 could be introduced here with the basic \"anonymous\" test in patch\n 6, but marked 'test_expect_failure'.\n2. test-http-server: add HTTP error response function (prev. patch 4)\n3. test-http-server: add HTTP request parsing (prev. patch 5)\n4. test-http-server: pass Git requests to http-backend (prev. patch 6)\n5. test-http-server: add simple authentication (prev. patch 7)\n6. http: read HTTP WWW-Authenticate response headers (prev. patch 1)\n7. credential: add WWW-Authenticate header to cred requests (prev patch 2)\n - Some/all of the tests from the current patch (patch 8) could be squashed\n into this one so that the tests exist directly alongside the new\n functionality they're testing.\n\n> \n> Signed-off-by: Matthew John Cheetham \n> ---\n> t/helper/test-credential-helper-replay.sh | 14 +++\n> t/t5556-http-auth.sh | 120 +++++++++++++++++++++-\n> 2 files changed, 133 insertions(+), 1 deletion(-)\n> create mode 100755 t/helper/test-credential-helper-replay.sh\n> \n> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\n> new file mode 100755\n> index 00000000000..03e5e63dad6\n> --- /dev/null\n> +++ b/t/helper/test-credential-helper-replay.sh\n\nI'm not sure a 't/helper' file is the right place for this - it's a pretty\nsimple shell script, but it defines a lot of information (namely 'teefile',\n'catfile') that is otherwise unexplained in 't5556'. \n\nWhat about something like 'lib-rebase.sh' and its 'set_fake_editor()'? You\ncould create a similar test lib ('lib-credential-helper.sh') and wrapper\nfunction (' that writes out a custom credential helper. Something like\n'set_fake_credential_helper()' could also take 'teefile' and 'catfile' as\narguments, making their names more transparent to 't5556'.\n\n> @@ -0,0 +1,14 @@\n> +cmd=$1\n> +teefile=$cmd-actual.cred\n> +catfile=$cmd-response.cred\n> +rm -f $teefile\n> +while read line;\n> +do\n> +\tif test -z \"$line\"; then\n> +\t\tbreak;\n> +\tfi\n> +\techo \"$line\" >> $teefile\n> +done\n> +if test \"$cmd\" = \"get\"; then\n> +\tcat $catfile\n> +fi\n> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n> index 78da151f122..541fa32bd77 100755\n> --- a/t/t5556-http-auth.sh\n> +++ b/t/t5556-http-auth.sh\n> @@ -26,6 +26,8 @@ PID_FILE=\"$(pwd)\"/pid-file.pid\n> SERVER_LOG=\"$(pwd)\"/OUT.server.log\n> \n> PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n> +CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n> +\t&& export CREDENTIAL_HELPER\n\nI see - this is how you connect the \"test\" credential helper to the HTTP\nserver and header parsing (as implemented in patches 1 & 2), so that the\nresults can be compared for correctness.\n\nnit: you can just 'export CREDENTIAL_HELPER=\"...\"', rather than breaking it\ninto two lines. You also shouldn't need to 'export' at all - the value will\nbe set in the context of the test.\n\n> \n> test_expect_success 'setup repos' '\n> \ttest_create_repo \"$REPO_DIR\" &&\n> @@ -91,7 +93,8 @@ start_http_server () {\n> \n> per_test_cleanup () {\n> \tstop_http_server &&\n> -\trm -f OUT.*\n> +\trm -f OUT.* &&\n> +\trm -f *.cred\n> }\n> \n> test_expect_success 'http auth anonymous no challenge' '\n> @@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '\n> \tgit ls-remote $ORIGIN_URL\n> '\n> \n> +test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n\n...\n\n> +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n\n...\n\n> +test_expect_success 'http auth www-auth headers to credential helper invalid' '\n\nThese tests all look good. That said, is there any way to test more\nbizarre/edge cases (headers too long to fit on one line, headers that end\nwith a long string of whitespace, etc.)?\n\n"},{"id":"469063","messageId":"xmqqpmcltt36.fsf@gitster.g","threadId":"58425","inReplyTo":"1dc44716-2550-47de-e666-9972b102905d@github.com","subject":"Re: [PATCH v4 8/8] t5556: add HTTP authentication tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-12-15T00:21:17Z","receivedAt":"2022-12-15T00:21:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> A general comment about this series - the way you have the patches organized\n> means that the \"feature\" content you're trying to integrate (the first two\n> patches) is contextually separated from these tests. For people that\n> learn/understand code via examples in tests, this makes it really difficult\n> to understand what's going on. To avoid that, I think you could rearrange\n> the patches pretty easily:\n> ...\n\nThanks for a thorough review of the entire series, with concrete\nsuggestions for improvements with encouragements sprinkled in.\n\nVery much appreciated.\n\n"},{"id":"469088","messageId":"221215.861qp13tfh.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"b5b56ccd9419353a4bf5bc9d751a711af07d2197.1670880984.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-15T09:27:48Z","receivedAt":"2022-12-15T09:31:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Dec 12 2022, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> [...]\n> /* Initialize a credential structure, setting all fields to empty. */\n> diff --git a/http.c b/http.c\n> index 8a5ba3f4776..c4e9cd73e14 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> \treturn nmemb;\n> }\n> \n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n\nJust out of general paranoia: use st_mult() here, not \"*\" (checks for\noverflows)?\n\n> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tconst char *val;\n> +\tconst char *z = NULL;\n\nWhy NULL-init the \"z\" here, but not the \"val\"? Both look like they\nshould be un-init'd. We also tend to call a throw-away char pointer \"p\",\nnot \"z\", but anyway (more below).... \n\n> +\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\tstrbuf_add(&buf, ptr, size);\n> +\n> +\t/* Strip the CRLF that should be present at the end of each field */\n> +\tstrbuf_trim_trailing_newline(&buf);\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n> +\t\twhile (isspace(*val))\n> +\t\t\tval++;\n\nAs we already have a \"struct strbuf\" here, maybe we can instead\nconsistently use the strbuf functions, e.g. strbuf_ltrim() in this case.\n\nI haven't reviewed this in detail, maybe it's not easy or worth it\nhere...\n\n> +\n> +\t\tstrvec_push(values, val);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t */\n> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n> +\t\tconst char **v = values->v + values->nr - 1;\n\nIt makes no difference to the compiler, but perhaps using []-indexing\nhere is more idiomatic, for getting the nth member of this strvec?\n\n> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n> +\n> +\t\tfree((void*)*v);\n\nIs this reaching into the strvec & manually memory-managing it\nunavoidable, or can we use strvec_pop() etc?\n\n> +\t\t*v = append;\n> +\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/* This is the start of a new header we don't care about */\n> +\thttp_auth.header_is_last_match = 0;\n> +\n> +\t/*\n> +\t * If this is a HTTP status line and not a header field, this signals\n> +\t * a different HTTP response. libcurl writes all the output of all\n> +\t * response headers of all responses, including redirects.\n> +\t * We only care about the last HTTP request response's headers so clear\n> +\t * the existing array.\n> +\t */\n> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n\n...Don't you want to just skip this \"z\" variable altogether and use\nistarts_with() instead? All you seem to care about is whether it starts\nwith it, not what the offset is.\n\n"},{"id":"470104","messageId":"AS2PR03MB9815AE0DC94C53B59E13C3C9C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"35352a10-3fc0-365f-11af-b821d114c409@github.com","subject":"Re: [PATCH v4 2/8] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T20:37:14Z","receivedAt":"2023-01-11T20:37:30Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:15, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Add the value of the WWW-Authenticate response header to credential\n>> requests. Credential helpers that understand and support HTTP\n>> authentication and authorization can use this standard header (RFC 2616\n>> Section 14.47 [1]) to generate valid credentials.\n>>\n>> WWW-Authenticate headers can contain information pertaining to the\n>> authority, authentication mechanism, or extra parameters/scopes that are\n>> required.\n>>\n>> The current I/O format for credential helpers only allows for unique\n>> names for properties/attributes, so in order to transmit multiple header\n>> values (with a specific order) we introduce a new convention whereby a\n>> C-style array syntax is used in the property name to denote multiple\n>> ordered values for the same property.\n>>\n>> In this case we send multiple `wwwauth[]` properties where the order\n>> that the repeated attributes appear in the conversation reflects the\n>> order that the WWW-Authenticate headers appeared in the HTTP response.\n>>\n>> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n> \n> ...\n> \n>> +Attributes with keys that end with C-style array brackets `[]` can have\n>> +multiple values. Each instance of a multi-valued attribute forms an\n>> +ordered list of values - the order of the repeated attributes defines\n>> +the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n>> +acts to clear any previous entries and reset the list.\n>> +\n> \n> The commit message & documentation changes (here and the 'www-auth[]'\n> definition below) are concise, easy-to-understand explanations of what\n> you're doing here with the 'www-authenticate' header values.\n> \n>> \n>> @@ -160,6 +166,16 @@ empty string.\n>> Components which are missing from the URL (e.g., there is no\n>> username in the example above) will be left unset.\n>> \n>> +`wwwauth[]`::\n>> +\n>> +\tWhen an HTTP response is received by Git that includes one or more\n>> +\t'WWW-Authenticate' authentication headers, these will be passed by Git\n>> +\tto credential helpers.\n>> +\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n>> +\tattribute 'wwwauth[]', where the order of the attributes is the same as\n>> +\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n>> +\tto pass additional information to credential helpers.\n> \n> nit: if you're trying to get a paragraph break between \"...to credential\n> helpers.\" and \"Each 'WWW-Authenticate' header value\", you need to add an\n> explicit break:\n> \n> -------- 8< --------\n> \n> diff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\n> index bf0de0e940..50759153ef 100644\n> --- a/Documentation/git-credential.txt\n> +++ b/Documentation/git-credential.txt\n> @@ -171,10 +171,11 @@ username in the example above) will be left unset.\n> \tWhen an HTTP response is received by Git that includes one or more\n> \t'WWW-Authenticate' authentication headers, these will be passed by Git\n> \tto credential helpers.\n> -\tEach 'WWW-Authenticate' header value is passed as a multi-valued\n> -\tattribute 'wwwauth[]', where the order of the attributes is the same as\n> -\tthey appear in the HTTP response. This attribute is 'one-way' from Git\n> -\tto pass additional information to credential helpers.\n> ++\n> +Each 'WWW-Authenticate' header value is passed as a multi-valued\n> +attribute 'wwwauth[]', where the order of the attributes is the same as\n> +they appear in the HTTP response. This attribute is 'one-way' from Git\n> +to pass additional information to credential helpers.\n> \n> Unrecognised attributes are silently discarded.\n> \n> -------- >8 --------\n> \n> You can test to see how the docs look by running 'make doc' from the\n> repository root and looking at the generated 'git-credential.html' (note\n> that, if you've installed Git dependencies with Homebrew, you might need to\n> specify 'XML_CATALOG_FILES=$(brew --prefix)/etc/xml/catalog' to get it to\n> work).\n\nThanks! Yes, I was intending there to be a line break. Thanks for the tip;\nwill be addressed in the next iteration.\n\n>> +\n>> Unrecognised attributes are silently discarded.\n>> \n>> GIT\n>> diff --git a/credential.c b/credential.c\n>> index 897b4679333..8a3ad6c0ae2 100644\n>> --- a/credential.c\n>> +++ b/credential.c\n>> @@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n>> \tfprintf(fp, \"%s=%s\\n\", key, value);\n>> }\n>> \n>> +static void credential_write_strvec(FILE *fp, const char *key,\n>> +\t\t\t\t const struct strvec *vec)\n>> +{\n>> +\tint i = 0;\n>> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n>> +\tfor (; i < vec->nr; i++) {\n>> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n>> +\t}\n>> +\tfree((void*)full_key);\n>> +}\n>> +\n>> void credential_write(const struct credential *c, FILE *fp)\n>> {\n>> \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n>> @@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n>> \tcredential_write_item(fp, \"path\", c->path, 0);\n>> \tcredential_write_item(fp, \"username\", c->username, 0);\n>> \tcredential_write_item(fp, \"password\", c->password, 0);\n>> +\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n> \n> This implementation looks good to me.\n> \n>> }\n>> \n>> static int run_credential_helper(struct credential *c,\n> \n\nThanks,\nMatthew\n"},{"id":"470105","messageId":"AS2PR03MB98150C33F9704D2CA10A2EF9C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"752da6b2-9c75-0f68-e507-cca02bf918ca@github.com","subject":"Re: [PATCH v4 3/8] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T20:46:42Z","receivedAt":"2023-01-11T20:47:26Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:16, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Introduce a mini HTTP server helper that in the future will be enhanced\n>> to provide a frontend for the git-http-backend, with support for\n>> arbitrary authentication schemes.\n> \n> I really like this approach, particularly because it opens up the\n> possibility of writing more fine-grained tests in other contexts (e.g.,\n> testing how a bundle-uri client handles different kinds of erroneous server\n> responses by intercepting and customizing those responses).\n\nHaving a mini server we can play around with makes it easier to simulate a\n'bad' server, rather than use a real one like Apache and try and coerce it\nin to doing 'bad' things.\n\n>>\n>> Right now, test-http-server is a pared-down copy of the git-daemon that\n>> always returns a 501 Not Implemented response to all callers.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> Makefile | 2 +\n>> contrib/buildsystems/CMakeLists.txt | 13 +\n>> t/helper/.gitignore | 1 +\n>> t/helper/test-http-server.c | 685 ++++++++++++++++++++++++++++\n>> 4 files changed, 701 insertions(+)\n>> create mode 100644 t/helper/test-http-server.c\n>>\n>> diff --git a/Makefile b/Makefile\n>> index b258fdbed86..1eb795bbfd4 100644\n>> --- a/Makefile\n>> +++ b/Makefile\n>> @@ -1611,6 +1611,8 @@ else\n>> \tendif\n>> \tBASIC_CFLAGS += $(CURL_CFLAGS)\n>> \n>> +\tTEST_PROGRAMS_NEED_X += test-http-server\n> \n> This works because all usage of 'TEST_PROGRAMS_NEED_X' are either lazily\n> evaluated (in the case of 'TEST_PROGRAMS') or are assigned later in the\n> 'Makefile' than the addition here (in the case of 'test_bindir_programs'). \n> \n> On a related note, I think it would be helpful to mention 'test-http-server'\n> in the \"=== Optional library: libcurl ===\" section of the documentation at\n> the top of the Makefile, to clarify that it (like 'git-http-fetch' and\n> 'git-http-push') are not built.\n\nUpon closer inspection I noticed we don't actuall depend on libcurl here.\nIn my next iteration I've reworked the test helper to share some code with\ndaemon.c and changed where we add `test-http-server` in the Makefiles to\nbe the same as `test-fake-ssh`.\n\n>> +\n>> \tREMOTE_CURL_PRIMARY = git-remote-http$X\n>> \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n>> \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\n>> diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\n>> index 2f6e0197ffa..e9b9bfbb437 100644\n>> --- a/contrib/buildsystems/CMakeLists.txt\n>> +++ b/contrib/buildsystems/CMakeLists.txt\n>> @@ -989,6 +989,19 @@ set(wrapper_scripts\n>> set(wrapper_test_scripts\n>> \ttest-fake-ssh test-tool)\n>> \n>> +if(CURL_FOUND)\n>> + list(APPEND wrapper_test_scripts test-http-server)\n>> +\n>> + add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n>> + target_link_libraries(test-http-server common-main)\n>> +\n>> + if(MSVC)\n>> + set_target_properties(test-http-server\n>> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n>> + set_target_properties(test-http-server\n>> + PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n>> + endif()\n>> +endif()\n>> \n>> foreach(script ${wrapper_scripts})\n>> \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\n>> diff --git a/t/helper/.gitignore b/t/helper/.gitignore\n>> index 8c2ddcce95f..9aa9c752997 100644\n>> --- a/t/helper/.gitignore\n>> +++ b/t/helper/.gitignore\n>> @@ -1,2 +1,3 @@\n>> /test-tool\n>> /test-fake-ssh\n>> +/test-http-server\n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n>> new file mode 100644\n>> index 00000000000..18f1f741305\n>> --- /dev/null\n>> +++ b/t/helper/test-http-server.c\n> \n> A lot of the functions in this file are modified versions of ones in\n> 'daemon.c'. It would help reviewers/future readers to mention that in the\n> commit message. \n\nI appreciate the thorough effort here in understanding what those daemon.c\nfunctions do. Hopefully the next iteration will help other reviewers as I'm\ngoing to be extracting the identical functions to share them between daemon.c\nand test-http-server.c.\n\n> My comments are mostly going to be around the similarities/differences from\n> 'daemon.c', hopefully to understand how 'test-http-server' is meant to be\n> used.\n> \n>> +static void logreport(const char *label, const char *err, va_list params)\n>> +{\n>> +\tstruct strbuf msg = STRBUF_INIT;\n>> +\n>> +\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n>> +\tstrbuf_vaddf(&msg, err, params);\n>> +\tstrbuf_addch(&msg, '\\n');\n>> +\n>> +\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n>> +\tfflush(stderr);\n>> +\n>> +\tstrbuf_release(&msg);\n> \n> This looks like the 'LOG_DESTINATION_STDERR' case of 'logreport()' in\n> 'daemon.c', but adds a \"label\" to represent the priority. Makes sense; these\n> logs will be helpful to have in stderr when running tests, and the priority\n> will be captured as well.\n> \n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void logerror(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"error\", err, params);\n>> +\tva_end(params);\n>> +}\n>> +\n>> +__attribute__((format (printf, 1, 2)))\n>> +static void loginfo(const char *err, ...)\n>> +{\n>> +\tva_list params;\n>> +\tif (!verbose)\n>> +\t\treturn;\n>> +\tva_start(params, err);\n>> +\tlogreport(\"info\", err, params);\n>> +\tva_end(params);\n>> +}\n> \n> These two functions replace the \"priority\" int with the \"label\" string, but\n> otherwise capture the same information.\n> \n>> +\n>> +static void set_keep_alive(int sockfd)\n> \n> This function is identical to its 'daemon.c' counterpart; its usage in\n> 'test-http-server.c' doesn't indicate any need to differ.\n> \n>> +\n>> +/*\n>> + * The code in this section is used by \"worker\" instances to service\n>> + * a single connection from a client. The worker talks to the client\n>> + * on 0 and 1.\n>> + */\n>> +\n>> +enum worker_result {\n>> +\t/*\n>> +\t * Operation successful.\n>> +\t * Caller *might* keep the socket open and allow keep-alive.\n>> +\t */\n>> +\tWR_OK = 0,\n>> +\n>> +\t/*\n>> +\t * Various errors while processing the request and/or the response.\n>> +\t * Close the socket and clean up.\n>> +\t * Exit child-process with non-zero status.\n>> +\t */\n>> +\tWR_IO_ERROR = 1<<0,\n>> +\n>> +\t/*\n>> +\t * Close the socket and clean up. Does not imply an error.\n>> +\t */\n>> +\tWR_HANGUP = 1<<1,\n>> +\n>> +\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n> \n> As much as I love the name, I'm not sure having this value defined makes\n> much sense as its own \"state\". AFAICT, 'WR_IO_ERROR' means \"error AND exit\",\n> but 'WR_HANGUP' just means \"exit\", so the latter is a superset of the\n> former. Even if you interpret 'WR_HANGUP' as \"*no* error and exit\", that\n> makes it and 'WR_IO_ERROR' mutually exclusive, so the \"combined\" state\n> doesn't represent anything \"real\".\n\nFair point. Will remove this extra value in next iteration.\n\n>> +};\n>> +\n>> +static enum worker_result worker(void)\n>> +{\n>> +\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n> \n> Here's the hardcoded 501 error, as mentioned in the commit message.\n> \n>> +\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n>> +\tchar *client_port = getenv(\"REMOTE_PORT\");\n>> +\tenum worker_result wr = WR_OK;\n>> +\n>> +\tif (client_addr)\n>> +\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n>> +\n>> +\tset_keep_alive(0);\n>> +\n>> +\twhile (1) {\n>> +\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n>> +\t\t\tlogerror(\"unable to write response\");\n>> +\t\t\twr = WR_IO_ERROR;\n>> +\t\t}\n> \n> This tries to write the response out to stdout (optional nit: you could use\n> 'STDOUT_FILENO' instead of '1' to make this clearer), and sets 'WR_IO_ERROR'\n> if it fails. \n\nGood point; will use `STDOUT_FILENO` in all applicable places in next iteration.\n\n>> +\n>> +\t\tif (wr & WR_STOP_THE_MUSIC)\n>> +\t\t\tbreak;\n> \n> This will trigger if 'wr' is 'WR_HANGUP' *or* 'WR_IO_ERROR'. Is that\n> intentional? If it is, I think 'wr != 'WR_OK' might make that more obvious?\n> \n>> +\t}\n>> +\n>> +\tclose(0);\n>> +\tclose(1);\n>> +\n>> +\treturn !!(wr & WR_IO_ERROR);\n> \n> Then finish by closing out 'stdin' and 'stdout', and returning '0' for \"no\n> error\", '1' for \"error\".\n> \n>> +}\n>> +\n>> +/*\n>> + * This section contains the listener and child-process management\n>> + * code used by the primary instance to accept incoming connections\n>> + * and dispatch them to async child process \"worker\" instances.\n>> + */\n>> +\n>> +static int addrcmp(const struct sockaddr_storage *s1,\n> \n> \n> Identical to 'daemon.c'.\n> \n>> +static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n>> +{\n>> +\tstruct child *newborn, **cradle;\n>> +\n>> +\tnewborn = xcalloc(1, sizeof(*newborn));\n>> +\tlive_children++;\n>> +\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n>> +\tmemcpy(&newborn->address, addr, addrlen);\n>> +\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n>> +\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n>> +\t\t\tbreak;\n>> +\tnewborn->next = *cradle;\n>> +\t*cradle = newborn;\n>> +}\n> \n> This is mostly the same as 'daemon.c', but uses 'xcalloc()' instead of\n> 'CALLOC_ARRAY()'. The latter is an alias for the former, so this is fine.\n> \n>> +static void kill_some_child(void)\n> \n> ...\n> \n>> +static void check_dead_children(void)\n> Both of these are identical to 'daemon.c'.\n> \n>> +\n>> +static struct strvec cld_argv = STRVEC_INIT;\n>> +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n> \n> This matches 'daemon.c' except for the addition of:\n> \n>> +\tif (cld.out < 0)\n>> +\t\tlogerror(\"could not dup() `incoming`\");\n> \n> The extra context provided by this message could be helpful in debugging. If\n> nothing else, it doesn't hurt.\n> \n>> +\telse if (start_command(&cld))\n>> +\t\tlogerror(\"unable to fork\");\n>> +\telse\n>> +\t\tadd_child(&cld, addr, addrlen);\n>> +}\n>> +\n>> +static void child_handler(int signo)\n> \n> ...\n> \n>> +static int set_reuse_addr(int sockfd)\n> \n> ...\n> \n>> +static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n> \n> ...\n> \n>> +#ifndef NO_IPV6\n>> +\n>> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n> ...\n> \n>> +#else /* NO_IPV6 */\n>> +\n>> +static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n> \n> All of these functions match 'daemon.c' (save for some whitespace fixups).\n> \n>> +\n>> +static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n>> +{\n>> +\tif (!listen_addr->nr)\n>> +\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n> \n> This is the only difference in this function from 'daemon.c' (there, the\n> first arg is 'NULL', which ends up mapping to 'INADDR_ANY'). Why the change\n> in default?\n\nNext iteration will share implementation with daemon.c.\n\n>> +\telse {\n>> +\t\tint i, socknum;\n>> +\t\tfor (i = 0; i < listen_addr->nr; i++) {\n>> +\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n>> +\t\t\t\t\t\t listen_port, socklist);\n>> +\n>> +\t\t\tif (socknum == 0)\n>> +\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n>> +\t\t\t\t\t listen_addr->items[i].string, listen_port);\n>> +\t\t}\n>> +\t}\n>> +}\n>> +\n>> +static int service_loop(struct socketlist *socklist)\n> \n> This function differs from 'daemon.c' by using removal of the 'pid_file' to\n> force a graceful shutdown of the server.\n> \n>> +{\n>> +\tstruct pollfd *pfd;\n>> +\tint i;\n>> +\n>> +\tCALLOC_ARRAY(pfd, socklist->nr);\n>> +\n>> +\tfor (i = 0; i < socklist->nr; i++) {\n>> +\t\tpfd[i].fd = socklist->list[i];\n>> +\t\tpfd[i].events = POLLIN;\n>> +\t}\n>> +\n>> +\tsignal(SIGCHLD, child_handler);\n>> +\n>> +\tfor (;;) {\n>> +\t\tint i;\n>> +\t\tint nr_ready;\n>> +\t\tint timeout = (pid_file ? 100 : -1);\n>> +\n>> +\t\tcheck_dead_children();\n>> +\n>> +\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n> \n> Setting a timeout here (if 'pid_file' is present) allows us to operate in a\n> mode where the removal of a 'pid_file' indicates that the server should shut\n> down.\n> \n>> +\t\tif (nr_ready < 0) {\n> \n> 'nr_ready < 0' indicates an error [1]; handle the same way as 'daemon.c'.\n> \n> [1] https://man7.org/linux/man-pages/man2/poll.2.html\n> \n>> +\t\t\tif (errno != EINTR) {\n>> +\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n>> +\t\t\t\t strerror(errno));\n>> +\t\t\t\tsleep(1);\n>> +\t\t\t}\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\telse if (nr_ready == 0) {\n> \n> 'nr_ready == 0' indicates a polling timeout (see [1] above)...\n> \n>> +\t\t\t/*\n>> +\t\t\t * If we have a pid_file, then we watch it.\n>> +\t\t\t * If someone deletes it, we shutdown the service.\n>> +\t\t\t * The shell scripts in the test suite will use this.\n>> +\t\t\t */\n>> +\t\t\tif (!pid_file || file_exists(pid_file))\n>> +\t\t\t\tcontinue;\n>> +\t\t\tgoto shutdown;\n> \n> ...and that timeout exists so that we can check whether the 'pid_file' still\n> exists and, if so, shut down gracefully.\n> \n>> +\t\t}\n>> +\n> \n> Otherwise, 'nr_ready > 1', so handle the polled events.\n> \n>> +\t\tfor (i = 0; i < socklist->nr; i++) {\n>> +\t\t\tif (pfd[i].revents & POLLIN) {\n>> +\t\t\t\tunion {\n>> +\t\t\t\t\tstruct sockaddr sa;\n>> +\t\t\t\t\tstruct sockaddr_in sai;\n>> +#ifndef NO_IPV6\n>> +\t\t\t\t\tstruct sockaddr_in6 sai6;\n>> +#endif\n>> +\t\t\t\t} ss;\n>> +\t\t\t\tsocklen_t sslen = sizeof(ss);\n>> +\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n>> +\t\t\t\tif (incoming < 0) {\n>> +\t\t\t\t\tswitch (errno) {\n>> +\t\t\t\t\tcase EAGAIN:\n>> +\t\t\t\t\tcase EINTR:\n>> +\t\t\t\t\tcase ECONNABORTED:\n>> +\t\t\t\t\t\tcontinue;\n>> +\t\t\t\t\tdefault:\n>> +\t\t\t\t\t\tdie_errno(\"accept returned\");\n>> +\t\t\t\t\t}\n>> +\t\t\t\t}\n>> +\t\t\t\thandle(incoming, &ss.sa, sslen);\n>> +\t\t\t}\n>> +\t\t}\n>> +\t}\n>> +\n>> +shutdown:\n>> +\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n>> +\tfor (i = 0; i < socklist->nr; i++)\n>> +\t\tclose(socklist->list[i]);\n>> +\n>> +\treturn 0;\n> \n> This addition logs the shutdown and closes out sockets. Looks good!\n> \n>> +}\n>> +\n>> +static int serve(struct string_list *listen_addr, int listen_port)\n>> +{\n>> +\tstruct socketlist socklist = { NULL, 0, 0 };\n>> +\n>> +\tsocksetup(listen_addr, listen_port, &socklist);\n>> +\tif (socklist.nr == 0)\n>> +\t\tdie(\"unable to allocate any listen sockets on port %u\",\n>> +\t\t listen_port);\n>> +\n>> +\tloginfo(\"Ready to rumble\");\n> \n> I thought this was a leftover debug printout, but it turns out that\n> 'serve()' in 'daemon.c' has the same message. :) \n\nIndeed! This made me chuckle when I first saw it..\n\n>> +\n>> +\t/*\n>> +\t * Wait to create the pid-file until we've setup the sockets\n>> +\t * and are open for business.\n>> +\t */\n>> +\tif (pid_file)\n>> +\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n>> +\n>> +\treturn service_loop(&socklist);\n>> +}\n>> +\n>> +/*\n>> + * This section is executed by both the primary instance and all\n>> + * worker instances. So, yes, each child-process re-parses the\n>> + * command line argument and re-discovers how it should behave.\n>> + */\n>> +\n>> +int cmd_main(int argc, const char **argv)\n>> +{\n>> +\tint listen_port = 0;\n>> +\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n>> +\tint worker_mode = 0;\n>> +\tint i;\n>> +\n>> +\ttrace2_cmd_name(\"test-http-server\");\n>> +\tsetup_git_directory_gently(NULL);\n> \n> Since this isn't part of 'test-tool', it needs to do its own trace2 setup,\n> but it seems to be missing some of the relevant function calls. Could you\n> include 'trace2_cmd_list_config()' and 'trace2_cmd_list_env_vars()' as well? \n\nSure!\n\n>> +\n>> +\tfor (i = 1; i < argc; i++) {\n> \n> Can this loop be replaced with 'parse_options()' and the appropriate 'struct\n> option[]'? Newer test helpers ('test-bundle-uri', 'test-cache-tree',\n> 'test-getcwd') have been using it, and it generally seems much easier to\n> work with/more flexible than a custom 'if()' block (handling option\n> negation, interpreting both '--option=' and '--option value' syntax\n> etc.).\n> \n> That said, it looks this was mostly pulled from 'daemon.c' (which might\n> predate 'parse_options()'), so I'd also understand if you want to keep it as\n> similar to that as possible. Up to you!\n\nFor now I think I'll keep it the same style as daemon.c.\n\n>> +\t/* avoid splitting a message in the middle */\n>> +\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n>> +\n>> +\tif (listen_port == 0)\n>> +\t\tlisten_port = DEFAULT_GIT_PORT;\n>> +\n>> +\t/*\n>> +\t * If no --listen= args are given, the setup_named_sock()\n>> +\t * code will use receive a NULL address and set INADDR_ANY.\n>> +\t * This exposes both internal and external interfaces on the\n>> +\t * port.\n>> +\t *\n>> +\t * Disallow that and default to the internal-use-only loopback\n>> +\t * address.\n>> +\t */\n>> +\tif (!listen_addr.nr)\n>> +\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n>> +\n>> +\t/*\n>> +\t * worker_mode is set in our own child process instances\n>> +\t * (that are bound to a connected socket from a client).\n>> +\t */\n>> +\tif (worker_mode)\n>> +\t\treturn worker();\n>> +\n>> +\t/*\n>> +\t * `cld_argv` is a bit of a clever hack. The top-level instance\n>> +\t * of test-http-server does the normal bind/listen/accept stuff.\n>> +\t * For each incoming socket, the top-level process spawns\n>> +\t * a child instance of test-http-server *WITH* the additional\n>> +\t * `--worker` argument. This causes the child to set `worker_mode`\n>> +\t * and immediately call `worker()` using the connected socket (and\n>> +\t * without the usual need for fork() or threads).\n>> +\t *\n>> +\t * The magic here is made possible because `cld_argv` is static\n>> +\t * and handle() (called by service_loop()) knows about it.\n>> +\t */\n>> +\tstrvec_push(&cld_argv, argv[0]);\n>> +\tstrvec_push(&cld_argv, \"--worker\");\n>> +\tfor (i = 1; i < argc; ++i)\n>> +\t\tstrvec_push(&cld_argv, argv[i]);\n>> +\n>> +\t/*\n>> +\t * Setup primary instance to listen for connections.\n>> +\t */\n>> +\treturn serve(&listen_addr, listen_port);\n> \n> The rest of the function is \"new\", but is well-documented and appears to\n> work as intended.\n> \n>> +}\n> \n> One last note/suggestion - while a lot of the functions in\n> 'test-http-server.c' are modified from those in 'daemon.c', there are a fair\n> number of identical functions as well. Would it be possible to libify some\n> of 'daemon.c's functions (mainly by creating a 'daemon.h' and making the\n> functions non-static) so that they don't need to be copied?\n> \n\nWatch for my next iteration for this!\n\nThanks,\nMatthew\n\n"},{"id":"470106","messageId":"AS2PR03MB9815A45BDA0DE776198B5042C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"e957d4f4-fa94-7a68-f378-38e6ed131244@github.com","subject":"Re: [PATCH v4 5/8] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T21:39:56Z","receivedAt":"2023-01-11T21:40:15Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:18, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> +/*\n>> + * Read the HTTP request up to the start of the optional message-body.\n>> + * We do this byte-by-byte because we have keep-alive turned on and\n>> + * cannot rely on an EOF.\n>> + *\n>> + * https://tools.ietf.org/html/rfc7230\n>> + *\n>> + * We cannot call die() here because our caller needs to properly\n>> + * respond to the client and/or close the socket before this\n>> + * child exits so that the client doesn't get a connection reset\n>> + * by peer error.\n>> + */\n>> +static enum worker_result req__read(struct req *req, int fd)\n>> +{\n>> +\tstruct strbuf h = STRBUF_INIT;\n>> +\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n>> +\tint nr_start_line_fields;\n>> +\tconst char *uri_target;\n>> +\tconst char *query;\n>> +\tchar *hp;\n>> +\tconst char *hv;\n>> +\n>> +\tenum worker_result result = WR_OK;\n>> +\n>> +\t/*\n>> +\t * Read line 0 of the request and split it into component parts:\n>> +\t *\n>> +\t * SP SP CRLF\n>> +\t *\n>> +\t */\n>> +\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n>> +\t\tresult = WR_OK | WR_HANGUP;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\tstrbuf_trim_trailing_newline(&req->start_line);\n>> +\n>> +\tnr_start_line_fields = string_list_split(&start_line_fields,\n>> +\t\t\t\t\t\t req->start_line.buf,\n>> +\t\t\t\t\t\t ' ', -1);\n>> +\tif (nr_start_line_fields != 3) {\n>> +\t\tlogerror(\"could not parse request start-line '%s'\",\n>> +\t\t\t req->start_line.buf);\n>> +\t\tresult = WR_IO_ERROR;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\treq->method = xstrdup(start_line_fields.items[0].string);\n>> +\treq->http_version = xstrdup(start_line_fields.items[2].string);\n>> +\n>> +\turi_target = start_line_fields.items[1].string;\n>> +\n>> +\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n>> +\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n>> +\t\t\t req->http_version);\n>> +\t\tresult = WR_IO_ERROR;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\tquery = strchr(uri_target, '?');\n>> +\n>> +\tif (query) {\n>> +\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n>> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n>> +\t\tstrbuf_addstr(&req->query_args, query + 1);\n>> +\t} else {\n>> +\t\tstrbuf_addstr(&req->uri_path, uri_target);\n>> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n>> +\t}\n> \n> This \"line 0\" parsing looks good, and aligns with the RFC you linked\n> (specifically section 3.1.1 [1]).\n> \n> [1] https://www.rfc-editor.org/rfc/rfc7230#section-3.1.1\n> \n>> +\n>> +\t/*\n>> +\t * Read the set of HTTP headers into a string-list.\n>> +\t */\n>> +\twhile (1) {\n>> +\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n>> +\t\t\tgoto done;\n>> +\t\tstrbuf_trim_trailing_newline(&h);\n>> +\n>> +\t\tif (!h.len)\n>> +\t\t\tgoto done; /* a blank line ends the header */\n>> +\n>> +\t\thp = strbuf_detach(&h, NULL);\n>> +\t\tstring_list_append(&req->header_list, hp);\n>> +\n>> +\t\t/* store common request headers separately */\n>> +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n>> +\t\t\treq->content_type = hv;\n>> +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n>> +\t\t\treq->content_length = strtol(hv, &hp, 10);\n>> +\t\t}\n> \n> The \"separately\" is somewhat confusing - you unconditionally add 'hp' to\n> 'req->header_list', so the \"Content-Type\" and \"Content-Length\" headers are\n> included there as well. If that's the desired behavior, a comment like \"Also\n> store common headers as 'req' fields\" might be clearer.\n\nWill clarify this comment in next roll. You are correct, we *also* store these\ncommon headers on `struct req`.\n\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * We do not attempt to read the , if it exists.\n>> +\t * We let our caller read/chunk it in as appropriate.\n>> +\t */\n>> +\n>> +done:\n>> +\tstring_list_clear(&start_line_fields, 0);\n>> +\n>> +\t/*\n>> +\t * This is useful for debugging the request, but very noisy.\n>> +\t */\n>> +\tif (trace2_is_enabled()) {\n> \n> 'trace2_printf()' is gated internally by 'trace2_enabled' anyway, so I don't\n> think this 'if()' is necessary. You could add a 'DEBUG_HTTP_SERVER'\n> preprocessor directive (like 'DEBUG_CACHE_TREE' in 'cache-tree.c') if you\n> wanted to prevent these printouts unless a developer sets it to '1'.\n\nThe overarching `trace2_is_enabled()` call is to avoid any possible repeated\nevaluation within `trace2_printf` for potentially multiple request headers.\n\n>> +\t\tstruct string_list_item *item;\n>> +\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n>> +\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n>> +\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n>> +\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n>> +\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n>> +\t\tif (req->content_length >= 0)\n>> +\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n>> +\t\tif (req->content_type)\n>> +\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n>> +\t\tfor_each_string_list_item(item, &req->header_list)\n>> +\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n>> +\t}\n>> +\n>> +\treturn result;\n>> +}\n>> +\n>> +static enum worker_result dispatch(struct req *req)\n>> +{\n>> +\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n>> +\t\t\t WR_OK | WR_HANGUP);\n> \n> Although the request is now being read & parsed, the response creation code\n> is still a hardcoded \"Not Implemented\". This means that the now-parsed 'req'\n> is be temporarily unused, but I think that's reasonable (since it allows for\n> breaking up the implementation of 'test-http-server' into multiple, less\n> overwhelming patches).\n> \n>> +}\n>> +\n>> static enum worker_result worker(void)\n>> {\n>> +\tstruct req req = REQ__INIT;\n>> \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n>> \tchar *client_port = getenv(\"REMOTE_PORT\");\n>> \tenum worker_result wr = WR_OK;\n>> @@ -160,8 +324,16 @@ static enum worker_result worker(void)\n>> \tset_keep_alive(0);\n>> \n>> \twhile (1) {\n>> -\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n>> -\t\t\tWR_OK | WR_HANGUP);\n>> +\t\treq__release(&req);\n>> +\n>> +\t\talarm(init_timeout ? init_timeout : timeout);\n>> +\t\twr = req__read(&req, 0);\n>> +\t\talarm(0);\n> \n> I know 'init_timeout' and 'timeout' were pulled from 'daemon.c', but what's\n> the difference between them/why do they both exist? It looks like\n> 'init_timeout' just acts as a permanent override to the value of 'timeout'.\n\nGood catch. This split made sense in daemon.c whereby the `--timeout` are would\nbe passed to the `git-upload-pack` command, and `--init-timeout` is used as the\ntimeout value for the daemon server itself.\n\nIn the test HTTP server we don't need the differentiation so I'll just use the\nsimpler `--timeout` arg.\n\n>> +\n>> +\t\tif (wr & WR_STOP_THE_MUSIC)\n>> +\t\t\tbreak;\n>> +\n>> +\t\twr = dispatch(&req);\n>> \t\tif (wr & WR_STOP_THE_MUSIC)\n>> \t\t\tbreak;\n>> \t}\n> \n\nThanks,\nMatthew\n"},{"id":"470107","messageId":"AS2PR03MB9815D6B888AD0E0E12C1C679C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"f99c381c-1d30-7c95-6158-cecd5321dafd@github.com","subject":"Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T21:45:32Z","receivedAt":"2023-01-11T21:45:49Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"\nOn 2022-12-14 15:20, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Teach the test-http-sever test helper to forward Git requests to the\n>> `git-http-backend`.\n>>\n>> Introduce a new test script t5556-http-auth.sh that spins up the test\n>> HTTP server and attempts an `ls-remote` on the served repository,\n>> without any authentication.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> t/helper/test-http-server.c | 56 +++++++++++++++++++\n>> t/t5556-http-auth.sh | 105 ++++++++++++++++++++++++++++++++++++\n>> 2 files changed, 161 insertions(+)\n>> create mode 100755 t/t5556-http-auth.sh\n>>\n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n>> index 7bde678e264..9f1d6b58067 100644\n>> --- a/t/helper/test-http-server.c\n>> +++ b/t/helper/test-http-server.c\n>> @@ -305,8 +305,64 @@ done:\n>> \treturn result;\n>> }\n>> \n>> +static int is_git_request(struct req *req)\n>> +{\n>> +\tstatic regex_t *smart_http_regex;\n>> +\tstatic int initialized;\n>> +\n>> +\tif (!initialized) {\n>> +\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n>> +\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n>> +\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n>> +\t\t\t REG_EXTENDED)) {\n> \n> Could you explain the reasoning behind this regex (e.g., in a comment)? What\n> sorts of valid/invalid requests does it represent? Is that the full set of\n> requests that are \"valid\" to Git, or is it a test-specific subset?\n\nExplanatory comment will be added in next iteration. These are the valid Git\nendpoints for the dumb and smart HTTP protocols as specified in the tech docs.\n\n>> +\t\t\twarning(\"could not compile smart HTTP regex\");\n>> +\t\t\tsmart_http_regex = NULL;\n>> +\t\t}\n>> +\t\tinitialized = 1;\n>> +\t}\n>> +\n>> +\treturn smart_http_regex &&\n>> +\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n>> +}\n>> +\n>> +static enum worker_result do__git(struct req *req, const char *user)\n>> +{\n>> +\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n>> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n>> +\tint res;\n>> +\n>> +\tif (write(1, ok, strlen(ok)) < 0)\n>> +\t\treturn error(_(\"could not send '%s'\"), ok);\n> \n> Is it correct to hardcode the response status to '200 OK'? Even when\n> 'http-backend' exits with an error?\n\nWe always respond with a 200 OK response even if the http-backend process exits\nwith an error. This helper is intended only to be used to exercise the HTTP\nauth handling in the Git client, and specifically around authentication (not\nhandled by http-backend).\n\nIf we wanted to respond with a more 'valid' HTTP response status then we'd need\nto buffer the output of http-backend, wait for and grok the exit status of the\nprocess, then write the HTTP status line followed by the http-backend output.\nThis is outside of the scope of this test helper's use at time of writing.\n\nImportant auth responses (401) we are handling prior to getting to this point.\n\nThe above will also be summarised in a comment on the next roll.\n\n>> +\n>> +\tif (user)\n>> +\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n> \n> I'm guessing that 'user' isn't used until a later patch? I think it might be\n> better to not introduce that arg at all until it's needed (it'll put the\n> usage of 'user' in context with how its value is determined), rather than\n> hardcode it to 'NULL' for now.\n\nGood point!\n\n>> +\n>> +\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n>> +\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n>> +\t\t\treq->uri_path.buf);\n>> +\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n>> +\tif (req->query_args.len)\n>> +\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n>> +\t\t\t\treq->query_args.buf);\n>> +\tif (req->content_type)\n>> +\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n>> +\t\t\t\treq->content_type);\n>> +\tif (req->content_length >= 0)\n>> +\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n>> +\t\t\t\t(intmax_t)req->content_length);\n>> +\tcp.git_cmd = 1;\n>> +\tstrvec_push(&cp.args, \"http-backend\");\n>> +\tres = run_command(&cp);\n> \n> I'm not super familiar with 'http-backend' but as long as it 1) uses the\n> content passed into the environment to parse the request, and 2) writes the\n> response to stdout, I think this is right.\n> \n>> +\tclose(1);\n>> +\tclose(0);\n>> +\treturn !!res;\n>> +}\n>> +\n>> static enum worker_result dispatch(struct req *req)\n>> {\n>> +\tif (is_git_request(req))\n>> +\t\treturn do__git(req, NULL);\n>> +\n>> \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n>> \t\t\t WR_OK | WR_HANGUP);\n>> }\n>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n>> new file mode 100755\n>> index 00000000000..78da151f122\n>> --- /dev/null\n>> +++ b/t/t5556-http-auth.sh\n>> @@ -0,0 +1,105 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='test http auth header and credential helper interop'\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n>> +\n>> +# Setup a repository\n>> +#\n>> +REPO_DIR=\"$(pwd)\"/repo\n> \n> nit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with\n> what I see in other tests. \n\nI don't see this? In fact I see more usages of `$(pwd)` than your suggestion.\n\n> Also, if you're creating a repo in its own subdirectory ('repo'), you can\n> set 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating\n> a repo at the root level of the test output dir - it can help avoid\n> potential weird/unexpected behavior as a result of being in a repo inside of\n> another repo.\n\nHowever.. after setting `TEST_NO_CREATE_REPO=1` I was getting CI failures\naround a missing PWD, so my next iteration uses the `$TRASH_DIRECTORY` variable\nexplicitly in paths instead :-)\n\n>> +\n>> +# Setup some lookback URLs where test-http-server will be listening.\n>> +# We will spawn it directly inside the repo directory, so we avoid\n>> +# any need to configure directory mappings etc - we only serve this\n>> +# repository from the root '/' of the server.\n>> +#\n>> +HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n>> +ORIGIN_URL=http://$HOST_PORT/\n>> +\n>> +# The pid-file is created by test-http-server when it starts.\n>> +# The server will shutdown if/when we delete it (this is easier than\n>> +# killing it by PID).\n>> +#\n>> +PID_FILE=\"$(pwd)\"/pid-file.pid\n>> +SERVER_LOG=\"$(pwd)\"/OUT.server.log\n>> +\n>> +PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n>> +\n>> +test_expect_success 'setup repos' '\n>> +\ttest_create_repo \"$REPO_DIR\" &&\n>> +\tgit -C \"$REPO_DIR\" branch -M main\n>> +'\n>> +\n>> +stop_http_server () {\n>> +\tif ! test -f \"$PID_FILE\"\n>> +\tthen\n>> +\t\treturn 0\n>> +\tfi\n>> +\t#\n>> +\t# The server will shutdown automatically when we delete the pid-file.\n>> +\t#\n>> +\trm -f \"$PID_FILE\"\n>> +\t#\n>> +\t# Give it a few seconds to shutdown (mainly to completely release the\n>> +\t# port before the next test start another instance and it attempts to\n>> +\t# bind to it).\n>> +\t#\n>> +\tfor k in 0 1 2 3 4\n>> +\tdo\n>> +\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n>> +\t\tthen\n>> +\t\t\treturn 0\n>> +\t\tfi\n>> +\t\tsleep 1\n>> +\tdone\n>> +\n>> +\techo \"stop_http_server: timeout waiting for server shutdown\"\n>> +\treturn 1\n>> +}\n>> +\n>> +start_http_server () {\n>> +\t#\n>> +\t# Launch our server into the background in repo_dir.\n>> +\t#\n>> +\t(\n>> +\t\tcd \"$REPO_DIR\"\n>> +\t\ttest-http-server --verbose \\\n>> +\t\t\t--listen=127.0.0.1 \\\n>> +\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n>> +\t\t\t--reuseaddr \\\n>> +\t\t\t--pid-file=\"$PID_FILE\" \\\n>> +\t\t\t\"$@\" \\\n>> +\t\t\t2>\"$SERVER_LOG\" &\n>> +\t)\n>> +\t#\n>> +\t# Give it a few seconds to get started.\n>> +\t#\n>> +\tfor k in 0 1 2 3 4\n>> +\tdo\n>> +\t\tif test -f \"$PID_FILE\"\n>> +\t\tthen\n>> +\t\t\treturn 0\n>> +\t\tfi\n>> +\t\tsleep 1\n>> +\tdone\n>> +\n>> +\techo \"start_http_server: timeout waiting for server startup\"\n>> +\treturn 1\n>> +}\n> \n> These start/stop functions look good to me!\n> \n>> +\n>> +per_test_cleanup () {\n>> +\tstop_http_server &&\n>> +\trm -f OUT.*\n>> +}\n>> +\n>> +test_expect_success 'http auth anonymous no challenge' '\n>> +\ttest_when_finished \"per_test_cleanup\" &&\n>> +\tstart_http_server --allow-anonymous &&\n> \n> The '--allow-anonymous' option isn't added until patch 7 [1], so the test\n> will fail in this patch. I think the easiest way to solve that is to remove\n> it here (although I think it's fine to leave the title \"anonymous no\n> challenge\", though), then add it in patch 7. \n> \n> [1] https://lore.kernel.org/git/794256754c1f7d32e438dfb19a05444d423989aa.1670880984.git.gitgitgadget@gmail.com/\n\nGood catch! Will fix.\n\n>> +\n>> +\t# Attempt to read from a protected repository\n>> +\tgit ls-remote $ORIGIN_URL\n>> +'\n>> +\n>> +test_done\n> \n\nThanks,\nMatthew\n"},{"id":"470109","messageId":"AS2PR03MB981593EB3382F9738D2CA3D7C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com","subject":"Re: [PATCH v4 7/8] test-http-server: add simple authentication","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T22:00:32Z","receivedAt":"2023-01-11T22:00:50Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:23, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n>> +{\n>> +\tenum auth_result result = AUTH_UNKNOWN;\n>> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n>> +\tstruct auth_module *mod;\n>> +\n>> +\tstruct string_list_item *hdr;\n>> +\tstruct string_list_item *token;\n>> +\tconst char *v;\n>> +\tstruct strbuf **split = NULL;\n>> +\tint i;\n>> +\tchar *challenge;\n>> +\n>> +\t/*\n>> +\t * Check all auth modules and try to validate the request.\n>> +\t * The first module that matches a valid token approves the request.\n>> +\t * If no module is found, or if there is no valid token, then 401 error.\n>> +\t * Otherwise, only permit the request if anonymous auth is enabled.\n>> +\t */\n>> +\tfor_each_string_list_item(hdr, &req->header_list) {\n>> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n> \n> Is only one \"Authorization:\" header allowed? If so, adding a 'break;' at the\n> end of this if-statement would make that clearer. If not, what's the\n> expected allow/deny behavior if e.g. one header is ALLOW'd by one auth\n> module, and another header is DENY'd by a different auth module?\n\nYes, only one Authorization header *should* be passed.. but the RFCs are not very\nexplicit about that. The test server supports multiple, but will `ALLOW` or `DENY`\nbased on the first matching auth scheme (module).\n\n> \n>> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n>> +\t\t\tif (!split[0] || !split[1]) continue;\n>> +\n>> +\t\t\t/* trim trailing space ' ' */\n>> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n>> +\n>> +\t\t\tmod = get_auth_module(split[0]->buf);\n>> +\t\t\tif (mod) {\n>> +\t\t\t\tresult = AUTH_DENY;\n>> +\n>> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n>> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n>> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n>> +\t\t\t\t\t\tbreak;\n>> +\t\t\t\t\t}\n>> +\t\t\t\t}\n>> +\n>> +\t\t\t\tgoto done;\n>> +\t\t\t}\n>> +\t\t}\n>> +\t}\n>> +\n>> +done:\n>> +\tswitch (result) {\n>> +\tcase AUTH_ALLOW:\n>> +\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n>> +\t\t*user = \"VALID_TEST_USER\";\n>> +\t\t*wr = WR_OK;\n>> +\t\tbreak;\n>> +\n>> +\tcase AUTH_DENY:\n>> +\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n>> +\t\t/* fall-through */\n>> +\n>> +\tcase AUTH_UNKNOWN:\n>> +\t\tif (result != AUTH_DENY && allow_anonymous)\n>> +\t\t\tbreak;\n> \n> I think this just needs to be 'if (allow_anonymous)' - we already know\n> 'result' is 'AUTH_UNKNOWN' once we reach this block.\n\nNote that `AUTH_DENY` falls-through to the `AUTH_UNKNOWN` case.\nThe only time we *DON'T* want to output the auth challenge response headers is\nwhen there was no challenge provided (`AUTH_UNKNOWN`) *and* we are permitting\nanonymous users.\n\n result | allow_anoymous | Output Challenge?\n---------------------------------------------------\n AUTH_DENY | 1 | Yes\n AUTH_DENY | 0 | Yes\n AUTH_UNKNOWN | 1 | No\n AUTH_UNKNOWN | 0 | Yes\n\n>> +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n>> +\t\t\tmod = auth_modules[i];\n>> +\t\t\tif (mod->challenge_params)\n>> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n>> +\t\t\t\t\t\t mod->scheme,\n>> +\t\t\t\t\t\t mod->challenge_params);\n>> +\t\t\telse\n>> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n>> +\t\t\t\t\t\t mod->scheme);\n>> +\t\t\tstring_list_append(&hdrs, challenge);\n>> +\t\t}\n>> +\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n>> +\t}\n>> +\n>> +\tstrbuf_list_free(split);\n>> +\tstring_list_clear(&hdrs, 0);\n>> +\n>> +\treturn result == AUTH_ALLOW ||\n>> +\t (result == AUTH_UNKNOWN && allow_anonymous);\n> \n> So if a user is explicitly denied, even with 'allow_anonymous', this fails?\n> Is there a test case that uses that behavior and/or is that standard auth\n> behavior? Otherwise, it'd be simpler to skip the 'is_authed()' check (in\n> 'dispatch()') altogether if 'allow_anonymous' is enabled.\n\nIf the user is being denied by a module we should always deny access.\n\nAdmittedly, for this simple authentication scenario it's kind of silly to deny\na user who is trying to identify themselves, but permit an anoymous user.\nHowever, if this was an authorization failure then denying a user based on their\ntoken may be totally valid. Right now, we're only concerned about authentication\nand not authorization, so I could move this check to `dispatch()` if you feel\nstrongly about it.\n\n>> +}\n>> +\n>> static enum worker_result dispatch(struct req *req)\n>> {\n>> +\tenum worker_result wr = WR_OK;\n>> +\tconst char *user = NULL;\n>> +\n>> +\tif (!is_authed(req, &user, &wr))\n>> +\t\treturn wr;\n>> +\n>> \tif (is_git_request(req))\n>> -\t\treturn do__git(req, NULL);\n>> +\t\treturn do__git(req, user);\n>> \n>> \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n>> \t\t\t WR_OK | WR_HANGUP);\n>> @@ -854,6 +982,7 @@ int cmd_main(int argc, const char **argv)\n>> \tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n>> \tint worker_mode = 0;\n>> \tint i;\n>> +\tstruct auth_module *mod = NULL;\n>> \n>> \ttrace2_cmd_name(\"test-http-server\");\n>> \tsetup_git_directory_gently(NULL);\n>> @@ -906,6 +1035,63 @@ int cmd_main(int argc, const char **argv)\n>> \t\t\tpid_file = v;\n>> \t\t\tcontinue;\n>> \t\t}\n>> +\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n>> +\t\t\tallow_anonymous = 1;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n> ...\n> \n>> +\t\t}\n>> +\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n>> +\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n>> +\t\t\tif (!p[0]) {\n>> +\t\t\t\terror(\"invalid argument '%s'\", v);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\tif (!p[1]) {\n>> +\t\t\t\terror(\"missing token value '%s'\\n\", v);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n>> +\n>> +\t\t\t/* trim trailing ':' */\n>> +\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n>> +\n>> +\t\t\tmod = get_auth_module(p[0]->buf);\n>> +\t\t\tif (!mod) {\n>> +\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n>> +\t\t\t\tusage(test_http_auth_usage);\n>> +\t\t\t}\n> \n> Does this mean that '--auth' needs to be specified before '--auth-token' to\n> avoid the \"auth scheme not defined\" error? If so, this could be made less\n> fragile by just setting the string value of the arg in this 'if()' block,\n> then processing the value after the option-parsing loop.\n\nYes, `--auth` needs to come first and 'setup' the module and challenge.\n\n>> +\n>> +\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n>> +\t\t\tstrbuf_list_free(p);\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> \n>> \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n>> \t\tusage(test_http_auth_usage);\n> \n> I think a test (in this patch) showing how the auth headers are handled by\n> this HTTP server would be really helpful in demonstrating/exercising the\n> intended behavior. \n> \n\nThanks,\nMatthew\n"},{"id":"470110","messageId":"AS2PR03MB981512D8EF6775A9D6D9DEB6C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"1dc44716-2550-47de-e666-9972b102905d@github.com","subject":"Re: [PATCH v4 8/8] t5556: add HTTP authentication tests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T22:04:46Z","receivedAt":"2023-01-11T22:05:07Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:48, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Add a series of tests to exercise the HTTP authentication header parsing\n>> and the interop with credential helpers. Credential helpers will receive\n>> WWW-Authenticate information in credential requests.\n> \n> A general comment about this series - the way you have the patches organized\n> means that the \"feature\" content you're trying to integrate (the first two\n> patches) is contextually separated from these tests. For people that\n> learn/understand code via examples in tests, this makes it really difficult\n> to understand what's going on. To avoid that, I think you could rearrange\n> the patches pretty easily:\n> \n> 1. test-http-server: add stub HTTP server test helper (prev. patch 3)\n> - t5556 could be introduced here with the basic \"anonymous\" test in patch\n> 6, but marked 'test_expect_failure'.\n> 2. test-http-server: add HTTP error response function (prev. patch 4)\n> 3. test-http-server: add HTTP request parsing (prev. patch 5)\n> 4. test-http-server: pass Git requests to http-backend (prev. patch 6)\n> 5. test-http-server: add simple authentication (prev. patch 7)\n> 6. http: read HTTP WWW-Authenticate response headers (prev. patch 1)\n> 7. credential: add WWW-Authenticate header to cred requests (prev patch 2)\n> - Some/all of the tests from the current patch (patch 8) could be squashed\n> into this one so that the tests exist directly alongside the new\n> functionality they're testing.\n\n\nI think that order make sense - I'll rearrange for my next iteration.\nThanks!\n\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> t/helper/test-credential-helper-replay.sh | 14 +++\n>> t/t5556-http-auth.sh | 120 +++++++++++++++++++++-\n>> 2 files changed, 133 insertions(+), 1 deletion(-)\n>> create mode 100755 t/helper/test-credential-helper-replay.sh\n>>\n>> diff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\n>> new file mode 100755\n>> index 00000000000..03e5e63dad6\n>> --- /dev/null\n>> +++ b/t/helper/test-credential-helper-replay.sh\n> \n> I'm not sure a 't/helper' file is the right place for this - it's a pretty\n> simple shell script, but it defines a lot of information (namely 'teefile',\n> 'catfile') that is otherwise unexplained in 't5556'. \n> \n> What about something like 'lib-rebase.sh' and its 'set_fake_editor()'? You\n> could create a similar test lib ('lib-credential-helper.sh') and wrapper\n> function (' that writes out a custom credential helper. Something like\n> 'set_fake_credential_helper()' could also take 'teefile' and 'catfile' as\n> arguments, making their names more transparent to 't5556'.\n\nThe `lib-rebase.sh` script sets the fake editor by setting an environment\nvariable (from what I can see). Credential helpers can only be set via config\nor command-line arg. Would it be easier to move writing of the test credential\nhelper script to the t5556 test script setup?\n\n>> @@ -0,0 +1,14 @@\n>> +cmd=$1\n>> +teefile=$cmd-actual.cred\n>> +catfile=$cmd-response.cred\n>> +rm -f $teefile\n>> +while read line;\n>> +do\n>> +\tif test -z \"$line\"; then\n>> +\t\tbreak;\n>> +\tfi\n>> +\techo \"$line\" >> $teefile\n>> +done\n>> +if test \"$cmd\" = \"get\"; then\n>> +\tcat $catfile\n>> +fi\n>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n>> index 78da151f122..541fa32bd77 100755\n>> --- a/t/t5556-http-auth.sh\n>> +++ b/t/t5556-http-auth.sh\n>> @@ -26,6 +26,8 @@ PID_FILE=\"$(pwd)\"/pid-file.pid\n>> SERVER_LOG=\"$(pwd)\"/OUT.server.log\n>> \n>> PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n>> +CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n>> +\t&& export CREDENTIAL_HELPER\n> \n> I see - this is how you connect the \"test\" credential helper to the HTTP\n> server and header parsing (as implemented in patches 1 & 2), so that the\n> results can be compared for correctness.\n> \n> nit: you can just 'export CREDENTIAL_HELPER=\"...\"', rather than breaking it\n> into two lines. You also shouldn't need to 'export' at all - the value will\n> be set in the context of the test.\n\nI tried this originally, but got errors from one of the environments in CI that\nthis was not portable.\n\n>> \n>> test_expect_success 'setup repos' '\n>> \ttest_create_repo \"$REPO_DIR\" &&\n>> @@ -91,7 +93,8 @@ start_http_server () {\n>> \n>> per_test_cleanup () {\n>> \tstop_http_server &&\n>> -\trm -f OUT.*\n>> +\trm -f OUT.* &&\n>> +\trm -f *.cred\n>> }\n>> \n>> test_expect_success 'http auth anonymous no challenge' '\n>> @@ -102,4 +105,119 @@ test_expect_success 'http auth anonymous no challenge' '\n>> \tgit ls-remote $ORIGIN_URL\n>> '\n>> \n>> +test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n> \n> ...\n> \n>> +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n> \n> ...\n> \n>> +test_expect_success 'http auth www-auth headers to credential helper invalid' '\n> \n> These tests all look good. That said, is there any way to test more\n> bizarre/edge cases (headers too long to fit on one line, headers that end\n> with a long string of whitespace, etc.)?\n> \n\nThanks,\nMatthew\n"},{"id":"470111","messageId":"AS2PR03MB98153051196418108D6CCAEFC0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"xmqqpmcltt36.fsf@gitster.g","subject":"Re: [PATCH v4 8/8] t5556: add HTTP authentication tests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T22:05:49Z","receivedAt":"2023-01-11T22:06:24Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 16:21, Junio C Hamano wrote:\n\n> Victoria Dye writes:\n> \n>> A general comment about this series - the way you have the patches organized\n>> means that the \"feature\" content you're trying to integrate (the first two\n>> patches) is contextually separated from these tests. For people that\n>> learn/understand code via examples in tests, this makes it really difficult\n>> to understand what's going on. To avoid that, I think you could rearrange\n>> the patches pretty easily:\n>> ...\n> \n> Thanks for a thorough review of the entire series, with concrete\n> suggestions for improvements with encouragements sprinkled in.\n> \n> Very much appreciated.\n> \n\nYes! Thank you Victoria for the detailed and thorough review.\nI also too very much appreciate it :-)\n\nThanks,\nMatthew\n"},{"id":"470112","messageId":"AS2PR03MB9815234B918C9C566F21FABBC0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"c255896d-637d-f7b0-8698-10a2112852c1@github.com","subject":"Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T22:09:14Z","receivedAt":"2023-01-11T22:09:32Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-14 15:15, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> +{\n>> +\tsize_t size = eltsize * nmemb;\n>> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\tconst char *val;\n>> +\tconst char *z = NULL;\n>> +\n>> +\t/*\n>> +\t * Header lines may not come NULL-terminated from libcurl so we must\n>> +\t * limit all scans to the maximum length of the header line, or leverage\n>> +\t * strbufs for all operations.\n>> +\t *\n>> +\t * In addition, it is possible that header values can be split over\n>> +\t * multiple lines as per RFC 2616 (even though this has since been\n>> +\t * deprecated in RFC 7230). A continuation header field value is\n>> +\t * identified as starting with a space or horizontal tab.\n>> +\t *\n>> +\t * The formal definition of a header field as given in RFC 2616 is:\n>> +\t *\n>> +\t * message-header = field-name \":\" [ field-value ]\n>> +\t * field-name = token\n>> +\t * field-value = *( field-content | LWS )\n>> +\t * field-content = > +\t * and consisting of either *TEXT or combinations\n>> +\t * of token, separators, and quoted-string>\n>> +\t */\n>> +\n>> +\tstrbuf_add(&buf, ptr, size);\n>> +\n>> +\t/* Strip the CRLF that should be present at the end of each field */\n>> +\tstrbuf_trim_trailing_newline(&buf);\n>> +\n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n>> +\t\twhile (isspace(*val))\n>> +\t\t\tval++;\n> \n> Per the RFC [1]: \n> \n>> The field value MAY be preceded by any amount of LWS, though a single SP\n>> is preferred.\n> \n> And LWS (linear whitespace) is defined as:\n> \n>> CRLF = CR LF \n>> LWS = [CRLF] 1*( SP | HT )\n> \n> and 'isspace()' includes CR, LF, SP, and HT [2]. \n> \n> Looks good!\n> \n> [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4-2\n> [2] https://linux.die.net/man/3/isspace\n> \n>> +\n>> +\t\tstrvec_push(values, val);\n> \n> I had the same question about \"what happens with an empty 'val' here?\" as\n> Stolee did earlier [3], but I *think* the \"zero length\" (i.e., single null\n> terminator) will be copied successfully. It's probably worth testing that\n> explicitly, though (I see you set up tests in later patches - ideally a \n> \"www-authenticate:\" line could be tested there).\n> \n> [3] https://lore.kernel.org/git/9fded44b-c503-f8e5-c6a6-93e882d50e27@github.com/\n\nThere is a bug here. Empty header values would indeed be appended\nsuccessfully, but this eventually results in empty values for `wwwauth[]`\nbeing sent over to credential helpers (which should treat the empty value as\na reset of the existing list!!)\n\nReally, empty values should be ignored.\nMy next iteration should hopefully be a bit more careful around these cases.\n\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * This line could be a continuation of the previously matched header\n>> +\t * field. If this is the case then we should append this value to the\n>> +\t * end of the previously consumed value.\n>> +\t */\n>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>> +\t\tconst char **v = values->v + values->nr - 1;\n>> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n> \n> In this case (where the line is a continuation of a 'www-authenticate'\n> header), it looks like the code here expects *exactly* one LWS at the start\n> of the line ('isspace(*buf.buf)' requiring at least one space to append the\n> header, 'ptr + 1' skipping no more than one). But, according to the RFC, it\n> could be more than one:\n> \n>> Header fields can be extended over multiple lines by preceding each extra\n>> line with at least one SP or HT.\n> \n> So I think 'buf.buf' might need to have all preceding spaces removed, like\n> you did in the \"Start of a new WWW-Authenticate header\" block.\n> \n> Also, if you're copying 'ptr' into 'buf' to avoid issues from a missing null\n> terminator, wouldn't you want to use 'buf.buf' (instead of 'ptr') in\n> 'xstrfmt()'?\n\nSure! Good points.\n\n>> +\n>> +\t\tfree((void*)*v);\n>> +\t\t*v = append;\n> \n> I was about to suggest (optionally) rewriting this to use 'strvec_pop()' and\n> 'strvec_push_nodup()':\n> \n> \tstrvec_pop(values); \n> \tstrvec_push_nodup(values, append);\n> \n> to maybe make this a bit easier to follow, but unfortunately\n> 'strvec_push_nodup()' isn't available outside of 'strvec.c'. If you did want\n> to use 'strvec' functions, you could remove the 'static' from\n> 'strvec_push_nodup()' and add it to 'strvec.h' it in a later reroll, but I\n> don't consider that change \"blocking\" or even important enough to warrant\n> its own reroll. \n\nThat wouldn't be too much effort, and would help simplify overall the move\nto using `strbuf_` functions. Check my next iteration for this.\n\n>> +\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/* This is the start of a new header we don't care about */\n>> +\thttp_auth.header_is_last_match = 0;\n>> +\n>> +\t/*\n>> +\t * If this is a HTTP status line and not a header field, this signals\n>> +\t * a different HTTP response. libcurl writes all the output of all\n>> +\t * response headers of all responses, including redirects.\n>> +\t * We only care about the last HTTP request response's headers so clear\n>> +\t * the existing array.\n>> +\t */\n>> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n>> +\t\tstrvec_clear(values);\n> \n> The comments describing the intended behavior (as well as the commit\n> message) are clear and explain the somewhat esoteric (at least to my\n> untrained eye ;) ) code. Thanks!\n> \n>> +\n>> +exit:\n>> +\tstrbuf_release(&buf);\n>> +\treturn size;\n>> +}\n>> +\n>> size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n>> {\n>> \treturn nmemb;\n>> @@ -1864,6 +1940,8 @@ static int http_request(const char *url,\n>> \t\t\t\t\t fwrite_buffer);\n>> \t}\n>> \n>> +\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n>> +\n>> \taccept_language = http_get_accept_language_header();\n>> \n>> \tif (accept_language)\n> \n\nThanks,\nMatthew\n"},{"id":"470113","messageId":"AS2PR03MB98153C2C0EC082257F091B7DC0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"221215.861qp13tfh.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v4 1/8] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-11T22:11:04Z","receivedAt":"2023-01-11T22:11:21Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2022-12-15 01:27, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Mon, Dec 12 2022, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>> [...]\n>> /* Initialize a credential structure, setting all fields to empty. */\n>> diff --git a/http.c b/http.c\n>> index 8a5ba3f4776..c4e9cd73e14 100644\n>> --- a/http.c\n>> +++ b/http.c\n>> @@ -183,6 +183,82 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>> \treturn nmemb;\n>> }\n>> \n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> +{\n>> +\tsize_t size = eltsize * nmemb;\n> \n> Just out of general paranoia: use st_mult() here, not \"*\" (checks for\n> overflows)?\n\nSure! Good point.\n\n>> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\tconst char *val;\n>> +\tconst char *z = NULL;\n> \n> Why NULL-init the \"z\" here, but not the \"val\"? Both look like they\n> should be un-init'd. We also tend to call a throw-away char pointer \"p\",\n> not \"z\", but anyway (more below).... \n> \n>> +\n>> +\t/*\n>> +\t * Header lines may not come NULL-terminated from libcurl so we must\n>> +\t * limit all scans to the maximum length of the header line, or leverage\n>> +\t * strbufs for all operations.\n>> +\t *\n>> +\t * In addition, it is possible that header values can be split over\n>> +\t * multiple lines as per RFC 2616 (even though this has since been\n>> +\t * deprecated in RFC 7230). A continuation header field value is\n>> +\t * identified as starting with a space or horizontal tab.\n>> +\t *\n>> +\t * The formal definition of a header field as given in RFC 2616 is:\n>> +\t *\n>> +\t * message-header = field-name \":\" [ field-value ]\n>> +\t * field-name = token\n>> +\t * field-value = *( field-content | LWS )\n>> +\t * field-content = > +\t * and consisting of either *TEXT or combinations\n>> +\t * of token, separators, and quoted-string>\n>> +\t */\n>> +\n>> +\tstrbuf_add(&buf, ptr, size);\n>> +\n>> +\t/* Strip the CRLF that should be present at the end of each field */\n>> +\tstrbuf_trim_trailing_newline(&buf);\n>> +\n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n>> +\t\twhile (isspace(*val))\n>> +\t\t\tval++;\n> \n> As we already have a \"struct strbuf\" here, maybe we can instead\n> consistently use the strbuf functions, e.g. strbuf_ltrim() in this case.\n\nThat's a good point. I can move to using strbuf functions entirely.\n\n> I haven't reviewed this in detail, maybe it's not easy or worth it\n> here...\n> \n>> +\n>> +\t\tstrvec_push(values, val);\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * This line could be a continuation of the previously matched header\n>> +\t * field. If this is the case then we should append this value to the\n>> +\t * end of the previously consumed value.\n>> +\t */\n>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>> +\t\tconst char **v = values->v + values->nr - 1;\n> \n> It makes no difference to the compiler, but perhaps using []-indexing\n> here is more idiomatic, for getting the nth member of this strvec?\n\nSure!\n\n>> +\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n>> +\n>> +\t\tfree((void*)*v);\n> \n> Is this reaching into the strvec & manually memory-managing it\n> unavoidable, or can we use strvec_pop() etc?\n\nAgain, good point. I can rework this to pop and push a new, joined value.\n\n>> +\t\t*v = append;\n>> +\n>> +\t\tgoto exit;\n>> +\t}\n>> +\n>> +\t/* This is the start of a new header we don't care about */\n>> +\thttp_auth.header_is_last_match = 0;\n>> +\n>> +\t/*\n>> +\t * If this is a HTTP status line and not a header field, this signals\n>> +\t * a different HTTP response. libcurl writes all the output of all\n>> +\t * response headers of all responses, including redirects.\n>> +\t * We only care about the last HTTP request response's headers so clear\n>> +\t * the existing array.\n>> +\t */\n>> +\tif (skip_iprefix(buf.buf, \"http/\", &z))\n> \n> ...Don't you want to just skip this \"z\" variable altogether and use\n> istarts_with() instead? All you seem to care about is whether it starts\n> with it, not what the offset is.\n> \n\nAgain, a good point. Thanks for the suggestions. My next iteration will include\nthis.\n\nThanks,\nMatthew\n"},{"id":"470114","messageId":"74b0de14185120c9d53d7470e59f57fa20a1927f.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 01/10] daemon: libify socket setup and option functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:01Z","receivedAt":"2023-01-11T22:13:20Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions for setting up listening sockets and keep-alive options\nfrom `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct\ndependencies on global state by inlining the behaviour at the callsites\nfor all libified functions.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 23 ++++++\n daemon.c | 214 +------------------------------------------------\n 4 files changed, 237 insertions(+), 210 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n\ndiff --git a/Makefile b/Makefile\nindex b258fdbed86..2654094dbb5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o\n LIB_OBJS += csum-file.o\n LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n+LIB_OBJS += daemon-utils.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n LIB_OBJS += diagnose.o\ndiff --git a/daemon-utils.c b/daemon-utils.c\nnew file mode 100644\nindex 00000000000..b96b55962db\n--- /dev/null\n+++ b/daemon-utils.c\n@@ -0,0 +1,209 @@\n+#include \"cache.h\"\n+#include \"daemon-utils.h\"\n+\n+void set_keep_alive(int sockfd, log_fn logerror)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd, logerror);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd, logerror);\n+\n+\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(NULL, listen_port, socklist, reuseaddr,\n+\t\t\t\t logerror);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist, reuseaddr,\n+\t\t\t\t\t\t logerror);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nnew file mode 100644\nindex 00000000000..6710a2a6dc0\n--- /dev/null\n+++ b/daemon-utils.h\n@@ -0,0 +1,23 @@\n+#ifndef DAEMON_UTILS_H\n+#define DAEMON_UTILS_H\n+\n+#include \"git-compat-util.h\"\n+#include \"string-list.h\"\n+\n+typedef void (*log_fn)(const char *msg, ...);\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+/* Enable sending of keep-alive messages on the socket. */\n+void set_keep_alive(int sockfd, log_fn logerror);\n+\n+/* Setup a number of sockets to listen on the provided addresses. */\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror);\n+\n+#endif\ndiff --git a/daemon.c b/daemon.c\nindex 0ae7d12b5c1..1ed4e705680 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1,9 +1,9 @@\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"daemon-utils.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n-#include \"string-list.h\"\n \n #ifdef NO_INITGROUPS\n #define initgroups(x, y) (0) /* nothing */\n@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)\n \tstrbuf_release(&hi->tcp_port);\n }\n \n-static void set_keep_alive(int sockfd)\n-{\n-\tint ka = 1;\n-\n-\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n-\t\tif (errno != ENOTSOCK)\n-\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n-\t\t\t\tstrerror(errno));\n-\t}\n-}\n-\n static int execute(void)\n {\n \tchar *line = packet_buffer;\n@@ -759,7 +748,7 @@ static int execute(void)\n \tif (addr)\n \t\tloginfo(\"Connection from %s:%s\", addr, port);\n \n-\tset_keep_alive(0);\n+\tset_keep_alive(0, logerror);\n \talarm(init_timeout ? init_timeout : timeout);\n \tpktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);\n \talarm(0);\n@@ -938,202 +927,6 @@ static void child_handler(int signo)\n \tsignal(SIGCHLD, child_handler);\n }\n \n-static int set_reuse_addr(int sockfd)\n-{\n-\tint on = 1;\n-\n-\tif (!reuseaddr)\n-\t\treturn 0;\n-\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n-\t\t\t &on, sizeof(on));\n-}\n-\n-struct socketlist {\n-\tint *list;\n-\tsize_t nr;\n-\tsize_t alloc;\n-};\n-\n-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n-{\n-#ifdef NO_IPV6\n-\tstatic char ip[INET_ADDRSTRLEN];\n-#else\n-\tstatic char ip[INET6_ADDRSTRLEN];\n-#endif\n-\n-\tswitch (family) {\n-#ifndef NO_IPV6\n-\tcase AF_INET6:\n-\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n-\t\tbreak;\n-#endif\n-\tcase AF_INET:\n-\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n-\t\tbreak;\n-\tdefault:\n-\t\txsnprintf(ip, sizeof(ip), \"\");\n-\t}\n-\treturn ip;\n-}\n-\n-#ifndef NO_IPV6\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tint socknum = 0;\n-\tchar pbuf[NI_MAXSERV];\n-\tstruct addrinfo hints, *ai0, *ai;\n-\tint gai;\n-\tlong flags;\n-\n-\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n-\tmemset(&hints, 0, sizeof(hints));\n-\thints.ai_family = AF_UNSPEC;\n-\thints.ai_socktype = SOCK_STREAM;\n-\thints.ai_protocol = IPPROTO_TCP;\n-\thints.ai_flags = AI_PASSIVE;\n-\n-\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n-\tif (gai) {\n-\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n-\t\treturn 0;\n-\t}\n-\n-\tfor (ai = ai0; ai; ai = ai->ai_next) {\n-\t\tint sockfd;\n-\n-\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n-\t\tif (sockfd < 0)\n-\t\t\tcontinue;\n-\t\tif (sockfd >= FD_SETSIZE) {\n-\t\t\tlogerror(\"Socket descriptor too large\");\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-#ifdef IPV6_V6ONLY\n-\t\tif (ai->ai_family == AF_INET6) {\n-\t\t\tint on = 1;\n-\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n-\t\t\t\t &on, sizeof(on));\n-\t\t\t/* Note: error is not fatal */\n-\t\t}\n-#endif\n-\n-\t\tif (set_reuse_addr(sockfd)) {\n-\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tset_keep_alive(sockfd);\n-\n-\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n-\t\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\t\tif (listen(sockfd, 5) < 0) {\n-\t\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\n-\t\tflags = fcntl(sockfd, F_GETFD, 0);\n-\t\tif (flags >= 0)\n-\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\t\tsocklist->list[socklist->nr++] = sockfd;\n-\t\tsocknum++;\n-\t}\n-\n-\tfreeaddrinfo(ai0);\n-\n-\treturn socknum;\n-}\n-\n-#else /* NO_IPV6 */\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tstruct sockaddr_in sin;\n-\tint sockfd;\n-\tlong flags;\n-\n-\tmemset(&sin, 0, sizeof sin);\n-\tsin.sin_family = AF_INET;\n-\tsin.sin_port = htons(listen_port);\n-\n-\tif (listen_addr) {\n-\t\t/* Well, host better be an IP address here. */\n-\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n-\t\t\treturn 0;\n-\t} else {\n-\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n-\t}\n-\n-\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n-\tif (sockfd < 0)\n-\t\treturn 0;\n-\n-\tif (set_reuse_addr(sockfd)) {\n-\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tset_keep_alive(sockfd);\n-\n-\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n-\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tif (listen(sockfd, 5) < 0) {\n-\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tflags = fcntl(sockfd, F_GETFD, 0);\n-\tif (flags >= 0)\n-\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\tsocklist->list[socklist->nr++] = sockfd;\n-\treturn 1;\n-}\n-\n-#endif\n-\n-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tif (!listen_addr->nr)\n-\t\tsetup_named_sock(NULL, listen_port, socklist);\n-\telse {\n-\t\tint i, socknum;\n-\t\tfor (i = 0; i < listen_addr->nr; i++) {\n-\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n-\t\t\t\t\t\t listen_port, socklist);\n-\n-\t\t\tif (socknum == 0)\n-\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n-\t\t\t\t\t listen_addr->items[i].string, listen_port);\n-\t\t}\n-\t}\n-}\n-\n static int service_loop(struct socketlist *socklist)\n {\n \tstruct pollfd *pfd;\n@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,\n {\n \tstruct socketlist socklist = { NULL, 0, 0 };\n \n-\tsocksetup(listen_addr, listen_port, &socklist);\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr,\n+\t\t logerror);\n \tif (socklist.nr == 0)\n \t\tdie(\"unable to allocate any listen sockets on port %u\",\n \t\t listen_port);\n-- \ngitgitgadget\n\n"},{"id":"470115","messageId":"bc972fc8d3d3a028d3d160aac354d2a13bad37ae.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 02/10] daemon: libify child process handling functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:02Z","receivedAt":"2023-01-11T22:13:24Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions and structures for managing child processes started\nfrom the parent daemon-like process from `daemon.c` to the new shared\n`daemon-utils.{c,h}` files.\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 15 ++++++++\n daemon.c | 92 +++-----------------------------------------------\n 3 files changed, 97 insertions(+), 87 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex b96b55962db..3804bc60973 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t\t}\n \t}\n }\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+ const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn , unsigned int *live_children)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tCALLOC_ARRAY(newborn, 1);\n+\tlive_children++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+void kill_some_child(struct child *firstborn)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tif (loginfo) {\n+\t\t\t\tconst char *dead = \"\";\n+\t\t\t\tif (status)\n+\t\t\t\t\tdead = \" (with error)\";\n+\t\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\",\n+\t\t\t\t\t(uintmax_t)pid, dead);\n+\t\t\t}\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\tlive_children--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex 6710a2a6dc0..fe8d9d05256 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -2,6 +2,7 @@\n #define DAEMON_UTILS_H\n \n #include \"git-compat-util.h\"\n+#include \"run-command.h\"\n #include \"string-list.h\"\n \n typedef void (*log_fn)(const char *msg, ...);\n@@ -20,4 +21,18 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t struct socketlist *socklist, int reuseaddr,\n \t log_fn logerror);\n \n+struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+};\n+\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn, unsigned int *live_children);\n+\n+void kill_some_child(struct child *firstborn);\n+\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo);\n+\n #endif\ndiff --git a/daemon.c b/daemon.c\nindex 1ed4e705680..ec3b407ecbc 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -785,93 +785,11 @@ static int execute(void)\n \treturn -1;\n }\n \n-static int addrcmp(const struct sockaddr_storage *s1,\n- const struct sockaddr_storage *s2)\n-{\n-\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n-\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n-\n-\tif (sa1->sa_family != sa2->sa_family)\n-\t\treturn sa1->sa_family - sa2->sa_family;\n-\tif (sa1->sa_family == AF_INET)\n-\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n-\t\t &((struct sockaddr_in *)s2)->sin_addr,\n-\t\t sizeof(struct in_addr));\n-#ifndef NO_IPV6\n-\tif (sa1->sa_family == AF_INET6)\n-\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n-\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n-\t\t sizeof(struct in6_addr));\n-#endif\n-\treturn 0;\n-}\n-\n static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child {\n-\tstruct child *next;\n-\tstruct child_process cld;\n-\tstruct sockaddr_storage address;\n-} *firstborn;\n-\n-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n-{\n-\tstruct child *newborn, **cradle;\n-\n-\tCALLOC_ARRAY(newborn, 1);\n-\tlive_children++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n-\t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n-}\n-\n-/*\n- * This gets called if the number of connections grows\n- * past \"max_connections\".\n- *\n- * We kill the newest connection from a duplicate IP.\n- */\n-static void kill_some_child(void)\n-{\n-\tconst struct child *blanket, *next;\n-\n-\tif (!(blanket = firstborn))\n-\t\treturn;\n-\n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n-\t\t\tbreak;\n-\t\t}\n-}\n-\n-static void check_dead_children(void)\n-{\n-\tint status;\n-\tpid_t pid;\n-\n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n-\t\t\tconst char *dead = \"\";\n-\t\t\tif (status)\n-\t\t\t\tdead = \" (with error)\";\n-\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n-\n-\t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n-\t\t\tlive_children--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n-\t\t} else\n-\t\t\tcradle = &blanket->next;\n-}\n+static struct child *firstborn;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child();\n+\t\tkill_some_child(firstborn);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen);\n+\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470116","messageId":"8f176d5955dfc83616a39622972aaa71a71f5599.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 03/10] daemon: rename some esoteric/laboured terminology","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:03Z","receivedAt":"2023-01-11T22:13:28Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRename some of the variables and function arguments used to manage child\nprocesses. The existing names are esoteric; stretching an analogy too\nfar to the point of being confusing to understand.\n\nRename \"firstborn\" to simply \"first\", \"newborn\" to \"new_cld\", \"blanket\"\nto \"current\" and \"cradle\" to \"ptr\".\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 46 +++++++++++++++++++++++-----------------------\n daemon-utils.h | 6 +++---\n daemon.c | 10 +++++-----\n 3 files changed, 31 insertions(+), 31 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex 3804bc60973..190da01aea9 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,\n }\n \n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn , unsigned int *live_children)\n+\t struct child *first, unsigned int *live_children)\n {\n-\tstruct child *newborn, **cradle;\n+\tstruct child *new_cld, **current;\n \n-\tCALLOC_ARRAY(newborn, 1);\n+\tCALLOC_ARRAY(new_cld, 1);\n \tlive_children++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\tmemcpy(&new_cld->cld, cld, sizeof(*cld));\n+\tmemcpy(&new_cld->address, addr, addrlen);\n+\tfor (current = &first; *current; current = &(*current)->next)\n+\t\tif (!addrcmp(&(*current)->address, &new_cld->address))\n \t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n+\tnew_cld->next = *current;\n+\t*current = new_cld;\n }\n \n-void kill_some_child(struct child *firstborn)\n+void kill_some_child(struct child *first)\n {\n-\tconst struct child *blanket, *next;\n+\tconst struct child *current, *next;\n \n-\tif (!(blanket = firstborn))\n+\tif (!(current = first))\n \t\treturn;\n \n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\tfor (; (next = current->next); current = next)\n+\t\tif (!addrcmp(¤t->address, &next->address)) {\n+\t\t\tkill(current->cld.pid, SIGTERM);\n \t\t\tbreak;\n \t\t}\n }\n \n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first, unsigned int *live_children,\n \t\t\t log_fn loginfo)\n {\n \tint status;\n \tpid_t pid;\n \n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\tstruct child **ptr, *current;\n+\tfor (ptr = &first; (current = *ptr);)\n+\t\tif ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {\n \t\t\tif (loginfo) {\n \t\t\t\tconst char *dead = \"\";\n \t\t\t\tif (status)\n@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,\n \t\t\t}\n \n \t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n+\t\t\t*ptr = current->next;\n \t\t\tlive_children--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n+\t\t\tchild_process_clear(¤t->cld);\n+\t\t\tfree(current);\n \t\t} else\n-\t\t\tcradle = &blanket->next;\n+\t\t\tptr = ¤t->next;\n }\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex fe8d9d05256..e87bc7b9567 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -28,11 +28,11 @@ struct child {\n };\n \n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn, unsigned int *live_children);\n+\t struct child *first, unsigned int *live_children);\n \n-void kill_some_child(struct child *firstborn);\n+void kill_some_child(struct child *first);\n \n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first, unsigned int *live_children,\n \t\t\t log_fn loginfo);\n \n #endif\ndiff --git a/daemon.c b/daemon.c\nindex ec3b407ecbc..d3e7d81de18 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -789,7 +789,7 @@ static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child *firstborn;\n+static struct child *first_child;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child(firstborn);\n+\t\tkill_some_child(first_child);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470118","messageId":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v4.git.1670880984.gitgitgadget@gmail.com","subject":"[PATCH v5 00/10] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:00Z","receivedAt":"2023-01-11T22:13:31Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server sharing code\nwith git-daemon, with simple authentication configurable by a config file.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\nMatthew John Cheetham (10):\n daemon: libify socket setup and option functions\n daemon: libify child process handling functions\n daemon: rename some esoteric/laboured terminology\n test-http-server: add stub HTTP server test helper\n test-http-server: add HTTP error response function\n test-http-server: add simple authentication\n http: replace unsafe size_t multiplication with st_mult\n strvec: expose strvec_push_nodup for external use\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n credential.c | 13 +\n credential.h | 15 +\n daemon-utils.c | 286 +++++++\n daemon-utils.h | 38 +\n daemon.c | 306 +------\n http.c | 102 ++-\n strvec.c | 2 +-\n strvec.h | 3 +\n t/helper/.gitignore | 1 +\n t/helper/test-credential-helper-replay.sh | 14 +\n t/helper/test-http-server.c | 920 ++++++++++++++++++++++\n t/t5556-http-auth.sh | 372 +++++++++\n 15 files changed, 1801 insertions(+), 303 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n create mode 100755 t/helper/test-credential-helper-replay.sh\n create mode 100644 t/helper/test-http-server.c\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v4:\n\n -: ----------- > 1: 74b0de14185 daemon: libify socket setup and option functions\n -: ----------- > 2: bc972fc8d3d daemon: libify child process handling functions\n -: ----------- > 3: 8f176d5955d daemon: rename some esoteric/laboured terminology\n 3: 07a1845ea56 ! 4: 706fb3781bd test-http-server: add stub HTTP server test helper\n @@ Commit message\n Signed-off-by: Matthew John Cheetham \n \n ## Makefile ##\n -@@ Makefile: else\n - \tendif\n - \tBASIC_CFLAGS += $(CURL_CFLAGS)\n +@@ Makefile: TEST_BUILTINS_OBJS += test-xml-encode.o\n + # Do not add more tests here unless they have extra dependencies. Add\n + # them in TEST_BUILTINS_OBJS above.\n + TEST_PROGRAMS_NEED_X += test-fake-ssh\n ++TEST_PROGRAMS_NEED_X += test-http-server\n + TEST_PROGRAMS_NEED_X += test-tool\n \n -+\tTEST_PROGRAMS_NEED_X += test-http-server\n -+\n - \tREMOTE_CURL_PRIMARY = git-remote-http$X\n - \tREMOTE_CURL_ALIASES = git-remote-https$X git-remote-ftp$X git-remote-ftps$X\n - \tREMOTE_CURL_NAMES = $(REMOTE_CURL_PRIMARY) $(REMOTE_CURL_ALIASES)\n + TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\n \n ## contrib/buildsystems/CMakeLists.txt ##\n +@@ contrib/buildsystems/CMakeLists.txt: if(BUILD_TESTING)\n + add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n + target_link_libraries(test-fake-ssh common-main)\n + \n ++add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n ++target_link_libraries(test-http-server common-main)\n ++\n + #reftable-tests\n + parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n + list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n +@@ contrib/buildsystems/CMakeLists.txt: if(MSVC)\n + \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n + \tset_target_properties(test-fake-ssh test-tool\n + \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n ++\n ++\tset_target_properties(test-http-server\n ++\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n ++\tset_target_properties(test-http-server\n ++\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n + endif()\n + \n + #wrapper scripts\n @@ contrib/buildsystems/CMakeLists.txt: set(wrapper_scripts\n - set(wrapper_test_scripts\n - \ttest-fake-ssh test-tool)\n + \tgit git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)\n \n -+if(CURL_FOUND)\n -+ list(APPEND wrapper_test_scripts test-http-server)\n -+\n -+ add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n -+ target_link_libraries(test-http-server common-main)\n -+\n -+ if(MSVC)\n -+ set_target_properties(test-http-server\n -+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n -+ set_target_properties(test-http-server\n -+ PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n -+ endif()\n -+endif()\n + set(wrapper_test_scripts\n +-\ttest-fake-ssh test-tool)\n +-\n ++\ttest-http-server test-fake-ssh test-tool)\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\n @@ t/helper/.gitignore\n \n ## t/helper/test-http-server.c (new) ##\n @@\n ++#include \"daemon-utils.h\"\n +#include \"config.h\"\n +#include \"run-command.h\"\n +#include \"strbuf.h\"\n @@ t/helper/test-http-server.c (new)\n +\n +static const char test_http_auth_usage[] =\n +\"http-server [--verbose]\\n\"\n -+\" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n ++\" [--timeout=] [--max-connections=]\\n\"\n +\" [--reuseaddr] [--pid-file=]\\n\"\n +\" [--listen=]* [--port=]\\n\"\n +;\n +\n -+/* Timeout, and initial timeout */\n +static unsigned int timeout;\n -+static unsigned int init_timeout;\n +\n +static void logreport(const char *label, const char *err, va_list params)\n +{\n @@ t/helper/test-http-server.c (new)\n +\tva_end(params);\n +}\n +\n -+static void set_keep_alive(int sockfd)\n -+{\n -+\tint ka = 1;\n -+\n -+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n -+\t\tif (errno != ENOTSOCK)\n -+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n -+\t\t\t\tstrerror(errno));\n -+\t}\n -+}\n -+\n +/*\n + * The code in this section is used by \"worker\" instances to service\n + * a single connection from a client. The worker talks to the client\n @@ t/helper/test-http-server.c (new)\n +\t * Close the socket and clean up. Does not imply an error.\n +\t */\n +\tWR_HANGUP = 1<<1,\n -+\n -+\tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n +};\n +\n +static enum worker_result worker(void)\n @@ t/helper/test-http-server.c (new)\n +\tif (client_addr)\n +\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n +\n -+\tset_keep_alive(0);\n ++\tset_keep_alive(0, logerror);\n +\n +\twhile (1) {\n -+\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n ++\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n +\t\t\tlogerror(\"unable to write response\");\n +\t\t\twr = WR_IO_ERROR;\n +\t\t}\n +\n -+\t\tif (wr & WR_STOP_THE_MUSIC)\n ++\t\tif (wr != WR_OK)\n +\t\t\tbreak;\n +\t}\n +\n -+\tclose(0);\n -+\tclose(1);\n ++\tclose(STDIN_FILENO);\n ++\tclose(STDOUT_FILENO);\n +\n +\treturn !!(wr & WR_IO_ERROR);\n +}\n +\n -+/*\n -+ * This section contains the listener and child-process management\n -+ * code used by the primary instance to accept incoming connections\n -+ * and dispatch them to async child process \"worker\" instances.\n -+ */\n -+\n -+static int addrcmp(const struct sockaddr_storage *s1,\n -+\t\t const struct sockaddr_storage *s2)\n -+{\n -+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n -+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n -+\n -+\tif (sa1->sa_family != sa2->sa_family)\n -+\t\treturn sa1->sa_family - sa2->sa_family;\n -+\tif (sa1->sa_family == AF_INET)\n -+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n -+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n -+\t\t sizeof(struct in_addr));\n -+#ifndef NO_IPV6\n -+\tif (sa1->sa_family == AF_INET6)\n -+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n -+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n -+\t\t sizeof(struct in6_addr));\n -+#endif\n -+\treturn 0;\n -+}\n -+\n +static int max_connections = 32;\n +\n +static unsigned int live_children;\n +\n -+static struct child {\n -+\tstruct child *next;\n -+\tstruct child_process cld;\n -+\tstruct sockaddr_storage address;\n -+} *firstborn;\n -+\n -+static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n -+{\n -+\tstruct child *newborn, **cradle;\n -+\n -+\tnewborn = xcalloc(1, sizeof(*newborn));\n -+\tlive_children++;\n -+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n -+\tmemcpy(&newborn->address, addr, addrlen);\n -+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n -+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n -+\t\t\tbreak;\n -+\tnewborn->next = *cradle;\n -+\t*cradle = newborn;\n -+}\n -+\n -+/*\n -+ * This gets called if the number of connections grows\n -+ * past \"max_connections\".\n -+ *\n -+ * We kill the newest connection from a duplicate IP.\n -+ */\n -+static void kill_some_child(void)\n -+{\n -+\tconst struct child *blanket, *next;\n -+\n -+\tif (!(blanket = firstborn))\n -+\t\treturn;\n -+\n -+\tfor (; (next = blanket->next); blanket = next)\n -+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n -+\t\t\tkill(blanket->cld.pid, SIGTERM);\n -+\t\t\tbreak;\n -+\t\t}\n -+}\n -+\n -+static void check_dead_children(void)\n -+{\n -+\tint status;\n -+\tpid_t pid;\n -+\n -+\tstruct child **cradle, *blanket;\n -+\tfor (cradle = &firstborn; (blanket = *cradle);)\n -+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n -+\t\t\tconst char *dead = \"\";\n -+\t\t\tif (status)\n -+\t\t\t\tdead = \" (with error)\";\n -+\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n -+\n -+\t\t\t/* remove the child */\n -+\t\t\t*cradle = blanket->next;\n -+\t\t\tlive_children--;\n -+\t\t\tchild_process_clear(&blanket->cld);\n -+\t\t\tfree(blanket);\n -+\t\t} else\n -+\t\t\tcradle = &blanket->next;\n -+}\n ++static struct child *first_child;\n +\n +static struct strvec cld_argv = STRVEC_INIT;\n +static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n @@ t/helper/test-http-server.c (new)\n +\tstruct child_process cld = CHILD_PROCESS_INIT;\n +\n +\tif (max_connections && live_children >= max_connections) {\n -+\t\tkill_some_child();\n ++\t\tkill_some_child(first_child);\n +\t\tsleep(1); /* give it some time to die */\n -+\t\tcheck_dead_children();\n ++\t\tcheck_dead_children(first_child, &live_children, loginfo);\n +\t\tif (live_children >= max_connections) {\n +\t\t\tclose(incoming);\n +\t\t\tlogerror(\"Too many children, dropping connection\");\n @@ t/helper/test-http-server.c (new)\n +\telse if (start_command(&cld))\n +\t\tlogerror(\"unable to fork\");\n +\telse\n -+\t\tadd_child(&cld, addr, addrlen);\n ++\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n +}\n +\n +static void child_handler(int signo)\n @@ t/helper/test-http-server.c (new)\n +\tsignal(SIGCHLD, child_handler);\n +}\n +\n -+static int set_reuse_addr(int sockfd)\n -+{\n -+\tint on = 1;\n -+\n -+\tif (!reuseaddr)\n -+\t\treturn 0;\n -+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n -+\t\t\t &on, sizeof(on));\n -+}\n -+\n -+struct socketlist {\n -+\tint *list;\n -+\tsize_t nr;\n -+\tsize_t alloc;\n -+};\n -+\n -+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n -+{\n -+#ifdef NO_IPV6\n -+\tstatic char ip[INET_ADDRSTRLEN];\n -+#else\n -+\tstatic char ip[INET6_ADDRSTRLEN];\n -+#endif\n -+\n -+\tswitch (family) {\n -+#ifndef NO_IPV6\n -+\tcase AF_INET6:\n -+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n -+\t\tbreak;\n -+#endif\n -+\tcase AF_INET:\n -+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n -+\t\tbreak;\n -+\tdefault:\n -+\t\txsnprintf(ip, sizeof(ip), \"\");\n -+\t}\n -+\treturn ip;\n -+}\n -+\n -+#ifndef NO_IPV6\n -+\n -+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n -+{\n -+\tint socknum = 0;\n -+\tchar pbuf[NI_MAXSERV];\n -+\tstruct addrinfo hints, *ai0, *ai;\n -+\tint gai;\n -+\tlong flags;\n -+\n -+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n -+\tmemset(&hints, 0, sizeof(hints));\n -+\thints.ai_family = AF_UNSPEC;\n -+\thints.ai_socktype = SOCK_STREAM;\n -+\thints.ai_protocol = IPPROTO_TCP;\n -+\thints.ai_flags = AI_PASSIVE;\n -+\n -+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n -+\tif (gai) {\n -+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n -+\t\tint sockfd;\n -+\n -+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n -+\t\tif (sockfd < 0)\n -+\t\t\tcontinue;\n -+\t\tif (sockfd >= FD_SETSIZE) {\n -+\t\t\tlogerror(\"Socket descriptor too large\");\n -+\t\t\tclose(sockfd);\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+#ifdef IPV6_V6ONLY\n -+\t\tif (ai->ai_family == AF_INET6) {\n -+\t\t\tint on = 1;\n -+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n -+\t\t\t\t &on, sizeof(on));\n -+\t\t\t/* Note: error is not fatal */\n -+\t\t}\n -+#endif\n -+\n -+\t\tif (set_reuse_addr(sockfd)) {\n -+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n -+\t\t\tclose(sockfd);\n -+\t\t\tcontinue;\n -+\t\t}\n -+\n -+\t\tset_keep_alive(sockfd);\n -+\n -+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n -+\t\t\tlogerror(\"Could not bind to %s: %s\",\n -+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n -+\t\t\t\t strerror(errno));\n -+\t\t\tclose(sockfd);\n -+\t\t\tcontinue;\t/* not fatal */\n -+\t\t}\n -+\t\tif (listen(sockfd, 5) < 0) {\n -+\t\t\tlogerror(\"Could not listen to %s: %s\",\n -+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n -+\t\t\t\t strerror(errno));\n -+\t\t\tclose(sockfd);\n -+\t\t\tcontinue;\t/* not fatal */\n -+\t\t}\n -+\n -+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n -+\t\tif (flags >= 0)\n -+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n -+\n -+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n -+\t\tsocklist->list[socklist->nr++] = sockfd;\n -+\t\tsocknum++;\n -+\t}\n -+\n -+\tfreeaddrinfo(ai0);\n -+\n -+\treturn socknum;\n -+}\n -+\n -+#else /* NO_IPV6 */\n -+\n -+static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n -+{\n -+\tstruct sockaddr_in sin;\n -+\tint sockfd;\n -+\tlong flags;\n -+\n -+\tmemset(&sin, 0, sizeof sin);\n -+\tsin.sin_family = AF_INET;\n -+\tsin.sin_port = htons(listen_port);\n -+\n -+\tif (listen_addr) {\n -+\t\t/* Well, host better be an IP address here. */\n -+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n -+\t\t\treturn 0;\n -+\t} else {\n -+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n -+\t}\n -+\n -+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n -+\tif (sockfd < 0)\n -+\t\treturn 0;\n -+\n -+\tif (set_reuse_addr(sockfd)) {\n -+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n -+\t\tclose(sockfd);\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tset_keep_alive(sockfd);\n -+\n -+\tif (bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0) {\n -+\t\tlogerror(\"Could not bind to %s: %s\",\n -+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n -+\t\t\t strerror(errno));\n -+\t\tclose(sockfd);\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tif (listen(sockfd, 5) < 0) {\n -+\t\tlogerror(\"Could not listen to %s: %s\",\n -+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n -+\t\t\t strerror(errno));\n -+\t\tclose(sockfd);\n -+\t\treturn 0;\n -+\t}\n -+\n -+\tflags = fcntl(sockfd, F_GETFD, 0);\n -+\tif (flags >= 0)\n -+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n -+\n -+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n -+\tsocklist->list[socklist->nr++] = sockfd;\n -+\treturn 1;\n -+}\n -+\n -+#endif\n -+\n -+static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n -+{\n -+\tif (!listen_addr->nr)\n -+\t\tsetup_named_sock(\"127.0.0.1\", listen_port, socklist);\n -+\telse {\n -+\t\tint i, socknum;\n -+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n -+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n -+\t\t\t\t\t\t listen_port, socklist);\n -+\n -+\t\t\tif (socknum == 0)\n -+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n -+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n -+\t\t}\n -+\t}\n -+}\n -+\n +static int service_loop(struct socketlist *socklist)\n +{\n +\tstruct pollfd *pfd;\n @@ t/helper/test-http-server.c (new)\n +\t\tint nr_ready;\n +\t\tint timeout = (pid_file ? 100 : -1);\n +\n -+\t\tcheck_dead_children();\n ++\t\tcheck_dead_children(first_child, &live_children, loginfo);\n +\n +\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n +\t\tif (nr_ready < 0) {\n @@ t/helper/test-http-server.c (new)\n +{\n +\tstruct socketlist socklist = { NULL, 0, 0 };\n +\n -+\tsocksetup(listen_addr, listen_port, &socklist);\n ++\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);\n +\tif (socklist.nr == 0)\n +\t\tdie(\"unable to allocate any listen sockets on port %u\",\n +\t\t listen_port);\n @@ t/helper/test-http-server.c (new)\n +\tint i;\n +\n +\ttrace2_cmd_name(\"test-http-server\");\n ++\ttrace2_cmd_list_config();\n ++\ttrace2_cmd_list_env_vars();\n +\tsetup_git_directory_gently(NULL);\n +\n +\tfor (i = 1; i < argc; i++) {\n @@ t/helper/test-http-server.c (new)\n +\t\t\ttimeout = atoi(v);\n +\t\t\tcontinue;\n +\t\t}\n -+\t\tif (skip_prefix(arg, \"--init-timeout=\", &v)) {\n -+\t\t\tinit_timeout = atoi(v);\n -+\t\t\tcontinue;\n -+\t\t}\n +\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n +\t\t\tmax_connections = atoi(v);\n +\t\t\tif (max_connections < 0)\n 5: 5c4e36e23ee ! 5: 6f66bf146b4 test-http-server: add HTTP request parsing\n @@ Metadata\n Author: Matthew John Cheetham \n \n ## Commit message ##\n - test-http-server: add HTTP request parsing\n + test-http-server: add HTTP error response function\n \n - Add ability to parse HTTP requests to the test-http-server test helper.\n + Introduce a function to the test-http-server test helper to write more\n + full and valid HTTP error responses, including all the standard response\n + headers like `Server` and `Date`.\n \n Signed-off-by: Matthew John Cheetham \n \n ## t/helper/test-http-server.c ##\n @@ t/helper/test-http-server.c: enum worker_result {\n - \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n + \tWR_HANGUP = 1<<1,\n };\n \n +/*\n @@ t/helper/test-http-server.c: enum worker_result {\n +\tstring_list_clear(&req->header_list, 0);\n +}\n +\n - static enum worker_result send_http_error(\n - \tint fd,\n - \tint http_code, const char *http_code_name,\n -@@ t/helper/test-http-server.c: done:\n - \treturn wr;\n - }\n - \n ++static enum worker_result send_http_error(\n ++\tint fd,\n ++\tint http_code, const char *http_code_name,\n ++\tint retry_after_seconds, struct string_list *response_headers,\n ++\tenum worker_result wr_in)\n ++{\n ++\tstruct strbuf response_header = STRBUF_INIT;\n ++\tstruct strbuf response_content = STRBUF_INIT;\n ++\tstruct string_list_item *h;\n ++\tenum worker_result wr;\n ++\n ++\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n ++\t\t http_code, http_code_name);\n ++\tif (retry_after_seconds > 0)\n ++\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n ++\t\t\t retry_after_seconds);\n ++\n ++\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n ++\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n ++\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n ++\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n ++\tif (retry_after_seconds > 0)\n ++\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n ++\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n ++\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n ++\tif (response_headers)\n ++\t\tfor_each_string_list_item(h, response_headers)\n ++\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n ++\tstrbuf_addstr(&response_header, \"\\r\\n\");\n ++\n ++\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n ++\t\tlogerror(\"unable to write response header\");\n ++\t\twr = WR_IO_ERROR;\n ++\t\tgoto done;\n ++\t}\n ++\n ++\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n ++\t\tlogerror(\"unable to write response content body\");\n ++\t\twr = WR_IO_ERROR;\n ++\t\tgoto done;\n ++\t}\n ++\n ++\twr = wr_in;\n ++\n ++done:\n ++\tstrbuf_release(&response_header);\n ++\tstrbuf_release(&response_content);\n ++\n ++\treturn wr;\n ++}\n ++\n +/*\n + * Read the HTTP request up to the start of the optional message-body.\n + * We do this byte-by-byte because we have keep-alive turned on and\n @@ t/helper/test-http-server.c: done:\n +\t\thp = strbuf_detach(&h, NULL);\n +\t\tstring_list_append(&req->header_list, hp);\n +\n -+\t\t/* store common request headers separately */\n ++\t\t/* also store common request headers as struct req members */\n +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n +\t\t\treq->content_type = hv;\n +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n @@ t/helper/test-http-server.c: done:\n +\treturn result;\n +}\n +\n ++static int is_git_request(struct req *req)\n ++{\n ++\tstatic regex_t *smart_http_regex;\n ++\tstatic int initialized;\n ++\n ++\tif (!initialized) {\n ++\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n ++\t\t/*\n ++\t\t * This regular expression matches all dumb and smart HTTP\n ++\t\t * requests that are currently in use, and defined in\n ++\t\t * Documentation/gitprotocol-http.txt.\n ++\t\t *\n ++\t\t */\n ++\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n ++\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n ++\t\t\t REG_EXTENDED)) {\n ++\t\t\twarning(\"could not compile smart HTTP regex\");\n ++\t\t\tsmart_http_regex = NULL;\n ++\t\t}\n ++\t\tinitialized = 1;\n ++\t}\n ++\n ++\treturn smart_http_regex &&\n ++\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n ++}\n ++\n ++static enum worker_result do__git(struct req *req)\n ++{\n ++\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n ++\tstruct child_process cp = CHILD_PROCESS_INIT;\n ++\tint res;\n ++\n ++\t/*\n ++\t * Note that we always respond with a 200 OK response even if the\n ++\t * http-backend process exits with an error. This helper is intended\n ++\t * only to be used to exercise the HTTP auth handling in the Git client,\n ++\t * and specifically around authentication (not handled by http-backend).\n ++\t *\n ++\t * If we wanted to respond with a more 'valid' HTTP response status then\n ++\t * we'd need to buffer the output of http-backend, wait for and grok the\n ++\t * exit status of the process, then write the HTTP status line followed\n ++\t * by the http-backend output. This is outside of the scope of this test\n ++\t * helper's use at time of writing.\n ++\t *\n ++\t * The important auth responses (401) we are handling prior to getting\n ++\t * to this point.\n ++\t */\n ++\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n ++\t\treturn error(_(\"could not send '%s'\"), ok);\n ++\n ++\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n ++\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n ++\t\t\treq->uri_path.buf);\n ++\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n ++\tif (req->query_args.len)\n ++\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n ++\t\t\t\treq->query_args.buf);\n ++\tif (req->content_type)\n ++\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n ++\t\t\t\treq->content_type);\n ++\tif (req->content_length >= 0)\n ++\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n ++\t\t\t\t(intmax_t)req->content_length);\n ++\tcp.git_cmd = 1;\n ++\tstrvec_push(&cp.args, \"http-backend\");\n ++\tres = run_command(&cp);\n ++\tclose(STDOUT_FILENO);\n ++\tclose(STDIN_FILENO);\n ++\treturn !!res;\n ++}\n ++\n +static enum worker_result dispatch(struct req *req)\n +{\n -+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n ++\tif (is_git_request(req))\n ++\t\treturn do__git(req);\n ++\n ++\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n +\t\t\t WR_OK | WR_HANGUP);\n +}\n +\n static enum worker_result worker(void)\n {\n +-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n +\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n - \tset_keep_alive(0);\n + \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n --\t\twr = send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n --\t\t\tWR_OK | WR_HANGUP);\n +-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n +-\t\t\tlogerror(\"unable to write response\");\n +-\t\t\twr = WR_IO_ERROR;\n +-\t\t}\n +\t\treq__release(&req);\n +\n -+\t\talarm(init_timeout ? init_timeout : timeout);\n ++\t\talarm(timeout);\n +\t\twr = req__read(&req, 0);\n +\t\talarm(0);\n +\n -+\t\tif (wr & WR_STOP_THE_MUSIC)\n ++\t\tif (wr != WR_OK)\n +\t\t\tbreak;\n -+\n + \n +\t\twr = dispatch(&req);\n - \t\tif (wr & WR_STOP_THE_MUSIC)\n + \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\n +\n + ## t/t5556-http-auth.sh (new) ##\n +@@\n ++#!/bin/sh\n ++\n ++test_description='test http auth header and credential helper interop'\n ++\n ++TEST_NO_CREATE_REPO=1\n ++. ./test-lib.sh\n ++\n ++test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n ++\n ++# Setup a repository\n ++#\n ++REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n ++\n ++# Setup some lookback URLs where test-http-server will be listening.\n ++# We will spawn it directly inside the repo directory, so we avoid\n ++# any need to configure directory mappings etc - we only serve this\n ++# repository from the root '/' of the server.\n ++#\n ++HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n ++ORIGIN_URL=http://$HOST_PORT/\n ++\n ++# The pid-file is created by test-http-server when it starts.\n ++# The server will shutdown if/when we delete it (this is easier than\n ++# killing it by PID).\n ++#\n ++PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n ++SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n ++\n ++PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n ++\n ++test_expect_success 'setup repos' '\n ++\ttest_create_repo \"$REPO_DIR\" &&\n ++\tgit -C \"$REPO_DIR\" branch -M main\n ++'\n ++\n ++stop_http_server () {\n ++\tif ! test -f \"$PID_FILE\"\n ++\tthen\n ++\t\treturn 0\n ++\tfi\n ++\t#\n ++\t# The server will shutdown automatically when we delete the pid-file.\n ++\t#\n ++\trm -f \"$PID_FILE\"\n ++\t#\n ++\t# Give it a few seconds to shutdown (mainly to completely release the\n ++\t# port before the next test start another instance and it attempts to\n ++\t# bind to it).\n ++\t#\n ++\tfor k in 0 1 2 3 4\n ++\tdo\n ++\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n ++\t\tthen\n ++\t\t\treturn 0\n ++\t\tfi\n ++\t\tsleep 1\n ++\tdone\n ++\n ++\techo \"stop_http_server: timeout waiting for server shutdown\"\n ++\treturn 1\n ++}\n ++\n ++start_http_server () {\n ++\t#\n ++\t# Launch our server into the background in repo_dir.\n ++\t#\n ++\t(\n ++\t\tcd \"$REPO_DIR\"\n ++\t\ttest-http-server --verbose \\\n ++\t\t\t--listen=127.0.0.1 \\\n ++\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n ++\t\t\t--reuseaddr \\\n ++\t\t\t--pid-file=\"$PID_FILE\" \\\n ++\t\t\t\"$@\" \\\n ++\t\t\t2>\"$SERVER_LOG\" &\n ++\t)\n ++\t#\n ++\t# Give it a few seconds to get started.\n ++\t#\n ++\tfor k in 0 1 2 3 4\n ++\tdo\n ++\t\tif test -f \"$PID_FILE\"\n ++\t\tthen\n ++\t\t\treturn 0\n ++\t\tfi\n ++\t\tsleep 1\n ++\tdone\n ++\n ++\techo \"start_http_server: timeout waiting for server startup\"\n ++\treturn 1\n ++}\n ++\n ++per_test_cleanup () {\n ++\tstop_http_server &&\n ++\trm -f OUT.*\n ++}\n ++\n ++test_expect_success 'http auth anonymous no challenge' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\tstart_http_server &&\n ++\n ++\t# Attempt to read from a protected repository\n ++\tgit ls-remote $ORIGIN_URL\n ++'\n ++\n ++test_done\n 7: 794256754c1 ! 6: c3c3d17a688 test-http-server: add simple authentication\n @@ Commit message\n \n Add simple authentication to the test-http-server test helper.\n Authentication schemes and sets of valid tokens can be specified via\n - command-line arguments. Incoming requests are compared against the set\n - of valid schemes and tokens and only approved if a matching token is\n - found, or if no auth was provided and anonymous auth is enabled.\n + a configuration file (in the normal gitconfig file format).\n + Incoming requests are compared against the set of valid schemes and\n + tokens and only approved if a matching token is found, or if no auth\n + was provided and anonymous auth is enabled.\n \n Signed-off-by: Matthew John Cheetham \n \n ## t/helper/test-http-server.c ##\n +@@\n + #include \"version.h\"\n + #include \"dir.h\"\n + #include \"date.h\"\n ++#include \"config.h\"\n + \n + #define TR2_CAT \"test-http-server\"\n + \n @@ t/helper/test-http-server.c: static const char test_http_auth_usage[] =\n - \" [--timeout=] [--init-timeout=] [--max-connections=]\\n\"\n + \" [--timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n -+\" [--anonymous-allowed]\\n\"\n -+\" [--auth=[:] [--auth-token=:]]*\\n\"\n ++\" [--auth-config=]\\n\"\n ;\n \n - /* Timeout, and initial timeout */\n -@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req, const char *user)\n + static unsigned int timeout;\n +@@ t/helper/test-http-server.c: static int is_git_request(struct req *req)\n + \t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n + }\n + \n +-static enum worker_result do__git(struct req *req)\n ++static enum worker_result do__git(struct req *req, const char *user)\n + {\n + \tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n + \tstruct child_process cp = CHILD_PROCESS_INIT;\n +@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n + \tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n + \t\treturn error(_(\"could not send '%s'\"), ok);\n + \n ++\tif (user)\n ++\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n ++\n + \tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n + \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n + \t\t\treq->uri_path.buf);\n +@@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n \treturn !!res;\n }\n \n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +static struct auth_module **auth_modules = NULL;\n +static size_t auth_modules_nr = 0;\n +static size_t auth_modules_alloc = 0;\n ++static struct strvec extra_headers = STRVEC_INIT;\n ++\n ++static struct auth_module *create_auth_module(const char *scheme,\n ++\t\t\t\t\t const char *challenge)\n ++{\n ++\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n ++\tmod->scheme = xstrdup(scheme);\n ++\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n ++\tCALLOC_ARRAY(mod->tokens, 1);\n ++\tstring_list_init_dup(mod->tokens);\n ++\treturn mod;\n ++}\n +\n +static struct auth_module *get_auth_module(const char *scheme)\n +{\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\treturn NULL;\n +}\n +\n -+static void add_auth_module(struct auth_module *mod)\n ++static int add_auth_module(struct auth_module *mod)\n +{\n ++\tif (get_auth_module(mod->scheme))\n ++\t\treturn error(\"duplicate auth scheme '%s'\\n\", mod->scheme);\n ++\n +\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n +\tauth_modules[auth_modules_nr++] = mod;\n ++\n ++\treturn 0;\n +}\n +\n +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\n +\t/*\n +\t * Check all auth modules and try to validate the request.\n -+\t * The first module that matches a valid token approves the request.\n ++\t * The first Authorization header that matches a known auth module\n ++\t * scheme will be consulted to either approve or deny the request.\n +\t * If no module is found, or if there is no valid token, then 401 error.\n +\t * Otherwise, only permit the request if anonymous auth is enabled.\n ++\t * It's atypical for user agents/clients to send multiple Authorization\n ++\t * headers, but not explicitly forbidden or defined.\n +\t */\n +\tfor_each_string_list_item(hdr, &req->header_list) {\n +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\tcase AUTH_UNKNOWN:\n +\t\tif (result != AUTH_DENY && allow_anonymous)\n +\t\t\tbreak;\n ++\n +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n +\t\t\tmod = auth_modules[i];\n +\t\t\tif (mod->challenge_params)\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\t\t\t\t\t\t mod->scheme);\n +\t\t\tstring_list_append(&hdrs, challenge);\n +\t\t}\n -+\t\t*wr = send_http_error(1, 401, \"Unauthorized\", -1, &hdrs, *wr);\n ++\n ++\t\tfor (i = 0; i < extra_headers.nr; i++)\n ++\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n ++\n ++\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n ++\t\t\t\t &hdrs, *wr);\n +\t}\n +\n +\tstrbuf_list_free(split);\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\treturn result == AUTH_ALLOW ||\n +\t (result == AUTH_UNKNOWN && allow_anonymous);\n +}\n ++\n ++static int split_auth_param(const char *str, char **scheme, char **val, int required_val)\n ++{\n ++\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n ++\n ++\tif (!p[0])\n ++\t\treturn -1;\n ++\n ++\t/* trim trailing ':' */\n ++\tif (p[1])\n ++\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n ++\n ++\tif (required_val && !p[1])\n ++\t\treturn -1;\n ++\n ++\t*scheme = strbuf_detach(p[0], NULL);\n ++\n ++\tif (p[1])\n ++\t\t*val = strbuf_detach(p[1], NULL);\n ++\n ++\tstrbuf_list_free(p);\n ++\treturn 0;\n ++}\n ++\n ++static int read_auth_config(const char *name, const char *val, void *data)\n ++{\n ++\tint ret = 0;\n ++\tchar *scheme = NULL;\n ++\tchar *token = NULL;\n ++\tchar *challenge = NULL;\n ++\tstruct auth_module *mod = NULL;\n ++\n ++\tif (!strcmp(name, \"auth.challenge\")) {\n ++\t\tif (split_auth_param(val, &scheme, &challenge, 0)) {\n ++\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n ++\t\t\tgoto cleanup;\n ++\t\t}\n ++\n ++\t\tmod = create_auth_module(scheme, challenge);\n ++\t\tif (add_auth_module(mod)) {\n ++\t\t\tret = error(\"failed to add auth module '%s'\", val);\n ++\t\t\tgoto cleanup;\n ++\t\t}\n ++\t}\n ++\tif (!strcmp(name, \"auth.token\")) {\n ++\t\tif (split_auth_param(val, &scheme, &token, 1)) {\n ++\t\t\tret = error(\"invalid auth token '%s'\", val);\n ++\t\t\tgoto cleanup;\n ++\t\t}\n ++\n ++\t\tmod = get_auth_module(scheme);\n ++\t\tif (!mod) {\n ++\t\t\tret = error(\"auth scheme not defined '%s'\\n\", scheme);\n ++\t\t\tgoto cleanup;\n ++\t\t}\n ++\n ++\t\tstring_list_append(mod->tokens, token);\n ++\t}\n ++\tif (!strcmp(name, \"auth.allowanonymous\")) {\n ++\t\tallow_anonymous = git_config_bool(name, val);\n ++\t}\n ++\tif (!strcmp(name, \"auth.extraheader\")) {\n ++\t\tstrvec_push(&extra_headers, val);\n ++\t}\n ++\n ++cleanup:\n ++\tfree(scheme);\n ++\tfree(token);\n ++\tfree(challenge);\n ++\n ++\treturn ret;\n ++}\n +\n static enum worker_result dispatch(struct req *req)\n {\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req,\n +\t\treturn wr;\n +\n \tif (is_git_request(req))\n --\t\treturn do__git(req, NULL);\n +-\t\treturn do__git(req);\n +\t\treturn do__git(req, user);\n \n - \treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n + \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n -@@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)\n - \tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n - \tint worker_mode = 0;\n - \tint i;\n -+\tstruct auth_module *mod = NULL;\n - \n - \ttrace2_cmd_name(\"test-http-server\");\n - \tsetup_git_directory_gently(NULL);\n @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n -+\t\tif (skip_prefix(arg, \"--allow-anonymous\", &v)) {\n -+\t\t\tallow_anonymous = 1;\n -+\t\t\tcontinue;\n -+\t\t}\n -+\t\tif (skip_prefix(arg, \"--auth=\", &v)) {\n -+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n -+\n -+\t\t\tif (!p[0]) {\n -+\t\t\t\terror(\"invalid argument '%s'\", v);\n ++\t\tif (skip_prefix(arg, \"--auth-config=\", &v)) {\n ++\t\t\tif (!strlen(v)) {\n ++\t\t\t\terror(\"invalid argument - missing file path\");\n +\t\t\t\tusage(test_http_auth_usage);\n +\t\t\t}\n +\n -+\t\t\t/* trim trailing ':' */\n -+\t\t\tif (p[1])\n -+\t\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n -+\n -+\t\t\tif (get_auth_module(p[0]->buf)) {\n -+\t\t\t\terror(\"duplicate auth scheme '%s'\\n\", p[0]->buf);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\tmod = xmalloc(sizeof(struct auth_module));\n -+\t\t\tmod->scheme = xstrdup(p[0]->buf);\n -+\t\t\tmod->challenge_params = p[1] ? xstrdup(p[1]->buf) : NULL;\n -+\t\t\tCALLOC_ARRAY(mod->tokens, 1);\n -+\t\t\tstring_list_init_dup(mod->tokens);\n -+\n -+\t\t\tadd_auth_module(mod);\n -+\n -+\t\t\tstrbuf_list_free(p);\n -+\t\t\tcontinue;\n -+\t\t}\n -+\t\tif (skip_prefix(arg, \"--auth-token=\", &v)) {\n -+\t\t\tstruct strbuf **p = strbuf_split_str(v, ':', 2);\n -+\t\t\tif (!p[0]) {\n -+\t\t\t\terror(\"invalid argument '%s'\", v);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\tif (!p[1]) {\n -+\t\t\t\terror(\"missing token value '%s'\\n\", v);\n -+\t\t\t\tusage(test_http_auth_usage);\n -+\t\t\t}\n -+\n -+\t\t\t/* trim trailing ':' */\n -+\t\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n -+\n -+\t\t\tmod = get_auth_module(p[0]->buf);\n -+\t\t\tif (!mod) {\n -+\t\t\t\terror(\"auth scheme not defined '%s'\\n\", p[0]->buf);\n ++\t\t\tif (git_config_from_file(read_auth_config, v, NULL)) {\n ++\t\t\t\terror(\"failed to read auth config file '%s'\", v);\n +\t\t\t\tusage(test_http_auth_usage);\n +\t\t\t}\n +\n -+\t\t\tstring_list_append(mod->tokens, p[1]->buf);\n -+\t\t\tstrbuf_list_free(p);\n +\t\t\tcontinue;\n +\t\t}\n \n -: ----------- > 7: 9c4d25945dd http: replace unsafe size_t multiplication with st_mult\n -: ----------- > 8: 65a620b08ef strvec: expose strvec_push_nodup for external use\n 1: b5b56ccd941 ! 9: bcfec529d95 http: read HTTP WWW-Authenticate response headers\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n \n +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n +{\n -+\tsize_t size = eltsize * nmemb;\n ++\tsize_t size = st_mult(eltsize, nmemb);\n +\tstruct strvec *values = &http_auth.wwwauth_headers;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tconst char *val;\n -+\tconst char *z = NULL;\n +\n +\t/*\n +\t * Header lines may not come NULL-terminated from libcurl so we must\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * This line could be a continuation of the previously matched header\n +\t * field. If this is the case then we should append this value to the\n +\t * end of the previously consumed value.\n ++\t * Continuation lines start with at least one whitespace, maybe more,\n ++\t * so we should collapse these down to a single SP (valid per the spec).\n +\t */\n +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n -+\t\tconst char **v = values->v + values->nr - 1;\n -+\t\tchar *append = xstrfmt(\"%s%.*s\", *v, (int)(size - 1), ptr + 1);\n ++\t\t/* Trim leading whitespace from this continuation hdr line. */\n ++\t\tstrbuf_ltrim(&buf);\n +\n -+\t\tfree((void*)*v);\n -+\t\t*v = append;\n ++\t\t/*\n ++\t\t * At this point we should always have at least one existing\n ++\t\t * value, even if it is empty. Do not bother appending the new\n ++\t\t * value if this continuation header is itself empty.\n ++\t\t */\n ++\t\tif (!values->nr) {\n ++\t\t\tBUG(\"should have at least one existing header value\");\n ++\t\t} else if (buf.len) {\n ++\t\t\tconst char *prev = values->v[values->nr - 1];\n ++\t\t\tstruct strbuf append = STRBUF_INIT;\n ++\t\t\tstrbuf_addstr(&append, prev);\n ++\n ++\t\t\t/* Join two non-empty values with a single space. */\n ++\t\t\tif (append.len)\n ++\t\t\t\tstrbuf_addch(&append, ' ');\n ++\n ++\t\t\tstrbuf_addbuf(&append, &buf);\n ++\n ++\t\t\tstrvec_pop(values);\n ++\t\t\tstrvec_push_nodup(values, strbuf_detach(&append, NULL));\n ++\t\t}\n +\n +\t\tgoto exit;\n +\t}\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * We only care about the last HTTP request response's headers so clear\n +\t * the existing array.\n +\t */\n -+\tif (skip_iprefix(buf.buf, \"http/\", &z))\n ++\tif (istarts_with(buf.buf, \"http/\"))\n +\t\tstrvec_clear(values);\n +\n +exit:\n 2: d02875dda7c < -: ----------- credential: add WWW-Authenticate header to cred requests\n 4: 98dd286db7c < -: ----------- test-http-server: add HTTP error response function\n 6: 0a0f4fd10c8 < -: ----------- test-http-server: pass Git requests to http-backend\n 8: 8ecf6383522 ! 10: af66d2d2ede t5556: add HTTP authentication tests\n @@ Metadata\n Author: Matthew John Cheetham \n \n ## Commit message ##\n - t5556: add HTTP authentication tests\n + credential: add WWW-Authenticate header to cred requests\n \n - Add a series of tests to exercise the HTTP authentication header parsing\n + Add the value of the WWW-Authenticate response header to credential\n + requests. Credential helpers that understand and support HTTP\n + authentication and authorization can use this standard header (RFC 2616\n + Section 14.47 [1]) to generate valid credentials.\n +\n + WWW-Authenticate headers can contain information pertaining to the\n + authority, authentication mechanism, or extra parameters/scopes that are\n + required.\n +\n + The current I/O format for credential helpers only allows for unique\n + names for properties/attributes, so in order to transmit multiple header\n + values (with a specific order) we introduce a new convention whereby a\n + C-style array syntax is used in the property name to denote multiple\n + ordered values for the same property.\n +\n + In this case we send multiple `wwwauth[]` properties where the order\n + that the repeated attributes appear in the conversation reflects the\n + order that the WWW-Authenticate headers appeared in the HTTP response.\n +\n + Add a set of tests to exercise the HTTP authentication header parsing\n and the interop with credential helpers. Credential helpers will receive\n WWW-Authenticate information in credential requests.\n \n + [1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n +\n Signed-off-by: Matthew John Cheetham \n \n + ## Documentation/git-credential.txt ##\n +@@ Documentation/git-credential.txt: separated by an `=` (equals) sign, followed by a newline.\n + The key may contain any bytes except `=`, newline, or NUL. The value may\n + contain any bytes except newline or NUL.\n + \n +-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n ++Attributes with keys that end with C-style array brackets `[]` can have\n ++multiple values. Each instance of a multi-valued attribute forms an\n ++ordered list of values - the order of the repeated attributes defines\n ++the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n ++acts to clear any previous entries and reset the list.\n ++\n ++In all cases, all bytes are treated as-is (i.e., there is no quoting,\n + and one cannot transmit a value with newline or NUL in it). The list of\n + attributes is terminated by a blank line or end-of-file.\n + \n +@@ Documentation/git-credential.txt: empty string.\n + Components which are missing from the URL (e.g., there is no\n + username in the example above) will be left unset.\n + \n ++`wwwauth[]`::\n ++\n ++\tWhen an HTTP response is received by Git that includes one or more\n ++\t'WWW-Authenticate' authentication headers, these will be passed by Git\n ++\tto credential helpers.\n +++\n ++Each 'WWW-Authenticate' header value is passed as a multi-valued\n ++attribute 'wwwauth[]', where the order of the attributes is the same as\n ++they appear in the HTTP response. This attribute is 'one-way' from Git\n ++to pass additional information to credential helpers.\n ++\n + Unrecognised attributes are silently discarded.\n + \n + GIT\n +\n + ## credential.c ##\n +@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,\n + \tfprintf(fp, \"%s=%s\\n\", key, value);\n + }\n + \n ++static void credential_write_strvec(FILE *fp, const char *key,\n ++\t\t\t\t const struct strvec *vec)\n ++{\n ++\tint i = 0;\n ++\tconst char *full_key = xstrfmt(\"%s[]\", key);\n ++\tfor (; i < vec->nr; i++) {\n ++\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n ++\t}\n ++\tfree((void*)full_key);\n ++}\n ++\n + void credential_write(const struct credential *c, FILE *fp)\n + {\n + \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n +@@ credential.c: void credential_write(const struct credential *c, FILE *fp)\n + \tcredential_write_item(fp, \"path\", c->path, 0);\n + \tcredential_write_item(fp, \"username\", c->username, 0);\n + \tcredential_write_item(fp, \"password\", c->password, 0);\n ++\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n + }\n + \n + static int run_credential_helper(struct credential *c,\n +\n ## t/helper/test-credential-helper-replay.sh (new) ##\n @@\n +cmd=$1\n @@ t/helper/test-credential-helper-replay.sh (new)\n +fi\n \n ## t/t5556-http-auth.sh ##\n -@@ t/t5556-http-auth.sh: PID_FILE=\"$(pwd)\"/pid-file.pid\n - SERVER_LOG=\"$(pwd)\"/OUT.server.log\n +@@ t/t5556-http-auth.sh: PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n + SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n \n PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n +CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n @@ t/t5556-http-auth.sh: start_http_server () {\n \tstop_http_server &&\n -\trm -f OUT.*\n +\trm -f OUT.* &&\n -+\trm -f *.cred\n ++\trm -f *.cred &&\n ++\trm -f auth.config\n }\n \n test_expect_success 'http auth anonymous no challenge' '\n -@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n + \ttest_when_finished \"per_test_cleanup\" &&\n +-\tstart_http_server &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t allowAnonymous = true\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n + \n + \t# Attempt to read from a protected repository\n \tgit ls-remote $ORIGIN_URL\n '\n \n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n +\texport USERPASS64 &&\n +\n -+\tstart_http_server \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=basic:$USERPASS64 &&\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n ++test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\t# base64(\"alice:secret-passwd\")\n ++\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n ++\texport USERPASS64 &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\t extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n ++\n ++\tcat >get-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\twwwauth[]=basic realm=\"example.com\"\n ++\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n ++\tEOF\n ++\n ++\tcat >store-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tcat >get-response.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n ++\n ++\ttest_cmp get-expected.cred get-actual.cred &&\n ++\ttest_cmp store-expected.cred store-actual.cred\n ++'\n ++\n ++test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\t# base64(\"alice:secret-passwd\")\n ++\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n ++\texport USERPASS64 &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n ++\n ++\tcat >get-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n ++\twwwauth[]=basic realm=\"example.com\"\n ++\tEOF\n ++\n ++\tcat >store-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tcat >get-response.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n ++\n ++\ttest_cmp get-expected.cred get-actual.cred &&\n ++\ttest_cmp store-expected.cred store-actual.cred\n ++'\n ++\n ++test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\t# base64(\"alice:secret-passwd\")\n ++\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n ++\texport USERPASS64 &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\t extraheader = \"WWW-Authenticate:\"\n ++\t extraheader = \" \"\n ++\t extraheader = \" bearer authority=\\\"id.example.com\\\"\"\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n ++\n ++\tcat >get-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\twwwauth[]=basic realm=\"example.com\"\n ++\twwwauth[]=bearer authority=\"id.example.com\"\n ++\tEOF\n ++\n ++\tcat >store-expected.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tcat >get-response.cred <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HOST_PORT\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++\n ++\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n ++\n ++\ttest_cmp get-expected.cred get-actual.cred &&\n ++\ttest_cmp store-expected.cred store-actual.cred\n ++'\n ++\n +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n +\t# base64(\"alice:secret-passwd\")\n +\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n +\texport USERPASS64 &&\n +\n -+\tstart_http_server \\\n -+\t\t--auth=foobar:alg=test\\ widget=1 \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=basic:$USERPASS64 &&\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = \"foobar:alg=test widget=1\"\n ++\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n +\t# base64(\"alice:secret-passwd\")\n +\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n +\texport USERPASS64 &&\n -+\tstart_http_server \\\n -+\t\t--auth=bearer:authority=\\\"id.example.com\\\"\\ q=1\\ p=0 \\\n -+\t\t--auth=basic:realm=\\\"example.com\\\" \\\n -+\t\t--auth-token=basic:$USERPASS64 &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n ++\t challenge = basic:realm=\\\"example.com\\\"\n ++\t token = basic:$USERPASS64\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tcat >get-expected.cred <<-EOF &&\n +\tprotocol=http\n\n-- \ngitgitgadget\n"},{"id":"470117","messageId":"706fb3781bd383380a7b1fd30495eb2da970b5ec.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 04/10] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:04Z","receivedAt":"2023-01-11T22:13:33Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a mini HTTP server helper that in the future will be enhanced\nto provide a frontend for the git-http-backend, with support for\narbitrary authentication schemes.\n\nRight now, test-http-server is a pared-down copy of the git-daemon that\nalways returns a 501 Not Implemented response to all callers.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++\n 4 files changed, 396 insertions(+), 2 deletions(-)\n create mode 100644 t/helper/test-http-server.c\n\ndiff --git a/Makefile b/Makefile\nindex 2654094dbb5..3cd61c792ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o\n # Do not add more tests here unless they have extra dependencies. Add\n # them in TEST_BUILTINS_OBJS above.\n TEST_PROGRAMS_NEED_X += test-fake-ssh\n+TEST_PROGRAMS_NEED_X += test-http-server\n TEST_PROGRAMS_NEED_X += test-tool\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 2f6e0197ffa..5d949dcb16c 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -961,6 +961,9 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+target_link_libraries(test-http-server common-main)\n+\n #reftable-tests\n parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n@@ -980,6 +983,11 @@ if(MSVC)\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n \tset_target_properties(test-fake-ssh test-tool\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n endif()\n \n #wrapper scripts\n@@ -987,8 +995,7 @@ set(wrapper_scripts\n \tgit git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)\n \n set(wrapper_test_scripts\n-\ttest-fake-ssh test-tool)\n-\n+\ttest-http-server test-fake-ssh test-tool)\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..9aa9c752997 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+/test-http-server\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..11071b1dd89\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,385 @@\n+#include \"daemon-utils.h\"\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+;\n+\n+static unsigned int timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+/*\n+ * The code in this section is used by \"worker\" instances to service\n+ * a single connection from a client. The worker talks to the client\n+ * on 0 and 1.\n+ */\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\n+\t/*\n+\t * Various errors while processing the request and/or the response.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_IO_ERROR = 1<<0,\n+\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 1<<1,\n+};\n+\n+static enum worker_result worker(void)\n+{\n+\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0, logerror);\n+\n+\twhile (1) {\n+\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n+\t\t\tlogerror(\"unable to write response\");\n+\t\t\twr = WR_IO_ERROR;\n+\t\t}\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(STDIN_FILENO);\n+\tclose(STDOUT_FILENO);\n+\n+\treturn !!(wr & WR_IO_ERROR);\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child *first_child;\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child(first_child);\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+/*\n+ * This section is executed by both the primary instance and all\n+ * worker instances. So, yes, each child-process re-parses the\n+ * command line argument and re-discovers how it should behave.\n+ */\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\ttrace2_cmd_list_config();\n+\ttrace2_cmd_list_env_vars();\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\n-- \ngitgitgadget\n\n"},{"id":"470119","messageId":"6f66bf146b4d6e4044b3c6c2224795918191bc3f.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 05/10] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:05Z","receivedAt":"2023-01-11T22:14:10Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a function to the test-http-server test helper to write more\nfull and valid HTTP error responses, including all the standard response\nheaders like `Server` and `Date`.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 303 +++++++++++++++++++++++++++++++++++-\n t/t5556-http-auth.sh | 106 +++++++++++++\n 2 files changed, 404 insertions(+), 5 deletions(-)\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 11071b1dd89..67bc16354a1 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -83,9 +83,297 @@ enum worker_result {\n \tWR_HANGUP = 1<<1,\n };\n \n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tssize_t content_length;\n+};\n+\n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = -1 \\\n+\t}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n+static enum worker_result send_http_error(\n+\tint fd,\n+\tint http_code, const char *http_code_name,\n+\tint retry_after_seconds, struct string_list *response_headers,\n+\tenum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n+\t\t http_code, http_code_name);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_OK | WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* also store common request headers as struct req members */\n+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\treq->content_length = strtol(hv, &hp, 10);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->content_length >= 0)\n+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\t/*\n+\t\t * This regular expression matches all dumb and smart HTTP\n+\t\t * requests that are currently in use, and defined in\n+\t\t * Documentation/gitprotocol-http.txt.\n+\t\t *\n+\t\t */\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\t/*\n+\t * Note that we always respond with a 200 OK response even if the\n+\t * http-backend process exits with an error. This helper is intended\n+\t * only to be used to exercise the HTTP auth handling in the Git client,\n+\t * and specifically around authentication (not handled by http-backend).\n+\t *\n+\t * If we wanted to respond with a more 'valid' HTTP response status then\n+\t * we'd need to buffer the output of http-backend, wait for and grok the\n+\t * exit status of the process, then write the HTTP status line followed\n+\t * by the http-backend output. This is outside of the scope of this test\n+\t * helper's use at time of writing.\n+\t *\n+\t * The important auth responses (401) we are handling prior to getting\n+\t * to this point.\n+\t */\n+\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n+\t\t\treq->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n+\t\t\t\treq->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n+\t\t\t\treq->content_type);\n+\tif (req->content_length >= 0)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n+\t\t\t\t(intmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(STDOUT_FILENO);\n+\tclose(STDIN_FILENO);\n+\treturn !!res;\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\tif (is_git_request(req))\n+\t\treturn do__git(req);\n+\n+\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_OK | WR_HANGUP);\n+}\n+\n static enum worker_result worker(void)\n {\n-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -96,11 +384,16 @@ static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n-\t\t\tlogerror(\"unable to write response\");\n-\t\t\twr = WR_IO_ERROR;\n-\t\t}\n+\t\treq__release(&req);\n+\n+\t\talarm(timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n \n+\t\twr = dispatch(&req);\n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..65105a5a6a9\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,106 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n+\n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n+per_test_cleanup () {\n+\tstop_http_server &&\n+\trm -f OUT.*\n+}\n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\tstart_http_server &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"470120","messageId":"9c4d25945dda0c5b5a16f0007dfb0575c59facf7.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 07/10] http: replace unsafe size_t multiplication with st_mult","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:07Z","receivedAt":"2023-01-11T22:14:27Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nReplace direct multiplication of two size_t parameters in curl response\nstream handling callback functions with `st_mult` to guard against\noverflows.\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..a2a80318bb2 100644\n--- a/http.c\n+++ b/http.c\n@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n \n size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct buffer *buffer = buffer_;\n \n \tif (size > buffer->buf.len - buffer->posn)\n@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n \n size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct strbuf *buffer = buffer_;\n \n \tstrbuf_add(buffer, ptr, size);\n-- \ngitgitgadget\n\n"},{"id":"470121","messageId":"c3c3d17a688963acc180e3bb7bbb4deb32a94304.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 06/10] test-http-server: add simple authentication","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:06Z","receivedAt":"2023-01-11T22:14:27Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd simple authentication to the test-http-server test helper.\nAuthentication schemes and sets of valid tokens can be specified via\na configuration file (in the normal gitconfig file format).\nIncoming requests are compared against the set of valid schemes and\ntokens and only approved if a matching token is found, or if no auth\nwas provided and anonymous auth is enabled.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 246 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 244 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 67bc16354a1..dcc326c8652 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -7,6 +7,7 @@\n #include \"version.h\"\n #include \"dir.h\"\n #include \"date.h\"\n+#include \"config.h\"\n \n #define TR2_CAT \"test-http-server\"\n \n@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =\n \" [--timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n+\" [--auth-config=]\\n\"\n ;\n \n static unsigned int timeout;\n@@ -317,7 +319,7 @@ static int is_git_request(struct req *req)\n \t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n }\n \n-static enum worker_result do__git(struct req *req)\n+static enum worker_result do__git(struct req *req, const char *user)\n {\n \tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -341,6 +343,9 @@ static enum worker_result do__git(struct req *req)\n \tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n \t\treturn error(_(\"could not send '%s'\"), ok);\n \n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n \tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n \t\t\treq->uri_path.buf);\n@@ -362,10 +367,234 @@ static enum worker_result do__git(struct req *req)\n \treturn !!res;\n }\n \n+enum auth_result {\n+\t/* No auth module matches the request. */\n+\tAUTH_UNKNOWN = 0,\n+\n+\t/* Auth module denied the request. */\n+\tAUTH_DENY = 1,\n+\n+\t/* Auth module successfully validated the request. */\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tchar *scheme;\n+\tchar *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+static struct strvec extra_headers = STRVEC_INIT;\n+\n+static struct auth_module *create_auth_module(const char *scheme,\n+\t\t\t\t\t const char *challenge)\n+{\n+\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n+\tmod->scheme = xstrdup(scheme);\n+\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n+\tCALLOC_ARRAY(mod->tokens, 1);\n+\tstring_list_init_dup(mod->tokens);\n+\treturn mod;\n+}\n+\n+static struct auth_module *get_auth_module(const char *scheme)\n+{\n+\tint i;\n+\tstruct auth_module *mod;\n+\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme))\n+\t\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int add_auth_module(struct auth_module *mod)\n+{\n+\tif (get_auth_module(mod->scheme))\n+\t\treturn error(\"duplicate auth scheme '%s'\\n\", mod->scheme);\n+\n+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\tauth_modules[auth_modules_nr++] = mod;\n+\n+\treturn 0;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/*\n+\t * Check all auth modules and try to validate the request.\n+\t * The first Authorization header that matches a known auth module\n+\t * scheme will be consulted to either approve or deny the request.\n+\t * If no module is found, or if there is no valid token, then 401 error.\n+\t * Otherwise, only permit the request if anonymous auth is enabled.\n+\t * It's atypical for user agents/clients to send multiple Authorization\n+\t * headers, but not explicitly forbidden or defined.\n+\t */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (!split[0] || !split[1]) continue;\n+\n+\t\t\t/* trim trailing space ' ' */\n+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(split[0]->buf);\n+\t\t\tif (mod) {\n+\t\t\t\tresult = AUTH_DENY;\n+\n+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (result != AUTH_DENY && allow_anonymous)\n+\t\t\tbreak;\n+\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\n+\t\tfor (i = 0; i < extra_headers.nr; i++)\n+\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n+\n+\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n+\t\t\t\t &hdrs, *wr);\n+\t}\n+\n+\tstrbuf_list_free(split);\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n+static int split_auth_param(const char *str, char **scheme, char **val, int required_val)\n+{\n+\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n+\n+\tif (!p[0])\n+\t\treturn -1;\n+\n+\t/* trim trailing ':' */\n+\tif (p[1])\n+\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\tif (required_val && !p[1])\n+\t\treturn -1;\n+\n+\t*scheme = strbuf_detach(p[0], NULL);\n+\n+\tif (p[1])\n+\t\t*val = strbuf_detach(p[1], NULL);\n+\n+\tstrbuf_list_free(p);\n+\treturn 0;\n+}\n+\n+static int read_auth_config(const char *name, const char *val, void *data)\n+{\n+\tint ret = 0;\n+\tchar *scheme = NULL;\n+\tchar *token = NULL;\n+\tchar *challenge = NULL;\n+\tstruct auth_module *mod = NULL;\n+\n+\tif (!strcmp(name, \"auth.challenge\")) {\n+\t\tif (split_auth_param(val, &scheme, &challenge, 0)) {\n+\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = create_auth_module(scheme, challenge);\n+\t\tif (add_auth_module(mod)) {\n+\t\t\tret = error(\"failed to add auth module '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t}\n+\tif (!strcmp(name, \"auth.token\")) {\n+\t\tif (split_auth_param(val, &scheme, &token, 1)) {\n+\t\t\tret = error(\"invalid auth token '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = get_auth_module(scheme);\n+\t\tif (!mod) {\n+\t\t\tret = error(\"auth scheme not defined '%s'\\n\", scheme);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tstring_list_append(mod->tokens, token);\n+\t}\n+\tif (!strcmp(name, \"auth.allowanonymous\")) {\n+\t\tallow_anonymous = git_config_bool(name, val);\n+\t}\n+\tif (!strcmp(name, \"auth.extraheader\")) {\n+\t\tstrvec_push(&extra_headers, val);\n+\t}\n+\n+cleanup:\n+\tfree(scheme);\n+\tfree(token);\n+\tfree(challenge);\n+\n+\treturn ret;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n \tif (is_git_request(req))\n-\t\treturn do__git(req);\n+\t\treturn do__git(req, user);\n \n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n@@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--auth-config=\", &v)) {\n+\t\t\tif (!strlen(v)) {\n+\t\t\t\terror(\"invalid argument - missing file path\");\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (git_config_from_file(read_auth_config, v, NULL)) {\n+\t\t\t\terror(\"failed to read auth config file '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tcontinue;\n+\t\t}\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\n-- \ngitgitgadget\n\n"},{"id":"470122","messageId":"bcfec529d9581db3181cc6f3b3d4b4b8d315ca0c.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:09Z","receivedAt":"2023-01-11T22:14:38Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++++\n http.c | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 114 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex a2a80318bb2..10882034145 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,102 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = st_mult(eltsize, nmemb);\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t * Continuation lines start with at least one whitespace, maybe more,\n+\t * so we should collapse these down to a single SP (valid per the spec).\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\t/* Trim leading whitespace from this continuation hdr line. */\n+\t\tstrbuf_ltrim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tconst char *prev = values->v[values->nr - 1];\n+\t\t\tstruct strbuf append = STRBUF_INIT;\n+\t\t\tstrbuf_addstr(&append, prev);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tif (append.len)\n+\t\t\t\tstrbuf_addch(&append, ' ');\n+\n+\t\t\tstrbuf_addbuf(&append, &buf);\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_push_nodup(values, strbuf_detach(&append, NULL));\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (istarts_with(buf.buf, \"http/\"))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1960,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"470123","messageId":"65a620b08ef359e29d678497f1b529e3ce6477b1.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 08/10] strvec: expose strvec_push_nodup for external use","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:08Z","receivedAt":"2023-01-11T22:14:38Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRemove the static modifier from the existing `strvec_push_nodup`\nfunction and define the function is `strvec.h` to make it available for\nother callers, making it now possible to append to a `struct strvec`\narray without duplication.\n\nSigned-off-by: Matthew John Cheetham \n---\n strvec.c | 2 +-\n strvec.h | 3 +++\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/strvec.c b/strvec.c\nindex 61a76ce6cb9..26e8751cae0 100644\n--- a/strvec.c\n+++ b/strvec.c\n@@ -10,7 +10,7 @@ void strvec_init(struct strvec *array)\n \tmemcpy(array, &blank, sizeof(*array));\n }\n \n-static void strvec_push_nodup(struct strvec *array, const char *value)\n+void strvec_push_nodup(struct strvec *array, const char *value)\n {\n \tif (array->v == empty_strvec)\n \t\tarray->v = NULL;\ndiff --git a/strvec.h b/strvec.h\nindex 9f55c8766ba..5d61dd73680 100644\n--- a/strvec.h\n+++ b/strvec.h\n@@ -43,6 +43,9 @@ struct strvec {\n */\n void strvec_init(struct strvec *);\n \n+/* Push a string onto the end of the array without copying. */\n+void strvec_push_nodup(struct strvec *array, const char *value);\n+\n /* Push a copy of a string onto the end of the array. */\n const char *strvec_push(struct strvec *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"470124","messageId":"af66d2d2ede2a502f32d74c86f302598c68d1476.1673475190.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-11T22:13:10Z","receivedAt":"2023-01-11T22:14:38Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 +-\n credential.c | 12 +\n t/helper/test-credential-helper-replay.sh | 14 ++\n t/t5556-http-auth.sh | 270 +++++++++++++++++++++-\n 4 files changed, 312 insertions(+), 3 deletions(-)\n create mode 100755 t/helper/test-credential-helper-replay.sh\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..8a3ad6c0ae2 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,17 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tint i = 0;\n+\tconst char *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree((void*)full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +281,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/helper/test-credential-helper-replay.sh b/t/helper/test-credential-helper-replay.sh\nnew file mode 100755\nindex 00000000000..03e5e63dad6\n--- /dev/null\n+++ b/t/helper/test-credential-helper-replay.sh\n@@ -0,0 +1,14 @@\n+cmd=$1\n+teefile=$cmd-actual.cred\n+catfile=$cmd-response.cred\n+rm -f $teefile\n+while read line;\n+do\n+\tif test -z \"$line\"; then\n+\t\tbreak;\n+\tfi\n+\techo \"$line\" >> $teefile\n+done\n+if test \"$cmd\" = \"get\"; then\n+\tcat $catfile\n+fi\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 65105a5a6a9..a8dbee6ca40 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -27,6 +27,8 @@ PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n \n PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n+\t&& export CREDENTIAL_HELPER\n \n test_expect_success 'setup repos' '\n \ttest_create_repo \"$REPO_DIR\" &&\n@@ -92,15 +94,279 @@ start_http_server () {\n \n per_test_cleanup () {\n \tstop_http_server &&\n-\trm -f OUT.*\n+\trm -f OUT.* &&\n+\trm -f *.cred &&\n+\trm -f auth.config\n }\n \n test_expect_success 'http auth anonymous no challenge' '\n \ttest_when_finished \"per_test_cleanup\" &&\n-\tstart_http_server &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t allowAnonymous = true\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n \n \t# Attempt to read from a protected repository\n \tgit ls-remote $ORIGIN_URL\n '\n \n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\t extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\t extraheader = \"WWW-Authenticate:\"\n+\t extraheader = \" \"\n+\t extraheader = \" bearer authority=\\\"id.example.com\\\"\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bearer authority=\"id.example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = \"foobar:alg=test widget=1\"\n+\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >store-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp store-expected.cred store-actual.cred\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t challenge = basic:realm=\\\"example.com\\\"\n+\t token = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcat >get-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >erase-expected.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\tcat >get-response.cred <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\tEOF\n+\n+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n+\n+\ttest_cmp get-expected.cred get-actual.cred &&\n+\ttest_cmp erase-expected.cred erase-actual.cred\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"470142","messageId":"230112.86wn5s2l5r.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"bcfec529d9581db3181cc6f3b3d4b4b8d315ca0c.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-12T08:41:08Z","receivedAt":"2023-01-12T08:51:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> [...]\n> +\t\t} else if (buf.len) {\n> +\t\t\tconst char *prev = values->v[values->nr - 1];\n> +\t\t\tstruct strbuf append = STRBUF_INIT;\n> +\t\t\tstrbuf_addstr(&append, prev);\n> +\n> +\t\t\t/* Join two non-empty values with a single space. */\n> +\t\t\tif (append.len)\n> +\t\t\t\tstrbuf_addch(&append, ' ');\n> +\n> +\t\t\tstrbuf_addbuf(&append, &buf);\n> +\n> +\t\t\tstrvec_pop(values);\n> +\t\t\tstrvec_push_nodup(values, strbuf_detach(&append, NULL));\n> +\t\t}\n> +\n\nI've written something like the strvec_push_nodup() patch that preceded\nthis myself for similar reasons, and as recently noted in [1] I think\nsuch a thing (although I implemented a different interface) might be\nuseful in general.\n\nBut this really doesn't seem like a good justification for adding this\nnew API. Let's instead do:\n\n\t} else if (buf.len) {\n\t\tconst char *prev = values->v[values->nr - 1];\n\t\t/* Join two non-empty values with a single space. */\n\t\tconst char *const sp = *prev ? \" \" : \"\"\n\n\t\tstrvec_pop(values);\n\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n\t}\n\nThere may be cases where a public strvec_push_nodup() simplifies things,\nbut this doesn't seem like such a case, just use strvec_pushf() directly\ninstead, and skip the strbuf & strbuf_detach().\n\nI haven't compiled/tested the above, so there may e.g. be a typo in\nthere. But I think the general concept should work in this case.\n\n1. https://lore.kernel.org/git/RFC-cover-0.5-00000000000-20221215T090226Z-avarab@gmail.com/\n"},{"id":"470143","messageId":"230112.86sfgg2kuj.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"af66d2d2ede2a502f32d74c86f302598c68d1476.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-12T08:48:43Z","receivedAt":"2023-01-12T08:59:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> [...]\n> +static void credential_write_strvec(FILE *fp, const char *key,\n> +\t\t\t\t const struct strvec *vec)\n> +{\n> +\tint i = 0;\n> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n> +\tfor (; i < vec->nr; i++) {\n> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n\nStyle: Don't mismatch types if there's no good reason. Use \"size_t i\" here, also let's do:\n\n\tfor (size_t i = 0; ....\n\nI.e. no reason to declare it earlier.\n\n> +\t}\n> +\tfree((void*)full_key);\n\nJust don't add a \"const\" to that \"full_key\" and skip the cast with\nfree() here.\n\n> +++ b/t/helper/test-credential-helper-replay.sh\n\nI see to my surprise that we have one existing *.sh helper in that\ndirectory, but in any case...\n\n> @@ -0,0 +1,14 @@\n> +cmd=$1\n> +teefile=$cmd-actual.cred\n> +catfile=$cmd-response.cred\n> +rm -f $teefile\n> +while read line;\n> +do\n> +\tif test -z \"$line\"; then\n> +\t\tbreak;\n> +\tfi\n> +\techo \"$line\" >> $teefile\n> +done\n\nIt looks like you're re-inventing \"sed\" here, isn't this whole loop just\n\n\tsed -n -e '/^$/q' -n 'p'\n\nAnd then you can skip the \"rm\" before, as you could just clobber the\nthing.\n\n> +if test \"$cmd\" = \"get\"; then\n> +\tcat $catfile\n> +fi\n> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n> index 65105a5a6a9..a8dbee6ca40 100755\n> --- a/t/t5556-http-auth.sh\n> +++ b/t/t5556-http-auth.sh\n> @@ -27,6 +27,8 @@ PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n> SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n> \n> PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n> +CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n> +\t&& export CREDENTIAL_HELPER\n\n...(continued from above): Let's just use write_script() here or\nwhatever, i.e. no reason to make this a global script, it's just used in\nthis one test, so it can set it up.\n> \n> test_expect_success 'setup repos' '\n> \ttest_create_repo \"$REPO_DIR\" &&\n> @@ -92,15 +94,279 @@ start_http_server () {\n> \n> per_test_cleanup () {\n> \tstop_http_server &&\n> -\trm -f OUT.*\n> +\trm -f OUT.* &&\n> +\trm -f *.cred &&\n> +\trm -f auth.config\n> }\n> \n> test_expect_success 'http auth anonymous no challenge' '\n> \ttest_when_finished \"per_test_cleanup\" &&\n> -\tstart_http_server &&\n> +\n> +\tcat >auth.config <<-EOF &&\n> +\t[auth]\n> +\t allowAnonymous = true\n\nMixed tab/space. Use \"\\t\" not 4x \" \" (ditto below).\n"},{"id":"470225","messageId":"e1b1a53b-69d7-6fd6-513d-0f6cb3751b51@github.com","threadId":"58425","inReplyTo":"74b0de14185120c9d53d7470e59f57fa20a1927f.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 01/10] daemon: libify socket setup and option functions","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T19:35:10Z","receivedAt":"2023-01-12T19:44:39Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Extract functions for setting up listening sockets and keep-alive options\n> from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct\n> dependencies on global state by inlining the behaviour at the callsites\n> for all libified functions.\n\nThanks for making this change, the reduced code duplication should make the\ncommon daemon-related code more maintainable.\n\nFor reference, I used \n\n'git blame -s -b -C -C -C master.. -- daemon-utils.c' \n\nto help identify which lines in 'daemon-utils.c' were changed from their\noriginal implementation in 'daemon.c'. I'll try to rearrange the diff to\nshow those differences more directly.\n\nThe first main change I see is that 'logerror' and 'reuseaddr' are changed\nfrom global references to arguments in 'set_keep_alive()',\n'setup_named_sock()' (same for 'NO_IPV6' defined and undefined), and\n'socksetup()':\n\n> -static void set_keep_alive(int sockfd)\n> +void set_keep_alive(int sockfd, log_fn logerror)\n\n> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n> +static int setup_named_sock(char *listen_addr, int listen_port,\n> +\t\t\t struct socketlist *socklist, int reuseaddr,\n> +\t\t\t log_fn logerror)\n\n> -static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n> +void socksetup(struct string_list *listen_addr, int listen_port,\n> +\t struct socketlist *socklist, int reuseaddr,\n> +\t log_fn logerror)\n\nThe external calls in 'daemon.c' to 'set_keep_alive()' and 'socksetup()' are\nupdated to pass the 'logerror()' function and global 'reuseaddr' as\narguments, so there isn't any change in behavior.\n\n> @@ -759,7 +748,7 @@ static int execute(void)\n> \tif (addr)\n> \t\tloginfo(\"Connection from %s:%s\", addr, port);\n> \n> -\tset_keep_alive(0);\n> +\tset_keep_alive(0, logerror);\n> \talarm(init_timeout ? init_timeout : timeout);\n> \tpktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);\n> \talarm(0);\n> @@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,\n> {\n> \tstruct socketlist socklist = { NULL, 0, 0 };\n> \n> -\tsocksetup(listen_addr, listen_port, &socklist);\n> +\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr,\n> +\t\t logerror);\n> \tif (socklist.nr == 0)\n> \t\tdie(\"unable to allocate any listen sockets on port %u\",\n> \t\t listen_port);\n\nThe other notable change is moving the 'if (!reusaddr) return 0' block in\n'set_reuse_addr()' to its callers in both 'setup_named_sock()'s:\n\n> +#ifndef NO_IPV6\n> +\n> +static int setup_named_sock(char *listen_addr, int listen_port,\n> +\t\t\t struct socketlist *socklist, int reuseaddr,\n> +\t\t\t log_fn logerror)\n> +{\n...\n> +\t\tif (reuseaddr && set_reuse_addr(sockfd)) {\n> +\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n> +\t\t\tclose(sockfd);\n> +\t\t\tcontinue;\n> +\t\t}\n...\n> +}\n> +\n> +#else /* NO_IPV6 */\n> +\n> +static int setup_named_sock(char *listen_addr, int listen_port,\n> +\t\t\t struct socketlist *socklist, int reuseaddr,\n> +\t\t\t log_fn logerror)\n> +{\n...\n> +\tif (reuseaddr && set_reuse_addr(sockfd)) {\n> +\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n> +\t\tclose(sockfd);\n> +\t\treturn 0;\n> +\t}\n...\n> +}\n> +\n> +#endif\n\nWhere, previously, that region looked like:\n\n> -#ifndef NO_IPV6\n> -\n> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n> -{\n...\n> -\t\tif (set_reuse_addr(sockfd)) {\n> -\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n> -\t\t\tclose(sockfd);\n> -\t\t\tcontinue;\n> -\t\t}\n...\n> -}\n> -\n> -#else /* NO_IPV6 */\n> -\n> -static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n> -{\n...\n> -\tif (set_reuse_addr(sockfd)) {\n> -\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n> -\t\tclose(sockfd);\n> -\t\treturn 0;\n> -\t}\n...\n> -}\n> -\n> -#endif\n\n'reuseaddr' is passed into 'setup_named_sock()' from 'socksetup()' calls in\n'daemon.c', so this also won't result in changed behavior.\n\nOtherwise, you only expose functions & types that are called in 'daemon.c'\n(the rest are still static), and everything else is a verbatim copy. Looks\ngood to me!\n\n"},{"id":"470226","messageId":"3a8d1b66-ed06-16a3-5459-9381faa69420@github.com","threadId":"58425","inReplyTo":"bc972fc8d3d3a028d3d160aac354d2a13bad37ae.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/10] daemon: libify child process handling functions","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T19:35:25Z","receivedAt":"2023-01-12T19:44:44Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Extract functions and structures for managing child processes started\n> from the parent daemon-like process from `daemon.c` to the new shared\n> `daemon-utils.{c,h}` files.\n\nAs with patch 1, it looks like the main changes here are changing global\nreferences to function arguments. Specifically, those variables are\n'firstborn', 'live_children', and 'loginfo':\n\n> -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n> +\t struct child *firstborn , unsigned int *live_children)\n\n> -static void kill_some_child(void)\n> +void kill_some_child(struct child *firstborn)\n\n> -static void check_dead_children(void)\n> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n> +\t\t\t log_fn loginfo)\n\nThose values are provided by the callers in 'daemon.c'. The major change\nhere is that 'live_children' is passed as a pointer, since its value is\nupdated by difference is passing 'live_children' as a pointer, since its\nvalue is updated by 'check_dead_children()' and 'add_child()':\n\n> @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n> \tstruct child_process cld = CHILD_PROCESS_INIT;\n> \n> \tif (max_connections && live_children >= max_connections) {\n> -\t\tkill_some_child();\n> +\t\tkill_some_child(firstborn);\n> \t\tsleep(1); /* give it some time to die */\n> -\t\tcheck_dead_children();\n> +\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n> \t\tif (live_children >= max_connections) {\n> \t\t\tclose(incoming);\n> \t\t\tlogerror(\"Too many children, dropping connection\");\n> @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n> \tif (start_command(&cld))\n> \t\tlogerror(\"unable to fork\");\n> \telse\n> -\t\tadd_child(&cld, addr, addrlen);\n> +\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n> }\n> \n> static void child_handler(int signo)\n> @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n> \tfor (;;) {\n> \t\tint i;\n> \n> -\t\tcheck_dead_children();\n> +\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n> \n> \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n> \t\t\tif (errno != EINTR) {\n\nHowever, I think that change to 'live_children' may have caused a bug. In\n'check_dead_children()', you decrement the 'live_children' *pointer*. That\nchanges its address, not its value:\n\n> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n> +\t\t\t log_fn loginfo)\n> +{\n...\n> +\t\t\tlive_children--;\n...\n> +}\n\nSame thing in 'add_child()':\n\n> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n> +\t struct child *firstborn , unsigned int *live_children)\n> +{\n...\n> +\tlive_children++;\n...\n> +}\n\nThese should be changed to '(*live_children)--' and '(*live_children)++',\nrespectively.\n\nThere's also one minor functional change in 'check_dead_children()', where\nan 'if (loginfo)' check is added guarding the call to 'loginfo()':\n\n> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n> +\t\t\t log_fn loginfo)\n> +{\n...\n> +\t\t\tif (loginfo) {\n> +\t\t\t\tconst char *dead = \"\";\n> +\t\t\t\tif (status)\n> +\t\t\t\t\tdead = \" (with error)\";\n> +\t\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\",\n> +\t\t\t\t\t(uintmax_t)pid, dead);\n> +\t\t\t}\n...\n> +}\n\nI'm guessing this is done because a caller later in the series won't provide\na 'loginfo', but if that's the case, it would help to note that in this\npatch's commit message.\n\nThe one other thing I noticed is that you removed the function documentation\nfor 'kill_some_child()':\n\n> -/*\n> - * This gets called if the number of connections grows\n> - * past \"max_connections\".\n> - *\n> - * We kill the newest connection from a duplicate IP.\n> - */\n\nIs there a reason why you removed it? Otherwise, it should be added back in\n- probably in 'daemon-utils.h'?\n\nEverything else here looks good.\n"},{"id":"470227","messageId":"da31180a-5beb-4f0e-667b-ddceba941e9f@github.com","threadId":"58425","inReplyTo":"8f176d5955dfc83616a39622972aaa71a71f5599.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T19:44:00Z","receivedAt":"2023-01-12T19:48:19Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Rename some of the variables and function arguments used to manage child\n> processes. The existing names are esoteric; stretching an analogy too\n> far to the point of being confusing to understand.\n> \n> Rename \"firstborn\" to simply \"first\", \"newborn\" to \"new_cld\", \"blanket\"\n> to \"current\" and \"cradle\" to \"ptr\".\n\nThanks for this, I agree that the new names make the code much easier to\nread.\n\n> diff --git a/daemon.c b/daemon.c\n> index ec3b407ecbc..d3e7d81de18 100644\n> --- a/daemon.c\n> +++ b/daemon.c\n> @@ -789,7 +789,7 @@ static int max_connections = 32;\n> \n> static unsigned int live_children;\n> \n> -static struct child *firstborn;\n> +static struct child *first_child;\n\nminor nit: you changed \"firstborn\" to \"first\" in 'daemon-utils.c' (aligning\nwith the commit message), but it's \"first_child\" here. If you end up\nre-rolling, it would be nice to make the names consistent across both files\n(could be 'first', 'first_child', 'first_cld', or anything really).\n\n"},{"id":"470229","messageId":"b5c0ba73-c1a7-293e-4594-b8ee291152de@github.com","threadId":"58425","inReplyTo":"706fb3781bd383380a7b1fd30495eb2da970b5ec.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 04/10] test-http-server: add stub HTTP server test helper","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T19:57:57Z","receivedAt":"2023-01-12T20:05:43Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Introduce a mini HTTP server helper that in the future will be enhanced\n> to provide a frontend for the git-http-backend, with support for\n> arbitrary authentication schemes.\n> \n> Right now, test-http-server is a pared-down copy of the git-daemon that\n> always returns a 501 Not Implemented response to all callers.\n\nBetween your earlier response [1] and this iteration of the patch, all of\nthe comments from my previous review [2] have been addressed. The changes to\ndrop the dependency on cURL also look correct to me. Thanks!\n\n[1] https://lore.kernel.org/git/AS2PR03MB98150C33F9704D2CA10A2EF9C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/\n[2] https://lore.kernel.org/git/752da6b2-9c75-0f68-e507-cca02bf918ca@github.com/\n\n> \n> Signed-off-by: Matthew John Cheetham \n> ---\n> Makefile | 1 +\n> contrib/buildsystems/CMakeLists.txt | 11 +-\n> t/helper/.gitignore | 1 +\n> t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++\n> 4 files changed, 396 insertions(+), 2 deletions(-)\n> create mode 100644 t/helper/test-http-server.c\n\n"},{"id":"470230","messageId":"2f40f8f7-7323-94ac-b423-9d35583f0dc3@github.com","threadId":"58425","inReplyTo":"e1b1a53b-69d7-6fd6-513d-0f6cb3751b51@github.com","subject":"Re: [PATCH v5 01/10] daemon: libify socket setup and option functions","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-12T20:22:59Z","receivedAt":"2023-01-12T20:52:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/12/2023 2:35 PM, Victoria Dye wrote:\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Extract functions for setting up listening sockets and keep-alive options\n>> from `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct\n>> dependencies on global state by inlining the behaviour at the callsites\n>> for all libified functions.\n> \n> Thanks for making this change, the reduced code duplication should make the\n> common daemon-related code more maintainable.\n> \n> For reference, I used \n> \n> 'git blame -s -b -C -C -C master.. -- daemon-utils.c' \n> \n> to help identify which lines in 'daemon-utils.c' were changed from their\n> original implementation in 'daemon.c'.\n\nNeat trick! Thanks for sharing. Using --color-moved was giving similar\nresults, but with a lot more tracking back-and-forth to see what the\ndifferences were.\n\nI agree with your assessment on this patch that the differences are\nvalid, safe, and desired.\n\nThanks,\n-Stolee\n"},{"id":"470231","messageId":"0a5f4195-600f-d099-5879-bbd7629285b2@github.com","threadId":"58425","inReplyTo":"6f66bf146b4d6e4044b3c6c2224795918191bc3f.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 05/10] test-http-server: add HTTP error response function","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T20:35:19Z","receivedAt":"2023-01-12T20:55:40Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> From: Matthew John Cheetham \n> \n> Introduce a function to the test-http-server test helper to write more\n> full and valid HTTP error responses, including all the standard response\n> headers like `Server` and `Date`.\n\nIt took me a second to figure out, but this patch combines the content of\npatches 4, 5, and 6 from the last iteration. After squashing those three\npatches from v4 together locally, the range-diff is actually pretty simple\n(see below). \n\nOut of curiosity, why did you combine those patches? I don't feel strongly\nabout changing it, but the smaller, incremental patches in the previous\nversion were a bit easier to review.\n\nIn any case, this version addresses my feedback from [1], [2], and [3] - the\nexplanatory comments are particularly helpful. Thanks!\n\n[1] https://lore.kernel.org/git/7b7d1059-cecf-744d-6927-b41963b9e5a8@github.com/\n[2] https://lore.kernel.org/git/e957d4f4-fa94-7a68-f378-38e6ed131244@github.com/\n[3] https://lore.kernel.org/git/f99c381c-1d30-7c95-6158-cecd5321dafd@github.com/\n\nRange diff v4 (patches 4-6, squashed) vs. v5 (this patch)\n\n4: 127827637e ! 5: 6f66bf146b test-http-server: add HTTP error response function\n @@ Commit message\n \n ## t/helper/test-http-server.c ##\n @@ t/helper/test-http-server.c: enum worker_result {\n - \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n + \tWR_HANGUP = 1<<1,\n };\n \n +/*\n @@ t/helper/test-http-server.c: enum worker_result {\n +\t\thp = strbuf_detach(&h, NULL);\n +\t\tstring_list_append(&req->header_list, hp);\n +\n -+\t\t/* store common request headers separately */\n ++\t\t/* also store common request headers as struct req members */\n +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n +\t\t\treq->content_type = hv;\n +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n @@ t/helper/test-http-server.c: enum worker_result {\n +\n +\tif (!initialized) {\n +\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n ++\t\t/*\n ++\t\t * This regular expression matches all dumb and smart HTTP\n ++\t\t * requests that are currently in use, and defined in\n ++\t\t * Documentation/gitprotocol-http.txt.\n ++\t\t *\n ++\t\t */\n +\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n +\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n +\t\t\t REG_EXTENDED)) {\n @@ t/helper/test-http-server.c: enum worker_result {\n +\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n +}\n +\n -+static enum worker_result do__git(struct req *req, const char *user)\n ++static enum worker_result do__git(struct req *req)\n +{\n +\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n +\tstruct child_process cp = CHILD_PROCESS_INIT;\n +\tint res;\n +\n -+\tif (write(1, ok, strlen(ok)) < 0)\n ++\t/*\n ++\t * Note that we always respond with a 200 OK response even if the\n ++\t * http-backend process exits with an error. This helper is intended\n ++\t * only to be used to exercise the HTTP auth handling in the Git client,\n ++\t * and specifically around authentication (not handled by http-backend).\n ++\t *\n ++\t * If we wanted to respond with a more 'valid' HTTP response status then\n ++\t * we'd need to buffer the output of http-backend, wait for and grok the\n ++\t * exit status of the process, then write the HTTP status line followed\n ++\t * by the http-backend output. This is outside of the scope of this test\n ++\t * helper's use at time of writing.\n ++\t *\n ++\t * The important auth responses (401) we are handling prior to getting\n ++\t * to this point.\n ++\t */\n ++\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n +\t\treturn error(_(\"could not send '%s'\"), ok);\n +\n -+\tif (user)\n -+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n -+\n +\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n +\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n +\t\t\treq->uri_path.buf);\n @@ t/helper/test-http-server.c: enum worker_result {\n +\tcp.git_cmd = 1;\n +\tstrvec_push(&cp.args, \"http-backend\");\n +\tres = run_command(&cp);\n -+\tclose(1);\n -+\tclose(0);\n ++\tclose(STDOUT_FILENO);\n ++\tclose(STDIN_FILENO);\n +\treturn !!res;\n +}\n +\n +static enum worker_result dispatch(struct req *req)\n +{\n +\tif (is_git_request(req))\n -+\t\treturn do__git(req, NULL);\n ++\t\treturn do__git(req);\n +\n -+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n ++\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n +\t\t\t WR_OK | WR_HANGUP);\n +}\n +\n @@ t/helper/test-http-server.c: enum worker_result {\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n - \tset_keep_alive(0);\n + \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n --\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n +-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n -\t\t\tlogerror(\"unable to write response\");\n -\t\t\twr = WR_IO_ERROR;\n -\t\t}\n +\t\treq__release(&req);\n +\n -+\t\talarm(init_timeout ? init_timeout : timeout);\n ++\t\talarm(timeout);\n +\t\twr = req__read(&req, 0);\n +\t\talarm(0);\n +\n -+\t\tif (wr & WR_STOP_THE_MUSIC)\n ++\t\tif (wr != WR_OK)\n +\t\t\tbreak;\n \n +\t\twr = dispatch(&req);\n - \t\tif (wr & WR_STOP_THE_MUSIC)\n + \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\n \n @@ t/t5556-http-auth.sh (new)\n +\n +test_description='test http auth header and credential helper interop'\n +\n ++TEST_NO_CREATE_REPO=1\n +. ./test-lib.sh\n +\n +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n +\n +# Setup a repository\n +#\n -+REPO_DIR=\"$(pwd)\"/repo\n ++REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n +\n +# Setup some lookback URLs where test-http-server will be listening.\n +# We will spawn it directly inside the repo directory, so we avoid\n @@ t/t5556-http-auth.sh (new)\n +# The server will shutdown if/when we delete it (this is easier than\n +# killing it by PID).\n +#\n -+PID_FILE=\"$(pwd)\"/pid-file.pid\n -+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n ++PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n ++SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n +\n +PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n +\n @@ t/t5556-http-auth.sh (new)\n +\n +test_expect_success 'http auth anonymous no challenge' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server --allow-anonymous &&\n ++\tstart_http_server &&\n +\n +\t# Attempt to read from a protected repository\n +\tgit ls-remote $ORIGIN_URL\n\n"},{"id":"470232","messageId":"98940e93-c4c5-01ec-54b2-b6015f488ad0@github.com","threadId":"58425","inReplyTo":"af66d2d2ede2a502f32d74c86f302598c68d1476.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-01-12T20:41:03Z","receivedAt":"2023-01-12T20:58:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2023 5:13 PM, Matthew John Cheetham via GitGitGadget wrote:\n\n> +static void credential_write_strvec(FILE *fp, const char *key,\n> +\t\t\t\t const struct strvec *vec)\n> +{\n> +\tint i = 0;\n> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n> +\tfor (; i < vec->nr; i++) {\n\nstyle nit: use \"int i;\" and \"for (i = 0; ...\"\n\n> test_expect_success 'http auth anonymous no challenge' '\n> \ttest_when_finished \"per_test_cleanup\" &&\n> -\tstart_http_server &&\n> +\n> +\tcat >auth.config <<-EOF &&\n> +\t[auth]\n> +\t allowAnonymous = true\n> +\tEOF\n> +\n> +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n\nI see that you added auth.allowAnonymous and --auth-config options\nin Patch 6, so perhaps this test change could move to that patch.\n\nThanks,\n-Stolee\n"},{"id":"470233","messageId":"3858d972-6659-76c5-8d13-b9c803bc32c8@github.com","threadId":"58425","inReplyTo":"AS2PR03MB9815D6B888AD0E0E12C1C679C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com","subject":"Re: [PATCH v4 6/8] test-http-server: pass Git requests to http-backend","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-12T20:54:30Z","receivedAt":"2023-01-12T21:10:39Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham wrote:\n> \n> On 2022-12-14 15:20, Victoria Dye wrote:\n>> nit: '$TEST_OUTPUT_DIRECTORY' instead of '$(pwd)' is more consistent with\n>> what I see in other tests. \n> \n> I don't see this? In fact I see more usages of `$(pwd)` than your suggestion.\n\nTo be honest, I'm not sure how I missed this. '$(pwd)' *is* quite common in\nthe tests, although it does seem to be used mostly in individual tests\nrather than file-level variables (although that's not universally true, e.g.\nusing it to set 'CURR_DIR' in 't9400-diff-highlight.sh'). \n\nSo, contrary to my earlier comment, this seems best left up to (your)\npersonal preference than any concrete rule.\n\n> \n>> Also, if you're creating a repo in its own subdirectory ('repo'), you can\n>> set 'TEST_NO_CREATE_REPO=1' before importing './test-lib' to avoid creating\n>> a repo at the root level of the test output dir - it can help avoid\n>> potential weird/unexpected behavior as a result of being in a repo inside of\n>> another repo.\n> \n> However.. after setting `TEST_NO_CREATE_REPO=1` I was getting CI failures\n> around a missing PWD, so my next iteration uses the `$TRASH_DIRECTORY` variable\n> explicitly in paths instead :-)\n\nYou're right, I was completely misreading the purpose of\n'TEST_OUTPUT_DIRECTORY' in 'test-lib.sh':\n\n> if test -z \"$TEST_OUTPUT_DIRECTORY\"\n> then\n> \t# Similarly, override this to store the test-results subdir\n> \t# elsewhere\n> \tTEST_OUTPUT_DIRECTORY=$TEST_DIRECTORY\n> fi\n\n\"the test-results subdir\" != \"the test working directory\". As you pointed\nout, '$TRASH_DIRECTORY' would be the variable to use here.\n\n"},{"id":"470296","messageId":"2a0b5f3f-7ab2-bc9e-76ac-93a52b4d32d0@github.com","threadId":"58425","inReplyTo":"c3c3d17a688963acc180e3bb7bbb4deb32a94304.1673475190.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 06/10] test-http-server: add simple authentication","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-13T18:10:03Z","receivedAt":"2023-01-13T18:18:47Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> +static struct auth_module *create_auth_module(const char *scheme,\n> +\t\t\t\t\t const char *challenge)\n> +{\n> +\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n> +\tmod->scheme = xstrdup(scheme);\n> +\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n> +\tCALLOC_ARRAY(mod->tokens, 1);\n> +\tstring_list_init_dup(mod->tokens);\n> +\treturn mod;\n> +}\n\n> +\n> +static int add_auth_module(struct auth_module *mod)\n> +{\n> +\tif (get_auth_module(mod->scheme))\n> +\t\treturn error(\"duplicate auth scheme '%s'\\n\", mod->scheme);\n> +\n> +\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n> +\tauth_modules[auth_modules_nr++] = mod;\n> +\n> +\treturn 0;\n> +}\n\n> +static int split_auth_param(const char *str, char **scheme, char **val, int required_val)\n> +{\n> +\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n> +\n> +\tif (!p[0])\n> +\t\treturn -1;\n> +\n> +\t/* trim trailing ':' */\n> +\tif (p[1])\n> +\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n> +\n> +\tif (required_val && !p[1])\n> +\t\treturn -1;\n> +\n> +\t*scheme = strbuf_detach(p[0], NULL);\n> +\n> +\tif (p[1])\n> +\t\t*val = strbuf_detach(p[1], NULL);\n> +\n> +\tstrbuf_list_free(p);\n> +\treturn 0;\n> +}\n\nThere's nothing really *new* in these functions in this iteration, just code\nmoved from the option parsing/handling in 'cmd_main()' into dedicated\nfunctions. Looks good!\n\n> +\tswitch (result) {\n> +\tcase AUTH_ALLOW:\n> +\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n> +\t\t*user = \"VALID_TEST_USER\";\n> +\t\t*wr = WR_OK;\n> +\t\tbreak;\n> +\n> +\tcase AUTH_DENY:\n> +\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n> +\t\t/* fall-through */\n> +\n> +\tcase AUTH_UNKNOWN:\n> +\t\tif (result != AUTH_DENY && allow_anonymous)\n> +\t\t\tbreak;\n\nI completely missed the \"fall-through\" comment in my last review [1], as you\nkindly pointed out [2]. ;) Given that, this makes sense to me.\n\n[1] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/\n[2] https://lore.kernel.org/git/AS2PR03MB981593EB3382F9738D2CA3D7C0FC9@AS2PR03MB9815.eurprd03.prod.outlook.com/\n\n> +\n> +\t\tfor (i = 0; i < auth_modules_nr; i++) {\n> +\t\t\tmod = auth_modules[i];\n> +\t\t\tif (mod->challenge_params)\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n> +\t\t\t\t\t\t mod->scheme,\n> +\t\t\t\t\t\t mod->challenge_params);\n> +\t\t\telse\n> +\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n> +\t\t\t\t\t\t mod->scheme);\n> +\t\t\tstring_list_append(&hdrs, challenge);\n> +\t\t}\n> +\n> +\t\tfor (i = 0; i < extra_headers.nr; i++)\n> +\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n> +\n> +\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n> +\t\t\t\t &hdrs, *wr);\n\nThe \"extra_headers\" configuration is new, and helps make the test server\nmore flexible. \n\n> +static int read_auth_config(const char *name, const char *val, void *data)\n> +{\n> +\tint ret = 0;\n> +\tchar *scheme = NULL;\n> +\tchar *token = NULL;\n> +\tchar *challenge = NULL;\n> +\tstruct auth_module *mod = NULL;\n> +\n> +\tif (!strcmp(name, \"auth.challenge\")) {\n> +\t\tif (split_auth_param(val, &scheme, &challenge, 0)) {\n> +\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\n> +\t\tmod = create_auth_module(scheme, challenge);\n> +\t\tif (add_auth_module(mod)) {\n> +\t\t\tret = error(\"failed to add auth module '%s'\", val);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\t}\n> +\tif (!strcmp(name, \"auth.token\")) {\n> +\t\tif (split_auth_param(val, &scheme, &token, 1)) {\n> +\t\t\tret = error(\"invalid auth token '%s'\", val);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\n> +\t\tmod = get_auth_module(scheme);\n> +\t\tif (!mod) {\n> +\t\t\tret = error(\"auth scheme not defined '%s'\\n\", scheme);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\n> +\t\tstring_list_append(mod->tokens, token);\n> +\t}\n\nI don't think this addresses the implicit option ordering requirement noted\nin [3]; instead of needing '--auth' before '--auth-token', this now needs\n'auth.challenge' before 'auth.token' in the config file. While I'd prefer it\nif this could be rearranged so that the auth setup happens after all config\nparsing (so the order doesn't matter), if you want to leave it as-is please\nadd a comment somewhere in this file explaining that requirement and/or add\na note to the \"auth scheme not defined\" error message. \n\n[3] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/\n\n> +\tif (!strcmp(name, \"auth.allowanonymous\")) {\n> +\t\tallow_anonymous = git_config_bool(name, val);\n> +\t}\n> +\tif (!strcmp(name, \"auth.extraheader\")) {\n> +\t\tstrvec_push(&extra_headers, val);\n> +\t}\n\nIs it worth printing a warning if the option found isn't any of the above?\nSomething like \"ignoring \". This is a test helper, so\nuser-friendliness isn't quite as important as it is for builtins, but the\nwarning might be helpful to developers trying to use it in the future.\n\n> +\n> +cleanup:\n> +\tfree(scheme);\n> +\tfree(token);\n> +\tfree(challenge);\n> +\n> +\treturn ret;\n> +}\n> +\n> static enum worker_result dispatch(struct req *req)\n> {\n> +\tenum worker_result wr = WR_OK;\n> +\tconst char *user = NULL;\n> +\n> +\tif (!is_authed(req, &user, &wr))\n> +\t\treturn wr;\n> +\n> \tif (is_git_request(req))\n> -\t\treturn do__git(req);\n> +\t\treturn do__git(req, user);\n> \n> \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n> \t\t\t WR_OK | WR_HANGUP);\n> @@ -624,6 +853,19 @@ int cmd_main(int argc, const char **argv)\n> \t\t\tpid_file = v;\n> \t\t\tcontinue;\n> \t\t}\n> +\t\tif (skip_prefix(arg, \"--auth-config=\", &v)) {\n> +\t\t\tif (!strlen(v)) {\n> +\t\t\t\terror(\"invalid argument - missing file path\");\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tif (git_config_from_file(read_auth_config, v, NULL)) {\n> +\t\t\t\terror(\"failed to read auth config file '%s'\", v);\n> +\t\t\t\tusage(test_http_auth_usage);\n> +\t\t\t}\n> +\n> +\t\t\tcontinue;\n> +\t\t}\n> \n> \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n> \t\tusage(test_http_auth_usage);\n\n"},{"id":"470310","messageId":"xmqq8ri616ww.fsf@gitster.g","threadId":"58425","inReplyTo":"2a0b5f3f-7ab2-bc9e-76ac-93a52b4d32d0@github.com","subject":"Re: [PATCH v5 06/10] test-http-server: add simple authentication","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-13T21:06:07Z","receivedAt":"2023-01-13T21:06:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n>> +static int split_auth_param(const char *str, char **scheme, char **val, int required_val)\n>> +{\n>> ...\n>> +}\n>\n> There's nothing really *new* in these functions in this iteration, just code\n> moved from the option parsing/handling in 'cmd_main()' into dedicated\n> functions. Looks good!\n\n> ...\n>\n> I completely missed the \"fall-through\" comment in my last review [1], as you\n> kindly pointed out [2]. ;) Given that, this makes sense to me.\n\n>> +\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n>> +\t\t\t\t &hdrs, *wr);\n>\n> The \"extra_headers\" configuration is new, and helps make the test server\n> more flexible. \n\nThis is not limited to this single review message, but it is good to\nsee \"this part of the patch is good because ...\" explicitly stated.\nI wish more people did so, in addition to pointing out what needs to\nbe improved.\n\nThanks.\n"},{"id":"470551","messageId":"AS2PR03MB9815C5C045F3138287070F1AC0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230112.86wn5s2l5r.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 09/10] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:51:59Z","receivedAt":"2023-01-17T22:41:28Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 00:41, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>> [...]\n>> +\t\t} else if (buf.len) {\n>> +\t\t\tconst char *prev = values->v[values->nr - 1];\n>> +\t\t\tstruct strbuf append = STRBUF_INIT;\n>> +\t\t\tstrbuf_addstr(&append, prev);\n>> +\n>> +\t\t\t/* Join two non-empty values with a single space. */\n>> +\t\t\tif (append.len)\n>> +\t\t\t\tstrbuf_addch(&append, ' ');\n>> +\n>> +\t\t\tstrbuf_addbuf(&append, &buf);\n>> +\n>> +\t\t\tstrvec_pop(values);\n>> +\t\t\tstrvec_push_nodup(values, strbuf_detach(&append, NULL));\n>> +\t\t}\n>> +\n> \n> I've written something like the strvec_push_nodup() patch that preceded\n> this myself for similar reasons, and as recently noted in [1] I think\n> such a thing (although I implemented a different interface) might be\n> useful in general.\n\nA fair point, and reading [1] I see there's some concerns about making the\nstrvec interface more complicated w.r.t. ownership vs saving a `xstrdup`.\nIn light of this, I'll drop the commit to add `strvec_push_nodup`.\n\n> But this really doesn't seem like a good justification for adding this\n> new API. Let's instead do:\n> \n> \t} else if (buf.len) {\n> \t\tconst char *prev = values->v[values->nr - 1];\n> \t\t/* Join two non-empty values with a single space. */\n> \t\tconst char *const sp = *prev ? \" \" : \"\"\n> \n> \t\tstrvec_pop(values);\n> \t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n> \t}\n> \n> There may be cases where a public strvec_push_nodup() simplifies things,\n> but this doesn't seem like such a case, just use strvec_pushf() directly\n> instead, and skip the strbuf & strbuf_detach().\n> \n> I haven't compiled/tested the above, so there may e.g. be a typo in\n> there. But I think the general concept should work in this case.\n> \n> 1. https://lore.kernel.org/git/RFC-cover-0.5-00000000000-20221215T090226Z-avarab@gmail.com/\n\n\nThere's a bug in your suggestion. We're `strvec_pop`-ing from the array\nwhich also frees the previous value that we want to use to append to in\nthe next call to `strvec_pushf`. We need to keep a copy of the previous\nheader value around.\n\nThis should work instead (adding an `xstrdup` and `free`):\n\n\t} else if (buf.len) {\n\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n\n\t\t/* Join two non-empty values with a single space. */\n\t\tconst char *const sp = *prev ? \" \" : \"\";\n\n\t\tstrvec_pop(values);\n\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n\t\tfree(prev);\n\t}\n\nThanks,\nMatthew\n"},{"id":"470556","messageId":"AS2PR03MB981518197F0DC6413F07ED5CC0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230112.86sfgg2kuj.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:35:39Z","receivedAt":"2023-01-17T23:05:54Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 00:48, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 11 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>> [...]\n>> +static void credential_write_strvec(FILE *fp, const char *key,\n>> +\t\t\t\t const struct strvec *vec)\n>> +{\n>> +\tint i = 0;\n>> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n>> +\tfor (; i < vec->nr; i++) {\n>> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n> \n> Style: Don't mismatch types if there's no good reason. Use \"size_t i\" here, also let's do:\n> \n> \tfor (size_t i = 0; ....\n> \n> I.e. no reason to declare it earlier.\n> \n>> +\t}\n>> +\tfree((void*)full_key);\n> \n> Just don't add a \"const\" to that \"full_key\" and skip the cast with\n> free() here.\n\nBoth good points! Thanks - will take this onboard in next iteration.\n\n>> +++ b/t/helper/test-credential-helper-replay.sh\n> \n> I see to my surprise that we have one existing *.sh helper in that\n> directory, but in any case...\n> \n>> @@ -0,0 +1,14 @@\n>> +cmd=$1\n>> +teefile=$cmd-actual.cred\n>> +catfile=$cmd-response.cred\n>> +rm -f $teefile\n>> +while read line;\n>> +do\n>> +\tif test -z \"$line\"; then\n>> +\t\tbreak;\n>> +\tfi\n>> +\techo \"$line\" >> $teefile\n>> +done\n> \n> It looks like you're re-inventing \"sed\" here, isn't this whole loop just\n> \n> \tsed -n -e '/^$/q' -n 'p'\n\nTrue; `sed -n -e '/^$/q' -e 'p'` is equivalent here.\n\n> And then you can skip the \"rm\" before, as you could just clobber the\n> thing.\n> \n>> +if test \"$cmd\" = \"get\"; then\n>> +\tcat $catfile\n>> +fi\n>> diff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\n>> index 65105a5a6a9..a8dbee6ca40 100755\n>> --- a/t/t5556-http-auth.sh\n>> +++ b/t/t5556-http-auth.sh\n>> @@ -27,6 +27,8 @@ PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n>> SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n>> \n>> PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n>> +CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n>> +\t&& export CREDENTIAL_HELPER\n> \n> ...(continued from above): Let's just use write_script() here or\n> whatever, i.e. no reason to make this a global script, it's just used in\n> this one test, so it can set it up.\n\nIn the next iteration I will move to using write_script; thanks!\n\n>> test_expect_success 'setup repos' '\n>> \ttest_create_repo \"$REPO_DIR\" &&\n>> @@ -92,15 +94,279 @@ start_http_server () {\n>> \n>> per_test_cleanup () {\n>> \tstop_http_server &&\n>> -\trm -f OUT.*\n>> +\trm -f OUT.* &&\n>> +\trm -f *.cred &&\n>> +\trm -f auth.config\n>> }\n>> \n>> test_expect_success 'http auth anonymous no challenge' '\n>> \ttest_when_finished \"per_test_cleanup\" &&\n>> -\tstart_http_server &&\n>> +\n>> +\tcat >auth.config <<-EOF &&\n>> +\t[auth]\n>> +\t allowAnonymous = true\n> \n> Mixed tab/space. Use \"\\t\" not 4x \" \" (ditto below).\n\nSure!\n"},{"id":"470558","messageId":"AS2PR03MB9815335902DC90737D178E7CC0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"3a8d1b66-ed06-16a3-5459-9381faa69420@github.com","subject":"Re: [PATCH v5 02/10] daemon: libify child process handling functions","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:14:32Z","receivedAt":"2023-01-17T23:23:35Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 11:35, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Extract functions and structures for managing child processes started\n>> from the parent daemon-like process from `daemon.c` to the new shared\n>> `daemon-utils.{c,h}` files.\n> \n> As with patch 1, it looks like the main changes here are changing global\n> references to function arguments. Specifically, those variables are\n> 'firstborn', 'live_children', and 'loginfo':\n> \n>> -static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n>> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n>> +\t struct child *firstborn , unsigned int *live_children)\n> \n>> -static void kill_some_child(void)\n>> +void kill_some_child(struct child *firstborn)\n> \n>> -static void check_dead_children(void)\n>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n>> +\t\t\t log_fn loginfo)\n> \n> Those values are provided by the callers in 'daemon.c'. The major change\n> here is that 'live_children' is passed as a pointer, since its value is\n> updated by difference is passing 'live_children' as a pointer, since its\n> value is updated by 'check_dead_children()' and 'add_child()':\n> \n>> @@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>> \tstruct child_process cld = CHILD_PROCESS_INIT;\n>> \n>> \tif (max_connections && live_children >= max_connections) {\n>> -\t\tkill_some_child();\n>> +\t\tkill_some_child(firstborn);\n>> \t\tsleep(1); /* give it some time to die */\n>> -\t\tcheck_dead_children();\n>> +\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n>> \t\tif (live_children >= max_connections) {\n>> \t\t\tclose(incoming);\n>> \t\t\tlogerror(\"Too many children, dropping connection\");\n>> @@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n>> \tif (start_command(&cld))\n>> \t\tlogerror(\"unable to fork\");\n>> \telse\n>> -\t\tadd_child(&cld, addr, addrlen);\n>> +\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n>> }\n>> \n>> static void child_handler(int signo)\n>> @@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n>> \tfor (;;) {\n>> \t\tint i;\n>> \n>> -\t\tcheck_dead_children();\n>> +\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n>> \n>> \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n>> \t\t\tif (errno != EINTR) {\n> \n> However, I think that change to 'live_children' may have caused a bug. In\n> 'check_dead_children()', you decrement the 'live_children' *pointer*. That\n> changes its address, not its value:\n> \n>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n>> +\t\t\t log_fn loginfo)\n>> +{\n> ...\n>> +\t\t\tlive_children--;\n> ...\n>> +}\n> \n> Same thing in 'add_child()':\n> \n>> +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n>> +\t struct child *firstborn , unsigned int *live_children)\n>> +{\n> ...\n>> +\tlive_children++;\n> ...\n>> +}\n> \n> These should be changed to '(*live_children)--' and '(*live_children)++',\n> respectively.\n\nAh! You are correct; my bad. Will correct this in v6.\n\n> There's also one minor functional change in 'check_dead_children()', where\n> an 'if (loginfo)' check is added guarding the call to 'loginfo()':\n> \n>> +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n>> +\t\t\t log_fn loginfo)\n>> +{\n> ...\n>> +\t\t\tif (loginfo) {\n>> +\t\t\t\tconst char *dead = \"\";\n>> +\t\t\t\tif (status)\n>> +\t\t\t\t\tdead = \" (with error)\";\n>> +\t\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\",\n>> +\t\t\t\t\t(uintmax_t)pid, dead);\n>> +\t\t\t}\n> ...\n>> +}\n> \n> I'm guessing this is done because a caller later in the series won't provide\n> a 'loginfo', but if that's the case, it would help to note that in this\n> patch's commit message.\n\nWill call this out in the commit message in v6.\n\n> The one other thing I noticed is that you removed the function documentation\n> for 'kill_some_child()':\n> \n>> -/*\n>> - * This gets called if the number of connections grows\n>> - * past \"max_connections\".\n>> - *\n>> - * We kill the newest connection from a duplicate IP.\n>> - */\n> \n> Is there a reason why you removed it? Otherwise, it should be added back in\n> - probably in 'daemon-utils.h'?\n\nI removed it initially as it was referencing things like `max_connections`\nwhich no longer existed in the context of `daemon-utils.{c,h}`.\n\nNext iteration I can restore the spirit of the comment, that this should be\ncalled when the maximimum number of connections has been reached, in order\nto kill the newest connection from a duplicate IP.\n\n> Everything else here looks good.\n\nThanks,\nMatthew\n"},{"id":"470559","messageId":"AS2PR03MB98150A57C1F9729CCEFE4EB3C0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"da31180a-5beb-4f0e-667b-ddceba941e9f@github.com","subject":"Re: [PATCH v5 03/10] daemon: rename some esoteric/laboured terminology","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:16:53Z","receivedAt":"2023-01-17T23:25:27Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 11:44, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Rename some of the variables and function arguments used to manage child\n>> processes. The existing names are esoteric; stretching an analogy too\n>> far to the point of being confusing to understand.\n>>\n>> Rename \"firstborn\" to simply \"first\", \"newborn\" to \"new_cld\", \"blanket\"\n>> to \"current\" and \"cradle\" to \"ptr\".\n> \n> Thanks for this, I agree that the new names make the code much easier to\n> read.\n> \n>> diff --git a/daemon.c b/daemon.c\n>> index ec3b407ecbc..d3e7d81de18 100644\n>> --- a/daemon.c\n>> +++ b/daemon.c\n>> @@ -789,7 +789,7 @@ static int max_connections = 32;\n>> \n>> static unsigned int live_children;\n>> \n>> -static struct child *firstborn;\n>> +static struct child *first_child;\n> \n> minor nit: you changed \"firstborn\" to \"first\" in 'daemon-utils.c' (aligning\n> with the commit message), but it's \"first_child\" here. If you end up\n> re-rolling, it would be nice to make the names consistent across both files\n> (could be 'first', 'first_child', 'first_cld', or anything really).\n> \n\nFair point. There's no technical reason to keep them named differently between\nthe 'libified' functions, and the actual variables for the callers.\nI shall align these to `first_child` in the next iteration.\n\nThanks,\nMatthew\n"},{"id":"470560","messageId":"AS2PR03MB9815EC09D465E123F15BA4B3C0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"0a5f4195-600f-d099-5879-bbd7629285b2@github.com","subject":"Re: [PATCH v5 05/10] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:23:29Z","receivedAt":"2023-01-17T23:27:11Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 12:35, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> From: Matthew John Cheetham \n>>\n>> Introduce a function to the test-http-server test helper to write more\n>> full and valid HTTP error responses, including all the standard response\n>> headers like `Server` and `Date`.\n> \n> It took me a second to figure out, but this patch combines the content of\n> patches 4, 5, and 6 from the last iteration. After squashing those three\n> patches from v4 together locally, the range-diff is actually pretty simple\n> (see below). \n> \n> Out of curiosity, why did you combine those patches? I don't feel strongly\n> about changing it, but the smaller, incremental patches in the previous\n> version were a bit easier to review.\n\nThis is my mistake. I didn't intend to do this, and agree splitting them is\neasier to grok. I will restore this! My apologies! :-(\n\n> In any case, this version addresses my feedback from [1], [2], and [3] - the\n> explanatory comments are particularly helpful. Thanks!\n> \n> [1] https://lore.kernel.org/git/7b7d1059-cecf-744d-6927-b41963b9e5a8@github.com/\n> [2] https://lore.kernel.org/git/e957d4f4-fa94-7a68-f378-38e6ed131244@github.com/\n> [3] https://lore.kernel.org/git/f99c381c-1d30-7c95-6158-cecd5321dafd@github.com/\n> \n> Range diff v4 (patches 4-6, squashed) vs. v5 (this patch)\n> \n> 4: 127827637e ! 5: 6f66bf146b test-http-server: add HTTP error response function\n> @@ Commit message\n> \n> ## t/helper/test-http-server.c ##\n> @@ t/helper/test-http-server.c: enum worker_result {\n> - \tWR_STOP_THE_MUSIC = (WR_IO_ERROR | WR_HANGUP),\n> + \tWR_HANGUP = 1<<1,\n> };\n> \n> +/*\n> @@ t/helper/test-http-server.c: enum worker_result {\n> +\t\thp = strbuf_detach(&h, NULL);\n> +\t\tstring_list_append(&req->header_list, hp);\n> +\n> -+\t\t/* store common request headers separately */\n> ++\t\t/* also store common request headers as struct req members */\n> +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n> +\t\t\treq->content_type = hv;\n> +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n> @@ t/helper/test-http-server.c: enum worker_result {\n> +\n> +\tif (!initialized) {\n> +\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n> ++\t\t/*\n> ++\t\t * This regular expression matches all dumb and smart HTTP\n> ++\t\t * requests that are currently in use, and defined in\n> ++\t\t * Documentation/gitprotocol-http.txt.\n> ++\t\t *\n> ++\t\t */\n> +\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n> +\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n> +\t\t\t REG_EXTENDED)) {\n> @@ t/helper/test-http-server.c: enum worker_result {\n> +\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n> +}\n> +\n> -+static enum worker_result do__git(struct req *req, const char *user)\n> ++static enum worker_result do__git(struct req *req)\n> +{\n> +\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n> +\tint res;\n> +\n> -+\tif (write(1, ok, strlen(ok)) < 0)\n> ++\t/*\n> ++\t * Note that we always respond with a 200 OK response even if the\n> ++\t * http-backend process exits with an error. This helper is intended\n> ++\t * only to be used to exercise the HTTP auth handling in the Git client,\n> ++\t * and specifically around authentication (not handled by http-backend).\n> ++\t *\n> ++\t * If we wanted to respond with a more 'valid' HTTP response status then\n> ++\t * we'd need to buffer the output of http-backend, wait for and grok the\n> ++\t * exit status of the process, then write the HTTP status line followed\n> ++\t * by the http-backend output. This is outside of the scope of this test\n> ++\t * helper's use at time of writing.\n> ++\t *\n> ++\t * The important auth responses (401) we are handling prior to getting\n> ++\t * to this point.\n> ++\t */\n> ++\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n> +\t\treturn error(_(\"could not send '%s'\"), ok);\n> +\n> -+\tif (user)\n> -+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n> -+\n> +\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n> +\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n> +\t\t\treq->uri_path.buf);\n> @@ t/helper/test-http-server.c: enum worker_result {\n> +\tcp.git_cmd = 1;\n> +\tstrvec_push(&cp.args, \"http-backend\");\n> +\tres = run_command(&cp);\n> -+\tclose(1);\n> -+\tclose(0);\n> ++\tclose(STDOUT_FILENO);\n> ++\tclose(STDIN_FILENO);\n> +\treturn !!res;\n> +}\n> +\n> +static enum worker_result dispatch(struct req *req)\n> +{\n> +\tif (is_git_request(req))\n> -+\t\treturn do__git(req, NULL);\n> ++\t\treturn do__git(req);\n> +\n> -+\treturn send_http_error(1, 501, \"Not Implemented\", -1, NULL,\n> ++\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n> +\t\t\t WR_OK | WR_HANGUP);\n> +}\n> +\n> @@ t/helper/test-http-server.c: enum worker_result {\n> \tchar *client_port = getenv(\"REMOTE_PORT\");\n> \tenum worker_result wr = WR_OK;\n> @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n> - \tset_keep_alive(0);\n> + \tset_keep_alive(0, logerror);\n> \n> \twhile (1) {\n> --\t\tif (write_in_full(1, response, strlen(response)) < 0) {\n> +-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n> -\t\t\tlogerror(\"unable to write response\");\n> -\t\t\twr = WR_IO_ERROR;\n> -\t\t}\n> +\t\treq__release(&req);\n> +\n> -+\t\talarm(init_timeout ? init_timeout : timeout);\n> ++\t\talarm(timeout);\n> +\t\twr = req__read(&req, 0);\n> +\t\talarm(0);\n> +\n> -+\t\tif (wr & WR_STOP_THE_MUSIC)\n> ++\t\tif (wr != WR_OK)\n> +\t\t\tbreak;\n> \n> +\t\twr = dispatch(&req);\n> - \t\tif (wr & WR_STOP_THE_MUSIC)\n> + \t\tif (wr != WR_OK)\n> \t\t\tbreak;\n> \t}\n> \n> @@ t/t5556-http-auth.sh (new)\n> +\n> +test_description='test http auth header and credential helper interop'\n> +\n> ++TEST_NO_CREATE_REPO=1\n> +. ./test-lib.sh\n> +\n> +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n> +\n> +# Setup a repository\n> +#\n> -+REPO_DIR=\"$(pwd)\"/repo\n> ++REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n> +\n> +# Setup some lookback URLs where test-http-server will be listening.\n> +# We will spawn it directly inside the repo directory, so we avoid\n> @@ t/t5556-http-auth.sh (new)\n> +# The server will shutdown if/when we delete it (this is easier than\n> +# killing it by PID).\n> +#\n> -+PID_FILE=\"$(pwd)\"/pid-file.pid\n> -+SERVER_LOG=\"$(pwd)\"/OUT.server.log\n> ++PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n> ++SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n> +\n> +PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n> +\n> @@ t/t5556-http-auth.sh (new)\n> +\n> +test_expect_success 'http auth anonymous no challenge' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> -+\tstart_http_server --allow-anonymous &&\n> ++\tstart_http_server &&\n> +\n> +\t# Attempt to read from a protected repository\n> +\tgit ls-remote $ORIGIN_URL\n> \n"},{"id":"470562","messageId":"AS2PR03MB981566FB36C357D009B64D4FC0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"2a0b5f3f-7ab2-bc9e-76ac-93a52b4d32d0@github.com","subject":"Re: [PATCH v5 06/10] test-http-server: add simple authentication","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:21:29Z","receivedAt":"2023-01-17T23:35:11Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-13 10:10, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> +static int read_auth_config(const char *name, const char *val, void *data)\n>> +{\n>> +\tint ret = 0;\n>> +\tchar *scheme = NULL;\n>> +\tchar *token = NULL;\n>> +\tchar *challenge = NULL;\n>> +\tstruct auth_module *mod = NULL;\n>> +\n>> +\tif (!strcmp(name, \"auth.challenge\")) {\n>> +\t\tif (split_auth_param(val, &scheme, &challenge, 0)) {\n>> +\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\n>> +\t\tmod = create_auth_module(scheme, challenge);\n>> +\t\tif (add_auth_module(mod)) {\n>> +\t\t\tret = error(\"failed to add auth module '%s'\", val);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\t}\n>> +\tif (!strcmp(name, \"auth.token\")) {\n>> +\t\tif (split_auth_param(val, &scheme, &token, 1)) {\n>> +\t\t\tret = error(\"invalid auth token '%s'\", val);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\n>> +\t\tmod = get_auth_module(scheme);\n>> +\t\tif (!mod) {\n>> +\t\t\tret = error(\"auth scheme not defined '%s'\\n\", scheme);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\n>> +\t\tstring_list_append(mod->tokens, token);\n>> +\t}\n> \n> I don't think this addresses the implicit option ordering requirement noted\n> in [3]; instead of needing '--auth' before '--auth-token', this now needs\n> 'auth.challenge' before 'auth.token' in the config file. While I'd prefer it\n> if this could be rearranged so that the auth setup happens after all config\n> parsing (so the order doesn't matter), if you want to leave it as-is please\n> add a comment somewhere in this file explaining that requirement and/or add\n> a note to the \"auth scheme not defined\" error message. \n> \n> [3] https://lore.kernel.org/git/2a5d6586-3d2c-8af4-12be-a5a106f966b5@github.com/\n> \n>> +\tif (!strcmp(name, \"auth.allowanonymous\")) {\n>> +\t\tallow_anonymous = git_config_bool(name, val);\n>> +\t}\n>> +\tif (!strcmp(name, \"auth.extraheader\")) {\n>> +\t\tstrvec_push(&extra_headers, val);\n>> +\t}\n> \n> Is it worth printing a warning if the option found isn't any of the above?\n> Something like \"ignoring \". This is a test helper, so\n> user-friendliness isn't quite as important as it is for builtins, but the\n> warning might be helpful to developers trying to use it in the future.\n\nI tried this suggestion of adding a warning, but it felt wrong. You are correct\nin the first instance that it should really \"just work\" when specified in any\norder. Watch for the next iteration where I'll make it such you can specify them\nin any order :-)\n\nThanks,\nMatthew\n"},{"id":"470563","messageId":"AS2PR03MB9815C6A25546FD8CAD4D0CD6C0C69@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"98940e93-c4c5-01ec-54b2-b6015f488ad0@github.com","subject":"Re: [PATCH v5 10/10] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-17T21:18:55Z","receivedAt":"2023-01-17T23:35:17Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-12 12:41, Derrick Stolee wrote:\n\n> On 1/11/2023 5:13 PM, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> +static void credential_write_strvec(FILE *fp, const char *key,\n>> +\t\t\t\t const struct strvec *vec)\n>> +{\n>> +\tint i = 0;\n>> +\tconst char *full_key = xstrfmt(\"%s[]\", key);\n>> +\tfor (; i < vec->nr; i++) {\n> \n> style nit: use \"int i;\" and \"for (i = 0; ...\"\n\nThanks for pointing this out; I missed that C99 style for-loops\nwere allowed now. As Ævar pointed out, this should also be `size_t`\nand not `int`.\n\n>> test_expect_success 'http auth anonymous no challenge' '\n>> \ttest_when_finished \"per_test_cleanup\" &&\n>> -\tstart_http_server &&\n>> +\n>> +\tcat >auth.config <<-EOF &&\n>> +\t[auth]\n>> +\t allowAnonymous = true\n>> +\tEOF\n>> +\n>> +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n> \n> I see that you added auth.allowAnonymous and --auth-config options\n> in Patch 6, so perhaps this test change could move to that patch.\n\nGood point; will update on reroll.\n\n> Thanks,\n> -Stolee\n"},{"id":"470567","messageId":"74b0de14185120c9d53d7470e59f57fa20a1927f.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 01/12] daemon: libify socket setup and option functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:06Z","receivedAt":"2023-01-18T03:30:37Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions for setting up listening sockets and keep-alive options\nfrom `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct\ndependencies on global state by inlining the behaviour at the callsites\nfor all libified functions.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 23 ++++++\n daemon.c | 214 +------------------------------------------------\n 4 files changed, 237 insertions(+), 210 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n\ndiff --git a/Makefile b/Makefile\nindex b258fdbed86..2654094dbb5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o\n LIB_OBJS += csum-file.o\n LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n+LIB_OBJS += daemon-utils.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n LIB_OBJS += diagnose.o\ndiff --git a/daemon-utils.c b/daemon-utils.c\nnew file mode 100644\nindex 00000000000..b96b55962db\n--- /dev/null\n+++ b/daemon-utils.c\n@@ -0,0 +1,209 @@\n+#include \"cache.h\"\n+#include \"daemon-utils.h\"\n+\n+void set_keep_alive(int sockfd, log_fn logerror)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd, logerror);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd, logerror);\n+\n+\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(NULL, listen_port, socklist, reuseaddr,\n+\t\t\t\t logerror);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist, reuseaddr,\n+\t\t\t\t\t\t logerror);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nnew file mode 100644\nindex 00000000000..6710a2a6dc0\n--- /dev/null\n+++ b/daemon-utils.h\n@@ -0,0 +1,23 @@\n+#ifndef DAEMON_UTILS_H\n+#define DAEMON_UTILS_H\n+\n+#include \"git-compat-util.h\"\n+#include \"string-list.h\"\n+\n+typedef void (*log_fn)(const char *msg, ...);\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+/* Enable sending of keep-alive messages on the socket. */\n+void set_keep_alive(int sockfd, log_fn logerror);\n+\n+/* Setup a number of sockets to listen on the provided addresses. */\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror);\n+\n+#endif\ndiff --git a/daemon.c b/daemon.c\nindex 0ae7d12b5c1..1ed4e705680 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1,9 +1,9 @@\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"daemon-utils.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n-#include \"string-list.h\"\n \n #ifdef NO_INITGROUPS\n #define initgroups(x, y) (0) /* nothing */\n@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)\n \tstrbuf_release(&hi->tcp_port);\n }\n \n-static void set_keep_alive(int sockfd)\n-{\n-\tint ka = 1;\n-\n-\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n-\t\tif (errno != ENOTSOCK)\n-\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n-\t\t\t\tstrerror(errno));\n-\t}\n-}\n-\n static int execute(void)\n {\n \tchar *line = packet_buffer;\n@@ -759,7 +748,7 @@ static int execute(void)\n \tif (addr)\n \t\tloginfo(\"Connection from %s:%s\", addr, port);\n \n-\tset_keep_alive(0);\n+\tset_keep_alive(0, logerror);\n \talarm(init_timeout ? init_timeout : timeout);\n \tpktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);\n \talarm(0);\n@@ -938,202 +927,6 @@ static void child_handler(int signo)\n \tsignal(SIGCHLD, child_handler);\n }\n \n-static int set_reuse_addr(int sockfd)\n-{\n-\tint on = 1;\n-\n-\tif (!reuseaddr)\n-\t\treturn 0;\n-\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n-\t\t\t &on, sizeof(on));\n-}\n-\n-struct socketlist {\n-\tint *list;\n-\tsize_t nr;\n-\tsize_t alloc;\n-};\n-\n-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n-{\n-#ifdef NO_IPV6\n-\tstatic char ip[INET_ADDRSTRLEN];\n-#else\n-\tstatic char ip[INET6_ADDRSTRLEN];\n-#endif\n-\n-\tswitch (family) {\n-#ifndef NO_IPV6\n-\tcase AF_INET6:\n-\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n-\t\tbreak;\n-#endif\n-\tcase AF_INET:\n-\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n-\t\tbreak;\n-\tdefault:\n-\t\txsnprintf(ip, sizeof(ip), \"\");\n-\t}\n-\treturn ip;\n-}\n-\n-#ifndef NO_IPV6\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tint socknum = 0;\n-\tchar pbuf[NI_MAXSERV];\n-\tstruct addrinfo hints, *ai0, *ai;\n-\tint gai;\n-\tlong flags;\n-\n-\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n-\tmemset(&hints, 0, sizeof(hints));\n-\thints.ai_family = AF_UNSPEC;\n-\thints.ai_socktype = SOCK_STREAM;\n-\thints.ai_protocol = IPPROTO_TCP;\n-\thints.ai_flags = AI_PASSIVE;\n-\n-\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n-\tif (gai) {\n-\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n-\t\treturn 0;\n-\t}\n-\n-\tfor (ai = ai0; ai; ai = ai->ai_next) {\n-\t\tint sockfd;\n-\n-\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n-\t\tif (sockfd < 0)\n-\t\t\tcontinue;\n-\t\tif (sockfd >= FD_SETSIZE) {\n-\t\t\tlogerror(\"Socket descriptor too large\");\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-#ifdef IPV6_V6ONLY\n-\t\tif (ai->ai_family == AF_INET6) {\n-\t\t\tint on = 1;\n-\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n-\t\t\t\t &on, sizeof(on));\n-\t\t\t/* Note: error is not fatal */\n-\t\t}\n-#endif\n-\n-\t\tif (set_reuse_addr(sockfd)) {\n-\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tset_keep_alive(sockfd);\n-\n-\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n-\t\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\t\tif (listen(sockfd, 5) < 0) {\n-\t\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\n-\t\tflags = fcntl(sockfd, F_GETFD, 0);\n-\t\tif (flags >= 0)\n-\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\t\tsocklist->list[socklist->nr++] = sockfd;\n-\t\tsocknum++;\n-\t}\n-\n-\tfreeaddrinfo(ai0);\n-\n-\treturn socknum;\n-}\n-\n-#else /* NO_IPV6 */\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tstruct sockaddr_in sin;\n-\tint sockfd;\n-\tlong flags;\n-\n-\tmemset(&sin, 0, sizeof sin);\n-\tsin.sin_family = AF_INET;\n-\tsin.sin_port = htons(listen_port);\n-\n-\tif (listen_addr) {\n-\t\t/* Well, host better be an IP address here. */\n-\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n-\t\t\treturn 0;\n-\t} else {\n-\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n-\t}\n-\n-\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n-\tif (sockfd < 0)\n-\t\treturn 0;\n-\n-\tif (set_reuse_addr(sockfd)) {\n-\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tset_keep_alive(sockfd);\n-\n-\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n-\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tif (listen(sockfd, 5) < 0) {\n-\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tflags = fcntl(sockfd, F_GETFD, 0);\n-\tif (flags >= 0)\n-\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\tsocklist->list[socklist->nr++] = sockfd;\n-\treturn 1;\n-}\n-\n-#endif\n-\n-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tif (!listen_addr->nr)\n-\t\tsetup_named_sock(NULL, listen_port, socklist);\n-\telse {\n-\t\tint i, socknum;\n-\t\tfor (i = 0; i < listen_addr->nr; i++) {\n-\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n-\t\t\t\t\t\t listen_port, socklist);\n-\n-\t\t\tif (socknum == 0)\n-\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n-\t\t\t\t\t listen_addr->items[i].string, listen_port);\n-\t\t}\n-\t}\n-}\n-\n static int service_loop(struct socketlist *socklist)\n {\n \tstruct pollfd *pfd;\n@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,\n {\n \tstruct socketlist socklist = { NULL, 0, 0 };\n \n-\tsocksetup(listen_addr, listen_port, &socklist);\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr,\n+\t\t logerror);\n \tif (socklist.nr == 0)\n \t\tdie(\"unable to allocate any listen sockets on port %u\",\n \t\t listen_port);\n-- \ngitgitgadget\n\n"},{"id":"470568","messageId":"b6ba344a671c674d1caf577194eddd66eb7e1415.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 02/12] daemon: libify child process handling functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:07Z","receivedAt":"2023-01-18T03:30:43Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions and structures for managing child processes started\nfrom the parent daemon-like process from `daemon.c` to the new shared\n`daemon-utils.{c,h}` files.\n\nOne minor functional change is introduced to `check_dead_children()`\nwhere the logging of a dead/disconnected child is now optional. With the\n'libification' of these functions we extract the call to `loginfo` to a\ncall to a function pointer, and guard the log message creation and\nlogging behind a `NULL` check. Callers can now skip logging by passing\n`NULL` as the `log_fn loginfo` argument.\nThe behaviour of callers in `daemon.c` remains the same (save one extra\nNULL check) however as a pointer to `loginfo` is always passed.\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 32 ++++++++++++++++++\n daemon.c | 92 +++-----------------------------------------------\n 3 files changed, 114 insertions(+), 87 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex b96b55962db..8506664b440 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t\t}\n \t}\n }\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+ const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn , unsigned int *live_children)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tCALLOC_ARRAY(newborn, 1);\n+\t(*live_children)++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+void kill_some_child(struct child *firstborn)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tif (loginfo) {\n+\t\t\t\tconst char *dead = \"\";\n+\t\t\t\tif (status)\n+\t\t\t\t\tdead = \" (with error)\";\n+\t\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\",\n+\t\t\t\t\t(uintmax_t)pid, dead);\n+\t\t\t}\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\t(*live_children)--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex 6710a2a6dc0..97e5cae20b8 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -2,6 +2,7 @@\n #define DAEMON_UTILS_H\n \n #include \"git-compat-util.h\"\n+#include \"run-command.h\"\n #include \"string-list.h\"\n \n typedef void (*log_fn)(const char *msg, ...);\n@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t struct socketlist *socklist, int reuseaddr,\n \t log_fn logerror);\n \n+struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+};\n+\n+/*\n+ * Add the child_process to the set of children and increment the number of\n+ * live children.\n+ */\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn, unsigned int *live_children);\n+\n+/*\n+ * Kill the newest connection from a duplicate IP.\n+ *\n+ * This function should be called if the number of connections grows\n+ * past the maximum number of allowed connections.\n+ */\n+void kill_some_child(struct child *firstborn);\n+\n+/*\n+ * Check for children that have disconnected and remove them from the\n+ * active set, decrementing the number of live children.\n+ *\n+ * Optionally log the child PID that disconnected by passing a loginfo\n+ * function.\n+ */\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo);\n+\n #endif\ndiff --git a/daemon.c b/daemon.c\nindex 1ed4e705680..ec3b407ecbc 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -785,93 +785,11 @@ static int execute(void)\n \treturn -1;\n }\n \n-static int addrcmp(const struct sockaddr_storage *s1,\n- const struct sockaddr_storage *s2)\n-{\n-\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n-\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n-\n-\tif (sa1->sa_family != sa2->sa_family)\n-\t\treturn sa1->sa_family - sa2->sa_family;\n-\tif (sa1->sa_family == AF_INET)\n-\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n-\t\t &((struct sockaddr_in *)s2)->sin_addr,\n-\t\t sizeof(struct in_addr));\n-#ifndef NO_IPV6\n-\tif (sa1->sa_family == AF_INET6)\n-\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n-\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n-\t\t sizeof(struct in6_addr));\n-#endif\n-\treturn 0;\n-}\n-\n static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child {\n-\tstruct child *next;\n-\tstruct child_process cld;\n-\tstruct sockaddr_storage address;\n-} *firstborn;\n-\n-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n-{\n-\tstruct child *newborn, **cradle;\n-\n-\tCALLOC_ARRAY(newborn, 1);\n-\tlive_children++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n-\t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n-}\n-\n-/*\n- * This gets called if the number of connections grows\n- * past \"max_connections\".\n- *\n- * We kill the newest connection from a duplicate IP.\n- */\n-static void kill_some_child(void)\n-{\n-\tconst struct child *blanket, *next;\n-\n-\tif (!(blanket = firstborn))\n-\t\treturn;\n-\n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n-\t\t\tbreak;\n-\t\t}\n-}\n-\n-static void check_dead_children(void)\n-{\n-\tint status;\n-\tpid_t pid;\n-\n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n-\t\t\tconst char *dead = \"\";\n-\t\t\tif (status)\n-\t\t\t\tdead = \" (with error)\";\n-\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n-\n-\t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n-\t\t\tlive_children--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n-\t\t} else\n-\t\t\tcradle = &blanket->next;\n-}\n+static struct child *firstborn;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child();\n+\t\tkill_some_child(firstborn);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen);\n+\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470569","messageId":"9967401c972cab547d7619a208c3a0e6a3923cd4.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 03/12] daemon: rename some esoteric/laboured terminology","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:08Z","receivedAt":"2023-01-18T03:30:48Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRename some of the variables and function arguments used to manage child\nprocesses. The existing names are esoteric; stretching an analogy too\nfar to the point of being confusing to understand.\n\nRename \"firstborn\" to \"first_child\", \"newborn\" to \"new_cld\", \"blanket\"\nto \"current\" and \"cradle\" to \"ptr\".\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 46 +++++++++++++++++++++++-----------------------\n daemon-utils.h | 6 +++---\n daemon.c | 10 +++++-----\n 3 files changed, 31 insertions(+), 31 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex 8506664b440..f23ea35ed7b 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,\n }\n \n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn , unsigned int *live_children)\n+\t struct child *first_child, unsigned int *live_children)\n {\n-\tstruct child *newborn, **cradle;\n+\tstruct child *new_cld, **current;\n \n-\tCALLOC_ARRAY(newborn, 1);\n+\tCALLOC_ARRAY(new_cld, 1);\n \t(*live_children)++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\tmemcpy(&new_cld->cld, cld, sizeof(*cld));\n+\tmemcpy(&new_cld->address, addr, addrlen);\n+\tfor (current = &first_child; *current; current = &(*current)->next)\n+\t\tif (!addrcmp(&(*current)->address, &new_cld->address))\n \t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n+\tnew_cld->next = *current;\n+\t*current = new_cld;\n }\n \n-void kill_some_child(struct child *firstborn)\n+void kill_some_child(struct child *first_child)\n {\n-\tconst struct child *blanket, *next;\n+\tconst struct child *current, *next;\n \n-\tif (!(blanket = firstborn))\n+\tif (!(current = first_child))\n \t\treturn;\n \n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\tfor (; (next = current->next); current = next)\n+\t\tif (!addrcmp(¤t->address, &next->address)) {\n+\t\t\tkill(current->cld.pid, SIGTERM);\n \t\t\tbreak;\n \t\t}\n }\n \n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo)\n {\n \tint status;\n \tpid_t pid;\n \n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\tstruct child **ptr, *current;\n+\tfor (ptr = &first_child; (current = *ptr);)\n+\t\tif ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {\n \t\t\tif (loginfo) {\n \t\t\t\tconst char *dead = \"\";\n \t\t\t\tif (status)\n@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,\n \t\t\t}\n \n \t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n+\t\t\t*ptr = current->next;\n \t\t\t(*live_children)--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n+\t\t\tchild_process_clear(¤t->cld);\n+\t\t\tfree(current);\n \t\t} else\n-\t\t\tcradle = &blanket->next;\n+\t\t\tptr = ¤t->next;\n }\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex 97e5cae20b8..c866e9c9a4e 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -32,7 +32,7 @@ struct child {\n * live children.\n */\n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn, unsigned int *live_children);\n+\t struct child *first_child, unsigned int *live_children);\n \n /*\n * Kill the newest connection from a duplicate IP.\n@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl\n * This function should be called if the number of connections grows\n * past the maximum number of allowed connections.\n */\n-void kill_some_child(struct child *firstborn);\n+void kill_some_child(struct child *first_child);\n \n /*\n * Check for children that have disconnected and remove them from the\n@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);\n * Optionally log the child PID that disconnected by passing a loginfo\n * function.\n */\n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo);\n \n #endif\ndiff --git a/daemon.c b/daemon.c\nindex ec3b407ecbc..d3e7d81de18 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -789,7 +789,7 @@ static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child *firstborn;\n+static struct child *first_child;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child(firstborn);\n+\t\tkill_some_child(first_child);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470570","messageId":"79805f042b984bb8ca7c9aaf6a15f8101037c375.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 05/12] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:10Z","receivedAt":"2023-01-18T03:30:50Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a function to the test-http-server test helper to write more\nfull and valid HTTP error responses, including all the standard response\nheaders like `Server` and `Date`.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----\n 1 file changed, 53 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 11071b1dd89..6cdac223a55 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -83,9 +83,59 @@ enum worker_result {\n \tWR_HANGUP = 1<<1,\n };\n \n+static enum worker_result send_http_error(\n+\tint fd,\n+\tint http_code, const char *http_code_name,\n+\tint retry_after_seconds, struct string_list *response_headers,\n+\tenum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n+\t\t http_code, http_code_name);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n static enum worker_result worker(void)\n {\n-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -96,10 +146,8 @@ static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n-\t\t\tlogerror(\"unable to write response\");\n-\t\t\twr = WR_IO_ERROR;\n-\t\t}\n+\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n+\t\t\t\t NULL, WR_OK | WR_HANGUP);\n \n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n-- \ngitgitgadget\n\n"},{"id":"470571","messageId":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v5.git.1673475190.gitgitgadget@gmail.com","subject":"[PATCH v6 00/12] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:05Z","receivedAt":"2023-01-18T03:30:51Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server sharing code\nwith git-daemon, with simple authentication configurable by a config file.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\nMatthew John Cheetham (12):\n daemon: libify socket setup and option functions\n daemon: libify child process handling functions\n daemon: rename some esoteric/laboured terminology\n test-http-server: add stub HTTP server test helper\n test-http-server: add HTTP error response function\n test-http-server: add HTTP request parsing\n test-http-server: pass Git requests to http-backend\n test-http-server: add simple authentication\n test-http-server: add sending of arbitrary headers\n http: replace unsafe size_t multiplication with st_mult\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n credential.c | 12 +\n credential.h | 15 +\n daemon-utils.c | 286 +++++++++\n daemon-utils.h | 55 ++\n daemon.c | 306 +---------\n http.c | 98 ++-\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 910 ++++++++++++++++++++++++++++\n t/lib-credential-helper.sh | 27 +\n t/t5556-http-auth.sh | 398 ++++++++++++\n 13 files changed, 1838 insertions(+), 302 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n create mode 100644 t/helper/test-http-server.c\n create mode 100644 t/lib-credential-helper.sh\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v5:\n\n 1: 74b0de14185 = 1: 74b0de14185 daemon: libify socket setup and option functions\n 2: bc972fc8d3d ! 2: b6ba344a671 daemon: libify child process handling functions\n @@ Commit message\n from the parent daemon-like process from `daemon.c` to the new shared\n `daemon-utils.{c,h}` files.\n \n + One minor functional change is introduced to `check_dead_children()`\n + where the logging of a dead/disconnected child is now optional. With the\n + 'libification' of these functions we extract the call to `loginfo` to a\n + call to a function pointer, and guard the log message creation and\n + logging behind a `NULL` check. Callers can now skip logging by passing\n + `NULL` as the `log_fn loginfo` argument.\n + The behaviour of callers in `daemon.c` remains the same (save one extra\n + NULL check) however as a pointer to `loginfo` is always passed.\n +\n Signed-off-by: Matthew John Cheetham \n \n ## daemon-utils.c ##\n @@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,\n +\tstruct child *newborn, **cradle;\n +\n +\tCALLOC_ARRAY(newborn, 1);\n -+\tlive_children++;\n ++\t(*live_children)++;\n +\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n +\tmemcpy(&newborn->address, addr, addrlen);\n +\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n @@ daemon-utils.c: void socksetup(struct string_list *listen_addr, int listen_port,\n +\n +\t\t\t/* remove the child */\n +\t\t\t*cradle = blanket->next;\n -+\t\t\tlive_children--;\n ++\t\t\t(*live_children)--;\n +\t\t\tchild_process_clear(&blanket->cld);\n +\t\t\tfree(blanket);\n +\t\t} else\n @@ daemon-utils.h: void socksetup(struct string_list *listen_addr, int listen_port,\n +\tstruct sockaddr_storage address;\n +};\n +\n ++/*\n ++ * Add the child_process to the set of children and increment the number of\n ++ * live children.\n ++ */\n +void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n +\t struct child *firstborn, unsigned int *live_children);\n +\n ++/*\n ++ * Kill the newest connection from a duplicate IP.\n ++ *\n ++ * This function should be called if the number of connections grows\n ++ * past the maximum number of allowed connections.\n ++ */\n +void kill_some_child(struct child *firstborn);\n +\n ++/*\n ++ * Check for children that have disconnected and remove them from the\n ++ * active set, decrementing the number of live children.\n ++ *\n ++ * Optionally log the child PID that disconnected by passing a loginfo\n ++ * function.\n ++ */\n +void check_dead_children(struct child *firstborn, unsigned int *live_children,\n +\t\t\t log_fn loginfo);\n +\n 3: 8f176d5955d ! 3: 9967401c972 daemon: rename some esoteric/laboured terminology\n @@ Commit message\n processes. The existing names are esoteric; stretching an analogy too\n far to the point of being confusing to understand.\n \n - Rename \"firstborn\" to simply \"first\", \"newborn\" to \"new_cld\", \"blanket\"\n + Rename \"firstborn\" to \"first_child\", \"newborn\" to \"new_cld\", \"blanket\"\n to \"current\" and \"cradle\" to \"ptr\".\n \n Signed-off-by: Matthew John Cheetham \n @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,\n \n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n -\t struct child *firstborn , unsigned int *live_children)\n -+\t struct child *first, unsigned int *live_children)\n ++\t struct child *first_child, unsigned int *live_children)\n {\n -\tstruct child *newborn, **cradle;\n +\tstruct child *new_cld, **current;\n \n -\tCALLOC_ARRAY(newborn, 1);\n +\tCALLOC_ARRAY(new_cld, 1);\n - \tlive_children++;\n + \t(*live_children)++;\n -\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n -\tmemcpy(&newborn->address, addr, addrlen);\n -\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n -\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n +\tmemcpy(&new_cld->cld, cld, sizeof(*cld));\n +\tmemcpy(&new_cld->address, addr, addrlen);\n -+\tfor (current = &first; *current; current = &(*current)->next)\n ++\tfor (current = &first_child; *current; current = &(*current)->next)\n +\t\tif (!addrcmp(&(*current)->address, &new_cld->address))\n \t\t\tbreak;\n -\tnewborn->next = *cradle;\n @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,\n }\n \n -void kill_some_child(struct child *firstborn)\n -+void kill_some_child(struct child *first)\n ++void kill_some_child(struct child *first_child)\n {\n -\tconst struct child *blanket, *next;\n +\tconst struct child *current, *next;\n \n -\tif (!(blanket = firstborn))\n -+\tif (!(current = first))\n ++\tif (!(current = first_child))\n \t\treturn;\n \n -\tfor (; (next = blanket->next); blanket = next)\n @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,\n }\n \n -void check_dead_children(struct child *firstborn, unsigned int *live_children,\n -+void check_dead_children(struct child *first, unsigned int *live_children,\n ++void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo)\n {\n \tint status;\n @@ daemon-utils.c: static int addrcmp(const struct sockaddr_storage *s1,\n -\tfor (cradle = &firstborn; (blanket = *cradle);)\n -\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n +\tstruct child **ptr, *current;\n -+\tfor (ptr = &first; (current = *ptr);)\n ++\tfor (ptr = &first_child; (current = *ptr);)\n +\t\tif ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {\n \t\t\tif (loginfo) {\n \t\t\t\tconst char *dead = \"\";\n @@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *\n \t\t\t/* remove the child */\n -\t\t\t*cradle = blanket->next;\n +\t\t\t*ptr = current->next;\n - \t\t\tlive_children--;\n + \t\t\t(*live_children)--;\n -\t\t\tchild_process_clear(&blanket->cld);\n -\t\t\tfree(blanket);\n +\t\t\tchild_process_clear(¤t->cld);\n @@ daemon-utils.c: void check_dead_children(struct child *firstborn, unsigned int *\n \n ## daemon-utils.h ##\n @@ daemon-utils.h: struct child {\n - };\n - \n + * live children.\n + */\n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n -\t struct child *firstborn, unsigned int *live_children);\n -+\t struct child *first, unsigned int *live_children);\n - \n ++\t struct child *first_child, unsigned int *live_children);\n + \n + /*\n + * Kill the newest connection from a duplicate IP.\n +@@ daemon-utils.h: void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl\n + * This function should be called if the number of connections grows\n + * past the maximum number of allowed connections.\n + */\n -void kill_some_child(struct child *firstborn);\n -+void kill_some_child(struct child *first);\n - \n ++void kill_some_child(struct child *first_child);\n + \n + /*\n + * Check for children that have disconnected and remove them from the\n +@@ daemon-utils.h: void kill_some_child(struct child *firstborn);\n + * Optionally log the child PID that disconnected by passing a loginfo\n + * function.\n + */\n -void check_dead_children(struct child *firstborn, unsigned int *live_children,\n -+void check_dead_children(struct child *first, unsigned int *live_children,\n ++void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo);\n \n #endif\n 4: 706fb3781bd = 4: d6e5e8825e8 test-http-server: add stub HTTP server test helper\n -: ----------- > 5: 79805f042b9 test-http-server: add HTTP error response function\n 5: 6f66bf146b4 ! 6: 252098db219 test-http-server: add HTTP error response function\n @@ Metadata\n Author: Matthew John Cheetham \n \n ## Commit message ##\n - test-http-server: add HTTP error response function\n + test-http-server: add HTTP request parsing\n \n - Introduce a function to the test-http-server test helper to write more\n - full and valid HTTP error responses, including all the standard response\n - headers like `Server` and `Date`.\n + Add ability to parse HTTP requests to the test-http-server test helper.\n \n Signed-off-by: Matthew John Cheetham \n \n @@ t/helper/test-http-server.c: enum worker_result {\n +\tstring_list_clear(&req->header_list, 0);\n +}\n +\n -+static enum worker_result send_http_error(\n -+\tint fd,\n -+\tint http_code, const char *http_code_name,\n -+\tint retry_after_seconds, struct string_list *response_headers,\n -+\tenum worker_result wr_in)\n -+{\n -+\tstruct strbuf response_header = STRBUF_INIT;\n -+\tstruct strbuf response_content = STRBUF_INIT;\n -+\tstruct string_list_item *h;\n -+\tenum worker_result wr;\n -+\n -+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n -+\t\t http_code, http_code_name);\n -+\tif (retry_after_seconds > 0)\n -+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n -+\t\t\t retry_after_seconds);\n -+\n -+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n -+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n -+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n -+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n -+\tif (retry_after_seconds > 0)\n -+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n -+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n -+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n -+\tif (response_headers)\n -+\t\tfor_each_string_list_item(h, response_headers)\n -+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n -+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n -+\n -+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n -+\t\tlogerror(\"unable to write response header\");\n -+\t\twr = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n -+\t\tlogerror(\"unable to write response content body\");\n -+\t\twr = WR_IO_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\twr = wr_in;\n -+\n -+done:\n -+\tstrbuf_release(&response_header);\n -+\tstrbuf_release(&response_content);\n -+\n -+\treturn wr;\n -+}\n -+\n + static enum worker_result send_http_error(\n + \tint fd,\n + \tint http_code, const char *http_code_name,\n +@@ t/helper/test-http-server.c: done:\n + \treturn wr;\n + }\n + \n +/*\n + * Read the HTTP request up to the start of the optional message-body.\n + * We do this byte-by-byte because we have keep-alive turned on and\n @@ t/helper/test-http-server.c: enum worker_result {\n +\treturn result;\n +}\n +\n -+static int is_git_request(struct req *req)\n -+{\n -+\tstatic regex_t *smart_http_regex;\n -+\tstatic int initialized;\n -+\n -+\tif (!initialized) {\n -+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n -+\t\t/*\n -+\t\t * This regular expression matches all dumb and smart HTTP\n -+\t\t * requests that are currently in use, and defined in\n -+\t\t * Documentation/gitprotocol-http.txt.\n -+\t\t *\n -+\t\t */\n -+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n -+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n -+\t\t\t REG_EXTENDED)) {\n -+\t\t\twarning(\"could not compile smart HTTP regex\");\n -+\t\t\tsmart_http_regex = NULL;\n -+\t\t}\n -+\t\tinitialized = 1;\n -+\t}\n -+\n -+\treturn smart_http_regex &&\n -+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n -+}\n -+\n -+static enum worker_result do__git(struct req *req)\n -+{\n -+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n -+\tstruct child_process cp = CHILD_PROCESS_INIT;\n -+\tint res;\n -+\n -+\t/*\n -+\t * Note that we always respond with a 200 OK response even if the\n -+\t * http-backend process exits with an error. This helper is intended\n -+\t * only to be used to exercise the HTTP auth handling in the Git client,\n -+\t * and specifically around authentication (not handled by http-backend).\n -+\t *\n -+\t * If we wanted to respond with a more 'valid' HTTP response status then\n -+\t * we'd need to buffer the output of http-backend, wait for and grok the\n -+\t * exit status of the process, then write the HTTP status line followed\n -+\t * by the http-backend output. This is outside of the scope of this test\n -+\t * helper's use at time of writing.\n -+\t *\n -+\t * The important auth responses (401) we are handling prior to getting\n -+\t * to this point.\n -+\t */\n -+\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n -+\t\treturn error(_(\"could not send '%s'\"), ok);\n -+\n -+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n -+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n -+\t\t\treq->uri_path.buf);\n -+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n -+\tif (req->query_args.len)\n -+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n -+\t\t\t\treq->query_args.buf);\n -+\tif (req->content_type)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n -+\t\t\t\treq->content_type);\n -+\tif (req->content_length >= 0)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n -+\t\t\t\t(intmax_t)req->content_length);\n -+\tcp.git_cmd = 1;\n -+\tstrvec_push(&cp.args, \"http-backend\");\n -+\tres = run_command(&cp);\n -+\tclose(STDOUT_FILENO);\n -+\tclose(STDIN_FILENO);\n -+\treturn !!res;\n -+}\n -+\n +static enum worker_result dispatch(struct req *req)\n +{\n -+\tif (is_git_request(req))\n -+\t\treturn do__git(req);\n -+\n +\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n +\t\t\t WR_OK | WR_HANGUP);\n +}\n +\n static enum worker_result worker(void)\n {\n --\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n +\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n --\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n --\t\t\tlogerror(\"unable to write response\");\n --\t\t\twr = WR_IO_ERROR;\n --\t\t}\n +-\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n +-\t\t\t\t NULL, WR_OK | WR_HANGUP);\n +\t\treq__release(&req);\n +\n +\t\talarm(timeout);\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\n -\n - ## t/t5556-http-auth.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description='test http auth header and credential helper interop'\n -+\n -+TEST_NO_CREATE_REPO=1\n -+. ./test-lib.sh\n -+\n -+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n -+\n -+# Setup a repository\n -+#\n -+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n -+\n -+# Setup some lookback URLs where test-http-server will be listening.\n -+# We will spawn it directly inside the repo directory, so we avoid\n -+# any need to configure directory mappings etc - we only serve this\n -+# repository from the root '/' of the server.\n -+#\n -+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n -+ORIGIN_URL=http://$HOST_PORT/\n -+\n -+# The pid-file is created by test-http-server when it starts.\n -+# The server will shutdown if/when we delete it (this is easier than\n -+# killing it by PID).\n -+#\n -+PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n -+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n -+\n -+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n -+\n -+test_expect_success 'setup repos' '\n -+\ttest_create_repo \"$REPO_DIR\" &&\n -+\tgit -C \"$REPO_DIR\" branch -M main\n -+'\n -+\n -+stop_http_server () {\n -+\tif ! test -f \"$PID_FILE\"\n -+\tthen\n -+\t\treturn 0\n -+\tfi\n -+\t#\n -+\t# The server will shutdown automatically when we delete the pid-file.\n -+\t#\n -+\trm -f \"$PID_FILE\"\n -+\t#\n -+\t# Give it a few seconds to shutdown (mainly to completely release the\n -+\t# port before the next test start another instance and it attempts to\n -+\t# bind to it).\n -+\t#\n -+\tfor k in 0 1 2 3 4\n -+\tdo\n -+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n -+\t\tthen\n -+\t\t\treturn 0\n -+\t\tfi\n -+\t\tsleep 1\n -+\tdone\n -+\n -+\techo \"stop_http_server: timeout waiting for server shutdown\"\n -+\treturn 1\n -+}\n -+\n -+start_http_server () {\n -+\t#\n -+\t# Launch our server into the background in repo_dir.\n -+\t#\n -+\t(\n -+\t\tcd \"$REPO_DIR\"\n -+\t\ttest-http-server --verbose \\\n -+\t\t\t--listen=127.0.0.1 \\\n -+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n -+\t\t\t--reuseaddr \\\n -+\t\t\t--pid-file=\"$PID_FILE\" \\\n -+\t\t\t\"$@\" \\\n -+\t\t\t2>\"$SERVER_LOG\" &\n -+\t)\n -+\t#\n -+\t# Give it a few seconds to get started.\n -+\t#\n -+\tfor k in 0 1 2 3 4\n -+\tdo\n -+\t\tif test -f \"$PID_FILE\"\n -+\t\tthen\n -+\t\t\treturn 0\n -+\t\tfi\n -+\t\tsleep 1\n -+\tdone\n -+\n -+\techo \"start_http_server: timeout waiting for server startup\"\n -+\treturn 1\n -+}\n -+\n -+per_test_cleanup () {\n -+\tstop_http_server &&\n -+\trm -f OUT.*\n -+}\n -+\n -+test_expect_success 'http auth anonymous no challenge' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\tstart_http_server &&\n -+\n -+\t# Attempt to read from a protected repository\n -+\tgit ls-remote $ORIGIN_URL\n -+'\n -+\n -+test_done\n -: ----------- > 7: ab06ac9b965 test-http-server: pass Git requests to http-backend\n 6: c3c3d17a688 ! 8: a1ff55dd6e2 test-http-server: add simple authentication\n @@ Commit message\n tokens and only approved if a matching token is found, or if no auth\n was provided and anonymous auth is enabled.\n \n + Configuration for auth includes a simple set of three options:\n +\n + [auth]\n + challenge = [:]\n + token = :[]*\n + allowAnonymous = \n +\n + `auth.challenge` allows you define what authentication schemes, and\n + optional challenge parameters the server should use. Scheme names are\n + unique and subsequently specified challenge parameters in the config\n + file will replace previously specified ones.\n +\n + `auth.token` allows you to define the set of value token values for an\n + authentication scheme. This is a multi-var and each entry in the\n + config file will append to the set of valid tokens for that scheme.\n + Specifying an empty token value will clear the list of tokens so far for\n + that scheme, i.e. `token = :`.\n +\n + `auth.allowAnonymous` controls whether or not unauthenticated requests\n + (those without any `Authorization` headers) should succeed or not, and\n + trigger a 401 Unauthorized response.\n +\n Signed-off-by: Matthew John Cheetham \n \n ## t/helper/test-http-server.c ##\n @@ t/helper/test-http-server.c: static int is_git_request(struct req *req)\n \tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n + \t * exit status of the process, then write the HTTP status line followed\n + \t * by the http-backend output. This is outside of the scope of this test\n + \t * helper's use at time of writing.\n ++\t *\n ++\t * The important auth responses (401) we are handling prior to getting\n ++\t * to this point.\n + \t */\n \tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n \t\treturn error(_(\"could not send '%s'\"), ok);\n \n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +static struct auth_module **auth_modules = NULL;\n +static size_t auth_modules_nr = 0;\n +static size_t auth_modules_alloc = 0;\n -+static struct strvec extra_headers = STRVEC_INIT;\n +\n -+static struct auth_module *create_auth_module(const char *scheme,\n -+\t\t\t\t\t const char *challenge)\n -+{\n -+\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n -+\tmod->scheme = xstrdup(scheme);\n -+\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n -+\tCALLOC_ARRAY(mod->tokens, 1);\n -+\tstring_list_init_dup(mod->tokens);\n -+\treturn mod;\n -+}\n -+\n -+static struct auth_module *get_auth_module(const char *scheme)\n ++static struct auth_module *get_auth_module(const char *scheme, int create)\n +{\n +\tint i;\n +\tstruct auth_module *mod;\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\t\treturn mod;\n +\t}\n +\n -+\treturn NULL;\n -+}\n ++\tif (create) {\n ++\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n ++\t\tmod->scheme = xstrdup(scheme);\n ++\t\tmod->challenge_params = NULL;\n ++\t\tCALLOC_ARRAY(mod->tokens, 1);\n ++\t\tstring_list_init_dup(mod->tokens);\n +\n -+static int add_auth_module(struct auth_module *mod)\n -+{\n -+\tif (get_auth_module(mod->scheme))\n -+\t\treturn error(\"duplicate auth scheme '%s'\\n\", mod->scheme);\n ++\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n ++\t\tauth_modules[auth_modules_nr++] = mod;\n +\n -+\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n -+\tauth_modules[auth_modules_nr++] = mod;\n ++\t\treturn mod;\n ++\t}\n +\n -+\treturn 0;\n ++\treturn NULL;\n +}\n +\n +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\t\t/* trim trailing space ' ' */\n +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n +\n -+\t\t\tmod = get_auth_module(split[0]->buf);\n ++\t\t\tmod = get_auth_module(split[0]->buf, 0);\n +\t\t\tif (mod) {\n +\t\t\t\tresult = AUTH_DENY;\n +\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\t\tstring_list_append(&hdrs, challenge);\n +\t\t}\n +\n -+\t\tfor (i = 0; i < extra_headers.nr; i++)\n -+\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n -+\n +\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n +\t\t\t\t &hdrs, *wr);\n +\t}\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t (result == AUTH_UNKNOWN && allow_anonymous);\n +}\n +\n -+static int split_auth_param(const char *str, char **scheme, char **val, int required_val)\n ++static int split_auth_param(const char *str, char **scheme, char **val)\n +{\n +\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n +\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\treturn -1;\n +\n +\t/* trim trailing ':' */\n -+\tif (p[1])\n ++\tif (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')\n +\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n +\n -+\tif (required_val && !p[1])\n -+\t\treturn -1;\n -+\n +\t*scheme = strbuf_detach(p[0], NULL);\n +\n +\tif (p[1])\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\tstruct auth_module *mod = NULL;\n +\n +\tif (!strcmp(name, \"auth.challenge\")) {\n -+\t\tif (split_auth_param(val, &scheme, &challenge, 0)) {\n ++\t\tif (split_auth_param(val, &scheme, &challenge)) {\n +\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n +\t\t\tgoto cleanup;\n +\t\t}\n +\n -+\t\tmod = create_auth_module(scheme, challenge);\n -+\t\tif (add_auth_module(mod)) {\n -+\t\t\tret = error(\"failed to add auth module '%s'\", val);\n -+\t\t\tgoto cleanup;\n -+\t\t}\n -+\t}\n -+\tif (!strcmp(name, \"auth.token\")) {\n -+\t\tif (split_auth_param(val, &scheme, &token, 1)) {\n -+\t\t\tret = error(\"invalid auth token '%s'\", val);\n -+\t\t\tgoto cleanup;\n -+\t\t}\n ++\t\tmod = get_auth_module(scheme, 1);\n +\n -+\t\tmod = get_auth_module(scheme);\n -+\t\tif (!mod) {\n -+\t\t\tret = error(\"auth scheme not defined '%s'\\n\", scheme);\n ++\t\t/* Replace any existing challenge parameters */\n ++\t\tfree(mod->challenge_params);\n ++\t\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n ++\t} else if (!strcmp(name, \"auth.token\")) {\n ++\t\tif (split_auth_param(val, &scheme, &token)) {\n ++\t\t\tret = error(\"invalid auth token '%s'\", val);\n +\t\t\tgoto cleanup;\n +\t\t}\n +\n -+\t\tstring_list_append(mod->tokens, token);\n -+\t}\n -+\tif (!strcmp(name, \"auth.allowanonymous\")) {\n ++\t\tmod = get_auth_module(scheme, 1);\n ++\n ++\t\t/*\n ++\t\t * Append to set of valid tokens unless an empty token value\n ++\t\t * is provided, then clear the existing list.\n ++\t\t */\n ++\t\tif (token)\n ++\t\t\tstring_list_append(mod->tokens, token);\n ++\t\telse\n ++\t\t\tstring_list_clear(mod->tokens, 1);\n ++\t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n +\t\tallow_anonymous = git_config_bool(name, val);\n -+\t}\n -+\tif (!strcmp(name, \"auth.extraheader\")) {\n -+\t\tstrvec_push(&extra_headers, val);\n ++\t} else {\n ++\t\twarning(\"unknown auth config '%s'\", name);\n +\t}\n +\n +cleanup:\n @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\n +\n + ## t/t5556-http-auth.sh ##\n +@@ t/t5556-http-auth.sh: per_test_cleanup () {\n + \trm -f OUT.*\n + }\n + \n ++test_expect_success CURL 'http auth server auth config' '\n ++\t#test_when_finished \"per_test_cleanup\" &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t\tchallenge = no-params\n ++\t\tchallenge = with-params:foo=\\\"bar\\\" p=1\n ++\t\tchallenge = with-params:foo=\\\"replaced\\\" q=1\n ++\n ++\t\ttoken = no-explicit-challenge:valid-token\n ++\t\ttoken = no-explicit-challenge:also-valid\n ++\t\ttoken = reset-tokens:these-tokens\n ++\t\ttoken = reset-tokens:will-be-reset\n ++\t\ttoken = reset-tokens:\n ++\t\ttoken = reset-tokens:the-only-valid-one\n ++\n ++\t\tallowAnonymous = false\n ++\tEOF\n ++\n ++\tcat >OUT.expected <<-EOF &&\n ++\tWWW-Authenticate: no-params\n ++\tWWW-Authenticate: with-params foo=\"replaced\" q=1\n ++\tWWW-Authenticate: no-explicit-challenge\n ++\tWWW-Authenticate: reset-tokens\n ++\n ++\tError: 401 Unauthorized\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n ++\n ++\tcurl --include $ORIGIN_URL >OUT.curl &&\n ++\ttr -d \"\\r\" OUT.actual &&\n ++\n ++\ttest_cmp OUT.expected OUT.actual\n ++'\n ++\n + test_expect_success 'http auth anonymous no challenge' '\n + \ttest_when_finished \"per_test_cleanup\" &&\n + \n +-\tstart_http_server &&\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t\tallowAnonymous = true\n ++\tEOF\n ++\n ++\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n + \n + \t# Attempt to read from a protected repository\n + \tgit ls-remote $ORIGIN_URL\n -: ----------- > 9: 76125cdf239 test-http-server: add sending of arbitrary headers\n 7: 9c4d25945dd = 10: cc9a220ed1f http: replace unsafe size_t multiplication with st_mult\n 8: 65a620b08ef < -: ----------- strvec: expose strvec_push_nodup for external use\n 9: bcfec529d95 ! 11: bc1ac8d3eb3 http: read HTTP WWW-Authenticate response headers\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t\tif (!values->nr) {\n +\t\t\tBUG(\"should have at least one existing header value\");\n +\t\t} else if (buf.len) {\n -+\t\t\tconst char *prev = values->v[values->nr - 1];\n -+\t\t\tstruct strbuf append = STRBUF_INIT;\n -+\t\t\tstrbuf_addstr(&append, prev);\n ++\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n +\n +\t\t\t/* Join two non-empty values with a single space. */\n -+\t\t\tif (append.len)\n -+\t\t\t\tstrbuf_addch(&append, ' ');\n -+\n -+\t\t\tstrbuf_addbuf(&append, &buf);\n ++\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n +\n +\t\t\tstrvec_pop(values);\n -+\t\t\tstrvec_push_nodup(values, strbuf_detach(&append, NULL));\n ++\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n ++\t\t\tfree(prev);\n +\t\t}\n +\n +\t\tgoto exit;\n 10: af66d2d2ede ! 12: 7c8229f0b11 credential: add WWW-Authenticate header to cred requests\n @@ credential.c: static void credential_write_item(FILE *fp, const char *key, const\n +static void credential_write_strvec(FILE *fp, const char *key,\n +\t\t\t\t const struct strvec *vec)\n +{\n -+\tint i = 0;\n -+\tconst char *full_key = xstrfmt(\"%s[]\", key);\n -+\tfor (; i < vec->nr; i++) {\n ++\tchar *full_key = xstrfmt(\"%s[]\", key);\n ++\tfor (size_t i = 0; i < vec->nr; i++) {\n +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n +\t}\n -+\tfree((void*)full_key);\n ++\tfree(full_key);\n +}\n +\n void credential_write(const struct credential *c, FILE *fp)\n @@ credential.c: void credential_write(const struct credential *c, FILE *fp)\n \n static int run_credential_helper(struct credential *c,\n \n - ## t/helper/test-credential-helper-replay.sh (new) ##\n + ## t/lib-credential-helper.sh (new) ##\n @@\n -+cmd=$1\n -+teefile=$cmd-actual.cred\n -+catfile=$cmd-response.cred\n -+rm -f $teefile\n -+while read line;\n -+do\n -+\tif test -z \"$line\"; then\n -+\t\tbreak;\n -+\tfi\n -+\techo \"$line\" >> $teefile\n -+done\n -+if test \"$cmd\" = \"get\"; then\n -+\tcat $catfile\n -+fi\n ++setup_credential_helper() {\n ++\ttest_expect_success 'setup credential helper' '\n ++\t\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/credential-helper.sh\" &&\n ++\t\texport CREDENTIAL_HELPER &&\n ++\t\techo $CREDENTIAL_HELPER &&\n ++\n ++\t\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n ++\t\tcmd=$1\n ++\t\tteefile=$cmd-query.cred\n ++\t\tcatfile=$cmd-reply.cred\n ++\t\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n ++\t\tif test \"$cmd\" = \"get\"; then\n ++\t\t\tcat $catfile\n ++\t\tfi\n ++\t\tEOF\n ++\t'\n ++}\n ++\n ++set_credential_reply() {\n ++\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n ++}\n ++\n ++expect_credential_query() {\n ++\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n ++\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n ++\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n ++}\n \n ## t/t5556-http-auth.sh ##\n -@@ t/t5556-http-auth.sh: PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n - SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n +@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'\n + \n + TEST_NO_CREATE_REPO=1\n + . ./test-lib.sh\n ++. \"$TEST_DIRECTORY\"/lib-credential-helper.sh\n \n - PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n -+CREDENTIAL_HELPER=\"$GIT_BUILD_DIR/t/helper/test-credential-helper-replay.sh\" \\\n -+\t&& export CREDENTIAL_HELPER\n + test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n \n - test_expect_success 'setup repos' '\n - \ttest_create_repo \"$REPO_DIR\" &&\n +@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '\n + \tgit -C \"$REPO_DIR\" branch -M main\n + '\n + \n ++setup_credential_helper\n ++\n + stop_http_server () {\n + \tif ! test -f \"$PID_FILE\"\n + \tthen\n @@ t/t5556-http-auth.sh: start_http_server () {\n \n per_test_cleanup () {\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\trm -f auth.config\n }\n \n - test_expect_success 'http auth anonymous no challenge' '\n - \ttest_when_finished \"per_test_cleanup\" &&\n --\tstart_http_server &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t allowAnonymous = true\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n - \n - \t# Attempt to read from a protected repository\n + test_expect_success CURL 'http auth server auth config' '\n +@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >store-expected.cred <<-EOF &&\n ++\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query store <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n +test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n -+\t extraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n ++\t\textraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >store-expected.cred <<-EOF &&\n ++\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\twwwauth[]=basic realm=\"example.com\"\n ++\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query store <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n +test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n ++\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >store-expected.cred <<-EOF &&\n ++\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n ++\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query store <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n +test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n -+\t extraheader = \"WWW-Authenticate:\"\n -+\t extraheader = \" \"\n -+\t extraheader = \" bearer authority=\\\"id.example.com\\\"\"\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n ++\t\textraheader = \"WWW-Authenticate:\"\n ++\t\textraheader = \" \"\n ++\t\textraheader = \" bearer authority=\\\"id.example.com\\\"\"\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\twwwauth[]=bearer authority=\"id.example.com\"\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >store-expected.cred <<-EOF &&\n ++\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\twwwauth[]=basic realm=\"example.com\"\n ++\twwwauth[]=bearer authority=\"id.example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query store <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n +test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = \"foobar:alg=test widget=1\"\n -+\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n ++\t\tchallenge = \"foobar:alg=test widget=1\"\n ++\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=foobar alg=test widget=1\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >store-expected.cred <<-EOF &&\n ++\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\twwwauth[]=foobar alg=test widget=1\n ++\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n ++\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query store <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n -+\n -+\tgit -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp store-expected.cred store-actual.cred\n +'\n +\n +test_expect_success 'http auth www-auth headers to credential helper invalid' '\n @@ t/t5556-http-auth.sh: start_http_server () {\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n -+\t challenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n -+\t challenge = basic:realm=\\\"example.com\\\"\n -+\t token = basic:$USERPASS64\n ++\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n ++\t\tchallenge = basic:realm=\\\"example.com\\\"\n ++\t\ttoken = basic:$USERPASS64\n +\tEOF\n +\n +\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n -+\tcat >get-expected.cred <<-EOF &&\n ++\tset_credential_reply get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\tusername=alice\n ++\tpassword=invalid-passwd\n +\tEOF\n +\n -+\tcat >erase-expected.cred <<-EOF &&\n ++\ttest_must_fail git -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n +\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=invalid-passwd\n +\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n +\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n +\n -+\tcat >get-response.cred <<-EOF &&\n ++\texpect_credential_query erase <<-EOF\n +\tprotocol=http\n +\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=invalid-passwd\n ++\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n ++\twwwauth[]=basic realm=\"example.com\"\n +\tEOF\n -+\n -+\ttest_must_fail git -c credential.helper=\"$CREDENTIAL_HELPER\" ls-remote $ORIGIN_URL &&\n -+\n -+\ttest_cmp get-expected.cred get-actual.cred &&\n -+\ttest_cmp erase-expected.cred erase-actual.cred\n +'\n +\n test_done\n\n-- \ngitgitgadget\n"},{"id":"470572","messageId":"d6e5e8825e8454242820738f0dfb03a9f1c01ced.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 04/12] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:09Z","receivedAt":"2023-01-18T03:30:54Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a mini HTTP server helper that in the future will be enhanced\nto provide a frontend for the git-http-backend, with support for\narbitrary authentication schemes.\n\nRight now, test-http-server is a pared-down copy of the git-daemon that\nalways returns a 501 Not Implemented response to all callers.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 385 ++++++++++++++++++++++++++++\n 4 files changed, 396 insertions(+), 2 deletions(-)\n create mode 100644 t/helper/test-http-server.c\n\ndiff --git a/Makefile b/Makefile\nindex 2654094dbb5..3cd61c792ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o\n # Do not add more tests here unless they have extra dependencies. Add\n # them in TEST_BUILTINS_OBJS above.\n TEST_PROGRAMS_NEED_X += test-fake-ssh\n+TEST_PROGRAMS_NEED_X += test-http-server\n TEST_PROGRAMS_NEED_X += test-tool\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 2f6e0197ffa..5d949dcb16c 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -961,6 +961,9 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+target_link_libraries(test-http-server common-main)\n+\n #reftable-tests\n parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n@@ -980,6 +983,11 @@ if(MSVC)\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n \tset_target_properties(test-fake-ssh test-tool\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n endif()\n \n #wrapper scripts\n@@ -987,8 +995,7 @@ set(wrapper_scripts\n \tgit git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)\n \n set(wrapper_test_scripts\n-\ttest-fake-ssh test-tool)\n-\n+\ttest-http-server test-fake-ssh test-tool)\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..9aa9c752997 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+/test-http-server\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..11071b1dd89\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,385 @@\n+#include \"daemon-utils.h\"\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+;\n+\n+static unsigned int timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+/*\n+ * The code in this section is used by \"worker\" instances to service\n+ * a single connection from a client. The worker talks to the client\n+ * on 0 and 1.\n+ */\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\n+\t/*\n+\t * Various errors while processing the request and/or the response.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_IO_ERROR = 1<<0,\n+\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 1<<1,\n+};\n+\n+static enum worker_result worker(void)\n+{\n+\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0, logerror);\n+\n+\twhile (1) {\n+\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n+\t\t\tlogerror(\"unable to write response\");\n+\t\t\twr = WR_IO_ERROR;\n+\t\t}\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(STDIN_FILENO);\n+\tclose(STDOUT_FILENO);\n+\n+\treturn !!(wr & WR_IO_ERROR);\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child *first_child;\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child(first_child);\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+/*\n+ * This section is executed by both the primary instance and all\n+ * worker instances. So, yes, each child-process re-parses the\n+ * command line argument and re-discovers how it should behave.\n+ */\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\ttrace2_cmd_list_config();\n+\ttrace2_cmd_list_env_vars();\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\n-- \ngitgitgadget\n\n"},{"id":"470573","messageId":"252098db219574527c587bc601565eab81b40c2c.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 06/12] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:11Z","receivedAt":"2023-01-18T03:30:56Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd ability to parse HTTP requests to the test-http-server test helper.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-\n 1 file changed, 173 insertions(+), 2 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 6cdac223a55..36f4a54fe6d 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -83,6 +83,42 @@ enum worker_result {\n \tWR_HANGUP = 1<<1,\n };\n \n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tssize_t content_length;\n+};\n+\n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = -1 \\\n+\t}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n static enum worker_result send_http_error(\n \tint fd,\n \tint http_code, const char *http_code_name,\n@@ -134,8 +170,136 @@ done:\n \treturn wr;\n }\n \n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_OK | WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_IO_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* also store common request headers as struct req members */\n+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\treq->content_length = strtol(hv, &hp, 10);\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->content_length >= 0)\n+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_OK | WR_HANGUP);\n+}\n+\n static enum worker_result worker(void)\n {\n+\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -146,9 +310,16 @@ static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n-\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n-\t\t\t\t NULL, WR_OK | WR_HANGUP);\n+\t\treq__release(&req);\n+\n+\t\talarm(timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n \n+\t\twr = dispatch(&req);\n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\n-- \ngitgitgadget\n\n"},{"id":"470574","messageId":"a1ff55dd6e25aa39f14b494f482720edf7d1eabd.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 08/12] test-http-server: add simple authentication","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:13Z","receivedAt":"2023-01-18T03:30:57Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd simple authentication to the test-http-server test helper.\nAuthentication schemes and sets of valid tokens can be specified via\na configuration file (in the normal gitconfig file format).\nIncoming requests are compared against the set of valid schemes and\ntokens and only approved if a matching token is found, or if no auth\nwas provided and anonymous auth is enabled.\n\nConfiguration for auth includes a simple set of three options:\n\n[auth]\n\tchallenge = [:]\n\ttoken = :[]*\n\tallowAnonymous = \n\n`auth.challenge` allows you define what authentication schemes, and\noptional challenge parameters the server should use. Scheme names are\nunique and subsequently specified challenge parameters in the config\nfile will replace previously specified ones.\n\n`auth.token` allows you to define the set of value token values for an\nauthentication scheme. This is a multi-var and each entry in the\nconfig file will append to the set of valid tokens for that scheme.\nSpecifying an empty token value will clear the list of tokens so far for\nthat scheme, i.e. `token = :`.\n\n`auth.allowAnonymous` controls whether or not unauthenticated requests\n(those without any `Authorization` headers) should succeed or not, and\ntrigger a 401 Unauthorized response.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 233 +++++++++++++++++++++++++++++++++++-\n t/t5556-http-auth.sh | 43 ++++++-\n 2 files changed, 273 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex ae17c738259..691fbfb51d6 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -7,6 +7,7 @@\n #include \"version.h\"\n #include \"dir.h\"\n #include \"date.h\"\n+#include \"config.h\"\n \n #define TR2_CAT \"test-http-server\"\n \n@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =\n \" [--timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n+\" [--auth-config=]\\n\"\n ;\n \n static unsigned int timeout;\n@@ -317,7 +319,7 @@ static int is_git_request(struct req *req)\n \t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n }\n \n-static enum worker_result do__git(struct req *req)\n+static enum worker_result do__git(struct req *req, const char *user)\n {\n \tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -334,10 +336,16 @@ static enum worker_result do__git(struct req *req)\n \t * exit status of the process, then write the HTTP status line followed\n \t * by the http-backend output. This is outside of the scope of this test\n \t * helper's use at time of writing.\n+\t *\n+\t * The important auth responses (401) we are handling prior to getting\n+\t * to this point.\n \t */\n \tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n \t\treturn error(_(\"could not send '%s'\"), ok);\n \n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n \tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n \t\t\treq->uri_path.buf);\n@@ -359,10 +367,218 @@ static enum worker_result do__git(struct req *req)\n \treturn !!res;\n }\n \n+enum auth_result {\n+\t/* No auth module matches the request. */\n+\tAUTH_UNKNOWN = 0,\n+\n+\t/* Auth module denied the request. */\n+\tAUTH_DENY = 1,\n+\n+\t/* Auth module successfully validated the request. */\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tchar *scheme;\n+\tchar *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+\n+static struct auth_module *get_auth_module(const char *scheme, int create)\n+{\n+\tint i;\n+\tstruct auth_module *mod;\n+\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme))\n+\t\t\treturn mod;\n+\t}\n+\n+\tif (create) {\n+\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n+\t\tmod->scheme = xstrdup(scheme);\n+\t\tmod->challenge_params = NULL;\n+\t\tCALLOC_ARRAY(mod->tokens, 1);\n+\t\tstring_list_init_dup(mod->tokens);\n+\n+\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\t\tauth_modules[auth_modules_nr++] = mod;\n+\n+\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/*\n+\t * Check all auth modules and try to validate the request.\n+\t * The first Authorization header that matches a known auth module\n+\t * scheme will be consulted to either approve or deny the request.\n+\t * If no module is found, or if there is no valid token, then 401 error.\n+\t * Otherwise, only permit the request if anonymous auth is enabled.\n+\t * It's atypical for user agents/clients to send multiple Authorization\n+\t * headers, but not explicitly forbidden or defined.\n+\t */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (!split[0] || !split[1]) continue;\n+\n+\t\t\t/* trim trailing space ' ' */\n+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n+\n+\t\t\tmod = get_auth_module(split[0]->buf, 0);\n+\t\t\tif (mod) {\n+\t\t\t\tresult = AUTH_DENY;\n+\n+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\tbreak;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (result != AUTH_DENY && allow_anonymous)\n+\t\t\tbreak;\n+\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\n+\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n+\t\t\t\t &hdrs, *wr);\n+\t}\n+\n+\tstrbuf_list_free(split);\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n+static int split_auth_param(const char *str, char **scheme, char **val)\n+{\n+\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n+\n+\tif (!p[0])\n+\t\treturn -1;\n+\n+\t/* trim trailing ':' */\n+\tif (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')\n+\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t*scheme = strbuf_detach(p[0], NULL);\n+\n+\tif (p[1])\n+\t\t*val = strbuf_detach(p[1], NULL);\n+\n+\tstrbuf_list_free(p);\n+\treturn 0;\n+}\n+\n+static int read_auth_config(const char *name, const char *val, void *data)\n+{\n+\tint ret = 0;\n+\tchar *scheme = NULL;\n+\tchar *token = NULL;\n+\tchar *challenge = NULL;\n+\tstruct auth_module *mod = NULL;\n+\n+\tif (!strcmp(name, \"auth.challenge\")) {\n+\t\tif (split_auth_param(val, &scheme, &challenge)) {\n+\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = get_auth_module(scheme, 1);\n+\n+\t\t/* Replace any existing challenge parameters */\n+\t\tfree(mod->challenge_params);\n+\t\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n+\t} else if (!strcmp(name, \"auth.token\")) {\n+\t\tif (split_auth_param(val, &scheme, &token)) {\n+\t\t\tret = error(\"invalid auth token '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = get_auth_module(scheme, 1);\n+\n+\t\t/*\n+\t\t * Append to set of valid tokens unless an empty token value\n+\t\t * is provided, then clear the existing list.\n+\t\t */\n+\t\tif (token)\n+\t\t\tstring_list_append(mod->tokens, token);\n+\t\telse\n+\t\t\tstring_list_clear(mod->tokens, 1);\n+\t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n+\t\tallow_anonymous = git_config_bool(name, val);\n+\t} else {\n+\t\twarning(\"unknown auth config '%s'\", name);\n+\t}\n+\n+cleanup:\n+\tfree(scheme);\n+\tfree(token);\n+\tfree(challenge);\n+\n+\treturn ret;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n \tif (is_git_request(req))\n-\t\treturn do__git(req);\n+\t\treturn do__git(req, user);\n \n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n@@ -621,6 +837,19 @@ int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--auth-config=\", &v)) {\n+\t\t\tif (!strlen(v)) {\n+\t\t\t\terror(\"invalid argument - missing file path\");\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (git_config_from_file(read_auth_config, v, NULL)) {\n+\t\t\t\terror(\"failed to read auth config file '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tcontinue;\n+\t\t}\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex ce1abffa6aa..cb5562a41bf 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -95,10 +95,51 @@ per_test_cleanup () {\n \trm -f OUT.*\n }\n \n+test_expect_success CURL 'http auth server auth config' '\n+\t#test_when_finished \"per_test_cleanup\" &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = no-params\n+\t\tchallenge = with-params:foo=\\\"bar\\\" p=1\n+\t\tchallenge = with-params:foo=\\\"replaced\\\" q=1\n+\n+\t\ttoken = no-explicit-challenge:valid-token\n+\t\ttoken = no-explicit-challenge:also-valid\n+\t\ttoken = reset-tokens:these-tokens\n+\t\ttoken = reset-tokens:will-be-reset\n+\t\ttoken = reset-tokens:\n+\t\ttoken = reset-tokens:the-only-valid-one\n+\n+\t\tallowAnonymous = false\n+\tEOF\n+\n+\tcat >OUT.expected <<-EOF &&\n+\tWWW-Authenticate: no-params\n+\tWWW-Authenticate: with-params foo=\"replaced\" q=1\n+\tWWW-Authenticate: no-explicit-challenge\n+\tWWW-Authenticate: reset-tokens\n+\n+\tError: 401 Unauthorized\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcurl --include $ORIGIN_URL >OUT.curl &&\n+\ttr -d \"\\r\" OUT.actual &&\n+\n+\ttest_cmp OUT.expected OUT.actual\n+'\n+\n test_expect_success 'http auth anonymous no challenge' '\n \ttest_when_finished \"per_test_cleanup\" &&\n \n-\tstart_http_server &&\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tallowAnonymous = true\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n \n \t# Attempt to read from a protected repository\n \tgit ls-remote $ORIGIN_URL\n-- \ngitgitgadget\n\n"},{"id":"470575","messageId":"76125cdf239df7bebc63a27099e68e71a8216798.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 09/12] test-http-server: add sending of arbitrary headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:14Z","receivedAt":"2023-01-18T03:30:58Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the ability to send arbitrary headers in HTTP responses from the\ntest-http-server. This is useful when we want to test 'malformed'\nresponse message handling.\n\nAdd the following option to the server auth config file:\n\n[auth]\n\textraHeader = []*\n\nEach `auth.extraHeader` value will be appended to the response headers\nverbatim.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 6 ++++++\n t/t5556-http-auth.sh | 7 +++++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 691fbfb51d6..cbaee4fc0f4 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -388,6 +388,7 @@ static int allow_anonymous;\n static struct auth_module **auth_modules = NULL;\n static size_t auth_modules_nr = 0;\n static size_t auth_modules_alloc = 0;\n+static struct strvec extra_headers = STRVEC_INIT;\n \n static struct auth_module *get_auth_module(const char *scheme, int create)\n {\n@@ -489,6 +490,9 @@ done:\n \t\t\tstring_list_append(&hdrs, challenge);\n \t\t}\n \n+\t\tfor (i = 0; i < extra_headers.nr; i++)\n+\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n+\n \t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n \t\t\t\t &hdrs, *wr);\n \t}\n@@ -557,6 +561,8 @@ static int read_auth_config(const char *name, const char *val, void *data)\n \t\t\tstring_list_clear(mod->tokens, 1);\n \t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n \t\tallow_anonymous = git_config_bool(name, val);\n+\t} else if (!strcmp(name, \"auth.extraheader\")) {\n+\t\tstrvec_push(&extra_headers, val);\n \t} else {\n \t\twarning(\"unknown auth config '%s'\", name);\n \t}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex cb5562a41bf..e36107ea95d 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -112,6 +112,10 @@ test_expect_success CURL 'http auth server auth config' '\n \t\ttoken = reset-tokens:the-only-valid-one\n \n \t\tallowAnonymous = false\n+\n+\t\textraHeader = X-Extra-Header: abc\n+\t\textraHeader = X-Extra-Header: 123\n+\t\textraHeader = X-Another: header\\twith\\twhitespace!\n \tEOF\n \n \tcat >OUT.expected <<-EOF &&\n@@ -119,6 +123,9 @@ test_expect_success CURL 'http auth server auth config' '\n \tWWW-Authenticate: with-params foo=\"replaced\" q=1\n \tWWW-Authenticate: no-explicit-challenge\n \tWWW-Authenticate: reset-tokens\n+\tX-Extra-Header: abc\n+\tX-Extra-Header: 123\n+\tX-Another: header\twith\twhitespace!\n \n \tError: 401 Unauthorized\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"470576","messageId":"cc9a220ed1f12aef2f4df940e71adc1fad917a6b.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:15Z","receivedAt":"2023-01-18T03:31:00Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nReplace direct multiplication of two size_t parameters in curl response\nstream handling callback functions with `st_mult` to guard against\noverflows.\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..a2a80318bb2 100644\n--- a/http.c\n+++ b/http.c\n@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n \n size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct buffer *buffer = buffer_;\n \n \tif (size > buffer->buf.len - buffer->posn)\n@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n \n size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct strbuf *buffer = buffer_;\n \n \tstrbuf_add(buffer, ptr, size);\n-- \ngitgitgadget\n\n"},{"id":"470577","messageId":"ab06ac9b965b827612594e3578b7be2a15ec1586.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 07/12] test-http-server: pass Git requests to http-backend","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:12Z","receivedAt":"2023-01-18T03:31:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nTeach the test-http-sever test helper to forward Git requests to the\n`git-http-backend`.\n\nIntroduce a new test script t5556-http-auth.sh that spins up the test\nHTTP server and attempts an `ls-remote` on the served repository,\nwithout any authentication.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 71 ++++++++++++++++++++++++\n t/t5556-http-auth.sh | 107 ++++++++++++++++++++++++++++++++++++\n 2 files changed, 178 insertions(+)\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 36f4a54fe6d..ae17c738259 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -291,8 +291,79 @@ done:\n \treturn result;\n }\n \n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\t/*\n+\t\t * This regular expression matches all dumb and smart HTTP\n+\t\t * requests that are currently in use, and defined in\n+\t\t * Documentation/gitprotocol-http.txt.\n+\t\t *\n+\t\t */\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\t/*\n+\t * Note that we always respond with a 200 OK response even if the\n+\t * http-backend process exits with an error. This helper is intended\n+\t * only to be used to exercise the HTTP auth handling in the Git client,\n+\t * and specifically around authentication (not handled by http-backend).\n+\t *\n+\t * If we wanted to respond with a more 'valid' HTTP response status then\n+\t * we'd need to buffer the output of http-backend, wait for and grok the\n+\t * exit status of the process, then write the HTTP status line followed\n+\t * by the http-backend output. This is outside of the scope of this test\n+\t * helper's use at time of writing.\n+\t */\n+\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n+\t\t\treq->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n+\t\t\t\treq->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n+\t\t\t\treq->content_type);\n+\tif (req->content_length >= 0)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n+\t\t\t\t(intmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(STDOUT_FILENO);\n+\tclose(STDIN_FILENO);\n+\treturn !!res;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tif (is_git_request(req))\n+\t\treturn do__git(req);\n+\n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_OK | WR_HANGUP);\n }\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..ce1abffa6aa\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,107 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n+\n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n+per_test_cleanup () {\n+\tstop_http_server &&\n+\trm -f OUT.*\n+}\n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tstart_http_server &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"470578","messageId":"bc1ac8d3eb3ac6e1161f6b6b67343874c10cd14d.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:16Z","receivedAt":"2023-01-18T03:31:30Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 +++++++++\n http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 110 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex a2a80318bb2..595c93bc7a3 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = st_mult(eltsize, nmemb);\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t * Continuation lines start with at least one whitespace, maybe more,\n+\t * so we should collapse these down to a single SP (valid per the spec).\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\t/* Trim leading whitespace from this continuation hdr line. */\n+\t\tstrbuf_ltrim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (istarts_with(buf.buf, \"http/\"))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1956,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"470579","messageId":"7c8229f0b11693310ae47551fcc5e58f0bb64a0a.1674012618.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v6 12/12] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-18T03:30:17Z","receivedAt":"2023-01-18T03:31:38Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 11 ++\n t/lib-credential-helper.sh | 27 ++++\n t/t5556-http-auth.sh | 245 ++++++++++++++++++++++++++++++-\n 4 files changed, 300 insertions(+), 2 deletions(-)\n create mode 100644 t/lib-credential-helper.sh\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..9f39ebc3c7e 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tchar *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (size_t i = 0; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree(full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh\nnew file mode 100644\nindex 00000000000..8b0e4414234\n--- /dev/null\n+++ b/t/lib-credential-helper.sh\n@@ -0,0 +1,27 @@\n+setup_credential_helper() {\n+\ttest_expect_success 'setup credential helper' '\n+\t\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/credential-helper.sh\" &&\n+\t\texport CREDENTIAL_HELPER &&\n+\t\techo $CREDENTIAL_HELPER &&\n+\n+\t\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\t\tcmd=$1\n+\t\tteefile=$cmd-query.cred\n+\t\tcatfile=$cmd-reply.cred\n+\t\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n+\t\tif test \"$cmd\" = \"get\"; then\n+\t\t\tcat $catfile\n+\t\tfi\n+\t\tEOF\n+\t'\n+}\n+\n+set_credential_reply() {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query() {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex e36107ea95d..79122c611a1 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop'\n \n TEST_NO_CREATE_REPO=1\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-credential-helper.sh\n \n test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n \n@@ -33,6 +34,8 @@ test_expect_success 'setup repos' '\n \tgit -C \"$REPO_DIR\" branch -M main\n '\n \n+setup_credential_helper\n+\n stop_http_server () {\n \tif ! test -f \"$PID_FILE\"\n \tthen\n@@ -92,7 +95,9 @@ start_http_server () {\n \n per_test_cleanup () {\n \tstop_http_server &&\n-\trm -f OUT.*\n+\trm -f OUT.* &&\n+\trm -f *.cred &&\n+\trm -f auth.config\n }\n \n test_expect_success CURL 'http auth server auth config' '\n@@ -152,4 +157,242 @@ test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\t\textraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\t\textraheader = \"WWW-Authenticate:\"\n+\t\textraheader = \" \"\n+\t\textraheader = \" bearer authority=\\\"id.example.com\\\"\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bearer authority=\"id.example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"foobar:alg=test widget=1\"\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\tEOF\n+\n+\ttest_must_fail git -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"470593","messageId":"230118.86k01kxfr7.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"d6e5e8825e8454242820738f0dfb03a9f1c01ced.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:04:24Z","receivedAt":"2023-01-18T11:48:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> [...]\n> +enum worker_result {\n> +\t/*\n> +\t * Operation successful.\n> +\t * Caller *might* keep the socket open and allow keep-alive.\n> +\t */\n> +\tWR_OK = 0,\n> [...]\n> +\tenum worker_result wr = WR_OK;\n> +\n> +\tif (client_addr)\n> +\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n> +\n> +\tset_keep_alive(0, logerror);\n> +\n> +\twhile (1) {\n> +\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n> +\t\t\tlogerror(\"unable to write response\");\n> +\t\t\twr = WR_IO_ERROR;\n> +\t\t}\n> +\n> +\t\tif (wr != WR_OK)\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\tclose(STDIN_FILENO);\n> +\tclose(STDOUT_FILENO);\n> +\n> +\treturn !!(wr & WR_IO_ERROR);\n> +}\n\nWe have cases where we assign \"0\" to a bitfield-looking structure like\nthis, but only in cases where we're planning to use it as a boolean too.\n\nOr, in other cases where we want some to be explicitly <-1.\n\nHere though we're adding a mixed \"OK\" and error use, which seems a bit\nodd. Shouldn't we pick one or the other?\n\nSo far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we\nalso use the bitfield-looking thing as a return value from main()....\n"},{"id":"470595","messageId":"230118.86fsc8xffg.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"79805f042b984bb8ca7c9aaf6a15f8101037c375.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 05/12] test-http-server: add HTTP error response function","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:07:52Z","receivedAt":"2023-01-18T11:58:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n>\n> Introduce a function to the test-http-server test helper to write more\n> full and valid HTTP error responses, including all the standard response\n> headers like `Server` and `Date`.\n>\n> Signed-off-by: Matthew John Cheetham \n> ---\n> t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----\n> 1 file changed, 53 insertions(+), 5 deletions(-)\n>\n> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> index 11071b1dd89..6cdac223a55 100644\n> --- a/t/helper/test-http-server.c\n> +++ b/t/helper/test-http-server.c\n> @@ -83,9 +83,59 @@ enum worker_result {\n> \tWR_HANGUP = 1<<1,\n> };\n\n...okey, this is the commit that makes use of WR_HANGUP. Whatever else\nwe do, let's then squash that addition into this change.\n\n> +static enum worker_result send_http_error(\n> +\tint fd,\n> +\tint http_code, const char *http_code_name,\n> +\tint retry_after_seconds, struct string_list *response_headers,\n> +\tenum worker_result wr_in)\n\nIn general in this series you are mis-indenting argument lists. Our\nusual style is to wrap at 79 characters, then to align (with tabs and\nspaces) with the \"(\".\n\nSo in this case:\n\nstatic enum worker_result send_http_error(int fd, int http_code,\n\t\t\t\t\t const char *http_code_name,\n\t\t\t\t\t int retry_after_seconds,\n\t\t\t\t\t struct string_list *response_headers,\n\t\t\t\t\t enum worker_result wr_in)\n\n> +{\n> +\tstruct strbuf response_header = STRBUF_INIT;\n> +\tstruct strbuf response_content = STRBUF_INIT;\n> +\tstruct string_list_item *h;\n> +\tenum worker_result wr;\n> +\n> +\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n> +\t\t http_code, http_code_name);\n\n\nDitto here, where \"http_code\" should go on the preceding line...\n\n> +\tif (retry_after_seconds > 0)\n> +\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n> +\t\t\t retry_after_seconds);\n> +\n> +\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n\n...and here there's a lack of such wrapping...\n\n> +\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n> +\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n> +\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n> +\tif (retry_after_seconds > 0)\n> +\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n> +\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n> +\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n\n...here you're adding strange whitespace at the start of an argument list...\n\n> +\tif (response_headers)\n> +\t\tfor_each_string_list_item(h, response_headers)\n> +\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n> +\tstrbuf_addstr(&response_header, \"\\r\\n\");\n\nTo comment on the code a bit, this whole thing would be more readable\nIMO if your own headers were also a \"struct string_list\". Yes we'd waste\na bit more memory, but in this case that's fine..\n\nI.e. don't add the \"\\r\\n\" every time, just:\n\n\tstring_list_append(&headers, \"Cache-Control: private\");\n\netc.\n\nThen at the end you'd do e.g.:\n\n\tadd_headers(&buf, &headers);\n\tif (response_headers)\n\t\tadd_headers(&buf, response_headers);\n\nWhere the add_headers() is a trivial \"static\" helper which does that\nfor_each_string_list_item() loop above.\n\n> \twhile (1) {\n> -\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n> -\t\t\tlogerror(\"unable to write response\");\n> -\t\t\twr = WR_IO_ERROR;\n> -\t\t}\n> +\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n> +\t\t\t\t NULL, WR_OK | WR_HANGUP);\n\nThis *does* use correct wrapping & indenation for a continuing argument\nlist.\n"},{"id":"470596","messageId":"230118.86bkmwxf6e.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"252098db219574527c587bc601565eab81b40c2c.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:14:55Z","receivedAt":"2023-01-18T12:03:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n>\n> Add ability to parse HTTP requests to the test-http-server test helper.\n>\n> Signed-off-by: Matthew John Cheetham \n> ---\n> t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-\n> 1 file changed, 173 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n> index 6cdac223a55..36f4a54fe6d 100644\n> --- a/t/helper/test-http-server.c\n> +++ b/t/helper/test-http-server.c\n> @@ -83,6 +83,42 @@ enum worker_result {\n> \tWR_HANGUP = 1<<1,\n> };\n> \n> +/*\n> + * Fields from a parsed HTTP request.\n> + */\n> +struct req {\n> +\tstruct strbuf start_line;\n> +\n> +\tconst char *method;\n> +\tconst char *http_version;\n> +\n> +\tstruct strbuf uri_path;\n> +\tstruct strbuf query_args;\n> +\n> +\tstruct string_list header_list;\n> +\tconst char *content_type;\n> +\tssize_t content_length;\n> +};\n> +\n> +#define REQ__INIT { \\\n> +\t.start_line = STRBUF_INIT, \\\n> +\t.uri_path = STRBUF_INIT, \\\n> +\t.query_args = STRBUF_INIT, \\\n> +\t.header_list = STRING_LIST_INIT_NODUP, \\\n> +\t.content_type = NULL, \\\n> +\t.content_length = -1 \\\n> +\t}\n\nStyle nit: Don't indent the trailing \"}\", and add a \",\" after the last\n\"content_length\" item.\n\nWe omit the comma by convention when there really should not be another\nitem, such as when we have a \"NULL\" terminator, here though we might add\na struct element at the end, so...\n\n> +static enum worker_result req__read(struct req *req, int fd)\n> +{\n> +\tstruct strbuf h = STRBUF_INIT;\n> +\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n> +\tint nr_start_line_fields;\n> +\tconst char *uri_target;\n> +\tconst char *query;\n> +\tchar *hp;\n> +\tconst char *hv;\n> +\n> +\tenum worker_result result = WR_OK;\n> +\n> +\t/*\n> +\t * Read line 0 of the request and split it into component parts:\n> +\t *\n> +\t * SP SP CRLF\n> +\t *\n> +\t */\n> +\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n> +\t\tresult = WR_OK | WR_HANGUP;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tstrbuf_trim_trailing_newline(&req->start_line);\n> +\n> +\tnr_start_line_fields = string_list_split(&start_line_fields,\n> +\t\t\t\t\t\t req->start_line.buf,\n> +\t\t\t\t\t\t ' ', -1);\n> +\tif (nr_start_line_fields != 3) {\n> +\t\tlogerror(\"could not parse request start-line '%s'\",\n> +\t\t\t req->start_line.buf);\n> +\t\tresult = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\treq->method = xstrdup(start_line_fields.items[0].string);\n> +\treq->http_version = xstrdup(start_line_fields.items[2].string);\n> +\n> +\turi_target = start_line_fields.items[1].string;\n> +\n> +\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n> +\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n> +\t\t\t req->http_version);\n> +\t\tresult = WR_IO_ERROR;\n> +\t\tgoto done;\n> +\t}\n> +\n> +\tquery = strchr(uri_target, '?');\n> +\n> +\tif (query) {\n> +\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n> +\t\tstrbuf_addstr(&req->query_args, query + 1);\n> +\t} else {\n> +\t\tstrbuf_addstr(&req->uri_path, uri_target);\n> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n> +\t}\n> +\n> +\t/*\n> +\t * Read the set of HTTP headers into a string-list.\n> +\t */\n> +\twhile (1) {\n> +\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n> +\t\t\tgoto done;\n> +\t\tstrbuf_trim_trailing_newline(&h);\n> +\n> +\t\tif (!h.len)\n> +\t\t\tgoto done; /* a blank line ends the header */\n> +\n> +\t\thp = strbuf_detach(&h, NULL);\n> +\t\tstring_list_append(&req->header_list, hp);\n> +\n> +\t\t/* also store common request headers as struct req members */\n> +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n> +\t\t\treq->content_type = hv;\n> +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n> +\t\t\treq->content_length = strtol(hv, &hp, 10);\n\nIn POSIX the \"ssize_t\" is not a \"this is the unsigned size_t\", but can\nbe a much smaller integer type (although in practice it tends to be the\nsigned version of \"size_t\".\n\nBut this seems like a potential overflow trap as a result, but sometimes\nwe need to live with \"ssize_t\".\n\nHowever, in this case it seems like we don't, as it seems the only\nreason you init'd this to -1 and then...\n\n> +\tif (trace2_is_enabled()) {\n> +\t\tstruct string_list_item *item;\n> +\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n> +\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n> +\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n> +\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n> +\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n> +\t\tif (req->content_length >= 0)\n> +\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n\n...use that \">= 0\" is to keep the state of \"did I assign to this above?\n\nSo firstly, shouldn't we error or something on a \"Content-Length: 0\",\nand aside from that wouldn't we just have a \"int have_content_length =\n0\" in this function that we'd then flip to 1?\n"},{"id":"470597","messageId":"230118.867cxkxece.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"a1ff55dd6e25aa39f14b494f482720edf7d1eabd.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 08/12] test-http-server: add simple authentication","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:21:29Z","receivedAt":"2023-01-18T12:17:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n\n> +static struct auth_module *get_auth_module(const char *scheme, int create)\n> +{\n> +\tint i;\n> +\tstruct auth_module *mod;\n> +\tfor (i = 0; i < auth_modules_nr; i++) {\n\nWe can use \"for (size_t i = 0\" syntax now, let's do that here to not mix\n\"size_t\" and \"int\" types needlessly.\n\n> +\tif (create) {\n> +\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n> +\t\tmod->scheme = xstrdup(scheme);\n> +\t\tmod->challenge_params = NULL;\n> +\t\tCALLOC_ARRAY(mod->tokens, 1);\n> +\t\tstring_list_init_dup(mod->tokens);\n\nDon't use CALLOC_ARRAY() if you're then going to use\nstring_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We\ndon't need to set the memory to 0, only to overwrite it entirely again.\n\n> +\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n> +\t\tauth_modules[auth_modules_nr++] = mod;\n\nI have not looked at the whole context here, but instead of:\n\n\tstruct auth_module {\n\t\tchar *scheme;\n\t\tchar *challenge_params;\n\t\tstruct string_list *tokens;\n\t};\n\nWhy not:\n\n\tstruct auth_module {\n\t\tchar *challenge_params;\n\t\tstruct string_list *tokens;\n\t};\n\nThen you could use a \"struct string_list\" for this, make the \"scheme\" be\nthe \"string\" member, and stick the remaining two fields in the \"util\",\nand thus save yourself the manual memory management etc.\n\n> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n> +{\n> +\tenum auth_result result = AUTH_UNKNOWN;\n> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n> +\tstruct auth_module *mod;\n> +\n> +\tstruct string_list_item *hdr;\n> +\tstruct string_list_item *token;\n> +\tconst char *v;\n> +\tstruct strbuf **split = NULL;\n> +\tint i;\n> +\tchar *challenge;\n> +\n> +\t/*\n> +\t * Check all auth modules and try to validate the request.\n> +\t * The first Authorization header that matches a known auth module\n> +\t * scheme will be consulted to either approve or deny the request.\n> +\t * If no module is found, or if there is no valid token, then 401 error.\n> +\t * Otherwise, only permit the request if anonymous auth is enabled.\n> +\t * It's atypical for user agents/clients to send multiple Authorization\n> +\t * headers, but not explicitly forbidden or defined.\n> +\t */\n> +\tfor_each_string_list_item(hdr, &req->header_list) {\n> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n> +\t\t\tif (!split[0] || !split[1]) continue;\n> +\n> +\t\t\t/* trim trailing space ' ' */\n> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n> +\n> +\t\t\tmod = get_auth_module(split[0]->buf, 0);\n> +\t\t\tif (mod) {\n> +\t\t\t\tresult = AUTH_DENY;\n> +\n> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n> +\t\t\t\t\t\tbreak;\n> +\t\t\t\t\t}\n> +\t\t\t\t}\n> +\n> +\t\t\t\tgoto done;\n\nSometimes we need a strbuf_split_str, but in this case couldn't you use\nthe in-place \"struct string_list\" variant of that instead, and just\ncarry a \"size_t len\" here for it, which you'd then pass to\nget_auth_module() (which this commit adds)?\n\nAlso, you \"split\" in the loop, but...\n\n> +\tstrbuf_list_free(split);\n...only free() the last one here, isn't this leaking?\n\n> +static int split_auth_param(const char *str, char **scheme, char **val)\n> +{\n> +\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n> +\n> +\tif (!p[0])\n> +\t\treturn -1;\n> +\n> +\t/* trim trailing ':' */\n> +\tif (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')\n\nDon't compare unsigned length fields to \"> 0\", just do \"if (len &&\n....)\".\n\nAlso, maybe I'm just groggy today, but how do we have a trailing \":\" if\nwe just split on \":\", and with a limit such that...\n\n> +\tif (p[1])\n> +\t\t*val = strbuf_detach(p[1], NULL);\n\n...we have an item after that?\n\n\n> +static int read_auth_config(const char *name, const char *val, void *data)\n> +{\n> +\tint ret = 0;\n> +\tchar *scheme = NULL;\n\nDon't init this to NULL, instead the split_auth_param() return value\nshould be trusted, the compiler will then help us catch errors, no?\n\n> +\tchar *token = NULL;\n> +\tchar *challenge = NULL;\n\nIn this case it *is* needed though, as the function will return\nnon-errors, but *maybe* give us the second out parameter.\n\nFor such a function though, isn't just assigning \"*second_param = NULL\"\nat the start of it less of a \"running with scissors\" pattern?\n\n> +\tstruct auth_module *mod = NULL;\n\nThis NULL assignment can be dropped, we assign to it below\nunconditionally before using it.\n\n> +\n> +\tif (!strcmp(name, \"auth.challenge\")) {\n> +\t\tif (split_auth_param(val, &scheme, &challenge)) {\n> +\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\n> +\t\tmod = get_auth_module(scheme, 1);\n> +\n> +\t\t/* Replace any existing challenge parameters */\n> +\t\tfree(mod->challenge_params);\n> +\t\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n> +\t} else if (!strcmp(name, \"auth.token\")) {\n> +\t\tif (split_auth_param(val, &scheme, &token)) {\n> +\t\t\tret = error(\"invalid auth token '%s'\", val);\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\n> +\t\tmod = get_auth_module(scheme, 1);\n> +\n> +\t\t/*\n> +\t\t * Append to set of valid tokens unless an empty token value\n> +\t\t * is provided, then clear the existing list.\n> +\t\t */\n> +\t\tif (token)\n> +\t\t\tstring_list_append(mod->tokens, token);\n> +\t\telse\n> +\t\t\tstring_list_clear(mod->tokens, 1);\n> +\t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n> +\t\tallow_anonymous = git_config_bool(name, val);\n> +\t} else {\n> +\t\twarning(\"unknown auth config '%s'\", name);\n> +\t}\n> +\n> +cleanup:\n> +\tfree(scheme);\n> +\tfree(token);\n> +\tfree(challenge);\n> +\n> +\treturn ret;\n> +}\n> +\n"},{"id":"470598","messageId":"230118.863588xeat.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"cc9a220ed1f12aef2f4df940e71adc1fad917a6b.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:38:34Z","receivedAt":"2023-01-18T12:17:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n>\n> Replace direct multiplication of two size_t parameters in curl response\n> stream handling callback functions with `st_mult` to guard against\n> overflows.\n>\n> Signed-off-by: Matthew John Cheetham \n> ---\n> http.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/http.c b/http.c\n> index 8a5ba3f4776..a2a80318bb2 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n> \n> size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> {\n> -\tsize_t size = eltsize * nmemb;\n> +\tsize_t size = st_mult(eltsize, nmemb);\n> \tstruct buffer *buffer = buffer_;\n> \n> \tif (size > buffer->buf.len - buffer->posn)\n> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n> \n> size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> {\n> -\tsize_t size = eltsize * nmemb;\n> +\tsize_t size = st_mult(eltsize, nmemb);\n> \tstruct strbuf *buffer = buffer_;\n> \n> \tstrbuf_add(buffer, ptr, size);\n\nThis is a really worthwhile fix, but shouldn't this be split into its\nown stand-alone patch? It applies on \"master\", and seems like something\nthat's a good idea outside of this \"test-http-server\" topic.\n"},{"id":"470599","messageId":"230118.86y1q0vzhh.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"bc1ac8d3eb3ac6e1161f6b6b67343874c10cd14d.1674012618.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T11:42:01Z","receivedAt":"2023-01-18T12:24:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n\n> +\tstrbuf_add(&buf, ptr, size);\n> +\n> +\t/* Strip the CRLF that should be present at the end of each field */\n> +\tstrbuf_trim_trailing_newline(&buf);\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n> +\t\twhile (isspace(*val))\n> +\t\t\tval++;\n> +\n> +\t\tstrvec_push(values, val);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> [...]\n> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n> +\t\t/* Trim leading whitespace from this continuation hdr line. */\n> +\t\tstrbuf_ltrim(&buf);\n\n\nThe mixture of this isspace() loop and then strbuf_ltrim() seems odd,\nwhy not stick with the strbuf API?\n\nI.e. after skip_iprefix() strbuf_splice() the start of the string away,\nthen use strbuf_ltrim() in the first \"if\" branch here?\n\nLikewise this is open-coding the \"isspace\" in strbuf_ltrim() for the\nsecond \"if\". Maybe run the strbuf_ltrim() unconditionally, save away the\nlength before, and then:\n\n\tif (http_auth.header_is_last_match && prev_len != buf.len) { ...\n\n?\n"},{"id":"470651","messageId":"aa8abc8d-284b-b87e-f594-27ee40cc4bec@github.com","threadId":"58425","inReplyTo":"230118.863588xeat.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-18T17:28:46Z","receivedAt":"2023-01-18T17:28:55Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>>\n>> Replace direct multiplication of two size_t parameters in curl response\n>> stream handling callback functions with `st_mult` to guard against\n>> overflows.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> http.c | 4 ++--\n>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/http.c b/http.c\n>> index 8a5ba3f4776..a2a80318bb2 100644\n>> --- a/http.c\n>> +++ b/http.c\n>> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n>> \n>> size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>> {\n>> -\tsize_t size = eltsize * nmemb;\n>> +\tsize_t size = st_mult(eltsize, nmemb);\n>> \tstruct buffer *buffer = buffer_;\n>> \n>> \tif (size > buffer->buf.len - buffer->posn)\n>> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n>> \n>> size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>> {\n>> -\tsize_t size = eltsize * nmemb;\n>> +\tsize_t size = st_mult(eltsize, nmemb);\n>> \tstruct strbuf *buffer = buffer_;\n>> \n>> \tstrbuf_add(buffer, ptr, size);\n> \n> This is a really worthwhile fix, but shouldn't this be split into its\n> own stand-alone patch? It applies on \"master\", and seems like something\n> that's a good idea outside of this \"test-http-server\" topic.\n\nWhile it's this change *can* stand alone, please keep in mind that\nsuggestions like this (recommending a series be split and resubmitted) can\nbe highly disruptive to the in-flight topic and the original contributor.\n\nMonitoring and iterating on multiple series at once is time-consuming for\nthe contributor and reviewers, and often (although not in this case) it\ncreates a dependency of one series on another, which comes with a cost to\nthe maintainer's time. Not to say those recommendations should never be made\n(e.g. in a clearly too-long series early in its review cycle, or when\ncertain patches lead to excessive context switching while reviewing), just\nthat they should be made more carefully, with consideration for the time of\nother contributors.\n\nSo, with that in mind, I don't think this patch is critical enough to\nseparate into an independent submission, and (subjectively) it does not\ndisrupt the flow of this series.\n\n"},{"id":"470682","messageId":"230119.864jsnwhxm.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"aa8abc8d-284b-b87e-f594-27ee40cc4bec@github.com","subject":"Re: [PATCH v6 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-01-18T23:16:41Z","receivedAt":"2023-01-18T23:18:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 18 2023, Victoria Dye wrote:\n\n> Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n>> \n>>> From: Matthew John Cheetham \n>>>\n>>> Replace direct multiplication of two size_t parameters in curl response\n>>> stream handling callback functions with `st_mult` to guard against\n>>> overflows.\n>>>\n>>> Signed-off-by: Matthew John Cheetham \n>>> ---\n>>> http.c | 4 ++--\n>>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>>\n>>> diff --git a/http.c b/http.c\n>>> index 8a5ba3f4776..a2a80318bb2 100644\n>>> --- a/http.c\n>>> +++ b/http.c\n>>> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n>>> \n>>> size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>>> {\n>>> -\tsize_t size = eltsize * nmemb;\n>>> +\tsize_t size = st_mult(eltsize, nmemb);\n>>> \tstruct buffer *buffer = buffer_;\n>>> \n>>> \tif (size > buffer->buf.len - buffer->posn)\n>>> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n>>> \n>>> size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>>> {\n>>> -\tsize_t size = eltsize * nmemb;\n>>> +\tsize_t size = st_mult(eltsize, nmemb);\n>>> \tstruct strbuf *buffer = buffer_;\n>>> \n>>> \tstrbuf_add(buffer, ptr, size);\n>> \n>> This is a really worthwhile fix, but shouldn't this be split into its\n>> own stand-alone patch? It applies on \"master\", and seems like something\n>> that's a good idea outside of this \"test-http-server\" topic.\n>\n> While it's this change *can* stand alone, please keep in mind that\n> suggestions like this (recommending a series be split and resubmitted) can\n> be highly disruptive to the in-flight topic and the original contributor.\n>\n> Monitoring and iterating on multiple series at once is time-consuming for\n> the contributor and reviewers, and often (although not in this case) it\n> creates a dependency of one series on another, which comes with a cost to\n> the maintainer's time. Not to say those recommendations should never be made\n> (e.g. in a clearly too-long series early in its review cycle, or when\n> certain patches lead to excessive context switching while reviewing), just\n> that they should be made more carefully, with consideration for the time of\n> other contributors.\n>\n> So, with that in mind, I don't think this patch is critical enough to\n> separate into an independent submission, and (subjectively) it does not\n> disrupt the flow of this series.\n\nYes, I take your general point, it's not always the right thing,\nsometimes a while-at-it cleanup is better than a split-out etc.\n\nIn this case the split-out seemed like it wouldn't create a dependency\nbetween topics, as the rest of the series didn't rely on the overflow\nsanity check being added, it's just a good idea to do it in general.\n\n"},{"id":"470805","messageId":"AS2PR03MB9815437B9BE892A9F0AC564BC0C59@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230118.86k01kxfr7.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 04/12] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-20T22:05:12Z","receivedAt":"2023-01-20T22:05:24Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-18 03:04, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>> [...]\n>> +enum worker_result {\n>> +\t/*\n>> +\t * Operation successful.\n>> +\t * Caller *might* keep the socket open and allow keep-alive.\n>> +\t */\n>> +\tWR_OK = 0,\n>> [...]\n>> +\tenum worker_result wr = WR_OK;\n>> +\n>> +\tif (client_addr)\n>> +\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n>> +\n>> +\tset_keep_alive(0, logerror);\n>> +\n>> +\twhile (1) {\n>> +\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n>> +\t\t\tlogerror(\"unable to write response\");\n>> +\t\t\twr = WR_IO_ERROR;\n>> +\t\t}\n>> +\n>> +\t\tif (wr != WR_OK)\n>> +\t\t\tbreak;\n>> +\t}\n>> +\n>> +\tclose(STDIN_FILENO);\n>> +\tclose(STDOUT_FILENO);\n>> +\n>> +\treturn !!(wr & WR_IO_ERROR);\n>> +}\n> \n> We have cases where we assign \"0\" to a bitfield-looking structure like\n> this, but only in cases where we're planning to use it as a boolean too.\n> \n> Or, in other cases where we want some to be explicitly <-1.\n> \n> Here though we're adding a mixed \"OK\" and error use, which seems a bit\n> odd. Shouldn't we pick one or the other?\n\nYou make a fair point about bitfields vs simple integer values. This was a\nholdover from previous early hacking on this work where I had the bitfield\nserve as a way to communicate the aspects of \"does this count as an error?\"\nand \"should we close the connection?\".\n\nUpon second thought, I think just simple integer values would be fine as\nreally only an \"OK\" and \"HANGUP\" are non-errors (the latter being the case\nthat the client gracefully ended the connection without an error and we\nshould exit).\n\nCheck for my next iteration for a rework on these `worker_result` values.\n\n> So far (maybe in later commits?) nothing uses WR_HANGUP, and oddly we\n> also use the bitfield-looking thing as a return value from main()....\n\nWe don't use the `enum worker_result` values as a return from `main`. We only\never return 0 or 1 as we `return worker()` from `main`, and the only `return`\nfrom `worker()` is `!!(wr & WR_IO_ERROR)` - 1 if we have `WR_IO_ERROR` set,\notherwise 0.\n\nThanks,\nMatthew\n"},{"id":"470806","messageId":"AS2PR03MB9815D62E17111924A002A714C0C59@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230118.86fsc8xffg.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 05/12] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-20T22:05:20Z","receivedAt":"2023-01-20T22:05:36Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-18 03:07, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>>\n>> Introduce a function to the test-http-server test helper to write more\n>> full and valid HTTP error responses, including all the standard response\n>> headers like `Server` and `Date`.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> t/helper/test-http-server.c | 58 +++++++++++++++++++++++++++++++++----\n>> 1 file changed, 53 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n>> index 11071b1dd89..6cdac223a55 100644\n>> --- a/t/helper/test-http-server.c\n>> +++ b/t/helper/test-http-server.c\n>> @@ -83,9 +83,59 @@ enum worker_result {\n>> \tWR_HANGUP = 1<<1,\n>> };\n> \n> ...okey, this is the commit that makes use of WR_HANGUP. Whatever else\n> we do, let's then squash that addition into this change.\n> \n>> +static enum worker_result send_http_error(\n>> +\tint fd,\n>> +\tint http_code, const char *http_code_name,\n>> +\tint retry_after_seconds, struct string_list *response_headers,\n>> +\tenum worker_result wr_in)\n> \n> In general in this series you are mis-indenting argument lists. Our\n> usual style is to wrap at 79 characters, then to align (with tabs and\n> spaces) with the \"(\".\n> \n> So in this case:\n> \n> static enum worker_result send_http_error(int fd, int http_code,\n> \t\t\t\t\t const char *http_code_name,\n> \t\t\t\t\t int retry_after_seconds,\n> \t\t\t\t\t struct string_list *response_headers,\n> \t\t\t\t\t enum worker_result wr_in)\n> \n>> +{\n>> +\tstruct strbuf response_header = STRBUF_INIT;\n>> +\tstruct strbuf response_content = STRBUF_INIT;\n>> +\tstruct string_list_item *h;\n>> +\tenum worker_result wr;\n>> +\n>> +\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n>> +\t\t http_code, http_code_name);\n> \n> \n> Ditto here, where \"http_code\" should go on the preceding line...\n> \n>> +\tif (retry_after_seconds > 0)\n>> +\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n>> +\t\t\t retry_after_seconds);\n>> +\n>> +\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n> \n> ...and here there's a lack of such wrapping...\n> \n>> +\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n>> +\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n>> +\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n>> +\tif (retry_after_seconds > 0)\n>> +\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n>> +\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n>> +\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n> \n> ...here you're adding strange whitespace at the start of an argument list...\n> \n>> +\tif (response_headers)\n>> +\t\tfor_each_string_list_item(h, response_headers)\n>> +\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n>> +\tstrbuf_addstr(&response_header, \"\\r\\n\");\n\nArgh! Thanks again for catching these. I shall address them.\n\n> To comment on the code a bit, this whole thing would be more readable\n> IMO if your own headers were also a \"struct string_list\". Yes we'd waste\n> a bit more memory, but in this case that's fine..\n> \n> I.e. don't add the \"\\r\\n\" every time, just:\n> \n> \tstring_list_append(&headers, \"Cache-Control: private\");\n> \n> etc.\n> \n> Then at the end you'd do e.g.:\n> \n> \tadd_headers(&buf, &headers);\n> \tif (response_headers)\n> \t\tadd_headers(&buf, response_headers);\n> \n> Where the add_headers() is a trivial \"static\" helper which does that\n> for_each_string_list_item() loop above.\n\nIn reality this only helps simplify the code in the case of a simple static\nheader like \"Cache-Control: private\". There's no `string_list_appendf` or\nsimilar where I need to append a header that contains dynamic information\n(date, content length, etc).\n\nBuilding the `strbuf` directly, and specifying the CRLF seems a lot easier IMO.\n\n>> \twhile (1) {\n>> -\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n>> -\t\t\tlogerror(\"unable to write response\");\n>> -\t\t\twr = WR_IO_ERROR;\n>> -\t\t}\n>> +\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n>> +\t\t\t\t NULL, WR_OK | WR_HANGUP);\n> \n> This *does* use correct wrapping & indenation for a continuing argument\n> list.\n\n\nThanks,\nMatthew\n"},{"id":"470807","messageId":"AS2PR03MB98154B064352CEDF76E2924FC0C59@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230118.86bkmwxf6e.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 06/12] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-20T22:05:28Z","receivedAt":"2023-01-20T22:05:42Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-18 03:14, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>>\n>> Add ability to parse HTTP requests to the test-http-server test helper.\n>>\n>> Signed-off-by: Matthew John Cheetham \n>> ---\n>> t/helper/test-http-server.c | 175 +++++++++++++++++++++++++++++++++++-\n>> 1 file changed, 173 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\n>> index 6cdac223a55..36f4a54fe6d 100644\n>> --- a/t/helper/test-http-server.c\n>> +++ b/t/helper/test-http-server.c\n>> @@ -83,6 +83,42 @@ enum worker_result {\n>> \tWR_HANGUP = 1<<1,\n>> };\n>> \n>> +/*\n>> + * Fields from a parsed HTTP request.\n>> + */\n>> +struct req {\n>> +\tstruct strbuf start_line;\n>> +\n>> +\tconst char *method;\n>> +\tconst char *http_version;\n>> +\n>> +\tstruct strbuf uri_path;\n>> +\tstruct strbuf query_args;\n>> +\n>> +\tstruct string_list header_list;\n>> +\tconst char *content_type;\n>> +\tssize_t content_length;\n>> +};\n>> +\n>> +#define REQ__INIT { \\\n>> +\t.start_line = STRBUF_INIT, \\\n>> +\t.uri_path = STRBUF_INIT, \\\n>> +\t.query_args = STRBUF_INIT, \\\n>> +\t.header_list = STRING_LIST_INIT_NODUP, \\\n>> +\t.content_type = NULL, \\\n>> +\t.content_length = -1 \\\n>> +\t}\n> \n> Style nit: Don't indent the trailing \"}\", and add a \",\" after the last\n> \"content_length\" item.\n> \n> We omit the comma by convention when there really should not be another\n> item, such as when we have a \"NULL\" terminator, here though we might add\n> a struct element at the end, so...\n\nSure.\n\n>> +static enum worker_result req__read(struct req *req, int fd)\n>> +{\n>> +\tstruct strbuf h = STRBUF_INIT;\n>> +\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n>> +\tint nr_start_line_fields;\n>> +\tconst char *uri_target;\n>> +\tconst char *query;\n>> +\tchar *hp;\n>> +\tconst char *hv;\n>> +\n>> +\tenum worker_result result = WR_OK;\n>> +\n>> +\t/*\n>> +\t * Read line 0 of the request and split it into component parts:\n>> +\t *\n>> +\t * SP SP CRLF\n>> +\t *\n>> +\t */\n>> +\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n>> +\t\tresult = WR_OK | WR_HANGUP;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\tstrbuf_trim_trailing_newline(&req->start_line);\n>> +\n>> +\tnr_start_line_fields = string_list_split(&start_line_fields,\n>> +\t\t\t\t\t\t req->start_line.buf,\n>> +\t\t\t\t\t\t ' ', -1);\n>> +\tif (nr_start_line_fields != 3) {\n>> +\t\tlogerror(\"could not parse request start-line '%s'\",\n>> +\t\t\t req->start_line.buf);\n>> +\t\tresult = WR_IO_ERROR;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\treq->method = xstrdup(start_line_fields.items[0].string);\n>> +\treq->http_version = xstrdup(start_line_fields.items[2].string);\n>> +\n>> +\turi_target = start_line_fields.items[1].string;\n>> +\n>> +\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n>> +\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n>> +\t\t\t req->http_version);\n>> +\t\tresult = WR_IO_ERROR;\n>> +\t\tgoto done;\n>> +\t}\n>> +\n>> +\tquery = strchr(uri_target, '?');\n>> +\n>> +\tif (query) {\n>> +\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n>> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n>> +\t\tstrbuf_addstr(&req->query_args, query + 1);\n>> +\t} else {\n>> +\t\tstrbuf_addstr(&req->uri_path, uri_target);\n>> +\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n>> +\t}\n>> +\n>> +\t/*\n>> +\t * Read the set of HTTP headers into a string-list.\n>> +\t */\n>> +\twhile (1) {\n>> +\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n>> +\t\t\tgoto done;\n>> +\t\tstrbuf_trim_trailing_newline(&h);\n>> +\n>> +\t\tif (!h.len)\n>> +\t\t\tgoto done; /* a blank line ends the header */\n>> +\n>> +\t\thp = strbuf_detach(&h, NULL);\n>> +\t\tstring_list_append(&req->header_list, hp);\n>> +\n>> +\t\t/* also store common request headers as struct req members */\n>> +\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n>> +\t\t\treq->content_type = hv;\n>> +\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n>> +\t\t\treq->content_length = strtol(hv, &hp, 10);\n> \n> In POSIX the \"ssize_t\" is not a \"this is the unsigned size_t\", but can\n> be a much smaller integer type (although in practice it tends to be the\n> signed version of \"size_t\".\n> \n> But this seems like a potential overflow trap as a result, but sometimes\n> we need to live with \"ssize_t\".\n> \n> However, in this case it seems like we don't, as it seems the only\n> reason you init'd this to -1 and then...\n> \n>> +\tif (trace2_is_enabled()) {\n>> +\t\tstruct string_list_item *item;\n>> +\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n>> +\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n>> +\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n>> +\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n>> +\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n>> +\t\tif (req->content_length >= 0)\n>> +\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n> \n> ...use that \">= 0\" is to keep the state of \"did I assign to this above?\n> \n> So firstly, shouldn't we error or something on a \"Content-Length: 0\",\n> and aside from that wouldn't we just have a \"int have_content_length =\n> 0\" in this function that we'd then flip to 1?\n\nIt seems like the perfect type for such a non-zero-or-error size value; from\nPOSIX specifications[1]:\n\n> ...\n> size_t\n>\tUsed for sizes of objects.\n> ssize_t\n>\tUsed for a count of bytes or an error indication.\n> ...\n\nBut you're probably right here that `ssize_t` isn't that suitable in practice\ndue to the comically low minimum size of the `SSIZE_MAX` (2^15 I believe).\n\nRFC 9110 §8.6 [2] addresses the `Content-Length` HTTP header and says that its\nvalue should be non-negative, but also have no upper bound; we're gonna have to\nset at least some practical limit.\n\nLibcurl handles this by writing it's own parsing function that's good up to\na max 64-bit integer value [3][4].\n\nGiven this is for a test helper and only going to be receiving data from tests,\nI propose just using something like `uintmax_t` and storing a bit with\n`unsigned has_content_length:1;` to show if we actually got a header in the\nrequest or not.\n\nThanks,\nMatthew\n\n[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/sys_types.h.html\n[2] https://www.rfc-editor.org/rfc/rfc9110.html#section-8.6\n[3] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/http.c#L3348-L3349\n[4] https://github.com/curl/curl/blob/6113dec2a829d4ab766428ccca9535b7a5efd012/lib/strtoofft.c#L214-L218\n"},{"id":"470808","messageId":"AS2PR03MB9815A64884689C30112BCA11C0C59@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230118.867cxkxece.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 08/12] test-http-server: add simple authentication","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-20T22:05:38Z","receivedAt":"2023-01-20T22:05:54Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-18 03:21, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n> \n>> +static struct auth_module *get_auth_module(const char *scheme, int create)\n>> +{\n>> +\tint i;\n>> +\tstruct auth_module *mod;\n>> +\tfor (i = 0; i < auth_modules_nr; i++) {\n> \n> We can use \"for (size_t i = 0\" syntax now, let's do that here to not mix\n> \"size_t\" and \"int\" types needlessly.\n\nYep!\n\n>> +\tif (create) {\n>> +\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n>> +\t\tmod->scheme = xstrdup(scheme);\n>> +\t\tmod->challenge_params = NULL;\n>> +\t\tCALLOC_ARRAY(mod->tokens, 1);\n>> +\t\tstring_list_init_dup(mod->tokens);\n> \n> Don't use CALLOC_ARRAY() if you're then going to use\n> string_list_init_dup() (which is good!), use ALLOC_ARRAY() instead. We\n> don't need to set the memory to 0, only to overwrite it entirely again.\n\nSure.\n\n>> +\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n>> +\t\tauth_modules[auth_modules_nr++] = mod;\n> \n> I have not looked at the whole context here, but instead of:\n> \n> \tstruct auth_module {\n> \t\tchar *scheme;\n> \t\tchar *challenge_params;\n> \t\tstruct string_list *tokens;\n> \t};\n> \n> Why not:\n> \n> \tstruct auth_module {\n> \t\tchar *challenge_params;\n> \t\tstruct string_list *tokens;\n> \t};\n> \n> Then you could use a \"struct string_list\" for this, make the \"scheme\" be\n> the \"string\" member, and stick the remaining two fields in the \"util\",\n> and thus save yourself the manual memory management etc.\n\nI looked at this, but this then means being more careful when looping over\ndifferent `struct auth_module`s to keep the current 'scheme' and `*mod` in\nsync/together. Just feels like overkill right now.\n\n>> +static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n>> +{\n>> +\tenum auth_result result = AUTH_UNKNOWN;\n>> +\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n>> +\tstruct auth_module *mod;\n>> +\n>> +\tstruct string_list_item *hdr;\n>> +\tstruct string_list_item *token;\n>> +\tconst char *v;\n>> +\tstruct strbuf **split = NULL;\n>> +\tint i;\n>> +\tchar *challenge;\n>> +\n>> +\t/*\n>> +\t * Check all auth modules and try to validate the request.\n>> +\t * The first Authorization header that matches a known auth module\n>> +\t * scheme will be consulted to either approve or deny the request.\n>> +\t * If no module is found, or if there is no valid token, then 401 error.\n>> +\t * Otherwise, only permit the request if anonymous auth is enabled.\n>> +\t * It's atypical for user agents/clients to send multiple Authorization\n>> +\t * headers, but not explicitly forbidden or defined.\n>> +\t */\n>> +\tfor_each_string_list_item(hdr, &req->header_list) {\n>> +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n>> +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n>> +\t\t\tif (!split[0] || !split[1]) continue;\n>> +\n>> +\t\t\t/* trim trailing space ' ' */\n>> +\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n>> +\n>> +\t\t\tmod = get_auth_module(split[0]->buf, 0);\n>> +\t\t\tif (mod) {\n>> +\t\t\t\tresult = AUTH_DENY;\n>> +\n>> +\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n>> +\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n>> +\t\t\t\t\t\tresult = AUTH_ALLOW;\n>> +\t\t\t\t\t\tbreak;\n>> +\t\t\t\t\t}\n>> +\t\t\t\t}\n>> +\n>> +\t\t\t\tgoto done;\n> \n> Sometimes we need a strbuf_split_str, but in this case couldn't you use\n> the in-place \"struct string_list\" variant of that instead, and just\n> carry a \"size_t len\" here for it, which you'd then pass to\n> get_auth_module() (which this commit adds)?\n\n`get_auth_module` taking a scheme name as parameter is a more sensible, IMO,\nthan a `string_list` or `string_list_item` and an offset. Given this is a test\nhelper, performance also isn't a priority. Readability wins here I think.\n\n> Also, you \"split\" in the loop, but...\n> \n>> +\tstrbuf_list_free(split);\n> ...only free() the last one here, isn't this leaking?\n\nYes, it is. Will fix in next iteration.\n\n>> +static int split_auth_param(const char *str, char **scheme, char **val)\n>> +{\n>> +\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n>> +\n>> +\tif (!p[0])\n>> +\t\treturn -1;\n>> +\n>> +\t/* trim trailing ':' */\n>> +\tif (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')\n> \n> Don't compare unsigned length fields to \"> 0\", just do \"if (len &&\n> ....)\".\n\nSure!\n\n> Also, maybe I'm just groggy today, but how do we have a trailing \":\" if\n> we just split on \":\", and with a limit such that...\n> \n>> +\tif (p[1])\n>> +\t\t*val = strbuf_detach(p[1], NULL);\n> \n> ...we have an item after that?\n\nBecause that's how the `strbuf_split_str` function works. The comments\nin the header file even call that out. \"The substrings include the\nterminator\". From strbuf.h:\n\n/**\n * Split str (of length slen) at the specified terminator character.\n * Return a null-terminated array of pointers to strbuf objects\n * holding the substrings. The substrings include the terminator,\n * except for the last substring, which might be unterminated if the\n * original string did not end with a terminator. [cut] ...\n ...\n */\nstruct strbuf **strbuf_split_buf(const char *str, size_t len,\n\t\t\t\t int terminator, int max);\n\n>> +static int read_auth_config(const char *name, const char *val, void *data)\n>> +{\n>> +\tint ret = 0;\n>> +\tchar *scheme = NULL;\n> \n> Don't init this to NULL, instead the split_auth_param() return value\n> should be trusted, the compiler will then help us catch errors, no?\n> \n>> +\tchar *token = NULL;\n>> +\tchar *challenge = NULL;\n> \n> In this case it *is* needed though, as the function will return\n> non-errors, but *maybe* give us the second out parameter.\n> \n> For such a function though, isn't just assigning \"*second_param = NULL\"\n> at the start of it less of a \"running with scissors\" pattern?\n> \n>> +\tstruct auth_module *mod = NULL;\n> \n> This NULL assignment can be dropped, we assign to it below\n> unconditionally before using it.\n\nAll of these variables need to be initialised to NULL because not all\narms of the `if-elseif` chain assign to all of these variables, but\nwe always `free` all of them at the function exit.\n\nFor example,\n\nchar *scheme = NULL;\nchar *token = NULL;\nchar *challenge = NULL;\n...\n} else if (!strcmp(name, \"auth.allowanonymous\")) {\n\tallow_anonymous = git_config_bool(name, val);\n} else {\n...\nfree(scheme);\nfree(token);\nfree(challenge);\n\n>> +\n>> +\tif (!strcmp(name, \"auth.challenge\")) {\n>> +\t\tif (split_auth_param(val, &scheme, &challenge)) {\n>> +\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\n>> +\t\tmod = get_auth_module(scheme, 1);\n>> +\n>> +\t\t/* Replace any existing challenge parameters */\n>> +\t\tfree(mod->challenge_params);\n>> +\t\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n>> +\t} else if (!strcmp(name, \"auth.token\")) {\n>> +\t\tif (split_auth_param(val, &scheme, &token)) {\n>> +\t\t\tret = error(\"invalid auth token '%s'\", val);\n>> +\t\t\tgoto cleanup;\n>> +\t\t}\n>> +\n>> +\t\tmod = get_auth_module(scheme, 1);\n>> +\n>> +\t\t/*\n>> +\t\t * Append to set of valid tokens unless an empty token value\n>> +\t\t * is provided, then clear the existing list.\n>> +\t\t */\n>> +\t\tif (token)\n>> +\t\t\tstring_list_append(mod->tokens, token);\n>> +\t\telse\n>> +\t\t\tstring_list_clear(mod->tokens, 1);\n>> +\t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n>> +\t\tallow_anonymous = git_config_bool(name, val);\n>> +\t} else {\n>> +\t\twarning(\"unknown auth config '%s'\", name);\n>> +\t}\n>> +\n>> +cleanup:\n>> +\tfree(scheme);\n>> +\tfree(token);\n>> +\tfree(challenge);\n>> +\n>> +\treturn ret;\n>> +}\n>> +\n\nThanks,\nMatthew\n"},{"id":"470809","messageId":"AS2PR03MB98152D5781CC52065A71C43EC0C59@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230118.86y1q0vzhh.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v6 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-01-20T22:05:45Z","receivedAt":"2023-01-20T22:06:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-18 03:42, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Jan 18 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n> \n>> +\tstrbuf_add(&buf, ptr, size);\n>> +\n>> +\t/* Strip the CRLF that should be present at the end of each field */\n>> +\tstrbuf_trim_trailing_newline(&buf);\n>> +\n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n>> +\t\twhile (isspace(*val))\n>> +\t\t\tval++;\n>> +\n>> +\t\tstrvec_push(values, val);\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n>> [...]\n>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>> +\t\t/* Trim leading whitespace from this continuation hdr line. */\n>> +\t\tstrbuf_ltrim(&buf);\n> \n> \n> The mixture of this isspace() loop and then strbuf_ltrim() seems odd,\n> why not stick with the strbuf API?\n> \n> I.e. after skip_iprefix() strbuf_splice() the start of the string away,\n> then use strbuf_ltrim() in the first \"if\" branch here?\n\nYou mean like this?\n\n size_t size = st_mult(eltsize, nmemb);\n struct strvec *values = &http_auth.wwwauth_headers;\n struct strbuf buf = STRBUF_INIT;\n- const char *val;\n \n /*\n * Header lines may not come NULL-terminated from libcurl so we must\n@@ -216,11 +215,11 @@ static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n strbuf_trim_trailing_newline(&buf);\n \n /* Start of a new WWW-Authenticate header */\n- if (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n- while (isspace(*val))\n- val++;\n+ if (istarts_with(buf.buf, \"www-authenticate:\")) {\n+ strbuf_splice(&buf, 0, 17, NULL, 0);\n+ strbuf_ltrim(&buf);\n \n- strvec_push(values, val);\n+ strvec_push(values, buf.buf);\n http_auth.header_is_last_match = 1;\n goto exit;\n }\n\n\nI don't particularly like this given we're now introducing the 'magic' number\n17 that's the length of `www-authenticate:`, plus `strbuf_splice` is doing\na lot more work moving pieces of memory around rather than just producing\na new starting pointer to the start of the value (skipping leading whitespace).\n\n> Likewise this is open-coding the \"isspace\" in strbuf_ltrim() for the\n> second \"if\". Maybe run the strbuf_ltrim() unconditionally, save away the\n> length before, and then:\n> \n> \tif (http_auth.header_is_last_match && prev_len != buf.len) { ...\n> \n> ?\n\nThe suggestion of trimming and comparing lengths just makes a piece of code\nhandling a little-known edge case less immediately obvious in its intent in\nmy opinion. The current implementation of \"if starts with a single space\"\nmatches the definition of continuation header lines, rather than re-piecing\ntogether this from \"why are we trimming and comparing lengths?\"\nPerf-wise the current implementation is only adding one extra `isspace`\ncall which we're potentially about to do in a loop inside of `strbuf_ltrim`\nanyway. Plus, the common case will be a single space anyway.\n\nThanks,\nMatthew\n"},{"id":"470811","messageId":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v6.git.1674012618.gitgitgadget@gmail.com","subject":"[PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:38Z","receivedAt":"2023-01-20T22:09:05Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I introduce a new test helper test-http-server\nthat acts as a frontend to git-http-backend; a mini HTTP server sharing code\nwith git-daemon, with simple authentication configurable by a config file.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\n\nUpdates in v7\n=============\n\n * Address several whitespace and arg/param list alignment issues.\n\n * Rethink the test-http-helper worker-mode error and result enum to be more\n simple and more informative to the nature of the error.\n\n * Use uintmax_t to store the Content-Length of a request in the helper\n test-http-server. Maintain a bit flag to store if we received such a\n header.\n\n * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n in the test-http-server.\n\n * Add test case to cover request message parsing in test-http-server.\n\n * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n get_auth_module.\n\n * Correctly free the split strbufs created in the header parsing loop in\n test-http-server.\n\n * Avoid needless comparison > 0 for unsigned types.\n\n * Always set optional outputs to NULL if not present in test helper config\n value handling.\n\n * Remove an accidentally commented-out test cleanup line for one test case\n in t5556.\n\nMatthew John Cheetham (12):\n daemon: libify socket setup and option functions\n daemon: libify child process handling functions\n daemon: rename some esoteric/laboured terminology\n test-http-server: add stub HTTP server test helper\n test-http-server: add HTTP error response function\n test-http-server: add HTTP request parsing\n test-http-server: pass Git requests to http-backend\n test-http-server: add simple authentication\n test-http-server: add sending of arbitrary headers\n http: replace unsafe size_t multiplication with st_mult\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n Makefile | 2 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n credential.c | 12 +\n credential.h | 15 +\n daemon-utils.c | 286 +++++++++\n daemon-utils.h | 55 ++\n daemon.c | 306 +--------\n http.c | 98 ++-\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 943 ++++++++++++++++++++++++++++\n t/lib-credential-helper.sh | 27 +\n t/t5556-http-auth.sh | 463 ++++++++++++++\n 13 files changed, 1936 insertions(+), 302 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n create mode 100644 t/helper/test-http-server.c\n create mode 100644 t/lib-credential-helper.sh\n create mode 100755 t/t5556-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v7\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v7\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v6:\n\n 1: 74b0de14185 = 1: 74b0de14185 daemon: libify socket setup and option functions\n 2: b6ba344a671 = 2: b6ba344a671 daemon: libify child process handling functions\n 3: 9967401c972 = 3: 9967401c972 daemon: rename some esoteric/laboured terminology\n 4: d6e5e8825e8 ! 4: 17c890ee108 test-http-server: add stub HTTP server test helper\n @@ t/helper/test-http-server.c (new)\n +\n +/*\n + * The code in this section is used by \"worker\" instances to service\n -+ * a single connection from a client. The worker talks to the client\n -+ * on 0 and 1.\n ++ * a single connection from a client. The worker talks to the client\n ++ * on stdin and stdout.\n + */\n +\n +enum worker_result {\n @@ t/helper/test-http-server.c (new)\n +\t * Operation successful.\n +\t * Caller *might* keep the socket open and allow keep-alive.\n +\t */\n -+\tWR_OK = 0,\n ++\tWR_OK = 0,\n +\n +\t/*\n -+\t * Various errors while processing the request and/or the response.\n ++\t * Fatal error that is not recoverable.\n +\t * Close the socket and clean up.\n +\t * Exit child-process with non-zero status.\n +\t */\n -+\tWR_IO_ERROR = 1<<0,\n -+\n -+\t/*\n -+\t * Close the socket and clean up. Does not imply an error.\n -+\t */\n -+\tWR_HANGUP = 1<<1,\n ++\tWR_FATAL_ERROR = 1,\n +};\n +\n +static enum worker_result worker(void)\n @@ t/helper/test-http-server.c (new)\n +\twhile (1) {\n +\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n +\t\t\tlogerror(\"unable to write response\");\n -+\t\t\twr = WR_IO_ERROR;\n ++\t\t\twr = WR_FATAL_ERROR;\n +\t\t}\n +\n +\t\tif (wr != WR_OK)\n @@ t/helper/test-http-server.c (new)\n +\tclose(STDIN_FILENO);\n +\tclose(STDOUT_FILENO);\n +\n -+\treturn !!(wr & WR_IO_ERROR);\n ++\t/* Only WR_OK should result in a non-zero exit code */\n ++\treturn wr != WR_OK;\n +}\n +\n +static int max_connections = 32;\n 5: 79805f042b9 ! 5: 6e70e304cfe test-http-server: add HTTP error response function\n @@ Commit message\n \n ## t/helper/test-http-server.c ##\n @@ t/helper/test-http-server.c: enum worker_result {\n - \tWR_HANGUP = 1<<1,\n + \t * Exit child-process with non-zero status.\n + \t */\n + \tWR_FATAL_ERROR = 1,\n ++\n ++\t/*\n ++\t * Close the socket and clean up. Does not imply an error.\n ++\t */\n ++\tWR_HANGUP = 2,\n };\n \n -+static enum worker_result send_http_error(\n -+\tint fd,\n -+\tint http_code, const char *http_code_name,\n -+\tint retry_after_seconds, struct string_list *response_headers,\n -+\tenum worker_result wr_in)\n ++static enum worker_result send_http_error(int fd, int http_code,\n ++\t\t\t\t\t const char *http_code_name,\n ++\t\t\t\t\t int retry_after_seconds,\n ++\t\t\t\t\t struct string_list *response_headers,\n ++\t\t\t\t\t enum worker_result wr_in)\n +{\n +\tstruct strbuf response_header = STRBUF_INIT;\n +\tstruct strbuf response_content = STRBUF_INIT;\n +\tstruct string_list_item *h;\n +\tenum worker_result wr;\n +\n -+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\",\n -+\t\t http_code, http_code_name);\n ++\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\", http_code,\n ++\t\t http_code_name);\n ++\n +\tif (retry_after_seconds > 0)\n +\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n +\t\t\t retry_after_seconds);\n +\n -+\tstrbuf_addf (&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code, http_code_name);\n ++\tstrbuf_addf(&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code,\n ++\t\t http_code_name);\n +\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n -+\tstrbuf_addstr(&response_header,\t\"Content-Type: text/plain\\r\\n\");\n -+\tstrbuf_addf (&response_header,\t\"Content-Length: %d\\r\\n\", (int)response_content.len);\n ++\tstrbuf_addstr(&response_header, \"Content-Type: text/plain\\r\\n\");\n ++\tstrbuf_addf(&response_header, \"Content-Length: %\"PRIuMAX\"\\r\\n\",\n ++\t\t (uintmax_t)response_content.len);\n ++\n +\tif (retry_after_seconds > 0)\n -+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\", retry_after_seconds);\n -+\tstrbuf_addf( &response_header,\t\"Server: test-http-server/%s\\r\\n\", git_version_string);\n -+\tstrbuf_addf( &response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0, DATE_MODE(RFC2822)));\n ++\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\",\n ++\t\t\t retry_after_seconds);\n ++\n ++\tstrbuf_addf(&response_header, \"Server: test-http-server/%s\\r\\n\",\n ++\t\t git_version_string);\n ++\tstrbuf_addf(&response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0,\n ++\t\t DATE_MODE(RFC2822)));\n ++\n +\tif (response_headers)\n +\t\tfor_each_string_list_item(h, response_headers)\n +\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n @@ t/helper/test-http-server.c: enum worker_result {\n +\n +\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n +\t\tlogerror(\"unable to write response header\");\n -+\t\twr = WR_IO_ERROR;\n ++\t\twr = WR_FATAL_ERROR;\n +\t\tgoto done;\n +\t}\n +\n +\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n +\t\tlogerror(\"unable to write response content body\");\n -+\t\twr = WR_IO_ERROR;\n ++\t\twr = WR_FATAL_ERROR;\n +\t\tgoto done;\n +\t}\n +\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n \twhile (1) {\n -\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n -\t\t\tlogerror(\"unable to write response\");\n --\t\t\twr = WR_IO_ERROR;\n +-\t\t\twr = WR_FATAL_ERROR;\n -\t\t}\n +\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n -+\t\t\t\t NULL, WR_OK | WR_HANGUP);\n ++\t\t\t\t NULL, WR_HANGUP);\n \n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n +@@ t/helper/test-http-server.c: static enum worker_result worker(void)\n + \tclose(STDIN_FILENO);\n + \tclose(STDOUT_FILENO);\n + \n +-\t/* Only WR_OK should result in a non-zero exit code */\n +-\treturn wr != WR_OK;\n ++\t/* Only WR_OK and WR_HANGUP should result in a non-zero exit code */\n ++\treturn wr != WR_OK && wr != WR_HANGUP;\n + }\n + \n + static int max_connections = 32;\n 6: 252098db219 ! 6: 43f1cdcbb82 test-http-server: add HTTP request parsing\n @@ Commit message\n test-http-server: add HTTP request parsing\n \n Add ability to parse HTTP requests to the test-http-server test helper.\n + Introduce `struct req` to store request information including:\n +\n + * HTTP method & version\n + * Request path and query parameters\n + * Headers\n + * Content type and length (from `Content-Type` and `-Length` headers)\n +\n + Failure to parse the request results in a 400 Bad Request response to\n + the client. Note that we're not trying to support all possible requests\n + here, but just enough to exercise all code under test.\n \n Signed-off-by: Matthew John Cheetham \n \n ## t/helper/test-http-server.c ##\n @@ t/helper/test-http-server.c: enum worker_result {\n - \tWR_HANGUP = 1<<1,\n - };\n - \n + \t * Close the socket and clean up. Does not imply an error.\n + \t */\n + \tWR_HANGUP = 2,\n ++\n ++\t/*\n ++\t * Unexpected request message or error in request parsing.\n ++\t * Respond with an 400 error. Close the socket and cleanup.\n ++\t * Exit child-process with a non-zero status.\n ++\t */\n ++\tWR_CLIENT_ERROR = 3,\n ++};\n ++\n +/*\n + * Fields from a parsed HTTP request.\n + */\n @@ t/helper/test-http-server.c: enum worker_result {\n +\n +\tstruct string_list header_list;\n +\tconst char *content_type;\n -+\tssize_t content_length;\n -+};\n -+\n ++\tuintmax_t content_length;\n ++\tunsigned has_content_length:1;\n + };\n + \n +#define REQ__INIT { \\\n +\t.start_line = STRBUF_INIT, \\\n +\t.uri_path = STRBUF_INIT, \\\n +\t.query_args = STRBUF_INIT, \\\n +\t.header_list = STRING_LIST_INIT_NODUP, \\\n +\t.content_type = NULL, \\\n -+\t.content_length = -1 \\\n -+\t}\n ++\t.content_length = 0, \\\n ++\t.has_content_length = 0, \\\n ++}\n +\n +static void req__release(struct req *req)\n +{\n @@ t/helper/test-http-server.c: enum worker_result {\n +\tstring_list_clear(&req->header_list, 0);\n +}\n +\n - static enum worker_result send_http_error(\n - \tint fd,\n - \tint http_code, const char *http_code_name,\n + static enum worker_result send_http_error(int fd, int http_code,\n + \t\t\t\t\t const char *http_code_name,\n + \t\t\t\t\t int retry_after_seconds,\n @@ t/helper/test-http-server.c: done:\n \treturn wr;\n }\n @@ t/helper/test-http-server.c: done:\n +\t *\n +\t */\n +\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n -+\t\tresult = WR_OK | WR_HANGUP;\n ++\t\tresult = WR_HANGUP;\n +\t\tgoto done;\n +\t}\n +\n @@ t/helper/test-http-server.c: done:\n +\tif (nr_start_line_fields != 3) {\n +\t\tlogerror(\"could not parse request start-line '%s'\",\n +\t\t\t req->start_line.buf);\n -+\t\tresult = WR_IO_ERROR;\n ++\t\tresult = WR_CLIENT_ERROR;\n +\t\tgoto done;\n +\t}\n +\n @@ t/helper/test-http-server.c: done:\n +\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n +\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n +\t\t\t req->http_version);\n -+\t\tresult = WR_IO_ERROR;\n ++\t\tresult = WR_CLIENT_ERROR;\n +\t\tgoto done;\n +\t}\n +\n @@ t/helper/test-http-server.c: done:\n +\t\tstring_list_append(&req->header_list, hp);\n +\n +\t\t/* also store common request headers as struct req members */\n -+\t\tif (skip_prefix(hp, \"Content-Type: \", &hv)) {\n ++\t\tif (skip_iprefix(hp, \"Content-Type: \", &hv)) {\n +\t\t\treq->content_type = hv;\n -+\t\t} else if (skip_prefix(hp, \"Content-Length: \", &hv)) {\n -+\t\t\treq->content_length = strtol(hv, &hp, 10);\n ++\t\t} else if (skip_iprefix(hp, \"Content-Length: \", &hv)) {\n ++\t\t\t/*\n ++\t\t\t * Content-Length is always non-negative, but has no\n ++\t\t\t * upper bound according to RFC 7230 (§3.3.2).\n ++\t\t\t */\n ++\t\t\tintmax_t len = 0;\n ++\t\t\tif (sscanf(hv, \"%\"PRIdMAX, &len) != 1 || len < 0 ||\n ++\t\t\t len == INTMAX_MAX) {\n ++\t\t\t\tlogerror(\"invalid content-length: '%s'\", hv);\n ++\t\t\t\tresult = WR_CLIENT_ERROR;\n ++\t\t\t\tgoto done;\n ++\t\t\t}\n ++\n ++\t\t\treq->content_length = (uintmax_t)len;\n ++\t\t\treq->has_content_length = 1;\n +\t\t}\n +\t}\n +\n @@ t/helper/test-http-server.c: done:\n +\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n +\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n +\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n -+\t\tif (req->content_length >= 0)\n -+\t\t\ttrace2_printf(\"%s: clen: %d\", TR2_CAT, req->content_length);\n ++\t\tif (req->has_content_length)\n ++\t\t\ttrace2_printf(\"%s: clen: %\"PRIuMAX, TR2_CAT,\n ++\t\t\t\t req->content_length);\n +\t\tif (req->content_type)\n +\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n +\t\tfor_each_string_list_item(item, &req->header_list)\n @@ t/helper/test-http-server.c: done:\n +static enum worker_result dispatch(struct req *req)\n +{\n +\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n -+\t\t\t WR_OK | WR_HANGUP);\n ++\t\t\t WR_HANGUP);\n +}\n +\n static enum worker_result worker(void)\n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n \n \twhile (1) {\n -\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n --\t\t\t\t NULL, WR_OK | WR_HANGUP);\n +-\t\t\t\t NULL, WR_HANGUP);\n +\t\treq__release(&req);\n +\n +\t\talarm(timeout);\n +\t\twr = req__read(&req, 0);\n +\t\talarm(0);\n +\n ++\t\tif (wr == WR_CLIENT_ERROR)\n ++\t\t\twr = send_http_error(STDOUT_FILENO, 400, \"Bad Request\",\n ++\t\t\t\t\t -1, NULL, wr);\n ++\n +\t\tif (wr != WR_OK)\n +\t\t\tbreak;\n \n @@ t/helper/test-http-server.c: static enum worker_result worker(void)\n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\n +\n + ## t/t5556-http-auth.sh (new) ##\n +@@\n ++#!/bin/sh\n ++\n ++test_description='test http auth header and credential helper interop'\n ++\n ++TEST_NO_CREATE_REPO=1\n ++. ./test-lib.sh\n ++\n ++# Setup a repository\n ++#\n ++REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n ++\n ++SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n ++\n ++PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n ++\n ++test_expect_success 'setup repos' '\n ++\ttest_create_repo \"$REPO_DIR\" &&\n ++\tgit -C \"$REPO_DIR\" branch -M main\n ++'\n ++\n ++run_http_server_worker() {\n ++\t(\n ++\t\tcd \"$REPO_DIR\"\n ++\t\ttest-http-server --worker \"$@\" 2>\"$SERVER_LOG\" | tr -d \"\\r\"\n ++\t)\n ++}\n ++\n ++per_test_cleanup () {\n ++\trm -f OUT.* &&\n ++\trm -f IN.* &&\n ++}\n ++\n ++test_expect_success 'http auth server request parsing' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\n ++\tcat >auth.config <<-EOF &&\n ++\t[auth]\n ++\t\tallowAnonymous = true\n ++\tEOF\n ++\n ++\techo \"HTTP/1.1 400 Bad Request\" >OUT.http400 &&\n ++\techo \"HTTP/1.1 200 OK\" >OUT.http200 &&\n ++\n ++\tcat >IN.http.valid <<-EOF &&\n ++\tGET /info/refs HTTP/1.1\n ++\tContent-Length: 0\n ++\tEOF\n ++\n ++\tcat >IN.http.badfirstline <<-EOF &&\n ++\t/info/refs GET HTTP\n ++\tEOF\n ++\n ++\tcat >IN.http.badhttpver <<-EOF &&\n ++\tGET /info/refs HTTP/999.9\n ++\tEOF\n ++\n ++\tcat >IN.http.ltzlen <<-EOF &&\n ++\tGET /info/refs HTTP/1.1\n ++\tContent-Length: -1\n ++\tEOF\n ++\n ++\tcat >IN.http.badlen <<-EOF &&\n ++\tGET /info/refs HTTP/1.1\n ++\tContent-Length: not-a-number\n ++\tEOF\n ++\n ++\tcat >IN.http.overlen <<-EOF &&\n ++\tGET /info/refs HTTP/1.1\n ++\tContent-Length: 9223372036854775807\n ++\tEOF\n ++\n ++\trun_http_server_worker \\\n ++\t\t--auth-config=\"$TRASH_DIRECTORY/auth.config\" OUT.actual &&\n ++\ttest_cmp OUT.http200 OUT.actual &&\n ++\n ++\trun_http_server_worker OUT.actual &&\n ++\ttest_cmp OUT.http400 OUT.actual &&\n ++\n ++\trun_http_server_worker OUT.actual &&\n ++\ttest_cmp OUT.http400 OUT.actual &&\n ++\n ++\trun_http_server_worker OUT.actual &&\n ++\ttest_cmp OUT.http400 OUT.actual &&\n ++\n ++\trun_http_server_worker OUT.actual &&\n ++\ttest_cmp OUT.http400 OUT.actual\n ++'\n ++\n ++test_done\n 7: ab06ac9b965 ! 7: ca9c2787248 test-http-server: pass Git requests to http-backend\n @@ t/helper/test-http-server.c: done:\n +\t\treturn error(_(\"could not send '%s'\"), ok);\n +\n +\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n -+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n -+\t\t\treq->uri_path.buf);\n ++\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\", req->uri_path.buf);\n +\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n +\tif (req->query_args.len)\n -+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\",\n -+\t\t\t\treq->query_args.buf);\n ++\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\", req->query_args.buf);\n +\tif (req->content_type)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\",\n -+\t\t\t\treq->content_type);\n -+\tif (req->content_length >= 0)\n -+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIdMAX,\n -+\t\t\t\t(intmax_t)req->content_length);\n ++\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\", req->content_type);\n ++\tif (req->has_content_length)\n ++\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIuMAX,\n ++\t\t\t(uintmax_t)req->content_length);\n +\tcp.git_cmd = 1;\n +\tstrvec_push(&cp.args, \"http-backend\");\n +\tres = run_command(&cp);\n @@ t/helper/test-http-server.c: done:\n +\t\treturn do__git(req);\n +\n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n - \t\t\t WR_OK | WR_HANGUP);\n + \t\t\t WR_HANGUP);\n }\n \n - ## t/t5556-http-auth.sh (new) ##\n -@@\n -+#!/bin/sh\n -+\n -+test_description='test http auth header and credential helper interop'\n -+\n -+TEST_NO_CREATE_REPO=1\n -+. ./test-lib.sh\n -+\n + ## t/t5556-http-auth.sh ##\n +@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'\n + TEST_NO_CREATE_REPO=1\n + . ./test-lib.sh\n + \n +test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n +\n -+# Setup a repository\n -+#\n -+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n -+\n + # Setup a repository\n + #\n + REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n + \n +# Setup some lookback URLs where test-http-server will be listening.\n +# We will spawn it directly inside the repo directory, so we avoid\n +# any need to configure directory mappings etc - we only serve this\n @@ t/t5556-http-auth.sh (new)\n +# killing it by PID).\n +#\n +PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n -+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n -+\n -+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n -+\n -+test_expect_success 'setup repos' '\n -+\ttest_create_repo \"$REPO_DIR\" &&\n -+\tgit -C \"$REPO_DIR\" branch -M main\n -+'\n -+\n + SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n + \n + PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n +@@ t/t5556-http-auth.sh: run_http_server_worker() {\n + \t)\n + }\n + \n +stop_http_server () {\n +\tif ! test -f \"$PID_FILE\"\n +\tthen\n @@ t/t5556-http-auth.sh (new)\n +\treturn 1\n +}\n +\n -+per_test_cleanup () {\n + per_test_cleanup () {\n +\tstop_http_server &&\n -+\trm -f OUT.*\n -+}\n + \trm -f OUT.* &&\n + \trm -f IN.* &&\n + }\n +@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '\n + \ttest_cmp OUT.http400 OUT.actual\n + '\n + \n +\n +test_expect_success 'http auth anonymous no challenge' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n @@ t/t5556-http-auth.sh (new)\n +\tgit ls-remote $ORIGIN_URL\n +'\n +\n -+test_done\n + test_done\n 8: a1ff55dd6e2 ! 8: b8d3e81b553 test-http-server: add simple authentication\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n +\n \tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n - \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\",\n - \t\t\treq->uri_path.buf);\n + \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\", req->uri_path.buf);\n + \tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n \treturn !!res;\n }\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\n +static struct auth_module *get_auth_module(const char *scheme, int create)\n +{\n -+\tint i;\n +\tstruct auth_module *mod;\n -+\tfor (i = 0; i < auth_modules_nr; i++) {\n ++\tfor (size_t i = 0; i < auth_modules_nr; i++) {\n +\t\tmod = auth_modules[i];\n +\t\tif (!strcasecmp(mod->scheme, scheme))\n +\t\t\treturn mod;\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n +\t\tmod->scheme = xstrdup(scheme);\n +\t\tmod->challenge_params = NULL;\n -+\t\tCALLOC_ARRAY(mod->tokens, 1);\n ++\t\tALLOC_ARRAY(mod->tokens, 1);\n +\t\tstring_list_init_dup(mod->tokens);\n +\n +\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\tfor_each_string_list_item(hdr, &req->header_list) {\n +\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n +\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n -+\t\t\tif (!split[0] || !split[1]) continue;\n -+\n -+\t\t\t/* trim trailing space ' ' */\n -+\t\t\tstrbuf_setlen(split[0], split[0]->len - 1);\n -+\n -+\t\t\tmod = get_auth_module(split[0]->buf, 0);\n -+\t\t\tif (mod) {\n -+\t\t\t\tresult = AUTH_DENY;\n -+\n -+\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n -+\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n -+\t\t\t\t\t\tresult = AUTH_ALLOW;\n -+\t\t\t\t\t\tbreak;\n ++\t\t\tif (split[0] && split[1]) {\n ++\t\t\t\t/* trim trailing space ' ' */\n ++\t\t\t\tstrbuf_rtrim(split[0]);\n ++\n ++\t\t\t\tmod = get_auth_module(split[0]->buf, 0);\n ++\t\t\t\tif (mod) {\n ++\t\t\t\t\tresult = AUTH_DENY;\n ++\n ++\t\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n ++\t\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n ++\t\t\t\t\t\t\tresult = AUTH_ALLOW;\n ++\t\t\t\t\t\t\tbreak;\n ++\t\t\t\t\t\t}\n +\t\t\t\t\t}\n -+\t\t\t\t}\n +\n -+\t\t\t\tgoto done;\n ++\t\t\t\t\tstrbuf_list_free(split);\n ++\t\t\t\t\tgoto done;\n ++\t\t\t\t}\n +\t\t\t}\n ++\n ++\t\t\tstrbuf_list_free(split);\n +\t\t}\n +\t}\n +\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\t\t\t &hdrs, *wr);\n +\t}\n +\n -+\tstrbuf_list_free(split);\n +\tstring_list_clear(&hdrs, 0);\n +\n +\treturn result == AUTH_ALLOW ||\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\treturn -1;\n +\n +\t/* trim trailing ':' */\n -+\tif (p[0]->len > 0 && p[0]->buf[p[0]->len - 1] == ':')\n ++\tif (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')\n +\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n +\n +\t*scheme = strbuf_detach(p[0], NULL);\n -+\n -+\tif (p[1])\n -+\t\t*val = strbuf_detach(p[1], NULL);\n ++\t*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;\n +\n +\tstrbuf_list_free(p);\n +\treturn 0;\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\tchar *scheme = NULL;\n +\tchar *token = NULL;\n +\tchar *challenge = NULL;\n -+\tstruct auth_module *mod = NULL;\n ++\tstruct auth_module *mod;\n +\n +\tif (!strcmp(name, \"auth.challenge\")) {\n +\t\tif (split_auth_param(val, &scheme, &challenge)) {\n @@ t/helper/test-http-server.c: static enum worker_result do__git(struct req *req)\n +\t\treturn do__git(req, user);\n \n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n - \t\t\t WR_OK | WR_HANGUP);\n + \t\t\t WR_HANGUP);\n @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n @@ t/helper/test-http-server.c: int cmd_main(int argc, const char **argv)\n \n ## t/t5556-http-auth.sh ##\n @@ t/t5556-http-auth.sh: per_test_cleanup () {\n - \trm -f OUT.*\n + \tstop_http_server &&\n + \trm -f OUT.* &&\n + \trm -f IN.* &&\n ++\trm -f auth.config\n }\n \n + test_expect_success 'http auth server request parsing' '\n +@@ t/t5556-http-auth.sh: test_expect_success 'http auth server request parsing' '\n + \ttest_cmp OUT.http400 OUT.actual\n + '\n + \n +test_expect_success CURL 'http auth server auth config' '\n -+\t#test_when_finished \"per_test_cleanup\" &&\n ++\ttest_when_finished \"per_test_cleanup\" &&\n +\n +\tcat >auth.config <<-EOF &&\n +\t[auth]\n @@ t/t5556-http-auth.sh: per_test_cleanup () {\n +\n +\ttest_cmp OUT.expected OUT.actual\n +'\n -+\n + \n test_expect_success 'http auth anonymous no challenge' '\n \ttest_when_finished \"per_test_cleanup\" &&\n \n 9: 76125cdf239 = 9: 2f97c94f679 test-http-server: add sending of arbitrary headers\n 10: cc9a220ed1f = 10: 4b1635b3f69 http: replace unsafe size_t multiplication with st_mult\n 11: bc1ac8d3eb3 = 11: 5f5e46038cf http: read HTTP WWW-Authenticate response headers\n 12: 7c8229f0b11 ! 12: 09164f77d56 credential: add WWW-Authenticate header to cred requests\n @@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '\n \n +setup_credential_helper\n +\n - stop_http_server () {\n - \tif ! test -f \"$PID_FILE\"\n - \tthen\n -@@ t/t5556-http-auth.sh: start_http_server () {\n - \n - per_test_cleanup () {\n + run_http_server_worker() {\n + \t(\n + \t\tcd \"$REPO_DIR\"\n +@@ t/t5556-http-auth.sh: per_test_cleanup () {\n \tstop_http_server &&\n --\trm -f OUT.*\n -+\trm -f OUT.* &&\n + \trm -f OUT.* &&\n + \trm -f IN.* &&\n +\trm -f *.cred &&\n -+\trm -f auth.config\n + \trm -f auth.config\n }\n \n - test_expect_success CURL 'http auth server auth config' '\n @@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n\n-- \ngitgitgadget\n"},{"id":"470810","messageId":"6e70e304cfe6372444a8070d27c7bcdc40ade046.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 05/12] test-http-server: add HTTP error response function","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:43Z","receivedAt":"2023-01-20T22:09:06Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a function to the test-http-server test helper to write more\nfull and valid HTTP error responses, including all the standard response\nheaders like `Server` and `Date`.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 76 +++++++++++++++++++++++++++++++++----\n 1 file changed, 69 insertions(+), 7 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 6e9a1c479ce..7ca4ddc7999 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -76,11 +76,75 @@ enum worker_result {\n \t * Exit child-process with non-zero status.\n \t */\n \tWR_FATAL_ERROR = 1,\n+\n+\t/*\n+\t * Close the socket and clean up. Does not imply an error.\n+\t */\n+\tWR_HANGUP = 2,\n };\n \n+static enum worker_result send_http_error(int fd, int http_code,\n+\t\t\t\t\t const char *http_code_name,\n+\t\t\t\t\t int retry_after_seconds,\n+\t\t\t\t\t struct string_list *response_headers,\n+\t\t\t\t\t enum worker_result wr_in)\n+{\n+\tstruct strbuf response_header = STRBUF_INIT;\n+\tstruct strbuf response_content = STRBUF_INIT;\n+\tstruct string_list_item *h;\n+\tenum worker_result wr;\n+\n+\tstrbuf_addf(&response_content, \"Error: %d %s\\r\\n\", http_code,\n+\t\t http_code_name);\n+\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_content, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf(&response_header, \"HTTP/1.1 %d %s\\r\\n\", http_code,\n+\t\t http_code_name);\n+\tstrbuf_addstr(&response_header, \"Cache-Control: private\\r\\n\");\n+\tstrbuf_addstr(&response_header, \"Content-Type: text/plain\\r\\n\");\n+\tstrbuf_addf(&response_header, \"Content-Length: %\"PRIuMAX\"\\r\\n\",\n+\t\t (uintmax_t)response_content.len);\n+\n+\tif (retry_after_seconds > 0)\n+\t\tstrbuf_addf(&response_header, \"Retry-After: %d\\r\\n\",\n+\t\t\t retry_after_seconds);\n+\n+\tstrbuf_addf(&response_header, \"Server: test-http-server/%s\\r\\n\",\n+\t\t git_version_string);\n+\tstrbuf_addf(&response_header, \"Date: %s\\r\\n\", show_date(time(NULL), 0,\n+\t\t DATE_MODE(RFC2822)));\n+\n+\tif (response_headers)\n+\t\tfor_each_string_list_item(h, response_headers)\n+\t\t\tstrbuf_addf(&response_header, \"%s\\r\\n\", h->string);\n+\tstrbuf_addstr(&response_header, \"\\r\\n\");\n+\n+\tif (write_in_full(fd, response_header.buf, response_header.len) < 0) {\n+\t\tlogerror(\"unable to write response header\");\n+\t\twr = WR_FATAL_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tif (write_in_full(fd, response_content.buf, response_content.len) < 0) {\n+\t\tlogerror(\"unable to write response content body\");\n+\t\twr = WR_FATAL_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\twr = wr_in;\n+\n+done:\n+\tstrbuf_release(&response_header);\n+\tstrbuf_release(&response_content);\n+\n+\treturn wr;\n+}\n+\n static enum worker_result worker(void)\n {\n-\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -91,10 +155,8 @@ static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n-\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n-\t\t\tlogerror(\"unable to write response\");\n-\t\t\twr = WR_FATAL_ERROR;\n-\t\t}\n+\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n+\t\t\t\t NULL, WR_HANGUP);\n \n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n@@ -103,8 +165,8 @@ static enum worker_result worker(void)\n \tclose(STDIN_FILENO);\n \tclose(STDOUT_FILENO);\n \n-\t/* Only WR_OK should result in a non-zero exit code */\n-\treturn wr != WR_OK;\n+\t/* Only WR_OK and WR_HANGUP should result in a non-zero exit code */\n+\treturn wr != WR_OK && wr != WR_HANGUP;\n }\n \n static int max_connections = 32;\n-- \ngitgitgadget\n\n"},{"id":"470812","messageId":"b6ba344a671c674d1caf577194eddd66eb7e1415.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 02/12] daemon: libify child process handling functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:40Z","receivedAt":"2023-01-20T22:09:06Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions and structures for managing child processes started\nfrom the parent daemon-like process from `daemon.c` to the new shared\n`daemon-utils.{c,h}` files.\n\nOne minor functional change is introduced to `check_dead_children()`\nwhere the logging of a dead/disconnected child is now optional. With the\n'libification' of these functions we extract the call to `loginfo` to a\ncall to a function pointer, and guard the log message creation and\nlogging behind a `NULL` check. Callers can now skip logging by passing\n`NULL` as the `log_fn loginfo` argument.\nThe behaviour of callers in `daemon.c` remains the same (save one extra\nNULL check) however as a pointer to `loginfo` is always passed.\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 77 ++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 32 ++++++++++++++++++\n daemon.c | 92 +++-----------------------------------------------\n 3 files changed, 114 insertions(+), 87 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex b96b55962db..8506664b440 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -207,3 +207,80 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t\t}\n \t}\n }\n+\n+static int addrcmp(const struct sockaddr_storage *s1,\n+ const struct sockaddr_storage *s2)\n+{\n+\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n+\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n+\n+\tif (sa1->sa_family != sa2->sa_family)\n+\t\treturn sa1->sa_family - sa2->sa_family;\n+\tif (sa1->sa_family == AF_INET)\n+\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n+\t\t &((struct sockaddr_in *)s2)->sin_addr,\n+\t\t sizeof(struct in_addr));\n+#ifndef NO_IPV6\n+\tif (sa1->sa_family == AF_INET6)\n+\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n+\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n+\t\t sizeof(struct in6_addr));\n+#endif\n+\treturn 0;\n+}\n+\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn , unsigned int *live_children)\n+{\n+\tstruct child *newborn, **cradle;\n+\n+\tCALLOC_ARRAY(newborn, 1);\n+\t(*live_children)++;\n+\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n+\tmemcpy(&newborn->address, addr, addrlen);\n+\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n+\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\t\t\tbreak;\n+\tnewborn->next = *cradle;\n+\t*cradle = newborn;\n+}\n+\n+void kill_some_child(struct child *firstborn)\n+{\n+\tconst struct child *blanket, *next;\n+\n+\tif (!(blanket = firstborn))\n+\t\treturn;\n+\n+\tfor (; (next = blanket->next); blanket = next)\n+\t\tif (!addrcmp(&blanket->address, &next->address)) {\n+\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\t\t\tbreak;\n+\t\t}\n+}\n+\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo)\n+{\n+\tint status;\n+\tpid_t pid;\n+\n+\tstruct child **cradle, *blanket;\n+\tfor (cradle = &firstborn; (blanket = *cradle);)\n+\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\t\t\tif (loginfo) {\n+\t\t\t\tconst char *dead = \"\";\n+\t\t\t\tif (status)\n+\t\t\t\t\tdead = \" (with error)\";\n+\t\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\",\n+\t\t\t\t\t(uintmax_t)pid, dead);\n+\t\t\t}\n+\n+\t\t\t/* remove the child */\n+\t\t\t*cradle = blanket->next;\n+\t\t\t(*live_children)--;\n+\t\t\tchild_process_clear(&blanket->cld);\n+\t\t\tfree(blanket);\n+\t\t} else\n+\t\t\tcradle = &blanket->next;\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex 6710a2a6dc0..97e5cae20b8 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -2,6 +2,7 @@\n #define DAEMON_UTILS_H\n \n #include \"git-compat-util.h\"\n+#include \"run-command.h\"\n #include \"string-list.h\"\n \n typedef void (*log_fn)(const char *msg, ...);\n@@ -20,4 +21,35 @@ void socksetup(struct string_list *listen_addr, int listen_port,\n \t struct socketlist *socklist, int reuseaddr,\n \t log_fn logerror);\n \n+struct child {\n+\tstruct child *next;\n+\tstruct child_process cld;\n+\tstruct sockaddr_storage address;\n+};\n+\n+/*\n+ * Add the child_process to the set of children and increment the number of\n+ * live children.\n+ */\n+void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n+\t struct child *firstborn, unsigned int *live_children);\n+\n+/*\n+ * Kill the newest connection from a duplicate IP.\n+ *\n+ * This function should be called if the number of connections grows\n+ * past the maximum number of allowed connections.\n+ */\n+void kill_some_child(struct child *firstborn);\n+\n+/*\n+ * Check for children that have disconnected and remove them from the\n+ * active set, decrementing the number of live children.\n+ *\n+ * Optionally log the child PID that disconnected by passing a loginfo\n+ * function.\n+ */\n+void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+\t\t\t log_fn loginfo);\n+\n #endif\ndiff --git a/daemon.c b/daemon.c\nindex 1ed4e705680..ec3b407ecbc 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -785,93 +785,11 @@ static int execute(void)\n \treturn -1;\n }\n \n-static int addrcmp(const struct sockaddr_storage *s1,\n- const struct sockaddr_storage *s2)\n-{\n-\tconst struct sockaddr *sa1 = (const struct sockaddr*) s1;\n-\tconst struct sockaddr *sa2 = (const struct sockaddr*) s2;\n-\n-\tif (sa1->sa_family != sa2->sa_family)\n-\t\treturn sa1->sa_family - sa2->sa_family;\n-\tif (sa1->sa_family == AF_INET)\n-\t\treturn memcmp(&((struct sockaddr_in *)s1)->sin_addr,\n-\t\t &((struct sockaddr_in *)s2)->sin_addr,\n-\t\t sizeof(struct in_addr));\n-#ifndef NO_IPV6\n-\tif (sa1->sa_family == AF_INET6)\n-\t\treturn memcmp(&((struct sockaddr_in6 *)s1)->sin6_addr,\n-\t\t &((struct sockaddr_in6 *)s2)->sin6_addr,\n-\t\t sizeof(struct in6_addr));\n-#endif\n-\treturn 0;\n-}\n-\n static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child {\n-\tstruct child *next;\n-\tstruct child_process cld;\n-\tstruct sockaddr_storage address;\n-} *firstborn;\n-\n-static void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen)\n-{\n-\tstruct child *newborn, **cradle;\n-\n-\tCALLOC_ARRAY(newborn, 1);\n-\tlive_children++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n-\t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n-}\n-\n-/*\n- * This gets called if the number of connections grows\n- * past \"max_connections\".\n- *\n- * We kill the newest connection from a duplicate IP.\n- */\n-static void kill_some_child(void)\n-{\n-\tconst struct child *blanket, *next;\n-\n-\tif (!(blanket = firstborn))\n-\t\treturn;\n-\n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n-\t\t\tbreak;\n-\t\t}\n-}\n-\n-static void check_dead_children(void)\n-{\n-\tint status;\n-\tpid_t pid;\n-\n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n-\t\t\tconst char *dead = \"\";\n-\t\t\tif (status)\n-\t\t\t\tdead = \" (with error)\";\n-\t\t\tloginfo(\"[%\"PRIuMAX\"] Disconnected%s\", (uintmax_t)pid, dead);\n-\n-\t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n-\t\t\tlive_children--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n-\t\t} else\n-\t\t\tcradle = &blanket->next;\n-}\n+static struct child *firstborn;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -879,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child();\n+\t\tkill_some_child(firstborn);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -914,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen);\n+\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -944,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children();\n+\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470813","messageId":"9967401c972cab547d7619a208c3a0e6a3923cd4.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 03/12] daemon: rename some esoteric/laboured terminology","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:41Z","receivedAt":"2023-01-20T22:09:08Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRename some of the variables and function arguments used to manage child\nprocesses. The existing names are esoteric; stretching an analogy too\nfar to the point of being confusing to understand.\n\nRename \"firstborn\" to \"first_child\", \"newborn\" to \"new_cld\", \"blanket\"\nto \"current\" and \"cradle\" to \"ptr\".\n\nSigned-off-by: Matthew John Cheetham \n---\n daemon-utils.c | 46 +++++++++++++++++++++++-----------------------\n daemon-utils.h | 6 +++---\n daemon.c | 10 +++++-----\n 3 files changed, 31 insertions(+), 31 deletions(-)\n\ndiff --git a/daemon-utils.c b/daemon-utils.c\nindex 8506664b440..f23ea35ed7b 100644\n--- a/daemon-utils.c\n+++ b/daemon-utils.c\n@@ -230,44 +230,44 @@ static int addrcmp(const struct sockaddr_storage *s1,\n }\n \n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn , unsigned int *live_children)\n+\t struct child *first_child, unsigned int *live_children)\n {\n-\tstruct child *newborn, **cradle;\n+\tstruct child *new_cld, **current;\n \n-\tCALLOC_ARRAY(newborn, 1);\n+\tCALLOC_ARRAY(new_cld, 1);\n \t(*live_children)++;\n-\tmemcpy(&newborn->cld, cld, sizeof(*cld));\n-\tmemcpy(&newborn->address, addr, addrlen);\n-\tfor (cradle = &firstborn; *cradle; cradle = &(*cradle)->next)\n-\t\tif (!addrcmp(&(*cradle)->address, &newborn->address))\n+\tmemcpy(&new_cld->cld, cld, sizeof(*cld));\n+\tmemcpy(&new_cld->address, addr, addrlen);\n+\tfor (current = &first_child; *current; current = &(*current)->next)\n+\t\tif (!addrcmp(&(*current)->address, &new_cld->address))\n \t\t\tbreak;\n-\tnewborn->next = *cradle;\n-\t*cradle = newborn;\n+\tnew_cld->next = *current;\n+\t*current = new_cld;\n }\n \n-void kill_some_child(struct child *firstborn)\n+void kill_some_child(struct child *first_child)\n {\n-\tconst struct child *blanket, *next;\n+\tconst struct child *current, *next;\n \n-\tif (!(blanket = firstborn))\n+\tif (!(current = first_child))\n \t\treturn;\n \n-\tfor (; (next = blanket->next); blanket = next)\n-\t\tif (!addrcmp(&blanket->address, &next->address)) {\n-\t\t\tkill(blanket->cld.pid, SIGTERM);\n+\tfor (; (next = current->next); current = next)\n+\t\tif (!addrcmp(¤t->address, &next->address)) {\n+\t\t\tkill(current->cld.pid, SIGTERM);\n \t\t\tbreak;\n \t\t}\n }\n \n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo)\n {\n \tint status;\n \tpid_t pid;\n \n-\tstruct child **cradle, *blanket;\n-\tfor (cradle = &firstborn; (blanket = *cradle);)\n-\t\tif ((pid = waitpid(blanket->cld.pid, &status, WNOHANG)) > 1) {\n+\tstruct child **ptr, *current;\n+\tfor (ptr = &first_child; (current = *ptr);)\n+\t\tif ((pid = waitpid(current->cld.pid, &status, WNOHANG)) > 1) {\n \t\t\tif (loginfo) {\n \t\t\t\tconst char *dead = \"\";\n \t\t\t\tif (status)\n@@ -277,10 +277,10 @@ void check_dead_children(struct child *firstborn, unsigned int *live_children,\n \t\t\t}\n \n \t\t\t/* remove the child */\n-\t\t\t*cradle = blanket->next;\n+\t\t\t*ptr = current->next;\n \t\t\t(*live_children)--;\n-\t\t\tchild_process_clear(&blanket->cld);\n-\t\t\tfree(blanket);\n+\t\t\tchild_process_clear(¤t->cld);\n+\t\t\tfree(current);\n \t\t} else\n-\t\t\tcradle = &blanket->next;\n+\t\t\tptr = ¤t->next;\n }\ndiff --git a/daemon-utils.h b/daemon-utils.h\nindex 97e5cae20b8..c866e9c9a4e 100644\n--- a/daemon-utils.h\n+++ b/daemon-utils.h\n@@ -32,7 +32,7 @@ struct child {\n * live children.\n */\n void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrlen,\n-\t struct child *firstborn, unsigned int *live_children);\n+\t struct child *first_child, unsigned int *live_children);\n \n /*\n * Kill the newest connection from a duplicate IP.\n@@ -40,7 +40,7 @@ void add_child(struct child_process *cld, struct sockaddr *addr, socklen_t addrl\n * This function should be called if the number of connections grows\n * past the maximum number of allowed connections.\n */\n-void kill_some_child(struct child *firstborn);\n+void kill_some_child(struct child *first_child);\n \n /*\n * Check for children that have disconnected and remove them from the\n@@ -49,7 +49,7 @@ void kill_some_child(struct child *firstborn);\n * Optionally log the child PID that disconnected by passing a loginfo\n * function.\n */\n-void check_dead_children(struct child *firstborn, unsigned int *live_children,\n+void check_dead_children(struct child *first_child, unsigned int *live_children,\n \t\t\t log_fn loginfo);\n \n #endif\ndiff --git a/daemon.c b/daemon.c\nindex ec3b407ecbc..d3e7d81de18 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -789,7 +789,7 @@ static int max_connections = 32;\n \n static unsigned int live_children;\n \n-static struct child *firstborn;\n+static struct child *first_child;\n \n static struct strvec cld_argv = STRVEC_INIT;\n static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n@@ -797,9 +797,9 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \n \tif (max_connections && live_children >= max_connections) {\n-\t\tkill_some_child(firstborn);\n+\t\tkill_some_child(first_child);\n \t\tsleep(1); /* give it some time to die */\n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \t\tif (live_children >= max_connections) {\n \t\t\tclose(incoming);\n \t\t\tlogerror(\"Too many children, dropping connection\");\n@@ -832,7 +832,7 @@ static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n \tif (start_command(&cld))\n \t\tlogerror(\"unable to fork\");\n \telse\n-\t\tadd_child(&cld, addr, addrlen, firstborn, &live_children);\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n }\n \n static void child_handler(int signo)\n@@ -862,7 +862,7 @@ static int service_loop(struct socketlist *socklist)\n \tfor (;;) {\n \t\tint i;\n \n-\t\tcheck_dead_children(firstborn, &live_children, loginfo);\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n \n \t\tif (poll(pfd, socklist->nr, -1) < 0) {\n \t\t\tif (errno != EINTR) {\n-- \ngitgitgadget\n\n"},{"id":"470814","messageId":"ca9c2787248688cd7d8e20043a6ed75d93654e35.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 07/12] test-http-server: pass Git requests to http-backend","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:45Z","receivedAt":"2023-01-20T22:09:11Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nTeach the test-http-sever test helper to forward Git requests to the\n`git-http-backend`.\n\nIntroduce a new test script t5556-http-auth.sh that spins up the test\nHTTP server and attempts an `ls-remote` on the served repository,\nwithout any authentication.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 68 ++++++++++++++++++++++++++++++\n t/t5556-http-auth.sh | 83 +++++++++++++++++++++++++++++++++++++\n 2 files changed, 151 insertions(+)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 900f5733cc1..4191daf3c64 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -323,8 +323,76 @@ done:\n \treturn result;\n }\n \n+static int is_git_request(struct req *req)\n+{\n+\tstatic regex_t *smart_http_regex;\n+\tstatic int initialized;\n+\n+\tif (!initialized) {\n+\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n+\t\t/*\n+\t\t * This regular expression matches all dumb and smart HTTP\n+\t\t * requests that are currently in use, and defined in\n+\t\t * Documentation/gitprotocol-http.txt.\n+\t\t *\n+\t\t */\n+\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n+\t\t\t REG_EXTENDED)) {\n+\t\t\twarning(\"could not compile smart HTTP regex\");\n+\t\t\tsmart_http_regex = NULL;\n+\t\t}\n+\t\tinitialized = 1;\n+\t}\n+\n+\treturn smart_http_regex &&\n+\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+}\n+\n+static enum worker_result do__git(struct req *req)\n+{\n+\tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tint res;\n+\n+\t/*\n+\t * Note that we always respond with a 200 OK response even if the\n+\t * http-backend process exits with an error. This helper is intended\n+\t * only to be used to exercise the HTTP auth handling in the Git client,\n+\t * and specifically around authentication (not handled by http-backend).\n+\t *\n+\t * If we wanted to respond with a more 'valid' HTTP response status then\n+\t * we'd need to buffer the output of http-backend, wait for and grok the\n+\t * exit status of the process, then write the HTTP status line followed\n+\t * by the http-backend output. This is outside of the scope of this test\n+\t * helper's use at time of writing.\n+\t */\n+\tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n+\t\treturn error(_(\"could not send '%s'\"), ok);\n+\n+\tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n+\tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\", req->uri_path.buf);\n+\tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n+\tif (req->query_args.len)\n+\t\tstrvec_pushf(&cp.env, \"QUERY_STRING=%s\", req->query_args.buf);\n+\tif (req->content_type)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_TYPE=%s\", req->content_type);\n+\tif (req->has_content_length)\n+\t\tstrvec_pushf(&cp.env, \"CONTENT_LENGTH=%\" PRIuMAX,\n+\t\t\t(uintmax_t)req->content_length);\n+\tcp.git_cmd = 1;\n+\tstrvec_push(&cp.args, \"http-backend\");\n+\tres = run_command(&cp);\n+\tclose(STDOUT_FILENO);\n+\tclose(STDIN_FILENO);\n+\treturn !!res;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tif (is_git_request(req))\n+\t\treturn do__git(req);\n+\n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_HANGUP);\n }\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 06efc85ca53..c0a47ce342b 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -5,10 +5,25 @@ test_description='test http auth header and credential helper interop'\n TEST_NO_CREATE_REPO=1\n . ./test-lib.sh\n \n+test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n+\n # Setup a repository\n #\n REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n \n+# Setup some lookback URLs where test-http-server will be listening.\n+# We will spawn it directly inside the repo directory, so we avoid\n+# any need to configure directory mappings etc - we only serve this\n+# repository from the root '/' of the server.\n+#\n+HOST_PORT=127.0.0.1:$GIT_TEST_HTTP_PROTOCOL_PORT\n+ORIGIN_URL=http://$HOST_PORT/\n+\n+# The pid-file is created by test-http-server when it starts.\n+# The server will shutdown if/when we delete it (this is easier than\n+# killing it by PID).\n+#\n+PID_FILE=\"$TRASH_DIRECTORY\"/pid-file.pid\n SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n \n PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n@@ -25,7 +40,65 @@ run_http_server_worker() {\n \t)\n }\n \n+stop_http_server () {\n+\tif ! test -f \"$PID_FILE\"\n+\tthen\n+\t\treturn 0\n+\tfi\n+\t#\n+\t# The server will shutdown automatically when we delete the pid-file.\n+\t#\n+\trm -f \"$PID_FILE\"\n+\t#\n+\t# Give it a few seconds to shutdown (mainly to completely release the\n+\t# port before the next test start another instance and it attempts to\n+\t# bind to it).\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif grep -q \"Starting graceful shutdown\" \"$SERVER_LOG\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"stop_http_server: timeout waiting for server shutdown\"\n+\treturn 1\n+}\n+\n+start_http_server () {\n+\t#\n+\t# Launch our server into the background in repo_dir.\n+\t#\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --verbose \\\n+\t\t\t--listen=127.0.0.1 \\\n+\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n+\t\t\t--reuseaddr \\\n+\t\t\t--pid-file=\"$PID_FILE\" \\\n+\t\t\t\"$@\" \\\n+\t\t\t2>\"$SERVER_LOG\" &\n+\t)\n+\t#\n+\t# Give it a few seconds to get started.\n+\t#\n+\tfor k in 0 1 2 3 4\n+\tdo\n+\t\tif test -f \"$PID_FILE\"\n+\t\tthen\n+\t\t\treturn 0\n+\t\tfi\n+\t\tsleep 1\n+\tdone\n+\n+\techo \"start_http_server: timeout waiting for server startup\"\n+\treturn 1\n+}\n+\n per_test_cleanup () {\n+\tstop_http_server &&\n \trm -f OUT.* &&\n \trm -f IN.* &&\n }\n@@ -87,4 +160,14 @@ test_expect_success 'http auth server request parsing' '\n \ttest_cmp OUT.http400 OUT.actual\n '\n \n+\n+test_expect_success 'http auth anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tstart_http_server &&\n+\n+\t# Attempt to read from a protected repository\n+\tgit ls-remote $ORIGIN_URL\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"470815","messageId":"17c890ee1080abc81267e44a1eaff4609ee41690.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 04/12] test-http-server: add stub HTTP server test helper","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:42Z","receivedAt":"2023-01-20T22:09:13Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nIntroduce a mini HTTP server helper that in the future will be enhanced\nto provide a frontend for the git-http-backend, with support for\narbitrary authentication schemes.\n\nRight now, test-http-server is a pared-down copy of the git-daemon that\nalways returns a 501 Not Implemented response to all callers.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n contrib/buildsystems/CMakeLists.txt | 11 +-\n t/helper/.gitignore | 1 +\n t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++\n 4 files changed, 392 insertions(+), 2 deletions(-)\n create mode 100644 t/helper/test-http-server.c\n\ndiff --git a/Makefile b/Makefile\nindex 2654094dbb5..3cd61c792ac 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -865,6 +865,7 @@ TEST_BUILTINS_OBJS += test-xml-encode.o\n # Do not add more tests here unless they have extra dependencies. Add\n # them in TEST_BUILTINS_OBJS above.\n TEST_PROGRAMS_NEED_X += test-fake-ssh\n+TEST_PROGRAMS_NEED_X += test-http-server\n TEST_PROGRAMS_NEED_X += test-tool\n \n TEST_PROGRAMS = $(patsubst %,t/helper/%$X,$(TEST_PROGRAMS_NEED_X))\ndiff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt\nindex 2f6e0197ffa..5d949dcb16c 100644\n--- a/contrib/buildsystems/CMakeLists.txt\n+++ b/contrib/buildsystems/CMakeLists.txt\n@@ -961,6 +961,9 @@ if(BUILD_TESTING)\n add_executable(test-fake-ssh ${CMAKE_SOURCE_DIR}/t/helper/test-fake-ssh.c)\n target_link_libraries(test-fake-ssh common-main)\n \n+add_executable(test-http-server ${CMAKE_SOURCE_DIR}/t/helper/test-http-server.c)\n+target_link_libraries(test-http-server common-main)\n+\n #reftable-tests\n parse_makefile_for_sources(test-reftable_SOURCES \"REFTABLE_TEST_OBJS\")\n list(TRANSFORM test-reftable_SOURCES PREPEND \"${CMAKE_SOURCE_DIR}/\")\n@@ -980,6 +983,11 @@ if(MSVC)\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n \tset_target_properties(test-fake-ssh test-tool\n \t\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n+\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG ${CMAKE_BINARY_DIR}/t/helper)\n+\tset_target_properties(test-http-server\n+\t\t\tPROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_BINARY_DIR}/t/helper)\n endif()\n \n #wrapper scripts\n@@ -987,8 +995,7 @@ set(wrapper_scripts\n \tgit git-upload-pack git-receive-pack git-upload-archive git-shell git-remote-ext scalar)\n \n set(wrapper_test_scripts\n-\ttest-fake-ssh test-tool)\n-\n+\ttest-http-server test-fake-ssh test-tool)\n \n foreach(script ${wrapper_scripts})\n \tfile(STRINGS ${CMAKE_SOURCE_DIR}/wrap-for-bin.sh content NEWLINE_CONSUME)\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex 8c2ddcce95f..9aa9c752997 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -1,2 +1,3 @@\n /test-tool\n /test-fake-ssh\n+/test-http-server\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nnew file mode 100644\nindex 00000000000..6e9a1c479ce\n--- /dev/null\n+++ b/t/helper/test-http-server.c\n@@ -0,0 +1,381 @@\n+#include \"daemon-utils.h\"\n+#include \"config.h\"\n+#include \"run-command.h\"\n+#include \"strbuf.h\"\n+#include \"string-list.h\"\n+#include \"trace2.h\"\n+#include \"version.h\"\n+#include \"dir.h\"\n+#include \"date.h\"\n+\n+#define TR2_CAT \"test-http-server\"\n+\n+static const char *pid_file;\n+static int verbose;\n+static int reuseaddr;\n+\n+static const char test_http_auth_usage[] =\n+\"http-server [--verbose]\\n\"\n+\" [--timeout=] [--max-connections=]\\n\"\n+\" [--reuseaddr] [--pid-file=]\\n\"\n+\" [--listen=]* [--port=]\\n\"\n+;\n+\n+static unsigned int timeout;\n+\n+static void logreport(const char *label, const char *err, va_list params)\n+{\n+\tstruct strbuf msg = STRBUF_INIT;\n+\n+\tstrbuf_addf(&msg, \"[%\"PRIuMAX\"] %s: \", (uintmax_t)getpid(), label);\n+\tstrbuf_vaddf(&msg, err, params);\n+\tstrbuf_addch(&msg, '\\n');\n+\n+\tfwrite(msg.buf, sizeof(char), msg.len, stderr);\n+\tfflush(stderr);\n+\n+\tstrbuf_release(&msg);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void logerror(const char *err, ...)\n+{\n+\tva_list params;\n+\tva_start(params, err);\n+\tlogreport(\"error\", err, params);\n+\tva_end(params);\n+}\n+\n+__attribute__((format (printf, 1, 2)))\n+static void loginfo(const char *err, ...)\n+{\n+\tva_list params;\n+\tif (!verbose)\n+\t\treturn;\n+\tva_start(params, err);\n+\tlogreport(\"info\", err, params);\n+\tva_end(params);\n+}\n+\n+/*\n+ * The code in this section is used by \"worker\" instances to service\n+ * a single connection from a client. The worker talks to the client\n+ * on stdin and stdout.\n+ */\n+\n+enum worker_result {\n+\t/*\n+\t * Operation successful.\n+\t * Caller *might* keep the socket open and allow keep-alive.\n+\t */\n+\tWR_OK = 0,\n+\n+\t/*\n+\t * Fatal error that is not recoverable.\n+\t * Close the socket and clean up.\n+\t * Exit child-process with non-zero status.\n+\t */\n+\tWR_FATAL_ERROR = 1,\n+};\n+\n+static enum worker_result worker(void)\n+{\n+\tconst char *response = \"HTTP/1.1 501 Not Implemented\\r\\n\";\n+\tchar *client_addr = getenv(\"REMOTE_ADDR\");\n+\tchar *client_port = getenv(\"REMOTE_PORT\");\n+\tenum worker_result wr = WR_OK;\n+\n+\tif (client_addr)\n+\t\tloginfo(\"Connection from %s:%s\", client_addr, client_port);\n+\n+\tset_keep_alive(0, logerror);\n+\n+\twhile (1) {\n+\t\tif (write_in_full(STDOUT_FILENO, response, strlen(response)) < 0) {\n+\t\t\tlogerror(\"unable to write response\");\n+\t\t\twr = WR_FATAL_ERROR;\n+\t\t}\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n+\t}\n+\n+\tclose(STDIN_FILENO);\n+\tclose(STDOUT_FILENO);\n+\n+\t/* Only WR_OK should result in a non-zero exit code */\n+\treturn wr != WR_OK;\n+}\n+\n+static int max_connections = 32;\n+\n+static unsigned int live_children;\n+\n+static struct child *first_child;\n+\n+static struct strvec cld_argv = STRVEC_INIT;\n+static void handle(int incoming, struct sockaddr *addr, socklen_t addrlen)\n+{\n+\tstruct child_process cld = CHILD_PROCESS_INIT;\n+\n+\tif (max_connections && live_children >= max_connections) {\n+\t\tkill_some_child(first_child);\n+\t\tsleep(1); /* give it some time to die */\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\t\tif (live_children >= max_connections) {\n+\t\t\tclose(incoming);\n+\t\t\tlogerror(\"Too many children, dropping connection\");\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n+\tif (addr->sa_family == AF_INET) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in *sin_addr = (void *) addr;\n+\t\tinet_ntop(addr->sa_family, &sin_addr->sin_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=%s\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin_addr->sin_port));\n+#ifndef NO_IPV6\n+\t} else if (addr->sa_family == AF_INET6) {\n+\t\tchar buf[128] = \"\";\n+\t\tstruct sockaddr_in6 *sin6_addr = (void *) addr;\n+\t\tinet_ntop(AF_INET6, &sin6_addr->sin6_addr, buf, sizeof(buf));\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_ADDR=[%s]\", buf);\n+\t\tstrvec_pushf(&cld.env, \"REMOTE_PORT=%d\",\n+\t\t\t\t ntohs(sin6_addr->sin6_port));\n+#endif\n+\t}\n+\n+\tstrvec_pushv(&cld.args, cld_argv.v);\n+\tcld.in = incoming;\n+\tcld.out = dup(incoming);\n+\n+\tif (cld.out < 0)\n+\t\tlogerror(\"could not dup() `incoming`\");\n+\telse if (start_command(&cld))\n+\t\tlogerror(\"unable to fork\");\n+\telse\n+\t\tadd_child(&cld, addr, addrlen, first_child, &live_children);\n+}\n+\n+static void child_handler(int signo)\n+{\n+\t/*\n+\t * Otherwise empty handler because systemcalls will get interrupted\n+\t * upon signal receipt\n+\t * SysV needs the handler to be rearmed\n+\t */\n+\tsignal(SIGCHLD, child_handler);\n+}\n+\n+static int service_loop(struct socketlist *socklist)\n+{\n+\tstruct pollfd *pfd;\n+\tint i;\n+\n+\tCALLOC_ARRAY(pfd, socklist->nr);\n+\n+\tfor (i = 0; i < socklist->nr; i++) {\n+\t\tpfd[i].fd = socklist->list[i];\n+\t\tpfd[i].events = POLLIN;\n+\t}\n+\n+\tsignal(SIGCHLD, child_handler);\n+\n+\tfor (;;) {\n+\t\tint i;\n+\t\tint nr_ready;\n+\t\tint timeout = (pid_file ? 100 : -1);\n+\n+\t\tcheck_dead_children(first_child, &live_children, loginfo);\n+\n+\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n+\t\tif (nr_ready < 0) {\n+\t\t\tif (errno != EINTR) {\n+\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n+\t\t\t\t strerror(errno));\n+\t\t\t\tsleep(1);\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\t\telse if (nr_ready == 0) {\n+\t\t\t/*\n+\t\t\t * If we have a pid_file, then we watch it.\n+\t\t\t * If someone deletes it, we shutdown the service.\n+\t\t\t * The shell scripts in the test suite will use this.\n+\t\t\t */\n+\t\t\tif (!pid_file || file_exists(pid_file))\n+\t\t\t\tcontinue;\n+\t\t\tgoto shutdown;\n+\t\t}\n+\n+\t\tfor (i = 0; i < socklist->nr; i++) {\n+\t\t\tif (pfd[i].revents & POLLIN) {\n+\t\t\t\tunion {\n+\t\t\t\t\tstruct sockaddr sa;\n+\t\t\t\t\tstruct sockaddr_in sai;\n+#ifndef NO_IPV6\n+\t\t\t\t\tstruct sockaddr_in6 sai6;\n+#endif\n+\t\t\t\t} ss;\n+\t\t\t\tsocklen_t sslen = sizeof(ss);\n+\t\t\t\tint incoming = accept(pfd[i].fd, &ss.sa, &sslen);\n+\t\t\t\tif (incoming < 0) {\n+\t\t\t\t\tswitch (errno) {\n+\t\t\t\t\tcase EAGAIN:\n+\t\t\t\t\tcase EINTR:\n+\t\t\t\t\tcase ECONNABORTED:\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tdefault:\n+\t\t\t\t\t\tdie_errno(\"accept returned\");\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t\thandle(incoming, &ss.sa, sslen);\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+shutdown:\n+\tloginfo(\"Starting graceful shutdown (pid-file gone)\");\n+\tfor (i = 0; i < socklist->nr; i++)\n+\t\tclose(socklist->list[i]);\n+\n+\treturn 0;\n+}\n+\n+static int serve(struct string_list *listen_addr, int listen_port)\n+{\n+\tstruct socketlist socklist = { NULL, 0, 0 };\n+\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr, logerror);\n+\tif (socklist.nr == 0)\n+\t\tdie(\"unable to allocate any listen sockets on port %u\",\n+\t\t listen_port);\n+\n+\tloginfo(\"Ready to rumble\");\n+\n+\t/*\n+\t * Wait to create the pid-file until we've setup the sockets\n+\t * and are open for business.\n+\t */\n+\tif (pid_file)\n+\t\twrite_file(pid_file, \"%\"PRIuMAX, (uintmax_t) getpid());\n+\n+\treturn service_loop(&socklist);\n+}\n+\n+/*\n+ * This section is executed by both the primary instance and all\n+ * worker instances. So, yes, each child-process re-parses the\n+ * command line argument and re-discovers how it should behave.\n+ */\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint listen_port = 0;\n+\tstruct string_list listen_addr = STRING_LIST_INIT_NODUP;\n+\tint worker_mode = 0;\n+\tint i;\n+\n+\ttrace2_cmd_name(\"test-http-server\");\n+\ttrace2_cmd_list_config();\n+\ttrace2_cmd_list_env_vars();\n+\tsetup_git_directory_gently(NULL);\n+\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tconst char *v;\n+\n+\t\tif (skip_prefix(arg, \"--listen=\", &v)) {\n+\t\t\tstring_list_append(&listen_addr, xstrdup_tolower(v));\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--port=\", &v)) {\n+\t\t\tchar *end;\n+\t\t\tunsigned long n;\n+\t\t\tn = strtoul(v, &end, 0);\n+\t\t\tif (*v && !*end) {\n+\t\t\t\tlisten_port = n;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n+\t\tif (!strcmp(arg, \"--worker\")) {\n+\t\t\tworker_mode = 1;\n+\t\t\ttrace2_cmd_mode(\"worker\");\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--verbose\")) {\n+\t\t\tverbose = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n+\t\t\ttimeout = atoi(v);\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n+\t\t\tmax_connections = atoi(v);\n+\t\t\tif (max_connections < 0)\n+\t\t\t\tmax_connections = 0; /* unlimited */\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(arg, \"--reuseaddr\")) {\n+\t\t\treuseaddr = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (skip_prefix(arg, \"--pid-file=\", &v)) {\n+\t\t\tpid_file = v;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n+\t\tusage(test_http_auth_usage);\n+\t}\n+\n+\t/* avoid splitting a message in the middle */\n+\tsetvbuf(stderr, NULL, _IOFBF, 4096);\n+\n+\tif (listen_port == 0)\n+\t\tlisten_port = DEFAULT_GIT_PORT;\n+\n+\t/*\n+\t * If no --listen= args are given, the setup_named_sock()\n+\t * code will use receive a NULL address and set INADDR_ANY.\n+\t * This exposes both internal and external interfaces on the\n+\t * port.\n+\t *\n+\t * Disallow that and default to the internal-use-only loopback\n+\t * address.\n+\t */\n+\tif (!listen_addr.nr)\n+\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n+\n+\t/*\n+\t * worker_mode is set in our own child process instances\n+\t * (that are bound to a connected socket from a client).\n+\t */\n+\tif (worker_mode)\n+\t\treturn worker();\n+\n+\t/*\n+\t * `cld_argv` is a bit of a clever hack. The top-level instance\n+\t * of test-http-server does the normal bind/listen/accept stuff.\n+\t * For each incoming socket, the top-level process spawns\n+\t * a child instance of test-http-server *WITH* the additional\n+\t * `--worker` argument. This causes the child to set `worker_mode`\n+\t * and immediately call `worker()` using the connected socket (and\n+\t * without the usual need for fork() or threads).\n+\t *\n+\t * The magic here is made possible because `cld_argv` is static\n+\t * and handle() (called by service_loop()) knows about it.\n+\t */\n+\tstrvec_push(&cld_argv, argv[0]);\n+\tstrvec_push(&cld_argv, \"--worker\");\n+\tfor (i = 1; i < argc; ++i)\n+\t\tstrvec_push(&cld_argv, argv[i]);\n+\n+\t/*\n+\t * Setup primary instance to listen for connections.\n+\t */\n+\treturn serve(&listen_addr, listen_port);\n+}\n-- \ngitgitgadget\n\n"},{"id":"470816","messageId":"74b0de14185120c9d53d7470e59f57fa20a1927f.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 01/12] daemon: libify socket setup and option functions","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:39Z","receivedAt":"2023-01-20T22:09:14Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nExtract functions for setting up listening sockets and keep-alive options\nfrom `daemon.c` to new `daemon-utils.{c,h}` files. Remove direct\ndependencies on global state by inlining the behaviour at the callsites\nfor all libified functions.\n\nSigned-off-by: Matthew John Cheetham \n---\n Makefile | 1 +\n daemon-utils.c | 209 +++++++++++++++++++++++++++++++++++++++++++++++\n daemon-utils.h | 23 ++++++\n daemon.c | 214 +------------------------------------------------\n 4 files changed, 237 insertions(+), 210 deletions(-)\n create mode 100644 daemon-utils.c\n create mode 100644 daemon-utils.h\n\ndiff --git a/Makefile b/Makefile\nindex b258fdbed86..2654094dbb5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1003,6 +1003,7 @@ LIB_OBJS += credential.o\n LIB_OBJS += csum-file.o\n LIB_OBJS += ctype.o\n LIB_OBJS += date.o\n+LIB_OBJS += daemon-utils.o\n LIB_OBJS += decorate.o\n LIB_OBJS += delta-islands.o\n LIB_OBJS += diagnose.o\ndiff --git a/daemon-utils.c b/daemon-utils.c\nnew file mode 100644\nindex 00000000000..b96b55962db\n--- /dev/null\n+++ b/daemon-utils.c\n@@ -0,0 +1,209 @@\n+#include \"cache.h\"\n+#include \"daemon-utils.h\"\n+\n+void set_keep_alive(int sockfd, log_fn logerror)\n+{\n+\tint ka = 1;\n+\n+\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n+\t\tif (errno != ENOTSOCK)\n+\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n+\t\t\t\tstrerror(errno));\n+\t}\n+}\n+\n+static int set_reuse_addr(int sockfd)\n+{\n+\tint on = 1;\n+\n+\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n+\t\t\t &on, sizeof(on));\n+}\n+\n+static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n+{\n+#ifdef NO_IPV6\n+\tstatic char ip[INET_ADDRSTRLEN];\n+#else\n+\tstatic char ip[INET6_ADDRSTRLEN];\n+#endif\n+\n+\tswitch (family) {\n+#ifndef NO_IPV6\n+\tcase AF_INET6:\n+\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n+\t\tbreak;\n+#endif\n+\tcase AF_INET:\n+\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n+\t\tbreak;\n+\tdefault:\n+\t\txsnprintf(ip, sizeof(ip), \"\");\n+\t}\n+\treturn ip;\n+}\n+\n+#ifndef NO_IPV6\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tint socknum = 0;\n+\tchar pbuf[NI_MAXSERV];\n+\tstruct addrinfo hints, *ai0, *ai;\n+\tint gai;\n+\tlong flags;\n+\n+\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n+\tmemset(&hints, 0, sizeof(hints));\n+\thints.ai_family = AF_UNSPEC;\n+\thints.ai_socktype = SOCK_STREAM;\n+\thints.ai_protocol = IPPROTO_TCP;\n+\thints.ai_flags = AI_PASSIVE;\n+\n+\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n+\tif (gai) {\n+\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n+\t\treturn 0;\n+\t}\n+\n+\tfor (ai = ai0; ai; ai = ai->ai_next) {\n+\t\tint sockfd;\n+\n+\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n+\t\tif (sockfd < 0)\n+\t\t\tcontinue;\n+\t\tif (sockfd >= FD_SETSIZE) {\n+\t\t\tlogerror(\"Socket descriptor too large\");\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+#ifdef IPV6_V6ONLY\n+\t\tif (ai->ai_family == AF_INET6) {\n+\t\t\tint on = 1;\n+\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n+\t\t\t\t &on, sizeof(on));\n+\t\t\t/* Note: error is not fatal */\n+\t\t}\n+#endif\n+\n+\t\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tset_keep_alive(sockfd, logerror);\n+\n+\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n+\t\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\t\tif (listen(sockfd, 5) < 0) {\n+\t\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n+\t\t\t\t strerror(errno));\n+\t\t\tclose(sockfd);\n+\t\t\tcontinue;\t/* not fatal */\n+\t\t}\n+\n+\t\tflags = fcntl(sockfd, F_GETFD, 0);\n+\t\tif (flags >= 0)\n+\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\t\tsocklist->list[socklist->nr++] = sockfd;\n+\t\tsocknum++;\n+\t}\n+\n+\tfreeaddrinfo(ai0);\n+\n+\treturn socknum;\n+}\n+\n+#else /* NO_IPV6 */\n+\n+static int setup_named_sock(char *listen_addr, int listen_port,\n+\t\t\t struct socketlist *socklist, int reuseaddr,\n+\t\t\t log_fn logerror)\n+{\n+\tstruct sockaddr_in sin;\n+\tint sockfd;\n+\tlong flags;\n+\n+\tmemset(&sin, 0, sizeof sin);\n+\tsin.sin_family = AF_INET;\n+\tsin.sin_port = htons(listen_port);\n+\n+\tif (listen_addr) {\n+\t\t/* Well, host better be an IP address here. */\n+\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n+\t\t\treturn 0;\n+\t} else {\n+\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n+\t}\n+\n+\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n+\tif (sockfd < 0)\n+\t\treturn 0;\n+\n+\tif (reuseaddr && set_reuse_addr(sockfd)) {\n+\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tset_keep_alive(sockfd, logerror);\n+\n+\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n+\t\tlogerror(\"Could not bind to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tif (listen(sockfd, 5) < 0) {\n+\t\tlogerror(\"Could not listen to %s: %s\",\n+\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n+\t\t\t strerror(errno));\n+\t\tclose(sockfd);\n+\t\treturn 0;\n+\t}\n+\n+\tflags = fcntl(sockfd, F_GETFD, 0);\n+\tif (flags >= 0)\n+\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n+\n+\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n+\tsocklist->list[socklist->nr++] = sockfd;\n+\treturn 1;\n+}\n+\n+#endif\n+\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror)\n+{\n+\tif (!listen_addr->nr)\n+\t\tsetup_named_sock(NULL, listen_port, socklist, reuseaddr,\n+\t\t\t\t logerror);\n+\telse {\n+\t\tint i, socknum;\n+\t\tfor (i = 0; i < listen_addr->nr; i++) {\n+\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n+\t\t\t\t\t\t listen_port, socklist, reuseaddr,\n+\t\t\t\t\t\t logerror);\n+\n+\t\t\tif (socknum == 0)\n+\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n+\t\t\t\t\t listen_addr->items[i].string, listen_port);\n+\t\t}\n+\t}\n+}\ndiff --git a/daemon-utils.h b/daemon-utils.h\nnew file mode 100644\nindex 00000000000..6710a2a6dc0\n--- /dev/null\n+++ b/daemon-utils.h\n@@ -0,0 +1,23 @@\n+#ifndef DAEMON_UTILS_H\n+#define DAEMON_UTILS_H\n+\n+#include \"git-compat-util.h\"\n+#include \"string-list.h\"\n+\n+typedef void (*log_fn)(const char *msg, ...);\n+\n+struct socketlist {\n+\tint *list;\n+\tsize_t nr;\n+\tsize_t alloc;\n+};\n+\n+/* Enable sending of keep-alive messages on the socket. */\n+void set_keep_alive(int sockfd, log_fn logerror);\n+\n+/* Setup a number of sockets to listen on the provided addresses. */\n+void socksetup(struct string_list *listen_addr, int listen_port,\n+\t struct socketlist *socklist, int reuseaddr,\n+\t log_fn logerror);\n+\n+#endif\ndiff --git a/daemon.c b/daemon.c\nindex 0ae7d12b5c1..1ed4e705680 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -1,9 +1,9 @@\n #include \"cache.h\"\n #include \"config.h\"\n+#include \"daemon-utils.h\"\n #include \"pkt-line.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n-#include \"string-list.h\"\n \n #ifdef NO_INITGROUPS\n #define initgroups(x, y) (0) /* nothing */\n@@ -737,17 +737,6 @@ static void hostinfo_clear(struct hostinfo *hi)\n \tstrbuf_release(&hi->tcp_port);\n }\n \n-static void set_keep_alive(int sockfd)\n-{\n-\tint ka = 1;\n-\n-\tif (setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &ka, sizeof(ka)) < 0) {\n-\t\tif (errno != ENOTSOCK)\n-\t\t\tlogerror(\"unable to set SO_KEEPALIVE on socket: %s\",\n-\t\t\t\tstrerror(errno));\n-\t}\n-}\n-\n static int execute(void)\n {\n \tchar *line = packet_buffer;\n@@ -759,7 +748,7 @@ static int execute(void)\n \tif (addr)\n \t\tloginfo(\"Connection from %s:%s\", addr, port);\n \n-\tset_keep_alive(0);\n+\tset_keep_alive(0, logerror);\n \talarm(init_timeout ? init_timeout : timeout);\n \tpktlen = packet_read(0, packet_buffer, sizeof(packet_buffer), 0);\n \talarm(0);\n@@ -938,202 +927,6 @@ static void child_handler(int signo)\n \tsignal(SIGCHLD, child_handler);\n }\n \n-static int set_reuse_addr(int sockfd)\n-{\n-\tint on = 1;\n-\n-\tif (!reuseaddr)\n-\t\treturn 0;\n-\treturn setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR,\n-\t\t\t &on, sizeof(on));\n-}\n-\n-struct socketlist {\n-\tint *list;\n-\tsize_t nr;\n-\tsize_t alloc;\n-};\n-\n-static const char *ip2str(int family, struct sockaddr *sin, socklen_t len)\n-{\n-#ifdef NO_IPV6\n-\tstatic char ip[INET_ADDRSTRLEN];\n-#else\n-\tstatic char ip[INET6_ADDRSTRLEN];\n-#endif\n-\n-\tswitch (family) {\n-#ifndef NO_IPV6\n-\tcase AF_INET6:\n-\t\tinet_ntop(family, &((struct sockaddr_in6*)sin)->sin6_addr, ip, len);\n-\t\tbreak;\n-#endif\n-\tcase AF_INET:\n-\t\tinet_ntop(family, &((struct sockaddr_in*)sin)->sin_addr, ip, len);\n-\t\tbreak;\n-\tdefault:\n-\t\txsnprintf(ip, sizeof(ip), \"\");\n-\t}\n-\treturn ip;\n-}\n-\n-#ifndef NO_IPV6\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tint socknum = 0;\n-\tchar pbuf[NI_MAXSERV];\n-\tstruct addrinfo hints, *ai0, *ai;\n-\tint gai;\n-\tlong flags;\n-\n-\txsnprintf(pbuf, sizeof(pbuf), \"%d\", listen_port);\n-\tmemset(&hints, 0, sizeof(hints));\n-\thints.ai_family = AF_UNSPEC;\n-\thints.ai_socktype = SOCK_STREAM;\n-\thints.ai_protocol = IPPROTO_TCP;\n-\thints.ai_flags = AI_PASSIVE;\n-\n-\tgai = getaddrinfo(listen_addr, pbuf, &hints, &ai0);\n-\tif (gai) {\n-\t\tlogerror(\"getaddrinfo() for %s failed: %s\", listen_addr, gai_strerror(gai));\n-\t\treturn 0;\n-\t}\n-\n-\tfor (ai = ai0; ai; ai = ai->ai_next) {\n-\t\tint sockfd;\n-\n-\t\tsockfd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol);\n-\t\tif (sockfd < 0)\n-\t\t\tcontinue;\n-\t\tif (sockfd >= FD_SETSIZE) {\n-\t\t\tlogerror(\"Socket descriptor too large\");\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-#ifdef IPV6_V6ONLY\n-\t\tif (ai->ai_family == AF_INET6) {\n-\t\t\tint on = 1;\n-\t\t\tsetsockopt(sockfd, IPPROTO_IPV6, IPV6_V6ONLY,\n-\t\t\t\t &on, sizeof(on));\n-\t\t\t/* Note: error is not fatal */\n-\t\t}\n-#endif\n-\n-\t\tif (set_reuse_addr(sockfd)) {\n-\t\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tset_keep_alive(sockfd);\n-\n-\t\tif (bind(sockfd, ai->ai_addr, ai->ai_addrlen) < 0) {\n-\t\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\t\tif (listen(sockfd, 5) < 0) {\n-\t\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t\t ip2str(ai->ai_family, ai->ai_addr, ai->ai_addrlen),\n-\t\t\t\t strerror(errno));\n-\t\t\tclose(sockfd);\n-\t\t\tcontinue;\t/* not fatal */\n-\t\t}\n-\n-\t\tflags = fcntl(sockfd, F_GETFD, 0);\n-\t\tif (flags >= 0)\n-\t\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\t\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\t\tsocklist->list[socklist->nr++] = sockfd;\n-\t\tsocknum++;\n-\t}\n-\n-\tfreeaddrinfo(ai0);\n-\n-\treturn socknum;\n-}\n-\n-#else /* NO_IPV6 */\n-\n-static int setup_named_sock(char *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tstruct sockaddr_in sin;\n-\tint sockfd;\n-\tlong flags;\n-\n-\tmemset(&sin, 0, sizeof sin);\n-\tsin.sin_family = AF_INET;\n-\tsin.sin_port = htons(listen_port);\n-\n-\tif (listen_addr) {\n-\t\t/* Well, host better be an IP address here. */\n-\t\tif (inet_pton(AF_INET, listen_addr, &sin.sin_addr.s_addr) <= 0)\n-\t\t\treturn 0;\n-\t} else {\n-\t\tsin.sin_addr.s_addr = htonl(INADDR_ANY);\n-\t}\n-\n-\tsockfd = socket(AF_INET, SOCK_STREAM, 0);\n-\tif (sockfd < 0)\n-\t\treturn 0;\n-\n-\tif (set_reuse_addr(sockfd)) {\n-\t\tlogerror(\"Could not set SO_REUSEADDR: %s\", strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tset_keep_alive(sockfd);\n-\n-\tif ( bind(sockfd, (struct sockaddr *)&sin, sizeof sin) < 0 ) {\n-\t\tlogerror(\"Could not bind to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tif (listen(sockfd, 5) < 0) {\n-\t\tlogerror(\"Could not listen to %s: %s\",\n-\t\t\t ip2str(AF_INET, (struct sockaddr *)&sin, sizeof(sin)),\n-\t\t\t strerror(errno));\n-\t\tclose(sockfd);\n-\t\treturn 0;\n-\t}\n-\n-\tflags = fcntl(sockfd, F_GETFD, 0);\n-\tif (flags >= 0)\n-\t\tfcntl(sockfd, F_SETFD, flags | FD_CLOEXEC);\n-\n-\tALLOC_GROW(socklist->list, socklist->nr + 1, socklist->alloc);\n-\tsocklist->list[socklist->nr++] = sockfd;\n-\treturn 1;\n-}\n-\n-#endif\n-\n-static void socksetup(struct string_list *listen_addr, int listen_port, struct socketlist *socklist)\n-{\n-\tif (!listen_addr->nr)\n-\t\tsetup_named_sock(NULL, listen_port, socklist);\n-\telse {\n-\t\tint i, socknum;\n-\t\tfor (i = 0; i < listen_addr->nr; i++) {\n-\t\t\tsocknum = setup_named_sock(listen_addr->items[i].string,\n-\t\t\t\t\t\t listen_port, socklist);\n-\n-\t\t\tif (socknum == 0)\n-\t\t\t\tlogerror(\"unable to allocate any listen sockets for host %s on port %u\",\n-\t\t\t\t\t listen_addr->items[i].string, listen_port);\n-\t\t}\n-\t}\n-}\n-\n static int service_loop(struct socketlist *socklist)\n {\n \tstruct pollfd *pfd;\n@@ -1246,7 +1039,8 @@ static int serve(struct string_list *listen_addr, int listen_port,\n {\n \tstruct socketlist socklist = { NULL, 0, 0 };\n \n-\tsocksetup(listen_addr, listen_port, &socklist);\n+\tsocksetup(listen_addr, listen_port, &socklist, reuseaddr,\n+\t\t logerror);\n \tif (socklist.nr == 0)\n \t\tdie(\"unable to allocate any listen sockets on port %u\",\n \t\t listen_port);\n-- \ngitgitgadget\n\n"},{"id":"470817","messageId":"43f1cdcbb82022521558dc649213eb4538364870.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 06/12] test-http-server: add HTTP request parsing","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:44Z","receivedAt":"2023-01-20T22:09:15Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd ability to parse HTTP requests to the test-http-server test helper.\nIntroduce `struct req` to store request information including:\n\n * HTTP method & version\n * Request path and query parameters\n * Headers\n * Content type and length (from `Content-Type` and `-Length` headers)\n\nFailure to parse the request results in a 400 Bad Request response to\nthe client. Note that we're not trying to support all possible requests\nhere, but just enough to exercise all code under test.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 202 +++++++++++++++++++++++++++++++++++-\n t/t5556-http-auth.sh | 90 ++++++++++++++++\n 2 files changed, 290 insertions(+), 2 deletions(-)\n create mode 100755 t/t5556-http-auth.sh\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 7ca4ddc7999..900f5733cc1 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -81,8 +81,53 @@ enum worker_result {\n \t * Close the socket and clean up. Does not imply an error.\n \t */\n \tWR_HANGUP = 2,\n+\n+\t/*\n+\t * Unexpected request message or error in request parsing.\n+\t * Respond with an 400 error. Close the socket and cleanup.\n+\t * Exit child-process with a non-zero status.\n+\t */\n+\tWR_CLIENT_ERROR = 3,\n+};\n+\n+/*\n+ * Fields from a parsed HTTP request.\n+ */\n+struct req {\n+\tstruct strbuf start_line;\n+\n+\tconst char *method;\n+\tconst char *http_version;\n+\n+\tstruct strbuf uri_path;\n+\tstruct strbuf query_args;\n+\n+\tstruct string_list header_list;\n+\tconst char *content_type;\n+\tuintmax_t content_length;\n+\tunsigned has_content_length:1;\n };\n \n+#define REQ__INIT { \\\n+\t.start_line = STRBUF_INIT, \\\n+\t.uri_path = STRBUF_INIT, \\\n+\t.query_args = STRBUF_INIT, \\\n+\t.header_list = STRING_LIST_INIT_NODUP, \\\n+\t.content_type = NULL, \\\n+\t.content_length = 0, \\\n+\t.has_content_length = 0, \\\n+}\n+\n+static void req__release(struct req *req)\n+{\n+\tstrbuf_release(&req->start_line);\n+\n+\tstrbuf_release(&req->uri_path);\n+\tstrbuf_release(&req->query_args);\n+\n+\tstring_list_clear(&req->header_list, 0);\n+}\n+\n static enum worker_result send_http_error(int fd, int http_code,\n \t\t\t\t\t const char *http_code_name,\n \t\t\t\t\t int retry_after_seconds,\n@@ -143,8 +188,150 @@ done:\n \treturn wr;\n }\n \n+/*\n+ * Read the HTTP request up to the start of the optional message-body.\n+ * We do this byte-by-byte because we have keep-alive turned on and\n+ * cannot rely on an EOF.\n+ *\n+ * https://tools.ietf.org/html/rfc7230\n+ *\n+ * We cannot call die() here because our caller needs to properly\n+ * respond to the client and/or close the socket before this\n+ * child exits so that the client doesn't get a connection reset\n+ * by peer error.\n+ */\n+static enum worker_result req__read(struct req *req, int fd)\n+{\n+\tstruct strbuf h = STRBUF_INIT;\n+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n+\tint nr_start_line_fields;\n+\tconst char *uri_target;\n+\tconst char *query;\n+\tchar *hp;\n+\tconst char *hv;\n+\n+\tenum worker_result result = WR_OK;\n+\n+\t/*\n+\t * Read line 0 of the request and split it into component parts:\n+\t *\n+\t * SP SP CRLF\n+\t *\n+\t */\n+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n+\t\tresult = WR_HANGUP;\n+\t\tgoto done;\n+\t}\n+\n+\tstrbuf_trim_trailing_newline(&req->start_line);\n+\n+\tnr_start_line_fields = string_list_split(&start_line_fields,\n+\t\t\t\t\t\t req->start_line.buf,\n+\t\t\t\t\t\t ' ', -1);\n+\tif (nr_start_line_fields != 3) {\n+\t\tlogerror(\"could not parse request start-line '%s'\",\n+\t\t\t req->start_line.buf);\n+\t\tresult = WR_CLIENT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\treq->method = xstrdup(start_line_fields.items[0].string);\n+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n+\n+\turi_target = start_line_fields.items[1].string;\n+\n+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n+\t\t\t req->http_version);\n+\t\tresult = WR_CLIENT_ERROR;\n+\t\tgoto done;\n+\t}\n+\n+\tquery = strchr(uri_target, '?');\n+\n+\tif (query) {\n+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t\tstrbuf_addstr(&req->query_args, query + 1);\n+\t} else {\n+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n+\t}\n+\n+\t/*\n+\t * Read the set of HTTP headers into a string-list.\n+\t */\n+\twhile (1) {\n+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n+\t\t\tgoto done;\n+\t\tstrbuf_trim_trailing_newline(&h);\n+\n+\t\tif (!h.len)\n+\t\t\tgoto done; /* a blank line ends the header */\n+\n+\t\thp = strbuf_detach(&h, NULL);\n+\t\tstring_list_append(&req->header_list, hp);\n+\n+\t\t/* also store common request headers as struct req members */\n+\t\tif (skip_iprefix(hp, \"Content-Type: \", &hv)) {\n+\t\t\treq->content_type = hv;\n+\t\t} else if (skip_iprefix(hp, \"Content-Length: \", &hv)) {\n+\t\t\t/*\n+\t\t\t * Content-Length is always non-negative, but has no\n+\t\t\t * upper bound according to RFC 7230 (§3.3.2).\n+\t\t\t */\n+\t\t\tintmax_t len = 0;\n+\t\t\tif (sscanf(hv, \"%\"PRIdMAX, &len) != 1 || len < 0 ||\n+\t\t\t len == INTMAX_MAX) {\n+\t\t\t\tlogerror(\"invalid content-length: '%s'\", hv);\n+\t\t\t\tresult = WR_CLIENT_ERROR;\n+\t\t\t\tgoto done;\n+\t\t\t}\n+\n+\t\t\treq->content_length = (uintmax_t)len;\n+\t\t\treq->has_content_length = 1;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * We do not attempt to read the , if it exists.\n+\t * We let our caller read/chunk it in as appropriate.\n+\t */\n+\n+done:\n+\tstring_list_clear(&start_line_fields, 0);\n+\n+\t/*\n+\t * This is useful for debugging the request, but very noisy.\n+\t */\n+\tif (trace2_is_enabled()) {\n+\t\tstruct string_list_item *item;\n+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n+\t\tif (req->has_content_length)\n+\t\t\ttrace2_printf(\"%s: clen: %\"PRIuMAX, TR2_CAT,\n+\t\t\t\t req->content_length);\n+\t\tif (req->content_type)\n+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n+\t\tfor_each_string_list_item(item, &req->header_list)\n+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n+\t}\n+\n+\treturn result;\n+}\n+\n+static enum worker_result dispatch(struct req *req)\n+{\n+\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n+\t\t\t WR_HANGUP);\n+}\n+\n static enum worker_result worker(void)\n {\n+\tstruct req req = REQ__INIT;\n \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n \tchar *client_port = getenv(\"REMOTE_PORT\");\n \tenum worker_result wr = WR_OK;\n@@ -155,9 +342,20 @@ static enum worker_result worker(void)\n \tset_keep_alive(0, logerror);\n \n \twhile (1) {\n-\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n-\t\t\t\t NULL, WR_HANGUP);\n+\t\treq__release(&req);\n+\n+\t\talarm(timeout);\n+\t\twr = req__read(&req, 0);\n+\t\talarm(0);\n+\n+\t\tif (wr == WR_CLIENT_ERROR)\n+\t\t\twr = send_http_error(STDOUT_FILENO, 400, \"Bad Request\",\n+\t\t\t\t\t -1, NULL, wr);\n+\n+\t\tif (wr != WR_OK)\n+\t\t\tbreak;\n \n+\t\twr = dispatch(&req);\n \t\tif (wr != WR_OK)\n \t\t\tbreak;\n \t}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nnew file mode 100755\nindex 00000000000..06efc85ca53\n--- /dev/null\n+++ b/t/t5556-http-auth.sh\n@@ -0,0 +1,90 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+TEST_NO_CREATE_REPO=1\n+. ./test-lib.sh\n+\n+# Setup a repository\n+#\n+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n+\n+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n+\n+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n+\n+test_expect_success 'setup repos' '\n+\ttest_create_repo \"$REPO_DIR\" &&\n+\tgit -C \"$REPO_DIR\" branch -M main\n+'\n+\n+run_http_server_worker() {\n+\t(\n+\t\tcd \"$REPO_DIR\"\n+\t\ttest-http-server --worker \"$@\" 2>\"$SERVER_LOG\" | tr -d \"\\r\"\n+\t)\n+}\n+\n+per_test_cleanup () {\n+\trm -f OUT.* &&\n+\trm -f IN.* &&\n+}\n+\n+test_expect_success 'http auth server request parsing' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tallowAnonymous = true\n+\tEOF\n+\n+\techo \"HTTP/1.1 400 Bad Request\" >OUT.http400 &&\n+\techo \"HTTP/1.1 200 OK\" >OUT.http200 &&\n+\n+\tcat >IN.http.valid <<-EOF &&\n+\tGET /info/refs HTTP/1.1\n+\tContent-Length: 0\n+\tEOF\n+\n+\tcat >IN.http.badfirstline <<-EOF &&\n+\t/info/refs GET HTTP\n+\tEOF\n+\n+\tcat >IN.http.badhttpver <<-EOF &&\n+\tGET /info/refs HTTP/999.9\n+\tEOF\n+\n+\tcat >IN.http.ltzlen <<-EOF &&\n+\tGET /info/refs HTTP/1.1\n+\tContent-Length: -1\n+\tEOF\n+\n+\tcat >IN.http.badlen <<-EOF &&\n+\tGET /info/refs HTTP/1.1\n+\tContent-Length: not-a-number\n+\tEOF\n+\n+\tcat >IN.http.overlen <<-EOF &&\n+\tGET /info/refs HTTP/1.1\n+\tContent-Length: 9223372036854775807\n+\tEOF\n+\n+\trun_http_server_worker \\\n+\t\t--auth-config=\"$TRASH_DIRECTORY/auth.config\" OUT.actual &&\n+\ttest_cmp OUT.http200 OUT.actual &&\n+\n+\trun_http_server_worker OUT.actual &&\n+\ttest_cmp OUT.http400 OUT.actual &&\n+\n+\trun_http_server_worker OUT.actual &&\n+\ttest_cmp OUT.http400 OUT.actual &&\n+\n+\trun_http_server_worker OUT.actual &&\n+\ttest_cmp OUT.http400 OUT.actual &&\n+\n+\trun_http_server_worker OUT.actual &&\n+\ttest_cmp OUT.http400 OUT.actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"470818","messageId":"b8d3e81b5534148359c7e92807cf1e2795480ddf.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 08/12] test-http-server: add simple authentication","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:46Z","receivedAt":"2023-01-20T22:09:17Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd simple authentication to the test-http-server test helper.\nAuthentication schemes and sets of valid tokens can be specified via\na configuration file (in the normal gitconfig file format).\nIncoming requests are compared against the set of valid schemes and\ntokens and only approved if a matching token is found, or if no auth\nwas provided and anonymous auth is enabled.\n\nConfiguration for auth includes a simple set of three options:\n\n[auth]\n\tchallenge = [:]\n\ttoken = :[]*\n\tallowAnonymous = \n\n`auth.challenge` allows you define what authentication schemes, and\noptional challenge parameters the server should use. Scheme names are\nunique and subsequently specified challenge parameters in the config\nfile will replace previously specified ones.\n\n`auth.token` allows you to define the set of value token values for an\nauthentication scheme. This is a multi-var and each entry in the\nconfig file will append to the set of valid tokens for that scheme.\nSpecifying an empty token value will clear the list of tokens so far for\nthat scheme, i.e. `token = :`.\n\n`auth.allowAnonymous` controls whether or not unauthenticated requests\n(those without any `Authorization` headers) should succeed or not, and\ntrigger a 401 Unauthorized response.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 232 +++++++++++++++++++++++++++++++++++-\n t/t5556-http-auth.sh | 43 ++++++-\n 2 files changed, 272 insertions(+), 3 deletions(-)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 4191daf3c64..72c6cca7e5c 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -7,6 +7,7 @@\n #include \"version.h\"\n #include \"dir.h\"\n #include \"date.h\"\n+#include \"config.h\"\n \n #define TR2_CAT \"test-http-server\"\n \n@@ -19,6 +20,7 @@ static const char test_http_auth_usage[] =\n \" [--timeout=] [--max-connections=]\\n\"\n \" [--reuseaddr] [--pid-file=]\\n\"\n \" [--listen=]* [--port=]\\n\"\n+\" [--auth-config=]\\n\"\n ;\n \n static unsigned int timeout;\n@@ -349,7 +351,7 @@ static int is_git_request(struct req *req)\n \t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n }\n \n-static enum worker_result do__git(struct req *req)\n+static enum worker_result do__git(struct req *req, const char *user)\n {\n \tconst char *ok = \"HTTP/1.1 200 OK\\r\\n\";\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -366,10 +368,16 @@ static enum worker_result do__git(struct req *req)\n \t * exit status of the process, then write the HTTP status line followed\n \t * by the http-backend output. This is outside of the scope of this test\n \t * helper's use at time of writing.\n+\t *\n+\t * The important auth responses (401) we are handling prior to getting\n+\t * to this point.\n \t */\n \tif (write(STDOUT_FILENO, ok, strlen(ok)) < 0)\n \t\treturn error(_(\"could not send '%s'\"), ok);\n \n+\tif (user)\n+\t\tstrvec_pushf(&cp.env, \"REMOTE_USER=%s\", user);\n+\n \tstrvec_pushf(&cp.env, \"REQUEST_METHOD=%s\", req->method);\n \tstrvec_pushf(&cp.env, \"PATH_TRANSLATED=%s\", req->uri_path.buf);\n \tstrvec_push(&cp.env, \"SERVER_PROTOCOL=HTTP/1.1\");\n@@ -388,10 +396,217 @@ static enum worker_result do__git(struct req *req)\n \treturn !!res;\n }\n \n+enum auth_result {\n+\t/* No auth module matches the request. */\n+\tAUTH_UNKNOWN = 0,\n+\n+\t/* Auth module denied the request. */\n+\tAUTH_DENY = 1,\n+\n+\t/* Auth module successfully validated the request. */\n+\tAUTH_ALLOW = 2,\n+};\n+\n+struct auth_module {\n+\tchar *scheme;\n+\tchar *challenge_params;\n+\tstruct string_list *tokens;\n+};\n+\n+static int allow_anonymous;\n+static struct auth_module **auth_modules = NULL;\n+static size_t auth_modules_nr = 0;\n+static size_t auth_modules_alloc = 0;\n+\n+static struct auth_module *get_auth_module(const char *scheme, int create)\n+{\n+\tstruct auth_module *mod;\n+\tfor (size_t i = 0; i < auth_modules_nr; i++) {\n+\t\tmod = auth_modules[i];\n+\t\tif (!strcasecmp(mod->scheme, scheme))\n+\t\t\treturn mod;\n+\t}\n+\n+\tif (create) {\n+\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n+\t\tmod->scheme = xstrdup(scheme);\n+\t\tmod->challenge_params = NULL;\n+\t\tALLOC_ARRAY(mod->tokens, 1);\n+\t\tstring_list_init_dup(mod->tokens);\n+\n+\t\tALLOC_GROW(auth_modules, auth_modules_nr + 1, auth_modules_alloc);\n+\t\tauth_modules[auth_modules_nr++] = mod;\n+\n+\t\treturn mod;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n+static int is_authed(struct req *req, const char **user, enum worker_result *wr)\n+{\n+\tenum auth_result result = AUTH_UNKNOWN;\n+\tstruct string_list hdrs = STRING_LIST_INIT_NODUP;\n+\tstruct auth_module *mod;\n+\n+\tstruct string_list_item *hdr;\n+\tstruct string_list_item *token;\n+\tconst char *v;\n+\tstruct strbuf **split = NULL;\n+\tint i;\n+\tchar *challenge;\n+\n+\t/*\n+\t * Check all auth modules and try to validate the request.\n+\t * The first Authorization header that matches a known auth module\n+\t * scheme will be consulted to either approve or deny the request.\n+\t * If no module is found, or if there is no valid token, then 401 error.\n+\t * Otherwise, only permit the request if anonymous auth is enabled.\n+\t * It's atypical for user agents/clients to send multiple Authorization\n+\t * headers, but not explicitly forbidden or defined.\n+\t */\n+\tfor_each_string_list_item(hdr, &req->header_list) {\n+\t\tif (skip_iprefix(hdr->string, \"Authorization: \", &v)) {\n+\t\t\tsplit = strbuf_split_str(v, ' ', 2);\n+\t\t\tif (split[0] && split[1]) {\n+\t\t\t\t/* trim trailing space ' ' */\n+\t\t\t\tstrbuf_rtrim(split[0]);\n+\n+\t\t\t\tmod = get_auth_module(split[0]->buf, 0);\n+\t\t\t\tif (mod) {\n+\t\t\t\t\tresult = AUTH_DENY;\n+\n+\t\t\t\t\tfor_each_string_list_item(token, mod->tokens) {\n+\t\t\t\t\t\tif (!strcmp(split[1]->buf, token->string)) {\n+\t\t\t\t\t\t\tresult = AUTH_ALLOW;\n+\t\t\t\t\t\t\tbreak;\n+\t\t\t\t\t\t}\n+\t\t\t\t\t}\n+\n+\t\t\t\t\tstrbuf_list_free(split);\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tstrbuf_list_free(split);\n+\t\t}\n+\t}\n+\n+done:\n+\tswitch (result) {\n+\tcase AUTH_ALLOW:\n+\t\ttrace2_printf(\"%s: auth '%s' ALLOW\", TR2_CAT, mod->scheme);\n+\t\t*user = \"VALID_TEST_USER\";\n+\t\t*wr = WR_OK;\n+\t\tbreak;\n+\n+\tcase AUTH_DENY:\n+\t\ttrace2_printf(\"%s: auth '%s' DENY\", TR2_CAT, mod->scheme);\n+\t\t/* fall-through */\n+\n+\tcase AUTH_UNKNOWN:\n+\t\tif (result != AUTH_DENY && allow_anonymous)\n+\t\t\tbreak;\n+\n+\t\tfor (i = 0; i < auth_modules_nr; i++) {\n+\t\t\tmod = auth_modules[i];\n+\t\t\tif (mod->challenge_params)\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s %s\",\n+\t\t\t\t\t\t mod->scheme,\n+\t\t\t\t\t\t mod->challenge_params);\n+\t\t\telse\n+\t\t\t\tchallenge = xstrfmt(\"WWW-Authenticate: %s\",\n+\t\t\t\t\t\t mod->scheme);\n+\t\t\tstring_list_append(&hdrs, challenge);\n+\t\t}\n+\n+\t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n+\t\t\t\t &hdrs, *wr);\n+\t}\n+\n+\tstring_list_clear(&hdrs, 0);\n+\n+\treturn result == AUTH_ALLOW ||\n+\t (result == AUTH_UNKNOWN && allow_anonymous);\n+}\n+\n+static int split_auth_param(const char *str, char **scheme, char **val)\n+{\n+\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n+\n+\tif (!p[0])\n+\t\treturn -1;\n+\n+\t/* trim trailing ':' */\n+\tif (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')\n+\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n+\n+\t*scheme = strbuf_detach(p[0], NULL);\n+\t*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;\n+\n+\tstrbuf_list_free(p);\n+\treturn 0;\n+}\n+\n+static int read_auth_config(const char *name, const char *val, void *data)\n+{\n+\tint ret = 0;\n+\tchar *scheme = NULL;\n+\tchar *token = NULL;\n+\tchar *challenge = NULL;\n+\tstruct auth_module *mod;\n+\n+\tif (!strcmp(name, \"auth.challenge\")) {\n+\t\tif (split_auth_param(val, &scheme, &challenge)) {\n+\t\t\tret = error(\"invalid auth challenge '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = get_auth_module(scheme, 1);\n+\n+\t\t/* Replace any existing challenge parameters */\n+\t\tfree(mod->challenge_params);\n+\t\tmod->challenge_params = challenge ? xstrdup(challenge) : NULL;\n+\t} else if (!strcmp(name, \"auth.token\")) {\n+\t\tif (split_auth_param(val, &scheme, &token)) {\n+\t\t\tret = error(\"invalid auth token '%s'\", val);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n+\t\tmod = get_auth_module(scheme, 1);\n+\n+\t\t/*\n+\t\t * Append to set of valid tokens unless an empty token value\n+\t\t * is provided, then clear the existing list.\n+\t\t */\n+\t\tif (token)\n+\t\t\tstring_list_append(mod->tokens, token);\n+\t\telse\n+\t\t\tstring_list_clear(mod->tokens, 1);\n+\t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n+\t\tallow_anonymous = git_config_bool(name, val);\n+\t} else {\n+\t\twarning(\"unknown auth config '%s'\", name);\n+\t}\n+\n+cleanup:\n+\tfree(scheme);\n+\tfree(token);\n+\tfree(challenge);\n+\n+\treturn ret;\n+}\n+\n static enum worker_result dispatch(struct req *req)\n {\n+\tenum worker_result wr = WR_OK;\n+\tconst char *user = NULL;\n+\n+\tif (!is_authed(req, &user, &wr))\n+\t\treturn wr;\n+\n \tif (is_git_request(req))\n-\t\treturn do__git(req);\n+\t\treturn do__git(req, user);\n \n \treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n \t\t\t WR_HANGUP);\n@@ -655,6 +870,19 @@ int cmd_main(int argc, const char **argv)\n \t\t\tpid_file = v;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, \"--auth-config=\", &v)) {\n+\t\t\tif (!strlen(v)) {\n+\t\t\t\terror(\"invalid argument - missing file path\");\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tif (git_config_from_file(read_auth_config, v, NULL)) {\n+\t\t\t\terror(\"failed to read auth config file '%s'\", v);\n+\t\t\t\tusage(test_http_auth_usage);\n+\t\t\t}\n+\n+\t\t\tcontinue;\n+\t\t}\n \n \t\tfprintf(stderr, \"error: unknown argument '%s'\\n\", arg);\n \t\tusage(test_http_auth_usage);\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex c0a47ce342b..20fd9b09aef 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -101,6 +101,7 @@ per_test_cleanup () {\n \tstop_http_server &&\n \trm -f OUT.* &&\n \trm -f IN.* &&\n+\trm -f auth.config\n }\n \n test_expect_success 'http auth server request parsing' '\n@@ -160,11 +161,51 @@ test_expect_success 'http auth server request parsing' '\n \ttest_cmp OUT.http400 OUT.actual\n '\n \n+test_expect_success CURL 'http auth server auth config' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = no-params\n+\t\tchallenge = with-params:foo=\\\"bar\\\" p=1\n+\t\tchallenge = with-params:foo=\\\"replaced\\\" q=1\n+\n+\t\ttoken = no-explicit-challenge:valid-token\n+\t\ttoken = no-explicit-challenge:also-valid\n+\t\ttoken = reset-tokens:these-tokens\n+\t\ttoken = reset-tokens:will-be-reset\n+\t\ttoken = reset-tokens:\n+\t\ttoken = reset-tokens:the-only-valid-one\n+\n+\t\tallowAnonymous = false\n+\tEOF\n+\n+\tcat >OUT.expected <<-EOF &&\n+\tWWW-Authenticate: no-params\n+\tWWW-Authenticate: with-params foo=\"replaced\" q=1\n+\tWWW-Authenticate: no-explicit-challenge\n+\tWWW-Authenticate: reset-tokens\n+\n+\tError: 401 Unauthorized\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tcurl --include $ORIGIN_URL >OUT.curl &&\n+\ttr -d \"\\r\" OUT.actual &&\n+\n+\ttest_cmp OUT.expected OUT.actual\n+'\n \n test_expect_success 'http auth anonymous no challenge' '\n \ttest_when_finished \"per_test_cleanup\" &&\n \n-\tstart_http_server &&\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tallowAnonymous = true\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n \n \t# Attempt to read from a protected repository\n \tgit ls-remote $ORIGIN_URL\n-- \ngitgitgadget\n\n"},{"id":"470819","messageId":"2f97c94f67981dcfbbed00a9800cec4cd26c594d.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 09/12] test-http-server: add sending of arbitrary headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:47Z","receivedAt":"2023-01-20T22:09:18Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the ability to send arbitrary headers in HTTP responses from the\ntest-http-server. This is useful when we want to test 'malformed'\nresponse message handling.\n\nAdd the following option to the server auth config file:\n\n[auth]\n\textraHeader = []*\n\nEach `auth.extraHeader` value will be appended to the response headers\nverbatim.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/helper/test-http-server.c | 6 ++++++\n t/t5556-http-auth.sh | 7 +++++++\n 2 files changed, 13 insertions(+)\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 72c6cca7e5c..70bf15c3fa1 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -417,6 +417,7 @@ static int allow_anonymous;\n static struct auth_module **auth_modules = NULL;\n static size_t auth_modules_nr = 0;\n static size_t auth_modules_alloc = 0;\n+static struct strvec extra_headers = STRVEC_INIT;\n \n static struct auth_module *get_auth_module(const char *scheme, int create)\n {\n@@ -520,6 +521,9 @@ done:\n \t\t\tstring_list_append(&hdrs, challenge);\n \t\t}\n \n+\t\tfor (i = 0; i < extra_headers.nr; i++)\n+\t\t\tstring_list_append(&hdrs, extra_headers.v[i]);\n+\n \t\t*wr = send_http_error(STDOUT_FILENO, 401, \"Unauthorized\", -1,\n \t\t\t\t &hdrs, *wr);\n \t}\n@@ -585,6 +589,8 @@ static int read_auth_config(const char *name, const char *val, void *data)\n \t\t\tstring_list_clear(mod->tokens, 1);\n \t} else if (!strcmp(name, \"auth.allowanonymous\")) {\n \t\tallow_anonymous = git_config_bool(name, val);\n+\t} else if (!strcmp(name, \"auth.extraheader\")) {\n+\t\tstrvec_push(&extra_headers, val);\n \t} else {\n \t\twarning(\"unknown auth config '%s'\", name);\n \t}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 20fd9b09aef..2c16c8f72a5 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -178,6 +178,10 @@ test_expect_success CURL 'http auth server auth config' '\n \t\ttoken = reset-tokens:the-only-valid-one\n \n \t\tallowAnonymous = false\n+\n+\t\textraHeader = X-Extra-Header: abc\n+\t\textraHeader = X-Extra-Header: 123\n+\t\textraHeader = X-Another: header\\twith\\twhitespace!\n \tEOF\n \n \tcat >OUT.expected <<-EOF &&\n@@ -185,6 +189,9 @@ test_expect_success CURL 'http auth server auth config' '\n \tWWW-Authenticate: with-params foo=\"replaced\" q=1\n \tWWW-Authenticate: no-explicit-challenge\n \tWWW-Authenticate: reset-tokens\n+\tX-Extra-Header: abc\n+\tX-Extra-Header: 123\n+\tX-Another: header\twith\twhitespace!\n \n \tError: 401 Unauthorized\n \tEOF\n-- \ngitgitgadget\n\n"},{"id":"470820","messageId":"5f5e46038cf526714f3c5b89ffef2b895b503242.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:49Z","receivedAt":"2023-01-20T22:09:20Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 +++++++++\n http.c | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 110 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/http.c b/http.c\nindex a2a80318bb2..595c93bc7a3 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,98 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = st_mult(eltsize, nmemb);\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\tstrbuf_add(&buf, ptr, size);\n+\n+\t/* Strip the CRLF that should be present at the end of each field */\n+\tstrbuf_trim_trailing_newline(&buf);\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n+\t\twhile (isspace(*val))\n+\t\t\tval++;\n+\n+\t\tstrvec_push(values, val);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t * Continuation lines start with at least one whitespace, maybe more,\n+\t * so we should collapse these down to a single SP (valid per the spec).\n+\t */\n+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n+\t\t/* Trim leading whitespace from this continuation hdr line. */\n+\t\tstrbuf_ltrim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (istarts_with(buf.buf, \"http/\"))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1956,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"470821","messageId":"4b1635b3f6968f8d755bdf6bc4ec7af77aefd315.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:48Z","receivedAt":"2023-01-20T22:09:21Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nReplace direct multiplication of two size_t parameters in curl response\nstream handling callback functions with `st_mult` to guard against\noverflows.\n\nSigned-off-by: Matthew John Cheetham \n---\n http.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..a2a80318bb2 100644\n--- a/http.c\n+++ b/http.c\n@@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n \n size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct buffer *buffer = buffer_;\n \n \tif (size > buffer->buf.len - buffer->posn)\n@@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n \n size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n {\n-\tsize_t size = eltsize * nmemb;\n+\tsize_t size = st_mult(eltsize, nmemb);\n \tstruct strbuf *buffer = buffer_;\n \n \tstrbuf_add(buffer, ptr, size);\n-- \ngitgitgadget\n\n"},{"id":"470822","messageId":"09164f77d56e8efd1450091cf1b12af2bc6cf2f5.1674252531.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-01-20T22:08:50Z","receivedAt":"2023-01-20T22:09:24Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 11 ++\n t/lib-credential-helper.sh | 27 ++++\n t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++\n 4 files changed, 298 insertions(+), 1 deletion(-)\n create mode 100644 t/lib-credential-helper.sh\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..9f39ebc3c7e 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tchar *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (size_t i = 0; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree(full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/lib-credential-helper.sh b/t/lib-credential-helper.sh\nnew file mode 100644\nindex 00000000000..8b0e4414234\n--- /dev/null\n+++ b/t/lib-credential-helper.sh\n@@ -0,0 +1,27 @@\n+setup_credential_helper() {\n+\ttest_expect_success 'setup credential helper' '\n+\t\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/credential-helper.sh\" &&\n+\t\texport CREDENTIAL_HELPER &&\n+\t\techo $CREDENTIAL_HELPER &&\n+\n+\t\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\t\tcmd=$1\n+\t\tteefile=$cmd-query.cred\n+\t\tcatfile=$cmd-reply.cred\n+\t\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n+\t\tif test \"$cmd\" = \"get\"; then\n+\t\t\tcat $catfile\n+\t\tfi\n+\t\tEOF\n+\t'\n+}\n+\n+set_credential_reply() {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query() {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\ndiff --git a/t/t5556-http-auth.sh b/t/t5556-http-auth.sh\nindex 2c16c8f72a5..93b7c178da6 100755\n--- a/t/t5556-http-auth.sh\n+++ b/t/t5556-http-auth.sh\n@@ -4,6 +4,7 @@ test_description='test http auth header and credential helper interop'\n \n TEST_NO_CREATE_REPO=1\n . ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-credential-helper.sh\n \n test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n \n@@ -33,6 +34,8 @@ test_expect_success 'setup repos' '\n \tgit -C \"$REPO_DIR\" branch -M main\n '\n \n+setup_credential_helper\n+\n run_http_server_worker() {\n \t(\n \t\tcd \"$REPO_DIR\"\n@@ -101,6 +104,7 @@ per_test_cleanup () {\n \tstop_http_server &&\n \trm -f OUT.* &&\n \trm -f IN.* &&\n+\trm -f *.cred &&\n \trm -f auth.config\n }\n \n@@ -218,4 +222,242 @@ test_expect_success 'http auth anonymous no challenge' '\n \tgit ls-remote $ORIGIN_URL\n '\n \n+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\t\textraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\t\textraheader = \"WWW-Authenticate:\"\n+\t\textraheader = \" \"\n+\t\textraheader = \" bearer authority=\\\"id.example.com\\\"\"\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=basic realm=\"example.com\"\n+\twwwauth[]=bearer authority=\"id.example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"foobar:alg=test widget=1\"\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=foobar alg=test widget=1\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\t# base64(\"alice:secret-passwd\")\n+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n+\texport USERPASS64 &&\n+\n+\tcat >auth.config <<-EOF &&\n+\t[auth]\n+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n+\t\tchallenge = basic:realm=\\\"example.com\\\"\n+\t\ttoken = basic:$USERPASS64\n+\tEOF\n+\n+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\tEOF\n+\n+\ttest_must_fail git -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HOST_PORT\n+\tusername=alice\n+\tpassword=invalid-passwd\n+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n+\twwwauth[]=basic realm=\"example.com\"\n+\tEOF\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"470991","messageId":"e57c1ca3-c21c-db41-a386-e5887f46055c@github.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-01-24T17:30:02Z","receivedAt":"2023-01-24T17:30:10Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> Updates in v6\n> =============\n> \n> * Clarify the change to make logging optional in the check_dead_children()\n> function during libification of daemon.c.\n> \n> * Fix missing pointer dereference bugs identified in libification of child\n> process handling functions for daemon.c.\n> \n> * Add doc comments to child process handling function declarations in the\n> daemon-utils.h header.\n> \n> * Align function parameter names with variable names at callsites for\n> libified daemon functions.\n> \n> * Re-split out the test-http-server test helper commits in to smaller\n> patches: error response handling, request parsing, http-backend\n> pass-through, simple authentication, arbitrary header support.\n> \n> * Call out auth configuration file format for test-http-server test helper\n> and supported options in commit messages, as well as a test to exercise\n> and demonstrate these options.\n> \n> * Permit auth.token and auth.challenge to appear in any order; create the\n> struct auth_module just-in-time as options for that scheme are read. This\n> simplifies the configuration authoring of the test-http-server test\n> helper.\n> \n> * Update tests to use auth.allowAnoymous in the patch that introduces the\n> new test helper option.\n> \n> * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n> request header line folding to use xstrdup and strvec_pop and _pushf.\n> \n> * Use size_t instead of int in credential.c when iterating over the struct\n> strvec credential members. Also drop the not required const and cast from\n> the full_key definition and free.\n> \n> * Replace in-tree test-credential-helper-reply.sh test cred helper script\n> with the lib-credential-helper.sh reusable 'lib' test script and shell\n> functions to configure the helper behaviour.\n> \n> * Leverage sed over the while read $line loop in the test credential helper\n> script.\n> \n> \n> Updates in v7\n> =============\n> \n> * Address several whitespace and arg/param list alignment issues.\n> \n> * Rethink the test-http-helper worker-mode error and result enum to be more\n> simple and more informative to the nature of the error.\n> \n> * Use uintmax_t to store the Content-Length of a request in the helper\n> test-http-server. Maintain a bit flag to store if we received such a\n> header.\n> \n> * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n> in the test-http-server.\n> \n> * Add test case to cover request message parsing in test-http-server.\n> \n> * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n> get_auth_module.\n> \n> * Correctly free the split strbufs created in the header parsing loop in\n> test-http-server.\n> \n> * Avoid needless comparison > 0 for unsigned types.\n> \n> * Always set optional outputs to NULL if not present in test helper config\n> value handling.\n> \n> * Remove an accidentally commented-out test cleanup line for one test case\n> in t5556.\nI've re-read the patches in this version; all of my comments from v5 have\nbeen addressed, and the additional updates w.r.t. other reviewer feedback\nall look good as well. At this point, I think the series is ready for\n'next'.\n\nThanks!\n\n"},{"id":"470994","messageId":"xmqqwn5bg695.fsf@gitster.g","threadId":"58425","inReplyTo":"e57c1ca3-c21c-db41-a386-e5887f46055c@github.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-24T18:03:02Z","receivedAt":"2023-01-24T18:03:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Victoria Dye writes:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> Updates in v6\n>> =============\n>> ...\n> I've re-read the patches in this version; all of my comments from v5 have\n> been addressed, and the additional updates w.r.t. other reviewer feedback\n> all look good as well. At this point, I think the series is ready for\n> 'next'.\n>\n> Thanks!\n\nThanks, both. Let's merge it down.\n"},{"id":"471053","messageId":"Y9JA0UCRh7qUqKQI@coredump.intra.peff.net","threadId":"58425","inReplyTo":"17c890ee1080abc81267e44a1eaff4609ee41690.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 04/12] test-http-server: add stub HTTP server test helper","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T08:58:57Z","receivedAt":"2023-01-26T08:59:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:42PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> Introduce a mini HTTP server helper that in the future will be enhanced\n> to provide a frontend for the git-http-backend, with support for\n> arbitrary authentication schemes.\n> \n> Right now, test-http-server is a pared-down copy of the git-daemon that\n> always returns a 501 Not Implemented response to all callers.\n\nThis may be a dumb question, but I didn't see it raised or answered in\nthe cover letter or earlier in the thread: what does this custom server\ngive us that our current use of apache in the tests does not?\n\nI'd imagine the answer is along the lines of: configuring apache to\nrespond to auth in the way we'd like is hard and/or impossible. And if\nso, and if it's just \"hard\", I'd ask \"how hard?\".\n\nBecause I see a few downsides to introducing a custom server here:\n\n 1. It may or may not behave like real-world servers, which makes the\n test slightly less good. Not that apache can claim to cover all\n real-world behavior, but it's probably closer to reality (and that\n has flushed out interesting bugs and behaviors before).\n\n 2. It's a non-trivial amount of code, doing tricky things like\n daemonizing, socket setup and I/O, pidfiles, and so on. For\n example, it handles multiple listen addresses, and ipv6. Do we\n really need that? And we take shortcuts around things like CGI\n output buffering. I do see that you tried to reuse some existing\n code, but...\n\n 3. You're reusing parts of git-daemon, which I personally consider to\n be one of the absolute low-points of code quality inside git.git. I\n know that's a subjective statement, but my experience running it\n within GitHub was that there were a lot of rough edges, and we\n ended up rewriting several parts of it. In particular, the\n child-handling is inefficient (I seem to recall that it's quadratic\n in several places) and has odd behaviors (its kill_some_child() is\n basically nonsense, and can starve requests). Probably none of that\n matters for your use case in tests, which is likely doing one\n request at a time.\n\n But then I'd wonder: do we really need those bits at all, then? In\n fact, would it be sufficient to write the server to handle one\n request at a time, without spawning a worker child at all?\n\nI dunno. I know I am showing up to review quite late in the life of this\npatch series, and that probably makes me a bad person to start the\nreview with \"and could you re-do the whole test infrastructure\". So if\nyou want to tell me to get lost, I'd understand. But I had hoped that\none day we could just delete all of daemon.c, and this moves in the\nopposite direction.\n\nI think my order of preference (if you care ;) ) is:\n\n 1. Can we do it with apache?\n\n 2. If not, could we do it with a trivial application of some existing\n http server framework? I know that may mean extra dependencies, but\n there's a lot of perl in the test suite already, and it doesn't\n seem too terrible to me to require it for these tests.\n\n 3. If not, can we make the http-server code even more minimal?\n\n> Makefile | 1 +\n> contrib/buildsystems/CMakeLists.txt | 11 +-\n> t/helper/.gitignore | 1 +\n> t/helper/test-http-server.c | 381 ++++++++++++++++++++++++++++\n> 4 files changed, 392 insertions(+), 2 deletions(-)\n> create mode 100644 t/helper/test-http-server.c\n\nIf we do use this code, here are a few small bits I noticed:\n\n> +static void child_handler(int signo)\n> +{\n> +\t/*\n> +\t * Otherwise empty handler because systemcalls will get interrupted\n> +\t * upon signal receipt\n> +\t * SysV needs the handler to be rearmed\n> +\t */\n> +\tsignal(SIGCHLD, child_handler);\n> +}\n\ndaemon.c has this, too. If we're going to share its child-handling code,\nshould it maybe just handle this part, too?\n\n> +static int service_loop(struct socketlist *socklist)\n> +{\n> +\tstruct pollfd *pfd;\n> +\tint i;\n> +\n> +\tCALLOC_ARRAY(pfd, socklist->nr);\n\n(Actually, Coverity noticed this, not me).\n\nThis pfd is never freed. I know this is copied from daemon.c, but in\nthat file we never return from the function. Here you do break out of\nthe loop and try to clean up; you'd want to free(pfd) there.\n\n> +\tfor (;;) {\n> +\t\tint i;\n> +\t\tint nr_ready;\n> +\t\tint timeout = (pid_file ? 100 : -1);\n> +\n> +\t\tcheck_dead_children(first_child, &live_children, loginfo);\n> +\n> +\t\tnr_ready = poll(pfd, socklist->nr, timeout);\n> +\t\tif (nr_ready < 0) {\n> +\t\t\tif (errno != EINTR) {\n> +\t\t\t\tlogerror(\"Poll failed, resuming: %s\",\n> +\t\t\t\t strerror(errno));\n> +\t\t\t\tsleep(1);\n> +\t\t\t}\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\telse if (nr_ready == 0) {\n> +\t\t\t/*\n> +\t\t\t * If we have a pid_file, then we watch it.\n> +\t\t\t * If someone deletes it, we shutdown the service.\n> +\t\t\t * The shell scripts in the test suite will use this.\n> +\t\t\t */\n> +\t\t\tif (!pid_file || file_exists(pid_file))\n> +\t\t\t\tcontinue;\n> +\t\t\tgoto shutdown;\n> +\t\t}\n\nI wondered how this would work, since removal of the pid file won't\ntrigger poll(). But it looks like you set the timeout unconditionally in\nthat case, so we're effectively polling for its removal every 100ms.\nIt's not beautiful, but it should work reliably.\n\nThat also made me wonder about this timeout:\n\n> +\t\tif (skip_prefix(arg, \"--timeout=\", &v)) {\n> +\t\t\ttimeout = atoi(v);\n> +\t\t\tcontinue;\n> +\t\t}\n\nbut it is not used. The \"timeout\" in service_loop shadows the global,\nand nobody ever looks at the global (however, it looks like a later\npatch adds an alarm() which uses it).\n\n> +\t\tif (skip_prefix(arg, \"--max-connections=\", &v)) {\n> +\t\t\tmax_connections = atoi(v);\n> +\t\t\tif (max_connections < 0)\n> +\t\t\t\tmax_connections = 0; /* unlimited */\n> +\t\t\tcontinue;\n> +\t\t}\n\nI don't think any caller ever uses --max-connections, though. This could\nbe dropped, and that would simplify service_loop a bit.\n\n> +\tif (listen_port == 0)\n> +\t\tlisten_port = DEFAULT_GIT_PORT;\n\nThat's a funny default. Surely \"80\" or even \"8080\" would make more\nsense. But really, since our purpose is tests, isn't it a\nmisconfiguration if the test does not tell us which port (which is\ngenerally dynamic based on the test number), and we should bail?\n\n> +\t/*\n> +\t * If no --listen= args are given, the setup_named_sock()\n> +\t * code will use receive a NULL address and set INADDR_ANY.\n> +\t * This exposes both internal and external interfaces on the\n> +\t * port.\n> +\t *\n> +\t * Disallow that and default to the internal-use-only loopback\n> +\t * address.\n> +\t */\n> +\tif (!listen_addr.nr)\n> +\t\tstring_list_append(&listen_addr, \"127.0.0.1\");\n\nLikewise, it seems like you could probably ditch --listen entirely, and\njust always listen on 127.0.0.1, for the purposes of the tests.\n\n-Peff\n"},{"id":"471056","messageId":"Y9JIHV7et/8bMvZY@coredump.intra.peff.net","threadId":"58425","inReplyTo":"43f1cdcbb82022521558dc649213eb4538364870.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 06/12] test-http-server: add HTTP request parsing","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T09:30:05Z","receivedAt":"2023-01-26T09:30:18Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:44PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> +#define REQ__INIT { \\\n> + .start_line = STRBUF_INIT, \\\n> + .uri_path = STRBUF_INIT, \\\n> + .query_args = STRBUF_INIT, \\\n> + .header_list = STRING_LIST_INIT_NODUP, \\\n> + .content_type = NULL, \\\n> + .content_length = 0, \\\n> + .has_content_length = 0, \\\n> +}\n\nWe declare header_list as nodup, but later we put actual duplicated\nstrings in it:\n\n> + hp = strbuf_detach(&h, NULL);\n> + string_list_append(&req->header_list, hp);\n\nSo later when we free it:\n\n> +static void req__release(struct req *req)\n> +{\n> + strbuf_release(&req->start_line);\n> +\n> + strbuf_release(&req->uri_path);\n> + strbuf_release(&req->query_args);\n> +\n> + string_list_clear(&req->header_list, 0);\n> +}\n\nthe strings will be leaked. There are a lot of solutions here, including\nsetting strdup_strings right before freeing. But it's probably\nreasonable to just use INIT_DUP, and then when storing, just do:\n\n string_list_append(&req->header_list, h.buf);\n\nSince \"h\" is filled by strbuf_getwholeline(), there's no need to erase\nthe contents. It should reset the buffer itself (and so you end up\nre-using the same buffer, rather than freeing it for each loop). You\nwill have to remember to strbuf_release() after the loop, though.\n\nThe leak isn't very big, and we hold onto it until the process ends\nanyway, but it will probably cause the leak-detector to complain.\n\n(Yet another solution would just be to dump the trace as we parse the\nheaders, rather than holding them, since that appears to be the only use\nof header_list).\n\n> +\t/*\n> +\t * Read the set of HTTP headers into a string-list.\n> +\t */\n> +\twhile (1) {\n> +\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n> +\t\t\tgoto done;\n> +\t\tstrbuf_trim_trailing_newline(&h);\n> +\n> +\t\tif (!h.len)\n> +\t\t\tgoto done; /* a blank line ends the header */\n> +\n> +\t\thp = strbuf_detach(&h, NULL);\n> +\t\tstring_list_append(&req->header_list, hp);\n> +\n> +\t\t/* also store common request headers as struct req members */\n> +\t\tif (skip_iprefix(hp, \"Content-Type: \", &hv)) {\n> +\t\t\treq->content_type = hv;\n\nI think this is stricter than necessary. The whitespace after the colon\nis optional, but can also be longer than just one space (or could be a\ntab). It's probably OK to be picky here since this is just for tests,\nbut we'd want to make sure we're not this picky on the client side.\n\n> +\t\t} else if (skip_iprefix(hp, \"Content-Length: \", &hv)) {\n> +\t\t\t/*\n> +\t\t\t * Content-Length is always non-negative, but has no\n> +\t\t\t * upper bound according to RFC 7230 (§3.3.2).\n> +\t\t\t */\n> +\t\t\tintmax_t len = 0;\n> +\t\t\tif (sscanf(hv, \"%\"PRIdMAX, &len) != 1 || len < 0 ||\n> +\t\t\t len == INTMAX_MAX) {\n> +\t\t\t\tlogerror(\"invalid content-length: '%s'\", hv);\n> +\t\t\t\tresult = WR_CLIENT_ERROR;\n> +\t\t\t\tgoto done;\n> +\t\t\t}\n\nWe usually avoid sscanf because it's error-checking sucks. For example,\nthis will accept \"123.garbage\", but you can't tell because you have no\nclue how far it got. Something like strtoimax() is better. It probably\ndoesn't matter much since this is test code, though I do think in the\nlong run it would be nice to add scanf(), etc, to our list of banned\nfunctions (there are one or two other uses currently, though, so that\nisn't imminent).\n\n-Peff\n"},{"id":"471058","messageId":"Y9JJ2moUulG8gTba@coredump.intra.peff.net","threadId":"58425","inReplyTo":"ca9c2787248688cd7d8e20043a6ed75d93654e35.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 07/12] test-http-server: pass Git requests to http-backend","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T09:37:30Z","receivedAt":"2023-01-26T09:37:35Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:45PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> +static int is_git_request(struct req *req)\n> +{\n> +\tstatic regex_t *smart_http_regex;\n> +\tstatic int initialized;\n> +\n> +\tif (!initialized) {\n> +\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n> +\t\t/*\n> +\t\t * This regular expression matches all dumb and smart HTTP\n> +\t\t * requests that are currently in use, and defined in\n> +\t\t * Documentation/gitprotocol-http.txt.\n> +\t\t *\n> +\t\t */\n> +\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n> +\t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n> +\t\t\t REG_EXTENDED)) {\n> +\t\t\twarning(\"could not compile smart HTTP regex\");\n> +\t\t\tsmart_http_regex = NULL;\n> +\t\t}\n> +\t\tinitialized = 1;\n> +\t}\n> +\n> +\treturn smart_http_regex &&\n> +\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n> +}\n\nAssigning NULL to smart_http_regex leaks the earlier allocation. You\ncould free it, but I have to wonder why it is on the heap in the first\nplace. Yes, you check for NULL and return 0 if it failed to compile,\nbut...why would it? It's hard-coded. And if it does fail, wouldn't you\nwant to fail immediately and loudly, because it means all of the tests\nare broken?\n\nI.e., something like this is a bit simpler:\n\ndiff --git a/t/helper/test-http-server.c b/t/helper/test-http-server.c\nindex 14d170e640..8048ba1636 100644\n--- a/t/helper/test-http-server.c\n+++ b/t/helper/test-http-server.c\n@@ -327,28 +327,25 @@ static enum worker_result req__read(struct req *req, int fd)\n \n static int is_git_request(struct req *req)\n {\n-\tstatic regex_t *smart_http_regex;\n+\tstatic regex_t smart_http_regex;\n \tstatic int initialized;\n \n \tif (!initialized) {\n-\t\tsmart_http_regex = xmalloc(sizeof(*smart_http_regex));\n \t\t/*\n \t\t * This regular expression matches all dumb and smart HTTP\n \t\t * requests that are currently in use, and defined in\n \t\t * Documentation/gitprotocol-http.txt.\n \t\t *\n \t\t */\n-\t\tif (regcomp(smart_http_regex, \"^/(HEAD|info/refs|\"\n+\t\tif (regcomp(&smart_http_regex, \"^/(HEAD|info/refs|\"\n \t\t\t \"objects/info/[^/]+|git-(upload|receive)-pack)$\",\n \t\t\t REG_EXTENDED)) {\n-\t\t\twarning(\"could not compile smart HTTP regex\");\n-\t\t\tsmart_http_regex = NULL;\n+\t\t\tdie(\"could not compile smart HTTP regex\");\n \t\t}\n \t\tinitialized = 1;\n \t}\n \n-\treturn smart_http_regex &&\n-\t\t!regexec(smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n+\treturn !regexec(&smart_http_regex, req->uri_path.buf, 0, NULL, 0);\n }\n \n static enum worker_result do__git(struct req *req, const char *user)\n\n> +start_http_server () {\n> +\t#\n> +\t# Launch our server into the background in repo_dir.\n> +\t#\n> +\t(\n> +\t\tcd \"$REPO_DIR\"\n> +\t\ttest-http-server --verbose \\\n> +\t\t\t--listen=127.0.0.1 \\\n> +\t\t\t--port=$GIT_TEST_HTTP_PROTOCOL_PORT \\\n> +\t\t\t--reuseaddr \\\n> +\t\t\t--pid-file=\"$PID_FILE\" \\\n> +\t\t\t\"$@\" \\\n> +\t\t\t2>\"$SERVER_LOG\" &\n> +\t)\n> +\t#\n> +\t# Give it a few seconds to get started.\n> +\t#\n> +\tfor k in 0 1 2 3 4\n> +\tdo\n> +\t\tif test -f \"$PID_FILE\"\n> +\t\tthen\n> +\t\t\treturn 0\n> +\t\tfi\n> +\t\tsleep 1\n> +\tdone\n\nYuck. This makes the test take a long time to run, since it will almost\nalways \"sleep 1\" each time (and it looks like you bring the server up\nand down in several tests). Worse, it's at risk of failing racily if it\never takes more than 5 seconds to start up. That should be uncommon, I'd\nthink, but could happen on a heavily loaded system.\n\nThere's a race-less solution using fifos in lib-git-daemon.sh, where we\nwait for the \"ready to rumble\" line. It's kind of horrific, but it does\nwork and is battle-tested.\n\n-Peff\n"},{"id":"471059","messageId":"Y9JPslSoEayaCJ3n@coredump.intra.peff.net","threadId":"58425","inReplyTo":"b8d3e81b5534148359c7e92807cf1e2795480ddf.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 08/12] test-http-server: add simple authentication","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T10:02:26Z","receivedAt":"2023-01-26T10:02:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:46PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> +struct auth_module {\n> +\tchar *scheme;\n> +\tchar *challenge_params;\n> +\tstruct string_list *tokens;\n> +};\n\nThis is a really minor nit, but: why is \"tokens\" a pointer? It's always\ninitialized, so you never need or want to test it for NULL.\n\nThat would make this:\n\n> +\tif (create) {\n> +\t\tstruct auth_module *mod = xmalloc(sizeof(struct auth_module));\n> +\t\tmod->scheme = xstrdup(scheme);\n> +\t\tmod->challenge_params = NULL;\n> +\t\tALLOC_ARRAY(mod->tokens, 1);\n> +\t\tstring_list_init_dup(mod->tokens);\n\nsimplify to:\n\n string_list_init_dup(&mod->tokens);\n\nand one does not have to wonder why we use ALLOC_ARRAY() there, but not\nwhen allocating the module itself. :)\n\nLikewise you could skip freeing it, but since the memory is held until\nprogram end anyway, that doesn't happen either way.\n\nCertainly what you have won't behave wrong; I'd consider this more like\na coding style thing.\n\n> +\tcat >auth.config <<-EOF &&\n> +\t[auth]\n> +\t\tchallenge = no-params\n> +\t\tchallenge = with-params:foo=\\\"bar\\\" p=1\n> +\t\tchallenge = with-params:foo=\\\"replaced\\\" q=1\n> +\n> +\t\ttoken = no-explicit-challenge:valid-token\n> +\t\ttoken = no-explicit-challenge:also-valid\n> +\t\ttoken = reset-tokens:these-tokens\n> +\t\ttoken = reset-tokens:will-be-reset\n> +\t\ttoken = reset-tokens:\n> +\t\ttoken = reset-tokens:the-only-valid-one\n> +\n> +\t\tallowAnonymous = false\n> +\tEOF\n> +\n> +\tcat >OUT.expected <<-EOF &&\n> +\tWWW-Authenticate: no-params\n> +\tWWW-Authenticate: with-params foo=\"replaced\" q=1\n> +\tWWW-Authenticate: no-explicit-challenge\n> +\tWWW-Authenticate: reset-tokens\n> +\n> +\tError: 401 Unauthorized\n> +\tEOF\n\nOK, so I think now we are getting to the interesting part of what your\ncustom http-server does compared to something like apache. And the\nanswer so far is: custom WWW-Authenticate lines.\n\nI think we could do that with mod_headers pretty easily. But presumably\nwe also want to check that we are getting the correct tokens, generate a\n401, etc.\n\nI suspect this could all be done as a CGI wrapping git-http-backend. You\ncan influence the HTTP response code by sending:\n\n Status: 401 Authorization Required\n WWW-Authenticate: whatever you want\n\nAnd likewise you can see what the client sends by putting something like\nthis in apache.conf:\n\n SetEnvIf Authorization \"(.*)\" HTTP_AUTHORIZATION=$1\n\nand then reading $HTTP_AUTHORIZATION as you like. At that point, it\nfeels like a simple shell or perl script could then decide whether to\nreturn a 401 or not (and if not, then just exec git-http-backend to do\nthe rest). And the scripts would be simple enough that you could have\nindividual scripts to implement various schemes, rather than\nimplementing this configuration scheme. You can control which script is\nrun based on the URL; see the way we match /broken_smart/, etc, in\nt/lib-httpd/apache.conf.\n\n-Peff\n"},{"id":"471060","messageId":"Y9JRX02RLHmIKzwo@coredump.intra.peff.net","threadId":"58425","inReplyTo":"4b1635b3f6968f8d755bdf6bc4ec7af77aefd315.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 10/12] http: replace unsafe size_t multiplication with st_mult","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T10:09:35Z","receivedAt":"2023-01-26T10:09:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:48PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> Replace direct multiplication of two size_t parameters in curl response\n> stream handling callback functions with `st_mult` to guard against\n> overflows.\n\nHmm. So part of me says that more overflow detection is better than\nless, but...I really doubt this is doing anything, and it feels odd to\nme to do overflow checks when there is no allocation.\n\nThere are tons of integer multiplications in Git. Our usual strategy is\nto try to handle overflow like this when we're about to allocate a\nbuffer, with the idea that we'll avoid a truncated size (that we may\nlater fill with too many bytes).\n\nIn these cases, we could possibly avoid a weird or wrong result due to\ntruncation, but I don't see how that is different than most of the rest\nof Git. What makes these worth touching?\n\nMoreover...\n\n> @@ -176,7 +176,7 @@ curlioerr ioctl_buffer(CURL *handle, int cmd, void *clientp)\n> \n> size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> {\n> -\tsize_t size = eltsize * nmemb;\n> +\tsize_t size = st_mult(eltsize, nmemb);\n> \tstruct strbuf *buffer = buffer_;\n> \n> \tstrbuf_add(buffer, ptr, size);\n\nThe caller is already claiming to have eltsize*nmemb bytes accessible\nvia \"ptr\". How did it get such a buffer if that overflows size_t?\n\n> diff --git a/http.c b/http.c\n> index 8a5ba3f4776..a2a80318bb2 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -146,7 +146,7 @@ static int http_schannel_use_ssl_cainfo;\n> \n> size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> {\n> -\tsize_t size = eltsize * nmemb;\n> +\tsize_t size = st_mult(eltsize, nmemb);\n> \tstruct buffer *buffer = buffer_;\n> \n> \tif (size > buffer->buf.len - buffer->posn)\n\nLikewise the caller is asking us to fill a buffer that is eltsize*nmemb.\nSo they must have allocated it already. How can it be bigger than a\nsize_t?\n\nIn practice, of course, these are both coming from curl, and I strongly\nsuspect that curl always sets \"1\" for eltsize anyway, since it's working\nwith bytes. The two fields only exist to conform to the weird fread()\ninterface for historical reasons.\n\nSo I don't think this patch is really hurting much. It just feels like a\nweird one-off that makes the code inconsistent. If somebody who was\nwanting to write similar code later asks \"why is this one st_mult(), but\nnot other multiplications\", I wouldn't have an answer for them.\n\n-Peff\n"},{"id":"471061","messageId":"Y9JWnQeEV0weV4yu@coredump.intra.peff.net","threadId":"58425","inReplyTo":"5f5e46038cf526714f3c5b89ffef2b895b503242.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T10:31:57Z","receivedAt":"2023-01-26T10:32:01Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:49PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> \n> Read and store the HTTP WWW-Authenticate response headers made for\n> a particular request.\n> \n> This will allow us to pass important authentication challenge\n> information to credential helpers or others that would otherwise have\n> been lost.\n\nMakes sense, and the code looks pretty reasonable overall.\n\nA few observations:\n\n> @@ -115,6 +116,19 @@ struct credential {\n> \t */\n> \tstruct string_list helpers;\n> \n> +\t/**\n> +\t * A `strvec` of WWW-Authenticate header values. Each string\n> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n> +\t * in the order they were received in the response.\n> +\t */\n> +\tstruct strvec wwwauth_headers;\n> +\n> +\t/**\n> +\t * Internal use only. Used to keep track of split header fields\n> +\t * in order to fold multiple lines into one value.\n> +\t */\n> +\tunsigned header_is_last_match:1;\n> +\n\nStuffing this into a \"struct credential\" feels a little weird, just\nbecause it's specific to http parsing (especially this internal flag).\nAnd the credential code is seeing full header lines, not broken down at\nall.\n\nI guess I would have expected some level of abstraction here between the\ncredential subsystem and the http subsystem, where the latter is parsing\nand then sticking opaque data into the credential to ferry to the\nhelpers.\n\nBut it probably isn't that big a deal either way. Even though there are\nnon-http credentials, it's not too unreasonable for the credential\nsystem to be aware of http specifically.\n\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = st_mult(eltsize, nmemb);\n\nHere's that st_mult() again. Same comment as the previous patch. :)\n\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\tstrbuf_add(&buf, ptr, size);\n\nOK, so we just copy the buffer. I don't think it would be too hard to\nhandle the buffer as-is, but this does make things a bit easier. Given\nthat we're going to immediately throw away the copy for anything except\nwww-authenticate, we could perhaps wait until we've matched it. That\ndoes mean trimming the CRLF ourselves and using skip_prefix_mem() to\nmatch the start (you'd want skip_iprefix_mem(), of course, but it\ndoesn't yet exist; I'll leave that as an exercise).\n\nMaybe not worth it to save a few allocations, as an http request is\nalready pretty heavyweight. Mostly I flagged it because this is going to\nrun for every header of every request, even though most requests won't\ntrigger it at all.\n\n> +\t/* Strip the CRLF that should be present at the end of each field */\n> +\tstrbuf_trim_trailing_newline(&buf);\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n> +\t\twhile (isspace(*val))\n> +\t\t\tval++;\n> +\n> +\t\tstrvec_push(values, val);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> +\t}\n\nOK, this looks correct from my knowledge of the RFCs. I saw something\nabout isspace() matching newlines, etc, in an earlier thread, but I\nthink we'd never see a newline here, as we're expecting curl to be\nsplitting on our behalf.\n\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t * Continuation lines start with at least one whitespace, maybe more,\n> +\t * so we should collapse these down to a single SP (valid per the spec).\n> +\t */\n> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n> +\t\t/* Trim leading whitespace from this continuation hdr line. */\n> +\t\tstrbuf_ltrim(&buf);\n\nOK, makes sense. This will memmove(), which is needlessly inefficient\n(we could just advance a pointer), but probably not a big deal in\npractice. Using the strbuf functions is a nice simplification.\n\n> +\t\t/*\n> +\t\t * At this point we should always have at least one existing\n> +\t\t * value, even if it is empty. Do not bother appending the new\n> +\t\t * value if this continuation header is itself empty.\n> +\t\t */\n> +\t\tif (!values->nr) {\n> +\t\t\tBUG(\"should have at least one existing header value\");\n> +\t\t} else if (buf.len) {\n> +\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n> +\n> +\t\t\t/* Join two non-empty values with a single space. */\n> +\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n> +\n> +\t\t\tstrvec_pop(values);\n> +\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n> +\t\t\tfree(prev);\n> +\t\t}\n\nLikewise here we end up with an extra allocation of \"prev\", just because\nwe can't pop/push in the right order. But that's probably OK in\npractice, as this is triggering only for the header we care about.\n\nThe concatenation itself makes the whole thing quadratic, but unless we\nare worried about a malicious server DoS-ing us with a billion\nwww-authenticate continuations, I think we can disregard that.\n\n-Peff\n"},{"id":"471062","messageId":"Y9JjRfhl1H4Julv3@coredump.intra.peff.net","threadId":"58425","inReplyTo":"09164f77d56e8efd1450091cf1b12af2bc6cf2f5.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T11:25:57Z","receivedAt":"2023-01-26T11:26:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:50PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n> \n> Add the value of the WWW-Authenticate response header to credential\n> requests. Credential helpers that understand and support HTTP\n> authentication and authorization can use this standard header (RFC 2616\n> Section 14.47 [1]) to generate valid credentials.\n> \n> WWW-Authenticate headers can contain information pertaining to the\n> authority, authentication mechanism, or extra parameters/scopes that are\n> required.\n\nI'm definitely on board with sending these to the helpers. It does feel\na bit weird that we don't parse them at all, and just foist that on the\nhelpers.\n\nIf I understand the RFC correctly, you can have multiple challenges per\nheader, but also multiple headers. So:\n\n WWW-Authenticate: Basic realm=\"foo\", OtherAuth realm=\"bar\"\n WWW-Authenticate: YetAnotherScheme some-token\n\ncould be normalized as:\n\n www-auth-challenge=Basic realm=\"foo\"\n www-auth-challenge=OtherAuth realm=\"bar\"\n www-auth-challenge=YetAnotherScheme some-token\n\nwhich saves each helper from having to do the same work. Likewise, we\ncan do a _little_ more parsing to get:\n\n www-auth-basic=realm=\"foo\"\n www-auth-otherauth=realm=\"bar\"\n www-auth-yetanotherscheme=some-token\n\nI don't think we can go beyond there, though, without understanding the\nsyntax of individual schemes. Which is a shame, as one of the goals of\nthe credential format was to let the helpers do as little as possible\n(so they can't get it wrong!). But helpers are stuck doing things like\nhandling backslashed double-quotes, soaking up extra whitespace, etc.\n\nI'm not really sure what we expect to see in the real world. I guess for\nyour purposes, you are working on an already-big helper that is happy to\njust get the raw values and process them according to the rfc. I'm just\nwondering if there are use cases where somebody might want to do\nsomething with this header, but in a quick shell script kind of way. For\nexample, my credential config is still:\n\n [credential \"https://github.com\"]\n username = peff\n helper = \"!f() { test $1 = get && echo password=$(pass ...); }; f\"\n\nThat's an extreme example, but I'm wondering if there's _anything_\nuseful somebody would want to do in a similar quick-and-dirty kind of\nway. For example, deciding which cred to use based on basic realm, like:\n\n realm=foo\n while read line; do\n case \"$line\" in\n www-auth-basic=)\n value=${line#*=}\n\t# oops, we're just assuming it's realm= here, and we're\n\t# not handling quotes at all. I think it could technically be\n\t# realm=foo or realm=\"foo\"\n\trealm=${value#realm=}\n\t;;\n esac\n done\n echo password=$(pass \"pats-by-realm/$realm\")\n\nwhich could be made a lot easier if we did more parsing (e.g.,\nwww-auth-basic-realm or something). I dunno. Maybe that is just opening\nup a can of worms, as we're stuffing structured data into a linearized\nkey-value list. The nice thing about your proposal is that Git does not\neven have to know anything about these schemes; it's all the problem of\nthe helper. My biggest fear is just that we'll want to shift that later,\nand we'll be stuck with this microformat forever.\n\n> The current I/O format for credential helpers only allows for unique\n> names for properties/attributes, so in order to transmit multiple header\n> values (with a specific order) we introduce a new convention whereby a\n> C-style array syntax is used in the property name to denote multiple\n> ordered values for the same property.\n\nI don't know if this is strictly necessary. The semantics of duplicate\nkeys are not really defined anywhere, and just because the\nimplementations of current readers happen to replace duplicates for the\ncurrent set of keys doesn't mean everything has to. So you could just\ndefine \"wwwauth\" to behave differently. But I don't mind having a\nsyntactic marker to indicate this new type.\n\nIf you're at all convinced by what I said above, then we also might be\nable to get away with having unique keys anyway.\n\n> Documentation/git-credential.txt | 19 ++-\n> credential.c | 11 ++\n> t/lib-credential-helper.sh | 27 ++++\n> t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++\n> 4 files changed, 298 insertions(+), 1 deletion(-)\n> create mode 100644 t/lib-credential-helper.sh\n\nThe patch itself looks pretty reasonable to me.\n\nOne small thing I noticed:\n\n> +\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n\nAs you undoubtedly figured out, the helper path is fed to the shell, so\nspaces in the trash directory are a problem. You've solved it here by\nadding a layer of double quotes, which handles spaces. But you'd run\ninto problems if the absolute path that somebody is using for the test\nsuite has a backslash or a double quote in it.\n\nI don't know how careful we want to be here (or how careful we already\nare[1]), but one simple-ish solution is:\n\n export CREDENTIAL_HELPER\n git -c \"credential.helper=!\\\"\\$CREDENTIAL_HELPER\\\"\" ...\n\nI.e., letting the inner shell expand the variable itself. Another option\nis to put the helper into $TRASH_DIRECTORY/bin and add that to the\n$PATH.\n\nI also wondered if it was worth having setup_credential_helper() just\nstick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't\nhave to keep doing that ugly \"-c\" invocation. Or if you really want to\nhave each test enable it, perhaps have set_credential_reply() turn it on\nvia test_config (which will auto-remove it at the end of the test).\n\n-Peff\n\n[1] Curious, I tried cloning git into this directory:\n\n mkdir '/tmp/foo/\"horrible \\\"path\\\"'\n\n and we do indeed already fail. The first breakage I saw was recent,\n but going further back, it looks like bin-wrappers don't correctly\n handle this case anyway. So maybe that's evidence that nobody would\n do something so ridiculous in practice.\n"},{"id":"471063","messageId":"Y9JkMLueCwjkLHOr@coredump.intra.peff.net","threadId":"58425","inReplyTo":"xmqqwn5bg695.fsf@gitster.g","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T11:29:52Z","receivedAt":"2023-01-26T11:30:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 24, 2023 at 10:03:02AM -0800, Junio C Hamano wrote:\n\n> Victoria Dye writes:\n> \n> > Matthew John Cheetham via GitGitGadget wrote:\n> >> Updates in v6\n> >> =============\n> >> ...\n> > I've re-read the patches in this version; all of my comments from v5 have\n> > been addressed, and the additional updates w.r.t. other reviewer feedback\n> > all look good as well. At this point, I think the series is ready for\n> > 'next'.\n> >\n> > Thanks!\n> \n> Thanks, both. Let's merge it down.\n\nSorry, I'm a bit late to the party, but I left some comments just now\n(this topic had been on my review backlog for ages, but I never quite\ngot to it).\n\nMany of my comments were small bits that could be fixed on top (tiny\nleaks, etc). But some of my comments were of the form \"no, do it totally\ndifferently\". It may simply be too late for those ones, but let's see if\nMatthew finds anything compelling in them.\n\n-Peff\n"},{"id":"471070","messageId":"xmqqfsbxcmdd.fsf@gitster.g","threadId":"58425","inReplyTo":"Y9JkMLueCwjkLHOr@coredump.intra.peff.net","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-26T16:05:18Z","receivedAt":"2023-01-26T16:05:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> Thanks, both. Let's merge it down.\n>\n> Sorry, I'm a bit late to the party, but I left some comments just now\n> (this topic had been on my review backlog for ages, but I never quite\n> got to it).\n>\n> Many of my comments were small bits that could be fixed on top (tiny\n> leaks, etc). But some of my comments were of the form \"no, do it totally\n> differently\". It may simply be too late for those ones, but let's see if\n> Matthew finds anything compelling in them.\n\nI do not mind reverting the merge to 'next' to have an improved\nversion. Your \"do we really want to add a custom server based on\nquestionable codebase whose quality as a test-bed for real world\nusage is dubious?\" is a valid concern.\n"},{"id":"471077","messageId":"Y9LjmMWjBQgNTsQq@coredump.intra.peff.net","threadId":"58425","inReplyTo":"b8d3e81b5534148359c7e92807cf1e2795480ddf.1674252531.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v7 08/12] test-http-server: add simple authentication","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T20:33:28Z","receivedAt":"2023-01-26T20:33:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 20, 2023 at 10:08:46PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> +static int split_auth_param(const char *str, char **scheme, char **val)\n> +{\n> +\tstruct strbuf **p = strbuf_split_str(str, ':', 2);\n> +\n> +\tif (!p[0])\n> +\t\treturn -1;\n> +\n> +\t/* trim trailing ':' */\n> +\tif (p[0]->len && p[0]->buf[p[0]->len - 1] == ':')\n> +\t\tstrbuf_setlen(p[0], p[0]->len - 1);\n> +\n> +\t*scheme = strbuf_detach(p[0], NULL);\n> +\t*val = p[1] ? strbuf_detach(p[1], NULL) : NULL;\n> +\n> +\tstrbuf_list_free(p);\n> +\treturn 0;\n> +}\n\nOh, I forgot one more Coverity-detected problem here when reviewing last\nnight. The early \"return -1\" here leaks \"p\" (there are no strbufs in the\nresulting array, but strbuf_split_str() will still have allocated the\narray). It needs a call to strbuf_list_free(p) there.\n\n-Peff\n"},{"id":"471080","messageId":"Y9LvFMzriAWUsS58@coredump.intra.peff.net","threadId":"58425","inReplyTo":"Y9JPslSoEayaCJ3n@coredump.intra.peff.net","subject":"Re: [PATCH v7 08/12] test-http-server: add simple authentication","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-01-26T21:22:28Z","receivedAt":"2023-01-26T21:22:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 26, 2023 at 05:02:27AM -0500, Jeff King wrote:\n\n> I suspect this could all be done as a CGI wrapping git-http-backend. You\n> can influence the HTTP response code by sending:\n> \n> Status: 401 Authorization Required\n> WWW-Authenticate: whatever you want\n> \n> And likewise you can see what the client sends by putting something like\n> this in apache.conf:\n> \n> SetEnvIf Authorization \"(.*)\" HTTP_AUTHORIZATION=$1\n> \n> and then reading $HTTP_AUTHORIZATION as you like. At that point, it\n> feels like a simple shell or perl script could then decide whether to\n> return a 401 or not (and if not, then just exec git-http-backend to do\n> the rest). And the scripts would be simple enough that you could have\n> individual scripts to implement various schemes, rather than\n> implementing this configuration scheme. You can control which script is\n> run based on the URL; see the way we match /broken_smart/, etc, in\n> t/lib-httpd/apache.conf.\n\nAnd here's a minimally worked-out example of that approach. It's on top\nof your patches so I could use your credential-helper infrastructure in\nthe test, but the intent is that it would replace all of the test-tool\nserver patches and be rolled into t5556 as appropriate.\n\n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 ++++++\n t/lib-httpd/custom-auth.sh | 18 ++++++++++++++++\n t/t5563-simple-http-auth.sh | 42 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 67 insertions(+)\n create mode 100644 t/lib-httpd/custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 608949ea80..ab255bdbc5 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -137,6 +137,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n+\tinstall_script custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0294739a77..4b2256363f 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n \tSetEnv GIT_HTTP_EXPORT_ALL\n \tSetEnv GIT_PROTOCOL\n \n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tCGIPassAuth on\n+\n ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/custom-auth.sh b/t/lib-httpd/custom-auth.sh\nnew file mode 100644\nindex 0000000000..686895ee8c\n--- /dev/null\n+++ b/t/lib-httpd/custom-auth.sh\n@@ -0,0 +1,18 @@\n+#!/bin/sh\n+\n+# Our acceptable auth here is hard-coded, but we could\n+# read it from a file provided by individual tests, etc.\n+#\n+# base64(\"alice:secret-passwd\")\n+USERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\n+case \"$HTTP_AUTHORIZATION\" in\n+\"Basic $USERPASS64\")\n+\texec \"$GIT_EXEC_PATH\"/git-http-backend\n+\t;;\n+*)\n+\techo 'Status: 401 Auth Required'\n+\techo 'WWW-Authenticate: Basic realm=\"whatever\"'\n+\techo\n+\t;;\n+esac\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nnew file mode 100755\nindex 0000000000..314f9217e6\n--- /dev/null\n+++ b/t/t5563-simple-http-auth.sh\n@@ -0,0 +1,42 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+. \"$TEST_DIRECTORY\"/lib-credential-helper.sh\n+\n+start_httpd\n+\n+setup_credential_helper\n+\n+test_expect_success 'setup repository' '\n+\ttest_commit foo &&\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n+'\n+\n+test_expect_success 'access using custom auth' '\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote \\\n+\t\t\"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"whatever\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_done\n-- \n2.39.1.738.g5e5f8a2714\n\n"},{"id":"471082","messageId":"xmqqwn599bk7.fsf@gitster.g","threadId":"58425","inReplyTo":"Y9LvFMzriAWUsS58@coredump.intra.peff.net","subject":"Re: [PATCH v7 08/12] test-http-server: add simple authentication","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-01-26T22:27:04Z","receivedAt":"2023-01-26T22:27:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> I suspect this could all be done as a CGI wrapping git-http-backend. You\n>> can influence the HTTP response code by sending:\n> ...\n> And here's a minimally worked-out example of that approach. It's on top\n> of your patches so I could use your credential-helper infrastructure in\n> the test, but the intent is that it would replace all of the test-tool\n> server patches and be rolled into t5556 as appropriate.\n\nThanks for helping Matthew's topic move forward. I very much like\nseeing apache used for tests in this sample approach, like all the\nother http tests we do with apache, instead of a custom server that\nwe need to ensure that it mimics the real-world use cases and we\nhave to maintain.\n"},{"id":"471116","messageId":"20230128142827.17397-1-mirth.hickford@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"M Hickford","fromEmail":"mirth.hickford@gmail.com","sentAt":"2023-01-28T14:28:27Z","receivedAt":"2023-01-28T14:28:33Z","isPatch":true,"sender":{"key":"mirth.hickford@gmail.com","avatar":"https://avatars.githubusercontent.com/u/105314?v=4"},"body":"> Future work\n> ===========\n> \n> In the future we can further expand the protocol to allow credential helpers\n> decide the best authentication scheme. Today credential helpers are still\n> only expected to return a username/password pair to Git, meaning the other\n> authentication schemes that may be offered still need challenge responses\n> sent via a Basic Authorization header. The changes outlined above still\n> permit helpers to select and configure an available authentication mode, but\n> require the remote for example to unpack a bearer token from a basic\n> challenge.\n> \n> More careful consideration is required in the handling of custom\n> authentication schemes which may not have a username, or may require\n> arbitrary additional request header values be set.\n> \n> For example imagine a new \"FooBar\" authentication scheme that is surfaced in\n> the following response:\n> \n> HTTP/1.1 401 Unauthorized\n> WWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n> \n> \n> With support for arbitrary authentication schemes, Git would call credential\n> helpers with the following over standard input:\n> \n> protocol=https\n> host=example.com\n> wwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n> \n> \n> And then an enlightened credential helper could return over standard output:\n> \n> protocol=https\n> host=example.com\n> authtype=FooBar\n> username=bob@id.example.com\n> password=\n> header[]=X-FooBar: 12345\n> header[]=X-FooBar-Alt: ABCDEF\n> \n> \n> Git would be expected to attach this authorization header to the next\n> request:\n> \n> GET /info/refs?service=git-upload-pack HTTP/1.1\n> Host: git.example\n> Git-Protocol: version=2\n> Authorization: FooBar \n> X-FooBar: 12345\n> X-FooBar-Alt: ABCDEF\n\nInteresting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar). \n\nChallenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.\n\n[credential]\n helper = storage # eg. cache or osxkeychain\n helper = challenge-response # eg. oauth-dpop\n\nStorage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/pull.1443.git.git.1674914650588.gitgitgadget@gmail.com/T/#u\n\nA monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.\n\nDeclaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth\n\n[1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8\n"},{"id":"471264","messageId":"AS2PR03MB9815DEDEA3CF40F3D54624E0C0D19@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"20230128142827.17397-1-mirth.hickford@gmail.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-01T20:15:17Z","receivedAt":"2023-02-01T20:15:33Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-28 06:28, M Hickford wrote:\n\n>> Future work\n>> ===========\n>>\n>> In the future we can further expand the protocol to allow credential helpers\n>> decide the best authentication scheme. Today credential helpers are still\n>> only expected to return a username/password pair to Git, meaning the other\n>> authentication schemes that may be offered still need challenge responses\n>> sent via a Basic Authorization header. The changes outlined above still\n>> permit helpers to select and configure an available authentication mode, but\n>> require the remote for example to unpack a bearer token from a basic\n>> challenge.\n>>\n>> More careful consideration is required in the handling of custom\n>> authentication schemes which may not have a username, or may require\n>> arbitrary additional request header values be set.\n>>\n>> For example imagine a new \"FooBar\" authentication scheme that is surfaced in\n>> the following response:\n>>\n>> HTTP/1.1 401 Unauthorized\n>> WWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n>>\n>>\n>> With support for arbitrary authentication schemes, Git would call credential\n>> helpers with the following over standard input:\n>>\n>> protocol=https\n>> host=example.com\n>> wwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n>>\n>>\n>> And then an enlightened credential helper could return over standard output:\n>>\n>> protocol=https\n>> host=example.com\n>> authtype=FooBar\n>> username=bob@id.example.com\n>> password=\n>> header[]=X-FooBar: 12345\n>> header[]=X-FooBar-Alt: ABCDEF\n>>\n>>\n>> Git would be expected to attach this authorization header to the next\n>> request:\n>>\n>> GET /info/refs?service=git-upload-pack HTTP/1.1\n>> Host: git.example\n>> Git-Protocol: version=2\n>> Authorization: FooBar \n>> X-FooBar: 12345\n>> X-FooBar-Alt: ABCDEF\n> \n> Interesting! Can you tell us more about how you hope to use this at GitHub? Could this be used for OAuth 2.0 Demonstrating Proof-of-Possession at the Application Layer (DPoP)? https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop (some of the fields in your example look familiar). \n\nThis would be exactly the sort of thing that this would enable. DPoP is one\nexample where the correct auth response requires more than just a username/\npassword pair in the Authorization header. We should also be returning\nstandard headers like 'Authenticate-Info' on 200 OK responses along side the\n'store' calls to helpers; they could contain nonces or other important auth\ninformation.\n\nMy end goal here is to extend the credential helper protocol such that that\nhelpers can see more of the initial request challenge, and then modify the\nsubsequent request directly, or configure and let curl handle it (the latter\npart not present in this iteration).\n\nOne thing not called out in this doc is really the need for some capability\nadvertisement between Git and helpers - for example if the curl version in\nuse supports CURLOPT_XOAUTH2_BEARER for bearer tokens.\n\n> Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.\n> \n> [credential]\n> helper = storage # eg. cache or osxkeychain\n> helper = challenge-response # eg. oauth-dpop\n\nI think really this sort of thing is where the credential helper protocol\nisn't designed for credential-generating helpers in mind, but only simple\nstorage-only helpers. There is no affinity between get/erase/store commands\nmeaning one helper may return a credential for another helper to store it.\nNot sure if this was ever the intention, over just the need to consult a\nlist of helpers for a stored credential.\n\n> Storage may return an expired challenge response stored earlier. This could be avoided by introducing an expiry attribute to the credential protocol. https://lore.kernel.org/git/pull.1443.git.git.1674914650588.gitgitgadget@gmail.com/T/#u\n> \n> A monolithic helper configured alone doesn't have this problem -- it knows which parts of its output to store or discard.\n> \n> Declaration of interest: I maintain a credential-generating OAuth helper composable with any storage helper. https://github.com/hickford/git-credential-oauth\n> \n> [1] https://datatracker.ietf.org/doc/html/draft-ietf-oauth-dpop#section-8\n"},{"id":"471295","messageId":"Y9sAysjR8jCbETxy@coredump.intra.peff.net","threadId":"58425","inReplyTo":"AS2PR03MB9815DEDEA3CF40F3D54624E0C0D19@AS2PR03MB9815.eurprd03.prod.outlook.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-02T00:16:10Z","receivedAt":"2023-02-02T00:16:14Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Feb 01, 2023 at 12:15:17PM -0800, Matthew John Cheetham wrote:\n\n> > Challenge responses are typically short lived [1]. What happens if a storage helper is configured before a challenge-response helper? We want to maintain composability of helpers.\n> > \n> > [credential]\n> > helper = storage # eg. cache or osxkeychain\n> > helper = challenge-response # eg. oauth-dpop\n> \n> I think really this sort of thing is where the credential helper protocol\n> isn't designed for credential-generating helpers in mind, but only simple\n> storage-only helpers. There is no affinity between get/erase/store commands\n> meaning one helper may return a credential for another helper to store it.\n> Not sure if this was ever the intention, over just the need to consult a\n> list of helpers for a stored credential.\n\nI actually had envisioned helpers generating credentials. In fact, in\nthe first iteration of the series, Git did not prompt for passwords at\nall! It would depend on git-credential-prompt to do so. But I ended up\nfolding that in for simplicity.\n\nI could well believe that there is not enough context passed around for\nhelpers to make good decisions, though. It's both a feature and a bug\nthat credentials from one helper get passed to another. It's good if you\nwant to cache a generated credential. It's bad if you don't want\ncredentials from one helper to leak to another, less-secure one.\n\nSo I'm open to improvements that help define and communicate that\ncontext.\n\n-Peff\n"},{"id":"471345","messageId":"6f83ed25-a7e1-06dd-f180-d70c7e1b1973@gmx.de","threadId":"58425","inReplyTo":"xmqqfsbxcmdd.fsf@gitster.g","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-02-02T10:14:33Z","receivedAt":"2023-02-02T10:14:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio & Peff,\n\nOn Thu, 26 Jan 2023, Junio C Hamano wrote:\n\n> Jeff King writes:\n>\n> >> Thanks, both. Let's merge it down.\n> >\n> > Sorry, I'm a bit late to the party, but I left some comments just now\n> > (this topic had been on my review backlog for ages, but I never quite\n> > got to it).\n> >\n> > Many of my comments were small bits that could be fixed on top (tiny\n> > leaks, etc). But some of my comments were of the form \"no, do it totally\n> > differently\". It may simply be too late for those ones, but let's see if\n> > Matthew finds anything compelling in them.\n>\n> I do not mind reverting the merge to 'next' to have an improved\n> version. Your \"do we really want to add a custom server based on\n> questionable codebase whose quality as a test-bed for real world\n> usage is dubious?\" is a valid concern.\n\nExcept.\n\nExcept that this code base would have made for a fine base to potentially\nimplement an HTTPS-based replacement for the aging and insecure\ngit-daemon.\n\nThat code base (which is hardly as questionable codebase as you make it\nsound because it has been in use for years in a slightly different form)\nwould have had the opportunity to mature in a relatively safe environment:\nour test suite. And eventually, once robust enough, it could have been\nextended to allow for easy and painless yet secure ad-hoc serving of Git\nrepositories, addressing the security concerns around git-daemon.\n\nAnd now that we're throwing out that code we don't have that opportunity,\nmaking the goal to deprecate the git-daemon and replace it by something\nthat is as easy to set up but talks HTTPS instead much, much harder to\nreach.\n\nIn addition, it causes a loss of test coverage because Apache is not\navailable in all the setups where the \"questionable\" code would have had\nno problem being built and validating the credential code.\n\nWindows, for example, will now go completely uncovered in CI regarding the\nnew code.\n\nCiao,\nJohannes\n"},{"id":"471349","messageId":"230202.86edr8pax5.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"6f83ed25-a7e1-06dd-f180-d70c7e1b1973@gmx.de","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-02T11:04:48Z","receivedAt":"2023-02-02T11:24:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 02 2023, Johannes Schindelin wrote:\n\n> Hi Junio & Peff,\n>\n> On Thu, 26 Jan 2023, Junio C Hamano wrote:\n>\n>> Jeff King writes:\n>>\n>> >> Thanks, both. Let's merge it down.\n>> >\n>> > Sorry, I'm a bit late to the party, but I left some comments just now\n>> > (this topic had been on my review backlog for ages, but I never quite\n>> > got to it).\n>> >\n>> > Many of my comments were small bits that could be fixed on top (tiny\n>> > leaks, etc). But some of my comments were of the form \"no, do it totally\n>> > differently\". It may simply be too late for those ones, but let's see if\n>> > Matthew finds anything compelling in them.\n>>\n>> I do not mind reverting the merge to 'next' to have an improved\n>> version. Your \"do we really want to add a custom server based on\n>> questionable codebase whose quality as a test-bed for real world\n>> usage is dubious?\" is a valid concern.\n>\n> Except.\n>\n> Except that this code base would have made for a fine base to potentially\n> implement an HTTPS-based replacement for the aging and insecure\n> git-daemon.\n>\n> That code base (which is hardly as questionable codebase as you make it\n> sound because it has been in use for years in a slightly different form)\n> would have had the opportunity to mature in a relatively safe environment:\n> our test suite. And eventually, once robust enough, it could have been\n> extended to allow for easy and painless yet secure ad-hoc serving of Git\n> repositories, addressing the security concerns around git-daemon.\n>\n> And now that we're throwing out that code we don't have that opportunity,\n> making the goal to deprecate the git-daemon and replace it by something\n> that is as easy to set up but talks HTTPS instead much, much harder to\n> reach.\n\nThere's many reasons for why you almost never see a git:// URL in the\nwild anymore.\n\nBut if \"easy and painless\" was synonymous with \"built with git\" or\n\"ships with git\" as you seem to be using it, surely it would be more\ncommon than doing the same with http or https, which requires an\nexternal server?\n\nSo, easy for whom? Just us and our own test suite?\n\nHaving read both your reply & Jeff's[1] I don't think you're addressing\nthe thrust of his argument.\n\nYou can share all those goals without the method of getting there\nrequiring us to start maintaining our own webserver.\n\n> In addition, it causes a loss of test coverage because Apache is not\n> available in all the setups where the \"questionable\" code would have had\n> no problem being built and validating the credential code.\n>\n> Windows, for example, will now go completely uncovered in CI regarding the\n> new code.\n\nI have not set up Apache on Windows, but binaries seem to be available\nfor it[2]. We don't use those now, but is downloading, setting up &\nrunning them in CI really harder than emarking on a project of\nmaintaining our own webserver, especially we've got an eye towards\nnon-test suite use?\n\nEven if we think that we'd like to have a webserver built when you \"make\ngit\" I don't see why we'd go the NIH route of writing our own. Unlike\nthe git:// protocol there's a *lot* of implementations of http(s)://\nservers.\n\nIf we think apache is too heavyweight for whatever reason, can't we add\none of the many light http servers out there to contrib/ use it it from\nthere?\n\n1. https://lore.kernel.org/git/Y9JA0UCRh7qUqKQI@coredump.intra.peff.net/\n2. https://httpd.apache.org/docs/2.4/platform/windows.html\n"},{"id":"471361","messageId":"60c3f1d0-2858-8811-7eb0-d6f586bf2ab8@gmx.de","threadId":"58425","inReplyTo":"230202.86edr8pax5.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-02-02T13:51:20Z","receivedAt":"2023-02-02T13:53:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Thu, 2 Feb 2023, Ævar Arnfjörð Bjarmason wrote:\n\n> On Thu, Feb 02 2023, Johannes Schindelin wrote:\n>\n> > On Thu, 26 Jan 2023, Junio C Hamano wrote:\n> >\n> >> Jeff King writes:\n> >>\n> >> >> Thanks, both. Let's merge it down.\n> >> >\n> >> > Sorry, I'm a bit late to the party, but I left some comments just now\n> >> > (this topic had been on my review backlog for ages, but I never quite\n> >> > got to it).\n> >> >\n> >> > Many of my comments were small bits that could be fixed on top (tiny\n> >> > leaks, etc). But some of my comments were of the form \"no, do it totally\n> >> > differently\". It may simply be too late for those ones, but let's see if\n> >> > Matthew finds anything compelling in them.\n> >>\n> >> I do not mind reverting the merge to 'next' to have an improved\n> >> version. Your \"do we really want to add a custom server based on\n> >> questionable codebase whose quality as a test-bed for real world\n> >> usage is dubious?\" is a valid concern.\n> >\n> > Except.\n> >\n> > Except that this code base would have made for a fine base to potentially\n> > implement an HTTPS-based replacement for the aging and insecure\n> > git-daemon.\n> >\n> > That code base (which is hardly as questionable codebase as you make it\n> > sound because it has been in use for years in a slightly different form)\n> > would have had the opportunity to mature in a relatively safe environment:\n> > our test suite. And eventually, once robust enough, it could have been\n> > extended to allow for easy and painless yet secure ad-hoc serving of Git\n> > repositories, addressing the security concerns around git-daemon.\n> >\n> > And now that we're throwing out that code we don't have that opportunity,\n> > making the goal to deprecate the git-daemon and replace it by something\n> > that is as easy to set up but talks HTTPS instead much, much harder to\n> > reach.\n>\n> There's many reasons for why you almost never see a git:// URL in the\n> wild anymore.\n\nI am unwilling to accept that statement without any source to back it up.\nThin air is no substitute for reliable evidence.\n\n> But if \"easy and painless\" was synonymous with \"built with git\" or\n> \"ships with git\" as you seem to be using it, surely it would be more\n> common than doing the same with http or https, which requires an\n> external server?\n\nOh whoa... \"requires an external server\"?\n\nMy entire point was to suggest a way forward for an _internal_ server that\nspeaks https:// instead of git://.\n\nSo I am not suggesting what you seem to have understood me to suggest.\n\nCiao,\nJohannes\n"},{"id":"471449","messageId":"Y91FjhNgZGz6foFl@coredump.intra.peff.net","threadId":"58425","inReplyTo":"6f83ed25-a7e1-06dd-f180-d70c7e1b1973@gmx.de","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-03T17:34:06Z","receivedAt":"2023-02-03T17:36:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 02, 2023 at 11:14:33AM +0100, Johannes Schindelin wrote:\n\n> > I do not mind reverting the merge to 'next' to have an improved\n> > version. Your \"do we really want to add a custom server based on\n> > questionable codebase whose quality as a test-bed for real world\n> > usage is dubious?\" is a valid concern.\n> \n> Except.\n> \n> Except that this code base would have made for a fine base to potentially\n> implement an HTTPS-based replacement for the aging and insecure\n> git-daemon.\n\nI'm skeptical that it is a good idea for Git to implement a custom http\nserver from scratch. There are a lot of extended features people would\nwant in a production-ready server. TLS, HTTP/2, and so on. The code\nunder discussion is pretty stripped-down. A network service is also a\npretty big attack surface for buffer overflows, etc.\n\nIt feels to me like the resources required to make it good enough for\nnormal users to run would be substantial. And we'd be better off trying\nto integrate with an existing project that provides a web server\n(whether it's a lightweight server that supports us as a CGI, or a\nlibrary that does most of the heavy lifting).\n\n> That code base (which is hardly as questionable codebase as you make it\n> sound because it has been in use for years in a slightly different form)\n\nPerhaps I'm being too hard on git-daemon. But my operational experience\nwith it is that it has several flaws, mostly around the child-management\ncode. We rewrote that code totally to make it usable at GitHub.\n\nAs a concrete example, the parent daemon process will do a linear walk\nover all children, calling waitpid() on each one. This makes handling N\nchildren quadratic, and the daemon grinds to a halt when there are many\nclients.\n\n> In addition, it causes a loss of test coverage because Apache is not\n> available in all the setups where the \"questionable\" code would have had\n> no problem being built and validating the credential code.\n> \n> Windows, for example, will now go completely uncovered in CI regarding the\n> new code.\n\nI'm sympathetic there, though it's a problem for all of the existing\nhttp code, too. Is there some server option that would be easier to run\neverywhere, but that doesn't involve us writing a server from scratch?\nCertainly I have no particular love for apache or its configuration\nlanguage.\n\n-Peff\n"},{"id":"471587","messageId":"DB9PR03MB9831A708EA98E198591F6632C0DA9@DB9PR03MB9831.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"Y9JjRfhl1H4Julv3@coredump.intra.peff.net","subject":"Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-06T19:18:03Z","receivedAt":"2023-02-06T19:18:35Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-26 03:25, Jeff King wrote:\n\n> On Fri, Jan 20, 2023 at 10:08:50PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>>\n>> Add the value of the WWW-Authenticate response header to credential\n>> requests. Credential helpers that understand and support HTTP\n>> authentication and authorization can use this standard header (RFC 2616\n>> Section 14.47 [1]) to generate valid credentials.\n>>\n>> WWW-Authenticate headers can contain information pertaining to the\n>> authority, authentication mechanism, or extra parameters/scopes that are\n>> required.\n> \n> I'm definitely on board with sending these to the helpers. It does feel\n> a bit weird that we don't parse them at all, and just foist that on the\n> helpers.\n> \n> If I understand the RFC correctly, you can have multiple challenges per\n> header, but also multiple headers. So:\n> \n> WWW-Authenticate: Basic realm=\"foo\", OtherAuth realm=\"bar\"\n> WWW-Authenticate: YetAnotherScheme some-token\n\nThat is correct. It would be strange that server would respond with a mix\nof styles, but I guess it's not forbidden.\n\n> could be normalized as:\n> \n> www-auth-challenge=Basic realm=\"foo\"\n> www-auth-challenge=OtherAuth realm=\"bar\"\n> www-auth-challenge=YetAnotherScheme some-token\n> \n> which saves each helper from having to do the same work. Likewise, we\n> can do a _little_ more parsing to get:\n> \n> www-auth-basic=realm=\"foo\"\n> www-auth-otherauth=realm=\"bar\"\n> www-auth-yetanotherscheme=some-token\n> \n> I don't think we can go beyond there, though, without understanding the\n> syntax of individual schemes. Which is a shame, as one of the goals of\n> the credential format was to let the helpers do as little as possible\n> (so they can't get it wrong!). But helpers are stuck doing things like\n> handling backslashed double-quotes, soaking up extra whitespace, etc.\n\nThis key format wouldn't make it obviously easier for simple helpers to\nunderstand. Now they no longer have well-known keys but a key prefix.\n\nMy overall goal here is to have Git know less about auth, so it treats\nall values as totally opaque. The only logic added is around reconstructing\nfolded headers, which is just HTTP and not auth specific.\n\n> I'm not really sure what we expect to see in the real world. I guess for\n> your purposes, you are working on an already-big helper that is happy to\n> just get the raw values and process them according to the rfc. I'm just\n> wondering if there are use cases where somebody might want to do\n> something with this header, but in a quick shell script kind of way. For\n> example, my credential config is still:\n> \n> [credential \"https://github.com\"]\n> username = peff\n> helper = \"!f() { test $1 = get && echo password=$(pass ...); }; f\"\n> \n> That's an extreme example, but I'm wondering if there's _anything_\n> useful somebody would want to do in a similar quick-and-dirty kind of\n> way. For example, deciding which cred to use based on basic realm, like:\n> \n> realm=foo\n> while read line; do\n> case \"$line\" in\n> www-auth-basic=)\n> value=${line#*=}\n> \t# oops, we're just assuming it's realm= here, and we're\n> \t# not handling quotes at all. I think it could technically be\n> \t# realm=foo or realm=\"foo\"\n> \trealm=${value#realm=}\n> \t;;\n> esac\n> done\n> echo password=$(pass \"pats-by-realm/$realm\")\n> \n> which could be made a lot easier if we did more parsing (e.g.,\n> www-auth-basic-realm or something). I dunno. Maybe that is just opening\n> up a can of worms, as we're stuffing structured data into a linearized\n> key-value list. The nice thing about your proposal is that Git does not\n> even have to know anything about these schemes; it's all the problem of\n> the helper. My biggest fear is just that we'll want to shift that later,\n> and we'll be stuck with this microformat forever.\n\nI'm not sure there's such a continuous scale between simple and 'complex'\nhelpers that would mean there'd be a simple shell script generating\nOAuth or DPoP credentials instead of a helper written in a higher-level\nlanguage where parsing the headers is one of the simpler challenges faced.\n\n>> The current I/O format for credential helpers only allows for unique\n>> names for properties/attributes, so in order to transmit multiple header\n>> values (with a specific order) we introduce a new convention whereby a\n>> C-style array syntax is used in the property name to denote multiple\n>> ordered values for the same property.\n> \n> I don't know if this is strictly necessary. The semantics of duplicate\n> keys are not really defined anywhere, and just because the\n> implementations of current readers happen to replace duplicates for the\n> current set of keys doesn't mean everything has to. So you could just\n> define \"wwwauth\" to behave differently. But I don't mind having a\n> syntactic marker to indicate this new type.\n\nI had considered another model whereby we forgo the key=value line model,\nand hide another format behind the 'final' terminating new-line. However\nI thought this would be even more distuptive.\n\n> If you're at all convinced by what I said above, then we also might be\n> able to get away with having unique keys anyway.\n> \n>> Documentation/git-credential.txt | 19 ++-\n>> credential.c | 11 ++\n>> t/lib-credential-helper.sh | 27 ++++\n>> t/t5556-http-auth.sh | 242 +++++++++++++++++++++++++++++++\n>> 4 files changed, 298 insertions(+), 1 deletion(-)\n>> create mode 100644 t/lib-credential-helper.sh\n> \n> The patch itself looks pretty reasonable to me.\n> \n> One small thing I noticed:\n> \n>> +\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n> \n> As you undoubtedly figured out, the helper path is fed to the shell, so\n> spaces in the trash directory are a problem. You've solved it here by\n> adding a layer of double quotes, which handles spaces. But you'd run\n> into problems if the absolute path that somebody is using for the test\n> suite has a backslash or a double quote in it.\n> \n> I don't know how careful we want to be here (or how careful we already\n> are[1]), but one simple-ish solution is:\n> \n> export CREDENTIAL_HELPER\n> git -c \"credential.helper=!\\\"\\$CREDENTIAL_HELPER\\\"\" ...\n> \n> I.e., letting the inner shell expand the variable itself. Another option\n> is to put the helper into $TRASH_DIRECTORY/bin and add that to the\n> $PATH.\n> \n> I also wondered if it was worth having setup_credential_helper() just\n> stick it in $TRASH_DIRECTORY/.gitconfig so that individual tests don't\n> have to keep doing that ugly \"-c\" invocation. Or if you really want to\n> have each test enable it, perhaps have set_credential_reply() turn it on\n> via test_config (which will auto-remove it at the end of the test).\n\nGood ideas! I shall try those.\n\n> -Peff\n> \n> [1] Curious, I tried cloning git into this directory:\n> \n> mkdir '/tmp/foo/\"horrible \\\"path\\\"'\n> \n> and we do indeed already fail. The first breakage I saw was recent,\n> but going further back, it looks like bin-wrappers don't correctly\n> handle this case anyway. So maybe that's evidence that nobody would\n> do something so ridiculous in practice.\n"},{"id":"471588","messageId":"DB9PR03MB983150E73B1C963C628CBE75C0DA9@DB9PR03MB9831.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"Y9JWnQeEV0weV4yu@coredump.intra.peff.net","subject":"Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-06T19:25:49Z","receivedAt":"2023-02-06T19:26:03Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-01-26 02:31, Jeff King wrote:\n\n> On Fri, Jan 20, 2023 at 10:08:49PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n>>\n>> Read and store the HTTP WWW-Authenticate response headers made for\n>> a particular request.\n>>\n>> This will allow us to pass important authentication challenge\n>> information to credential helpers or others that would otherwise have\n>> been lost.\n> \n> Makes sense, and the code looks pretty reasonable overall.\n> \n> A few observations:\n> \n>> @@ -115,6 +116,19 @@ struct credential {\n>> \t */\n>> \tstruct string_list helpers;\n>> \n>> +\t/**\n>> +\t * A `strvec` of WWW-Authenticate header values. Each string\n>> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n>> +\t * in the order they were received in the response.\n>> +\t */\n>> +\tstruct strvec wwwauth_headers;\n>> +\n>> +\t/**\n>> +\t * Internal use only. Used to keep track of split header fields\n>> +\t * in order to fold multiple lines into one value.\n>> +\t */\n>> +\tunsigned header_is_last_match:1;\n>> +\n> \n> Stuffing this into a \"struct credential\" feels a little weird, just\n> because it's specific to http parsing (especially this internal flag).\n> And the credential code is seeing full header lines, not broken down at\n> all.\n> \n> I guess I would have expected some level of abstraction here between the\n> credential subsystem and the http subsystem, where the latter is parsing\n> and then sticking opaque data into the credential to ferry to the\n> helpers.\n> \n> But it probably isn't that big a deal either way. Even though there are\n> non-http credentials, it's not too unreasonable for the credential\n> system to be aware of http specifically.\n\nI had considered possibly introducing an opaque property-bag style of\n'protocol-specific properties' that, for example, http.c would add the\nWWW-Authenticate headers to as something like `http.wwwauth[]`.\nOther protocols (like smtp:// or cert://) could add their own properties\nif they needed or wanted to also.\n\nThoughts?\n\n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> +{\n>> +\tsize_t size = st_mult(eltsize, nmemb);\n> \n> Here's that st_mult() again. Same comment as the previous patch. :)\n\nYeah I'm gonna drop this. Your arguments make sense; it's not going to be a\nproblem in reality :-)\n\n>> +\t/*\n>> +\t * Header lines may not come NULL-terminated from libcurl so we must\n>> +\t * limit all scans to the maximum length of the header line, or leverage\n>> +\t * strbufs for all operations.\n>> +\t *\n>> +\t * In addition, it is possible that header values can be split over\n>> +\t * multiple lines as per RFC 2616 (even though this has since been\n>> +\t * deprecated in RFC 7230). A continuation header field value is\n>> +\t * identified as starting with a space or horizontal tab.\n>> +\t *\n>> +\t * The formal definition of a header field as given in RFC 2616 is:\n>> +\t *\n>> +\t * message-header = field-name \":\" [ field-value ]\n>> +\t * field-name = token\n>> +\t * field-value = *( field-content | LWS )\n>> +\t * field-content = > +\t * and consisting of either *TEXT or combinations\n>> +\t * of token, separators, and quoted-string>\n>> +\t */\n>> +\n>> +\tstrbuf_add(&buf, ptr, size);\n> \n> OK, so we just copy the buffer. I don't think it would be too hard to\n> handle the buffer as-is, but this does make things a bit easier. Given\n> that we're going to immediately throw away the copy for anything except\n> www-authenticate, we could perhaps wait until we've matched it. That\n> does mean trimming the CRLF ourselves and using skip_prefix_mem() to\n> match the start (you'd want skip_iprefix_mem(), of course, but it\n> doesn't yet exist; I'll leave that as an exercise).\n\nFair point! I can replace most of these with operations over the curl ptr.\n\n> Maybe not worth it to save a few allocations, as an http request is\n> already pretty heavyweight. Mostly I flagged it because this is going to\n> run for every header of every request, even though most requests won't\n> trigger it at all.\n> \n>> +\t/* Strip the CRLF that should be present at the end of each field */\n>> +\tstrbuf_trim_trailing_newline(&buf);\n>> +\n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n>> +\t\twhile (isspace(*val))\n>> +\t\t\tval++;\n>> +\n>> +\t\tstrvec_push(values, val);\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n>> +\t}\n> \n> OK, this looks correct from my knowledge of the RFCs. I saw something\n> about isspace() matching newlines, etc, in an earlier thread, but I\n> think we'd never see a newline here, as we're expecting curl to be\n> splitting on our behalf.\n> \n>> +\t/*\n>> +\t * This line could be a continuation of the previously matched header\n>> +\t * field. If this is the case then we should append this value to the\n>> +\t * end of the previously consumed value.\n>> +\t * Continuation lines start with at least one whitespace, maybe more,\n>> +\t * so we should collapse these down to a single SP (valid per the spec).\n>> +\t */\n>> +\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n>> +\t\t/* Trim leading whitespace from this continuation hdr line. */\n>> +\t\tstrbuf_ltrim(&buf);\n> \n> OK, makes sense. This will memmove(), which is needlessly inefficient\n> (we could just advance a pointer), but probably not a big deal in\n> practice. Using the strbuf functions is a nice simplification.\n> \n>> +\t\t/*\n>> +\t\t * At this point we should always have at least one existing\n>> +\t\t * value, even if it is empty. Do not bother appending the new\n>> +\t\t * value if this continuation header is itself empty.\n>> +\t\t */\n>> +\t\tif (!values->nr) {\n>> +\t\t\tBUG(\"should have at least one existing header value\");\n>> +\t\t} else if (buf.len) {\n>> +\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n>> +\n>> +\t\t\t/* Join two non-empty values with a single space. */\n>> +\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n>> +\n>> +\t\t\tstrvec_pop(values);\n>> +\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n>> +\t\t\tfree(prev);\n>> +\t\t}\n> \n> Likewise here we end up with an extra allocation of \"prev\", just because\n> we can't pop/push in the right order. But that's probably OK in\n> practice, as this is triggering only for the header we care about.\n> \n> The concatenation itself makes the whole thing quadratic, but unless we\n> are worried about a malicious server DoS-ing us with a billion\n> www-authenticate continuations, I think we can disregard that.\n> \n> -Peff\n"},{"id":"471589","messageId":"d362f7016d34c4803adf42a88012997c66e0bde8.1675711789.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"[PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-06T19:29:47Z","receivedAt":"2023-02-06T19:29:57Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a test showing simple anoymous HTTP access to an unprotected\nrepository, that results in no credential helper invocations.\nAlso add a test demonstrating simple basic authentication with\nsimple credential helper support.\n\nLeverage a no-parsed headers (NPH) CGI script so that we can directly\ncontrol the HTTP responses to simulate a multitude of good, bad and ugly\nremote server implementations around auth.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +++\n t/lib-httpd/nph-custom-auth.sh | 42 +++++++++++++++++\n t/t5563-simple-http-auth.sh | 86 ++++++++++++++++++++++++++++++++++\n 4 files changed, 135 insertions(+)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 608949ea80b..2c49569f675 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -137,6 +137,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n+\tinstall_script nph-custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0294739a77a..76335cdb24d 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n \tSetEnv GIT_HTTP_EXPORT_ALL\n \tSetEnv GIT_PROTOCOL\n \n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tCGIPassAuth on\n+\n ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\nnew file mode 100755\nindex 00000000000..8f851aebac4\n--- /dev/null\n+++ b/t/lib-httpd/nph-custom-auth.sh\n@@ -0,0 +1,42 @@\n+#!/bin/sh\n+\n+VALID_CREDS_FILE=custom-auth.valid\n+CHALLENGE_FILE=custom-auth.challenge\n+ANONYMOUS_FILE=custom-auth.anonymous\n+\n+#\n+# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n+#\n+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n+# credential for the current request. Each line in the file is considered a\n+# valid HTTP Authorization header value. For example:\n+#\n+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+#\n+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n+# in a 401 response if no valid authentication credentials were included in the\n+# request. For example:\n+#\n+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+# WWW-Authenticate: Basic realm=\"example.com\"\n+#\n+\n+if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n+\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n+then\n+\t# Note that although git-http-backend returns a status line, it\n+\t# does so using a CGI 'Status' header. Because this script is an\n+\t# No Parsed Headers (NPH) script, we must return a real HTTP\n+\t# status line.\n+\t# This is only a test script, so we don't bother to check for\n+\t# the actual status from git-http-backend and always return 200.\n+\techo 'HTTP/1.1 200 OK'\n+\texec \"$GIT_EXEC_PATH\"/git-http-backend\n+fi\n+\n+echo 'HTTP/1.1 401 Authorization Required'\n+if test -f \"$CHALLENGE_FILE\"\n+then\n+\tcat \"$CHALLENGE_FILE\"\n+fi\n+echo\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nnew file mode 100755\nindex 00000000000..004eac5d1ed\n--- /dev/null\n+++ b/t/t5563-simple-http-auth.sh\n@@ -0,0 +1,86 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+test_expect_success 'setup_credential_helper' '\n+\tmkdir -p \"$TRASH_DIRECTORY/bin\" &&\n+\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n+\texport PATH &&\n+\n+\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n+\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\tcmd=$1\n+\tteefile=$cmd-query.cred\n+\tcatfile=$cmd-reply.cred\n+\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n+\tif test \"$cmd\" = \"get\"; then\n+\t\tcat $catfile\n+\tfi\n+\tEOF\n+'\n+\n+set_credential_reply() {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query() {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\n+\n+per_test_cleanup () {\n+\trm -f *.cred &&\n+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n+}\n+\n+test_expect_success 'setup repository' '\n+\ttest_commit foo &&\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n+'\n+\n+test_expect_success 'access anonymous no challenge' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n+'\n+\n+test_expect_success 'access using basic auth' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"471590","messageId":"cd9a02ba94e9eb59b07d0b95140d5b880f122941.1675711789.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"[PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-06T19:29:48Z","receivedAt":"2023-02-06T19:29:59Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++\n git-compat-util.h | 22 +++++++++\n http.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 158 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a76d0526f79..f11c44517d7 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1266,6 +1266,28 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n \treturn 0;\n }\n \n+/*\n+ * Like skip_prefix_mem, but compare case-insensitively. Note that the\n+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n+ * characters or locale-specific conversions).\n+ */\n+static inline int skip_iprefix_mem(const char *buf, size_t len,\n+\t\t\t\t const char *prefix,\n+\t\t\t\t const char **out, size_t *outlen)\n+{\n+\tsize_t prefix_len = strlen(prefix);\n+\tif (len < prefix_len)\n+\t\treturn 0;\n+\n+\tif (!strncasecmp(buf, prefix, prefix_len)){\n+\t\t*out = buf + prefix_len;\n+\t\t*outlen = len - prefix_len;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n {\n \tunsigned long ul;\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..7a56a3db5f7 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+/*\n+ * A folded header continuation line starts with at least one single whitespace\n+ * character. It is not a continuation line if the line is *just* a newline.\n+ * The RFC for HTTP states that CRLF is the header field line ending, but some\n+ * servers may use LF only; we accept both.\n+ */\n+static inline int is_hdr_continuation(const char *ptr, const size_t size)\n+{\n+\t/* totally empty line or normal header */\n+\tif (!size || !isspace(*ptr))\n+\t\treturn 0;\n+\n+\t/* empty line with LF line ending */\n+\tif (size == 1 && ptr[0] == '\\n')\n+\t\treturn 0;\n+\n+\t/* empty line with CRLF line ending */\n+\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tsize_t val_len;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n+\t\tstrbuf_add(&buf, val, val_len);\n+\n+\t\t/*\n+\t\t * Strip the CRLF that should be present at the end of each\n+\t\t * field as well as any trailing or leading whitespace from the\n+\t\t * value.\n+\t\t */\n+\t\tstrbuf_trim(&buf);\n+\n+\t\tstrvec_push(values, buf.buf);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n+\t\t/*\n+\t\t * Trim the CRLF and any leading or trailing from this line.\n+\t\t */\n+\t\tstrbuf_add(&buf, ptr, size);\n+\t\tstrbuf_trim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (!strncasecmp(ptr, \"http/\", 5))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1982,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"471591","messageId":"149aedf55010718e22669575a148988eed0d8dcb.1675711789.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"[PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-06T19:29:49Z","receivedAt":"2023-02-06T19:30:07Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 11 ++\n t/t5563-simple-http-auth.sh | 236 +++++++++++++++++++++++++++++++\n 3 files changed, 265 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..9f39ebc3c7e 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n \tfprintf(fp, \"%s=%s\\n\", key, value);\n }\n \n+static void credential_write_strvec(FILE *fp, const char *key,\n+\t\t\t\t const struct strvec *vec)\n+{\n+\tchar *full_key = xstrfmt(\"%s[]\", key);\n+\tfor (size_t i = 0; i < vec->nr; i++) {\n+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n+\t}\n+\tfree(full_key);\n+}\n+\n void credential_write(const struct credential *c, FILE *fp)\n {\n \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -270,6 +280,7 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nindex 004eac5d1ed..a7b1e5bd1af 100755\n--- a/t/t5563-simple-http-auth.sh\n+++ b/t/t5563-simple-http-auth.sh\n@@ -73,6 +73,242 @@ test_expect_success 'access using basic auth' '\n \texpect_credential_query get <<-EOF &&\n \tprotocol=http\n \thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth invalid credentials' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\ttest_must_fail git ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with extra challenges' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\" param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth mixed-case wwwauth header name' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\twww-authenticate: foobar param1=\"value1\" param2=\"value2\"\n+\tWWW-AUTHENTICATE: BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\tWwW-aUtHeNtIcAtE: baSiC realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=foobar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=baSiC realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\"\n+\t param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\n+\t p=1\n+\t q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header empty continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" param2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Bearer authorize_uri=\\\"id.example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" p=1\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" q=0\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \"\\tparam2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Basic realm=\"example.com\"\n \tEOF\n \n \texpect_credential_query store <<-EOF\n-- \ngitgitgadget\n"},{"id":"471592","messageId":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v7.git.1674252530.gitgitgadget@gmail.com","subject":"[PATCH v8 0/3] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-06T19:29:46Z","receivedAt":"2023-02-06T19:30:09Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I use a small CGI shell script that acts as a\nfrontend to git-http-backend; simple authentication is configurable by\nfiles.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\n\nUpdates in v7\n=============\n\n * Address several whitespace and arg/param list alignment issues.\n\n * Rethink the test-http-helper worker-mode error and result enum to be more\n simple and more informative to the nature of the error.\n\n * Use uintmax_t to store the Content-Length of a request in the helper\n test-http-server. Maintain a bit flag to store if we received such a\n header.\n\n * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n in the test-http-server.\n\n * Add test case to cover request message parsing in test-http-server.\n\n * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n get_auth_module.\n\n * Correctly free the split strbufs created in the header parsing loop in\n test-http-server.\n\n * Avoid needless comparison > 0 for unsigned types.\n\n * Always set optional outputs to NULL if not present in test helper config\n value handling.\n\n * Remove an accidentally commented-out test cleanup line for one test case\n in t5556.\n\n\nUpdates in v8\n=============\n\n * Drop custom HTTP test helper tool in favour of using a CGI shell script\n and Apache; avoiding the need to implement an HTTP server.\n\n * Avoid allocations in header reading callback unless we have a header we\n care about; act on the char* from libcurl directly rather than create a\n strbuf for each header.\n\n * Drop st_mult overflow guarding function in curl callback functions; we're\n not allocating memory based on the resulting value and just adds to\n potential confusion in the future.\n\nMatthew John Cheetham (3):\n t5563: add tests for basic and anoymous HTTP access\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n credential.c | 12 ++\n credential.h | 15 ++\n git-compat-util.h | 22 +++\n http.c | 120 ++++++++++++\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +\n t/lib-httpd/nph-custom-auth.sh | 42 ++++\n t/t5563-simple-http-auth.sh | 322 +++++++++++++++++++++++++++++++\n 9 files changed, 558 insertions(+), 1 deletion(-)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v8\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v8\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v7:\n\n 1: 74b0de14185 < -: ----------- daemon: libify socket setup and option functions\n 2: b6ba344a671 < -: ----------- daemon: libify child process handling functions\n 3: 9967401c972 < -: ----------- daemon: rename some esoteric/laboured terminology\n 4: 17c890ee108 < -: ----------- test-http-server: add stub HTTP server test helper\n 5: 6e70e304cfe < -: ----------- test-http-server: add HTTP error response function\n 6: 43f1cdcbb82 ! 1: d362f7016d3 test-http-server: add HTTP request parsing\n @@ Metadata\n Author: Matthew John Cheetham \n \n ## Commit message ##\n - test-http-server: add HTTP request parsing\n + t5563: add tests for basic and anoymous HTTP access\n \n - Add ability to parse HTTP requests to the test-http-server test helper.\n - Introduce `struct req` to store request information including:\n + Add a test showing simple anoymous HTTP access to an unprotected\n + repository, that results in no credential helper invocations.\n + Also add a test demonstrating simple basic authentication with\n + simple credential helper support.\n \n - * HTTP method & version\n - * Request path and query parameters\n - * Headers\n - * Content type and length (from `Content-Type` and `-Length` headers)\n -\n - Failure to parse the request results in a 400 Bad Request response to\n - the client. Note that we're not trying to support all possible requests\n - here, but just enough to exercise all code under test.\n + Leverage a no-parsed headers (NPH) CGI script so that we can directly\n + control the HTTP responses to simulate a multitude of good, bad and ugly\n + remote server implementations around auth.\n \n Signed-off-by: Matthew John Cheetham \n \n - ## t/helper/test-http-server.c ##\n -@@ t/helper/test-http-server.c: enum worker_result {\n - \t * Close the socket and clean up. Does not imply an error.\n - \t */\n - \tWR_HANGUP = 2,\n -+\n -+\t/*\n -+\t * Unexpected request message or error in request parsing.\n -+\t * Respond with an 400 error. Close the socket and cleanup.\n -+\t * Exit child-process with a non-zero status.\n -+\t */\n -+\tWR_CLIENT_ERROR = 3,\n -+};\n -+\n -+/*\n -+ * Fields from a parsed HTTP request.\n -+ */\n -+struct req {\n -+\tstruct strbuf start_line;\n -+\n -+\tconst char *method;\n -+\tconst char *http_version;\n -+\n -+\tstruct strbuf uri_path;\n -+\tstruct strbuf query_args;\n -+\n -+\tstruct string_list header_list;\n -+\tconst char *content_type;\n -+\tuintmax_t content_length;\n -+\tunsigned has_content_length:1;\n - };\n + ## t/lib-httpd.sh ##\n +@@ t/lib-httpd.sh: prepare_httpd() {\n + \tinstall_script error-smart-http.sh\n + \tinstall_script error.sh\n + \tinstall_script apply-one-time-perl.sh\n ++\tinstall_script nph-custom-auth.sh\n \n -+#define REQ__INIT { \\\n -+\t.start_line = STRBUF_INIT, \\\n -+\t.uri_path = STRBUF_INIT, \\\n -+\t.query_args = STRBUF_INIT, \\\n -+\t.header_list = STRING_LIST_INIT_NODUP, \\\n -+\t.content_type = NULL, \\\n -+\t.content_length = 0, \\\n -+\t.has_content_length = 0, \\\n -+}\n -+\n -+static void req__release(struct req *req)\n -+{\n -+\tstrbuf_release(&req->start_line);\n -+\n -+\tstrbuf_release(&req->uri_path);\n -+\tstrbuf_release(&req->query_args);\n -+\n -+\tstring_list_clear(&req->header_list, 0);\n -+}\n -+\n - static enum worker_result send_http_error(int fd, int http_code,\n - \t\t\t\t\t const char *http_code_name,\n - \t\t\t\t\t int retry_after_seconds,\n -@@ t/helper/test-http-server.c: done:\n - \treturn wr;\n - }\n - \n -+/*\n -+ * Read the HTTP request up to the start of the optional message-body.\n -+ * We do this byte-by-byte because we have keep-alive turned on and\n -+ * cannot rely on an EOF.\n -+ *\n -+ * https://tools.ietf.org/html/rfc7230\n -+ *\n -+ * We cannot call die() here because our caller needs to properly\n -+ * respond to the client and/or close the socket before this\n -+ * child exits so that the client doesn't get a connection reset\n -+ * by peer error.\n -+ */\n -+static enum worker_result req__read(struct req *req, int fd)\n -+{\n -+\tstruct strbuf h = STRBUF_INIT;\n -+\tstruct string_list start_line_fields = STRING_LIST_INIT_DUP;\n -+\tint nr_start_line_fields;\n -+\tconst char *uri_target;\n -+\tconst char *query;\n -+\tchar *hp;\n -+\tconst char *hv;\n -+\n -+\tenum worker_result result = WR_OK;\n -+\n -+\t/*\n -+\t * Read line 0 of the request and split it into component parts:\n -+\t *\n -+\t * SP SP CRLF\n -+\t *\n -+\t */\n -+\tif (strbuf_getwholeline_fd(&req->start_line, fd, '\\n') == EOF) {\n -+\t\tresult = WR_HANGUP;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tstrbuf_trim_trailing_newline(&req->start_line);\n -+\n -+\tnr_start_line_fields = string_list_split(&start_line_fields,\n -+\t\t\t\t\t\t req->start_line.buf,\n -+\t\t\t\t\t\t ' ', -1);\n -+\tif (nr_start_line_fields != 3) {\n -+\t\tlogerror(\"could not parse request start-line '%s'\",\n -+\t\t\t req->start_line.buf);\n -+\t\tresult = WR_CLIENT_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\treq->method = xstrdup(start_line_fields.items[0].string);\n -+\treq->http_version = xstrdup(start_line_fields.items[2].string);\n -+\n -+\turi_target = start_line_fields.items[1].string;\n -+\n -+\tif (strcmp(req->http_version, \"HTTP/1.1\")) {\n -+\t\tlogerror(\"unsupported version '%s' (expecting HTTP/1.1)\",\n -+\t\t\t req->http_version);\n -+\t\tresult = WR_CLIENT_ERROR;\n -+\t\tgoto done;\n -+\t}\n -+\n -+\tquery = strchr(uri_target, '?');\n -+\n -+\tif (query) {\n -+\t\tstrbuf_add(&req->uri_path, uri_target, (query - uri_target));\n -+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n -+\t\tstrbuf_addstr(&req->query_args, query + 1);\n -+\t} else {\n -+\t\tstrbuf_addstr(&req->uri_path, uri_target);\n -+\t\tstrbuf_trim_trailing_dir_sep(&req->uri_path);\n -+\t}\n -+\n -+\t/*\n -+\t * Read the set of HTTP headers into a string-list.\n -+\t */\n -+\twhile (1) {\n -+\t\tif (strbuf_getwholeline_fd(&h, fd, '\\n') == EOF)\n -+\t\t\tgoto done;\n -+\t\tstrbuf_trim_trailing_newline(&h);\n -+\n -+\t\tif (!h.len)\n -+\t\t\tgoto done; /* a blank line ends the header */\n -+\n -+\t\thp = strbuf_detach(&h, NULL);\n -+\t\tstring_list_append(&req->header_list, hp);\n -+\n -+\t\t/* also store common request headers as struct req members */\n -+\t\tif (skip_iprefix(hp, \"Content-Type: \", &hv)) {\n -+\t\t\treq->content_type = hv;\n -+\t\t} else if (skip_iprefix(hp, \"Content-Length: \", &hv)) {\n -+\t\t\t/*\n -+\t\t\t * Content-Length is always non-negative, but has no\n -+\t\t\t * upper bound according to RFC 7230 (§3.3.2).\n -+\t\t\t */\n -+\t\t\tintmax_t len = 0;\n -+\t\t\tif (sscanf(hv, \"%\"PRIdMAX, &len) != 1 || len < 0 ||\n -+\t\t\t len == INTMAX_MAX) {\n -+\t\t\t\tlogerror(\"invalid content-length: '%s'\", hv);\n -+\t\t\t\tresult = WR_CLIENT_ERROR;\n -+\t\t\t\tgoto done;\n -+\t\t\t}\n -+\n -+\t\t\treq->content_length = (uintmax_t)len;\n -+\t\t\treq->has_content_length = 1;\n -+\t\t}\n -+\t}\n -+\n -+\t/*\n -+\t * We do not attempt to read the , if it exists.\n -+\t * We let our caller read/chunk it in as appropriate.\n -+\t */\n -+\n -+done:\n -+\tstring_list_clear(&start_line_fields, 0);\n -+\n -+\t/*\n -+\t * This is useful for debugging the request, but very noisy.\n -+\t */\n -+\tif (trace2_is_enabled()) {\n -+\t\tstruct string_list_item *item;\n -+\t\ttrace2_printf(\"%s: %s\", TR2_CAT, req->start_line.buf);\n -+\t\ttrace2_printf(\"%s: hver: %s\", TR2_CAT, req->http_version);\n -+\t\ttrace2_printf(\"%s: hmth: %s\", TR2_CAT, req->method);\n -+\t\ttrace2_printf(\"%s: path: %s\", TR2_CAT, req->uri_path.buf);\n -+\t\ttrace2_printf(\"%s: qury: %s\", TR2_CAT, req->query_args.buf);\n -+\t\tif (req->has_content_length)\n -+\t\t\ttrace2_printf(\"%s: clen: %\"PRIuMAX, TR2_CAT,\n -+\t\t\t\t req->content_length);\n -+\t\tif (req->content_type)\n -+\t\t\ttrace2_printf(\"%s: ctyp: %s\", TR2_CAT, req->content_type);\n -+\t\tfor_each_string_list_item(item, &req->header_list)\n -+\t\t\ttrace2_printf(\"%s: hdrs: %s\", TR2_CAT, item->string);\n -+\t}\n -+\n -+\treturn result;\n -+}\n -+\n -+static enum worker_result dispatch(struct req *req)\n -+{\n -+\treturn send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1, NULL,\n -+\t\t\t WR_HANGUP);\n -+}\n -+\n - static enum worker_result worker(void)\n - {\n -+\tstruct req req = REQ__INIT;\n - \tchar *client_addr = getenv(\"REMOTE_ADDR\");\n - \tchar *client_port = getenv(\"REMOTE_PORT\");\n - \tenum worker_result wr = WR_OK;\n -@@ t/helper/test-http-server.c: static enum worker_result worker(void)\n - \tset_keep_alive(0, logerror);\n + \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \n - \twhile (1) {\n --\t\twr = send_http_error(STDOUT_FILENO, 501, \"Not Implemented\", -1,\n --\t\t\t\t NULL, WR_HANGUP);\n -+\t\treq__release(&req);\n +\n + ## t/lib-httpd/apache.conf ##\n +@@ t/lib-httpd/apache.conf: Alias /auth/dumb/ www/auth/dumb/\n + \tSetEnv GIT_HTTP_EXPORT_ALL\n + \tSetEnv GIT_PROTOCOL\n + \n ++\n ++\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n ++\tSetEnv GIT_HTTP_EXPORT_ALL\n ++\tCGIPassAuth on\n ++\n + ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n + ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n + ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n +@@ t/lib-httpd/apache.conf: ScriptAlias /broken_smart/ broken-smart-http.sh/\n + ScriptAlias /error_smart/ error-smart-http.sh/\n + ScriptAlias /error/ error.sh/\n + ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n ++ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n + \n + \tOptions FollowSymlinks\n + \n +\n + ## t/lib-httpd/nph-custom-auth.sh (new) ##\n +@@\n ++#!/bin/sh\n +\n -+\t\talarm(timeout);\n -+\t\twr = req__read(&req, 0);\n -+\t\talarm(0);\n ++VALID_CREDS_FILE=custom-auth.valid\n ++CHALLENGE_FILE=custom-auth.challenge\n ++ANONYMOUS_FILE=custom-auth.anonymous\n +\n -+\t\tif (wr == WR_CLIENT_ERROR)\n -+\t\t\twr = send_http_error(STDOUT_FILENO, 400, \"Bad Request\",\n -+\t\t\t\t\t -1, NULL, wr);\n ++#\n ++# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n ++#\n ++# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n ++# credential for the current request. Each line in the file is considered a\n ++# valid HTTP Authorization header value. For example:\n ++#\n ++# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++#\n ++# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n ++# in a 401 response if no valid authentication credentials were included in the\n ++# request. For example:\n ++#\n ++# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n ++# WWW-Authenticate: Basic realm=\"example.com\"\n ++#\n +\n -+\t\tif (wr != WR_OK)\n -+\t\t\tbreak;\n - \n -+\t\twr = dispatch(&req);\n - \t\tif (wr != WR_OK)\n - \t\t\tbreak;\n - \t}\n ++if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n ++\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n ++then\n ++\t# Note that although git-http-backend returns a status line, it\n ++\t# does so using a CGI 'Status' header. Because this script is an\n ++\t# No Parsed Headers (NPH) script, we must return a real HTTP\n ++\t# status line.\n ++\t# This is only a test script, so we don't bother to check for\n ++\t# the actual status from git-http-backend and always return 200.\n ++\techo 'HTTP/1.1 200 OK'\n ++\texec \"$GIT_EXEC_PATH\"/git-http-backend\n ++fi\n ++\n ++echo 'HTTP/1.1 401 Authorization Required'\n ++if test -f \"$CHALLENGE_FILE\"\n ++then\n ++\tcat \"$CHALLENGE_FILE\"\n ++fi\n ++echo\n \n - ## t/t5556-http-auth.sh (new) ##\n + ## t/t5563-simple-http-auth.sh (new) ##\n @@\n +#!/bin/sh\n +\n +test_description='test http auth header and credential helper interop'\n +\n -+TEST_NO_CREATE_REPO=1\n +. ./test-lib.sh\n -+\n -+# Setup a repository\n -+#\n -+REPO_DIR=\"$TRASH_DIRECTORY\"/repo\n -+\n -+SERVER_LOG=\"$TRASH_DIRECTORY\"/OUT.server.log\n -+\n -+PATH=\"$GIT_BUILD_DIR/t/helper/:$PATH\" && export PATH\n -+\n -+test_expect_success 'setup repos' '\n -+\ttest_create_repo \"$REPO_DIR\" &&\n -+\tgit -C \"$REPO_DIR\" branch -M main\n ++. \"$TEST_DIRECTORY\"/lib-httpd.sh\n ++\n ++start_httpd\n ++\n ++test_expect_success 'setup_credential_helper' '\n ++\tmkdir -p \"$TRASH_DIRECTORY/bin\" &&\n ++\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n ++\texport PATH &&\n ++\n ++\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n ++\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n ++\tcmd=$1\n ++\tteefile=$cmd-query.cred\n ++\tcatfile=$cmd-reply.cred\n ++\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n ++\tif test \"$cmd\" = \"get\"; then\n ++\t\tcat $catfile\n ++\tfi\n ++\tEOF\n +'\n +\n -+run_http_server_worker() {\n -+\t(\n -+\t\tcd \"$REPO_DIR\"\n -+\t\ttest-http-server --worker \"$@\" 2>\"$SERVER_LOG\" | tr -d \"\\r\"\n -+\t)\n ++set_credential_reply() {\n ++\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n ++}\n ++\n ++expect_credential_query() {\n ++\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n ++\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n ++\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n +}\n +\n +per_test_cleanup () {\n -+\trm -f OUT.* &&\n -+\trm -f IN.* &&\n ++\trm -f *.cred &&\n ++\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n +}\n +\n -+test_expect_success 'http auth server request parsing' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n ++test_expect_success 'setup repository' '\n ++\ttest_commit foo &&\n ++\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n ++\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n ++'\n +\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tallowAnonymous = true\n -+\tEOF\n ++test_expect_success 'access anonymous no challenge' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n ++\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n ++'\n +\n -+\techo \"HTTP/1.1 400 Bad Request\" >OUT.http400 &&\n -+\techo \"HTTP/1.1 200 OK\" >OUT.http200 &&\n ++test_expect_success 'access using basic auth' '\n ++\ttest_when_finished \"per_test_cleanup\" &&\n +\n -+\tcat >IN.http.valid <<-EOF &&\n -+\tGET /info/refs HTTP/1.1\n -+\tContent-Length: 0\n ++\tset_credential_reply get <<-EOF &&\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\tcat >IN.http.badfirstline <<-EOF &&\n -+\t/info/refs GET HTTP\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n +\n -+\tcat >IN.http.badhttpver <<-EOF &&\n -+\tGET /info/refs HTTP/999.9\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n ++\tWWW-Authenticate: Basic realm=\"example.com\"\n +\tEOF\n +\n -+\tcat >IN.http.ltzlen <<-EOF &&\n -+\tGET /info/refs HTTP/1.1\n -+\tContent-Length: -1\n -+\tEOF\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n -+\tcat >IN.http.badlen <<-EOF &&\n -+\tGET /info/refs HTTP/1.1\n -+\tContent-Length: not-a-number\n ++\texpect_credential_query get <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HTTPD_DEST\n +\tEOF\n +\n -+\tcat >IN.http.overlen <<-EOF &&\n -+\tGET /info/refs HTTP/1.1\n -+\tContent-Length: 9223372036854775807\n ++\texpect_credential_query store <<-EOF\n ++\tprotocol=http\n ++\thost=$HTTPD_DEST\n ++\tusername=alice\n ++\tpassword=secret-passwd\n +\tEOF\n -+\n -+\trun_http_server_worker \\\n -+\t\t--auth-config=\"$TRASH_DIRECTORY/auth.config\" OUT.actual &&\n -+\ttest_cmp OUT.http200 OUT.actual &&\n -+\n -+\trun_http_server_worker OUT.actual &&\n -+\ttest_cmp OUT.http400 OUT.actual &&\n -+\n -+\trun_http_server_worker OUT.actual &&\n -+\ttest_cmp OUT.http400 OUT.actual &&\n -+\n -+\trun_http_server_worker OUT.actual &&\n -+\ttest_cmp OUT.http400 OUT.actual &&\n -+\n -+\trun_http_server_worker OUT.actual &&\n -+\ttest_cmp OUT.http400 OUT.actual\n +'\n +\n +test_done\n 7: ca9c2787248 < -: ----------- test-http-server: pass Git requests to http-backend\n 8: b8d3e81b553 < -: ----------- test-http-server: add simple authentication\n 9: 2f97c94f679 < -: ----------- test-http-server: add sending of arbitrary headers\n 10: 4b1635b3f69 < -: ----------- http: replace unsafe size_t multiplication with st_mult\n 11: 5f5e46038cf ! 2: cd9a02ba94e http: read HTTP WWW-Authenticate response headers\n @@ credential.h: struct credential {\n \n /* Initialize a credential structure, setting all fields to empty. */\n \n + ## git-compat-util.h ##\n +@@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *prefix,\n + \treturn 0;\n + }\n + \n ++/*\n ++ * Like skip_prefix_mem, but compare case-insensitively. Note that the\n ++ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n ++ * characters or locale-specific conversions).\n ++ */\n ++static inline int skip_iprefix_mem(const char *buf, size_t len,\n ++\t\t\t\t const char *prefix,\n ++\t\t\t\t const char **out, size_t *outlen)\n ++{\n ++\tsize_t prefix_len = strlen(prefix);\n ++\tif (len < prefix_len)\n ++\t\treturn 0;\n ++\n ++\tif (!strncasecmp(buf, prefix, prefix_len)){\n ++\t\t*out = buf + prefix_len;\n ++\t\t*outlen = len - prefix_len;\n ++\t\treturn 1;\n ++\t}\n ++\n ++\treturn 0;\n ++}\n ++\n + static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n + {\n + \tunsigned long ul;\n +\n ## http.c ##\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n ++/*\n ++ * A folded header continuation line starts with at least one single whitespace\n ++ * character. It is not a continuation line if the line is *just* a newline.\n ++ * The RFC for HTTP states that CRLF is the header field line ending, but some\n ++ * servers may use LF only; we accept both.\n ++ */\n ++static inline int is_hdr_continuation(const char *ptr, const size_t size)\n ++{\n ++\t/* totally empty line or normal header */\n ++\tif (!size || !isspace(*ptr))\n ++\t\treturn 0;\n ++\n ++\t/* empty line with LF line ending */\n ++\tif (size == 1 && ptr[0] == '\\n')\n ++\t\treturn 0;\n ++\n ++\t/* empty line with CRLF line ending */\n ++\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n ++\t\treturn 0;\n ++\n ++\treturn 1;\n ++}\n ++\n +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n +{\n -+\tsize_t size = st_mult(eltsize, nmemb);\n ++\tsize_t size = eltsize * nmemb;\n +\tstruct strvec *values = &http_auth.wwwauth_headers;\n +\tstruct strbuf buf = STRBUF_INIT;\n +\tconst char *val;\n ++\tsize_t val_len;\n +\n +\t/*\n +\t * Header lines may not come NULL-terminated from libcurl so we must\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * of token, separators, and quoted-string>\n +\t */\n +\n -+\tstrbuf_add(&buf, ptr, size);\n -+\n -+\t/* Strip the CRLF that should be present at the end of each field */\n -+\tstrbuf_trim_trailing_newline(&buf);\n -+\n +\t/* Start of a new WWW-Authenticate header */\n -+\tif (skip_iprefix(buf.buf, \"www-authenticate:\", &val)) {\n -+\t\twhile (isspace(*val))\n -+\t\t\tval++;\n ++\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n ++\t\tstrbuf_add(&buf, val, val_len);\n +\n -+\t\tstrvec_push(values, val);\n ++\t\t/*\n ++\t\t * Strip the CRLF that should be present at the end of each\n ++\t\t * field as well as any trailing or leading whitespace from the\n ++\t\t * value.\n ++\t\t */\n ++\t\tstrbuf_trim(&buf);\n ++\n ++\t\tstrvec_push(values, buf.buf);\n +\t\thttp_auth.header_is_last_match = 1;\n +\t\tgoto exit;\n +\t}\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * This line could be a continuation of the previously matched header\n +\t * field. If this is the case then we should append this value to the\n +\t * end of the previously consumed value.\n -+\t * Continuation lines start with at least one whitespace, maybe more,\n -+\t * so we should collapse these down to a single SP (valid per the spec).\n +\t */\n -+\tif (http_auth.header_is_last_match && isspace(*buf.buf)) {\n -+\t\t/* Trim leading whitespace from this continuation hdr line. */\n -+\t\tstrbuf_ltrim(&buf);\n ++\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n ++\t\t/*\n ++\t\t * Trim the CRLF and any leading or trailing from this line.\n ++\t\t */\n ++\t\tstrbuf_add(&buf, ptr, size);\n ++\t\tstrbuf_trim(&buf);\n +\n +\t\t/*\n +\t\t * At this point we should always have at least one existing\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * We only care about the last HTTP request response's headers so clear\n +\t * the existing array.\n +\t */\n -+\tif (istarts_with(buf.buf, \"http/\"))\n ++\tif (!strncasecmp(ptr, \"http/\", 5))\n +\t\tstrvec_clear(values);\n +\n +exit:\n 12: 09164f77d56 ! 3: 149aedf5501 credential: add WWW-Authenticate header to cred requests\n @@ credential.c: void credential_write(const struct credential *c, FILE *fp)\n \n static int run_credential_helper(struct credential *c,\n \n - ## t/lib-credential-helper.sh (new) ##\n -@@\n -+setup_credential_helper() {\n -+\ttest_expect_success 'setup credential helper' '\n -+\t\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/credential-helper.sh\" &&\n -+\t\texport CREDENTIAL_HELPER &&\n -+\t\techo $CREDENTIAL_HELPER &&\n -+\n -+\t\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n -+\t\tcmd=$1\n -+\t\tteefile=$cmd-query.cred\n -+\t\tcatfile=$cmd-reply.cred\n -+\t\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n -+\t\tif test \"$cmd\" = \"get\"; then\n -+\t\t\tcat $catfile\n -+\t\tfi\n -+\t\tEOF\n -+\t'\n -+}\n + ## t/t5563-simple-http-auth.sh ##\n +@@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n + \texpect_credential_query get <<-EOF &&\n + \tprotocol=http\n + \thost=$HTTPD_DEST\n ++\twwwauth[]=Basic realm=\"example.com\"\n ++\tEOF\n +\n -+set_credential_reply() {\n -+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n -+}\n ++\texpect_credential_query store <<-EOF\n ++\tprotocol=http\n ++\thost=$HTTPD_DEST\n ++\tusername=alice\n ++\tpassword=secret-passwd\n ++\tEOF\n ++'\n +\n -+expect_credential_query() {\n -+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n -+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n -+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n -+}\n -\n - ## t/t5556-http-auth.sh ##\n -@@ t/t5556-http-auth.sh: test_description='test http auth header and credential helper interop'\n - \n - TEST_NO_CREATE_REPO=1\n - . ./test-lib.sh\n -+. \"$TEST_DIRECTORY\"/lib-credential-helper.sh\n - \n - test_set_port GIT_TEST_HTTP_PROTOCOL_PORT\n - \n -@@ t/t5556-http-auth.sh: test_expect_success 'setup repos' '\n - \tgit -C \"$REPO_DIR\" branch -M main\n - '\n - \n -+setup_credential_helper\n -+\n - run_http_server_worker() {\n - \t(\n - \t\tcd \"$REPO_DIR\"\n -@@ t/t5556-http-auth.sh: per_test_cleanup () {\n - \tstop_http_server &&\n - \trm -f OUT.* &&\n - \trm -f IN.* &&\n -+\trm -f *.cred &&\n - \trm -f auth.config\n - }\n - \n -@@ t/t5556-http-auth.sh: test_expect_success 'http auth anonymous no challenge' '\n - \tgit ls-remote $ORIGIN_URL\n - '\n - \n -+test_expect_success 'http auth www-auth headers to credential helper basic valid' '\n ++test_expect_success 'access using basic auth invalid credentials' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n ++\n ++\tset_credential_reply get <<-EOF &&\n ++\tusername=baduser\n ++\tpassword=wrong-passwd\n +\tEOF\n +\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++\tEOF\n +\n -+\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n ++\tWWW-Authenticate: Basic realm=\"example.com\"\n +\tEOF\n +\n -+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\ttest_config_global credential.helper test-helper &&\n ++\ttest_must_fail git ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n +\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=Basic realm=\"example.com\"\n +\tEOF\n +\n -+\texpect_credential_query store <<-EOF\n ++\texpect_credential_query erase <<-EOF\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=secret-passwd\n ++\thost=$HTTPD_DEST\n ++\tusername=baduser\n ++\tpassword=wrong-passwd\n ++\twwwauth[]=Basic realm=\"example.com\"\n +\tEOF\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper ignore case valid' '\n ++test_expect_success 'access using basic auth with extra challenges' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n -+\t\textraHeader = wWw-aUtHeNtIcAtE: bEaRer auThoRiTy=\\\"id.example.com\\\"\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +\n -+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++\tEOF\n ++\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n ++\tWWW-Authenticate: FooBar param1=\"value1\" param2=\"value2\"\n ++\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n ++\tWWW-Authenticate: Basic realm=\"example.com\"\n ++\tEOF\n ++\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n +\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\twwwauth[]=bEaRer auThoRiTy=\"id.example.com\"\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n ++\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n ++\twwwauth[]=Basic realm=\"example.com\"\n +\tEOF\n +\n +\texpect_credential_query store <<-EOF\n +\tprotocol=http\n -+\thost=$HOST_PORT\n ++\thost=$HTTPD_DEST\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper continuation hdr' '\n ++test_expect_success 'access using basic auth mixed-case wwwauth header name' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\"\\\\n q=1\\\\n \\\\t p=0\"\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +\n -+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++\tEOF\n ++\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n ++\twww-authenticate: foobar param1=\"value1\" param2=\"value2\"\n ++\tWWW-AUTHENTICATE: BEARER authorize_uri=\"id.example.com\" p=1 q=0\n ++\tWwW-aUtHeNtIcAtE: baSiC realm=\"example.com\"\n ++\tEOF\n ++\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n +\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=foobar param1=\"value1\" param2=\"value2\"\n ++\twwwauth[]=BEARER authorize_uri=\"id.example.com\" p=1 q=0\n ++\twwwauth[]=baSiC realm=\"example.com\"\n +\tEOF\n +\n +\texpect_credential_query store <<-EOF\n +\tprotocol=http\n -+\thost=$HOST_PORT\n ++\thost=$HTTPD_DEST\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper empty continuation hdrs' '\n ++test_expect_success 'access using basic auth with wwwauth header continuations' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n -+\t\textraheader = \"WWW-Authenticate:\"\n -+\t\textraheader = \" \"\n -+\t\textraheader = \" bearer authority=\\\"id.example.com\\\"\"\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +\n -+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++\tEOF\n ++\n ++\t# Note that leading and trailing whitespace is important to correctly\n ++\t# simulate a continuation/folded header.\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n ++\tWWW-Authenticate: FooBar param1=\"value1\"\n ++\t param2=\"value2\"\n ++\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\n ++\t p=1\n ++\t q=0\n ++\tWWW-Authenticate: Basic realm=\"example.com\"\n ++\tEOF\n ++\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n +\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\twwwauth[]=bearer authority=\"id.example.com\"\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n ++\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n ++\twwwauth[]=Basic realm=\"example.com\"\n +\tEOF\n +\n +\texpect_credential_query store <<-EOF\n +\tprotocol=http\n -+\thost=$HOST_PORT\n ++\thost=$HTTPD_DEST\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper custom schemes' '\n ++test_expect_success 'access using basic auth with wwwauth header empty continuations' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = \"foobar:alg=test widget=1\"\n -+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +\n -+\tgit -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n ++\tEOF\n ++\n ++\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n ++\n ++\t# Note that leading and trailing whitespace is important to correctly\n ++\t# simulate a continuation/folded header.\n ++\tprintf \"\">$CHALLENGE &&\n ++\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n ++\tprintf \" \\r\\n\" >>$CHALLENGE &&\n ++\tprintf \" param2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n ++\tprintf \"WWW-Authenticate: Bearer authorize_uri=\\\"id.example.com\\\"\\r\\n\" >>$CHALLENGE &&\n ++\tprintf \" p=1\\r\\n\" >>$CHALLENGE &&\n ++\tprintf \" \\r\\n\" >>$CHALLENGE &&\n ++\tprintf \" q=0\\r\\n\" >>$CHALLENGE &&\n ++\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\\r\\n\" >>$CHALLENGE &&\n ++\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n +\n +\texpect_credential_query get <<-EOF &&\n +\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=foobar alg=test widget=1\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n ++\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n ++\twwwauth[]=Basic realm=\"example.com\"\n +\tEOF\n +\n +\texpect_credential_query store <<-EOF\n +\tprotocol=http\n -+\thost=$HOST_PORT\n ++\thost=$HTTPD_DEST\n +\tusername=alice\n +\tpassword=secret-passwd\n +\tEOF\n +'\n +\n -+test_expect_success 'http auth www-auth headers to credential helper invalid' '\n ++test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n -+\t# base64(\"alice:secret-passwd\")\n -+\tUSERPASS64=YWxpY2U6c2VjcmV0LXBhc3N3ZA== &&\n -+\texport USERPASS64 &&\n -+\n -+\tcat >auth.config <<-EOF &&\n -+\t[auth]\n -+\t\tchallenge = \"bearer:authority=\\\"id.example.com\\\" q=1 p=0\"\n -+\t\tchallenge = basic:realm=\\\"example.com\\\"\n -+\t\ttoken = basic:$USERPASS64\n -+\tEOF\n -+\n -+\tstart_http_server --auth-config=\"$TRASH_DIRECTORY/auth.config\" &&\n +\n +\tset_credential_reply get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n +\tusername=alice\n -+\tpassword=invalid-passwd\n ++\tpassword=secret-passwd\n +\tEOF\n +\n -+\ttest_must_fail git -c \"credential.helper=!\\\"$CREDENTIAL_HELPER\\\"\" ls-remote $ORIGIN_URL &&\n -+\n -+\texpect_credential_query get <<-EOF &&\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n ++\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n ++\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n +\n -+\texpect_credential_query erase <<-EOF\n -+\tprotocol=http\n -+\thost=$HOST_PORT\n -+\tusername=alice\n -+\tpassword=invalid-passwd\n -+\twwwauth[]=bearer authority=\"id.example.com\" q=1 p=0\n -+\twwwauth[]=basic realm=\"example.com\"\n -+\tEOF\n -+'\n ++\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n ++\n ++\t# Note that leading and trailing whitespace is important to correctly\n ++\t# simulate a continuation/folded header.\n ++\tprintf \"\">$CHALLENGE &&\n ++\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n ++\tprintf \" \\r\\n\" >>$CHALLENGE &&\n ++\tprintf \"\\tparam2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n ++\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\" >>$CHALLENGE &&\n +\n - test_done\n ++\ttest_config_global credential.helper test-helper &&\n ++\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n ++\n ++\texpect_credential_query get <<-EOF &&\n ++\tprotocol=http\n ++\thost=$HTTPD_DEST\n ++\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n ++\twwwauth[]=Basic realm=\"example.com\"\n + \tEOF\n + \n + \texpect_credential_query store <<-EOF\n\n-- \ngitgitgadget\n"},{"id":"471596","messageId":"230206.86fsbi5y63.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"d362f7016d34c4803adf42a88012997c66e0bde8.1675711789.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T20:32:53Z","receivedAt":"2023-02-06T20:36:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n>\n> Add a test showing simple anoymous HTTP access to an unprotected\n> repository, that results in no credential helper invocations.\n> Also add a test demonstrating simple basic authentication with\n> simple credential helper support.\n>\n> Leverage a no-parsed headers (NPH) CGI script so that we can directly\n> control the HTTP responses to simulate a multitude of good, bad and ugly\n> remote server implementations around auth.\n>\n> Signed-off-by: Matthew John Cheetham \n> ---\n> t/lib-httpd.sh | 1 +\n> t/lib-httpd/apache.conf | 6 +++\n> t/lib-httpd/nph-custom-auth.sh | 42 +++++++++++++++++\n> t/t5563-simple-http-auth.sh | 86 ++++++++++++++++++++++++++++++++++\n> 4 files changed, 135 insertions(+)\n> create mode 100755 t/lib-httpd/nph-custom-auth.sh\n> create mode 100755 t/t5563-simple-http-auth.sh\n>\n> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> index 608949ea80b..2c49569f675 100644\n> --- a/t/lib-httpd.sh\n> +++ b/t/lib-httpd.sh\n> @@ -137,6 +137,7 @@ prepare_httpd() {\n> \tinstall_script error-smart-http.sh\n> \tinstall_script error.sh\n> \tinstall_script apply-one-time-perl.sh\n> +\tinstall_script nph-custom-auth.sh\n> \n> \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n> \n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 0294739a77a..76335cdb24d 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n> \tSetEnv GIT_HTTP_EXPORT_ALL\n> \tSetEnv GIT_PROTOCOL\n> \n> +\n> +\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n> +\tSetEnv GIT_HTTP_EXPORT_ALL\n> +\tCGIPassAuth on\n> +\n> ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n> ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n> ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n> @@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n> ScriptAlias /error_smart/ error-smart-http.sh/\n> ScriptAlias /error/ error.sh/\n> ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n> \n> \tOptions FollowSymlinks\n> \n> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\n> new file mode 100755\n> index 00000000000..8f851aebac4\n> --- /dev/null\n> +++ b/t/lib-httpd/nph-custom-auth.sh\n> @@ -0,0 +1,42 @@\n> +#!/bin/sh\n> +\n> +VALID_CREDS_FILE=custom-auth.valid\n> +CHALLENGE_FILE=custom-auth.challenge\n> +ANONYMOUS_FILE=custom-auth.anonymous\n> +\n> +#\n> +# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n> +#\n> +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n> +# credential for the current request. Each line in the file is considered a\n> +# valid HTTP Authorization header value. For example:\n> +#\n> +# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n> +#\n> +# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n> +# in a 401 response if no valid authentication credentials were included in the\n> +# request. For example:\n> +#\n> +# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n> +# WWW-Authenticate: Basic realm=\"example.com\"\n> +#\n> +\n> +if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n> +\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n\nRather than \"test -f \"$f\" & grep ... \"$f\" I think you can just use only\n\"grep\", if the file doesn't exist it'll give you an error.\n\nIf you don't want to see that error just pipe it to /dev/null, in case\nthat's what you were trying to avoid with the \"check if it exists\nfirst\".\n\n> +echo 'HTTP/1.1 401 Authorization Required'\n> +if test -f \"$CHALLENGE_FILE\"\n> +then\n> +\tcat \"$CHALLENGE_FILE\"\n\nMaybe the same here, i.e. just:\n\n\tcat \"$f\" 2>/dev/null\n\n> +test_expect_success 'setup_credential_helper' '\n> +\tmkdir -p \"$TRASH_DIRECTORY/bin\" &&\n\nThe \"$TRASH_DIRECTORY\" is already created for you, so don't use \"-p\",\nunless something went wrong here..\n\n> +\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n> +\texport PATH &&\n> +\n> +\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n> +\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n> +\tcmd=$1\n> +\tteefile=$cmd-query.cred\n> +\tcatfile=$cmd-reply.cred\n> +\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n\nStyle: \">>$f\", not \">> $f\"\n\n> +\tif test \"$cmd\" = \"get\"; then\n\nStyle: We usually use \"\\nthen\", not \"; then\".\n"},{"id":"471598","messageId":"230206.86bkm65y2c.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"cd9a02ba94e9eb59b07d0b95140d5b880f122941.1675711789.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T20:36:41Z","receivedAt":"2023-02-06T20:38:53Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n\n> +static inline int skip_iprefix_mem(const char *buf, size_t len,\n> +\t\t\t\t const char *prefix,\n> +\t\t\t\t const char **out, size_t *outlen)\n> +{\n> +\tsize_t prefix_len = strlen(prefix);\n\nStyle nit: We typically use \\n\\n after the end of decls, so an extra\nnewline here before the code.\n\n> +\tif (len < prefix_len)\n> +\t\treturn 0;\n> +\n> +\tif (!strncasecmp(buf, prefix, prefix_len)){\n\nStyle: \")) {\", not \")){\".\n"},{"id":"471599","messageId":"230206.867cwu5xmu.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"149aedf55010718e22669575a148988eed0d8dcb.1675711789.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T20:45:49Z","receivedAt":"2023-02-06T20:48:02Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> From: Matthew John Cheetham \n\n> @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n> \tfprintf(fp, \"%s=%s\\n\", key, value);\n> }\n> \n> +static void credential_write_strvec(FILE *fp, const char *key,\n> +\t\t\t\t const struct strvec *vec)\n> +{\n> +\tchar *full_key = xstrfmt(\"%s[]\", key);\n\nFWIW you could avoid this allocation if you just renamed the current\n\"credential_write_item()\" to \"credential_write_fmt()\", and had it take a\nformat instead of its current hardcoded \"%s=%s\\n\".\n\nThen you could have two wrappers, credential_write_item() and\ncredential_write_items() (instead of \"strvec\"), the first passing\n\"%s=%s\\n\", the other \"%s[]=%s\\n\".\n\nJust a thought.\n\n> +\tfor (size_t i = 0; i < vec->nr; i++) {\n> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n\nThe {} here can be dropped in any case.\n\n"},{"id":"471607","messageId":"230206.86357i5vki.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T20:59:24Z","receivedAt":"2023-02-06T21:32:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:\n\n> Updates in v8\n> =============\n>\n> * Drop custom HTTP test helper tool in favour of using a CGI shell script\n> and Apache; avoiding the need to implement an HTTP server.\n>\n> * Avoid allocations in header reading callback unless we have a header we\n> care about; act on the char* from libcurl directly rather than create a\n> strbuf for each header.\n>\n> * Drop st_mult overflow guarding function in curl callback functions; we're\n> not allocating memory based on the resulting value and just adds to\n> potential confusion in the future.\n\nI just had some nit-y and other trivial comments spotted on a\nread-through, but with the caveat that I'm not too familiar with the\ncredential infrastructure this looks good to me.\n\nI'm rather neutral on the whole question of whether we eventually ship a\nhttpd in-tree, but I think the v7 to v8 clearly demonstrates that\nwhatever we do there, this topic is much improved by having that\nquestion un-tangled from the credential improvements here.\n\nThanks!\n\n"},{"id":"471611","messageId":"230206.86y1pa4gdh.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"60c3f1d0-2858-8811-7eb0-d6f586bf2ab8@gmx.de","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T21:32:45Z","receivedAt":"2023-02-06T21:46:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 02 2023, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> On Thu, 2 Feb 2023, Ævar Arnfjörð Bjarmason wrote:\n>\n>> On Thu, Feb 02 2023, Johannes Schindelin wrote:\n>>\n>> > On Thu, 26 Jan 2023, Junio C Hamano wrote:\n>> >\n>> >> Jeff King writes:\n>> >>\n>> >> >> Thanks, both. Let's merge it down.\n>> >> >\n>> >> > Sorry, I'm a bit late to the party, but I left some comments just now\n>> >> > (this topic had been on my review backlog for ages, but I never quite\n>> >> > got to it).\n>> >> >\n>> >> > Many of my comments were small bits that could be fixed on top (tiny\n>> >> > leaks, etc). But some of my comments were of the form \"no, do it totally\n>> >> > differently\". It may simply be too late for those ones, but let's see if\n>> >> > Matthew finds anything compelling in them.\n>> >>\n>> >> I do not mind reverting the merge to 'next' to have an improved\n>> >> version. Your \"do we really want to add a custom server based on\n>> >> questionable codebase whose quality as a test-bed for real world\n>> >> usage is dubious?\" is a valid concern.\n>> >\n>> > Except.\n>> >\n>> > Except that this code base would have made for a fine base to potentially\n>> > implement an HTTPS-based replacement for the aging and insecure\n>> > git-daemon.\n>> >\n>> > That code base (which is hardly as questionable codebase as you make it\n>> > sound because it has been in use for years in a slightly different form)\n>> > would have had the opportunity to mature in a relatively safe environment:\n>> > our test suite. And eventually, once robust enough, it could have been\n>> > extended to allow for easy and painless yet secure ad-hoc serving of Git\n>> > repositories, addressing the security concerns around git-daemon.\n>> >\n>> > And now that we're throwing out that code we don't have that opportunity,\n>> > making the goal to deprecate the git-daemon and replace it by something\n>> > that is as easy to set up but talks HTTPS instead much, much harder to\n>> > reach.\n>>\n>> There's many reasons for why you almost never see a git:// URL in the\n>> wild anymore.\n>\n> I am unwilling to accept that statement without any source to back it up.\n> Thin air is no substitute for reliable evidence.\n\nMost people exposing git over the Internet use the ssh or http\ntransport, and our own \"git\" protocol is relatively obscure.\n\nIf you need data I think major hosting sites not offering it, or\ndeprecating it, is a pretty strong signal, e.g. Microsoft with:\nhttps://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git\n\nBut if you'll grant me that it's 50/50 git/other protocols (I think it's\na *lot* more lopsided), then clearly combining git with 3rd party server\ncomponents isn't the limiting factor on deploying it.\n\nWhich is the point I was going for.\n\n>> But if \"easy and painless\" was synonymous with \"built with git\" or\n>> \"ships with git\" as you seem to be using it, surely it would be more\n>> common than doing the same with http or https, which requires an\n>> external server?\n>\n> Oh whoa... \"requires an external server\"?\n>\n> My entire point was to suggest a way forward for an _internal_ server that\n> speaks https:// instead of git://.\n\nI understand that.\n\n> So I am not suggesting what you seem to have understood me to suggest.\n\nI wasn't suggesting that, and you seem to have not read my reply to the\nend, which should have addressed that.\n\nBriefly, we'd like to be guaranteed to have regcomp() and regexec(), but\ndid the Git project write its own regex engine?\n\nNo, we imported (with some minor tweaks) one from glibc/gawk (whatever\ncurrent issues have cropped up with it lately...).\n\nSo can't we do the same for a httpd? If it really comes to \"we must have\nit in-tree\"?\n\nIt seems to me that there's a continuum here, which is at the very\nleast:\n\n1) We require an external package (e.g. ssh, or apache/httpd)\n2) We require an external package *or* built-in (e.g. our SHA-1\n implementations)\n3) We use an external package as-is (sha1dc)\n4) We adapt an external codebase, and perma-fork it (git-imap-send,\n although that example also kind of sucks)\n5) We write it \"in-house\" from scratch.\n\nIt seems to me from reading the upthread that we're jumping straight\nfrom #1 to #5, and it's not clear to me why that is.\n\nNot even that, we currently have CI tests running Apache on *nix boxes,\nbut you're suggesting a loss of coverage on Windows \n\nIs it really harder to just install (or even ship our own package of)\nApache for Windows than it is to embark on PID file handling, logging,\ntimeout management and the long tail of \"80% is easy, the rest is really\nhard\" of writing our own production-class httpd (as the suggestion is to\nhave it eventually mature beyond the test suite)?\n\nMaybe, all I'm saying (in trying to mediate the discussion between you\nand Jeff) is that it's not obvious to me why that is...\n"},{"id":"471794","messageId":"87f79e79-1591-ca28-4975-3bca5b8f7266@github.com","threadId":"58425","inReplyTo":"d362f7016d34c4803adf42a88012997c66e0bde8.1675711789.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-02-08T20:24:30Z","receivedAt":"2023-02-08T20:24:54Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> index 608949ea80b..2c49569f675 100644\n> --- a/t/lib-httpd.sh\n> +++ b/t/lib-httpd.sh\n> @@ -137,6 +137,7 @@ prepare_httpd() {\n> \tinstall_script error-smart-http.sh\n> \tinstall_script error.sh\n> \tinstall_script apply-one-time-perl.sh\n> +\tinstall_script nph-custom-auth.sh\n> \n> \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n> \n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 0294739a77a..76335cdb24d 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n> \tSetEnv GIT_HTTP_EXPORT_ALL\n> \tSetEnv GIT_PROTOCOL\n> \n> +\n> +\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n> +\tSetEnv GIT_HTTP_EXPORT_ALL\n> +\tCGIPassAuth on\n> +\n> ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n> ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n> ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n> @@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n> ScriptAlias /error_smart/ error-smart-http.sh/\n> ScriptAlias /error/ error.sh/\n> ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n\nThis setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'\nscript) is nice and straightforward. \n\n> \n> \tOptions FollowSymlinks\n> \n> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\n> new file mode 100755\n> index 00000000000..8f851aebac4\n> --- /dev/null\n> +++ b/t/lib-httpd/nph-custom-auth.sh\n> @@ -0,0 +1,42 @@\n> +#!/bin/sh\n> +\n> +VALID_CREDS_FILE=custom-auth.valid\n> +CHALLENGE_FILE=custom-auth.challenge\n> +ANONYMOUS_FILE=custom-auth.anonymous\n> +\n> +#\n> +# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n> +#\n> +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n> +# credential for the current request. Each line in the file is considered a\n> +# valid HTTP Authorization header value. For example:\n> +#\n> +# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n> +#\n> +# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n> +# in a 401 response if no valid authentication credentials were included in the\n> +# request. For example:\n> +#\n> +# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n> +# WWW-Authenticate: Basic realm=\"example.com\"\n> +#\n> +\n> +if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n> +\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n\nSo there are two cases where you want to return a '200 OK' response:\n\n1. anonymous access is allowed (indicated by $ANONYMOUS_FILE existing)\n2. anonymous access is *not* allowed, 'HTTP_AUTHORIZATION' is non-empty, and\n it matches at least one line in $VALID_CREDS_FILE\n\nDoes the '$' at the end of \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" need to be\nescaped? I'm guessing it doesn't *need* to be based on the fact that the\ntests are passing, but it might be safer to escape it anyway.\n\nI see what you're going for with the \"nopenopenope\" substitution, but I\nthink you could be more explicit about requiring that 'HTTP_AUTHORIZATION'\nis set without the need for a special invalid value fallback:\n\n if test -f \"$ANONYMOUS_FILE\" || (test -n \"$HTTP_AUTHORIZATION\" && \\\n \tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\")\n\nNote the addition of '-s' to 'grep' - it seems cleaner than redirecting to\n'/dev/null' (as Ævar suggested [1]) while achieving the same result.\n\n[1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/\n\n> +then\n> +\t# Note that although git-http-backend returns a status line, it\n> +\t# does so using a CGI 'Status' header. Because this script is an\n> +\t# No Parsed Headers (NPH) script, we must return a real HTTP\n> +\t# status line.\n> +\t# This is only a test script, so we don't bother to check for\n> +\t# the actual status from git-http-backend and always return 200.\n> +\techo 'HTTP/1.1 200 OK'\n> +\texec \"$GIT_EXEC_PATH\"/git-http-backend\n\nI'm not familiar with 'exec', but a cursory look at the documentation shows\nthat, because this replaces the current shell, it will exit with the code\nfrom 'git-http-backend', so there's no risk of continuing on to print the\n'401 Authorization Required' response & challenge handling. \n\n> +fi\n> +\n> +echo 'HTTP/1.1 401 Authorization Required'\n> +if test -f \"$CHALLENGE_FILE\"\n> +then\n> +\tcat \"$CHALLENGE_FILE\"\n> +fi\n\nIn contrast to Ævar's comments in the review linked earlier, I like having\nthe explicit 'test -f' (to sort of \"self-document\" that the challenge is\nonly issued if $CHALLENGE_FILE exists). I think you're fine keeping this\nas-is or changing it, depending on your preference.\n\n> +test_expect_success 'access anonymous no challenge' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> +\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n> +\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n> +'\n> +\n> +test_expect_success 'access using basic auth' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> +\n> +\tset_credential_reply get <<-EOF &&\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tEOF\n> +\n> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n> +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n> +\tEOF\n> +\n> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n> +\tWWW-Authenticate: Basic realm=\"example.com\"\n> +\tEOF\n> +\n> +\ttest_config_global credential.helper test-helper &&\n> +\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n> +\n> +\texpect_credential_query get <<-EOF &&\n> +\tprotocol=http\n> +\thost=$HTTPD_DEST\n> +\tEOF\n> +\n> +\texpect_credential_query store <<-EOF\n> +\tprotocol=http\n> +\thost=$HTTPD_DEST\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tEOF\n> +'\n> +\n> +test_done\n\nAnd these tests properly exercise the custom auth handling. \n\nWhile I wasn't as opposed to the custom HTTP handler as others that have\ncommented, I do appreciate the relative simplicity of this new Apache setup\nand like that it's still pretty easy to test. Nice work!\n\n"},{"id":"471799","messageId":"983fc35b-55e8-50df-5035-191a10b4ddac@github.com","threadId":"58425","inReplyTo":"cd9a02ba94e9eb59b07d0b95140d5b880f122941.1675711789.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-02-08T21:05:52Z","receivedAt":"2023-02-08T21:06:16Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index a76d0526f79..f11c44517d7 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -1266,6 +1266,28 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n> \treturn 0;\n> }\n> \n> +/*\n> + * Like skip_prefix_mem, but compare case-insensitively. Note that the\n> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n> + * characters or locale-specific conversions).\n> + */\n> +static inline int skip_iprefix_mem(const char *buf, size_t len,\n> +\t\t\t\t const char *prefix,\n> +\t\t\t\t const char **out, size_t *outlen)\n> +{\n> +\tsize_t prefix_len = strlen(prefix);\n> +\tif (len < prefix_len)\n> +\t\treturn 0;\n> +\n> +\tif (!strncasecmp(buf, prefix, prefix_len)){\n> +\t\t*out = buf + prefix_len;\n> +\t\t*outlen = len - prefix_len;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n> +\n> static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n> {\n> \tunsigned long ul;\n> diff --git a/http.c b/http.c\n> index 8a5ba3f4776..7a56a3db5f7 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> \treturn nmemb;\n> }\n> \n> +/*\n> + * A folded header continuation line starts with at least one single whitespace\n> + * character. It is not a continuation line if the line is *just* a newline.\n> + * The RFC for HTTP states that CRLF is the header field line ending, but some\n> + * servers may use LF only; we accept both.\n> + */\n> +static inline int is_hdr_continuation(const char *ptr, const size_t size)\n> +{\n> +\t/* totally empty line or normal header */\n> +\tif (!size || !isspace(*ptr))\n> +\t\treturn 0;\n> +\n> +\t/* empty line with LF line ending */\n> +\tif (size == 1 && ptr[0] == '\\n')\n> +\t\treturn 0;\n> +\n> +\t/* empty line with CRLF line ending */\n> +\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n> +\t\treturn 0;\n> +\n> +\treturn 1;\n> +}\n> +\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tconst char *val;\n> +\tsize_t val_len;\n> +\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n> +\t\tstrbuf_add(&buf, val, val_len);\n> +\n> +\t\t/*\n> +\t\t * Strip the CRLF that should be present at the end of each\n> +\t\t * field as well as any trailing or leading whitespace from the\n> +\t\t * value.\n> +\t\t */\n> +\t\tstrbuf_trim(&buf);\n> +\n> +\t\tstrvec_push(values, buf.buf);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t */\n> +\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n> +\t\t/*\n> +\t\t * Trim the CRLF and any leading or trailing from this line.\n> +\t\t */\n> +\t\tstrbuf_add(&buf, ptr, size);\n> +\t\tstrbuf_trim(&buf);\n> +\n> +\t\t/*\n> +\t\t * At this point we should always have at least one existing\n> +\t\t * value, even if it is empty. Do not bother appending the new\n> +\t\t * value if this continuation header is itself empty.\n> +\t\t */\n> +\t\tif (!values->nr) {\n> +\t\t\tBUG(\"should have at least one existing header value\");\n> +\t\t} else if (buf.len) {\n> +\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n> +\n> +\t\t\t/* Join two non-empty values with a single space. */\n> +\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n> +\n> +\t\t\tstrvec_pop(values);\n> +\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n> +\t\t\tfree(prev);\n> +\t\t}\n> +\n> +\t\tgoto exit;\n> +\t}\n> +\n> +\t/* This is the start of a new header we don't care about */\n> +\thttp_auth.header_is_last_match = 0;\n> +\n> +\t/*\n> +\t * If this is a HTTP status line and not a header field, this signals\n> +\t * a different HTTP response. libcurl writes all the output of all\n> +\t * response headers of all responses, including redirects.\n> +\t * We only care about the last HTTP request response's headers so clear\n> +\t * the existing array.\n> +\t */\n> +\tif (!strncasecmp(ptr, \"http/\", 5))\n> +\t\tstrvec_clear(values);\n\nI found this updated version of 'fwrite_wwwauth()' (using\n'skip_iprefix_mem()', 'is_hdr_continuation()', and 'strncasecmp()') a bit\neasier to read than previous iterations - possibly because all the\nprefix-skipping is done before adding to 'buf', so 'buf' represents *only*\nthe line's header value (possibly with leading/trailing whitespace, which is\ntrimmed). Plus, avoiding unnecessary allocations is always nice. \n\n> +\n> +exit:\n> +\tstrbuf_release(&buf);\n> +\treturn size;\n> +}\n\n"},{"id":"471805","messageId":"9012b8a3-4abe-d3c6-41ee-f28931869ad7@github.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers","fromName":"Victoria Dye","fromEmail":"vdye@github.com","sentAt":"2023-02-08T21:29:15Z","receivedAt":"2023-02-08T21:29:20Z","isPatch":true,"sender":{"key":"vdye@github.com","avatar":"https://avatars.githubusercontent.com/u/3619353?v=4"},"body":"Matthew John Cheetham via GitGitGadget wrote:\n> Updates in v8\n> =============\n> \n> * Drop custom HTTP test helper tool in favour of using a CGI shell script\n> and Apache; avoiding the need to implement an HTTP server.\n> \n> * Avoid allocations in header reading callback unless we have a header we\n> care about; act on the char* from libcurl directly rather than create a\n> strbuf for each header.\n> \n> * Drop st_mult overflow guarding function in curl callback functions; we're\n> not allocating memory based on the resulting value and just adds to\n> potential confusion in the future.\n> \nThe core functionality change in this series (passing through\n'WWW-Authenticate' headers to credential requests) didn't change much\nbetween v7 and v8, and what was updated was a net improvement [1].\nOtherwise, the new test infrastructure is concise and made for an easy\nre-review; I had one small suggestion for patch 1 [2], but it's not critical\nenough on its own to hold this series up. I think this could probably be\nmerged to 'next' as-is, but a final re-roll with some of the minor fixups\nsuggested in other reviews wouldn't hurt either. \n\nThanks again for the time & effort you've put into perfecting these patches!\n\n[1] https://lore.kernel.org/git/983fc35b-55e8-50df-5035-191a10b4ddac@github.com/\n[2] https://lore.kernel.org/git/87f79e79-1591-ca28-4975-3bca5b8f7266@github.com/\n\n"},{"id":"471806","messageId":"xmqqh6vvq0uy.fsf@gitster.g","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"Re: [PATCH v8 0/3] Enhance credential helper protocol to include auth headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-08T21:54:45Z","receivedAt":"2023-02-08T21:54:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> In this patch series I update the existing credential helper design in order\n> to allow for some new scenarios, and future evolution of auth methods that\n> Git hosts may wish to provide. I outline the background, summary of changes\n> and some challenges below.\n>\n> Testing these new additions, I use a small CGI shell script that acts as a\n> frontend to git-http-backend; simple authentication is configurable by\n> files.\n\nThanks for an update, and thanks Victoria and Ævar for your prompt\nreviews.\n\nQueued on 'seen', anticipating a small finishing touches before\nmerging to 'next'.\n\n"},{"id":"471850","messageId":"230209.86v8kbvz51.gmgdl@evledraar.gmail.com","threadId":"58425","inReplyTo":"87f79e79-1591-ca28-4975-3bca5b8f7266@github.com","subject":"Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T11:19:03Z","receivedAt":"2023-02-09T11:58:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 08 2023, Victoria Dye wrote:\n\n> Matthew John Cheetham via GitGitGadget wrote:\n>> ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n>> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n>\n> This setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'\n> script) is nice and straightforward. \n\n*nod*\n\n> [...]\n>> +if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n>> +\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n> [...]\n> if test -f \"$ANONYMOUS_FILE\" || (test -n \"$HTTP_AUTHORIZATION\" && \\\n> \tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\")\n>\n> Note the addition of '-s' to 'grep' - it seems cleaner than redirecting to\n> '/dev/null' (as Ævar suggested [1]) while achieving the same result.\n>\n> [1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/\n\nI wondered if it's in POSIX, turns out it is!:\nhttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html\n\nBut we don't have any existing use of it, even for things in POSIX it's\noften a gamble what the exact semantics are on our long tail of *nix,\ne.g. old AIX.\n\nIn general I'd think we could just avoid \"-s\" or piping to \"/dev/null\"\nhere, i.e. under \"-x\" or whatever it's informative to know it doesn't\nexist from the stderr, but on second look I think both of us long track\nof a larger issue here...\n> [...]\n>> +fi\n>> +\n>> +echo 'HTTP/1.1 401 Authorization Required'\n>> +if test -f \"$CHALLENGE_FILE\"\n>> +then\n>> +\tcat \"$CHALLENGE_FILE\"\n>> +fi\n>\n> In contrast to Ævar's comments in the review linked earlier, I like having\n> the explicit 'test -f' (to sort of \"self-document\" that the challenge is\n> only issued if $CHALLENGE_FILE exists). I think you're fine keeping this\n> as-is or changing it, depending on your preference.\n\nLooking at this again I think we should just have it be unconditional\nhere. I.e. it looks like we both assumed that this needs to be a\nconditional, but actually every /custom_auth/ test also sets up this\n\"$CHALLENGE_FILE\".\n\nSo this \"test -f\" seems to only serve the purpose of burying an error\nunder the rug if things have already gone wrong.\n\nBut if we're making these requests why are we writing a script that\nhandles the combination of 3 parameters, and needs to second guess\nthings? We can just create N urls and N scripts instead. So I tried this\nfix-up instead:\n\t\n\tdiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n\tindex 7979605d344..c25e3000db0 100644\n\t--- a/t/lib-httpd.sh\n\t+++ b/t/lib-httpd.sh\n\t@@ -141,6 +141,7 @@ prepare_httpd() {\n\t \tinstall_script error.sh\n\t \tinstall_script apply-one-time-perl.sh\n\t \tinstall_script nph-custom-auth.sh\n\t+\tinstall_script nph-custom-auth-anon.sh\n\t \n\t \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n\t \n\tdiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n\tindex 2aac922376c..7a63a9169c3 100644\n\t--- a/t/lib-httpd/apache.conf\n\t+++ b/t/lib-httpd/apache.conf\n\t@@ -140,6 +140,7 @@ ScriptAlias /error_smart/ error-smart-http.sh/\n\t ScriptAlias /error/ error.sh/\n\t ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n\t ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n\t+ScriptAliasMatch /custom_auth_anon/(.*) nph-custom-auth-anon.sh/$1\n\t \n\t \tOptions FollowSymlinks\n\t \n\tdiff --git a/t/lib-httpd/nph-custom-auth-anon.sh b/t/lib-httpd/nph-custom-auth-anon.sh\n\tnew file mode 100755\n\tindex 00000000000..3c7a24fed6b\n\t--- /dev/null\n\t+++ b/t/lib-httpd/nph-custom-auth-anon.sh\n\t@@ -0,0 +1,4 @@\n\t+#!/bin/sh\n\t+\n\t+echo 'HTTP/1.1 200 OK'\n\t+exec \"$GIT_EXEC_PATH\"/git-http-backend\n\tdiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\n\tindex 8f851aebac4..e3ee61c8c9e 100755\n\t--- a/t/lib-httpd/nph-custom-auth.sh\n\t+++ b/t/lib-httpd/nph-custom-auth.sh\n\t@@ -1,28 +1,15 @@\n\t #!/bin/sh\n\t \n\t+set -e\n\t+\n\t VALID_CREDS_FILE=custom-auth.valid\n\t-CHALLENGE_FILE=custom-auth.challenge\n\t-ANONYMOUS_FILE=custom-auth.anonymous\n\t \n\t-#\n\t-# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n\t-#\n\t # If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n\t # credential for the current request. Each line in the file is considered a\n\t # valid HTTP Authorization header value. For example:\n\t #\n\t # Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n\t-#\n\t-# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n\t-# in a 401 response if no valid authentication credentials were included in the\n\t-# request. For example:\n\t-#\n\t-# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n\t-# WWW-Authenticate: Basic realm=\"example.com\"\n\t-#\n\t-\n\t-if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n\t-\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n\t+if grep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\"\n\t then\n\t \t# Note that although git-http-backend returns a status line, it\n\t \t# does so using a CGI 'Status' header. Because this script is an\n\t@@ -31,12 +18,15 @@ then\n\t \t# This is only a test script, so we don't bother to check for\n\t \t# the actual status from git-http-backend and always return 200.\n\t \techo 'HTTP/1.1 200 OK'\n\t-\texec \"$GIT_EXEC_PATH\"/git-http-backend\n\t+\texit 1\n\t fi\n\t \n\t+# Output of our challenge file as headers\n\t+# in a 401 response if no valid authentication credentials were included in the\n\t+# request. For example:\n\t+#\n\t+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n\t+# WWW-Authenticate: Basic realm=\"example.com\"\n\t echo 'HTTP/1.1 401 Authorization Required'\n\t-if test -f \"$CHALLENGE_FILE\"\n\t-then\n\t-\tcat \"$CHALLENGE_FILE\"\n\t-fi\n\t+cat custom-auth.challenge\n\t echo\n\tdiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\n\tindex a7b1e5bd1af..feb8149de8f 100755\n\t--- a/t/t5563-simple-http-auth.sh\n\t+++ b/t/t5563-simple-http-auth.sh\n\t@@ -47,8 +47,7 @@ test_expect_success 'setup repository' '\n\t \n\t test_expect_success 'access anonymous no challenge' '\n\t \ttest_when_finished \"per_test_cleanup\" &&\n\t-\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n\t-\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n\t+\tgit ls-remote \"$HTTPD_URL/custom_auth_anon/repo.git\"\n\t '\n\t \n\t test_expect_success 'access using basic auth' '\n\nI think that's much better, now we just have a 2-line script to handle\nthis \"anon auth\" case. Instead of creating a \"custom-auth.anonymous\"\nfile to communicate how the remote end should behave, let's just\ncommunicate that by requesting a different URL, one that accepts\nanonymous authentication.\n\nI did insert a deliberate bug here, or:\n\n\t-\texec \"$GIT_EXEC_PATH\"/git-http-backend\n\t+\texit 1\n\nSo aside from your \"exec\" comment it seems both of us missed that this\n\"exec\" does nothing useful, the test will fail if we emit different\nheaders, but it doesn't matter that we execute the git-http-backend.\n\nOr maybe it does, but the tests aren't good enough to spot the\ndifference.\n\nThe above is a rough WIP, I'm leaving it here for Matthew to follow-up\non. I think it might benefit from being further split-up, i.e. we know\nwhich URLs we expect to fail auth, so if we just had another URL for\n\"the auth response fails here\" we'd have 3x trivial scripts with no\nif/else; but maybe that sucks, I didn't try it.\n"},{"id":"471854","messageId":"Y+Twa22Gw2nzV8sG@coredump.intra.peff.net","threadId":"58425","inReplyTo":"DB9PR03MB9831A708EA98E198591F6632C0DA9@DB9PR03MB9831.eurprd03.prod.outlook.com","subject":"Re: [PATCH v7 12/12] credential: add WWW-Authenticate header to cred requests","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-09T13:08:59Z","receivedAt":"2023-02-09T13:09:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 06, 2023 at 11:18:03AM -0800, Matthew John Cheetham wrote:\n\n> > could be normalized as:\n> > \n> > www-auth-challenge=Basic realm=\"foo\"\n> > www-auth-challenge=OtherAuth realm=\"bar\"\n> > www-auth-challenge=YetAnotherScheme some-token\n> > \n> > which saves each helper from having to do the same work. Likewise, we\n> > can do a _little_ more parsing to get:\n> > \n> > www-auth-basic=realm=\"foo\"\n> > www-auth-otherauth=realm=\"bar\"\n> > www-auth-yetanotherscheme=some-token\n> > \n> > I don't think we can go beyond there, though, without understanding the\n> > syntax of individual schemes. Which is a shame, as one of the goals of\n> > the credential format was to let the helpers do as little as possible\n> > (so they can't get it wrong!). But helpers are stuck doing things like\n> > handling backslashed double-quotes, soaking up extra whitespace, etc.\n> \n> This key format wouldn't make it obviously easier for simple helpers to\n> understand. Now they no longer have well-known keys but a key prefix.\n\nYes, though I don't think that's particularly complicated to parse.\nEither way we're just flattening a tuple of (a, b, c) from \"a=b c\" to\n\"a-b=c\". The value is in normalizing the syntax, so that helpers don't\nhave to deal with both \"a=b c d e\" and (\"a=b c\", \"a=d e\") themselves.\n\nAnother way to do that normalization would be to have Git convert:\n\n WWW-Authenticate: Basic realm=\"foo\" OtherAuth realm=\"bar\"\n\ninto:\n\n WWW-Authenticate: Basic realm=\"foo\"\n WWW-Authenticate: OtherAuth realm=\"bar\"\n\nwhich then becomes (at the credential level):\n\n www-auth[]=Basic realm=\"foo\"\n www-auth[]=OtherAuth realm=\"bar\"\n\nAnd likewise to normalize whitespace, etc, so each individual helper\ndoesn't have to (or risk getting confused/exploited). That said...\n\n> My overall goal here is to have Git know less about auth, so it treats\n> all values as totally opaque. The only logic added is around reconstructing\n> folded headers, which is just HTTP and not auth specific.\n\nYeah, in general I agree with the notion that Git is mostly just passing\naround opaque tokens. We do have to understand some syntax (like\nfolding!) at the HTTP level, so I think some syntactic normalization /\nsimplification is reasonable.\n\nBUT. I think you are right that embedding it into the schema of the\nhelper protocol is probably bad. If the point is that the two forms of\nmy Basic / OtherAuth example are semantically equivalent, then we can\nalways decide later to convert between one and the other as a favor to\nhelpers. Whereas baking it into the schema is a promise for Git to\nalways parse and understand the headers.\n\nSo let me retract my suggestion, and we can leave \"maybe normalize\nheaders to save helpers some work\" as a possible topic for later (if\nindeed it ever even becomes a problem in practice).\n\n> > realm=foo\n> > while read line; do\n> > case \"$line\" in\n> > www-auth-basic=)\n> > value=${line#*=}\n> > \t# oops, we're just assuming it's realm= here, and we're\n> > \t# not handling quotes at all. I think it could technically be\n> > \t# realm=foo or realm=\"foo\"\n> > \trealm=${value#realm=}\n> > \t;;\n> > esac\n> > done\n> > echo password=$(pass \"pats-by-realm/$realm\")\n> > \n> > which could be made a lot easier if we did more parsing (e.g.,\n> > www-auth-basic-realm or something). I dunno. Maybe that is just opening\n> > up a can of worms, as we're stuffing structured data into a linearized\n> > key-value list. The nice thing about your proposal is that Git does not\n> > even have to know anything about these schemes; it's all the problem of\n> > the helper. My biggest fear is just that we'll want to shift that later,\n> > and we'll be stuck with this microformat forever.\n> \n> I'm not sure there's such a continuous scale between simple and 'complex'\n> helpers that would mean there'd be a simple shell script generating\n> OAuth or DPoP credentials instead of a helper written in a higher-level\n> language where parsing the headers is one of the simpler challenges faced.\n\nFor the most part, yeah. I tried to form the above example as something\nthat was really just relying on \"basic\", but taking in more information\n/ context than we currently provide (and that your patch would provide).\nI admit it's a stretch, though. Are there any servers which actually use\na Basic realm to distinguish between two credential's you'd want to\nprovide? I don't think I've seen one.\n\n(Not to mention that people scripting helpers like this is probably\npretty rare; I do, but you can probably consider me a special case. And\nif things got more complicated I'd just turn to Perl anyway. ;) ).\n\n> I had considered another model whereby we forgo the key=value line model,\n> and hide another format behind the 'final' terminating new-line. However\n> I thought this would be even more distuptive.\n\nYeah, if we can shoe-horn this into the existing key/value model, that's\nmuch better. The original intent with the final newline is that you\ncould read multiple credentials in a list, though in the end I don't\nrecall that we ever used that feature anyway.\n\n-Peff\n"},{"id":"471855","messageId":"Y+TxRDatM9Iirwwu@coredump.intra.peff.net","threadId":"58425","inReplyTo":"DB9PR03MB983150E73B1C963C628CBE75C0DA9@DB9PR03MB9831.eurprd03.prod.outlook.com","subject":"Re: [PATCH v7 11/12] http: read HTTP WWW-Authenticate response headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-09T13:12:36Z","receivedAt":"2023-02-09T13:12:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 06, 2023 at 11:25:49AM -0800, Matthew John Cheetham wrote:\n\n> > I guess I would have expected some level of abstraction here between the\n> > credential subsystem and the http subsystem, where the latter is parsing\n> > and then sticking opaque data into the credential to ferry to the\n> > helpers.\n> > \n> > But it probably isn't that big a deal either way. Even though there are\n> > non-http credentials, it's not too unreasonable for the credential\n> > system to be aware of http specifically.\n> \n> I had considered possibly introducing an opaque property-bag style of\n> 'protocol-specific properties' that, for example, http.c would add the\n> WWW-Authenticate headers to as something like `http.wwwauth[]`.\n> Other protocols (like smtp:// or cert://) could add their own properties\n> if they needed or wanted to also.\n> \n> Thoughts?\n\nAt the protocol level, I don't see much point. wwwauth sufficiently\nimplies \"http\", and any helper is free to ignore or respect keys as\nappropriate to what it can handle. A flat namespace is fine.\n\nHere I was more talking about the internal implementation. Mostly it was\njust funky that this internal http state flag was stuck into the\ncredential struct. I think it could be removed with some minor pain, but\nit's probably not too big a deal (the pain at all is only because we are\nhaving to bring this state across multiple curl callbacks).\n\nSo let's go with it for now.\n\n-Peff\n"},{"id":"472157","messageId":"AS2PR03MB9815361C9F2E52C71F7284D0C0A39@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230206.867cwu5xmu.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v8 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-15T19:19:20Z","receivedAt":"2023-02-15T19:19:39Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-06 12:45, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Mon, Feb 06 2023, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> From: Matthew John Cheetham \n> \n>> @@ -263,6 +263,16 @@ static void credential_write_item(FILE *fp, const char *key, const char *value,\n>> \tfprintf(fp, \"%s=%s\\n\", key, value);\n>> }\n>> \n>> +static void credential_write_strvec(FILE *fp, const char *key,\n>> +\t\t\t\t const struct strvec *vec)\n>> +{\n>> +\tchar *full_key = xstrfmt(\"%s[]\", key);\n> \n> FWIW you could avoid this allocation if you just renamed the current\n> \"credential_write_item()\" to \"credential_write_fmt()\", and had it take a\n> format instead of its current hardcoded \"%s=%s\\n\".\n> \n> Then you could have two wrappers, credential_write_item() and\n> credential_write_items() (instead of \"strvec\"), the first passing\n> \"%s=%s\\n\", the other \"%s[]=%s\\n\".\n> \n> Just a thought.\n\nGiven this is the only `items/strvec` that I'm writing out, just inlining\nthe for-loop and calling `credential_write_item` directly with the key as\n\"wwwauth[]\" would avoid the allocation without needing to make the write_item\nfunction more complicated.\n\n-static void credential_write_strvec(FILE *fp, const char *key,\n- const struct strvec *vec)\n-{\n- char *full_key = xstrfmt(\"%s[]\", key);\n- for (size_t i = 0; i < vec->nr; i++) {\n- credential_write_item(fp, full_key, vec->v[i], 0);\n- }\n- free(full_key);\n-}\n-\n void credential_write(const struct credential *c, FILE *fp)\n {\n credential_write_item(fp, \"protocol\", c->protocol, 1);\n@@ -280,7 +270,10 @@ void credential_write(const struct credential *c, FILE *fp)\n credential_write_item(fp, \"path\", c->path, 0);\n credential_write_item(fp, \"username\", c->username, 0);\n credential_write_item(fp, \"password\", c->password, 0);\n- credential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n+ for (size_t i = 0; i < c->wwwauth_headers.nr; i++)\n+ credential_write_item(fp, \"wwwauth[]\", c->wwwauth_headers.v[i],\n+ 0);\n }\n\n\n>> +\tfor (size_t i = 0; i < vec->nr; i++) {\n>> +\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n> \n> The {} here can be dropped in any case.\n> \n"},{"id":"472158","messageId":"AS2PR03MB981540DF75A83F637F715463C0A39@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"230209.86v8kbvz51.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v8 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-15T19:32:11Z","receivedAt":"2023-02-15T19:32:27Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-09 03:19, Ævar Arnfjörð Bjarmason wrote:\n\n> \n> On Wed, Feb 08 2023, Victoria Dye wrote:\n> \n>> Matthew John Cheetham via GitGitGadget wrote:\n>>> ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n>>> +ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n>>\n>> This setup (redirecting '/custom_auth/' routes to the 'nph-custom-auth.sh'\n>> script) is nice and straightforward. \n> \n> *nod*\n> \n>> [...]\n>>> +if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n>>> +\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n>> [...]\n>> if test -f \"$ANONYMOUS_FILE\" || (test -n \"$HTTP_AUTHORIZATION\" && \\\n>> \tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\")\n>>\n>> Note the addition of '-s' to 'grep' - it seems cleaner than redirecting to\n>> '/dev/null' (as Ævar suggested [1]) while achieving the same result.\n>>\n>> [1] https://lore.kernel.org/git/230206.86fsbi5y63.gmgdl@evledraar.gmail.com/\n> \n> I wondered if it's in POSIX, turns out it is!:\n> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html\n> \n> But we don't have any existing use of it, even for things in POSIX it's\n> often a gamble what the exact semantics are on our long tail of *nix,\n> e.g. old AIX.\n> \n> In general I'd think we could just avoid \"-s\" or piping to \"/dev/null\"\n> here, i.e. under \"-x\" or whatever it's informative to know it doesn't\n> exist from the stderr, but on second look I think both of us long track\n> of a larger issue here...\n>> [...]\n>>> +fi\n>>> +\n>>> +echo 'HTTP/1.1 401 Authorization Required'\n>>> +if test -f \"$CHALLENGE_FILE\"\n>>> +then\n>>> +\tcat \"$CHALLENGE_FILE\"\n>>> +fi\n>>\n>> In contrast to Ævar's comments in the review linked earlier, I like having\n>> the explicit 'test -f' (to sort of \"self-document\" that the challenge is\n>> only issued if $CHALLENGE_FILE exists). I think you're fine keeping this\n>> as-is or changing it, depending on your preference.\n> \n> Looking at this again I think we should just have it be unconditional\n> here. I.e. it looks like we both assumed that this needs to be a\n> conditional, but actually every /custom_auth/ test also sets up this\n> \"$CHALLENGE_FILE\".\n> \n> So this \"test -f\" seems to only serve the purpose of burying an error\n> under the rug if things have already gone wrong.\n> \n> But if we're making these requests why are we writing a script that\n> handles the combination of 3 parameters, and needs to second guess\n> things? We can just create N urls and N scripts instead. So I tried this\n> fix-up instead:\n> \t\n> \tdiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> \tindex 7979605d344..c25e3000db0 100644\n> \t--- a/t/lib-httpd.sh\n> \t+++ b/t/lib-httpd.sh\n> \t@@ -141,6 +141,7 @@ prepare_httpd() {\n> \t \tinstall_script error.sh\n> \t \tinstall_script apply-one-time-perl.sh\n> \t \tinstall_script nph-custom-auth.sh\n> \t+\tinstall_script nph-custom-auth-anon.sh\n> \t \n> \t \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n> \t \n> \tdiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> \tindex 2aac922376c..7a63a9169c3 100644\n> \t--- a/t/lib-httpd/apache.conf\n> \t+++ b/t/lib-httpd/apache.conf\n> \t@@ -140,6 +140,7 @@ ScriptAlias /error_smart/ error-smart-http.sh/\n> \t ScriptAlias /error/ error.sh/\n> \t ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n> \t ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n> \t+ScriptAliasMatch /custom_auth_anon/(.*) nph-custom-auth-anon.sh/$1\n> \t \n> \t \tOptions FollowSymlinks\n> \t \n> \tdiff --git a/t/lib-httpd/nph-custom-auth-anon.sh b/t/lib-httpd/nph-custom-auth-anon.sh\n> \tnew file mode 100755\n> \tindex 00000000000..3c7a24fed6b\n> \t--- /dev/null\n> \t+++ b/t/lib-httpd/nph-custom-auth-anon.sh\n> \t@@ -0,0 +1,4 @@\n> \t+#!/bin/sh\n> \t+\n> \t+echo 'HTTP/1.1 200 OK'\n> \t+exec \"$GIT_EXEC_PATH\"/git-http-backend\n> \tdiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\n> \tindex 8f851aebac4..e3ee61c8c9e 100755\n> \t--- a/t/lib-httpd/nph-custom-auth.sh\n> \t+++ b/t/lib-httpd/nph-custom-auth.sh\n> \t@@ -1,28 +1,15 @@\n> \t #!/bin/sh\n> \t \n> \t+set -e\n> \t+\n> \t VALID_CREDS_FILE=custom-auth.valid\n> \t-CHALLENGE_FILE=custom-auth.challenge\n> \t-ANONYMOUS_FILE=custom-auth.anonymous\n> \t \n> \t-#\n> \t-# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n> \t-#\n> \t # If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n> \t # credential for the current request. Each line in the file is considered a\n> \t # valid HTTP Authorization header value. For example:\n> \t #\n> \t # Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n> \t-#\n> \t-# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n> \t-# in a 401 response if no valid authentication credentials were included in the\n> \t-# request. For example:\n> \t-#\n> \t-# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n> \t-# WWW-Authenticate: Basic realm=\"example.com\"\n> \t-#\n> \t-\n> \t-if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n> \t-\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n> \t+if grep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\"\n> \t then\n> \t \t# Note that although git-http-backend returns a status line, it\n> \t \t# does so using a CGI 'Status' header. Because this script is an\n> \t@@ -31,12 +18,15 @@ then\n> \t \t# This is only a test script, so we don't bother to check for\n> \t \t# the actual status from git-http-backend and always return 200.\n> \t \techo 'HTTP/1.1 200 OK'\n> \t-\texec \"$GIT_EXEC_PATH\"/git-http-backend\n> \t+\texit 1\n> \t fi\n> \t \n> \t+# Output of our challenge file as headers\n> \t+# in a 401 response if no valid authentication credentials were included in the\n> \t+# request. For example:\n> \t+#\n> \t+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n> \t+# WWW-Authenticate: Basic realm=\"example.com\"\n> \t echo 'HTTP/1.1 401 Authorization Required'\n> \t-if test -f \"$CHALLENGE_FILE\"\n> \t-then\n> \t-\tcat \"$CHALLENGE_FILE\"\n> \t-fi\n> \t+cat custom-auth.challenge\n> \t echo\n> \tdiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\n> \tindex a7b1e5bd1af..feb8149de8f 100755\n> \t--- a/t/t5563-simple-http-auth.sh\n> \t+++ b/t/t5563-simple-http-auth.sh\n> \t@@ -47,8 +47,7 @@ test_expect_success 'setup repository' '\n> \t \n> \t test_expect_success 'access anonymous no challenge' '\n> \t \ttest_when_finished \"per_test_cleanup\" &&\n> \t-\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n> \t-\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n> \t+\tgit ls-remote \"$HTTPD_URL/custom_auth_anon/repo.git\"\n> \t '\n> \t \n> \t test_expect_success 'access using basic auth' '\n> \n> I think that's much better, now we just have a 2-line script to handle\n> this \"anon auth\" case. Instead of creating a \"custom-auth.anonymous\"\n> file to communicate how the remote end should behave, let's just\n> communicate that by requesting a different URL, one that accepts\n> anonymous authentication.\n\nActually, we don't really need to test the anonymous auth case at all\nbecause all other tests that try accessing a remote repository over HTTP\nare already exercising this. See t5551-http-fetch-smart for example..\nhere we're performing various requests without auth.\nShould we be erronously issuing credential helper challenges in these\nscenarios then the tests would fail with an askpass prompt.\n\nI will drop the anon auth test and script support.\n\n> I did insert a deliberate bug here, or:\n> \n> \t-\texec \"$GIT_EXEC_PATH\"/git-http-backend\n> \t+\texit 1\n> \n> So aside from your \"exec\" comment it seems both of us missed that this\n> \"exec\" does nothing useful, the test will fail if we emit different\n> headers, but it doesn't matter that we execute the git-http-backend.\n> \n> Or maybe it does, but the tests aren't good enough to spot the\n> difference.\n> \n> The above is a rough WIP, I'm leaving it here for Matthew to follow-up\n> on. I think it might benefit from being further split-up, i.e. we know\n> which URLs we expect to fail auth, so if we just had another URL for\n> \"the auth response fails here\" we'd have 3x trivial scripts with no\n> if/else; but maybe that sucks, I didn't try it.\n"},{"id":"472161","messageId":"05449ec892b1205c1e1c90d15facd812b5cbbe3c.1676496846.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v9.git.1676496846.gitgitgadget@gmail.com","subject":"[PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-15T21:34:04Z","receivedAt":"2023-02-15T21:34:13Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a test showing simple anoymous HTTP access to an unprotected\nrepository, that results in no credential helper invocations.\nAlso add a test demonstrating simple basic authentication with\nsimple credential helper support.\n\nLeverage a no-parsed headers (NPH) CGI script so that we can directly\ncontrol the HTTP responses to simulate a multitude of good, bad and ugly\nremote server implementations around auth.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +++\n t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++\n t/t5563-simple-http-auth.sh | 81 ++++++++++++++++++++++++++++++++++\n 4 files changed, 127 insertions(+)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 608949ea80b..2c49569f675 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -137,6 +137,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n+\tinstall_script nph-custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0294739a77a..76335cdb24d 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n \tSetEnv GIT_HTTP_EXPORT_ALL\n \tSetEnv GIT_PROTOCOL\n \n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tCGIPassAuth on\n+\n ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\nnew file mode 100755\nindex 00000000000..2dd35d6fa39\n--- /dev/null\n+++ b/t/lib-httpd/nph-custom-auth.sh\n@@ -0,0 +1,39 @@\n+#!/bin/sh\n+\n+VALID_CREDS_FILE=custom-auth.valid\n+CHALLENGE_FILE=custom-auth.challenge\n+\n+#\n+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n+# credential for the current request. Each line in the file is considered a\n+# valid HTTP Authorization header value. For example:\n+#\n+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+#\n+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n+# in a 401 response if no valid authentication credentials were included in the\n+# request. For example:\n+#\n+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+# WWW-Authenticate: Basic realm=\"example.com\"\n+#\n+\n+if test -n \"$HTTP_AUTHORIZATION\" && \\\n+\tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\"\n+then\n+\t# Note that although git-http-backend returns a status line, it\n+\t# does so using a CGI 'Status' header. Because this script is an\n+\t# No Parsed Headers (NPH) script, we must return a real HTTP\n+\t# status line.\n+\t# This is only a test script, so we don't bother to check for\n+\t# the actual status from git-http-backend and always return 200.\n+\techo 'HTTP/1.1 200 OK'\n+\texec \"$GIT_EXEC_PATH\"/git-http-backend\n+fi\n+\n+echo 'HTTP/1.1 401 Authorization Required'\n+if test -f \"$CHALLENGE_FILE\"\n+then\n+\tcat \"$CHALLENGE_FILE\"\n+fi\n+echo\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nnew file mode 100755\nindex 00000000000..e0682039de7\n--- /dev/null\n+++ b/t/t5563-simple-http-auth.sh\n@@ -0,0 +1,81 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+test_expect_success 'setup_credential_helper' '\n+\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n+\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n+\texport PATH &&\n+\n+\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n+\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\tcmd=$1\n+\tteefile=$cmd-query.cred\n+\tcatfile=$cmd-reply.cred\n+\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n+\tif test \"$cmd\" = \"get\"\n+\tthen\n+\t\tcat $catfile\n+\tfi\n+\tEOF\n+'\n+\n+set_credential_reply() {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query() {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\n+\n+per_test_cleanup () {\n+\trm -f *.cred &&\n+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n+}\n+\n+test_expect_success 'setup repository' '\n+\ttest_commit foo &&\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n+'\n+\n+test_expect_success 'access using basic auth' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"472162","messageId":"pull.1352.v9.git.1676496846.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v8.git.1675711789.gitgitgadget@gmail.com","subject":"[PATCH v9 0/3] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-15T21:34:03Z","receivedAt":"2023-02-15T21:34:16Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I use a small CGI shell script that acts as a\nfrontend to git-http-backend; simple authentication is configurable by\nfiles.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\n\nUpdates in v7\n=============\n\n * Address several whitespace and arg/param list alignment issues.\n\n * Rethink the test-http-helper worker-mode error and result enum to be more\n simple and more informative to the nature of the error.\n\n * Use uintmax_t to store the Content-Length of a request in the helper\n test-http-server. Maintain a bit flag to store if we received such a\n header.\n\n * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n in the test-http-server.\n\n * Add test case to cover request message parsing in test-http-server.\n\n * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n get_auth_module.\n\n * Correctly free the split strbufs created in the header parsing loop in\n test-http-server.\n\n * Avoid needless comparison > 0 for unsigned types.\n\n * Always set optional outputs to NULL if not present in test helper config\n value handling.\n\n * Remove an accidentally commented-out test cleanup line for one test case\n in t5556.\n\n\nUpdates in v8\n=============\n\n * Drop custom HTTP test helper tool in favour of using a CGI shell script\n and Apache; avoiding the need to implement an HTTP server.\n\n * Avoid allocations in header reading callback unless we have a header we\n care about; act on the char* from libcurl directly rather than create a\n strbuf for each header.\n\n * Drop st_mult overflow guarding function in curl callback functions; we're\n not allocating memory based on the resulting value and just adds to\n potential confusion in the future.\n\n\nUpdates in v9\n=============\n\n * Drop anoynmous auth tests as these cases are already covered by all other\n tests that perform HTTP interactions with a remote today.\n\n * In the custom auth CGI script, avoid the empty-substitution in favour of\n testing explicitly for an empty string. Also simplify some other\n conditional expressions.\n\n * Avoid an allocation on each wwwauth[] credential helper key-value pair\n write.\n\n * Various style fixups.\n\nMatthew John Cheetham (3):\n t5563: add tests for basic and anoymous HTTP access\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n credential.c | 4 +\n credential.h | 15 ++\n git-compat-util.h | 23 +++\n http.c | 120 ++++++++++++\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +\n t/lib-httpd/nph-custom-auth.sh | 39 ++++\n t/t5563-simple-http-auth.sh | 317 +++++++++++++++++++++++++++++++\n 9 files changed, 543 insertions(+), 1 deletion(-)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v9\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v9\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v8:\n\n 1: d362f7016d3 ! 1: 05449ec892b t5563: add tests for basic and anoymous HTTP access\n @@ t/lib-httpd/nph-custom-auth.sh (new)\n +\n +VALID_CREDS_FILE=custom-auth.valid\n +CHALLENGE_FILE=custom-auth.challenge\n -+ANONYMOUS_FILE=custom-auth.anonymous\n +\n +#\n -+# If $ANONYMOUS_FILE exists in $HTTPD_ROOT_PATH, allow anonymous access.\n -+#\n +# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n +# credential for the current request. Each line in the file is considered a\n +# valid HTTP Authorization header value. For example:\n @@ t/lib-httpd/nph-custom-auth.sh (new)\n +# WWW-Authenticate: Basic realm=\"example.com\"\n +#\n +\n -+if test -f \"$ANONYMOUS_FILE\" || (test -f \"$VALID_CREDS_FILE\" && \\\n -+\tgrep -qi \"^${HTTP_AUTHORIZATION:-nopenopnope}$\" \"$VALID_CREDS_FILE\")\n ++if test -n \"$HTTP_AUTHORIZATION\" && \\\n ++\tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\"\n +then\n +\t# Note that although git-http-backend returns a status line, it\n +\t# does so using a CGI 'Status' header. Because this script is an\n @@ t/t5563-simple-http-auth.sh (new)\n +start_httpd\n +\n +test_expect_success 'setup_credential_helper' '\n -+\tmkdir -p \"$TRASH_DIRECTORY/bin\" &&\n ++\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n +\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n +\texport PATH &&\n +\n @@ t/t5563-simple-http-auth.sh (new)\n +\tcmd=$1\n +\tteefile=$cmd-query.cred\n +\tcatfile=$cmd-reply.cred\n -+\tsed -n -e \"/^$/q\" -e \"p\" >> $teefile\n -+\tif test \"$cmd\" = \"get\"; then\n ++\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n ++\tif test \"$cmd\" = \"get\"\n ++\tthen\n +\t\tcat $catfile\n +\tfi\n +\tEOF\n @@ t/t5563-simple-http-auth.sh (new)\n +\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n +'\n +\n -+test_expect_success 'access anonymous no challenge' '\n -+\ttest_when_finished \"per_test_cleanup\" &&\n -+\ttouch \"$HTTPD_ROOT_PATH/custom-auth.anonymous\" &&\n -+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\"\n -+'\n -+\n +test_expect_success 'access using basic auth' '\n +\ttest_when_finished \"per_test_cleanup\" &&\n +\n 2: cd9a02ba94e ! 2: f3f28e508c1 http: read HTTP WWW-Authenticate response headers\n @@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *p\n +\t\t\t\t const char **out, size_t *outlen)\n +{\n +\tsize_t prefix_len = strlen(prefix);\n ++\n +\tif (len < prefix_len)\n +\t\treturn 0;\n +\n -+\tif (!strncasecmp(buf, prefix, prefix_len)){\n ++\tif (!strncasecmp(buf, prefix, prefix_len)) {\n +\t\t*out = buf + prefix_len;\n +\t\t*outlen = len - prefix_len;\n +\t\treturn 1;\n 3: 149aedf5501 ! 3: eba58c0d08d credential: add WWW-Authenticate header to cred requests\n @@ Documentation/git-credential.txt: empty string.\n GIT\n \n ## credential.c ##\n -@@ credential.c: static void credential_write_item(FILE *fp, const char *key, const char *value,\n - \tfprintf(fp, \"%s=%s\\n\", key, value);\n - }\n - \n -+static void credential_write_strvec(FILE *fp, const char *key,\n -+\t\t\t\t const struct strvec *vec)\n -+{\n -+\tchar *full_key = xstrfmt(\"%s[]\", key);\n -+\tfor (size_t i = 0; i < vec->nr; i++) {\n -+\t\tcredential_write_item(fp, full_key, vec->v[i], 0);\n -+\t}\n -+\tfree(full_key);\n -+}\n -+\n - void credential_write(const struct credential *c, FILE *fp)\n - {\n - \tcredential_write_item(fp, \"protocol\", c->protocol, 1);\n @@ credential.c: void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n -+\tcredential_write_strvec(fp, \"wwwauth\", &c->wwwauth_headers);\n ++\tfor (size_t i = 0; i < c->wwwauth_headers.nr; i++)\n ++\t\tcredential_write_item(fp, \"wwwauth[]\", c->wwwauth_headers.v[i],\n ++\t\t\t\t 0);\n }\n \n static int run_credential_helper(struct credential *c,\n\n-- \ngitgitgadget\n"},{"id":"472163","messageId":"f3f28e508c1792cbc8f8d3bd56099c659743ed3e.1676496846.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v9.git.1676496846.gitgitgadget@gmail.com","subject":"[PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-15T21:34:05Z","receivedAt":"2023-02-15T21:34:18Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nAccording to RFC2616 Section 4.2 [1], header field names are not\ncase-sensitive meaning when collecting multiple values for the same\nfield name, we can just use the case of the first observed instance of\neach field name and no normalisation is required.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us one line at\na time.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 15 ++++++\n git-compat-util.h | 23 +++++++++\n http.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 159 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..6f2e5bc610b 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,19 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Used to keep track of split header fields\n+\t * in order to fold multiple lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +144,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a76d0526f79..a59230564e8 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n \treturn 0;\n }\n \n+/*\n+ * Like skip_prefix_mem, but compare case-insensitively. Note that the\n+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n+ * characters or locale-specific conversions).\n+ */\n+static inline int skip_iprefix_mem(const char *buf, size_t len,\n+\t\t\t\t const char *prefix,\n+\t\t\t\t const char **out, size_t *outlen)\n+{\n+\tsize_t prefix_len = strlen(prefix);\n+\n+\tif (len < prefix_len)\n+\t\treturn 0;\n+\n+\tif (!strncasecmp(buf, prefix, prefix_len)) {\n+\t\t*out = buf + prefix_len;\n+\t\t*outlen = len - prefix_len;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n {\n \tunsigned long ul;\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..7a56a3db5f7 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+/*\n+ * A folded header continuation line starts with at least one single whitespace\n+ * character. It is not a continuation line if the line is *just* a newline.\n+ * The RFC for HTTP states that CRLF is the header field line ending, but some\n+ * servers may use LF only; we accept both.\n+ */\n+static inline int is_hdr_continuation(const char *ptr, const size_t size)\n+{\n+\t/* totally empty line or normal header */\n+\tif (!size || !isspace(*ptr))\n+\t\treturn 0;\n+\n+\t/* empty line with LF line ending */\n+\tif (size == 1 && ptr[0] == '\\n')\n+\t\treturn 0;\n+\n+\t/* empty line with CRLF line ending */\n+\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tsize_t val_len;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 2616 (even though this has since been\n+\t * deprecated in RFC 7230). A continuation header field value is\n+\t * identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 2616 is:\n+\t *\n+\t * message-header = field-name \":\" [ field-value ]\n+\t * field-name = token\n+\t * field-value = *( field-content | LWS )\n+\t * field-content = \n+\t */\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n+\t\tstrbuf_add(&buf, val, val_len);\n+\n+\t\t/*\n+\t\t * Strip the CRLF that should be present at the end of each\n+\t\t * field as well as any trailing or leading whitespace from the\n+\t\t * value.\n+\t\t */\n+\t\tstrbuf_trim(&buf);\n+\n+\t\tstrvec_push(values, buf.buf);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n+\t\t/*\n+\t\t * Trim the CRLF and any leading or trailing from this line.\n+\t\t */\n+\t\tstrbuf_add(&buf, ptr, size);\n+\t\tstrbuf_trim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* This is the start of a new header we don't care about */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (!strncasecmp(ptr, \"http/\", 5))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1982,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"472164","messageId":"eba58c0d08d06daa3d2f0ce5914c8a78d66648f7.1676496846.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v9.git.1676496846.gitgitgadget@gmail.com","subject":"[PATCH v9 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-15T21:34:06Z","receivedAt":"2023-02-15T21:34:20Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 3 +\n t/t5563-simple-http-auth.sh | 236 +++++++++++++++++++++++++++++++\n 3 files changed, 257 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..f566c8ab195 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tfor (size_t i = 0; i < c->wwwauth_headers.nr; i++)\n+\t\tcredential_write_item(fp, \"wwwauth[]\", c->wwwauth_headers.v[i],\n+\t\t\t\t 0);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nindex e0682039de7..7b390a7bf95 100755\n--- a/t/t5563-simple-http-auth.sh\n+++ b/t/t5563-simple-http-auth.sh\n@@ -68,6 +68,242 @@ test_expect_success 'access using basic auth' '\n \texpect_credential_query get <<-EOF &&\n \tprotocol=http\n \thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth invalid credentials' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\ttest_must_fail git ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with extra challenges' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\" param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth mixed-case wwwauth header name' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\twww-authenticate: foobar param1=\"value1\" param2=\"value2\"\n+\tWWW-AUTHENTICATE: BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\tWwW-aUtHeNtIcAtE: baSiC realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=foobar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=baSiC realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\"\n+\t param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\n+\t p=1\n+\t q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header empty continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" param2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Bearer authorize_uri=\\\"id.example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" p=1\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" q=0\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \"\\tparam2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Basic realm=\"example.com\"\n \tEOF\n \n \texpect_credential_query store <<-EOF\n-- \ngitgitgadget\n"},{"id":"472167","messageId":"xmqqy1oywp78.fsf@gitster.g","threadId":"58425","inReplyTo":"05449ec892b1205c1e1c90d15facd812b5cbbe3c.1676496846.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T22:15:07Z","receivedAt":"2023-02-15T22:15:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> +if test -n \"$HTTP_AUTHORIZATION\" && \\\n> +\tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\"\n\nDo we require a regexp match (and worry about metacharacters in\nHTTP_AUTHORIZATION variable), or would we want to use \"grep -F -x\"\nhere to force match with the entire line?\n\n> +then\n> +\t# Note that although git-http-backend returns a status line, it\n> +\t# does so using a CGI 'Status' header. Because this script is an\n> +\t# No Parsed Headers (NPH) script, we must return a real HTTP\n> +\t# status line.\n> +\t# This is only a test script, so we don't bother to check for\n> +\t# the actual status from git-http-backend and always return 200.\n> +\techo 'HTTP/1.1 200 OK'\n> +\texec \"$GIT_EXEC_PATH\"/git-http-backend\n> +fi\n\nOK. That's the successful auth case. Otherwise ...\n\n> +echo 'HTTP/1.1 401 Authorization Required'\n> +if test -f \"$CHALLENGE_FILE\"\n> +then\n> +\tcat \"$CHALLENGE_FILE\"\n> +fi\n> +echo\n\nOK. We'll just give a challenge.\n\n> diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\n> new file mode 100755\n> index 00000000000..e0682039de7\n> --- /dev/null\n> +++ b/t/t5563-simple-http-auth.sh\n> @@ -0,0 +1,81 @@\n> +#!/bin/sh\n> +\n> +test_description='test http auth header and credential helper interop'\n> +\n> +. ./test-lib.sh\n> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n> +\n> +start_httpd\n> +\n> +test_expect_success 'setup_credential_helper' '\n> +\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n> +\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n> +\texport PATH &&\n> +\n> +\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n> +\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n> +\tcmd=$1\n> +\tteefile=$cmd-query.cred\n> +\tcatfile=$cmd-reply.cred\n> +\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n> +\tif test \"$cmd\" = \"get\"\n> +\tthen\n> +\t\tcat $catfile\n> +\tfi\n> +\tEOF\n> +'\n> +\n> +set_credential_reply() {\n\nStyle. Have SP before \"()\" as well as after.\n\n> +\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n> +}\n> +\n> +expect_credential_query() {\n\nStyle. Have SP before \"()\" as well as after.\n\n> +\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n> +\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n> +\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n> +}\n> +\n> +per_test_cleanup () {\n> +\trm -f *.cred &&\n> +\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n> +}\n> +\n> +test_expect_success 'setup repository' '\n> +\ttest_commit foo &&\n> +\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n> +\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n> +'\n\nOK.\n\n> +test_expect_success 'access using basic auth' '\n> +\ttest_when_finished \"per_test_cleanup\" &&\n> +\n> +\tset_credential_reply get <<-EOF &&\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tEOF\n> +\n> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n> +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n> +\tEOF\n\nPerhaps we want to note that this matches the \"alice:secret-passwd\"\nwe prepared earlier?\n\n> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n> +\tWWW-Authenticate: Basic realm=\"example.com\"\n> +\tEOF\n\nOK.\n\n> +\ttest_config_global credential.helper test-helper &&\n> +\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n> +\n> +\texpect_credential_query get <<-EOF &&\n> +\tprotocol=http\n> +\thost=$HTTPD_DEST\n> +\tEOF\n> +\n> +\texpect_credential_query store <<-EOF\n> +\tprotocol=http\n> +\thost=$HTTPD_DEST\n> +\tusername=alice\n> +\tpassword=secret-passwd\n> +\tEOF\n> +'\n\nOK.\n\n> +test_done\n"},{"id":"472172","messageId":"xmqqy1oyv7ck.fsf@gitster.g","threadId":"58425","inReplyTo":"f3f28e508c1792cbc8f8d3bd56099c659743ed3e.1676496846.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-15T23:26:03Z","receivedAt":"2023-02-15T23:26:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Matthew John Cheetham via GitGitGadget\" \nwrites:\n\n> According to RFC2616 Section 4.2 [1], header field names are not\n> case-sensitive meaning when collecting multiple values for the same\n> field name, we can just use the case of the first observed instance of\n> each field name and no normalisation is required.\n\nIf the names are not case-sensitive, you can choose to first\ndowncase the names you see, and use that consistently, and the\nresult would still be valid. IOW, \"not case-sensitive\" does not at\nall mean you have to use the first observed instance without\nnormalization. You are allowed to choose such an implementation,\nbut \"not case-sensitive\" is not a justification to choose such an\nimplementation among possible implementation that would be allowed\nunder the rule.\n\n> The collection of all header values matching the WWW-Authenticate\n> header is complicated by the fact that it is legal for header fields to\n> be continued over multiple lines, but libcurl only gives us one line at\n> a time.\n\nSaying \"one physical line\" at a time may make it clear what you are\npointing out as a weak point in the interface libcURL gives us (I\nthink you are getting at \"if they handled header folding for us and\nfed us one logical line at a time, it would have been nicer\").\n\n> @@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n> \tfree(c->username);\n> \tfree(c->password);\n> \tstring_list_clear(&c->helpers, 0);\n> +\tstrvec_clear(&c->wwwauth_headers);\n> \n> \tcredential_init(c);\n> }\n> diff --git a/credential.h b/credential.h\n> index f430e77fea4..6f2e5bc610b 100644\n> --- a/credential.h\n> +++ b/credential.h\n> @@ -2,6 +2,7 @@\n> #define CREDENTIAL_H\n> \n> #include \"string-list.h\"\n> +#include \"strvec.h\"\n> \n> /**\n> * The credentials API provides an abstracted way of gathering username and\n> @@ -115,6 +116,19 @@ struct credential {\n> \t */\n> \tstruct string_list helpers;\n> \n> +\t/**\n> +\t * A `strvec` of WWW-Authenticate header values. Each string\n> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n> +\t * in the order they were received in the response.\n> +\t */\n> +\tstruct strvec wwwauth_headers;\n> +\n> +\t/**\n> +\t * Internal use only. Used to keep track of split header fields\n\nThe technical term for what you call \"split header\" here seems to be\n\"line folding\" (RFC 7230, which deprecates it).\n\n> +\t * in order to fold multiple lines into one value.\n> +\t */\n> +\tunsigned header_is_last_match:1;\n> +\n> \tunsigned approved:1,\n> \t\t configured:1,\n> \t\t quit:1,\n> @@ -130,6 +144,7 @@ struct credential {\n> \n> #define CREDENTIAL_INIT { \\\n> \t.helpers = STRING_LIST_INIT_DUP, \\\n> +\t.wwwauth_headers = STRVEC_INIT, \\\n> }\n> \n> /* Initialize a credential structure, setting all fields to empty. */\n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index a76d0526f79..a59230564e8 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n> \treturn 0;\n> }\n> \n> +/*\n> + * Like skip_prefix_mem, but compare case-insensitively. Note that the\n> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n> + * characters or locale-specific conversions).\n> + */\n> +static inline int skip_iprefix_mem(const char *buf, size_t len,\n> +\t\t\t\t const char *prefix,\n> +\t\t\t\t const char **out, size_t *outlen)\n> +{\n> +\tsize_t prefix_len = strlen(prefix);\n> +\n> +\tif (len < prefix_len)\n> +\t\treturn 0;\n> +\n> +\tif (!strncasecmp(buf, prefix, prefix_len)) {\n> +\t\t*out = buf + prefix_len;\n> +\t\t*outlen = len - prefix_len;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nOK.\n\n> diff --git a/http.c b/http.c\n> index 8a5ba3f4776..7a56a3db5f7 100644\n> --- a/http.c\n> +++ b/http.c\n> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n> \treturn nmemb;\n> }\n> \n> +/*\n> + * A folded header continuation line starts with at least one single whitespace\n> + * character. It is not a continuation line if the line is *just* a newline.\n> + * The RFC for HTTP states that CRLF is the header field line ending, but some\n> + * servers may use LF only; we accept both.\n> + */\n\nNice.\n\n> +static inline int is_hdr_continuation(const char *ptr, const size_t size)\n> +{\n> +\t/* totally empty line or normal header */\n> +\tif (!size || !isspace(*ptr))\n> +\t\treturn 0;\n\nobs-fold (RFC7230) begins the next line with SP or HTAB, but\nisspace() allows not just SP and HT but also CR and LF. So\nthis is a bit pessimistic but rejects what is not a folded\ncontinuation line reliably.\n\n> +\t/* empty line with LF line ending */\n> +\tif (size == 1 && ptr[0] == '\\n')\n> +\t\treturn 0;\n\nAnd this is a blank line after the headers, with LF (not conforming\nbut is OK).\n\n> +\t/* empty line with CRLF line ending */\n> +\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n> +\t\treturn 0;\n\nAnd this is another form of a blank line after the headers, with\nCRLF.\n\n> +\treturn 1;\n> +}\n\nAfter rejecting the above two \"blank\", it is a folded continuation\nline. OK.\n\nI wonder if\n\n\tstatic inline int ... () {\n\t \treturn (size && (*ptr == ' ' || *ptr == '\\t'));\n\t}\n\nsufficient and easier to grok, though.\n\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> +{\n> +\tsize_t size = eltsize * nmemb;\n> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tconst char *val;\n> +\tsize_t val_len;\n> +\n> +\t/*\n> +\t * Header lines may not come NULL-terminated from libcurl so we must\n> +\t * limit all scans to the maximum length of the header line, or leverage\n> +\t * strbufs for all operations.\n> +\t *\n> +\t * In addition, it is possible that header values can be split over\n> +\t * multiple lines as per RFC 2616 (even though this has since been\n> +\t * deprecated in RFC 7230). A continuation header field value is\n> +\t * identified as starting with a space or horizontal tab.\n> +\t *\n> +\t * The formal definition of a header field as given in RFC 2616 is:\n> +\t *\n> +\t * message-header = field-name \":\" [ field-value ]\n> +\t * field-name = token\n> +\t * field-value = *( field-content | LWS )\n> +\t * field-content = +\t * and consisting of either *TEXT or combinations\n> +\t * of token, separators, and quoted-string>\n> +\t */\n> +\n> +\t/* Start of a new WWW-Authenticate header */\n> +\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n> +\t\tstrbuf_add(&buf, val, val_len);\n> +\n> +\t\t/*\n> +\t\t * Strip the CRLF that should be present at the end of each\n> +\t\t * field as well as any trailing or leading whitespace from the\n> +\t\t * value.\n> +\t\t */\n> +\t\tstrbuf_trim(&buf);\n> +\n> +\t\tstrvec_push(values, buf.buf);\n> +\t\thttp_auth.header_is_last_match = 1;\n> +\t\tgoto exit;\n\nOK. We remember that we have seen the beginning of a header we are\ninterested in (so that we can append if it is a continuation we see\nnext). Good.\n\n> +\t}\n> +\n> +\t/*\n> +\t * This line could be a continuation of the previously matched header\n> +\t * field. If this is the case then we should append this value to the\n> +\t * end of the previously consumed value.\n> +\t */\n> +\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n> +\t\t/*\n> +\t\t * Trim the CRLF and any leading or trailing from this line.\n> +\t\t */\n> +\t\tstrbuf_add(&buf, ptr, size);\n> +\t\tstrbuf_trim(&buf);\n> +\n> +\t\t/*\n> +\t\t * At this point we should always have at least one existing\n> +\t\t * value, even if it is empty. Do not bother appending the new\n> +\t\t * value if this continuation header is itself empty.\n> +\t\t */\n> +\t\tif (!values->nr) {\n> +\t\t\tBUG(\"should have at least one existing header value\");\n\nOK, we should set _is_last_match to true only after we recorded the\nheader that might see a continuation, so it would be a bug if we\ndidn't have anything there. Good.\n\n> +\t\t} else if (buf.len) {\n> +\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n> +\n> +\t\t\t/* Join two non-empty values with a single space. */\n> +\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n> +\n> +\t\t\tstrvec_pop(values);\n> +\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n> +\t\t\tfree(prev);\n> +\t\t}\n> +\n> +\t\tgoto exit;\n\nGood that we are prepared to see a logical line split over more than\ntwo lines (i.e. by not toggling _is_last_match off prematurely here).\n\n> +\t}\n> +\n> +\t/* This is the start of a new header we don't care about */\n> +\thttp_auth.header_is_last_match = 0;\n\nOr what we just saw and ignored could be a continuation line of a\nheader we ignored. The comment is slightly misleading.\n\nOther than that, looking good.\n\nThanks.\n"},{"id":"472230","messageId":"AS2PR03MB9815447A4B519DE1ECC29595C0A09@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"xmqqy1oywp78.fsf@gitster.g","subject":"Re: [PATCH v9 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-16T22:25:16Z","receivedAt":"2023-02-16T22:25:41Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-15 14:15, Junio C Hamano wrote:\n\n> \"Matthew John Cheetham via GitGitGadget\" \n> writes:\n> \n>> +if test -n \"$HTTP_AUTHORIZATION\" && \\\n>> +\tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\"\n> \n> Do we require a regexp match (and worry about metacharacters in\n> HTTP_AUTHORIZATION variable), or would we want to use \"grep -F -x\"\n> here to force match with the entire line?\n\nYou're right. We don't need a regex match here. Will fix.\n\n>> +then\n>> +\t# Note that although git-http-backend returns a status line, it\n>> +\t# does so using a CGI 'Status' header. Because this script is an\n>> +\t# No Parsed Headers (NPH) script, we must return a real HTTP\n>> +\t# status line.\n>> +\t# This is only a test script, so we don't bother to check for\n>> +\t# the actual status from git-http-backend and always return 200.\n>> +\techo 'HTTP/1.1 200 OK'\n>> +\texec \"$GIT_EXEC_PATH\"/git-http-backend\n>> +fi\n> \n> OK. That's the successful auth case. Otherwise ...\n> \n>> +echo 'HTTP/1.1 401 Authorization Required'\n>> +if test -f \"$CHALLENGE_FILE\"\n>> +then\n>> +\tcat \"$CHALLENGE_FILE\"\n>> +fi\n>> +echo\n> \n> OK. We'll just give a challenge.\n> \n>> diff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\n>> new file mode 100755\n>> index 00000000000..e0682039de7\n>> --- /dev/null\n>> +++ b/t/t5563-simple-http-auth.sh\n>> @@ -0,0 +1,81 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='test http auth header and credential helper interop'\n>> +\n>> +. ./test-lib.sh\n>> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n>> +\n>> +start_httpd\n>> +\n>> +test_expect_success 'setup_credential_helper' '\n>> +\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n>> +\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n>> +\texport PATH &&\n>> +\n>> +\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n>> +\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n>> +\tcmd=$1\n>> +\tteefile=$cmd-query.cred\n>> +\tcatfile=$cmd-reply.cred\n>> +\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n>> +\tif test \"$cmd\" = \"get\"\n>> +\tthen\n>> +\t\tcat $catfile\n>> +\tfi\n>> +\tEOF\n>> +'\n>> +\n>> +set_credential_reply() {\n> \n> Style. Have SP before \"()\" as well as after.\n\nWill fix.\n\n>> +\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n>> +}\n>> +\n>> +expect_credential_query() {\n> \n> Style. Have SP before \"()\" as well as after.\n\nDitto.\n\n>> +\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n>> +\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n>> +\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n>> +}\n>> +\n>> +per_test_cleanup () {\n>> +\trm -f *.cred &&\n>> +\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n>> +}\n>> +\n>> +test_expect_success 'setup repository' '\n>> +\ttest_commit foo &&\n>> +\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n>> +\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n>> +'\n> \n> OK.\n> \n>> +test_expect_success 'access using basic auth' '\n>> +\ttest_when_finished \"per_test_cleanup\" &&\n>> +\n>> +\tset_credential_reply get <<-EOF &&\n>> +\tusername=alice\n>> +\tpassword=secret-passwd\n>> +\tEOF\n>> +\n>> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n>> +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n>> +\tEOF\n> \n> Perhaps we want to note that this matches the \"alice:secret-passwd\"\n> we prepared earlier?\n\nGood point. It's helpful. Will re-introduce.\n\n>> +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n>> +\tWWW-Authenticate: Basic realm=\"example.com\"\n>> +\tEOF\n> \n> OK.\n> \n>> +\ttest_config_global credential.helper test-helper &&\n>> +\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n>> +\n>> +\texpect_credential_query get <<-EOF &&\n>> +\tprotocol=http\n>> +\thost=$HTTPD_DEST\n>> +\tEOF\n>> +\n>> +\texpect_credential_query store <<-EOF\n>> +\tprotocol=http\n>> +\thost=$HTTPD_DEST\n>> +\tusername=alice\n>> +\tpassword=secret-passwd\n>> +\tEOF\n>> +'\n> \n> OK.\n> \n>> +test_done\n"},{"id":"472231","messageId":"AS2PR03MB981587537C03857C389BF6F4C0A09@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"xmqqy1oyv7ck.fsf@gitster.g","subject":"Re: [PATCH v9 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-16T22:29:25Z","receivedAt":"2023-02-16T22:29:39Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-15 15:26, Junio C Hamano wrote:\n\n> \"Matthew John Cheetham via GitGitGadget\" \n> writes:\n> \n>> According to RFC2616 Section 4.2 [1], header field names are not\n>> case-sensitive meaning when collecting multiple values for the same\n>> field name, we can just use the case of the first observed instance of\n>> each field name and no normalisation is required.\n> \n> If the names are not case-sensitive, you can choose to first\n> downcase the names you see, and use that consistently, and the\n> result would still be valid. IOW, \"not case-sensitive\" does not at\n> all mean you have to use the first observed instance without\n> normalization. You are allowed to choose such an implementation,\n> but \"not case-sensitive\" is not a justification to choose such an\n> implementation among possible implementation that would be allowed\n> under the rule.\n\nRe-reading this paragraph, it doens't really need to even be here. This was\nan artefact of a time when I was storing all headers, including keys and\nvalues. Since we're only interested now in the WWW-Authenticate header\n_values_, there's no need to call out this out. Will drop this paragraph.\n\n>> The collection of all header values matching the WWW-Authenticate\n>> header is complicated by the fact that it is legal for header fields to\n>> be continued over multiple lines, but libcurl only gives us one line at\n>> a time.\n> \n> Saying \"one physical line\" at a time may make it clear what you are\n> pointing out as a weak point in the interface libcURL gives us (I\n> think you are getting at \"if they handled header folding for us and\n> fed us one logical line at a time, it would have been nicer\").\n\nLogical header fields vs physical header lines is useful and clearer\nterminology - I will update the commit message to reflect in the next\niteration. Thanks!\n\n>> @@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n>> \tfree(c->username);\n>> \tfree(c->password);\n>> \tstring_list_clear(&c->helpers, 0);\n>> +\tstrvec_clear(&c->wwwauth_headers);\n>> \n>> \tcredential_init(c);\n>> }\n>> diff --git a/credential.h b/credential.h\n>> index f430e77fea4..6f2e5bc610b 100644\n>> --- a/credential.h\n>> +++ b/credential.h\n>> @@ -2,6 +2,7 @@\n>> #define CREDENTIAL_H\n>> \n>> #include \"string-list.h\"\n>> +#include \"strvec.h\"\n>> \n>> /**\n>> * The credentials API provides an abstracted way of gathering username and\n>> @@ -115,6 +116,19 @@ struct credential {\n>> \t */\n>> \tstruct string_list helpers;\n>> \n>> +\t/**\n>> +\t * A `strvec` of WWW-Authenticate header values. Each string\n>> +\t * is the value of a WWW-Authenticate header in an HTTP response,\n>> +\t * in the order they were received in the response.\n>> +\t */\n>> +\tstruct strvec wwwauth_headers;\n>> +\n>> +\t/**\n>> +\t * Internal use only. Used to keep track of split header fields\n> \n> The technical term for what you call \"split header\" here seems to be\n> \"line folding\" (RFC 7230, which deprecates it).\n> \n>> +\t * in order to fold multiple lines into one value.\n>> +\t */\n>> +\tunsigned header_is_last_match:1;\n>> +\n>> \tunsigned approved:1,\n>> \t\t configured:1,\n>> \t\t quit:1,\n>> @@ -130,6 +144,7 @@ struct credential {\n>> \n>> #define CREDENTIAL_INIT { \\\n>> \t.helpers = STRING_LIST_INIT_DUP, \\\n>> +\t.wwwauth_headers = STRVEC_INIT, \\\n>> }\n>> \n>> /* Initialize a credential structure, setting all fields to empty. */\n>> diff --git a/git-compat-util.h b/git-compat-util.h\n>> index a76d0526f79..a59230564e8 100644\n>> --- a/git-compat-util.h\n>> +++ b/git-compat-util.h\n>> @@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n>> \treturn 0;\n>> }\n>> \n>> +/*\n>> + * Like skip_prefix_mem, but compare case-insensitively. Note that the\n>> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n>> + * characters or locale-specific conversions).\n>> + */\n>> +static inline int skip_iprefix_mem(const char *buf, size_t len,\n>> +\t\t\t\t const char *prefix,\n>> +\t\t\t\t const char **out, size_t *outlen)\n>> +{\n>> +\tsize_t prefix_len = strlen(prefix);\n>> +\n>> +\tif (len < prefix_len)\n>> +\t\treturn 0;\n>> +\n>> +\tif (!strncasecmp(buf, prefix, prefix_len)) {\n>> +\t\t*out = buf + prefix_len;\n>> +\t\t*outlen = len - prefix_len;\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\treturn 0;\n>> +}\n> \n> OK.\n> \n>> diff --git a/http.c b/http.c\n>> index 8a5ba3f4776..7a56a3db5f7 100644\n>> --- a/http.c\n>> +++ b/http.c\n>> @@ -183,6 +183,124 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n>> \treturn nmemb;\n>> }\n>> \n>> +/*\n>> + * A folded header continuation line starts with at least one single whitespace\n>> + * character. It is not a continuation line if the line is *just* a newline.\n>> + * The RFC for HTTP states that CRLF is the header field line ending, but some\n>> + * servers may use LF only; we accept both.\n>> + */\n> \n> Nice.\n> \n>> +static inline int is_hdr_continuation(const char *ptr, const size_t size)\n>> +{\n>> +\t/* totally empty line or normal header */\n>> +\tif (!size || !isspace(*ptr))\n>> +\t\treturn 0;\n> \n> obs-fold (RFC7230) begins the next line with SP or HTAB, but\n> isspace() allows not just SP and HT but also CR and LF. So\n> this is a bit pessimistic but rejects what is not a folded\n> continuation line reliably.\n> \n>> +\t/* empty line with LF line ending */\n>> +\tif (size == 1 && ptr[0] == '\\n')\n>> +\t\treturn 0;\n> \n> And this is a blank line after the headers, with LF (not conforming\n> but is OK).\n> \n>> +\t/* empty line with CRLF line ending */\n>> +\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n>> +\t\treturn 0;\n> \n> And this is another form of a blank line after the headers, with\n> CRLF.\n> \n>> +\treturn 1;\n>> +}\n> \n> After rejecting the above two \"blank\", it is a folded continuation\n> line. OK.\n> \n> I wonder if\n> \n> \tstatic inline int ... () {\n> \t \treturn (size && (*ptr == ' ' || *ptr == '\\t'));\n> \t}\n> \n> sufficient and easier to grok, though.\n\nYou're correct. This implementation is 'more correct' and easier to grok.\n\n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> +{\n>> +\tsize_t size = eltsize * nmemb;\n>> +\tstruct strvec *values = &http_auth.wwwauth_headers;\n>> +\tstruct strbuf buf = STRBUF_INIT;\n>> +\tconst char *val;\n>> +\tsize_t val_len;\n>> +\n>> +\t/*\n>> +\t * Header lines may not come NULL-terminated from libcurl so we must\n>> +\t * limit all scans to the maximum length of the header line, or leverage\n>> +\t * strbufs for all operations.\n>> +\t *\n>> +\t * In addition, it is possible that header values can be split over\n>> +\t * multiple lines as per RFC 2616 (even though this has since been\n>> +\t * deprecated in RFC 7230). A continuation header field value is\n>> +\t * identified as starting with a space or horizontal tab.\n>> +\t *\n>> +\t * The formal definition of a header field as given in RFC 2616 is:\n>> +\t *\n>> +\t * message-header = field-name \":\" [ field-value ]\n>> +\t * field-name = token\n>> +\t * field-value = *( field-content | LWS )\n>> +\t * field-content = > +\t * and consisting of either *TEXT or combinations\n>> +\t * of token, separators, and quoted-string>\n>> +\t */\n>> +\n>> +\t/* Start of a new WWW-Authenticate header */\n>> +\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n>> +\t\tstrbuf_add(&buf, val, val_len);\n>> +\n>> +\t\t/*\n>> +\t\t * Strip the CRLF that should be present at the end of each\n>> +\t\t * field as well as any trailing or leading whitespace from the\n>> +\t\t * value.\n>> +\t\t */\n>> +\t\tstrbuf_trim(&buf);\n>> +\n>> +\t\tstrvec_push(values, buf.buf);\n>> +\t\thttp_auth.header_is_last_match = 1;\n>> +\t\tgoto exit;\n> \n> OK. We remember that we have seen the beginning of a header we are\n> interested in (so that we can append if it is a continuation we see\n> next). Good.\n> \n>> +\t}\n>> +\n>> +\t/*\n>> +\t * This line could be a continuation of the previously matched header\n>> +\t * field. If this is the case then we should append this value to the\n>> +\t * end of the previously consumed value.\n>> +\t */\n>> +\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n>> +\t\t/*\n>> +\t\t * Trim the CRLF and any leading or trailing from this line.\n>> +\t\t */\n>> +\t\tstrbuf_add(&buf, ptr, size);\n>> +\t\tstrbuf_trim(&buf);\n>> +\n>> +\t\t/*\n>> +\t\t * At this point we should always have at least one existing\n>> +\t\t * value, even if it is empty. Do not bother appending the new\n>> +\t\t * value if this continuation header is itself empty.\n>> +\t\t */\n>> +\t\tif (!values->nr) {\n>> +\t\t\tBUG(\"should have at least one existing header value\");\n> \n> OK, we should set _is_last_match to true only after we recorded the\n> header that might see a continuation, so it would be a bug if we\n> didn't have anything there. Good.\n> \n>> +\t\t} else if (buf.len) {\n>> +\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n>> +\n>> +\t\t\t/* Join two non-empty values with a single space. */\n>> +\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n>> +\n>> +\t\t\tstrvec_pop(values);\n>> +\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n>> +\t\t\tfree(prev);\n>> +\t\t}\n>> +\n>> +\t\tgoto exit;\n> \n> Good that we are prepared to see a logical line split over more than\n> two lines (i.e. by not toggling _is_last_match off prematurely here).\n> \n>> +\t}\n>> +\n>> +\t/* This is the start of a new header we don't care about */\n>> +\thttp_auth.header_is_last_match = 0;\n> \n> Or what we just saw and ignored could be a continuation line of a\n> header we ignored. The comment is slightly misleading.\n\nI'll try and reword this to make it more accurate - we have determined\nthis line is not a continuation of the previous WWW-Authenticate header.\n\n> Other than that, looking good.\n> \n> Thanks.\n"},{"id":"472232","messageId":"f3ccc53055acf5d5c25d0ad3eed3867ea8670e55.1676586881.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v10.git.1676586881.gitgitgadget@gmail.com","subject":"[PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-16T22:34:39Z","receivedAt":"2023-02-16T22:34:49Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a test showing simple anoymous HTTP access to an unprotected\nrepository, that results in no credential helper invocations.\nAlso add a test demonstrating simple basic authentication with\nsimple credential helper support.\n\nLeverage a no-parsed headers (NPH) CGI script so that we can directly\ncontrol the HTTP responses to simulate a multitude of good, bad and ugly\nremote server implementations around auth.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +++\n t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++\n t/t5563-simple-http-auth.sh | 82 ++++++++++++++++++++++++++++++++++\n 4 files changed, 128 insertions(+)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 608949ea80b..2c49569f675 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -137,6 +137,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n+\tinstall_script nph-custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0294739a77a..76335cdb24d 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n \tSetEnv GIT_HTTP_EXPORT_ALL\n \tSetEnv GIT_PROTOCOL\n \n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tCGIPassAuth on\n+\n ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\nnew file mode 100755\nindex 00000000000..f5345e775e4\n--- /dev/null\n+++ b/t/lib-httpd/nph-custom-auth.sh\n@@ -0,0 +1,39 @@\n+#!/bin/sh\n+\n+VALID_CREDS_FILE=custom-auth.valid\n+CHALLENGE_FILE=custom-auth.challenge\n+\n+#\n+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n+# credential for the current request. Each line in the file is considered a\n+# valid HTTP Authorization header value. For example:\n+#\n+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+#\n+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n+# in a 401 response if no valid authentication credentials were included in the\n+# request. For example:\n+#\n+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+# WWW-Authenticate: Basic realm=\"example.com\"\n+#\n+\n+if test -n \"$HTTP_AUTHORIZATION\" && \\\n+\tgrep -Fqsx \"${HTTP_AUTHORIZATION}\" \"$VALID_CREDS_FILE\"\n+then\n+\t# Note that although git-http-backend returns a status line, it\n+\t# does so using a CGI 'Status' header. Because this script is an\n+\t# No Parsed Headers (NPH) script, we must return a real HTTP\n+\t# status line.\n+\t# This is only a test script, so we don't bother to check for\n+\t# the actual status from git-http-backend and always return 200.\n+\techo 'HTTP/1.1 200 OK'\n+\texec \"$GIT_EXEC_PATH\"/git-http-backend\n+fi\n+\n+echo 'HTTP/1.1 401 Authorization Required'\n+if test -f \"$CHALLENGE_FILE\"\n+then\n+\tcat \"$CHALLENGE_FILE\"\n+fi\n+echo\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nnew file mode 100755\nindex 00000000000..40f1b381d1b\n--- /dev/null\n+++ b/t/t5563-simple-http-auth.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+test_expect_success 'setup_credential_helper' '\n+\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n+\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n+\texport PATH &&\n+\n+\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n+\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\tcmd=$1\n+\tteefile=$cmd-query.cred\n+\tcatfile=$cmd-reply.cred\n+\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n+\tif test \"$cmd\" = \"get\"\n+\tthen\n+\t\tcat $catfile\n+\tfi\n+\tEOF\n+'\n+\n+set_credential_reply () {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query () {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\n+\n+per_test_cleanup () {\n+\trm -f *.cred &&\n+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n+}\n+\n+test_expect_success 'setup repository' '\n+\ttest_commit foo &&\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n+'\n+\n+test_expect_success 'access using basic auth' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"472233","messageId":"703ac15222fdcfc98751b11af725cc1395134bd1.1676586881.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v10.git.1676586881.gitgitgadget@gmail.com","subject":"[PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-16T22:34:40Z","receivedAt":"2023-02-16T22:34:50Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us each\nphysical line a time, not each logical header. This line folding feature\nis deprecated in RFC 7230 [1] but older servers may still emit them, so\nwe need to handle them.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 16 +++++++\n git-compat-util.h | 23 ++++++++++\n http.c | 111 ++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 151 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..3756a54c74d 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,20 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Keeps track of if we previously matched against a\n+\t * WWW-Authenticate header line in order to re-fold future continuation\n+\t * lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +145,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a76d0526f79..a59230564e8 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1266,6 +1266,29 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n \treturn 0;\n }\n \n+/*\n+ * Like skip_prefix_mem, but compare case-insensitively. Note that the\n+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n+ * characters or locale-specific conversions).\n+ */\n+static inline int skip_iprefix_mem(const char *buf, size_t len,\n+\t\t\t\t const char *prefix,\n+\t\t\t\t const char **out, size_t *outlen)\n+{\n+\tsize_t prefix_len = strlen(prefix);\n+\n+\tif (len < prefix_len)\n+\t\treturn 0;\n+\n+\tif (!strncasecmp(buf, prefix, prefix_len)) {\n+\t\t*out = buf + prefix_len;\n+\t\t*outlen = len - prefix_len;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n {\n \tunsigned long ul;\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..3ff570ee3a9 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,115 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+/*\n+ * A folded header continuation line starts with any number of spaces or\n+ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.\n+ * It is not a continuation line if the line starts with any other character.\n+ */\n+static inline int is_hdr_continuation(const char *ptr, const size_t size)\n+{\n+\treturn size && (*ptr == ' ' || *ptr == '\\t');\n+}\n+\n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tsize_t val_len;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 7230. 'Line folding' has been deprecated\n+\t * but older servers may still emit them. A continuation header field\n+\t * value is identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 7230 is:\n+\t *\n+\t * header-field = field-name \":\" OWS field-value OWS\n+\t *\n+\t * field-name = token\n+\t * field-value = *( field-content / obs-fold )\n+\t * field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ]\n+\t * field-vchar = VCHAR / obs-text\n+\t *\n+\t * obs-fold = CRLF 1*( SP / HTAB )\n+\t * ; obsolete line folding\n+\t * ; see Section 3.2.4\n+\t */\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n+\t\tstrbuf_add(&buf, val, val_len);\n+\n+\t\t/*\n+\t\t * Strip the CRLF that should be present at the end of each\n+\t\t * field as well as any trailing or leading whitespace from the\n+\t\t * value.\n+\t\t */\n+\t\tstrbuf_trim(&buf);\n+\n+\t\tstrvec_push(values, buf.buf);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n+\t\t/*\n+\t\t * Trim the CRLF and any leading or trailing from this line.\n+\t\t */\n+\t\tstrbuf_add(&buf, ptr, size);\n+\t\tstrbuf_trim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* Not a continuation of a previously matched auth header line. */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (!strncasecmp(ptr, \"http/\", 5))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1973,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"472234","messageId":"186da54fd3b2fec061769360b8da4635f6c161bf.1676586881.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v10.git.1676586881.gitgitgadget@gmail.com","subject":"[PATCH v10 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-16T22:34:41Z","receivedAt":"2023-02-16T22:34:56Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 3 +\n t/t5563-simple-http-auth.sh | 242 +++++++++++++++++++++++++++++++\n 3 files changed, 263 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..f566c8ab195 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tfor (size_t i = 0; i < c->wwwauth_headers.nr; i++)\n+\t\tcredential_write_item(fp, \"wwwauth[]\", c->wwwauth_headers.v[i],\n+\t\t\t\t 0);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nindex 40f1b381d1b..64d2acd0328 100755\n--- a/t/t5563-simple-http-auth.sh\n+++ b/t/t5563-simple-http-auth.sh\n@@ -69,6 +69,248 @@ test_expect_success 'access using basic auth' '\n \texpect_credential_query get <<-EOF &&\n \tprotocol=http\n \thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth invalid credentials' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\ttest_must_fail git ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with extra challenges' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\" param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth mixed-case wwwauth header name' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\twww-authenticate: foobar param1=\"value1\" param2=\"value2\"\n+\tWWW-AUTHENTICATE: BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\tWwW-aUtHeNtIcAtE: baSiC realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=foobar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=baSiC realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\"\n+\t param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\n+\t p=1\n+\t q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header empty continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" param2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Bearer authorize_uri=\\\"id.example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" p=1\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" q=0\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \"\\tparam2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Basic realm=\"example.com\"\n \tEOF\n \n \texpect_credential_query store <<-EOF\n-- \ngitgitgadget\n"},{"id":"472235","messageId":"pull.1352.v10.git.1676586881.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v9.git.1676496846.gitgitgadget@gmail.com","subject":"[PATCH v10 0/3] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-16T22:34:38Z","receivedAt":"2023-02-16T22:34:56Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I use a small CGI shell script that acts as a\nfrontend to git-http-backend; simple authentication is configurable by\nfiles.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\n\nUpdates in v7\n=============\n\n * Address several whitespace and arg/param list alignment issues.\n\n * Rethink the test-http-helper worker-mode error and result enum to be more\n simple and more informative to the nature of the error.\n\n * Use uintmax_t to store the Content-Length of a request in the helper\n test-http-server. Maintain a bit flag to store if we received such a\n header.\n\n * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n in the test-http-server.\n\n * Add test case to cover request message parsing in test-http-server.\n\n * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n get_auth_module.\n\n * Correctly free the split strbufs created in the header parsing loop in\n test-http-server.\n\n * Avoid needless comparison > 0 for unsigned types.\n\n * Always set optional outputs to NULL if not present in test helper config\n value handling.\n\n * Remove an accidentally commented-out test cleanup line for one test case\n in t5556.\n\n\nUpdates in v8\n=============\n\n * Drop custom HTTP test helper tool in favour of using a CGI shell script\n and Apache; avoiding the need to implement an HTTP server.\n\n * Avoid allocations in header reading callback unless we have a header we\n care about; act on the char* from libcurl directly rather than create a\n strbuf for each header.\n\n * Drop st_mult overflow guarding function in curl callback functions; we're\n not allocating memory based on the resulting value and just adds to\n potential confusion in the future.\n\n\nUpdates in v9\n=============\n\n * Drop anoynmous auth tests as these cases are already covered by all other\n tests that perform HTTP interactions with a remote today.\n\n * In the custom auth CGI script, avoid the empty-substitution in favour of\n testing explicitly for an empty string. Also simplify some other\n conditional expressions.\n\n * Avoid an allocation on each wwwauth[] credential helper key-value pair\n write.\n\n * Various style fixups.\n\n\nUpdates in v10\n==============\n\n * Style fixups.\n\n * Only consider space (SP ' ') and horizontal tab (HTAB '\\t') when\n detecting a header continuation line, as per the latest RFC on the\n matter.\n\n * Update references to old HTTP specs and formal grammars of header fields\n in comments.\n\n * Rewording of commit messages to remove confusing comment about the case\n sensitivity of header field names - this is not relevant with the current\n iteration of the header parsing code. Also update the message around\n libcurl header support to clarify that physical header lines are\n returned, but not 'logical' header lines.\n\n * Reword struct credential member doc comment to clarify the purpose of\n header_is_last_match is for re-folding lines of the WWW-Authenticate\n header.\n\n * Reintroduce helpful comments in tests to show the origin of the 'magic'\n base64 basic auth value.\n\n * Use grep -F to ensure we don't do regex matching; avoid interpreting\n special characters. Remove erronous insensitive comparison flag.\n\nMatthew John Cheetham (3):\n t5563: add tests for basic and anoymous HTTP access\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n credential.c | 4 +\n credential.h | 16 ++\n git-compat-util.h | 23 +++\n http.c | 111 +++++++++++\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +\n t/lib-httpd/nph-custom-auth.sh | 39 ++++\n t/t5563-simple-http-auth.sh | 324 +++++++++++++++++++++++++++++++\n 9 files changed, 542 insertions(+), 1 deletion(-)\n create mode 100755 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v10\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v10\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v9:\n\n 1: 05449ec892b ! 1: f3ccc53055a t5563: add tests for basic and anoymous HTTP access\n @@ t/lib-httpd/nph-custom-auth.sh (new)\n +#\n +\n +if test -n \"$HTTP_AUTHORIZATION\" && \\\n -+\tgrep -qsi \"^${HTTP_AUTHORIZATION}\\$\" \"$VALID_CREDS_FILE\"\n ++\tgrep -Fqsx \"${HTTP_AUTHORIZATION}\" \"$VALID_CREDS_FILE\"\n +then\n +\t# Note that although git-http-backend returns a status line, it\n +\t# does so using a CGI 'Status' header. Because this script is an\n @@ t/t5563-simple-http-auth.sh (new)\n +\tEOF\n +'\n +\n -+set_credential_reply() {\n ++set_credential_reply () {\n +\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n +}\n +\n -+expect_credential_query() {\n ++expect_credential_query () {\n +\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n +\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n +\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n @@ t/t5563-simple-http-auth.sh (new)\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n 2: f3f28e508c1 ! 2: 703ac15222f http: read HTTP WWW-Authenticate response headers\n @@ Commit message\n information to credential helpers or others that would otherwise have\n been lost.\n \n - According to RFC2616 Section 4.2 [1], header field names are not\n - case-sensitive meaning when collecting multiple values for the same\n - field name, we can just use the case of the first observed instance of\n - each field name and no normalisation is required.\n -\n libcurl only provides us with the ability to read all headers recieved\n for a particular request, including any intermediate redirect requests\n or proxies. The lines returned by libcurl include HTTP status lines\n @@ Commit message\n \n The collection of all header values matching the WWW-Authenticate\n header is complicated by the fact that it is legal for header fields to\n - be continued over multiple lines, but libcurl only gives us one line at\n - a time.\n + be continued over multiple lines, but libcurl only gives us each\n + physical line a time, not each logical header. This line folding feature\n + is deprecated in RFC 7230 [1] but older servers may still emit them, so\n + we need to handle them.\n \n In the future [2] we may be able to leverage functions to read headers\n from libcurl itself, but as of today we must do this ourselves.\n \n - [1] https://datatracker.ietf.org/doc/html/rfc2616#section-4.2\n + [1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2\n [2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n \n Signed-off-by: Matthew John Cheetham \n @@ credential.h: struct credential {\n +\tstruct strvec wwwauth_headers;\n +\n +\t/**\n -+\t * Internal use only. Used to keep track of split header fields\n -+\t * in order to fold multiple lines into one value.\n ++\t * Internal use only. Keeps track of if we previously matched against a\n ++\t * WWW-Authenticate header line in order to re-fold future continuation\n ++\t * lines into one value.\n +\t */\n +\tunsigned header_is_last_match:1;\n +\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n }\n \n +/*\n -+ * A folded header continuation line starts with at least one single whitespace\n -+ * character. It is not a continuation line if the line is *just* a newline.\n -+ * The RFC for HTTP states that CRLF is the header field line ending, but some\n -+ * servers may use LF only; we accept both.\n ++ * A folded header continuation line starts with any number of spaces or\n ++ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.\n ++ * It is not a continuation line if the line starts with any other character.\n + */\n +static inline int is_hdr_continuation(const char *ptr, const size_t size)\n +{\n -+\t/* totally empty line or normal header */\n -+\tif (!size || !isspace(*ptr))\n -+\t\treturn 0;\n -+\n -+\t/* empty line with LF line ending */\n -+\tif (size == 1 && ptr[0] == '\\n')\n -+\t\treturn 0;\n -+\n -+\t/* empty line with CRLF line ending */\n -+\tif (size == 2 && ptr[0] == '\\r' && ptr[1] == '\\n')\n -+\t\treturn 0;\n -+\n -+\treturn 1;\n ++\treturn size && (*ptr == ' ' || *ptr == '\\t');\n +}\n +\n +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * strbufs for all operations.\n +\t *\n +\t * In addition, it is possible that header values can be split over\n -+\t * multiple lines as per RFC 2616 (even though this has since been\n -+\t * deprecated in RFC 7230). A continuation header field value is\n -+\t * identified as starting with a space or horizontal tab.\n ++\t * multiple lines as per RFC 7230. 'Line folding' has been deprecated\n ++\t * but older servers may still emit them. A continuation header field\n ++\t * value is identified as starting with a space or horizontal tab.\n ++\t *\n ++\t * The formal definition of a header field as given in RFC 7230 is:\n ++\t *\n ++\t * header-field = field-name \":\" OWS field-value OWS\n +\t *\n -+\t * The formal definition of a header field as given in RFC 2616 is:\n ++\t * field-name = token\n ++\t * field-value = *( field-content / obs-fold )\n ++\t * field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ]\n ++\t * field-vchar = VCHAR / obs-text\n +\t *\n -+\t * message-header = field-name \":\" [ field-value ]\n -+\t * field-name = token\n -+\t * field-value = *( field-content | LWS )\n -+\t * field-content = \n ++\t * obs-fold = CRLF 1*( SP / HTAB )\n ++\t * ; obsolete line folding\n ++\t * ; see Section 3.2.4\n +\t */\n +\n +\t/* Start of a new WWW-Authenticate header */\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t\tgoto exit;\n +\t}\n +\n -+\t/* This is the start of a new header we don't care about */\n ++\t/* Not a continuation of a previously matched auth header line. */\n +\thttp_auth.header_is_last_match = 0;\n +\n +\t/*\n 3: eba58c0d08d ! 3: 186da54fd3b credential: add WWW-Authenticate header to cred requests\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=wrong-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n @@ t/t5563-simple-http-auth.sh: test_expect_success 'access using basic auth' '\n +\tpassword=secret-passwd\n +\tEOF\n +\n ++\t# Basic base64(alice:secret-passwd)\n +\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n +\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n +\tEOF\n\n-- \ngitgitgadget\n"},{"id":"472520","messageId":"Y/cu7K5uFjvOMXLu@coredump.intra.peff.net","threadId":"58425","inReplyTo":"f3ccc53055acf5d5c25d0ad3eed3867ea8670e55.1676586881.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-23T09:16:28Z","receivedAt":"2023-02-23T09:16:34Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 16, 2023 at 10:34:39PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> Leverage a no-parsed headers (NPH) CGI script so that we can directly\n> control the HTTP responses to simulate a multitude of good, bad and ugly\n> remote server implementations around auth.\n\nHmm, today I learned about NPH scripts.\n\nObviously it works here, but I have to wonder: is there a reason we need\nthis? AFAICT the only thing we do is set the HTTP response code, which\ncould also be done with a Status: header.\n\nI.e., this passes your test:\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex ccd5f3cf82..1eadfa4bbc 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -140,7 +140,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n-\tinstall_script nph-custom-auth.sh\n+\tinstall_script custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 2aac922376..0f9083dd6c 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -139,7 +139,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n-ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/custom-auth.sh\nsimilarity index 94%\nrename from t/lib-httpd/nph-custom-auth.sh\nrename to t/lib-httpd/custom-auth.sh\nindex f5345e775e..8bf07e9398 100755\n--- a/t/lib-httpd/nph-custom-auth.sh\n+++ b/t/lib-httpd/custom-auth.sh\n@@ -27,11 +27,10 @@ then\n \t# status line.\n \t# This is only a test script, so we don't bother to check for\n \t# the actual status from git-http-backend and always return 200.\n-\techo 'HTTP/1.1 200 OK'\n \texec \"$GIT_EXEC_PATH\"/git-http-backend\n fi\n \n-echo 'HTTP/1.1 401 Authorization Required'\n+echo 'Status: 401'\n if test -f \"$CHALLENGE_FILE\"\n then\n \tcat \"$CHALLENGE_FILE\"\n\n\nThe other, more invisible thing happening behind the scenes is that\nApache isn't adding any of its usual headers. But I don't know of any\nthat would interfere with our goal of doing auth here. Is there some\nfeature you're planning where it would?\n\nI think you could argue that it's mostly a matter of personal preference\nand doesn't matter much either way. But all things being equal, I'd\nusually go with the thing that is simpler and closer to the rest of the\nsystem (e.g., I think you kill the ability of http-backend to return a\nnon-200 status, though I doubt it matters much in practice).\n\nSo I dunno. We are on v10 and this is arguably a nit. Mostly I'm just\ncurious what led you in this direction in the first place.\n\n> ---\n> t/lib-httpd.sh | 1 +\n> t/lib-httpd/apache.conf | 6 +++\n> t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++\n> t/t5563-simple-http-auth.sh | 82 ++++++++++++++++++++++++++++++++++\n> 4 files changed, 128 insertions(+)\n> create mode 100755 t/lib-httpd/nph-custom-auth.sh\n\nMost of the other scripts here don't have an execute bit. They get one\nwhen they're copied by instal_script in lib-httpd.sh. The exception is\nerror.sh, but I don't think there's any good reason for it. So probably\nnot a big deal either way, but another nit. :)\n\nThe rest of it all looks quite nice to me.\n\n-Peff\n"},{"id":"472522","messageId":"Y/cz5g1PJoYeh0Fw@coredump.intra.peff.net","threadId":"58425","inReplyTo":"Y/cu7K5uFjvOMXLu@coredump.intra.peff.net","subject":"Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-23T09:37:42Z","receivedAt":"2023-02-23T09:38:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 23, 2023 at 04:16:28AM -0500, Jeff King wrote:\n\n> Hmm, today I learned about NPH scripts.\n> \n> Obviously it works here, but I have to wonder: is there a reason we need\n> this? AFAICT the only thing we do is set the HTTP response code, which\n> could also be done with a Status: header.\n> \n> I.e., this passes your test:\n\nHaving looked at patch 3 now, this also needs:\n\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nindex 64d2acd032..afdf388677 100755\n--- a/t/t5563-simple-http-auth.sh\n+++ b/t/t5563-simple-http-auth.sh\n@@ -37,7 +37,7 @@ expect_credential_query () {\n \n per_test_cleanup () {\n \trm -f *.cred &&\n-\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.valid \"$HTTPD_ROOT_PATH\"/custom-auth.challenge\n }\n \n test_expect_success 'setup repository' '\n\nor comedy ensues. But more importantly, realized why you want to use NPH\nhere. Apache will happily munge:\n\n WWW-Authenticate: foo\n WWW-Authenticate: bar\n\ninto:\n\n WWW-Authenticate: foo, bar\n\nand you want to stress the parser with specific syntactic forms. So that\nmakes sense, and I agree NPH is the right solution here.\n\nI think you did try to say this in the commit message as:\n\n Leverage a no-parsed headers (NPH) CGI script so that we can directly\n control the HTTP responses to simulate a multitude of good, bad and\n ugly remote server implementations around auth.\n\nbut I was too dense to realize quite what that meant. :)\n\n-Peff\n"},{"id":"472523","messageId":"Y/c1+Llb/D27eWdg@coredump.intra.peff.net","threadId":"58425","inReplyTo":"703ac15222fdcfc98751b11af725cc1395134bd1.1676586881.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-23T09:46:32Z","receivedAt":"2023-02-23T09:46:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Feb 16, 2023 at 10:34:40PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> +/*\n> + * Like skip_prefix_mem, but compare case-insensitively. Note that the\n> + * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n> + * characters or locale-specific conversions).\n> + */\n> +static inline int skip_iprefix_mem(const char *buf, size_t len,\n> +\t\t\t\t const char *prefix,\n> +\t\t\t\t const char **out, size_t *outlen)\n> +{\n> +\tsize_t prefix_len = strlen(prefix);\n> +\n> +\tif (len < prefix_len)\n> +\t\treturn 0;\n> +\n> +\tif (!strncasecmp(buf, prefix, prefix_len)) {\n> +\t\t*out = buf + prefix_len;\n> +\t\t*outlen = len - prefix_len;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nThe comment at the top of the function seems out of date. It's using\nstrncasecmp(), so it probably would be locale-dependent. I think that's\nprobably OK, but we should probably fix the comment.\n\nAlternatively, you could copy the tolower() loop from skip_iprefix().\nSomething like:\n \ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex 28456241b6..f671a0ec3f 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,\n \t\t\t\t const char *prefix,\n \t\t\t\t const char **out, size_t *outlen)\n {\n-\tsize_t prefix_len = strlen(prefix);\n-\n-\tif (len < prefix_len)\n-\t\treturn 0;\n-\n-\tif (!strncasecmp(buf, prefix, prefix_len)) {\n-\t\t*out = buf + prefix_len;\n-\t\t*outlen = len - prefix_len;\n-\t\treturn 1;\n-\t}\n-\n+\tdo {\n+\t\tif (!*prefix) {\n+\t\t\t*out = buf;\n+\t\t\t*outlen = len;\n+\t\t\treturn 1;\n+\t\t}\n+\t} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));\n \treturn 0;\n }\n \n\nlooks right to me, though only lightly tested (via t5563). I'm happy\nwith either implementation.\n\n> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n> [...]\n> +\t/*\n> +\t * If this is a HTTP status line and not a header field, this signals\n> +\t * a different HTTP response. libcurl writes all the output of all\n> +\t * response headers of all responses, including redirects.\n> +\t * We only care about the last HTTP request response's headers so clear\n> +\t * the existing array.\n> +\t */\n> +\tif (!strncasecmp(ptr, \"http/\", 5))\n> +\t\tstrvec_clear(values);\n\nSince \"ptr\" isn't NUL terminated, using strncasecmp() may walk off the\nend. I think you'd need to check that there are five bytes. You could\neven use skip_iprefix_mem(), though of course we'd throw away the output\nvalues. (For strings there is also istarts_with(), but I don't think we\nhave a \"mem\" equivalent).\n\nThe rest of the parsing looks good to me.\n\n-Peff\n"},{"id":"472568","messageId":"xmqqo7pk6u2h.fsf@gitster.g","threadId":"58425","inReplyTo":"Y/c1+Llb/D27eWdg@coredump.intra.peff.net","subject":"Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-23T19:49:26Z","receivedAt":"2023-02-23T19:49:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Alternatively, you could copy the tolower() loop from skip_iprefix().\n> Something like:\n> \n> diff --git a/git-compat-util.h b/git-compat-util.h\n> index 28456241b6..f671a0ec3f 100644\n> --- a/git-compat-util.h\n> +++ b/git-compat-util.h\n> @@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,\n> \t\t\t\t const char *prefix,\n> \t\t\t\t const char **out, size_t *outlen)\n> {\n> -\tsize_t prefix_len = strlen(prefix);\n> -\n> -\tif (len < prefix_len)\n> -\t\treturn 0;\n> -\n> -\tif (!strncasecmp(buf, prefix, prefix_len)) {\n> -\t\t*out = buf + prefix_len;\n> -\t\t*outlen = len - prefix_len;\n> -\t\treturn 1;\n> -\t}\n> -\n> +\tdo {\n> +\t\tif (!*prefix) {\n> +\t\t\t*out = buf;\n> +\t\t\t*outlen = len;\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));\n> \treturn 0;\n> }\n> \n>\n> looks right to me, though only lightly tested (via t5563). I'm happy\n> with either implementation.\n\nYeah, the alternative version looks clearer to me.\n\n>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>> [...]\n>> +\t/*\n>> +\t * If this is a HTTP status line and not a header field, this signals\n>> +\t * a different HTTP response. libcurl writes all the output of all\n>> +\t * response headers of all responses, including redirects.\n>> +\t * We only care about the last HTTP request response's headers so clear\n>> +\t * the existing array.\n>> +\t */\n>> +\tif (!strncasecmp(ptr, \"http/\", 5))\n>> +\t\tstrvec_clear(values);\n>\n> Since \"ptr\" isn't NUL terminated, using strncasecmp() may walk off the\n> end. I think you'd need to check that there are five bytes. You could\n> even use skip_iprefix_mem(), though of course we'd throw away the output\n> values. (For strings there is also istarts_with(), but I don't think we\n> have a \"mem\" equivalent).\n\nYuck, thank you very much for carefully reading. I missed this one\nwhen I queued it.\n\n> The rest of the parsing looks good to me.\n\nThanks.\n"},{"id":"472801","messageId":"AS2PR03MB9815D1DE582E0DC94C18535DC0AF9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"xmqqo7pk6u2h.fsf@gitster.g","subject":"Re: [PATCH v10 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-27T17:14:49Z","receivedAt":"2023-02-27T17:15:07Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-23 11:49, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n>> Alternatively, you could copy the tolower() loop from skip_iprefix().\n>> Something like:\n>> \n>> diff --git a/git-compat-util.h b/git-compat-util.h\n>> index 28456241b6..f671a0ec3f 100644\n>> --- a/git-compat-util.h\n>> +++ b/git-compat-util.h\n>> @@ -1296,17 +1296,13 @@ static inline int skip_iprefix_mem(const char *buf, size_t len,\n>> \t\t\t\t const char *prefix,\n>> \t\t\t\t const char **out, size_t *outlen)\n>> {\n>> -\tsize_t prefix_len = strlen(prefix);\n>> -\n>> -\tif (len < prefix_len)\n>> -\t\treturn 0;\n>> -\n>> -\tif (!strncasecmp(buf, prefix, prefix_len)) {\n>> -\t\t*out = buf + prefix_len;\n>> -\t\t*outlen = len - prefix_len;\n>> -\t\treturn 1;\n>> -\t}\n>> -\n>> +\tdo {\n>> +\t\tif (!*prefix) {\n>> +\t\t\t*out = buf;\n>> +\t\t\t*outlen = len;\n>> +\t\t\treturn 1;\n>> +\t\t}\n>> +\t} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));\n>> \treturn 0;\n>> }\n>> \n>>\n>> looks right to me, though only lightly tested (via t5563). I'm happy\n>> with either implementation.\n> \n> Yeah, the alternative version looks clearer to me.\n\nWill update - thanks!\n\n>>> +static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n>>> [...]\n>>> +\t/*\n>>> +\t * If this is a HTTP status line and not a header field, this signals\n>>> +\t * a different HTTP response. libcurl writes all the output of all\n>>> +\t * response headers of all responses, including redirects.\n>>> +\t * We only care about the last HTTP request response's headers so clear\n>>> +\t * the existing array.\n>>> +\t */\n>>> +\tif (!strncasecmp(ptr, \"http/\", 5))\n>>> +\t\tstrvec_clear(values);\n>>\n>> Since \"ptr\" isn't NUL terminated, using strncasecmp() may walk off the\n>> end. I think you'd need to check that there are five bytes. You could\n>> even use skip_iprefix_mem(), though of course we'd throw away the output\n>> values. (For strings there is also istarts_with(), but I don't think we\n>> have a \"mem\" equivalent).\n> \n> Yuck, thank you very much for carefully reading. I missed this one\n> when I queued it.\n\nOops! Will update in a v11\n\n>> The rest of the parsing looks good to me.\n> \n> Thanks.\n"},{"id":"472803","messageId":"AS2PR03MB98155ADC85C730AF0651A3C7C0AF9@AS2PR03MB9815.eurprd03.prod.outlook.com","threadId":"58425","inReplyTo":"Y/cu7K5uFjvOMXLu@coredump.intra.peff.net","subject":"Re: [PATCH v10 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham","fromEmail":"mjcheetham@outlook.com","sentAt":"2023-02-27T17:18:11Z","receivedAt":"2023-02-27T17:18:31Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"On 2023-02-23 01:16, Jeff King wrote:\n\n> On Thu, Feb 16, 2023 at 10:34:39PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n> \n>> Leverage a no-parsed headers (NPH) CGI script so that we can directly\n>> control the HTTP responses to simulate a multitude of good, bad and ugly\n>> remote server implementations around auth.\n> \n> Hmm, today I learned about NPH scripts.\n> \n> Obviously it works here, but I have to wonder: is there a reason we need\n> this? AFAICT the only thing we do is set the HTTP response code, which\n> could also be done with a Status: header.\n\nYep - I think you realised why in a later email. It's because Apache is\ndoing some CGI -> HTTP header normalisation, but we want to control the\nexact byte output of WWW-Authenticate headers for exercising the new code :-)\n\n> I.e., this passes your test:\n> \n> diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\n> index ccd5f3cf82..1eadfa4bbc 100644\n> --- a/t/lib-httpd.sh\n> +++ b/t/lib-httpd.sh\n> @@ -140,7 +140,7 @@ prepare_httpd() {\n> \tinstall_script error-smart-http.sh\n> \tinstall_script error.sh\n> \tinstall_script apply-one-time-perl.sh\n> -\tinstall_script nph-custom-auth.sh\n> +\tinstall_script custom-auth.sh\n> \n> \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n> \n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 2aac922376..0f9083dd6c 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -139,7 +139,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n> ScriptAlias /error_smart/ error-smart-http.sh/\n> ScriptAlias /error/ error.sh/\n> ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n> -ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n> +ScriptAliasMatch /custom_auth/(.*) custom-auth.sh/$1\n> \n> \tOptions FollowSymlinks\n> \n> diff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/custom-auth.sh\n> similarity index 94%\n> rename from t/lib-httpd/nph-custom-auth.sh\n> rename to t/lib-httpd/custom-auth.sh\n> index f5345e775e..8bf07e9398 100755\n> --- a/t/lib-httpd/nph-custom-auth.sh\n> +++ b/t/lib-httpd/custom-auth.sh\n> @@ -27,11 +27,10 @@ then\n> \t# status line.\n> \t# This is only a test script, so we don't bother to check for\n> \t# the actual status from git-http-backend and always return 200.\n> -\techo 'HTTP/1.1 200 OK'\n> \texec \"$GIT_EXEC_PATH\"/git-http-backend\n> fi\n> \n> -echo 'HTTP/1.1 401 Authorization Required'\n> +echo 'Status: 401'\n> if test -f \"$CHALLENGE_FILE\"\n> then\n> \tcat \"$CHALLENGE_FILE\"\n> \n> \n> The other, more invisible thing happening behind the scenes is that\n> Apache isn't adding any of its usual headers. But I don't know of any\n> that would interfere with our goal of doing auth here. Is there some\n> feature you're planning where it would?\n> \n> I think you could argue that it's mostly a matter of personal preference\n> and doesn't matter much either way. But all things being equal, I'd\n> usually go with the thing that is simpler and closer to the rest of the\n> system (e.g., I think you kill the ability of http-backend to return a\n> non-200 status, though I doubt it matters much in practice).\n> \n> So I dunno. We are on v10 and this is arguably a nit. Mostly I'm just\n> curious what led you in this direction in the first place.\n> \n>> ---\n>> t/lib-httpd.sh | 1 +\n>> t/lib-httpd/apache.conf | 6 +++\n>> t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++\n>> t/t5563-simple-http-auth.sh | 82 ++++++++++++++++++++++++++++++++++\n>> 4 files changed, 128 insertions(+)\n>> create mode 100755 t/lib-httpd/nph-custom-auth.sh\n> \n> Most of the other scripts here don't have an execute bit. They get one\n> when they're copied by instal_script in lib-httpd.sh. The exception is\n> error.sh, but I don't think there's any good reason for it. So probably\n> not a big deal either way, but another nit. :)\n\nOh.. that's something I missed. I just added the executable bit by habit.\nWill remove to match the others in lib-httpd/.\n\n> The rest of it all looks quite nice to me.\n> \n> -Peff\n"},{"id":"472807","messageId":"f7b234c4038345ec3bd1240aa9b85c90fdff181d.1677518420.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v11.git.1677518420.gitgitgadget@gmail.com","subject":"[PATCH v11 1/3] t5563: add tests for basic and anoymous HTTP access","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-27T17:20:18Z","receivedAt":"2023-02-27T17:20:31Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd a test showing simple anoymous HTTP access to an unprotected\nrepository, that results in no credential helper invocations.\nAlso add a test demonstrating simple basic authentication with\nsimple credential helper support.\n\nLeverage a no-parsed headers (NPH) CGI script so that we can directly\ncontrol the HTTP responses to simulate a multitude of good, bad and ugly\nremote server implementations around auth.\n\nSigned-off-by: Matthew John Cheetham \n---\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +++\n t/lib-httpd/nph-custom-auth.sh | 39 ++++++++++++++++\n t/t5563-simple-http-auth.sh | 83 ++++++++++++++++++++++++++++++++++\n 4 files changed, 129 insertions(+)\n create mode 100644 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\ndiff --git a/t/lib-httpd.sh b/t/lib-httpd.sh\nindex 608949ea80b..2c49569f675 100644\n--- a/t/lib-httpd.sh\n+++ b/t/lib-httpd.sh\n@@ -137,6 +137,7 @@ prepare_httpd() {\n \tinstall_script error-smart-http.sh\n \tinstall_script error.sh\n \tinstall_script apply-one-time-perl.sh\n+\tinstall_script nph-custom-auth.sh\n \n \tln -s \"$LIB_HTTPD_MODULE_PATH\" \"$HTTPD_ROOT_PATH/modules\"\n \ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0294739a77a..76335cdb24d 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -135,6 +135,11 @@ Alias /auth/dumb/ www/auth/dumb/\n \tSetEnv GIT_HTTP_EXPORT_ALL\n \tSetEnv GIT_PROTOCOL\n \n+\n+\tSetEnv GIT_EXEC_PATH ${GIT_EXEC_PATH}\n+\tSetEnv GIT_HTTP_EXPORT_ALL\n+\tCGIPassAuth on\n+\n ScriptAlias /smart/incomplete_length/git-upload-pack incomplete-length-upload-pack-v2-http.sh/\n ScriptAlias /smart/incomplete_body/git-upload-pack incomplete-body-upload-pack-v2-http.sh/\n ScriptAlias /smart/no_report/git-receive-pack error-no-report.sh/\n@@ -144,6 +149,7 @@ ScriptAlias /broken_smart/ broken-smart-http.sh/\n ScriptAlias /error_smart/ error-smart-http.sh/\n ScriptAlias /error/ error.sh/\n ScriptAliasMatch /one_time_perl/(.*) apply-one-time-perl.sh/$1\n+ScriptAliasMatch /custom_auth/(.*) nph-custom-auth.sh/$1\n \n \tOptions FollowSymlinks\n \ndiff --git a/t/lib-httpd/nph-custom-auth.sh b/t/lib-httpd/nph-custom-auth.sh\nnew file mode 100644\nindex 00000000000..f5345e775e4\n--- /dev/null\n+++ b/t/lib-httpd/nph-custom-auth.sh\n@@ -0,0 +1,39 @@\n+#!/bin/sh\n+\n+VALID_CREDS_FILE=custom-auth.valid\n+CHALLENGE_FILE=custom-auth.challenge\n+\n+#\n+# If $VALID_CREDS_FILE exists in $HTTPD_ROOT_PATH, consider each line as a valid\n+# credential for the current request. Each line in the file is considered a\n+# valid HTTP Authorization header value. For example:\n+#\n+# Basic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+#\n+# If $CHALLENGE_FILE exists in $HTTPD_ROOT_PATH, output the contents as headers\n+# in a 401 response if no valid authentication credentials were included in the\n+# request. For example:\n+#\n+# WWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+# WWW-Authenticate: Basic realm=\"example.com\"\n+#\n+\n+if test -n \"$HTTP_AUTHORIZATION\" && \\\n+\tgrep -Fqsx \"${HTTP_AUTHORIZATION}\" \"$VALID_CREDS_FILE\"\n+then\n+\t# Note that although git-http-backend returns a status line, it\n+\t# does so using a CGI 'Status' header. Because this script is an\n+\t# No Parsed Headers (NPH) script, we must return a real HTTP\n+\t# status line.\n+\t# This is only a test script, so we don't bother to check for\n+\t# the actual status from git-http-backend and always return 200.\n+\techo 'HTTP/1.1 200 OK'\n+\texec \"$GIT_EXEC_PATH\"/git-http-backend\n+fi\n+\n+echo 'HTTP/1.1 401 Authorization Required'\n+if test -f \"$CHALLENGE_FILE\"\n+then\n+\tcat \"$CHALLENGE_FILE\"\n+fi\n+echo\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nnew file mode 100755\nindex 00000000000..bc880bf80f9\n--- /dev/null\n+++ b/t/t5563-simple-http-auth.sh\n@@ -0,0 +1,83 @@\n+#!/bin/sh\n+\n+test_description='test http auth header and credential helper interop'\n+\n+. ./test-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+\n+start_httpd\n+\n+test_expect_success 'setup_credential_helper' '\n+\tmkdir \"$TRASH_DIRECTORY/bin\" &&\n+\tPATH=$PATH:\"$TRASH_DIRECTORY/bin\" &&\n+\texport PATH &&\n+\n+\tCREDENTIAL_HELPER=\"$TRASH_DIRECTORY/bin/git-credential-test-helper\" &&\n+\twrite_script \"$CREDENTIAL_HELPER\" <<-\\EOF\n+\tcmd=$1\n+\tteefile=$cmd-query.cred\n+\tcatfile=$cmd-reply.cred\n+\tsed -n -e \"/^$/q\" -e \"p\" >>$teefile\n+\tif test \"$cmd\" = \"get\"\n+\tthen\n+\t\tcat $catfile\n+\tfi\n+\tEOF\n+'\n+\n+set_credential_reply () {\n+\tcat >\"$TRASH_DIRECTORY/$1-reply.cred\"\n+}\n+\n+expect_credential_query () {\n+\tcat >\"$TRASH_DIRECTORY/$1-expect.cred\" &&\n+\ttest_cmp \"$TRASH_DIRECTORY/$1-expect.cred\" \\\n+\t\t \"$TRASH_DIRECTORY/$1-query.cred\"\n+}\n+\n+per_test_cleanup () {\n+\trm -f *.cred &&\n+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.valid \\\n+\t \"$HTTPD_ROOT_PATH\"/custom-auth.challenge\n+}\n+\n+test_expect_success 'setup repository' '\n+\ttest_commit foo &&\n+\tgit init --bare \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit push --mirror \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\"\n+'\n+\n+test_expect_success 'access using basic auth' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"472808","messageId":"pull.1352.v11.git.1677518420.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v10.git.1676586881.gitgitgadget@gmail.com","subject":"[PATCH v11 0/3] Enhance credential helper protocol to include auth headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-27T17:20:17Z","receivedAt":"2023-02-27T17:20:31Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"Following from my original RFC submission [0], this submission is considered\nready for full review. This patch series is now based on top of current\nmaster (9c32cfb49c60fa8173b9666db02efe3b45a8522f) that includes my now\nseparately submitted patches [1] to fix up the other credential helpers'\nbehaviour.\n\nIn this patch series I update the existing credential helper design in order\nto allow for some new scenarios, and future evolution of auth methods that\nGit hosts may wish to provide. I outline the background, summary of changes\nand some challenges below.\n\nTesting these new additions, I use a small CGI shell script that acts as a\nfrontend to git-http-backend; simple authentication is configurable by\nfiles.\n\n\nBackground\n==========\n\nGit uses a variety of protocols [2]: local, Smart HTTP, Dumb HTTP, SSH, and\nGit. Here I focus on the Smart HTTP protocol, and attempt to enhance the\nauthentication capabilities of this protocol to address limitations (see\nbelow).\n\nThe Smart HTTP protocol in Git supports a few different types of HTTP\nauthentication - Basic and Digest (RFC 2617) [3], and Negotiate (RFC 2478)\n[4]. Git uses a extensible model where credential helpers can provide\ncredentials for protocols [5]. Several helpers support alternatives such as\nOAuth authentication (RFC 6749) [6], but this is typically done as an\nextension. For example, a helper might use basic auth and set the password\nto an OAuth Bearer access token. Git uses standard input and output to\ncommunicate with credential helpers.\n\nAfter a HTTP 401 response, Git would call a credential helper with the\nfollowing over standard input:\n\nprotocol=https\nhost=example.com\n\n\nAnd then a credential helper would return over standard output:\n\nprotocol=https\nhost=example.com\nusername=bob@id.example.com\npassword=\n\n\nGit then the following request to the remote, including the standard HTTP\nAuthorization header (RFC 7235 Section 4.2) [7]:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: Basic base64(bob@id.example.com:)\n\n\nCredential helpers are encouraged (see gitcredentials.txt) to return the\nminimum information necessary.\n\n\nLimitations\n===========\n\nBecause this credential model was built mostly for password based\nauthentication systems, it's somewhat limited. In particular:\n\n 1. To generate valid credentials, additional information about the request\n (or indeed the requestee and their device) may be required. For example,\n OAuth is based around scopes. A scope, like \"git.read\", might be\n required to read data from the remote. However, the remote cannot tell\n the credential helper what scope is required for this request.\n\n 2. This system is not fully extensible. Each time a new type of\n authentication (like OAuth Bearer) is invented, Git needs updates before\n credential helpers can take advantage of it (or leverage a new\n capability in libcurl).\n\n\nGoals\n=====\n\n * As a user with multiple federated cloud identities:\n \n * Reach out to a remote and have my credential helper automatically\n prompt me for the correct identity.\n * Allow credential helpers to differentiate between different authorities\n or authentication/authorization challenge types, even from the same DNS\n hostname (and without needing to use credential.useHttpPath).\n * Leverage existing authentication systems built-in to many operating\n systems and devices to boost security and reduce reliance on passwords.\n\n * As a Git host and/or cloud identity provider:\n \n * Enforce security policies (like requiring two-factor authentication)\n dynamically.\n * Allow integration with third party standard based identity providers in\n enterprises allowing customers to have a single plane of control for\n critical identities with access to source code.\n\n\nDesign Principles\n=================\n\n * Use the existing infrastructure. Git credential helpers are an\n already-working model.\n * Follow widely-adopted time-proven open standards, avoid net new ideas in\n the authentication space.\n * Minimize knowledge of authentication in Git; maintain modularity and\n extensibility.\n\n\nProposed Changes\n================\n\n 1. Teach Git to read HTTP response headers, specifically the standard\n WWW-Authenticate (RFC 7235 Section 4.1) headers.\n\n 2. Teach Git to include extra information about HTTP responses that require\n authentication when calling credential helpers. Specifically the\n WWW-Authenticate header information.\n \n Because the extra information forms an ordered list, and the existing\n credential helper I/O format only provides for simple key=value pairs,\n we introduce a new convention for transmitting an ordered list of\n values. Key names that are suffixed with a C-style array syntax should\n have values considered to form an order list, i.e. key[]=value, where\n the order of the key=value pairs in the stream specifies the order.\n \n For the WWW-Authenticate header values we opt to use the key wwwauth[].\n\n\nHandling the WWW-Authenticate header in detail\n==============================================\n\nRFC 6750 [8] envisions that OAuth Bearer resource servers would give\nresponses that include WWW-Authenticate headers, for example:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nSpecifically, a WWW-Authenticate header consists of a scheme and arbitrary\nattributes, depending on the scheme. This pattern enables generic OAuth or\nOpenID Connect [9] authorities. Note that it is possible to have several\nWWW-Authenticate challenges in a response.\n\nFirst Git attempts to make a request, unauthenticated, which fails with a\n401 response and includes WWW-Authenticate header(s).\n\nNext, Git invokes a credential helper which may prompt the user. If the user\napproves, a credential helper can generate a token (or any auth challenge\nresponse) to be used for that request.\n\nFor example: with a remote that supports bearer tokens from an OpenID\nConnect [9] authority, a credential helper can use OpenID Connect's\nDiscovery [10] and Dynamic Client Registration [11] to register a client and\nmake a request with the correct permissions to access the remote. In this\nmanner, a user can be dynamically sent to the right federated identity\nprovider for a remote without any up-front configuration or manual\nprocesses.\n\nFollowing from the principle of keeping authentication knowledge in Git to a\nminimum, we modify Git to add all WWW-Authenticate values to the credential\nhelper call.\n\nGit sends over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=Bearer realm=\"login.example\", scope=\"git.readwrite\"\nwwwauth[]=Basic realm=\"login.example\"\n\n\nA credential helper that understands the extra wwwauth[n] property can\ndecide on the \"best\" or correct authentication scheme, generate credentials\nfor the request, and interact with the user.\n\nThe credential helper would then return over standard output:\n\nprotocol=https\nhost=example.com\npath=foo.git\nusername=bob@identity.example\npassword=\n\n\nNote that WWW-Authenticate supports multiple challenges, either in one\nheader:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\nor in multiple headers:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: Bearer realm=\"login.example\", scope=\"git.readwrite\"\nWWW-Authenticate: Basic realm=\"login.example\"\n\n\nThese have equivalent meaning (RFC 2616 Section 4.2 [12]). To simplify the\nimplementation, Git will not merge or split up any of these WWW-Authenticate\nheaders, and instead pass each header line as one credential helper\nproperty. The credential helper is responsible for splitting, merging, and\notherwise parsing these header values.\n\nAn alternative option to sending the header fields individually would be to\nmerge the header values in to one key=value property, for example:\n\n...\nwwwauth=Bearer realm=\"login.example\", scope=\"git.readwrite\", Basic realm=\"login.example\"\n\n\n\nFuture work\n===========\n\nIn the future we can further expand the protocol to allow credential helpers\ndecide the best authentication scheme. Today credential helpers are still\nonly expected to return a username/password pair to Git, meaning the other\nauthentication schemes that may be offered still need challenge responses\nsent via a Basic Authorization header. The changes outlined above still\npermit helpers to select and configure an available authentication mode, but\nrequire the remote for example to unpack a bearer token from a basic\nchallenge.\n\nMore careful consideration is required in the handling of custom\nauthentication schemes which may not have a username, or may require\narbitrary additional request header values be set.\n\nFor example imagine a new \"FooBar\" authentication scheme that is surfaced in\nthe following response:\n\nHTTP/1.1 401 Unauthorized\nWWW-Authenticate: FooBar realm=\"login.example\", algs=\"ES256 PS256\"\n\n\nWith support for arbitrary authentication schemes, Git would call credential\nhelpers with the following over standard input:\n\nprotocol=https\nhost=example.com\nwwwauth[]=FooBar realm=\"login.example\", algs=\"ES256 PS256\", nonce=\"abc123\"\n\n\nAnd then an enlightened credential helper could return over standard output:\n\nprotocol=https\nhost=example.com\nauthtype=FooBar\nusername=bob@id.example.com\npassword=\nheader[]=X-FooBar: 12345\nheader[]=X-FooBar-Alt: ABCDEF\n\n\nGit would be expected to attach this authorization header to the next\nrequest:\n\nGET /info/refs?service=git-upload-pack HTTP/1.1\nHost: git.example\nGit-Protocol: version=2\nAuthorization: FooBar \nX-FooBar: 12345\nX-FooBar-Alt: ABCDEF\n\n\n\nWhy not SSH?\n============\n\nThere's nothing wrong with SSH. However, Git's Smart HTTP transport is\nwidely used, often with OAuth Bearer tokens. Git's Smart HTTP transport\nsometimes requires less client setup than SSH transport, and works in\nenvironments when SSH ports may be blocked. As long as Git supports HTTP\ntransport, it should support common and popular HTTP authentication methods.\n\n\nReferences\n==========\n\n * [0] [PATCH 0/8] [RFC] Enhance credential helper protocol to include auth\n headers\n https://lore.kernel.org/git/pull.1352.git.1663097156.gitgitgadget@gmail.com/\n\n * [1] [PATCH 0/3] Correct credential helper discrepancies handling input\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * [2] Git on the Server - The Protocols\n https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols\n\n * [3] HTTP Authentication: Basic and Digest Access Authentication\n https://datatracker.ietf.org/doc/html/rfc2617\n\n * [4] The Simple and Protected GSS-API Negotiation Mechanism\n https://datatracker.ietf.org/doc/html/rfc2478\n\n * [5] Git Credentials - Custom Helpers\n https://git-scm.com/docs/gitcredentials#_custom_helpers\n\n * [6] The OAuth 2.0 Authorization Framework\n https://datatracker.ietf.org/doc/html/rfc6749\n\n * [7] Hypertext Transfer Protocol (HTTP/1.1): Authentication\n https://datatracker.ietf.org/doc/html/rfc7235\n\n * [8] The OAuth 2.0 Authorization Framework: Bearer Token Usage\n https://datatracker.ietf.org/doc/html/rfc6750\n\n * [9] OpenID Connect Core 1.0\n https://openid.net/specs/openid-connect-core-1_0.html\n\n * [10] OpenID Connect Discovery 1.0\n https://openid.net/specs/openid-connect-discovery-1_0.html\n\n * [11] OpenID Connect Dynamic Client Registration 1.0\n https://openid.net/specs/openid-connect-registration-1_0.html\n\n * [12] Hypertext Transfer Protocol (HTTP/1.1)\n https://datatracker.ietf.org/doc/html/rfc2616\n\n\nUpdates from RFC\n================\n\n * Submitted first three patches as separate submission:\n https://lore.kernel.org/git/pull.1363.git.1663865974.gitgitgadget@gmail.com/\n\n * Various style fixes and updates to- and addition of comments.\n\n * Drop the explicit integer index in new 'array' style credential helper\n attrbiutes (\"key[n]=value\" becomes just \"key[]=value\").\n\n * Added test helper; a mini HTTP server, and several tests.\n\n\nUpdates in v3\n=============\n\n * Split final patch that added the test-http-server in to several, easier\n to review patches.\n\n * Updated wording in git-credential.txt to clarify which side of the\n credential helper protocol is sending/receiving the new wwwauth and\n authtype attributes.\n\n\nUpdates in v4\n=============\n\n * Drop authentication scheme selection authtype attribute patches to\n greatly simplify the series; auth scheme selection is punted to a future\n series. This series still allows credential helpers to generate\n credentials and intelligently select correct identities for a given auth\n challenge.\n\n\nUpdates in v5\n=============\n\n * Libify parts of daemon.c and share implementation with test-http-server.\n\n * Clarify test-http-server Git request regex pattern and auth logic\n comments.\n\n * Use STD*_FILENO in place of 'magic' file descriptor numbers.\n\n * Use strbuf_* functions in continuation header parsing.\n\n * Use configuration file to configure auth for test-http-server rather than\n command-line arguments. Add ability to specify arbitrary extra headers\n that is useful for testing 'malformed' server responses.\n\n * Use st_mult over unchecked multiplication in http.c curl callback\n functions.\n\n * Fix some documentation line break issues.\n\n * Reorder some commits to bring in the tests and test-http-server helper\n first and, then the WWW-Authentication changes, alongside tests to cover.\n\n * Expose previously static strvec_push_nodup function.\n\n * Merge the two timeout args for test-http-server (--timeout and\n --init-timeout) that were a hang-over from the original daemon.c but are\n no longer required here.\n\n * Be more careful around continuation headers where they may be empty\n strings. Add more tests to cover these header types.\n\n * Include standard trace2 tracing calls at start of test-http-server\n helper.\n\n\nUpdates in v6\n=============\n\n * Clarify the change to make logging optional in the check_dead_children()\n function during libification of daemon.c.\n\n * Fix missing pointer dereference bugs identified in libification of child\n process handling functions for daemon.c.\n\n * Add doc comments to child process handling function declarations in the\n daemon-utils.h header.\n\n * Align function parameter names with variable names at callsites for\n libified daemon functions.\n\n * Re-split out the test-http-server test helper commits in to smaller\n patches: error response handling, request parsing, http-backend\n pass-through, simple authentication, arbitrary header support.\n\n * Call out auth configuration file format for test-http-server test helper\n and supported options in commit messages, as well as a test to exercise\n and demonstrate these options.\n\n * Permit auth.token and auth.challenge to appear in any order; create the\n struct auth_module just-in-time as options for that scheme are read. This\n simplifies the configuration authoring of the test-http-server test\n helper.\n\n * Update tests to use auth.allowAnoymous in the patch that introduces the\n new test helper option.\n\n * Drop the strvec_push_nodup() commit and update the implementation of HTTP\n request header line folding to use xstrdup and strvec_pop and _pushf.\n\n * Use size_t instead of int in credential.c when iterating over the struct\n strvec credential members. Also drop the not required const and cast from\n the full_key definition and free.\n\n * Replace in-tree test-credential-helper-reply.sh test cred helper script\n with the lib-credential-helper.sh reusable 'lib' test script and shell\n functions to configure the helper behaviour.\n\n * Leverage sed over the while read $line loop in the test credential helper\n script.\n\n\nUpdates in v7\n=============\n\n * Address several whitespace and arg/param list alignment issues.\n\n * Rethink the test-http-helper worker-mode error and result enum to be more\n simple and more informative to the nature of the error.\n\n * Use uintmax_t to store the Content-Length of a request in the helper\n test-http-server. Maintain a bit flag to store if we received such a\n header.\n\n * Return a \"400 Bad Request\" HTTP response if we fail to parse the request\n in the test-http-server.\n\n * Add test case to cover request message parsing in test-http-server.\n\n * Use size_t and ALLOC_ARRAY over int and CALLOC_ARRAY respectively in\n get_auth_module.\n\n * Correctly free the split strbufs created in the header parsing loop in\n test-http-server.\n\n * Avoid needless comparison > 0 for unsigned types.\n\n * Always set optional outputs to NULL if not present in test helper config\n value handling.\n\n * Remove an accidentally commented-out test cleanup line for one test case\n in t5556.\n\n\nUpdates in v8\n=============\n\n * Drop custom HTTP test helper tool in favour of using a CGI shell script\n and Apache; avoiding the need to implement an HTTP server.\n\n * Avoid allocations in header reading callback unless we have a header we\n care about; act on the char* from libcurl directly rather than create a\n strbuf for each header.\n\n * Drop st_mult overflow guarding function in curl callback functions; we're\n not allocating memory based on the resulting value and just adds to\n potential confusion in the future.\n\n\nUpdates in v9\n=============\n\n * Drop anoynmous auth tests as these cases are already covered by all other\n tests that perform HTTP interactions with a remote today.\n\n * In the custom auth CGI script, avoid the empty-substitution in favour of\n testing explicitly for an empty string. Also simplify some other\n conditional expressions.\n\n * Avoid an allocation on each wwwauth[] credential helper key-value pair\n write.\n\n * Various style fixups.\n\n\nUpdates in v10\n==============\n\n * Style fixups.\n\n * Only consider space (SP ' ') and horizontal tab (HTAB '\\t') when\n detecting a header continuation line, as per the latest RFC on the\n matter.\n\n * Update references to old HTTP specs and formal grammars of header fields\n in comments.\n\n * Rewording of commit messages to remove confusing comment about the case\n sensitivity of header field names - this is not relevant with the current\n iteration of the header parsing code. Also update the message around\n libcurl header support to clarify that physical header lines are\n returned, but not 'logical' header lines.\n\n * Reword struct credential member doc comment to clarify the purpose of\n header_is_last_match is for re-folding lines of the WWW-Authenticate\n header.\n\n * Reintroduce helpful comments in tests to show the origin of the 'magic'\n base64 basic auth value.\n\n * Use grep -F to ensure we don't do regex matching; avoid interpreting\n special characters. Remove erronous insensitive comparison flag.\n\n\nUpdates in v11\n==============\n\n * Delete custom-auth.valid and .challenge explicitly in test cleanup.\n\n * Use tolower over strncasecmp in implementation of skip_iprefix_mem.\n\n * Use skip_iprefix_mem to match \"HTTP/\" header lines.\n\nMatthew John Cheetham (3):\n t5563: add tests for basic and anoymous HTTP access\n http: read HTTP WWW-Authenticate response headers\n credential: add WWW-Authenticate header to cred requests\n\n Documentation/git-credential.txt | 19 +-\n credential.c | 4 +\n credential.h | 16 ++\n git-compat-util.h | 19 ++\n http.c | 111 +++++++++++\n t/lib-httpd.sh | 1 +\n t/lib-httpd/apache.conf | 6 +\n t/lib-httpd/nph-custom-auth.sh | 39 ++++\n t/t5563-simple-http-auth.sh | 325 +++++++++++++++++++++++++++++++\n 9 files changed, 539 insertions(+), 1 deletion(-)\n create mode 100644 t/lib-httpd/nph-custom-auth.sh\n create mode 100755 t/t5563-simple-http-auth.sh\n\n\nbase-commit: c48035d29b4e524aed3a32f0403676f0d9128863\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1352%2Fmjcheetham%2Femu-v11\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1352/mjcheetham/emu-v11\nPull-Request: https://github.com/gitgitgadget/git/pull/1352\n\nRange-diff vs v10:\n\n 1: f3ccc53055a ! 1: f7b234c4038 t5563: add tests for basic and anoymous HTTP access\n @@ t/t5563-simple-http-auth.sh (new)\n +\n +per_test_cleanup () {\n +\trm -f *.cred &&\n -+\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.*\n ++\trm -f \"$HTTPD_ROOT_PATH\"/custom-auth.valid \\\n ++\t \"$HTTPD_ROOT_PATH\"/custom-auth.challenge\n +}\n +\n +test_expect_success 'setup repository' '\n 2: 703ac15222f ! 2: 3cca9ea0736 http: read HTTP WWW-Authenticate response headers\n @@ git-compat-util.h: static inline int skip_iprefix(const char *str, const char *p\n +\t\t\t\t const char *prefix,\n +\t\t\t\t const char **out, size_t *outlen)\n +{\n -+\tsize_t prefix_len = strlen(prefix);\n -+\n -+\tif (len < prefix_len)\n -+\t\treturn 0;\n -+\n -+\tif (!strncasecmp(buf, prefix, prefix_len)) {\n -+\t\t*out = buf + prefix_len;\n -+\t\t*outlen = len - prefix_len;\n -+\t\treturn 1;\n -+\t}\n -+\n ++\tdo {\n ++\t\tif (!*prefix) {\n ++\t\t\t*out = buf;\n ++\t\t\t*outlen = len;\n ++\t\t\treturn 1;\n ++\t\t}\n ++\t} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));\n +\treturn 0;\n +}\n +\n @@ http.c: size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buff\n +\t * We only care about the last HTTP request response's headers so clear\n +\t * the existing array.\n +\t */\n -+\tif (!strncasecmp(ptr, \"http/\", 5))\n ++\tif (skip_iprefix_mem(ptr, size, \"http/\", &val, &val_len))\n +\t\tstrvec_clear(values);\n +\n +exit:\n 3: 186da54fd3b = 3: b774acf3896 credential: add WWW-Authenticate header to cred requests\n\n-- \ngitgitgadget\n"},{"id":"472809","messageId":"3cca9ea0736b6b58065ec03c052ff5acd1349dcc.1677518420.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v11.git.1677518420.gitgitgadget@gmail.com","subject":"[PATCH v11 2/3] http: read HTTP WWW-Authenticate response headers","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-27T17:20:19Z","receivedAt":"2023-02-27T17:20:33Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nRead and store the HTTP WWW-Authenticate response headers made for\na particular request.\n\nThis will allow us to pass important authentication challenge\ninformation to credential helpers or others that would otherwise have\nbeen lost.\n\nlibcurl only provides us with the ability to read all headers recieved\nfor a particular request, including any intermediate redirect requests\nor proxies. The lines returned by libcurl include HTTP status lines\ndelinating any intermediate requests such as \"HTTP/1.1 200\". We use\nthese lines to reset the strvec of WWW-Authenticate header values as\nwe encounter them in order to only capture the final response headers.\n\nThe collection of all header values matching the WWW-Authenticate\nheader is complicated by the fact that it is legal for header fields to\nbe continued over multiple lines, but libcurl only gives us each\nphysical line a time, not each logical header. This line folding feature\nis deprecated in RFC 7230 [1] but older servers may still emit them, so\nwe need to handle them.\n\nIn the future [2] we may be able to leverage functions to read headers\nfrom libcurl itself, but as of today we must do this ourselves.\n\n[1] https://www.rfc-editor.org/rfc/rfc7230#section-3.2\n[2] https://daniel.haxx.se/blog/2022/03/22/a-headers-api-for-libcurl/\n\nSigned-off-by: Matthew John Cheetham \n---\n credential.c | 1 +\n credential.h | 16 +++++++\n git-compat-util.h | 19 ++++++++\n http.c | 111 ++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 147 insertions(+)\n\ndiff --git a/credential.c b/credential.c\nindex f6389a50684..897b4679333 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -22,6 +22,7 @@ void credential_clear(struct credential *c)\n \tfree(c->username);\n \tfree(c->password);\n \tstring_list_clear(&c->helpers, 0);\n+\tstrvec_clear(&c->wwwauth_headers);\n \n \tcredential_init(c);\n }\ndiff --git a/credential.h b/credential.h\nindex f430e77fea4..3756a54c74d 100644\n--- a/credential.h\n+++ b/credential.h\n@@ -2,6 +2,7 @@\n #define CREDENTIAL_H\n \n #include \"string-list.h\"\n+#include \"strvec.h\"\n \n /**\n * The credentials API provides an abstracted way of gathering username and\n@@ -115,6 +116,20 @@ struct credential {\n \t */\n \tstruct string_list helpers;\n \n+\t/**\n+\t * A `strvec` of WWW-Authenticate header values. Each string\n+\t * is the value of a WWW-Authenticate header in an HTTP response,\n+\t * in the order they were received in the response.\n+\t */\n+\tstruct strvec wwwauth_headers;\n+\n+\t/**\n+\t * Internal use only. Keeps track of if we previously matched against a\n+\t * WWW-Authenticate header line in order to re-fold future continuation\n+\t * lines into one value.\n+\t */\n+\tunsigned header_is_last_match:1;\n+\n \tunsigned approved:1,\n \t\t configured:1,\n \t\t quit:1,\n@@ -130,6 +145,7 @@ struct credential {\n \n #define CREDENTIAL_INIT { \\\n \t.helpers = STRING_LIST_INIT_DUP, \\\n+\t.wwwauth_headers = STRVEC_INIT, \\\n }\n \n /* Initialize a credential structure, setting all fields to empty. */\ndiff --git a/git-compat-util.h b/git-compat-util.h\nindex a76d0526f79..62747bf6676 100644\n--- a/git-compat-util.h\n+++ b/git-compat-util.h\n@@ -1266,6 +1266,25 @@ static inline int skip_iprefix(const char *str, const char *prefix,\n \treturn 0;\n }\n \n+/*\n+ * Like skip_prefix_mem, but compare case-insensitively. Note that the\n+ * comparison is done via tolower(), so it is strictly ASCII (no multi-byte\n+ * characters or locale-specific conversions).\n+ */\n+static inline int skip_iprefix_mem(const char *buf, size_t len,\n+\t\t\t\t const char *prefix,\n+\t\t\t\t const char **out, size_t *outlen)\n+{\n+\tdo {\n+\t\tif (!*prefix) {\n+\t\t\t*out = buf;\n+\t\t\t*outlen = len;\n+\t\t\treturn 1;\n+\t\t}\n+\t} while (len-- > 0 && tolower(*buf++) == tolower(*prefix++));\n+\treturn 0;\n+}\n+\n static inline int strtoul_ui(char const *s, int base, unsigned int *result)\n {\n \tunsigned long ul;\ndiff --git a/http.c b/http.c\nindex 8a5ba3f4776..677266afff1 100644\n--- a/http.c\n+++ b/http.c\n@@ -183,6 +183,115 @@ size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *buffer_)\n \treturn nmemb;\n }\n \n+/*\n+ * A folded header continuation line starts with any number of spaces or\n+ * horizontal tab characters (SP or HTAB) as per RFC 7230 section 3.2.\n+ * It is not a continuation line if the line starts with any other character.\n+ */\n+static inline int is_hdr_continuation(const char *ptr, const size_t size)\n+{\n+\treturn size && (*ptr == ' ' || *ptr == '\\t');\n+}\n+\n+static size_t fwrite_wwwauth(char *ptr, size_t eltsize, size_t nmemb, void *p)\n+{\n+\tsize_t size = eltsize * nmemb;\n+\tstruct strvec *values = &http_auth.wwwauth_headers;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tconst char *val;\n+\tsize_t val_len;\n+\n+\t/*\n+\t * Header lines may not come NULL-terminated from libcurl so we must\n+\t * limit all scans to the maximum length of the header line, or leverage\n+\t * strbufs for all operations.\n+\t *\n+\t * In addition, it is possible that header values can be split over\n+\t * multiple lines as per RFC 7230. 'Line folding' has been deprecated\n+\t * but older servers may still emit them. A continuation header field\n+\t * value is identified as starting with a space or horizontal tab.\n+\t *\n+\t * The formal definition of a header field as given in RFC 7230 is:\n+\t *\n+\t * header-field = field-name \":\" OWS field-value OWS\n+\t *\n+\t * field-name = token\n+\t * field-value = *( field-content / obs-fold )\n+\t * field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ]\n+\t * field-vchar = VCHAR / obs-text\n+\t *\n+\t * obs-fold = CRLF 1*( SP / HTAB )\n+\t * ; obsolete line folding\n+\t * ; see Section 3.2.4\n+\t */\n+\n+\t/* Start of a new WWW-Authenticate header */\n+\tif (skip_iprefix_mem(ptr, size, \"www-authenticate:\", &val, &val_len)) {\n+\t\tstrbuf_add(&buf, val, val_len);\n+\n+\t\t/*\n+\t\t * Strip the CRLF that should be present at the end of each\n+\t\t * field as well as any trailing or leading whitespace from the\n+\t\t * value.\n+\t\t */\n+\t\tstrbuf_trim(&buf);\n+\n+\t\tstrvec_push(values, buf.buf);\n+\t\thttp_auth.header_is_last_match = 1;\n+\t\tgoto exit;\n+\t}\n+\n+\t/*\n+\t * This line could be a continuation of the previously matched header\n+\t * field. If this is the case then we should append this value to the\n+\t * end of the previously consumed value.\n+\t */\n+\tif (http_auth.header_is_last_match && is_hdr_continuation(ptr, size)) {\n+\t\t/*\n+\t\t * Trim the CRLF and any leading or trailing from this line.\n+\t\t */\n+\t\tstrbuf_add(&buf, ptr, size);\n+\t\tstrbuf_trim(&buf);\n+\n+\t\t/*\n+\t\t * At this point we should always have at least one existing\n+\t\t * value, even if it is empty. Do not bother appending the new\n+\t\t * value if this continuation header is itself empty.\n+\t\t */\n+\t\tif (!values->nr) {\n+\t\t\tBUG(\"should have at least one existing header value\");\n+\t\t} else if (buf.len) {\n+\t\t\tchar *prev = xstrdup(values->v[values->nr - 1]);\n+\n+\t\t\t/* Join two non-empty values with a single space. */\n+\t\t\tconst char *const sp = *prev ? \" \" : \"\";\n+\n+\t\t\tstrvec_pop(values);\n+\t\t\tstrvec_pushf(values, \"%s%s%s\", prev, sp, buf.buf);\n+\t\t\tfree(prev);\n+\t\t}\n+\n+\t\tgoto exit;\n+\t}\n+\n+\t/* Not a continuation of a previously matched auth header line. */\n+\thttp_auth.header_is_last_match = 0;\n+\n+\t/*\n+\t * If this is a HTTP status line and not a header field, this signals\n+\t * a different HTTP response. libcurl writes all the output of all\n+\t * response headers of all responses, including redirects.\n+\t * We only care about the last HTTP request response's headers so clear\n+\t * the existing array.\n+\t */\n+\tif (skip_iprefix_mem(ptr, size, \"http/\", &val, &val_len))\n+\t\tstrvec_clear(values);\n+\n+exit:\n+\tstrbuf_release(&buf);\n+\treturn size;\n+}\n+\n size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf)\n {\n \treturn nmemb;\n@@ -1864,6 +1973,8 @@ static int http_request(const char *url,\n \t\t\t\t\t fwrite_buffer);\n \t}\n \n+\tcurl_easy_setopt(slot->curl, CURLOPT_HEADERFUNCTION, fwrite_wwwauth);\n+\n \taccept_language = http_get_accept_language_header();\n \n \tif (accept_language)\n-- \ngitgitgadget\n\n"},{"id":"472810","messageId":"b774acf3896c4c1818f48776b1daba0ca6f74df8.1677518420.git.gitgitgadget@gmail.com","threadId":"58425","inReplyTo":"pull.1352.v11.git.1677518420.gitgitgadget@gmail.com","subject":"[PATCH v11 3/3] credential: add WWW-Authenticate header to cred requests","fromName":"Matthew John Cheetham via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2023-02-27T17:20:20Z","receivedAt":"2023-02-27T17:20:44Z","isPatch":true,"sender":{"key":"mjcheetham@outlook.com","avatar":"https://avatars.githubusercontent.com/u/5658207?v=4"},"body":"From: Matthew John Cheetham \n\nAdd the value of the WWW-Authenticate response header to credential\nrequests. Credential helpers that understand and support HTTP\nauthentication and authorization can use this standard header (RFC 2616\nSection 14.47 [1]) to generate valid credentials.\n\nWWW-Authenticate headers can contain information pertaining to the\nauthority, authentication mechanism, or extra parameters/scopes that are\nrequired.\n\nThe current I/O format for credential helpers only allows for unique\nnames for properties/attributes, so in order to transmit multiple header\nvalues (with a specific order) we introduce a new convention whereby a\nC-style array syntax is used in the property name to denote multiple\nordered values for the same property.\n\nIn this case we send multiple `wwwauth[]` properties where the order\nthat the repeated attributes appear in the conversation reflects the\norder that the WWW-Authenticate headers appeared in the HTTP response.\n\nAdd a set of tests to exercise the HTTP authentication header parsing\nand the interop with credential helpers. Credential helpers will receive\nWWW-Authenticate information in credential requests.\n\n[1] https://datatracker.ietf.org/doc/html/rfc2616#section-14.47\n\nSigned-off-by: Matthew John Cheetham \n---\n Documentation/git-credential.txt | 19 ++-\n credential.c | 3 +\n t/t5563-simple-http-auth.sh | 242 +++++++++++++++++++++++++++++++\n 3 files changed, 263 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-credential.txt b/Documentation/git-credential.txt\nindex ac2818b9f66..50759153ef1 100644\n--- a/Documentation/git-credential.txt\n+++ b/Documentation/git-credential.txt\n@@ -113,7 +113,13 @@ separated by an `=` (equals) sign, followed by a newline.\n The key may contain any bytes except `=`, newline, or NUL. The value may\n contain any bytes except newline or NUL.\n \n-In both cases, all bytes are treated as-is (i.e., there is no quoting,\n+Attributes with keys that end with C-style array brackets `[]` can have\n+multiple values. Each instance of a multi-valued attribute forms an\n+ordered list of values - the order of the repeated attributes defines\n+the order of the values. An empty multi-valued attribute (`key[]=\\n`)\n+acts to clear any previous entries and reset the list.\n+\n+In all cases, all bytes are treated as-is (i.e., there is no quoting,\n and one cannot transmit a value with newline or NUL in it). The list of\n attributes is terminated by a blank line or end-of-file.\n \n@@ -160,6 +166,17 @@ empty string.\n Components which are missing from the URL (e.g., there is no\n username in the example above) will be left unset.\n \n+`wwwauth[]`::\n+\n+\tWhen an HTTP response is received by Git that includes one or more\n+\t'WWW-Authenticate' authentication headers, these will be passed by Git\n+\tto credential helpers.\n++\n+Each 'WWW-Authenticate' header value is passed as a multi-valued\n+attribute 'wwwauth[]', where the order of the attributes is the same as\n+they appear in the HTTP response. This attribute is 'one-way' from Git\n+to pass additional information to credential helpers.\n+\n Unrecognised attributes are silently discarded.\n \n GIT\ndiff --git a/credential.c b/credential.c\nindex 897b4679333..f566c8ab195 100644\n--- a/credential.c\n+++ b/credential.c\n@@ -270,6 +270,9 @@ void credential_write(const struct credential *c, FILE *fp)\n \tcredential_write_item(fp, \"path\", c->path, 0);\n \tcredential_write_item(fp, \"username\", c->username, 0);\n \tcredential_write_item(fp, \"password\", c->password, 0);\n+\tfor (size_t i = 0; i < c->wwwauth_headers.nr; i++)\n+\t\tcredential_write_item(fp, \"wwwauth[]\", c->wwwauth_headers.v[i],\n+\t\t\t\t 0);\n }\n \n static int run_credential_helper(struct credential *c,\ndiff --git a/t/t5563-simple-http-auth.sh b/t/t5563-simple-http-auth.sh\nindex bc880bf80f9..ccf7e54b073 100755\n--- a/t/t5563-simple-http-auth.sh\n+++ b/t/t5563-simple-http-auth.sh\n@@ -70,6 +70,248 @@ test_expect_success 'access using basic auth' '\n \texpect_credential_query get <<-EOF &&\n \tprotocol=http\n \thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth invalid credentials' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\ttest_must_fail git ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query erase <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=baduser\n+\tpassword=wrong-passwd\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with extra challenges' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\" param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth mixed-case wwwauth header name' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\twww-authenticate: foobar param1=\"value1\" param2=\"value2\"\n+\tWWW-AUTHENTICATE: BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\tWwW-aUtHeNtIcAtE: baSiC realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=foobar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=BEARER authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=baSiC realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.challenge\" <<-EOF &&\n+\tWWW-Authenticate: FooBar param1=\"value1\"\n+\t param2=\"value2\"\n+\tWWW-Authenticate: Bearer authorize_uri=\"id.example.com\"\n+\t p=1\n+\t q=0\n+\tWWW-Authenticate: Basic realm=\"example.com\"\n+\tEOF\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header empty continuations' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" param2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Bearer authorize_uri=\\\"id.example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" p=1\\r\\n\" >>$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \" q=0\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\\r\\n\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Bearer authorize_uri=\"id.example.com\" p=1 q=0\n+\twwwauth[]=Basic realm=\"example.com\"\n+\tEOF\n+\n+\texpect_credential_query store <<-EOF\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+'\n+\n+test_expect_success 'access using basic auth with wwwauth header mixed line-endings' '\n+\ttest_when_finished \"per_test_cleanup\" &&\n+\n+\tset_credential_reply get <<-EOF &&\n+\tusername=alice\n+\tpassword=secret-passwd\n+\tEOF\n+\n+\t# Basic base64(alice:secret-passwd)\n+\tcat >\"$HTTPD_ROOT_PATH/custom-auth.valid\" <<-EOF &&\n+\tBasic YWxpY2U6c2VjcmV0LXBhc3N3ZA==\n+\tEOF\n+\n+\tCHALLENGE=\"$HTTPD_ROOT_PATH/custom-auth.challenge\" &&\n+\n+\t# Note that leading and trailing whitespace is important to correctly\n+\t# simulate a continuation/folded header.\n+\tprintf \"\">$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: FooBar param1=\\\"value1\\\"\\r\\n\" >$CHALLENGE &&\n+\tprintf \" \\r\\n\" >>$CHALLENGE &&\n+\tprintf \"\\tparam2=\\\"value2\\\"\\r\\n\" >>$CHALLENGE &&\n+\tprintf \"WWW-Authenticate: Basic realm=\\\"example.com\\\"\" >>$CHALLENGE &&\n+\n+\ttest_config_global credential.helper test-helper &&\n+\tgit ls-remote \"$HTTPD_URL/custom_auth/repo.git\" &&\n+\n+\texpect_credential_query get <<-EOF &&\n+\tprotocol=http\n+\thost=$HTTPD_DEST\n+\twwwauth[]=FooBar param1=\"value1\" param2=\"value2\"\n+\twwwauth[]=Basic realm=\"example.com\"\n \tEOF\n \n \texpect_credential_query store <<-EOF\n-- \ngitgitgadget\n"},{"id":"472822","messageId":"Y/0SNW4p0oS9RCbE@coredump.intra.peff.net","threadId":"58425","inReplyTo":"pull.1352.v11.git.1677518420.gitgitgadget@gmail.com","subject":"Re: [PATCH v11 0/3] Enhance credential helper protocol to include auth headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-02-27T20:27:33Z","receivedAt":"2023-02-27T20:27:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 27, 2023 at 05:20:17PM +0000, Matthew John Cheetham via GitGitGadget wrote:\n\n> Updates in v11\n> ==============\n> \n> * Delete custom-auth.valid and .challenge explicitly in test cleanup.\n> \n> * Use tolower over strncasecmp in implementation of skip_iprefix_mem.\n> \n> * Use skip_iprefix_mem to match \"HTTP/\" header lines.\n\nThanks, I looked over all three changes and the whole thing looks good\nto me. The first one isn't strictly necessary if we're not renaming the\nscript, but I agree that it is probably worth being a bit more strict\nwhen deleting in $HTTPD_ROOT_PATH.\n\n-Peff\n"},{"id":"474222","messageId":"a1d44a2f-fb3e-fa99-121d-804a884e6aed@gmx.de","threadId":"58425","inReplyTo":"230206.86y1pa4gdh.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-03-27T09:05:44Z","receivedAt":"2023-03-27T09:08:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Mon, 6 Feb 2023, Ævar Arnfjörð Bjarmason wrote:\n\n> we currently have CI tests running Apache on *nix boxes, but you're\n> suggesting a loss of coverage on Windows\n>\n> Is it really harder to just install (or even ship our own package of)\n> Apache for Windows than it is to embark on PID file handling, logging,\n> timeout management and the long tail of \"80% is easy, the rest is really\n> hard\" of writing our own production-class httpd (as the suggestion is to\n> have it eventually mature beyond the test suite)?\n\nYes, it _is_ that much harder, and it would result in yet more painful\nincreases of the build times which have really gotten out of hand in the\npast year.\n\nCiao,\nJohannes\n"},{"id":"474223","messageId":"0f94b998-e223-85cc-7730-f75675d5e649@gmx.de","threadId":"58425","inReplyTo":"Y91FjhNgZGz6foFl@coredump.intra.peff.net","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2023-03-27T09:10:40Z","receivedAt":"2023-03-27T09:11:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Fri, 3 Feb 2023, Jeff King wrote:\n\n> On Thu, Feb 02, 2023 at 11:14:33AM +0100, Johannes Schindelin wrote:\n>\n> > > I do not mind reverting the merge to 'next' to have an improved\n> > > version. Your \"do we really want to add a custom server based on\n> > > questionable codebase whose quality as a test-bed for real world\n> > > usage is dubious?\" is a valid concern.\n> >\n> > Except.\n> >\n> > Except that this code base would have made for a fine base to potentially\n> > implement an HTTPS-based replacement for the aging and insecure\n> > git-daemon.\n>\n> I'm skeptical that it is a good idea for Git to implement a custom http\n> server from scratch.\n\nTo be clear: I never suggested to implement a generic HTTP server.\n\nAll I wanted was to have a replacement for `git daemon` that speaks\nhttps:// instead of git://. It does not have to speak to every browser out\nthere, it only needs to respond well when speaking to Git clients. That is\na much, much smaller surface than \"production-ready server, HTTP/2 and so\non\".\n\nAnd while the proposed test helper was not quite complete in that way, and\nwhile it may have had much of the `git daemon` code that you would love to\nlose, it would have offered an incremental way forward.\n\nI am afraid that this way forward is now blocked, and we're further away\nfrom dropping that `git daemon` code you wanted to drop than ever.\n\nCiao,\nJohannes\n"},{"id":"474354","messageId":"20230328185520.GE18558@coredump.intra.peff.net","threadId":"58425","inReplyTo":"0f94b998-e223-85cc-7730-f75675d5e649@gmx.de","subject":"Re: [PATCH v7 00/12] Enhance credential helper protocol to include auth headers","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-28T18:55:20Z","receivedAt":"2023-03-28T18:55:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 27, 2023 at 11:10:40AM +0200, Johannes Schindelin wrote:\n\n> > I'm skeptical that it is a good idea for Git to implement a custom http\n> > server from scratch.\n> \n> To be clear: I never suggested to implement a generic HTTP server.\n> \n> All I wanted was to have a replacement for `git daemon` that speaks\n> https:// instead of git://. It does not have to speak to every browser out\n> there, it only needs to respond well when speaking to Git clients. That is\n> a much, much smaller surface than \"production-ready server, HTTP/2 and so\n> on\".\n\nI guess I don't see the point of having this in our test suite, though.\nWe do want to test things like HTTP/2, SSL, and so on in our test suite.\nSo either we have a split in our tests (some use apache, some don't,\nwhich presumably means many tests are still not run on Windows), or this\ncustom HTTP server eventually grows to do all of those other things.\n\nI can see the utility outside the tests of a quick \"let me stand up an\nHTTP server to access Git\" tool. But even there, I'd be considered with\nfeature creep as regular users ignore any warnings about its lack of\nencryption/robustness, and so on. And it feels like something that could\nutilize work already done by others in making a web server. Yes, that's\na new dependency for the tool, but there are a lot of options out there.\nSurely one of them is worth building on?\n\n> And while the proposed test helper was not quite complete in that way, and\n> while it may have had much of the `git daemon` code that you would love to\n> lose, it would have offered an incremental way forward.\n> \n> I am afraid that this way forward is now blocked, and we're further away\n> from dropping that `git daemon` code you wanted to drop than ever.\n\nI don't see how pushing the same code into an http server helps. If we\ncould have incrementally improved it there, we could incrementally\nimprove it in git-daemon, too.\n\n-Peff\n"}]}