threads / rfc / 30674

[RFC] Deal with HTTP 401 by requesting credentials.

Subject: [RFC] Deal with HTTP 401 by requesting credentials.

## tl;dr

8 messages between May 31, 2012 and Jun 5, 2012.

replies: 7people: 3as markdown or json

Kevin Stange· May 31, 2012, 22:24 UTC · lore

Request credentials from the user if none are already defined when a HTTP 401 is received on a restricted repository. Then, resubmit the request and return the final result.

This allows all webdav transactions to obtain credentials without having to individually handle the case in each request. Having push working with HTTP auth is needed for a use case I have where storing the credentials in .netrc or using SSH keys is inappropriate.

Apologies for anything wrong I might have done here. I'm not used to procedures for this sort of patch submission, or terribly familiar with the code base. I'm seeking advice on whether this approach is sane or completely crazy, and I'm willing to adjust it to make it suitable for inclusion.

Signed-off-by: Kevin <kevin@steadfast.net>
---
 http.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
diff --git a/http.c b/http.c
index 5cb87f1..e1c9e65 100644
--- a/http.c
+++ b/http.c
@@ -668,6 +668,21 @@ void finish_active_slot(struct active_request_slot
*slot)
 	closedown_active_slot(slot);
 	curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE, &slot->http_code);

+	if (slot->http_code == 401) {
+		if(!http_auth.username && !http_auth.password) {
+			active_requests++;
+			credential_fill(&http_auth);
+			init_curl_http_auth(slot->curl);
+			(*slot->finished) = 1;
+			if (start_active_slot(slot)) {
+				run_active_slot(slot);
+				return;
+			}
+		} else {
+			fprintf(stderr,"Authentication Failed!\n");
+		}
+	}
+
 	if (slot->finished != NULL)
 		(*slot->finished) = 1;

-- 
1.7.11.rc0.55.gb2478aa

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867
Kevin Stange· May 31, 2012, 23:18 UTC · re: Kevin Stange · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On 05/31/2012 05:24 PM, Kevin Stange wrote:
Show 5 quoted lines
> Apologies for anything wrong I might have done here.  I'm not used to
> procedures for this sort of patch submission, or terribly familiar with
> the code base.  I'm seeking advice on whether this approach is sane or
> completely crazy, and I'm willing to adjust it to make it suitable for
> inclusion.
Of course, I failed to get my email client to send the patch unwrapped.  Only one line appears to have been affected, but I don't think this is a final patch.  I'll be sure to make a proper submission for any future patches I send.  Sorry!
-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867
Jeff King· Jun 1, 2012, 08:35 UTC · re: Kevin Stange · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:
Show 8 quoted lines
> Request credentials from the user if none are already defined when a
> HTTP 401 is received on a restricted repository.  Then, resubmit the
> request and return the final result.
> 
> This allows all webdav transactions to obtain credentials without having
> to individually handle the case in each request.  Having push working
> with HTTP auth is needed for a use case I have where storing the
> credentials in .netrc or using SSH keys is inappropriate.

We already do this at a higher level in http_request, which in turns calls into finish_active_slot. So if we were going to go this route, wouldn't we also want to remove the 401 handling in http_request?

The dumb-http push code is the only thing that does not go through http_request these days. So another option would be to refactor it to go through that central point. I took a brief look at this when I was updating the credential code a few months ago, but didn't consider it a priority, as most people should be using smart http these days. Is there a reason you can't use smart-http? It's significantly more efficient.

You also don't necessarily need to handle 401 in every code path of http-push; once we see the credential once, we will use it everywhere, so you really only need to handle it on the initial request (assuming that all requests will have the same authorization requirements).

Show 8 quoted lines
> Apologies for anything wrong I might have done here.  I'm not used to
> procedures for this sort of patch submission, or terribly familiar with
> the code base.  I'm seeking advice on whether this approach is sane or
> completely crazy, and I'm willing to adjust it to make it suitable for
> inclusion.
> 
> Signed-off-by: Kevin <kevin@steadfast.net>
> ---

Cover letter material (i.e., anything that would not go into the commit message of the final commit) should go below the "---".

Show 23 quoted lines
> diff --git a/http.c b/http.c
> index 5cb87f1..e1c9e65 100644
> --- a/http.c
> +++ b/http.c
> @@ -668,6 +668,21 @@ void finish_active_slot(struct active_request_slot
> *slot)
>  	closedown_active_slot(slot);
>  	curl_easy_getinfo(slot->curl, CURLINFO_HTTP_CODE, &slot->http_code);
> 
> +	if (slot->http_code == 401) {
> +		if(!http_auth.username && !http_auth.password) {
> +			active_requests++;
> +			credential_fill(&http_auth);
> +			init_curl_http_auth(slot->curl);
> +			(*slot->finished) = 1;
> +			if (start_active_slot(slot)) {
> +				run_active_slot(slot);
> +				return;
> +			}
> +		} else {
> +			fprintf(stderr,"Authentication Failed!\n");
> +		}
> +	}

Is it safe to just run start_active_slot again without reinitializing the request? The 401-handling code in http_request actually restarts a new request. I don't immediately see any state that would need to be reset; we might have written some data to the output file if curl gave us any body data, but presumably it would not have done so for a 401.

In the "else" clause you add, I don't think there's any point in printing an error. The 401 should get propagated back to the caller, who will produce an error. However, you _should_ call credential_reject, since you know that the credential you have doesn't work.

Similarly, you would want to call credential_accept after a successful request, so that helpers can store it.

-Peff
Junio C Hamano· Jun 1, 2012, 17:01 UTC · re: Jeff King · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

Jeff King <peff@peff.net> writes:
Show 14 quoted lines
> On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:
>
>> Request credentials from the user if none are already defined when a
>> HTTP 401 is received on a restricted repository.  Then, resubmit the
>> request and return the final result.
>> 
>> This allows all webdav transactions to obtain credentials without having
>> to individually handle the case in each request.  Having push working
>> with HTTP auth is needed for a use case I have where storing the
>> credentials in .netrc or using SSH keys is inappropriate.
>
> We already do this at a higher level in http_request, which in turns
> calls into finish_active_slot. So if we were going to go this route,
> wouldn't we also want to remove the 401 handling in http_request?

Wouldn't the higher levels know a lot more about the context this request was made, though? What problem does the patch try to solve? Some higher level callers missing the "if we got 401, then reset and retry" logic? Wouldn't it be saner to fix the breakage there?

> The dumb-http push code is the only thing that does not go through
> http_request these days. So another option would be to refactor it to go
> through that central point.
It does sound like the right approach to take.
Thanks.
Jeff King· Jun 5, 2012, 16:23 UTC · re: Junio C Hamano · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On Fri, Jun 01, 2012 at 10:01:58AM -0700, Junio C Hamano wrote:
Show 17 quoted lines
> >> Request credentials from the user if none are already defined when a
> >> HTTP 401 is received on a restricted repository.  Then, resubmit the
> >> request and return the final result.
> >> 
> >> This allows all webdav transactions to obtain credentials without having
> >> to individually handle the case in each request.  Having push working
> >> with HTTP auth is needed for a use case I have where storing the
> >> credentials in .netrc or using SSH keys is inappropriate.
> >
> > We already do this at a higher level in http_request, which in turns
> > calls into finish_active_slot. So if we were going to go this route,
> > wouldn't we also want to remove the 401 handling in http_request?
> 
> Wouldn't the higher levels know a lot more about the context this
> request was made, though?  What problem does the patch try to solve?
> Some higher level callers missing the "if we got 401, then reset and
> retry" logic?  Wouldn't it be saner to fix the breakage there?

I don't know that the higher level really has much more context; the URL is really the only context we use. I think the problem is adequately solved in http_request; it is simply that the http-push code paths do not use it. The patch solves that by pushing it to a lower level.

-Peff
Kevin Stange· Jun 1, 2012, 17:02 UTC · re: Jeff King · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On 06/01/2012 03:35 AM, Jeff King wrote:
Show 14 quoted lines
> On Thu, May 31, 2012 at 05:24:55PM -0500, Kevin Stange wrote:
> 
>> Request credentials from the user if none are already defined when a
>> HTTP 401 is received on a restricted repository.  Then, resubmit the
>> request and return the final result.
>>
>> This allows all webdav transactions to obtain credentials without having
>> to individually handle the case in each request.  Having push working
>> with HTTP auth is needed for a use case I have where storing the
>> credentials in .netrc or using SSH keys is inappropriate.
> 
> We already do this at a higher level in http_request, which in turns
> calls into finish_active_slot. So if we were going to go this route,
> wouldn't we also want to remove the 401 handling in http_request?

I did see the work being done there, and considered removing it. In fact, I used it as a reference for working out what was going on in the authentication process. I decided not to do anything there until soliciting feedback and deciding whether my approach was reasonable.

Show 6 quoted lines
> The dumb-http push code is the only thing that does not go through
> http_request these days. So another option would be to refactor it to go
> through that central point. I took a brief look at this when I was
> updating the credential code a few months ago, but didn't consider it a
> priority, as most people should be using smart http these days. Is there
> a reason you can't use smart-http? It's significantly more efficient.

Smart HTTP didn't come up in any of my Google searches. With that as an option, I might just drop this work now. I'd rather see incomplete methods that aren't recommended go away than further facilitate their use, personally.

> You also don't necessarily need to handle 401 in every code path of
> http-push; once we see the credential once, we will use it everywhere,
> so you really only need to handle it on the initial request (assuming
> that all requests will have the same authorization requirements).

I made the change where I did because I wasn't sure if the push code was avoiding using http_request intentionally, and wasn't sure whether new code would be written that avoid it as well.

If that's not the case, then I gather http-push would be better rewritten to just use http_request, if anything.

Show 11 quoted lines
>> Apologies for anything wrong I might have done here.  I'm not used to
>> procedures for this sort of patch submission, or terribly familiar with
>> the code base.  I'm seeking advice on whether this approach is sane or
>> completely crazy, and I'm willing to adjust it to make it suitable for
>> inclusion.
>>
>> Signed-off-by: Kevin <kevin@steadfast.net>
>> ---
> 
> Cover letter material (i.e., anything that would not go into the commit
> message of the final commit) should go below the "---".
Thanks, will remember this for future reference.
Show 5 quoted lines
> Is it safe to just run start_active_slot again without reinitializing
> the request? The 401-handling code in http_request actually restarts a
> new request. I don't immediately see any state that would need to be
> reset; we might have written some data to the output file if curl gave
> us any body data, but presumably it would not have done so for a 401.

In my tests, this particular code flow never returns anything to the original request call because I interrupt it and start the request over. Now that I look at it again, there's a chance it leaks the curl response, but it doesn't return that response and the new request works fine, replacing the original.

Show 7 quoted lines
> In the "else" clause you add, I don't think there's any point in
> printing an error. The 401 should get propagated back to the caller, who
> will produce an error. However, you _should_ call credential_reject,
> since you know that the credential you have doesn't work.
> 
> Similarly, you would want to call credential_accept after a successful
> request, so that helpers can store it.

If I decide to continue working on this, I will keep these in mind. I'm pretty sure that if I can get smart HTTP working, there's no reason to even bother with this from my perspective, unless you think there's substantial value in it.

Thanks for the detailed feedback on the proposed change and suggestions on alternative options.

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867
Jeff King· Jun 5, 2012, 16:28 UTC · re: Kevin Stange · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On Fri, Jun 01, 2012 at 12:02:08PM -0500, Kevin Stange wrote:
Show 10 quoted lines
> > The dumb-http push code is the only thing that does not go through
> > http_request these days. So another option would be to refactor it to go
> > through that central point. I took a brief look at this when I was
> > updating the credential code a few months ago, but didn't consider it a
> > priority, as most people should be using smart http these days. Is there
> > a reason you can't use smart-http? It's significantly more efficient.
> 
> Smart HTTP didn't come up in any of my Google searches.  With that as an
> option, I might just drop this work now.  I'd rather see incomplete methods
> that aren't recommended go away than further facilitate their use, personally.

Me too. I would love it if dumb http push just went away. It's extremely neglected, and has very few advantages over smart http (really, the only advantage is that the server does not need to run git). However, we do get bug reports on it occasionally, so I think people are still using it.

So far our approach has mostly been to prevent any serious regressions, and otherwise not worry too much about dragging it along with new features.

Show 6 quoted lines
> I made the change where I did because I wasn't sure if the push code was
> avoiding using http_request intentionally, and wasn't sure whether new code
> would be written that avoid it as well.
> 
> If that's not the case, then I gather http-push would be better rewritten to
> just use http_request, if anything.

Yes, I think so. It's purely historical and accidental that the http-push code does not use it (and that the 401-handling happened to be put at that layer). The push and fetch sides grew up independently and organically; I think if somebody were writing it now, they would organize http.c quite differently.

> If I decide to continue working on this, I will keep these in mind.  I'm
> pretty sure that if I can get smart HTTP working, there's no reason to even
> bother with this from my perspective, unless you think there's substantial
> value in it.

No, I don't think there's substantial value. If you can move to smart http, you are much better off.

-Peff
Kevin Stange· Jun 5, 2012, 16:41 UTC · re: Jeff King · lore

Re: [RFC] Deal with HTTP 401 by requesting credentials.

On 06/05/2012 11:28 AM, Jeff King wrote:
Show 22 quoted lines
> On Fri, Jun 01, 2012 at 12:02:08PM -0500, Kevin Stange wrote:
> 
>>> The dumb-http push code is the only thing that does not go through
>>> http_request these days. So another option would be to refactor it to go
>>> through that central point. I took a brief look at this when I was
>>> updating the credential code a few months ago, but didn't consider it a
>>> priority, as most people should be using smart http these days. Is there
>>> a reason you can't use smart-http? It's significantly more efficient.
>>
>> Smart HTTP didn't come up in any of my Google searches.  With that as an
>> option, I might just drop this work now.  I'd rather see incomplete methods
>> that aren't recommended go away than further facilitate their use, personally.
> 
> Me too. I would love it if dumb http push just went away. It's extremely
> neglected, and has very few advantages over smart http (really, the only
> advantage is that the server does not need to run git). However, we do
> get bug reports on it occasionally, so I think people are still using
> it.
> 
> So far our approach has mostly been to prevent any serious regressions,
> and otherwise not worry too much about dragging it along with new
> features.

It seemed a lot of "guides" that I found while searching suggested using the "dumb" system. That may be why it remains in use. I can't imagine too many people have issues running a CGI to ensure everything works right.

Show 7 quoted lines
>> If I decide to continue working on this, I will keep these in mind.  I'm
>> pretty sure that if I can get smart HTTP working, there's no reason to even
>> bother with this from my perspective, unless you think there's substantial
>> value in it.
> 
> No, I don't think there's substantial value. If you can move to smart
> http, you are much better off.

I've gotten smart HTTP working perfectly. Thank you for pointing me to it. I am considering this resolved.

-- 
Kevin Stange
Chief Technology Officer
Steadfast Networks
http://steadfast.net
Phone: 312-602-2689 ext. 203 | Fax: 312-602-2688 | Cell: 312-320-5867

← back to recent threads