[PATCH] Enable info/refs gzip decompression in HTTP client

Shawn O. Pearce· Sep 19, 2012, 23:12 UTC · lore

From: "Shawn O. Pearce" <spearce@spearce.org>

Some HTTP servers try to use gzip compression on the /info/refs request to save transfer bandwidth. Repositories with many tags may find the /info/refs request can be gzipped to be 50% of the original size due to the few but often repeated bytes used (hex SHA-1 and commonly digits in tag names).

For most HTTP requests enable "Accept-Encoding: gzip" ensuring the /info/refs payload can use this encoding format.

Disable the Accept-Encoding header on probe RPCs as response bodies are supposed to be exactly 4 bytes long, "0000". The HTTP headers requesting and indicating compression use more space than the data transferred in the body.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 http.c                | 1 +
 remote-curl.c         | 4 ++--
 t/t5551-http-fetch.sh | 3 ++-
 3 files changed, 5 insertions(+), 3 deletions(-)

Show changes to 3 files +5 −3

http.c, remote-curl.c, t/t5551-http-fetch.sh

diff --git a/http.c b/http.c
index 9bac1d8..345c171 100644
--- a/http.c
+++ b/http.c
@@ -818,6 +818,7 @@ static int http_request(const char *url, void *result, int target, int options)
 
 	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
diff --git a/remote-curl.c b/remote-curl.c
index 3ec474f..4a0927e 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -393,7 +393,7 @@ static int probe_rpc(struct rpc_state *rpc)
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);
-	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, NULL);
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, "0000");
 	curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, 4);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
@@ -449,7 +449,7 @@ static int post_rpc(struct rpc_state *rpc)
 	curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
 	curl_easy_setopt(slot->curl, CURLOPT_POST, 1);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, rpc->service_url);
-	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "");
+	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
 
 	headers = curl_slist_append(headers, rpc->hdr_content_type);
 	headers = curl_slist_append(headers, rpc->hdr_accept);
diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
index 2db5c35..380c175 100755
--- a/t/t5551-http-fetch.sh
+++ b/t/t5551-http-fetch.sh
@@ -32,13 +32,14 @@ setup_askpass_helper
 cat >exp <<EOF
 > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
 > Accept: */*
+> Accept-Encoding: gzip
 > Pragma: no-cache
 < HTTP/1.1 200 OK
 < Pragma: no-cache
 < Cache-Control: no-cache, max-age=0, must-revalidate
 < Content-Type: application/x-git-upload-pack-advertisement
 > POST /smart/repo.git/git-upload-pack HTTP/1.1
-> Accept-Encoding: deflate, gzip
+> Accept-Encoding: gzip
 > Content-Type: application/x-git-upload-pack-request
 > Accept: application/x-git-upload-pack-result
 > Content-Length: xxx

-- 
1.7.12.1.510.g5dd77d8

Junio C Hamano· Sep 20, 2012, 00:43 UTC · re: Shawn O. Pearce · lore

Re: [PATCH] Enable info/refs gzip decompression in HTTP client

"Shawn O. Pearce" <spearce@spearce.org> writes:

Show 15 quoted lines

> From: "Shawn O. Pearce" <spearce@spearce.org>
>
> Some HTTP servers try to use gzip compression on the /info/refs
> request to save transfer bandwidth. Repositories with many tags
> may find the /info/refs request can be gzipped to be 50% of the
> original size due to the few but often repeated bytes used (hex
> SHA-1 and commonly digits in tag names).
>
> For most HTTP requests enable "Accept-Encoding: gzip" ensuring
> the /info/refs payload can use this encoding format.
>
> Disable the Accept-Encoding header on probe RPCs as response bodies
> are supposed to be exactly 4 bytes long, "0000". The HTTP headers
> requesting and indicating compression use more space than the data
> transferred in the body.

All of the above sounds very convincing, but ...

Show 17 quoted lines

> diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
> index 2db5c35..380c175 100755
> --- a/t/t5551-http-fetch.sh
> +++ b/t/t5551-http-fetch.sh
> @@ -32,13 +32,14 @@ setup_askpass_helper
>  cat >exp <<EOF
>  > GET /smart/repo.git/info/refs?service=git-upload-pack HTTP/1.1
>  > Accept: */*
> +> Accept-Encoding: gzip
>  > Pragma: no-cache
>  < HTTP/1.1 200 OK
>  < Pragma: no-cache
>  < Cache-Control: no-cache, max-age=0, must-revalidate
>  < Content-Type: application/x-git-upload-pack-advertisement
>  > POST /smart/repo.git/git-upload-pack HTTP/1.1
> -> Accept-Encoding: deflate, gzip
> +> Accept-Encoding: gzip

... was loss of "deflate" intended? If so why? Could you explain it in the log message?

>  > Content-Type: application/x-git-upload-pack-request
>  > Accept: application/x-git-upload-pack-result
>  > Content-Length: xxx

Shawn Pearce· Sep 20, 2012, 02:59 UTC · re: Junio C Hamano · lore

Re: [PATCH] Enable info/refs gzip decompression in HTTP client

On Wed, Sep 19, 2012 at 5:43 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 5 quoted lines

>> -> Accept-Encoding: deflate, gzip
>> +> Accept-Encoding: gzip
>
> ... was loss of "deflate" intended?  If so why?  Could you explain
> it in the log message?

Yes. I would add the following to the end of the commit message as a new paragraph, please amend this for me:

--8<-- Only request gzip encoding from servers. Although deflate is supported by libcurl, most servers have standardized on gzip encoding for compression as that is what most browsers support. Asking for deflate increases request sizes by a few bytes, but is unlikely to ever be used by a server. -->8--

patchEnable info/refs gzip decompression in HTTP client