{"thread":{"id":"1790","subject":"[PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","startedAt":"2005-09-13T15:38:58Z","lastAt":"2005-09-20T11:34:22Z","messageCount":9,"participants":["Sergey Vlasov","Junio C Hamano","sf","David Lang","Petr Baudis"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"8461","messageId":"20050913153858.GB24405@master.mivlgu.local","threadId":"1790","inReplyTo":null,"subject":"[PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-09-13T15:38:58Z","receivedAt":"2005-09-13T15:38:58Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"By default the curl library adds \"Pragma: no-cache\" header to all\nrequests, which disables caching by proxy servers. However, most\nfiles in a GIT repository are immutable, and caching them is safe and\ncould be useful.\n\nThis patch removes the \"Pragma: no-cache\" header from requests for all\nfiles except the pack list (objects/info/packs) and references\n(refs/*), which are really mutable and should not be cached.\n\nSigned-off-by: Sergey Vlasov \n\n\n---\n\n http-fetch.c | 8 ++++++++\n 1 files changed, 8 insertions(+), 0 deletions(-)\n\nfd3f39120636d1ef8834845aa80475c1664a3a3e\ndiff --git a/http-fetch.c b/http-fetch.c\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -14,6 +14,7 @@\n #endif\n \n static CURL *curl;\n+static struct curl_slist *no_pragma_header;\n \n static char *base;\n \n@@ -102,6 +103,7 @@ static int fetch_index(unsigned char *sh\n \tcurl_easy_setopt(curl, CURLOPT_FILE, indexfile);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \t\n \tif (curl_easy_perform(curl)) {\n \t\tfclose(indexfile);\n@@ -152,6 +154,7 @@ static int fetch_indices(void)\n \tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);\n \t\n \tif (curl_easy_perform(curl)) {\n \t\treturn error(\"Unable to get pack index %s\", url);\n@@ -215,6 +218,7 @@ static int fetch_pack(unsigned char *sha\n \tcurl_easy_setopt(curl, CURLOPT_FILE, packfile);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);\n \tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \t\n \tif (curl_easy_perform(curl)) {\n \t\tfclose(packfile);\n@@ -255,6 +259,7 @@ int fetch(unsigned char *sha1)\n \tcurl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);\n \tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);\n \n \turl = xmalloc(strlen(base) + 50);\n \tstrcpy(url, base);\n@@ -303,6 +308,7 @@ int fetch_ref(char *ref, unsigned char *\n \n curl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);\n \n url = xmalloc(strlen(base) + 6 + strlen(ref));\n strcpy(url, base);\n@@ -354,6 +360,7 @@ int main(int argc, char **argv)\n \tcurl_global_init(CURL_GLOBAL_ALL);\n \n \tcurl = curl_easy_init();\n+\tno_pragma_header = curl_slist_append(no_pragma_header, \"Pragma:\");\n \n \tcurl_ssl_verify = getenv(\"GIT_SSL_NO_VERIFY\") ? 0 : 1;\n \tcurl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);\n@@ -366,6 +373,7 @@ int main(int argc, char **argv)\n \tif (pull(commit_id))\n \t\treturn 1;\n \n+\tcurl_slist_free_all(no_pragma_header);\n \tcurl_global_cleanup();\n \treturn 0;\n }\n"},{"id":"8463","messageId":"7vfys93qn5.fsf@assigned-by-dhcp.cox.net","threadId":"1790","inReplyTo":"20050913153858.GB24405@master.mivlgu.local","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T15:59:26Z","receivedAt":"2005-09-13T15:59:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sergey Vlasov writes:\n\n> This patch removes the \"Pragma: no-cache\" header from requests for all\n> files except the pack list (objects/info/packs) and references\n> (refs/*), which are really mutable and should not be cached.\n\nThanks. What the patch does looks reasonable.\n\nDo you know if we can use it for any reasonably recent version\nof curl? I seem to recall we already do things slightly\ndifferently depending on LIBCURL_VERSION_NUM.\n"},{"id":"8518","messageId":"20050914131236.GH24405@master.mivlgu.local","threadId":"1790","inReplyTo":"7vfys93qn5.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","fromName":"Sergey Vlasov","fromEmail":"vsu@altlinux.ru","sentAt":"2005-09-14T13:12:36Z","receivedAt":"2005-09-14T13:12:36Z","isPatch":true,"sender":{"key":"vsu@altlinux.ru","avatar":"https://avatars.githubusercontent.com/u/616082?v=4"},"body":"On Tue, Sep 13, 2005 at 08:59:26AM -0700, Junio C Hamano wrote:\n> Sergey Vlasov writes:\n> \n> > This patch removes the \"Pragma: no-cache\" header from requests for all\n> > files except the pack list (objects/info/packs) and references\n> > (refs/*), which are really mutable and should not be cached.\n> \n> Thanks. What the patch does looks reasonable.\n> \n> Do you know if we can use it for any reasonably recent version\n> of curl? I seem to recall we already do things slightly\n> differently depending on LIBCURL_VERSION_NUM.\n\nhttp://cool.haxx.se/cvs.cgi/curl/include/curl/curl.h?rev=1.1&content-type=text/vnd.viewcvs-markup\nshows that CURLOPT_HTTPHEADER, curl_slist_append() and\ncurl_slist_free_all() were available in Dec 1999, curl 6.3.1. The FAQ\nentry about disabling \"Pragma: no-cache\" is from Aug 2, 2000.\n"},{"id":"8532","messageId":"7vu0gnobq7.fsf@assigned-by-dhcp.cox.net","threadId":"1790","inReplyTo":"20050914131236.GH24405@master.mivlgu.local","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-14T16:28:16Z","receivedAt":"2005-09-14T16:28:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sergey Vlasov writes:\n\n>> Do you know if we can use it for any reasonably recent version\n>> of curl? I seem to recall we already do things slightly\n>> differently depending on LIBCURL_VERSION_NUM.\n>\n> http://cool.haxx.se/cvs.cgi/curl/include/curl/curl.h?rev=1.1&content-type=text/vnd.viewcvs-markup\n> shows...\n\nCool. I'll not worry about version dependency for this one,\nthen.\n\nThanks for the pointer -- I really appreciate it when people\nteach others how to find out what was asked themselves next\ntime need arises.\n"},{"id":"8803","messageId":"43285B24.7050907@b-i-t.de","threadId":"1790","inReplyTo":"20050913153858.GB24405@master.mivlgu.local","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects by proxy servers","fromName":"sf","fromEmail":"sf@b-i-t.de","sentAt":"2005-09-14T17:17:24Z","receivedAt":"2005-09-14T17:17:24Z","isPatch":true,"sender":{"key":"sf@b-i-t.de","avatar":null},"body":"Sergey Vlasov wrote:\n> By default the curl library adds \"Pragma: no-cache\" header to all\n> requests, which disables caching by proxy servers. However, most\n> files in a GIT repository are immutable, and caching them is safe and\n> could be useful.\n\nIs caching really safe? Because of compression one git object can have \nmany file representations.\n\nRegards\n\tStephan\n"},{"id":"8819","messageId":"Pine.LNX.4.62.0509181720290.14003@qynat.qvtvafvgr.pbz","threadId":"1790","inReplyTo":"43285B24.7050907@b-i-t.de","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects byproxy servers","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2005-09-19T00:23:23Z","receivedAt":"2005-09-19T00:23:23Z","isPatch":true,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Wed, 14 Sep 2005, sf wrote:\n\n> Sergey Vlasov wrote:\n>> By default the curl library adds \"Pragma: no-cache\" header to all\n>> requests, which disables caching by proxy servers. However, most\n>> files in a GIT repository are immutable, and caching them is safe and\n>> could be useful.\n>\n> Is caching really safe? Because of compression one git object can have many \n> file representations.\n\nonly if you use different compression algorithums.\n\nremember that git objects are identified by their sha1, if the sha1 is \nwhat you want (and the file matches the sha1 after you decompress it) then \nit really doesn't matter what it's on-disk representation is.\n\nDavid Lang\n\n-- \nThere are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.\n -- C.A.R. Hoare\n"},{"id":"8849","messageId":"432E91ED.7050605@b-i-t.de","threadId":"1790","inReplyTo":"Pine.LNX.4.62.0509181720290.14003@qynat.qvtvafvgr.pbz","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects byproxy servers","fromName":"sf","fromEmail":"sf@b-i-t.de","sentAt":"2005-09-19T10:24:45Z","receivedAt":"2005-09-19T10:24:45Z","isPatch":true,"sender":{"key":"sf@b-i-t.de","avatar":null},"body":"David Lang wrote:\n> On Wed, 14 Sep 2005, sf wrote:\n> \n>> Sergey Vlasov wrote:\n>>\n>>> By default the curl library adds \"Pragma: no-cache\" header to all\n>>> requests, which disables caching by proxy servers. However, most\n>>> files in a GIT repository are immutable, and caching them is safe and\n>>> could be useful.\n>>\n>>\n>> Is caching really safe? Because of compression one git object can have \n>> many file representations.\n> \n> \n> only if you use different compression algorithums.\n\nEven different implementations and different compression levels can lead \nto different file representations.\n\n> remember that git objects are identified by their sha1, if the sha1 is \n> what you want (and the file matches the sha1 after you decompress it) \n> then it really doesn't matter what it's on-disk representation is.\n\nYou are arguing on the git tool level but we are talking about HTTP \nwhich knows nothing about the uncompressed sha1.\n\nThe OP assumed that \"files in a GIT repository are immutable\" which is \nnot true. If you consider the sequence\n\npack -> prune -> update zlib or git -> unpack\n\nyou can end up with different files if the new zlib implementation \nchanges imcompatibly (with respect to byte-by-byte compression results) \nor if git suddenly does not use the default compression level any more.\n\nAnd surely in the future there will be other git implementations than \nthis one which may not even use zlib.\n\nI do not say that caching is not possible at all but HTTP caching has \nits pitfalls so just be careful.\n\nRegards\n\n\tStephan\n"},{"id":"8853","messageId":"20050919133508.GA2903@pasky.or.cz","threadId":"1790","inReplyTo":"432E91ED.7050605@b-i-t.de","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects byproxy servers","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2005-09-19T13:35:08Z","receivedAt":"2005-09-19T13:35:08Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Sep 19, 2005 at 12:24:45PM CEST, I got a letter\nwhere sf told me that...\n> >remember that git objects are identified by their sha1, if the sha1 is \n> >what you want (and the file matches the sha1 after you decompress it) \n> >then it really doesn't matter what it's on-disk representation is.\n> \n> You are arguing on the git tool level but we are talking about HTTP \n> which knows nothing about the uncompressed sha1.\n> \n> The OP assumed that \"files in a GIT repository are immutable\" which is \n> not true. If you consider the sequence\n> \n> pack -> prune -> update zlib or git -> unpack\n> \n> you can end up with different files if the new zlib implementation \n> changes imcompatibly (with respect to byte-by-byte compression results) \n> or if git suddenly does not use the default compression level any more.\n\nYes, but why should this matter? It shouldn't matter if you get the old\n\"version\" or the new version of the file over HTTP, the actual object's\ncontents is still the same, and GIT shouldn't care.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nIf you want the holes in your knowledge showing up try teaching\nsomeone. -- Alan Cox\n"},{"id":"8985","messageId":"432FF3BE.8060501@b-i-t.de","threadId":"1790","inReplyTo":"20050919133508.GA2903@pasky.or.cz","subject":"Re: [PATCH] git-http-fetch: Allow caching of retrieved objects byproxy servers","fromName":"sf","fromEmail":"sf@b-i-t.de","sentAt":"2005-09-20T11:34:22Z","receivedAt":"2005-09-20T11:34:22Z","isPatch":true,"sender":{"key":"sf@b-i-t.de","avatar":null},"body":"Petr Baudis wrote:\n> Dear diary, on Mon, Sep 19, 2005 at 12:24:45PM CEST, I got a letter\n> where sf told me that...\n...\n>> The OP assumed that \"files in a GIT repository are immutable\" which is \n>> not true. If you consider the sequence\n>> \n>> pack -> prune -> update zlib or git -> unpack\n>> \n>> you can end up with different files if the new zlib implementation \n>> changes imcompatibly (with respect to byte-by-byte compression results) \n>> or if git suddenly does not use the default compression level any more.\n> \n> Yes, but why should this matter? It shouldn't matter if you get the old\n> \"version\" or the new version of the file over HTTP, the actual object's\n> contents is still the same, and GIT shouldn't care.\n> \n\nThis is correct as long as you take care to always get each file in one go.\n\nRecently there was talk about how git handles objects larger than 4GB. \nBut you do not have to go this far. Think about fetching 1MB (or 10MB or \n100MB) compressed objects over a slow link. If the transfer gets \ninterrupted some people or some clever piece of software - perhaps even \nin git-core - might try to continue the interrupted download. Now if the \nfile representation has changed in the meantime the downloaded file is \ngoing to be corrupt.\n\nThe git tools will of course take note of the corruption but then the \nhead scratching begins: \"What went wrong?\"\n\nThe more I think about this I realize that my worries have nothing to do \nwith caching but with HTTP fetching in general.\n\nRegards\n\n\tStephan\n"}]}