{"thread":{"id":"11433","subject":"git over webdav: what can I do for improving http-push ?","startedAt":"2007-12-30T22:59:15Z","lastAt":"2008-01-04T19:59:11Z","messageCount":14,"participants":["Grégoire Barbier","Daniel Barkalow","Graham Barr","Jan Hudec","Jakub Narebski","Linus Torvalds","Martin Langhoff"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"64242","messageId":"477822C3.9060002@gbarbier.org","threadId":"11433","inReplyTo":null,"subject":"git over webdav: what can I do for improving http-push ?","fromName":"Grégoire Barbier","fromEmail":"gb@gbarbier.org","sentAt":"2007-12-30T22:59:15Z","receivedAt":"2007-12-30T22:59:15Z","isPatch":false,"sender":{"key":"gb@gbarbier.org","avatar":null},"body":"Hi everybody.\n\nI've just subscribed to the list, therefore I think it should be rude \nnot to introduce myself. My name is Grégoire Barbier (non \nfrench-speaking people should call me Greg and don't bother with \nnon-ascii characters), I'm working mainly as a consultant (using \nPowerpoint and wearing a tie) but have some personal and professional \ninterest in programming, especially about middlewares. BTW I apologize \nfor my poor english.\n\nI'm using Git for a rather short time but enough to fall in love with \nit. For a few days I'm trying to use it over webdav, that is over \nhttp/https with write (push) access. As for me, the main rationale to \nuse http(s) rather than git or ssh is to get through corporate \nfirewalls, otherwise I would probably not bother with webdav.\n\nWith 1.5.3.6 and 1.5.4-rc2, I encounter severe issues that make me think \nthat http-push is not totally ready for production. That's why I would \nlike to have a discussion with some of you that use and maintain it, to \nsee what I can do to improve it or to help you improve it.\n\nHere are some issues I encountered:\n- http-push does not release locks when failing due to syntax error \n(e.g. if one types \"git push\" instead of \"git push origin master\")\n- http-push freezes with no message with urls not terminated by a slash\n- http-push does not create directory for the object (objects/xx/) and \nif the directory exists, it does not actually push objects without \nhaving USE_CURL_MULTI defined (which is not the compilation default)\n\nI've starting to look at the source code and make some little \nimprovements, but I feel that I should rather discuss with you to \nunderstand why there are two rather independant modes in http-push \n(USE_CURL_MULTI or not) and what is the real target (I don't want to \nwork twice, neither to mess up the work of someone else that would be \ncurrently reorginzing this part of the code).\n\nI attached some patches I did against 1.5.4-rc2, but I'm not sure they \nare doing it the good way, so I wouldn't be suprised if you were not \nokay to apply them as is.\n\n-- \nGrégoire Barbier - gb à gbarbier.org - +33 6 21 35 73 49\n\n\n>From 216f27db1f768fd80519a4e57dd835a1f497d902 Mon Sep 17 00:00:00 2001\nFrom: Gregoire Barbier, gb at gbarbier dot org \nDate: Sun, 30 Dec 2007 17:45:54 +0100\nSubject: [PATCH] Removed double-free() int http-push.c.\n\n---\n http-push.c | 2 --\n 1 files changed, 0 insertions(+), 2 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 64be904..55d0c94 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1979,7 +1979,6 @@ static int remote_exists(const char *path)\n \n \tif (start_active_slot(slot)) {\n \t\trun_active_slot(slot);\n-\t\tfree(url);\n \t\tif (results.http_code == 404)\n \t\t\tret = 0;\n \t\telse if (results.curl_result == CURLE_OK)\n@@ -1987,7 +1986,6 @@ static int remote_exists(const char *path)\n \t\telse\n \t\t\tfprintf(stderr, \"HEAD HTTP error %ld\\n\", results.http_code);\n \t} else {\n-\t\tfree(url);\n \t\tfprintf(stderr, \"Unable to start HEAD request\\n\");\n \t}\n \n-- \n1.5.4.rc2.4.gcef60-dirty\n\n\n\n>From 70226905d8f1dd6ed7d953285a6ee693f1e87b65 Mon Sep 17 00:00:00 2001\nFrom: Gregoire Barbier, gb at gbarbier dot org \nDate: Sun, 30 Dec 2007 17:48:07 +0100\nSubject: [PATCH] Making HTTP push more robust and more user-friendly: - fail when info/refs exists and is already locked (avoiding some repository corruption) - warn if the URL does not end with '/' (since 302 is not yet handled) - more explicit error message when the URL or password is not set correctly (instead of \"no DAV locking support\") - DAV locking time of 1 minute instead of 10 minutes (avoid waiting 10 minutes for a orphan lock to expire)\n\n---\n http-push.c | 17 ++++++++++++++++-\n http.c | 25 +++++++++++++++++++++++++\n http.h | 1 +\n 3 files changed, 42 insertions(+), 1 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex 55d0c94..c005903 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -57,7 +57,7 @@ enum XML_Status {\n #define PROPFIND_ALL_REQUEST \"\\n\\n\\n\"\n #define LOCK_REQUEST \"\\n\\n\\n\\n\\nmailto:%s\\n\\n\"\n \n-#define LOCK_TIME 600\n+#define LOCK_TIME 60\n #define LOCK_REFRESH 30\n \n /* bits #0-15 in revision.h */\n@@ -2224,6 +2224,16 @@ int main(int argc, char **argv)\n \n \tno_pragma_header = curl_slist_append(no_pragma_header, \"Pragma:\");\n \n+\t/* Verify connexion string (agains bad URLs or password errors) */\n+\tif (remote->url && remote->url[strlen(remote->url)-1] != '/') {\n+\t\tfprintf(stderr, \"Warning: remote URL does not end with a '/' which often leads to problems\\n\");\n+\t}\n+\tif (!http_test_connection(remote->url)) {\n+\t\tfprintf(stderr, \"Error: cannot access to remote URL (maybe malformed URL, network error or bad credentials)\\n\");\n+\t\trc = 1;\n+\t\tgoto cleanup;\n+\t}\n+\n \t/* Verify DAV compliance/lock support */\n \tif (!locking_available()) {\n \t\tfprintf(stderr, \"Error: no DAV locking support on remote repo %s\\n\", remote->url);\n@@ -2239,6 +2249,11 @@ int main(int argc, char **argv)\n \t\tinfo_ref_lock = lock_remote(\"info/refs\", LOCK_TIME);\n \t\tif (info_ref_lock)\n \t\t\tremote->can_update_info_refs = 1;\n+\t\telse {\n+\t\t\tfprintf(stderr, \"Error: cannot lock existing info/refs\\n\");\n+\t\t\trc = 1;\n+\t\t\tgoto cleanup;\n+\t\t}\n \t}\n \tif (remote->has_info_packs)\n \t\tfetch_indices();\ndiff --git a/http.c b/http.c\nindex d2c11ae..8b04ae9 100644\n--- a/http.c\n+++ b/http.c\n@@ -634,3 +634,28 @@ int http_fetch_ref(const char *base, const char *ref, unsigned char *sha1)\n \tfree(url);\n \treturn ret;\n }\n+\n+int http_test_connection(const char *url)\n+{\n+\tstruct strbuf buffer = STRBUF_INIT;\n+\tstruct active_request_slot *slot;\n+\tstruct slot_results results;\n+\tint ret = 0;\n+\n+\tslot = get_active_slot();\n+\tslot->results = &results;\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\tif (start_active_slot(slot)) {\n+\t\trun_active_slot(slot);\n+\t\tif (results.curl_result == CURLE_OK)\n+\t\t\tret = -1;\n+\t\telse\n+\t\t\terror(\"Cannot access to URL %s, return code %d\", url, results.curl_result);\n+\t} else\n+\t\terror(\"Unable to start request\");\n+\tstrbuf_release(&buffer);\n+\treturn ret;\n+}\ndiff --git a/http.h b/http.h\nindex aeba930..b353007 100644\n--- a/http.h\n+++ b/http.h\n@@ -77,6 +77,7 @@ extern void step_active_slots(void);\n \n extern void http_init(void);\n extern void http_cleanup(void);\n+extern int http_test_connection(const char *url);\n \n extern int data_received;\n extern int active_requests;\n-- \n1.5.4.rc2.4.gcef60-dirty\n\n\n\n>From e00ae0f4b9ed0e61088fa729a7cabbfcbd006b98 Mon Sep 17 00:00:00 2001\nFrom: Gregoire Barbier, gb at gbarbier dot org \nDate: Sun, 30 Dec 2007 19:35:31 +0100\nSubject: [PATCH] Releasing webdav lock even if push fails because of bad (or no) reference on command line.\n\n---\n http-push.c | 13 ++++++++-----\n 1 files changed, 8 insertions(+), 5 deletions(-)\n\ndiff --git a/http-push.c b/http-push.c\nindex c005903..cbbf432 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -2275,11 +2275,14 @@ int main(int argc, char **argv)\n \tif (!remote_tail)\n \t\tremote_tail = &remote_refs;\n \tif (match_refs(local_refs, remote_refs, &remote_tail,\n-\t\t nr_refspec, (const char **) refspec, push_all))\n-\t\treturn -1;\n+\t\t nr_refspec, (const char **) refspec, push_all)) {\n+\t\trc = -1;\n+\t\tgoto cleanup;\n+\t}\n \tif (!remote_refs) {\n \t\tfprintf(stderr, \"No refs in common and none specified; doing nothing.\\n\");\n-\t\treturn 0;\n+\t\trc = 0;\n+\t\tgoto cleanup;\n \t}\n \n \tnew_refs = 0;\n@@ -2410,10 +2413,10 @@ int main(int argc, char **argv)\n \t\t\tfprintf(stderr, \"Unable to update server info\\n\");\n \t\t}\n \t}\n-\tif (info_ref_lock)\n-\t\tunlock_remote(info_ref_lock);\n \n cleanup:\n+\tif (info_ref_lock)\n+\t\tunlock_remote(info_ref_lock);\n \tfree(remote);\n \n \tcurl_slist_free_all(no_pragma_header);\n-- \n1.5.4.rc2.4.gcef60-dirty\n\n\n\n>From cef60c7940008487547894855eeed34d2edeb48e Mon Sep 17 00:00:00 2001\nFrom: Gregoire Barbier, gb at gbarbier dot org \nDate: Sun, 30 Dec 2007 21:30:25 +0100\nSubject: [PATCH] Adding #define DEFAULT_MAX_REQUESTS for USE_CURL_MULTI mode\n\n---\n http.c | 1 +\n 1 files changed, 1 insertions(+), 0 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 8b04ae9..7b1bcb8 100644\n--- a/http.c\n+++ b/http.c\n@@ -4,6 +4,7 @@ int data_received;\n int active_requests = 0;\n \n #ifdef USE_CURL_MULTI\n+#define DEFAULT_MAX_REQUESTS 4\n static int max_requests = -1;\n static CURLM *curlm;\n #endif\n-- \n1.5.4.rc2.4.gcef60-dirty\n\n\n\n>From b34d81c1fff43a806cb91615effd00e424bd0e6b Mon Sep 17 00:00:00 2001\nFrom: Gregoire Barbier, gb at gbarbier dot org \nDate: Sun, 30 Dec 2007 23:40:18 +0100\nSubject: [PATCH] Moving #ifdef/#endif for making USE_CURL_MULTI code more visible.\n\n---\n http-push.c | 15 +++++++++++----\n 1 files changed, 11 insertions(+), 4 deletions(-)\n mode change 100644 => 100755 http-push.c\n\ndiff --git a/http-push.c b/http-push.c\nold mode 100644\nnew mode 100755\nindex cbbf432..6dd3c45\n--- a/http-push.c\n+++ b/http-push.c\n@@ -116,10 +116,12 @@ struct transfer_request\n \tstruct remote_lock *lock;\n \tstruct curl_slist *headers;\n \tstruct buffer buffer;\n+#ifdef USE_CURL_MULTI\n \tchar filename[PATH_MAX];\n \tchar tmpfile[PATH_MAX];\n \tint local_fileno;\n \tFILE *local_stream;\n+#endif\n \tenum transfer_state state;\n \tCURLcode curl_result;\n \tchar errorstr[CURL_ERROR_SIZE];\n@@ -175,6 +177,7 @@ struct remote_ls_ctx\n \tstruct remote_ls_ctx *parent;\n };\n \n+#ifdef USE_CURL_MULTI\n static void finish_request(struct transfer_request *request);\n static void release_request(struct transfer_request *request);\n \n@@ -186,7 +189,6 @@ static void process_response(void *callback_data)\n \tfinish_request(request);\n }\n \n-#ifdef USE_CURL_MULTI\n static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n \t\t\t void *data)\n {\n@@ -383,7 +385,6 @@ static void start_mkcol(struct transfer_request *request)\n \t\trequest->url = NULL;\n \t}\n }\n-#endif\n \n static void start_fetch_packed(struct transfer_request *request)\n {\n@@ -581,6 +582,7 @@ static void start_move(struct transfer_request *request)\n \t\trequest->url = NULL;\n \t}\n }\n+#endif\n \n static int refresh_lock(struct remote_lock *lock)\n {\n@@ -660,15 +662,18 @@ static void release_request(struct transfer_request *request)\n \t\t\tentry->next = entry->next->next;\n \t}\n \n+#ifdef USE_CURL_MULTI\n \tif (request->local_fileno != -1)\n \t\tclose(request->local_fileno);\n \tif (request->local_stream)\n \t\tfclose(request->local_stream);\n+#endif\n \tif (request->url != NULL)\n \t\tfree(request->url);\n \tfree(request);\n }\n \n+#ifdef USE_CURL_MULTI\n static void finish_request(struct transfer_request *request)\n {\n \tstruct stat st;\n@@ -793,7 +798,6 @@ static void finish_request(struct transfer_request *request)\n \t}\n }\n \n-#ifdef USE_CURL_MULTI\n static int fill_active_slot(void *unused)\n {\n \tstruct transfer_request *request = request_queue_head;\n@@ -841,8 +845,10 @@ static void add_fetch_request(struct object *obj)\n \trequest->url = NULL;\n \trequest->lock = NULL;\n \trequest->headers = NULL;\n+#ifdef USE_CURL_MULTI\n \trequest->local_fileno = -1;\n \trequest->local_stream = NULL;\n+#endif\n \trequest->state = NEED_FETCH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\n@@ -881,12 +887,13 @@ static int add_send_request(struct object *obj, struct remote_lock *lock)\n \trequest->url = NULL;\n \trequest->lock = lock;\n \trequest->headers = NULL;\n+#ifdef USE_CURL_MULTI\n \trequest->local_fileno = -1;\n \trequest->local_stream = NULL;\n+#endif\n \trequest->state = NEED_PUSH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\n-\n #ifdef USE_CURL_MULTI\n \tfill_active_slots();\n \tstep_active_slots();\n-- \n1.5.4.rc2.4.gcef60-dirty\n\n"},{"id":"64244","messageId":"alpine.LNX.1.00.0712302145500.13593@iabervon.org","threadId":"11433","inReplyTo":"477822C3.9060002@gbarbier.org","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2007-12-31T03:46:22Z","receivedAt":"2007-12-31T03:46:22Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 30 Dec 2007, Grégoire Barbier wrote:\n\n> Hi everybody.\n> \n> I've just subscribed to the list, therefore I think it should be rude not to\n> introduce myself. My name is Grégoire Barbier (non french-speaking people\n> should call me Greg and don't bother with non-ascii characters), I'm working\n> mainly as a consultant (using Powerpoint and wearing a tie) but have some\n> personal and professional interest in programming, especially about\n> middlewares. BTW I apologize for my poor english.\n> \n> I'm using Git for a rather short time but enough to fall in love with it. For\n> a few days I'm trying to use it over webdav, that is over http/https with\n> write (push) access. As for me, the main rationale to use http(s) rather than\n> git or ssh is to get through corporate firewalls, otherwise I would probably\n> not bother with webdav.\n\nIn general, we've been able to either get through firewalls with ssh or \nit's all in the same VPN. So it's kind of unloved at this point. People \npoke at it occasionally, but mostly in the context of other fixes, I \nthink.\n\n> With 1.5.3.6 and 1.5.4-rc2, I encounter severe issues that make me think that\n> http-push is not totally ready for production. That's why I would like to have\n> a discussion with some of you that use and maintain it, to see what I can do\n> to improve it or to help you improve it.\n> \n> Here are some issues I encountered:\n> - http-push does not release locks when failing due to syntax error (e.g. if\n> one types \"git push\" instead of \"git push origin master\")\n> - http-push freezes with no message with urls not terminated by a slash\n> - http-push does not create directory for the object (objects/xx/) and if the\n> directory exists, it does not actually push objects without having\n> USE_CURL_MULTI defined (which is not the compilation default)\n> \n> I've starting to look at the source code and make some little improvements,\n> but I feel that I should rather discuss with you to understand why there are\n> two rather independant modes in http-push (USE_CURL_MULTI or not) and what is\n> the real target (I don't want to work twice, neither to mess up the work of\n> someone else that would be currently reorginzing this part of the code).\n\nI think the issue is the CURL_MULTI library code is either not supported \nor is broken in versions of curl that many distros still ship, but we can \ndo a lot better with it, so the duplicate implementation is plausibly \nworthwhile.\n\nOne thing I personally thought would be worthwhile would be to separate \nout the logic for sending stuff like the fetching logic is in \nwalker.{c,h}, and include the necessary methods in struct walker. There \nwere people interested in sftp (for the case where you can get ssh through \nfirewalls, but you aren't allowed to install programs on the file server \nand git isn't installed system-wide).\n\nOne thing that's worth doing when looking at the code is using \"git blame\" \nto find out where the lines you're changing came from, and \"git log \n\" to find out what the person writing them was trying to do. This \nwill also turn up the people who've been working in the area, who you \nmight want to cc, since they'll be good reviewers.\n\n\t-Daniel\n*This .sig left intentionally blank*"},{"id":"64267","messageId":"47791F90.8030302@pobox.com","threadId":"11433","inReplyTo":"alpine.LNX.1.00.0712302145500.13593@iabervon.org","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Graham Barr","fromEmail":"gbarr@pobox.com","sentAt":"2007-12-31T16:57:52Z","receivedAt":"2007-12-31T16:57:52Z","isPatch":false,"sender":{"key":"gbarr@pobox.com","avatar":null},"body":"Daniel Barkalow wrote:\n> On Sun, 30 Dec 2007, Grégoire Barbier wrote:\n\n>> I'm using Git for a rather short time but enough to fall in love with it. For\n>> a few days I'm trying to use it over webdav, that is over http/https with\n>> write (push) access. As for me, the main rationale to use http(s) rather than\n>> git or ssh is to get through corporate firewalls, otherwise I would probably\n>> not bother with webdav.\n\n> In general, we've been able to either get through firewalls with ssh or \n> it's all in the same VPN. So it's kind of unloved at this point. People \n> poke at it occasionally, but mostly in the context of other fixes, I \n> think.\n\nIf you have a http proxy that you can use, the you can use ssh via that with\nsomething like corkscrew. http://wiki.kartbuilding.net/index.php/Corkscrew_-_ssh_over_https\n\nA simple shell script wrapper around ssh to detect when you are behind a firewall\ncan inject the ProxyCommand into the command line arguments with -o\n\nGraham.\n"},{"id":"64295","messageId":"20080101113301.GC9214@efreet.light.src","threadId":"11433","inReplyTo":"47791F90.8030302@pobox.com","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-01-01T11:33:01Z","receivedAt":"2008-01-01T11:33:01Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Mon, Dec 31, 2007 at 10:57:52 -0600, Graham Barr wrote:\n> Daniel Barkalow wrote:\n>> On Sun, 30 Dec 2007, Grégoire Barbier wrote:\n>\n>>> I'm using Git for a rather short time but enough to fall in love with it. For\n>>> a few days I'm trying to use it over webdav, that is over http/https with\n>>> write (push) access. As for me, the main rationale to use http(s) rather than\n>>> git or ssh is to get through corporate firewalls, otherwise I would probably\n>>> not bother with webdav.\n>\n>> In general, we've been able to either get through firewalls with ssh or \n>> it's all in the same VPN. So it's kind of unloved at this point. People \n>> poke at it occasionally, but mostly in the context of other fixes, I \n>> think.\n>\n> If you have a http proxy that you can use, the you can use ssh via that with\n> something like corkscrew. http://wiki.kartbuilding.net/index.php/Corkscrew_-_ssh_over_https\n\nThis, obviously, requires, that ssh is running on port 443, because most HTTP\nproxies won't let you CONNECT anywhere else. I have also heared of a HTTP\nproxy, that will check whether the session inside CONNECT starts with SSL\nhandshake and will break your connection if it does not.\n\n> A simple shell script wrapper around ssh to detect when you are behind a firewall\n> can inject the ProxyCommand into the command line arguments with -o\n\nMost of the time simply setting the parameter in .ssh/config works better\n-- because you are often behind a proxy for some sites only.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"},{"id":"64296","messageId":"477A26FD.7020408@gbarbier.org","threadId":"11433","inReplyTo":"20080101113301.GC9214@efreet.light.src","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Grégoire Barbier","fromEmail":"gb@gbarbier.org","sentAt":"2008-01-01T11:41:49Z","receivedAt":"2008-01-01T11:41:49Z","isPatch":false,"sender":{"key":"gb@gbarbier.org","avatar":null},"body":"Jan Hudec a écrit :\n> On Mon, Dec 31, 2007 at 10:57:52 -0600, Graham Barr wrote:\n> \n>> Daniel Barkalow wrote:\n>> \n>>> On Sun, 30 Dec 2007, Grégoire Barbier wrote:\n>>> \n>>>> As for me, the main rationale to use http(s) rather than\n>>>> git or ssh is to get through corporate firewalls, otherwise I would probably\n>>>> not bother with webdav.\n>>>> \n>>> In general, we've been able to either get through firewalls with ssh or \n>>> it's all in the same VPN. So it's kind of unloved at this point. People \n>>> poke at it occasionally, but mostly in the context of other fixes, I \n>>> think.\n>>> \n>> If you have a http proxy that you can use, the you can use ssh via that with\n>> something like corkscrew. http://wiki.kartbuilding.net/index.php/Corkscrew_-_ssh_over_https\n>> \n>\n> This, obviously, requires, that ssh is running on port 443, because most HTTP\n> proxies won't let you CONNECT anywhere else. I have also heared of a HTTP\n> proxy, that will check whether the session inside CONNECT starts with SSL\n> handshake and will break your connection if it does not.\n> \n\nHello Jan.\n\nI think we have similar experiences. I have personnaly be faced to \nproxies that not only scan for the SSL handshake but do \nman-in-the-middle \"attack\" to break the SSL into two parts, checking for \nHTTP inside it (and probably scanning for viruses and things like hat, I \nthink).\n\nI first replied privatly to Graham because I didn't think it was \ninteresting for the whole list.\nIt was a mistake, here is my answer:\n\nIn fact, I already use this hack where it is possible.\n\nHowever some well advised companies does not allow CONNECT through their HTTP proxy without some limitations that make this tip unusable (for instance: allowing only port 443, allowing only sites of a white-list, forcing a man-in-the-middle that not only breaks the confidentiality but too forbids the use of other protocols such as SSH, even on port 443).\n\nBTW such circumvention of the security facilities is often (at less where I live and with my clients) forbidden in some corporate rules, even when it is technically possible.\nTherefore I'm not allowed to do so and, furthermore, I cannot tell my clients to do so and write documents that tell it's the good way.\n\nI think that real HTTP support is better than all workarounds we will be able to find to get through firewalls (when CONNECT is not available, some awful VPNs that send Etherne over HTTP may work ;-)).\nThat's why I'm ok to work several hours on git code to enhance real HTTP(S) support.\n\n\n-- \nGrégoire Barbier - gb à gbarbier.org - +33 6 21 35 73 49\n"},{"id":"64300","messageId":"m3myrpo1p0.fsf@roke.D-201","threadId":"11433","inReplyTo":"477A26FD.7020408@gbarbier.org","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-01-01T18:12:28Z","receivedAt":"2008-01-01T18:12:28Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Grégoire Barbier writes:\n\n> I think that real HTTP support is better than all workarounds we\n> will be able to find to get through firewalls (when CONNECT is not\n> available, some awful VPNs that send Etherne over HTTP may work\n> ;-)). That's why I'm ok to work several hours on git code to\n> enhance real HTTP(S) support.\n\nThere was also an idea to create a CGI program, or enhance gitweb\nto use for pushing. I don't know if it would be better way to pursue\nto work around corporate firewalls, or not...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"64305","messageId":"20080101202352.GA4295@efreet.light.src","threadId":"11433","inReplyTo":"m3myrpo1p0.fsf@roke.D-201","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-01-01T20:23:52Z","receivedAt":"2008-01-01T20:23:52Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Tue, Jan 01, 2008 at 10:12:28 -0800, Jakub Narebski wrote:\n> Grégoire Barbier writes:\n> \n> > I think that real HTTP support is better than all workarounds we\n> > will be able to find to get through firewalls (when CONNECT is not\n> > available, some awful VPNs that send Etherne over HTTP may work\n> > ;-)). That's why I'm ok to work several hours on git code to\n> > enhance real HTTP(S) support.\n> \n> There was also an idea to create a CGI program, or enhance gitweb\n> to use for pushing. I don't know if it would be better way to pursue\n> to work around corporate firewalls, or not...\n\nIt is what bzr and mercurial do and I think it would be quite good way to go\nfor cases like this. Eg. while our corporate firewall does allow anything\nthrough connect on 443 (so I can use ssh that way), it does *not* support\nweb-dav in non-ssl mode. So I eg. can't even get from public subversion\nrepositories at work.\n\nI have also thought about optimizing download using CGI, but than I thought,\nthat maybe there is a way to statically generate packs so, that if the client\nwants n revisions, the number of revisions it downloads is O(n) and the\nnumber of packs it gets them from (and thus number of round-trips) is\nO(log(n)). Assuming the client always wants everything up to the tip, of\ncourse. Now this is trivial with linear history (pack first half, than half\nof what's left, etc., gives logarithmic number of packs and you always\ndownload at most twice as much as you need), but it would be nice if somebody\nfound a way (even one that satisfies the conditions on average only) to do\nthis with non-linear history, it would be very nice improvement to the http\ndownload -- native git server optimizes amount of data transfered very well,\nbut at the cost of quite heavy CPU load on the server.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"},{"id":"64390","messageId":"477D3401.2010005@gbarbier.org","threadId":"11433","inReplyTo":"20080101202352.GA4295@efreet.light.src","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Grégoire Barbier","fromEmail":"gb@gbarbier.org","sentAt":"2008-01-03T19:14:09Z","receivedAt":"2008-01-03T19:14:09Z","isPatch":false,"sender":{"key":"gb@gbarbier.org","avatar":null},"body":"Jan Hudec a écrit :\n> On Tue, Jan 01, 2008 at 10:12:28 -0800, Jakub Narebski wrote:\n> \n>> Grégoire Barbier writes:\n>>\n>> \n>>> I think that real HTTP support is better than all workarounds we\n>>> will be able to find to get through firewalls (when CONNECT is not\n>>> available, some awful VPNs that send Etherne over HTTP may work\n>>> ;-)). That's why I'm ok to work several hours on git code to\n>>> enhance real HTTP(S) support.\n>>> \n>> There was also an idea to create a CGI program, or enhance gitweb\n>> to use for pushing. I don't know if it would be better way to pursue\n>> to work around corporate firewalls, or not...\n>> \nI subscribe to this point of view.\nI will look at the list archive to search for what has been said before \nabout this.\n>\n> It is what bzr and mercurial do and I think it would be quite good way to go\n> for cases like this.\nOk, I will have to look at bzr and mercurial...\n\n> Eg. while our corporate firewall does allow anything\n> through connect on 443 (so I can use ssh that way), it does *not* support\n> web-dav in non-ssl mode. So I eg. can't even get from public subversion\n> repositories at work.\n>\n> I have also thought about optimizing download using CGI, but than I thought,\n> that maybe there is a way to statically generate packs so, that if the client\n> wants n revisions, the number of revisions it downloads is O(n) and the\n> number of packs it gets them from (and thus number of round-trips) is\n> O(log(n)). Assuming the client always wants everything up to the tip, of\n> course. Now this is trivial with linear history (pack first half, than half\n> of what's left, etc., gives logarithmic number of packs and you always\n> download at most twice as much as you need), but it would be nice if somebody\n> found a way (even one that satisfies the conditions on average only) to do\n> this with non-linear history, it would be very nice improvement to the http\n> download -- native git server optimizes amount of data transfered very well,\n> but at the cost of quite heavy CPU load on the server.\n> \nWell... frankly I don't think I'm able of such things.\nWriting a walker over webdav or a simple cgi is a thing I can do (I \nthink), but I'm not tought enough (or not ready to take the time \nneeded) to have a look on the internals of packing revisions (whereas I \ncan imagine it would means that \"my\" walker would be suitable only for \nsmall projects in terms of code amount and commit frequency).\n\nI had a quick look on bzr and hg, and it seems that bzr use the easy way \n(walker, no optimizations) and hg a cgi (therefore, maybe \noptimizations). By quick look I mean that I sniff the HTTP queries on \nthe network during a clone. I need to look harder...\n\nBTW I never looked at the git:// protocol. Do you think that by \ntunneling the git protocol in a cgi (hg uses URLs of the form \n\"/mycgi?cmd=mycommand&...\", therefore I think \"tunnel\" is not a bad \nword...) the performance would be good?\nMaybe it's not that hard to write a performant HTTP/CGI protocol for Git \nif it's based upon existing code such as the git protocol.\n\n-- \nGrégoire Barbier - gb à gbarbier.org - +33 6 21 35 73 49\n"},{"id":"64394","messageId":"20080103211521.GA4225@efreet.light.src","threadId":"11433","inReplyTo":"477D3401.2010005@gbarbier.org","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-01-03T21:15:21Z","receivedAt":"2008-01-03T21:15:21Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Thu, Jan 03, 2008 at 20:14:09 +0100, Grégoire Barbier wrote:\n> Jan Hudec a écrit :\n[...]\n>> It is what bzr and mercurial do and I think it would be quite good way to go\n>> for cases like this.\n> Ok, I will have to look at bzr and mercurial...\n\nBzr is quite far, design-wise, I fear. The mercurial might be a little more\ninteresting to study, but being in python and internally somewhat\nfile-oriented, I wouldn't think it is of much use.\n\nYou should start with upload, leaving the download direction to the dumb\nmachinery git currently uses.\n\n[...]\n>> I have also thought about optimizing download using CGI, but than I thought,\n>> that maybe there is a way to statically generate packs so, that if the client\n>> wants n revisions, the number of revisions it downloads is O(n) and the\n>> number of packs it gets them from (and thus number of round-trips) is\n>> O(log(n)). Assuming the client always wants everything up to the tip, of\n>> course. Now this is trivial with linear history (pack first half, than half\n>> of what's left, etc., gives logarithmic number of packs and you always\n>> download at most twice as much as you need), but it would be nice if somebody\n>> found a way (even one that satisfies the conditions on average only) to do\n>> this with non-linear history, it would be very nice improvement to the http\n>> download -- native git server optimizes amount of data transfered very well,\n>> but at the cost of quite heavy CPU load on the server.\n>> \n> Well... frankly I don't think I'm able of such things.\n> Writing a walker over webdav or a simple cgi is a thing I can do (I \n> think), but I'm not tought enough (or not ready to take the time needed) \n> to have a look on the internals of packing revisions (whereas I can imagine \n> it would means that \"my\" walker would be suitable only for small projects \n> in terms of code amount and commit frequency).\n\nWell, it does not depend on the walker -- the walker is quite simple and\nalready written anyway.\n\n> I had a quick look on bzr and hg, and it seems that bzr use the easy way \n> (walker, no optimizations)\n\nThat's not quite true -- bzr has both dumb (walker over plain HTTP) and smart\n(CGI) methods. But their CGI is really just tunelling their custom protocol\nover HTTP and that protocol will not be anywhere near what we want for git\nbecause of vastly different design of the storage.\n\n> and hg a cgi (therefore, maybe optimizations). \n> By quick look I mean that I sniff the HTTP queries on the network during a \n> clone. I need to look harder...\n\nYes, mercurial uses a CGI. But I am not sure how similar their approach is to\nanything that would make sense for git, so looking at the details might or\nmight not be useful.\n\n> BTW I never looked at the git:// protocol. Do you think that by tunneling \n> the git protocol in a cgi (hg uses URLs of the form \n> \"/mycgi?cmd=mycommand&...\", therefore I think \"tunnel\" is not a bad \n> word...) the performance would be good?\n\nIt would be pretty hard to tunnel it and it would loose all it's nice\nproperties. The git protocol, for pull, basically works like this:\n\n - server sends a list of it's refs\n - client tells server which ones it wants\n - client starts listing revisions it has, newest to oldest\n - server tells client whenever it finds common ancestor with one of the\n heads desired\n - client restarts the listing from next ref\n - server starts sending the data when client runs out of refs to list\n\nThe main point about the protocol is, that the client is listing the refs, as\nfast as it can and server will stop it when it sees a revision it knows.\nTherefore there will only be one round-trip to discover each common ancestor.\n\nHowever, you can't do this over HTTP, because response won't be started until\nthe request is received. You could be sending a lot of smallish requests and\nquick, often empty, responses to them. However, that will waste a lot of\nbandwidth (because of the HTTP overhead) and loose much of the speed anyway.\nAlso the HTTP protocol is stateless, but this is inherently stateful, so you\nwould have to work that around somehow too. Therefore a different approach is\npreferable on HTTP.\n\nNow to keep it stateless, I thought that:\n - client would first ask for list of refs\n - client would than ask for pack containing the first ref\n - server would respond with pack containing just that commit plus all\n objects that are not referenced by any of it's parents\n - if client does not have it's parent, it would ask for pack containing that\n - since it's second request, server would pack 2 revisions (with necessary\n objects) this time\n - if client still does not have all parents, it would again ask for a pack,\n receiving 4 revisions this time (next 8, 16, etc...)\n\nThis would guarantee, that when you want n revisions, you make at most\nlog2(n) requests and get at most 2*n revisions (well, the requests are for\neach ref separately, so it's more like m*log2(n) where m is number of refs,\nbut still). Additionally, it would be stateless, because the client would\nsimply say 'I want commit abcdef01 and this is my 3rd request' and server\nwould provide that commit and 7 it's parents (ie. 2^3 commits).\n\nNow generating the packs takes it's CPU. The servers like git.kernel.org have\nquite high CPU load. But in this schema, all clients would most of the time\nget the same packs (unlike native git protocol, where the client always gets\nsingle pack with exactly what it needs). So the idea struck me, that they\ncould simply be statically generated and fetched via the existing dumb\nprotocol. It would get the efficiency and save a lot of CPU power, which\nis allows one to serve quite busy git repository from a limited (and\ntherefore cheap) virtual machine or even (yes, I saw such idea on the list)\nserving any repository from NSLU2.\n\nNow to create a packing policy to create such packs, you don't actually need\nto touch any C -- because git-repack is still in shell -- and you don't\nreally need to touch any internals, because you only need to change how the\npack-objects command will be called and leave all the dirty details to that\ncommand.\n\nI would personally not re-split the already generated packs. Only find some\nalgorithm for choosing when packs are deep enough in history so they should\nbe merged together. It would also might not make sense to ever pack unpacked\nobjects to more than one pack -- a dumb-HTTP-served might have a requirement\nof running this kind of repack after every push and clients will rarely have\npart of single push to the server.\n\n> Maybe it's not that hard to write a performant HTTP/CGI protocol for Git if \n> it's based upon existing code such as the git protocol.\n\nFor push it might, or might not be easy. But in the worst case you should be\nable to calculate a pack to upload locally (fetching from the server\nbeforehead if necessary), upload that pack (or bundle) and update all the\nrefs.\n\nFor pull it certainly won't be. You might be able to reimplement the common\nref discovery using some kind of gradually increasing ref list and then\ngenerate a bundle for the server, but optimizing the dumb protocol seems more\nuseful to me. As I said, generating the packs only requires devising a way of\nselecting which objects should go together and git pack-objects will take\ncare of the dirty details of generating the packs and git update-server-info\nwill take care of the dirty details of presenting the list of packs to\nclient.\n\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"},{"id":"64399","messageId":"alpine.LFD.1.00.0801031342200.2811@woody.linux-foundation.org","threadId":"11433","inReplyTo":"20080103211521.GA4225@efreet.light.src","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-01-03T21:43:59Z","receivedAt":"2008-01-03T21:43:59Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 3 Jan 2008, Jan Hudec wrote:\n> \n> That's not quite true -- bzr has both dumb (walker over plain HTTP) and smart\n> (CGI) methods. But their CGI is really just tunelling their custom protocol\n> over HTTP and that protocol will not be anywhere near what we want for git\n> because of vastly different design of the storage.\n\nWell, tunnelling the native git protocol is *exactly* what you'd want to \ndo with some git CGI thing. So no, I don't think the actual stuff you \ntunnel would have any relationship, but the actual code to set up a tunnel \nand making it all look like some fake html sequence might be something \nthat can be used as a base.\n\n\t\tLinus\n"},{"id":"64400","messageId":"200801032247.02323.jnareb@gmail.com","threadId":"11433","inReplyTo":"20080103211521.GA4225@efreet.light.src","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-01-03T21:47:01Z","receivedAt":"2008-01-03T21:47:01Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jan Hudec wrote:\n> On Thu, Jan 03, 2008 at 20:14:09 +0100, Grégoire Barbier wrote:\n>\n> > I had a quick look on bzr and hg, and it seems that bzr use the easy way \n> > (walker, no optimizations)\n> \n> That's not quite true -- bzr has both dumb (walker over plain HTTP) and smart\n> (CGI) methods. But their CGI is really just tunelling their custom protocol\n> over HTTP and that protocol will not be anywhere near what we want for git\n> because of vastly different design of the storage.\n\nPerhaps we could also simply tunnel git protocol over HTTP / HTTPS?\n \n> > and hg a cgi (therefore, maybe optimizations). \n> > By quick look I mean that I sniff the HTTP queries on the network during a \n> > clone. I need to look harder...\n> \n> Yes, mercurial uses a CGI. But I am not sure how similar their approach is to\n> anything that would make sense for git, so looking at the details might or\n> might not be useful.\n> \n> > BTW I never looked at the git:// protocol. Do you think that by tunneling \n> > the git protocol in a cgi (hg uses URLs of the form \n> > \"/mycgi?cmd=mycommand&...\", therefore I think \"tunnel\" is not a bad \n> > word...) the performance would be good?\n> \n> It would be pretty hard to tunnel it and it would loose all it's nice\n> properties. The git protocol, for pull, basically works like this:\n> \n> - server sends a list of it's refs\n> - client tells server which ones it wants\n> - client starts listing revisions it has, newest to oldest\n> - server tells client whenever it finds common ancestor with one of the\n> heads desired\n> - client restarts the listing from next ref\n> - server starts sending the data when client runs out of refs to list\n> \n> The main point about the protocol is, that the client is listing the refs, as\n> fast as it can and server will stop it when it sees a revision it knows.\n> Therefore there will only be one round-trip to discover each common ancestor.\n> \n> However, you can't do this over HTTP, because response won't be started until\n> the request is received. You could be sending a lot of smallish requests and\n> quick, often empty, responses to them. However, that will waste a lot of\n> bandwidth (because of the HTTP overhead) and loose much of the speed anyway.\n> Also the HTTP protocol is stateless, but this is inherently stateful, so you\n> would have to work that around somehow too. Therefore a different approach is\n> preferable on HTTP.\n\nPerhaps we could use AJAX (XMLHttpRequest for communication, plain HTTP or\nIFRAMES for sending data) or something like this for git protocol tunneling?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"64403","messageId":"477D6FD9.20608@gbarbier.org","threadId":"11433","inReplyTo":"200801032247.02323.jnareb@gmail.com","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Grégoire Barbier","fromEmail":"gb@gbarbier.org","sentAt":"2008-01-03T23:29:29Z","receivedAt":"2008-01-03T23:29:29Z","isPatch":false,"sender":{"key":"gb@gbarbier.org","avatar":null},"body":"Jakub Narebski a écrit :\n> Perhaps we could use AJAX (XMLHttpRequest for communication, plain HTTP or\n> IFRAMES for sending data) or something like this for git protocol tunneling?\n> \nwell... I think I may manage to avoid javascript... ;-)\nmore seriously, I was thinking of using http in an automated, \nun-human-browsable manner, not a full html user interface.\n\n-- \nGrégoire Barbier - gb à gbarbier.org - +33 6 21 35 73 49\n"},{"id":"64404","messageId":"46a038f90801031554j6218f08cl6c9608b24e1675f8@mail.gmail.com","threadId":"11433","inReplyTo":"20080103211521.GA4225@efreet.light.src","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2008-01-03T23:54:58Z","receivedAt":"2008-01-03T23:54:58Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Jan 4, 2008 10:15 AM, Jan Hudec wrote:\n> Now to keep it stateless, I thought that:\n...\n> This would guarantee, that when you want n revisions, you make at most\n> log2(n) requests and get at most 2*n revisions (well, the requests are for\n\nThat is still a lot! How about, for each ref\n\n - Client sends a POST listing the ref and the latest related commit\nit has that the server is likely to have (from origin/heads/).\nOptionally, it can provide a blacklist of (where every\nobject refered is known) and blob sha1s.\n - Server sends the new sha1 of the ref, and a thin pack that covers the changes\n - The client can disconnect to stop the transaction. For example --\nif it sees the sha1 of a huge object that it already has. It can\nre-request, with a blacklist.\n\nA good number of objects will be sent unnecesarily - with no option to\nthe client to say \"I have this\" - but by using the hint of letting the\nserver know we have origin/heads/ I suspect that it will be\nminimal.\n\nAlso:\n - It will probably be useful to list all the refs the client knows\nfrom that server in the request.\n - If the ref has changed with a non-fast-forward, the server needs to\nsay so, and provide a listing of the commits. As soon as the client\nspots a common commit, it can close the connection -- it now knows\nwhat ref to tell the server about in a subsequent command.\n\nThis way, you ideally have 1 request per ref, 2 if it has been\nrebased/rewound. This can probably get reorganised to do several refs\nin one request.\n\ncheers,\n\n\nm\n"},{"id":"64459","messageId":"20080104195911.GA4055@efreet.light.src","threadId":"11433","inReplyTo":"46a038f90801031554j6218f08cl6c9608b24e1675f8@mail.gmail.com","subject":"Re: git over webdav: what can I do for improving http-push ?","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-01-04T19:59:11Z","receivedAt":"2008-01-04T19:59:11Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Fri, Jan 04, 2008 at 12:54:58 +1300, Martin Langhoff wrote:\n> On Jan 4, 2008 10:15 AM, Jan Hudec wrote:\n> > Now to keep it stateless, I thought that:\n> ...\n> > This would guarantee, that when you want n revisions, you make at most\n> > log2(n) requests and get at most 2*n revisions (well, the requests are for\n> \n> That is still a lot! How about, for each ref\n\nThe whole point of that is that the packs can be statically precomputed and\nserved with quite low CPU load, which is useful for serving from shared\ncomputers (like servers in school computer labs or cheapo web hosting) or\nslow servers like NSLU2. Also it makes HTTP caching actually useful, because\nthe set of possible requests is quite limited.\n\nAlso, while I said it's for each ref, the packs should really be optimized\nfor the common case of fetching all refs, which would really make it just\nlog2(n) packs and 2*n revisions for each whole download.\n\n> - Client sends a POST listing the ref and the latest related commit\n> it has that the server is likely to have (from origin/heads/).\n> Optionally, it can provide a blacklist of (where every\n> object refered is known) and blob sha1s.\n> - Server sends the new sha1 of the ref, and a thin pack that covers the changes\n> - The client can disconnect to stop the transaction. For example --\n> if it sees the sha1 of a huge object that it already has. It can\n> re-request, with a blacklist.\n> \n> A good number of objects will be sent unnecesarily - with no option to\n> the client to say \"I have this\" - but by using the hint of letting the\n> server know we have origin/heads/ I suspect that it will be\n> minimal.\n\nIt would be better to only unnecesarily send revlists. Since each HTTP packed\nwill likely have something like 1kb overhead, sending few kb worth of revlist\nis still pretty efficient. So just send part of revlist, than more revlist\nand so on until you find exactly which revisions you need and than ask for\nthem. That will save *both* bandwidth *and* server CPU. The only reason to\nwaste bandwidth is to save CPU and you are not doing that.\n\n> Also:\n> - It will probably be useful to list all the refs the client knows\n> from that server in the request.\n> - If the ref has changed with a non-fast-forward, the server needs to\n> say so, and provide a listing of the commits. As soon as the client\n> spots a common commit, it can close the connection -- it now knows\n> what ref to tell the server about in a subsequent command.\n> \n> This way, you ideally have 1 request per ref, 2 if it has been\n> rebased/rewound. This can probably get reorganised to do several refs\n> in one request.\n> \n> cheers,\n> \n> \n> m\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec \n"}]}