{"thread":{"id":"20616","subject":"git http-push and MKCOL error (22/409)","startedAt":"2009-08-16T13:57:26Z","lastAt":"2009-08-17T17:28:38Z","messageCount":12,"participants":["Thomas Schlichter","Tay Ray Chuan","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"120768","messageId":"200908161557.26962.thomas.schlichter@web.de","threadId":"20616","inReplyTo":null,"subject":"git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-16T13:57:26Z","receivedAt":"2009-08-16T13:57:26Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hello,\n\nI was using git version 1.6.4 and a remote WebDAV repository. But \nunfortunately, git-http-push always returns following error:\n\nschlicht@netbook:~/dummy$ git push\nFetching remote heads...\n  refs\n  refs/heads\n  refs/tags\n  refs/heads\n  refs/tags\nupdating 'refs/heads/master'\n  from 980385b1032efc0db665dff3ad54068f762af9aa\n  to   98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d\n    sending 1 objects\nMKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)\nUpdating remote server info\nerror: failed to push some refs to \n'https://webdav.smartdrive.web.de/dummy.git'\n\nCurrent \"master\" and \"next\" trees also have this problem. But as git version \n1.6.4 does not have this problem, I was able to bisect it down to commit:\n\n   5424bc557fc6414660830b470dd45774b8f5f281\n   http*: add helper methods for fetching objects (loose)\n\nI can always reproduce this problem, so I am willing to test patches to fix \nthis regression.\n\nKind regards,\nThomas Schlichter\n"},{"id":"120769","messageId":"be6fef0d0908160727r7dfa9b46l4d493e3954c21de3@mail.gmail.com","threadId":"20616","inReplyTo":"200908161557.26962.thomas.schlichter@web.de","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-08-16T14:27:40Z","receivedAt":"2009-08-16T14:27:40Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Sun, Aug 16, 2009 at 9:57 PM, Thomas\nSchlichter<thomas.schlichter@web.de> wrote:\n> Current \"master\" and \"next\" trees also have this problem. But as git version\n> 1.6.4 does not have this problem, I was able to bisect it down to commit:\n>\n>   5424bc557fc6414660830b470dd45774b8f5f281\n>   http*: add helper methods for fetching objects (loose)\n\nInteresting. Please do provide:\n\n -steps to reproduce,\n -your server's access log.\n\n\n-- \nCheers,\nRay Chuan\n"},{"id":"120770","messageId":"200908161652.37282.thomas.schlichter@web.de","threadId":"20616","inReplyTo":"be6fef0d0908160727r7dfa9b46l4d493e3954c21de3@mail.gmail.com","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-16T14:52:37Z","receivedAt":"2009-08-16T14:52:37Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hi,\nAm Sonntag 16 August 2009 16:27:40 schrieb Tay Ray Chuan:\n> Interesting. Please do provide:\n>\n>  -steps to reproduce,\n>  -your server's access log.\n\nUnfortunately I cannot provide the server's access log. It is a server \nprovided by the German E-Mail provider web.de. It allows to save files via a \nweb-interface and via WebDAV.\n\nSteps to reproduce:\n  1. locally set up a git archive:\n     mkdir dummy.git\n     cd dummy.git\n     git init --bare\n  2. Upload this directory to the server.\n     I did do this using KDE's dolphin via WebDAV.\n  3. Clone this remote repository:\n     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy\n  4. Create a local commit:\n     cd my_dummy\n     touch dummy.c\n     git commit -a\n  5. Push this commit up to the remote repository:\n     git push origin master\n\nThe last step fails es explained. During bisecting git I was able to commit \nseveral changes with versions before commit \n5424bc557fc6414660830b470dd45774b8f5f281.\n\nKind regards,\nThomas Schlichter\n\nP.S.: Maybe it is important that the server uses the secure https protocol, \ntherefore I set my username and password combination in my local ~/.netrc \nfile.\n"},{"id":"120776","messageId":"7veirb5z9a.fsf@alter.siamese.dyndns.org","threadId":"20616","inReplyTo":"be6fef0d0908160727r7dfa9b46l4d493e3954c21de3@mail.gmail.com","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-08-16T19:34:25Z","receivedAt":"2009-08-16T19:34:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tay Ray Chuan <rctay89@gmail.com> writes:\n\n> On Sun, Aug 16, 2009 at 9:57 PM, Thomas\n> Schlichter<thomas.schlichter@web.de> wrote:\n>> Current \"master\" and \"next\" trees also have this problem. But as git version\n>> 1.6.4 does not have this problem, I was able to bisect it down to commit:\n>>\n>>   5424bc557fc6414660830b470dd45774b8f5f281\n>>   http*: add helper methods for fetching objects (loose)\n>\n> Interesting. Please do provide:\n>\n>  -steps to reproduce,\n>  -your server's access log.\n\nThe report said:\n\n    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)\n\nAs far as I can see you are trying (in http-push.c::start_mkcol()) to\ncreate the two-hexdigit fan-out directory (i.e. \"98\" for this example); it\nis strange to see a request to create the full 40-hexdigit collection in\nthe first place.\n\nIn another thread you responded to earlier:\n\n    http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=125972\n\nthe original report did not give the exact error message, but in that one,\ninstead of failing to create 40-hexdigit collection like this, I am\nguessing that it fails with something like \"MKCOL 'refs' failed\".\n\nSo are these unrelated \"breakages\"[1]?\n\n[Foornote]\n\n*1* Not necessarily in the sense the client is broken but in the sense\nthat the server-client combination does not work for the reporter.\n"},{"id":"120821","messageId":"be6fef0d0908162152o54206acfge43f6ffa012bb307@mail.gmail.com","threadId":"20616","inReplyTo":"200908161652.37282.thomas.schlichter@web.de","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-08-17T04:52:32Z","receivedAt":"2009-08-17T04:52:32Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Sun, Aug 16, 2009 at 10:52 PM, Thomas\nSchlichter<thomas.schlichter@web.de> wrote:\n> Steps to reproduce:\n\nbefore I try this out, I have a few queries:\n\n>  1. locally set up a git archive:\n>     mkdir dummy.git\n>     cd dummy.git\n>     git init --bare\n>  2. Upload this directory to the server.\n>     I did do this using KDE's dolphin via WebDAV.\n>  3. Clone this remote repository:\n>     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy\n\nYou didn't run git update-server-info before doing the clone; cloning\nwouldn't work cos git can't find a info/refs file. Perhaps you did do\nit and unintentionally left it out from your procedure listed here?\n\n>  4. Create a local commit:\n>     cd my_dummy\n>     touch dummy.c\n>     git commit -a\n\nI assume you want to commit dummy.c? But you didn't tell git to track\nit in the first place (git add dummy.c).\n\n>  5. Push this commit up to the remote repository:\n>     git push origin master\n\nI also find it strange that git only pushed 1 object from your git\npush output, when usually there would be a tree, a blob and a commit\nto push for a new commit.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"120822","messageId":"be6fef0d0908162158y429c4b5fgb9138151446e39ef@mail.gmail.com","threadId":"20616","inReplyTo":"7veirb5z9a.fsf@alter.siamese.dyndns.org","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-08-17T04:58:25Z","receivedAt":"2009-08-17T04:58:25Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Mon, Aug 17, 2009 at 3:34 AM, Junio C Hamano<gitster@pobox.com> wrote:\n> The report said:\n>\n>    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)\n>\n> As far as I can see you are trying (in http-push.c::start_mkcol()) to\n> create the two-hexdigit fan-out directory (i.e. \"98\" for this example); it\n> is strange to see a request to create the full 40-hexdigit collection in\n> the first place.\n\nYes, you're right, but git prints out the full SHA-1 hash even though\nit is actually referring to the 2-hexdigit directory that it failed to\ncreate/verify, for whatever reason.\n\n> In another thread you responded to earlier:\n>\n>    http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=125972\n>\n> the original report did not give the exact error message, but in that one,\n> instead of failing to create 40-hexdigit collection like this, I am\n> guessing that it fails with something like \"MKCOL 'refs' failed\".\n\nI guess by \"original report\" you refer to Thomas' initial email: yes,\nhe (unwittingly) did provide theerror code in the subject line, the\npart which says (22/409). It actually means (<curl return code>/<http\nreturn code>). Referring again to\nhttp://www.webdav.org/specs/rfc4918.html#rfc.section.9.3.1:\n\n  409 (Conflict) - A collection cannot be made at the Request-URI until\n  one or more intermediate collections have been created. The server\n  must not create those intermediate collections automatically.\n\nmeaning one or more parent directories weren't created.\n\nBased on this and his procedure, I'm guessing that there's something\nwrong with his setup.\n\n> *1* Not necessarily in the sense the client is broken but in the sense\n> that the server-client combination does not work for the reporter.\n\nIndeed.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"120824","messageId":"200908170718.15423.thomas.schlichter@web.de","threadId":"20616","inReplyTo":"be6fef0d0908162152o54206acfge43f6ffa012bb307@mail.gmail.com","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-17T05:18:15Z","receivedAt":"2009-08-17T05:18:15Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hi,\n\nAm Montag 17 August 2009 06:52:32 schrieb Tay Ray Chuan:\n> Hi,\n>\n> On Sun, Aug 16, 2009 at 10:52 PM, Thomas\n>\n> Schlichter<thomas.schlichter@web.de> wrote:\n> > Steps to reproduce:\n>\n> before I try this out, I have a few queries:\n> >  1. locally set up a git archive:\n> >     mkdir dummy.git\n> >     cd dummy.git\n> >     git init --bare\n> >  2. Upload this directory to the server.\n> >     I did do this using KDE's dolphin via WebDAV.\n> >  3. Clone this remote repository:\n> >     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy\n>\n> You didn't run git update-server-info before doing the clone; cloning\n> wouldn't work cos git can't find a info/refs file. Perhaps you did do\n> it and unintentionally left it out from your procedure listed here?\n\nOh, yes, sorry.\nI did run it but unintentionally left it out. At least I had no problem with \nclone, it did clone an \"empty repository\" without a problem. Even during \nbisecting, I never had a problem with pulling from the server, only with \npushing to it.\n\n> >  4. Create a local commit:\n> >     cd my_dummy\n> >     touch dummy.c\n> >     git commit -a\n>\n> I assume you want to commit dummy.c?\n\nYes, indeed...\n\n> But you didn't tell git to track\n> it in the first place (git add dummy.c).\n\nI think that was not neccessary,  the parameter \"-a\" should do it for me...\n\n> >  5. Push this commit up to the remote repository:\n> >     git push origin master\n>\n> I also find it strange that git only pushed 1 object from your git\n> push output, when usually there would be a tree, a blob and a commit\n> to push for a new commit.\n\nYes, most of the times git tries to push three objects, and fails with all \nthree. As I wrote, during bisecting I was able to commit several times. The \nreported message was the last one failing. Unfortunately git only needed to \npush one object that time. I didn't think it was important if it were three or \njust one object?!\n\nKind regards,\nThomas Schlichter\n"},{"id":"120825","messageId":"200908170725.09592.thomas.schlichter@web.de","threadId":"20616","inReplyTo":"be6fef0d0908162158y429c4b5fgb9138151446e39ef@mail.gmail.com","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-17T05:25:09Z","receivedAt":"2009-08-17T05:25:09Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hi,\n\nAm Montag 17 August 2009 06:58:25 schrieb Tay Ray Chuan:\n> Hi,\n>\n> On Mon, Aug 17, 2009 at 3:34 AM, Junio C Hamano<gitster@pobox.com> wrote:\n> > The report said:\n> >\n> >    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting\n> > (22/409)\n> >\n> > As far as I can see you are trying (in http-push.c::start_mkcol()) to\n> > create the two-hexdigit fan-out directory (i.e. \"98\" for this example);\n> > it is strange to see a request to create the full 40-hexdigit collection\n> > in the first place.\n>\n> Yes, you're right, but git prints out the full SHA-1 hash even though\n> it is actually referring to the 2-hexdigit directory that it failed to\n> create/verify, for whatever reason.\n>\n> > In another thread you responded to earlier:\n> >\n> >  \n> >  http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=1259\n> >72\n> >\n> > the original report did not give the exact error message, but in that\n> > one, instead of failing to create 40-hexdigit collection like this, I am\n> > guessing that it fails with something like \"MKCOL 'refs' failed\".\n>\n> I guess by \"original report\" you refer to Thomas' initial email: yes,\n> he (unwittingly) did provide theerror code in the subject line, the\n> part which says (22/409). It actually means (<curl return code>/<http\n> return code>). Referring again to\n> http://www.webdav.org/specs/rfc4918.html#rfc.section.9.3.1:\n>\n>   409 (Conflict) - A collection cannot be made at the Request-URI until\n>   one or more intermediate collections have been created. The server\n>   must not create those intermediate collections automatically.\n>\n> meaning one or more parent directories weren't created.\n>\n> Based on this and his procedure, I'm guessing that there's something\n> wrong with his setup.\n>\n> > *1* Not necessarily in the sense the client is broken but in the sense\n> > that the server-client combination does not work for the reporter.\n>\n> Indeed.\n\nHmm. I hope you won't take this as an argument for not fixing it, because it \nis a clear regression! I tried git version 1.6.3.4, and it works flawlessly \nwith exactly this server! Even during bisecting, some versions worked, some \ndidn't (these after the mentioned commit...)\n\nSo I think this commit didn't only refactor things, but unintentionally \nchanged the behavior. And this must be the problem. As I'm not into this code, \nand the refactoring was not completely trivial, I was not able to quickly find \nthe place where the behavior was changed...\n\nKind regards,\nThomas Schlichter\n"},{"id":"120826","messageId":"200908170734.38838.thomas.schlichter@web.de","threadId":"20616","inReplyTo":"200908170718.15423.thomas.schlichter@web.de","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-17T05:34:38Z","receivedAt":"2009-08-17T05:34:38Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hi,\n\nAm Montag 17 August 2009 07:18:15 schrieb Thomas Schlichter:\n> Hi,\n>\n> Am Montag 17 August 2009 06:52:32 schrieb Tay Ray Chuan:\n> > I assume you want to commit dummy.c?\n>\n> Yes, indeed...\n>\n> > But you didn't tell git to track\n> > it in the first place (git add dummy.c).\n>\n> I think that was not neccessary,  the parameter \"-a\" should do it for me...\n\nOK, I tried it again, and the parameter \"-a\" was not sufficient. So I \nobviously did run a \"git add dummy.c\". Sorry for missing that.\n\nNow I do also have the error message for the first commit:\n\nschlicht@netbook:~/dummy2$ git push origin master\nFetching remote heads...\n  refs\n  refs/heads\n  refs/tags\n  refs/heads\n  refs/tags\nupdating 'refs/heads/master'\n  from 0000000000000000000000000000000000000000\n  to   60157fdfdca4693f703c744fed95e1b31743bb4e\n    sending 3 objects\nMKCOL e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 failed, aborting (22/409)\nMKCOL 4aefccd66b8e2bfa58079910045c30bc792877dc failed, aborting (22/409)\nMKCOL 60157fdfdca4693f703c744fed95e1b31743bb4e failed, aborting (22/409)\nUpdating remote server info\nerror: failed to push some refs to \n'https://webdav.smartdrive.web.de/dummy2.git'\n\nKind regards,\nThomas Schlichter\n"},{"id":"120829","messageId":"7vvdkn2crx.fsf@alter.siamese.dyndns.org","threadId":"20616","inReplyTo":"200908170725.09592.thomas.schlichter@web.de","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-08-17T06:08:18Z","receivedAt":"2009-08-17T06:08:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Schlichter <thomas.schlichter@web.de> writes:\n\n> Hmm. I hope you won't take this as an argument for not fixing it, because it \n> is a clear regression! I tried git version 1.6.3.4, and it works flawlessly \n> with exactly this server! Even during bisecting, some versions worked, some \n> didn't (these after the mentioned commit...)\n\nIf it works (without changing anything else) with one version and with a\nproblematic patch it doesn't, it is a clear regression.\n\nIf you revert 5424bc5 (http*: add helper methods for fetching objects\n(loose), 2009-06-06) from the tip of 'master', does it make the problem go\naway?  \n\nThe patch should look like something like the attached.\n\ndiff --git a/http-push.c b/http-push.c\nindex 00e83dc..d1ea702 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -115,10 +115,18 @@ struct transfer_request\n \tstruct remote_lock *lock;\n \tstruct curl_slist *headers;\n \tstruct buffer buffer;\n+\tchar filename[PATH_MAX];\n+\tchar tmpfile[PATH_MAX];\n+\tint local_fileno;\n \tenum transfer_state state;\n \tCURLcode curl_result;\n \tchar errorstr[CURL_ERROR_SIZE];\n \tlong http_code;\n+\tunsigned char real_sha1[20];\n+\tgit_SHA_CTX c;\n+\tz_stream stream;\n+\tint zret;\n+\tint rename;\n \tvoid *userData;\n \tstruct active_request_slot *slot;\n \tstruct transfer_request *next;\n@@ -226,6 +234,15 @@ static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum d\n \treturn dav_headers;\n }\n \n+static void append_remote_object_url(struct strbuf *buf, const char *url,\n+\t\t\t\t     const char *hex,\n+\t\t\t\t     int only_two_digit_prefix)\n+{\n+\tstrbuf_addf(buf, \"%sobjects/%.*s/\", url, 2, hex);\n+\tif (!only_two_digit_prefix)\n+\t\tstrbuf_addf(buf, \"%s\", hex+2);\n+}\n+\n static void finish_request(struct transfer_request *request);\n static void release_request(struct transfer_request *request);\n \n@@ -239,29 +256,169 @@ static void process_response(void *callback_data)\n \n #ifdef USE_CURL_MULTI\n \n+static char *get_remote_object_url(const char *url, const char *hex,\n+\t\t\t\t   int only_two_digit_prefix)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tappend_remote_object_url(&buf, url, hex, only_two_digit_prefix);\n+\treturn strbuf_detach(&buf, NULL);\n+}\n+\n+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n+\t\t\t       void *data)\n+{\n+\tunsigned char expn[4096];\n+\tsize_t size = eltsize * nmemb;\n+\tint posn = 0;\n+\tstruct transfer_request *request = (struct transfer_request *)data;\n+\tdo {\n+\t\tssize_t retval = xwrite(request->local_fileno,\n+\t\t\t\t\t(char *) ptr + posn, size - posn);\n+\t\tif (retval < 0)\n+\t\t\treturn posn;\n+\t\tposn += retval;\n+\t} while (posn < size);\n+\n+\trequest->stream.avail_in = size;\n+\trequest->stream.next_in = ptr;\n+\tdo {\n+\t\trequest->stream.next_out = expn;\n+\t\trequest->stream.avail_out = sizeof(expn);\n+\t\trequest->zret = git_inflate(&request->stream, Z_SYNC_FLUSH);\n+\t\tgit_SHA1_Update(&request->c, expn,\n+\t\t\t\tsizeof(expn) - request->stream.avail_out);\n+\t} while (request->stream.avail_in && request->zret == Z_OK);\n+\tdata_received++;\n+\treturn size;\n+}\n+\n static void start_fetch_loose(struct transfer_request *request)\n {\n+\tchar *hex = sha1_to_hex(request->obj->sha1);\n+\tchar *filename;\n+\tchar prevfile[PATH_MAX];\n+\tchar *url;\n+\tint prevlocal;\n+\tunsigned char prev_buf[PREV_BUF_SIZE];\n+\tssize_t prev_read = 0;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n \tstruct active_request_slot *slot;\n-\tstruct http_object_request *obj_req;\n \n-\tobj_req = new_http_object_request(repo->url, request->obj->sha1);\n-\tif (obj_req == NULL) {\n+\tfilename = sha1_file_name(request->obj->sha1);\n+\tsnprintf(request->filename, sizeof(request->filename), \"%s\", filename);\n+\tsnprintf(request->tmpfile, sizeof(request->tmpfile),\n+\t\t \"%s.temp\", filename);\n+\n+\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", request->filename);\n+\tunlink_or_warn(prevfile);\n+\trename(request->tmpfile, prevfile);\n+\tunlink_or_warn(request->tmpfile);\n+\n+\tif (request->local_fileno != -1)\n+\t\terror(\"fd leakage in start: %d\", request->local_fileno);\n+\trequest->local_fileno = open(request->tmpfile,\n+\t\t\t\t     O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t/*\n+\t * This could have failed due to the \"lazy directory creation\";\n+\t * try to mkdir the last path component.\n+\t */\n+\tif (request->local_fileno < 0 && errno == ENOENT) {\n+\t\tchar *dir = strrchr(request->tmpfile, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(request->tmpfile, 0777);\n+\t\t\t*dir = '/';\n+\t\t}\n+\t\trequest->local_fileno = open(request->tmpfile,\n+\t\t\t\t\t     O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t}\n+\n+\tif (request->local_fileno < 0) {\n \t\trequest->state = ABORTED;\n+\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n+\t\t      request->tmpfile, request->filename, strerror(errno));\n \t\treturn;\n \t}\n \n-\tslot = obj_req->slot;\n+\tmemset(&request->stream, 0, sizeof(request->stream));\n+\n+\tgit_inflate_init(&request->stream);\n+\n+\tgit_SHA1_Init(&request->c);\n+\n+\turl = get_remote_object_url(repo->url, hex, 0);\n+\trequest->url = xstrdup(url);\n+\n+\t/*\n+\t * If a previous temp file is present, process what was already\n+\t * fetched.\n+\t */\n+\tprevlocal = open(prevfile, O_RDONLY);\n+\tif (prevlocal != -1) {\n+\t\tdo {\n+\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n+\t\t\tif (prev_read>0) {\n+\t\t\t\tif (fwrite_sha1_file(prev_buf,\n+\t\t\t\t\t\t     1,\n+\t\t\t\t\t\t     prev_read,\n+\t\t\t\t\t\t     request) == prev_read)\n+\t\t\t\t\tprev_posn += prev_read;\n+\t\t\t\telse\n+\t\t\t\t\tprev_read = -1;\n+\t\t\t}\n+\t\t} while (prev_read > 0);\n+\t\tclose(prevlocal);\n+\t}\n+\tunlink_or_warn(prevfile);\n+\n+\t/*\n+\t * Reset inflate/SHA1 if there was an error reading the previous temp\n+\t * file; also rewind to the beginning of the local file.\n+\t */\n+\tif (prev_read == -1) {\n+\t\tmemset(&request->stream, 0, sizeof(request->stream));\n+\t\tgit_inflate_init(&request->stream);\n+\t\tgit_SHA1_Init(&request->c);\n+\t\tif (prev_posn>0) {\n+\t\t\tprev_posn = 0;\n+\t\t\tlseek(request->local_fileno, 0, SEEK_SET);\n+\t\t\tftruncate(request->local_fileno, 0);\n+\t\t}\n+\t}\n+\n+\tslot = get_active_slot();\n \tslot->callback_func = process_response;\n \tslot->callback_data = request;\n \trequest->slot = slot;\n-\trequest->userData = obj_req;\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, request);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n+\n+\t/*\n+\t * If we have successfully processed data from a previous fetch\n+\t * attempt, only fetch the data we don't already have.\n+\t */\n+\tif (prev_posn>0) {\n+\t\tif (push_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n+\t\t\t\thex, prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(slot->curl,\n+\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n+\t}\n \n \t/* Try to get the request started, abort the request on error */\n \trequest->state = RUN_FETCH_LOOSE;\n \tif (!start_active_slot(slot)) {\n \t\tfprintf(stderr, \"Unable to start GET request\\n\");\n \t\trepo->can_update_info_refs = 0;\n-\t\trelease_http_object_request(obj_req);\n \t\trelease_request(request);\n \t}\n }\n@@ -520,14 +677,16 @@ static void release_request(struct transfer_request *request)\n \t\t\tentry->next = entry->next->next;\n \t}\n \n+\tif (request->local_fileno != -1)\n+\t\tclose(request->local_fileno);\n \tfree(request->url);\n \tfree(request);\n }\n \n static void finish_request(struct transfer_request *request)\n {\n+\tstruct stat st;\n \tstruct http_pack_request *preq;\n-\tstruct http_object_request *obj_req;\n \n \trequest->curl_result = request->slot->curl_result;\n \trequest->http_code = request->slot->http_code;\n@@ -582,17 +741,39 @@ static void finish_request(struct transfer_request *request)\n \t\t\taborted = 1;\n \t\t}\n \t} else if (request->state == RUN_FETCH_LOOSE) {\n-\t\tobj_req = (struct http_object_request *)request->userData;\n-\n-\t\tif (finish_http_object_request(obj_req) == 0)\n-\t\t\tif (obj_req->rename == 0)\n-\t\t\t\trequest->obj->flags |= (LOCAL | REMOTE);\n+\t\tclose(request->local_fileno);\n+\t\trequest->local_fileno = -1;\n+\n+\t\tif (request->curl_result != CURLE_OK &&\n+\t\t    request->http_code != 416) {\n+\t\t\tif (stat(request->tmpfile, &st) == 0) {\n+\t\t\t\tif (st.st_size == 0)\n+\t\t\t\t\tunlink_or_warn(request->tmpfile);\n+\t\t\t}\n+\t\t} else {\n+\t\t\tif (request->http_code == 416)\n+\t\t\t\twarning(\"requested range invalid; we may already have all the data.\");\n+\n+\t\t\tgit_inflate_end(&request->stream);\n+\t\t\tgit_SHA1_Final(request->real_sha1, &request->c);\n+\t\t\tif (request->zret != Z_STREAM_END) {\n+\t\t\t\tunlink_or_warn(request->tmpfile);\n+\t\t\t} else if (hashcmp(request->obj->sha1, request->real_sha1)) {\n+\t\t\t\tunlink_or_warn(request->tmpfile);\n+\t\t\t} else {\n+\t\t\t\trequest->rename =\n+\t\t\t\t\tmove_temp_to_file(\n+\t\t\t\t\t\trequest->tmpfile,\n+\t\t\t\t\t\trequest->filename);\n+\t\t\t\tif (request->rename == 0)\n+\t\t\t\t\trequest->obj->flags |= (LOCAL | REMOTE);\n+\t\t\t}\n+\t\t}\n \n \t\t/* Try fetching packed if necessary */\n-\t\tif (request->obj->flags & LOCAL) {\n-\t\t\trelease_http_object_request(obj_req);\n+\t\tif (request->obj->flags & LOCAL)\n \t\t\trelease_request(request);\n-\t\t} else\n+\t\telse\n \t\t\tstart_fetch_packed(request);\n \n \t} else if (request->state == RUN_FETCH_PACKED) {\n@@ -664,6 +845,7 @@ static void add_fetch_request(struct object *obj)\n \trequest->url = NULL;\n \trequest->lock = NULL;\n \trequest->headers = NULL;\n+\trequest->local_fileno = -1;\n \trequest->state = NEED_FETCH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\n@@ -702,6 +884,7 @@ static int add_send_request(struct object *obj, struct remote_lock *lock)\n \trequest->url = NULL;\n \trequest->lock = lock;\n \trequest->headers = NULL;\n+\trequest->local_fileno = -1;\n \trequest->state = NEED_PUSH;\n \trequest->next = request_queue_head;\n \trequest_queue_head = request;\ndiff --git a/http-walker.c b/http-walker.c\nindex 700bc13..8f7a975 100644\n--- a/http-walker.c\n+++ b/http-walker.c\n@@ -3,6 +3,8 @@\n #include \"walker.h\"\n #include \"http.h\"\n \n+#define PREV_BUF_SIZE 4096\n+\n struct alt_base\n {\n \tchar *base;\n@@ -23,8 +25,20 @@ struct object_request\n \tstruct walker *walker;\n \tunsigned char sha1[20];\n \tstruct alt_base *repo;\n+\tchar *url;\n+\tchar filename[PATH_MAX];\n+\tchar tmpfile[PATH_MAX];\n+\tint local;\n \tenum object_request_state state;\n-\tstruct http_object_request *req;\n+\tCURLcode curl_result;\n+\tchar errorstr[CURL_ERROR_SIZE];\n+\tlong http_code;\n+\tunsigned char real_sha1[20];\n+\tgit_SHA_CTX c;\n+\tz_stream stream;\n+\tint zret;\n+\tint rename;\n+\tstruct active_request_slot *slot;\n \tstruct object_request *next;\n };\n \n@@ -45,6 +59,34 @@ struct walker_data {\n \n static struct object_request *object_queue_head;\n \n+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n+\t\t\t       void *data)\n+{\n+\tunsigned char expn[4096];\n+\tsize_t size = eltsize * nmemb;\n+\tint posn = 0;\n+\tstruct object_request *obj_req = (struct object_request *)data;\n+\tdo {\n+\t\tssize_t retval = xwrite(obj_req->local,\n+\t\t\t\t\t(char *) ptr + posn, size - posn);\n+\t\tif (retval < 0)\n+\t\t\treturn posn;\n+\t\tposn += retval;\n+\t} while (posn < size);\n+\n+\tobj_req->stream.avail_in = size;\n+\tobj_req->stream.next_in = ptr;\n+\tdo {\n+\t\tobj_req->stream.next_out = expn;\n+\t\tobj_req->stream.avail_out = sizeof(expn);\n+\t\tobj_req->zret = git_inflate(&obj_req->stream, Z_SYNC_FLUSH);\n+\t\tgit_SHA1_Update(&obj_req->c, expn,\n+\t\t\t\tsizeof(expn) - obj_req->stream.avail_out);\n+\t} while (obj_req->stream.avail_in && obj_req->zret == Z_OK);\n+\tdata_received++;\n+\treturn size;\n+}\n+\n static void fetch_alternates(struct walker *walker, const char *base);\n \n static void process_object_response(void *callback_data);\n@@ -52,35 +94,172 @@ static void process_object_response(void *callback_data);\n static void start_object_request(struct walker *walker,\n \t\t\t\t struct object_request *obj_req)\n {\n+\tchar *hex = sha1_to_hex(obj_req->sha1);\n+\tchar prevfile[PATH_MAX];\n+\tchar *url;\n+\tchar *posn;\n+\tint prevlocal;\n+\tunsigned char prev_buf[PREV_BUF_SIZE];\n+\tssize_t prev_read = 0;\n+\tlong prev_posn = 0;\n+\tchar range[RANGE_HEADER_SIZE];\n+\tstruct curl_slist *range_header = NULL;\n \tstruct active_request_slot *slot;\n-\tstruct http_object_request *req;\n \n-\treq = new_http_object_request(obj_req->repo->base, obj_req->sha1);\n-\tif (req == NULL) {\n+\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", obj_req->filename);\n+\tunlink_or_warn(prevfile);\n+\trename(obj_req->tmpfile, prevfile);\n+\tunlink_or_warn(obj_req->tmpfile);\n+\n+\tif (obj_req->local != -1)\n+\t\terror(\"fd leakage in start: %d\", obj_req->local);\n+\tobj_req->local = open(obj_req->tmpfile,\n+\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t/*\n+\t * This could have failed due to the \"lazy directory creation\";\n+\t * try to mkdir the last path component.\n+\t */\n+\tif (obj_req->local < 0 && errno == ENOENT) {\n+\t\tchar *dir = strrchr(obj_req->tmpfile, '/');\n+\t\tif (dir) {\n+\t\t\t*dir = 0;\n+\t\t\tmkdir(obj_req->tmpfile, 0777);\n+\t\t\t*dir = '/';\n+\t\t}\n+\t\tobj_req->local = open(obj_req->tmpfile,\n+\t\t\t\t      O_WRONLY | O_CREAT | O_EXCL, 0666);\n+\t}\n+\n+\tif (obj_req->local < 0) {\n \t\tobj_req->state = ABORTED;\n+\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n+\t\t      obj_req->tmpfile, obj_req->filename, strerror(errno));\n \t\treturn;\n \t}\n-\tobj_req->req = req;\n \n-\tslot = req->slot;\n+\tmemset(&obj_req->stream, 0, sizeof(obj_req->stream));\n+\n+\tgit_inflate_init(&obj_req->stream);\n+\n+\tgit_SHA1_Init(&obj_req->c);\n+\n+\turl = xmalloc(strlen(obj_req->repo->base) + 51);\n+\tobj_req->url = xmalloc(strlen(obj_req->repo->base) + 51);\n+\tstrcpy(url, obj_req->repo->base);\n+\tposn = url + strlen(obj_req->repo->base);\n+\tstrcpy(posn, \"/objects/\");\n+\tposn += 9;\n+\tmemcpy(posn, hex, 2);\n+\tposn += 2;\n+\t*(posn++) = '/';\n+\tstrcpy(posn, hex + 2);\n+\tstrcpy(obj_req->url, url);\n+\n+\t/*\n+\t * If a previous temp file is present, process what was already\n+\t * fetched.\n+\t */\n+\tprevlocal = open(prevfile, O_RDONLY);\n+\tif (prevlocal != -1) {\n+\t\tdo {\n+\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n+\t\t\tif (prev_read>0) {\n+\t\t\t\tif (fwrite_sha1_file(prev_buf,\n+\t\t\t\t\t\t     1,\n+\t\t\t\t\t\t     prev_read,\n+\t\t\t\t\t\t     obj_req) == prev_read)\n+\t\t\t\t\tprev_posn += prev_read;\n+\t\t\t\telse\n+\t\t\t\t\tprev_read = -1;\n+\t\t\t}\n+\t\t} while (prev_read > 0);\n+\t\tclose(prevlocal);\n+\t}\n+\tunlink_or_warn(prevfile);\n+\n+\t/*\n+\t * Reset inflate/SHA1 if there was an error reading the previous temp\n+\t * file; also rewind to the beginning of the local file.\n+\t */\n+\tif (prev_read == -1) {\n+\t\tmemset(&obj_req->stream, 0, sizeof(obj_req->stream));\n+\t\tgit_inflate_init(&obj_req->stream);\n+\t\tgit_SHA1_Init(&obj_req->c);\n+\t\tif (prev_posn>0) {\n+\t\t\tprev_posn = 0;\n+\t\t\tlseek(obj_req->local, 0, SEEK_SET);\n+\t\t\tftruncate(obj_req->local, 0);\n+\t\t}\n+\t}\n+\n+\tslot = get_active_slot();\n \tslot->callback_func = process_object_response;\n \tslot->callback_data = obj_req;\n+\tobj_req->slot = slot;\n+\n+\tcurl_easy_setopt(slot->curl, CURLOPT_FILE, obj_req);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, obj_req->errorstr);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_URL, url);\n+\tcurl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n+\n+\t/*\n+\t * If we have successfully processed data from a previous fetch\n+\t * attempt, only fetch the data we don't already have.\n+\t */\n+\tif (prev_posn>0) {\n+\t\tif (walker->get_verbosely)\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n+\t\t\t\thex, prev_posn);\n+\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n+\t\trange_header = curl_slist_append(range_header, range);\n+\t\tcurl_easy_setopt(slot->curl,\n+\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n+\t}\n \n \t/* Try to get the request started, abort the request on error */\n \tobj_req->state = ACTIVE;\n \tif (!start_active_slot(slot)) {\n \t\tobj_req->state = ABORTED;\n-\t\trelease_http_object_request(req);\n+\t\tobj_req->slot = NULL;\n+\t\tclose(obj_req->local);\n+\t\tobj_req->local = -1;\n+\t\tfree(obj_req->url);\n \t\treturn;\n \t}\n }\n \n static void finish_object_request(struct object_request *obj_req)\n {\n-\tif (finish_http_object_request(obj_req->req))\n+\tstruct stat st;\n+\n+\tclose(obj_req->local);\n+\tobj_req->local = -1;\n+\n+\tif (obj_req->http_code == 416) {\n+\t\tfprintf(stderr, \"Warning: requested range invalid; we may already have all the data.\\n\");\n+\t} else if (obj_req->curl_result != CURLE_OK) {\n+\t\tif (stat(obj_req->tmpfile, &st) == 0)\n+\t\t\tif (st.st_size == 0)\n+\t\t\t\tunlink_or_warn(obj_req->tmpfile);\n \t\treturn;\n+\t}\n+\n+\tgit_inflate_end(&obj_req->stream);\n+\tgit_SHA1_Final(obj_req->real_sha1, &obj_req->c);\n+\tif (obj_req->zret != Z_STREAM_END) {\n+\t\tunlink_or_warn(obj_req->tmpfile);\n+\t\treturn;\n+\t}\n+\tif (hashcmp(obj_req->sha1, obj_req->real_sha1)) {\n+\t\tunlink_or_warn(obj_req->tmpfile);\n+\t\treturn;\n+\t}\n+\tobj_req->rename =\n+\t\tmove_temp_to_file(obj_req->tmpfile, obj_req->filename);\n \n-\tif (obj_req->req->rename == 0)\n+\tif (obj_req->rename == 0)\n \t\twalker_say(obj_req->walker, \"got %s\\n\", sha1_to_hex(obj_req->sha1));\n }\n \n@@ -92,16 +271,19 @@ static void process_object_response(void *callback_data)\n \tstruct walker_data *data = walker->data;\n \tstruct alt_base *alt = data->alt;\n \n-\tprocess_http_object_request(obj_req->req);\n+\tobj_req->curl_result = obj_req->slot->curl_result;\n+\tobj_req->http_code = obj_req->slot->http_code;\n+\tobj_req->slot = NULL;\n \tobj_req->state = COMPLETE;\n \n \t/* Use alternates if necessary */\n-\tif (missing_target(obj_req->req)) {\n+\tif (missing_target(obj_req)) {\n \t\tfetch_alternates(walker, alt->base);\n \t\tif (obj_req->repo->next != NULL) {\n \t\t\tobj_req->repo =\n \t\t\t\tobj_req->repo->next;\n-\t\t\trelease_http_object_request(obj_req->req);\n+\t\t\tclose(obj_req->local);\n+\t\t\tobj_req->local = -1;\n \t\t\tstart_object_request(walker, obj_req);\n \t\t\treturn;\n \t\t}\n@@ -114,8 +296,8 @@ static void release_object_request(struct object_request *obj_req)\n {\n \tstruct object_request *entry = object_queue_head;\n \n-\tif (obj_req->req !=NULL && obj_req->req->localfile != -1)\n-\t\terror(\"fd leakage in release: %d\", obj_req->req->localfile);\n+\tif (obj_req->local != -1)\n+\t\terror(\"fd leakage in release: %d\", obj_req->local);\n \tif (obj_req == object_queue_head) {\n \t\tobject_queue_head = obj_req->next;\n \t} else {\n@@ -125,6 +307,7 @@ static void release_object_request(struct object_request *obj_req)\n \t\t\tentry->next = entry->next->next;\n \t}\n \n+\tfree(obj_req->url);\n \tfree(obj_req);\n }\n \n@@ -152,13 +335,19 @@ static void prefetch(struct walker *walker, unsigned char *sha1)\n \tstruct object_request *newreq;\n \tstruct object_request *tail;\n \tstruct walker_data *data = walker->data;\n+\tchar *filename = sha1_file_name(sha1);\n \n \tnewreq = xmalloc(sizeof(*newreq));\n \tnewreq->walker = walker;\n \thashcpy(newreq->sha1, sha1);\n \tnewreq->repo = data->alt;\n+\tnewreq->url = NULL;\n+\tnewreq->local = -1;\n \tnewreq->state = WAITING;\n-\tnewreq->req = NULL;\n+\tsnprintf(newreq->filename, sizeof(newreq->filename), \"%s\", filename);\n+\tsnprintf(newreq->tmpfile, sizeof(newreq->tmpfile),\n+\t\t \"%s.temp\", filename);\n+\tnewreq->slot = NULL;\n \tnewreq->next = NULL;\n \n \thttp_is_verbose = walker->get_verbosely;\n@@ -449,6 +638,15 @@ abort:\n \n static void abort_object_request(struct object_request *obj_req)\n {\n+\tif (obj_req->local >= 0) {\n+\t\tclose(obj_req->local);\n+\t\tobj_req->local = -1;\n+\t}\n+\tunlink_or_warn(obj_req->tmpfile);\n+\tif (obj_req->slot) {\n+\t\trelease_active_slot(obj_req->slot);\n+\t\tobj_req->slot = NULL;\n+\t}\n \trelease_object_request(obj_req);\n }\n \n@@ -457,7 +655,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \tchar *hex = sha1_to_hex(sha1);\n \tint ret = 0;\n \tstruct object_request *obj_req = object_queue_head;\n-\tstruct http_object_request *req;\n \n \twhile (obj_req != NULL && hashcmp(obj_req->sha1, sha1))\n \t\tobj_req = obj_req->next;\n@@ -465,8 +662,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \t\treturn error(\"Couldn't find request for %s in the queue\", hex);\n \n \tif (has_sha1_file(obj_req->sha1)) {\n-\t\tif (obj_req->req != NULL)\n-\t\t\tabort_http_object_request(obj_req->req);\n \t\tabort_object_request(obj_req);\n \t\treturn 0;\n \t}\n@@ -478,42 +673,34 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c\n \tstart_object_request(walker, obj_req);\n #endif\n \n-\t/*\n-\t * obj_req->req might change when fetching alternates in the callback\n-\t * process_object_response; therefore, the \"shortcut\" variable, req,\n-\t * is used only after we're done with slots.\n-\t */\n \twhile (obj_req->state == ACTIVE)\n-\t\trun_active_slot(obj_req->req->slot);\n-\n-\treq = obj_req->req;\n+\t\trun_active_slot(obj_req->slot);\n \n-\tif (req->localfile != -1) {\n-\t\tclose(req->localfile);\n-\t\treq->localfile = -1;\n+\tif (obj_req->local != -1) {\n+\t\tclose(obj_req->local);\n+\t\tobj_req->local = -1;\n \t}\n \n \tif (obj_req->state == ABORTED) {\n \t\tret = error(\"Request for %s aborted\", hex);\n-\t} else if (req->curl_result != CURLE_OK &&\n-\t\t   req->http_code != 416) {\n-\t\tif (missing_target(req))\n+\t} else if (obj_req->curl_result != CURLE_OK &&\n+\t\t   obj_req->http_code != 416) {\n+\t\tif (missing_target(obj_req))\n \t\t\tret = -1; /* Be silent, it is probably in a pack. */\n \t\telse\n \t\t\tret = error(\"%s (curl_result = %d, http_code = %ld, sha1 = %s)\",\n-\t\t\t\t    req->errorstr, req->curl_result,\n-\t\t\t\t    req->http_code, hex);\n-\t} else if (req->zret != Z_STREAM_END) {\n+\t\t\t\t    obj_req->errorstr, obj_req->curl_result,\n+\t\t\t\t    obj_req->http_code, hex);\n+\t} else if (obj_req->zret != Z_STREAM_END) {\n \t\twalker->corrupt_object_found++;\n-\t\tret = error(\"File %s (%s) corrupt\", hex, req->url);\n-\t} else if (hashcmp(obj_req->sha1, req->real_sha1)) {\n+\t\tret = error(\"File %s (%s) corrupt\", hex, obj_req->url);\n+\t} else if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {\n \t\tret = error(\"File %s has bad hash\", hex);\n-\t} else if (req->rename < 0) {\n+\t} else if (obj_req->rename < 0) {\n \t\tret = error(\"unable to write sha1 filename %s\",\n-\t\t\t    req->filename);\n+\t\t\t    obj_req->filename);\n \t}\n \n-\trelease_http_object_request(req);\n \trelease_object_request(obj_req);\n \treturn ret;\n }\ndiff --git a/http.c b/http.c\nindex 14d5357..ad7b6f8 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,10 +12,6 @@ static CURLM *curlm;\n #ifndef NO_CURL_EASY_DUPHANDLE\n static CURL *curl_default;\n #endif\n-\n-#define PREV_BUF_SIZE 4096\n-#define RANGE_HEADER_SIZE 30\n-\n char curl_errorstr[CURL_ERROR_SIZE];\n \n static int curl_ssl_verify = -1;\n@@ -45,7 +41,8 @@ static char *ssl_cert_password;\n static int ssl_cert_password_required;\n \n static struct curl_slist *pragma_header;\n-static struct curl_slist *no_pragma_header;\n+\n+struct curl_slist *no_pragma_header;\n \n static struct active_request_slot *active_queue_head;\n \n@@ -715,23 +712,6 @@ static char *quote_ref_url(const char *base, const char *ref)\n \treturn strbuf_detach(&buf, NULL);\n }\n \n-void append_remote_object_url(struct strbuf *buf, const char *url,\n-\t\t\t      const char *hex,\n-\t\t\t      int only_two_digit_prefix)\n-{\n-\tstrbuf_addf(buf, \"%s/objects/%.*s/\", url, 2, hex);\n-\tif (!only_two_digit_prefix)\n-\t\tstrbuf_addf(buf, \"%s\", hex+2);\n-}\n-\n-char *get_remote_object_url(const char *url, const char *hex,\n-\t\t\t    int only_two_digit_prefix)\n-{\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tappend_remote_object_url(&buf, url, hex, only_two_digit_prefix);\n-\treturn strbuf_detach(&buf, NULL);\n-}\n-\n /* http_request() targets */\n #define HTTP_REQUEST_STRBUF\t0\n #define HTTP_REQUEST_FILE\t1\n@@ -1061,231 +1041,3 @@ abort:\n \tfree(preq);\n \treturn NULL;\n }\n-\n-/* Helpers for fetching objects (loose) */\n-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,\n-\t\t\t       void *data)\n-{\n-\tunsigned char expn[4096];\n-\tsize_t size = eltsize * nmemb;\n-\tint posn = 0;\n-\tstruct http_object_request *freq =\n-\t\t(struct http_object_request *)data;\n-\tdo {\n-\t\tssize_t retval = xwrite(freq->localfile,\n-\t\t\t\t\t(char *) ptr + posn, size - posn);\n-\t\tif (retval < 0)\n-\t\t\treturn posn;\n-\t\tposn += retval;\n-\t} while (posn < size);\n-\n-\tfreq->stream.avail_in = size;\n-\tfreq->stream.next_in = ptr;\n-\tdo {\n-\t\tfreq->stream.next_out = expn;\n-\t\tfreq->stream.avail_out = sizeof(expn);\n-\t\tfreq->zret = git_inflate(&freq->stream, Z_SYNC_FLUSH);\n-\t\tgit_SHA1_Update(&freq->c, expn,\n-\t\t\t\tsizeof(expn) - freq->stream.avail_out);\n-\t} while (freq->stream.avail_in && freq->zret == Z_OK);\n-\tdata_received++;\n-\treturn size;\n-}\n-\n-struct http_object_request *new_http_object_request(const char *base_url,\n-\tunsigned char *sha1)\n-{\n-\tchar *hex = sha1_to_hex(sha1);\n-\tchar *filename;\n-\tchar prevfile[PATH_MAX];\n-\tint prevlocal;\n-\tunsigned char prev_buf[PREV_BUF_SIZE];\n-\tssize_t prev_read = 0;\n-\tlong prev_posn = 0;\n-\tchar range[RANGE_HEADER_SIZE];\n-\tstruct curl_slist *range_header = NULL;\n-\tstruct http_object_request *freq;\n-\n-\tfreq = xmalloc(sizeof(*freq));\n-\thashcpy(freq->sha1, sha1);\n-\tfreq->localfile = -1;\n-\n-\tfilename = sha1_file_name(sha1);\n-\tsnprintf(freq->filename, sizeof(freq->filename), \"%s\", filename);\n-\tsnprintf(freq->tmpfile, sizeof(freq->tmpfile),\n-\t\t \"%s.temp\", filename);\n-\n-\tsnprintf(prevfile, sizeof(prevfile), \"%s.prev\", filename);\n-\tunlink_or_warn(prevfile);\n-\trename(freq->tmpfile, prevfile);\n-\tunlink_or_warn(freq->tmpfile);\n-\n-\tif (freq->localfile != -1)\n-\t\terror(\"fd leakage in start: %d\", freq->localfile);\n-\tfreq->localfile = open(freq->tmpfile,\n-\t\t\t       O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t/*\n-\t * This could have failed due to the \"lazy directory creation\";\n-\t * try to mkdir the last path component.\n-\t */\n-\tif (freq->localfile < 0 && errno == ENOENT) {\n-\t\tchar *dir = strrchr(freq->tmpfile, '/');\n-\t\tif (dir) {\n-\t\t\t*dir = 0;\n-\t\t\tmkdir(freq->tmpfile, 0777);\n-\t\t\t*dir = '/';\n-\t\t}\n-\t\tfreq->localfile = open(freq->tmpfile,\n-\t\t\t\t       O_WRONLY | O_CREAT | O_EXCL, 0666);\n-\t}\n-\n-\tif (freq->localfile < 0) {\n-\t\terror(\"Couldn't create temporary file %s for %s: %s\",\n-\t\t      freq->tmpfile, freq->filename, strerror(errno));\n-\t\tgoto abort;\n-\t}\n-\n-\tmemset(&freq->stream, 0, sizeof(freq->stream));\n-\n-\tgit_inflate_init(&freq->stream);\n-\n-\tgit_SHA1_Init(&freq->c);\n-\n-\tfreq->url = get_remote_object_url(base_url, hex, 0);\n-\n-\t/*\n-\t * If a previous temp file is present, process what was already\n-\t * fetched.\n-\t */\n-\tprevlocal = open(prevfile, O_RDONLY);\n-\tif (prevlocal != -1) {\n-\t\tdo {\n-\t\t\tprev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);\n-\t\t\tif (prev_read>0) {\n-\t\t\t\tif (fwrite_sha1_file(prev_buf,\n-\t\t\t\t\t\t     1,\n-\t\t\t\t\t\t     prev_read,\n-\t\t\t\t\t\t     freq) == prev_read) {\n-\t\t\t\t\tprev_posn += prev_read;\n-\t\t\t\t} else {\n-\t\t\t\t\tprev_read = -1;\n-\t\t\t\t}\n-\t\t\t}\n-\t\t} while (prev_read > 0);\n-\t\tclose(prevlocal);\n-\t}\n-\tunlink_or_warn(prevfile);\n-\n-\t/*\n-\t * Reset inflate/SHA1 if there was an error reading the previous temp\n-\t * file; also rewind to the beginning of the local file.\n-\t */\n-\tif (prev_read == -1) {\n-\t\tmemset(&freq->stream, 0, sizeof(freq->stream));\n-\t\tgit_inflate_init(&freq->stream);\n-\t\tgit_SHA1_Init(&freq->c);\n-\t\tif (prev_posn>0) {\n-\t\t\tprev_posn = 0;\n-\t\t\tlseek(freq->localfile, 0, SEEK_SET);\n-\t\t\tif (ftruncate(freq->localfile, 0) < 0) {\n-\t\t\t\terror(\"Couldn't truncate temporary file %s for %s: %s\",\n-\t\t\t\t\t  freq->tmpfile, freq->filename, strerror(errno));\n-\t\t\t\tgoto abort;\n-\t\t\t}\n-\t\t}\n-\t}\n-\n-\tfreq->slot = get_active_slot();\n-\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);\n-\tcurl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);\n-\n-\t/*\n-\t * If we have successfully processed data from a previous fetch\n-\t * attempt, only fetch the data we don't already have.\n-\t */\n-\tif (prev_posn>0) {\n-\t\tif (http_is_verbose)\n-\t\t\tfprintf(stderr,\n-\t\t\t\t\"Resuming fetch of object %s at byte %ld\\n\",\n-\t\t\t\thex, prev_posn);\n-\t\tsprintf(range, \"Range: bytes=%ld-\", prev_posn);\n-\t\trange_header = curl_slist_append(range_header, range);\n-\t\tcurl_easy_setopt(freq->slot->curl,\n-\t\t\t\t CURLOPT_HTTPHEADER, range_header);\n-\t}\n-\n-\treturn freq;\n-\n-abort:\n-\tfree(filename);\n-\tfree(freq->url);\n-\tfree(freq);\n-\treturn NULL;\n-}\n-\n-void process_http_object_request(struct http_object_request *freq)\n-{\n-\tif (freq->slot == NULL)\n-\t\treturn;\n-\tfreq->curl_result = freq->slot->curl_result;\n-\tfreq->http_code = freq->slot->http_code;\n-\tfreq->slot = NULL;\n-}\n-\n-int finish_http_object_request(struct http_object_request *freq)\n-{\n-\tstruct stat st;\n-\n-\tclose(freq->localfile);\n-\tfreq->localfile = -1;\n-\n-\tprocess_http_object_request(freq);\n-\n-\tif (freq->http_code == 416) {\n-\t\tfprintf(stderr, \"Warning: requested range invalid; we may already have all the data.\\n\");\n-\t} else if (freq->curl_result != CURLE_OK) {\n-\t\tif (stat(freq->tmpfile, &st) == 0)\n-\t\t\tif (st.st_size == 0)\n-\t\t\t\tunlink_or_warn(freq->tmpfile);\n-\t\treturn -1;\n-\t}\n-\n-\tgit_inflate_end(&freq->stream);\n-\tgit_SHA1_Final(freq->real_sha1, &freq->c);\n-\tif (freq->zret != Z_STREAM_END) {\n-\t\tunlink_or_warn(freq->tmpfile);\n-\t\treturn -1;\n-\t}\n-\tif (hashcmp(freq->sha1, freq->real_sha1)) {\n-\t\tunlink_or_warn(freq->tmpfile);\n-\t\treturn -1;\n-\t}\n-\tfreq->rename =\n-\t\tmove_temp_to_file(freq->tmpfile, freq->filename);\n-\n-\treturn freq->rename;\n-}\n-\n-void abort_http_object_request(struct http_object_request *freq)\n-{\n-\tunlink_or_warn(freq->tmpfile);\n-\n-\trelease_http_object_request(freq);\n-}\n-\n-void release_http_object_request(struct http_object_request *freq)\n-{\n-\tif (freq->localfile != -1) {\n-\t\tclose(freq->localfile);\n-\t\tfreq->localfile = -1;\n-\t}\n-\tif (freq->url != NULL) {\n-\t\tfree(freq->url);\n-\t\tfreq->url = NULL;\n-\t}\n-\tfreq->slot = NULL;\n-}\ndiff --git a/http.h b/http.h\nindex 4c4e99c..511c0c4 100644\n--- a/http.h\n+++ b/http.h\n@@ -88,6 +88,10 @@ extern void add_fill_function(void *data, int (*fill)(void *));\n extern void step_active_slots(void);\n #endif\n \n+extern struct curl_slist *no_pragma_header;\n+\n+#define RANGE_HEADER_SIZE 30\n+\n extern void http_init(struct remote *remote);\n extern void http_cleanup(void);\n \n@@ -110,13 +114,6 @@ static inline int missing__target(int code, int result)\n \n #define missing_target(a) missing__target((a)->http_code, (a)->curl_result)\n \n-/* Helpers for modifying and creating URLs */\n-extern void append_remote_object_url(struct strbuf *buf, const char *url,\n-\t\t\t\t     const char *hex,\n-\t\t\t\t     int only_two_digit_prefix);\n-extern char *get_remote_object_url(const char *url, const char *hex,\n-\t\t\t\t   int only_two_digit_prefix);\n-\n /* Options for http_request_*() */\n #define HTTP_NO_CACHE\t\t1\n \n@@ -170,30 +167,4 @@ extern struct http_pack_request *new_http_pack_request(\n extern int finish_http_pack_request(struct http_pack_request *preq);\n extern void release_http_pack_request(struct http_pack_request *preq);\n \n-/* Helpers for fetching object */\n-struct http_object_request\n-{\n-\tchar *url;\n-\tchar filename[PATH_MAX];\n-\tchar tmpfile[PATH_MAX];\n-\tint localfile;\n-\tCURLcode curl_result;\n-\tchar errorstr[CURL_ERROR_SIZE];\n-\tlong http_code;\n-\tunsigned char sha1[20];\n-\tunsigned char real_sha1[20];\n-\tgit_SHA_CTX c;\n-\tz_stream stream;\n-\tint zret;\n-\tint rename;\n-\tstruct active_request_slot *slot;\n-};\n-\n-extern struct http_object_request *new_http_object_request(\n-\tconst char *base_url, unsigned char *sha1);\n-extern void process_http_object_request(struct http_object_request *freq);\n-extern int finish_http_object_request(struct http_object_request *freq);\n-extern void abort_http_object_request(struct http_object_request *freq);\n-extern void release_http_object_request(struct http_object_request *freq);\n-\n #endif /* HTTP_H */\n"},{"id":"120852","messageId":"20090817170943.180dbedd.rctay89@gmail.com","threadId":"20616","inReplyTo":"200908170725.09592.thomas.schlichter@web.de","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2009-08-17T09:09:43Z","receivedAt":"2009-08-17T09:09:43Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Mon, Aug 17, 2009 at 1:25 PM, Thomas Schlichter<thomas.schlichter@web.de> wrote:\n> Hmm. I hope you won't take this as an argument for not fixing it, because it\n> is a clear regression! I tried git version 1.6.3.4, and it works flawlessly\n> with exactly this server! Even during bisecting, some versions worked, some\n> didn't (these after the mentioned commit...)\n>\n> So I think this commit didn't only refactor things, but unintentionally\n> changed the behavior. And this must be the problem. As I'm not into this code,\n> and the refactoring was not completely trivial, I was not able to quickly find\n> the place where the behavior was changed...\n\nThomas, indeed you're right about this, the commit you noted in your\nearlier email (5424bc5) was indeed at fault.\n\nThe commit changed how urls was handled, and resulted (incorrectly) in\nan extra slash being added to the url. In other words, urls now look\nlike this:\n\n  http://server/repo.git//objects/2a/\n\nThis problem occurs only in http-push.c, because repo urls\n(http://server/repo.git) are always made to end with a slash there.\n\nBtw, on my Apache 2.2.x server, this didn't crop up when I tested,\nwhile Thomas worked with a 2.0.x.\n\nI've attached a patch below that hopefully fixes it.\n\n-- >8 --\n\nSubject: [PATCH] http.c: don't assume that urls don't end with slash\n\nMake append_remote_object_url() (and by implication,\nget_remote_object_url) use end_url_with_slash() to ensure that the url\nends with a slash.\n\nPreviously, they assumed that the url did not end with a slash and\nas a result appended a slash, sometimes errorneously.\n\nThis fixes an issue introduced in 5424bc5 (\"http*: add helper methods\nfor fetching objects (loose)\"), where the append_remote_object_url()\nimplementation in http-push.c, which assumed that urls end with a\nslash, was replaced by another one in http.c, which assumed urls did\nnot end with a slash.\n\nThe above issue was raised by Thomas Schlichter:\n\n  http://marc.info/?l=git&m=125043105231327\n\nSigned-off-by: Tay Ray Chuan <rctay89@gmail.com>\n---\n http.c |    4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/http.c b/http.c\nindex 14d5357..eb0c669 100644\n--- a/http.c\n+++ b/http.c\n@@ -719,7 +719,9 @@ void append_remote_object_url(struct strbuf *buf, const char *url,\n \t\t\t      const char *hex,\n \t\t\t      int only_two_digit_prefix)\n {\n-\tstrbuf_addf(buf, \"%s/objects/%.*s/\", url, 2, hex);\n+\tend_url_with_slash(buf, url);\n+\n+\tstrbuf_addf(buf, \"objects/%.*s/\", 2, hex);\n \tif (!only_two_digit_prefix)\n \t\tstrbuf_addf(buf, \"%s\", hex+2);\n }\n--\n1.6.4.dirty\n"},{"id":"120948","messageId":"200908171928.38999.thomas.schlichter@web.de","threadId":"20616","inReplyTo":"20090817170943.180dbedd.rctay89@gmail.com","subject":"Re: git http-push and MKCOL error (22/409)","fromName":"Thomas Schlichter","fromEmail":"thomas.schlichter@web.de","sentAt":"2009-08-17T17:28:38Z","receivedAt":"2009-08-17T17:28:38Z","isPatch":false,"sender":{"key":"thomas.schlichter@web.de","avatar":null},"body":"Hi,\n\nAm Montag 17 August 2009 11:09:43 schrieb Tay Ray Chuan:\n> Subject: [PATCH] http.c: don't assume that urls don't end with slash\n>\n> Make append_remote_object_url() (and by implication,\n> get_remote_object_url) use end_url_with_slash() to ensure that the url\n> ends with a slash.\n>\n> Previously, they assumed that the url did not end with a slash and\n> as a result appended a slash, sometimes errorneously.\n>\n> This fixes an issue introduced in 5424bc5 (\"http*: add helper methods\n> for fetching objects (loose)\"), where the append_remote_object_url()\n> implementation in http-push.c, which assumed that urls end with a\n> slash, was replaced by another one in http.c, which assumed urls did\n> not end with a slash.\n>\n> The above issue was raised by Thomas Schlichter:\n>\n>   http://marc.info/?l=git&m=125043105231327\n>\n> Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>\n\nThank you very much!\nThis patch indeed fixes my problem.\n\nYou may add my\n\n   Tested-by: Thomas Schlichter <thomas.schlichter@web.de>\n\nif you'd like to. I don't know how this is handled in the git-tree...\n\nIs there a plan in which version this patch will be released? Will there be a \n1.6.4.1 or is 1.6.5 next?\n\nKind regards,\nThomas Schlichter\n"}]}