{"thread":{"id":"39292","subject":"resume downloads","startedAt":"2015-05-10T21:55:27Z","lastAt":"2015-05-12T09:54:08Z","messageCount":3,"participants":["Thiago Farina","Junio C Hamano","Sitaram Chamarty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"260945","messageId":"CACnwZYeW3+LQZYV4D_+vWggeh_yoWa-C3KdssHO21sRbsEKrMg@mail.gmail.com","threadId":"39292","inReplyTo":null,"subject":"resume downloads","fromName":"Thiago Farina","fromEmail":"tfransosi@gmail.com","sentAt":"2015-05-10T21:55:27Z","receivedAt":"2015-05-10T21:55:27Z","isPatch":false,"sender":{"key":"tfransosi@gmail.com","avatar":"https://avatars.githubusercontent.com/u/970071?v=4"},"body":"Hi,\n\nIs there links to discussion on this? I mean, is resume downloads a\nfeature that is still being considered?\n\nBeing able to download huge repos like WebKit, Linux, LibreOffice in\nsmall parts seems like a good feature to me.\n\n-- \nThiago Farina\n"},{"id":"260946","messageId":"CAPc5daVwxEniz-s-6dcowQkE-bK50wJ4MOCWGkAM=u02BGtN+w@mail.gmail.com","threadId":"39292","inReplyTo":"CACnwZYeW3+LQZYV4D_+vWggeh_yoWa-C3KdssHO21sRbsEKrMg@mail.gmail.com","subject":"Re: resume downloads","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-05-10T22:19:34Z","receivedAt":"2015-05-10T22:19:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The current thinking is to model this after the \"repo\" tool.\nPrepare a reasonably up-to-date bundle file on the server side,\nadd a protocol capability to advertise the URL to download that\nbundle from upload-pack, and have \"git clone\" to pay attention to it.\n\nThen, a \"git clone\" could become:\n\n - If the capability advertises such a prebuilt bundle, spawn \"curl\"\n or \"wget\" internally to fetch it. This can be resumed when the\n connection goes down and will grab majority of the data necessary.\n\n - Extract the bundle into temporary area inside .git/refs/ to help\n the next step.\n\n - Internally do a \"git fetch\" to the original server. Thanks to the\n bundle transfer that has already happened, this step will become\n a small incremental update.\n\n - Then prune away the temporary .git/refs/ refs that were in the\n bundle, as these are not the up-to-date refs that exist on the\n server side.\n\nA few points that need to be considered by whoever is doing this\nare:\n\n - Where to download the bundle, so that after killing \"git clone\"\n that is still in the bundle-download phase, the next invocation\n of \"git clone\" can notice and resume the bundle-download;\n\n - What kind of transfer protocols do we want to support? Is http\n and https from CDN sufficient? In other words, what exactly\n should the new capability say to point at the prebuilt bundle?\n\nThese (and probably there are several others) are not something\nthat \"repo\" does not have to worry about, but would become\nissues when we try to fold this into \"git clone\".\n\n\n\nOn Sun, May 10, 2015 at 2:55 PM, Thiago Farina wrote:\n> Hi,\n>\n> Is there links to discussion on this? I mean, is resume downloads a\n> feature that is still being considered?\n>\n> Being able to download huge repos like WebKit, Linux, LibreOffice in\n> small parts seems like a good feature to me.\n>\n> --\n> Thiago Farina\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"261046","messageId":"5551CDC0.10101@gmail.com","threadId":"39292","inReplyTo":"CAPc5daVwxEniz-s-6dcowQkE-bK50wJ4MOCWGkAM=u02BGtN+w@mail.gmail.com","subject":"Re: resume downloads","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2015-05-12T09:54:08Z","receivedAt":"2015-05-12T09:54:08Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On 05/11/2015 03:49 AM, Junio C Hamano wrote:\n> The current thinking is to model this after the \"repo\" tool.\n> Prepare a reasonably up-to-date bundle file on the server side,\n\n\n\nFor people using gitolite, the server side issues of generating a\nreasonably up-to-date bundle *and* enabling it for resumable download\nusing rsync (with the same ssh key used to gain gitolite access), can\nall be handled by gitolite.\n\n\n\nOf course the client side issues still remain; gitolite can't help\nthere.\n\n> add a protocol capability to advertise the URL to download that\n> bundle from upload-pack, and have \"git clone\" to pay attention to it.\n> \n> Then, a \"git clone\" could become:\n> \n> - If the capability advertises such a prebuilt bundle, spawn \"curl\"\n> or \"wget\" internally to fetch it. This can be resumed when the\n> connection goes down and will grab majority of the data necessary.\n> \n> - Extract the bundle into temporary area inside .git/refs/ to help\n> the next step.\n> \n> - Internally do a \"git fetch\" to the original server. Thanks to the\n> bundle transfer that has already happened, this step will become\n> a small incremental update.\n> \n> - Then prune away the temporary .git/refs/ refs that were in the\n> bundle, as these are not the up-to-date refs that exist on the\n> server side.\n> \n> A few points that need to be considered by whoever is doing this\n> are:\n> \n> - Where to download the bundle, so that after killing \"git clone\"\n> that is still in the bundle-download phase, the next invocation\n> of \"git clone\" can notice and resume the bundle-download;\n> \n> - What kind of transfer protocols do we want to support? Is http\n> and https from CDN sufficient? In other words, what exactly\n> should the new capability say to point at the prebuilt bundle?\n> \n> These (and probably there are several others) are not something\n> that \"repo\" does not have to worry about, but would become\n> issues when we try to fold this into \"git clone\".\n> \n> \n> \n> On Sun, May 10, 2015 at 2:55 PM, Thiago Farina wrote:\n>> Hi,\n>>\n>> Is there links to discussion on this? I mean, is resume downloads a\n>> feature that is still being considered?\n>>\n>> Being able to download huge repos like WebKit, Linux, LibreOffice in\n>> small parts seems like a good feature to me.\n>>\n>> --\n>> Thiago Farina\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n> \n"}]}