{"thread":{"id":"34801","subject":"Stalled git cloning and possible solutions","startedAt":"2013-08-29T19:48:19Z","lastAt":"2013-09-04T01:06:48Z","messageCount":8,"participants":["V.Krishn","Jonathan Nieder","Junio C Hamano","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"226254","messageId":"201308300118.19166.vkrishn4@gmail.com","threadId":"34801","inReplyTo":null,"subject":"Stalled git cloning and possible solutions","fromName":"V.Krishn","fromEmail":"vkrishn4@gmail.com","sentAt":"2013-08-29T19:48:19Z","receivedAt":"2013-08-29T19:48:19Z","isPatch":false,"sender":{"key":"vkrishn4@gmail.com","avatar":null},"body":"Hi,\n\nQuite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans what \nbeen downloaded, and process needs re-start.\n\nIs there a way to recover or continue from already downloaded files during \ncloning ?\nPlease point me to an archive url if solution exists. (though I continue to \nsearch through them as I email this)\n\nCan there be something like:\ngit clone --use-method=rsync\n\n-- \nRegards.\nV.Krishn\n"},{"id":"226264","messageId":"20130829211034.GB4110@google.com","threadId":"34801","inReplyTo":"201308300118.19166.vkrishn4@gmail.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-08-29T21:10:34Z","receivedAt":"2013-08-29T21:10:34Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"V.Krishn wrote:\n\n> Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans what \n> been downloaded, and process needs re-start.\n>\n> Is there a way to recover or continue from already downloaded files during \n> cloning ?\n\nNo, sadly. The pack sent for a clone is generated dynamically, so\nthere's no easy way to support the equivalent of an HTTP Range request\nto resume. Someone might implement an appropriate protocol extension\nto tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\nbut for now it doesn't exist.\n\nWhat you *can* do today is create a bundle from the large repo\nsomewhere with a reliable connection and then grab that using a\nresumable transport such as HTTP. A kind person made a service to do\nthat.\n\n http://thread.gmane.org/gmane.comp.version-control.git/181380\n\nHope that helps,\nJonathan\n"},{"id":"226271","messageId":"201308300305.10440.vkrishn4@gmail.com","threadId":"34801","inReplyTo":"20130829211034.GB4110@google.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"V.Krishn","fromEmail":"vkrishn4@gmail.com","sentAt":"2013-08-29T21:35:10Z","receivedAt":"2013-08-29T21:35:10Z","isPatch":false,"sender":{"key":"vkrishn4@gmail.com","avatar":null},"body":"On Friday, August 30, 2013 02:40:34 AM you wrote:\n> V.Krishn wrote:\n> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans\n> > what been downloaded, and process needs re-start.\n> > \n> > Is there a way to recover or continue from already downloaded files\n> > during cloning ?\n> \n> No, sadly. The pack sent for a clone is generated dynamically, so\n> there's no easy way to support the equivalent of an HTTP Range request\n> to resume. Someone might implement an appropriate protocol extension\n> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\n> but for now it doesn't exist.\n\nThis is what I tried but then realized something more is needed:\n\nDuring stalled clone avoid Ctrl+c. \n1. Copy the content .i.e .git folder some other place.\n2. cd \n3. git config fetch.unpackLimit 999999\n4. git config transfer.unpackLimit 999999\n5. cat .git/config #to see if config went ok\n \n6. recover process:\n git unpack-objects -r --strict <.git/objects/pack/tmp_pack_0mSPsc\n\nTHEN... hopefully thought following should do the trick\n git pull\n OR\n git-fetch-pack\n OR\n git repack + git pull\n\nbut then something more is needed.... :)\nlike index/map file... etc for it work.\n\n> \n> What you *can* do today is create a bundle from the large repo\n> somewhere with a reliable connection and then grab that using a\n> resumable transport such as HTTP. A kind person made a service to do\n> that.\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/181380\n\nService looks nice. Hope its gets sponsors to keep it running.\n\n-- \nRegards.\nV.Krishn\n"},{"id":"226276","messageId":"xmqqvc2o16pn.fsf@gitster.dls.corp.google.com","threadId":"34801","inReplyTo":"201308300305.10440.vkrishn4@gmail.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-29T22:18:44Z","receivedAt":"2013-08-29T22:18:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"V.Krishn\" writes:\n\n> On Friday, August 30, 2013 02:40:34 AM you wrote:\n>> V.Krishn wrote:\n>> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans\n>> > what been downloaded, and process needs re-start.\n>> > \n>> > Is there a way to recover or continue from already downloaded files\n>> > during cloning ?\n>> \n>> No, sadly. The pack sent for a clone is generated dynamically, so\n>> there's no easy way to support the equivalent of an HTTP Range request\n>> to resume. Someone might implement an appropriate protocol extension\n>> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\n>> but for now it doesn't exist.\n>\n> This is what I tried but then realized something more is needed:\n>\n> During stalled clone avoid Ctrl+c. \n> 1. Copy the content .i.e .git folder some other place.\n> 2. cd \n> 3. git config fetch.unpackLimit 999999\n> 4. git config transfer.unpackLimit 999999\n\nThese two steps will not help, as negotiation between the sender and\nthe receiver is based on the commits that are known to be complete,\nand an earlier failed \"fetch\" will not (and should not) update refs\non the receiver's side.\n\n>> What you *can* do today is create a bundle from the large repo\n>> somewhere with a reliable connection and then grab that using a\n>> resumable transport such as HTTP.\n\nYes.\n\nAnother possibility is, if the project being cloned has a tag (or a\nbranch) that points at a commit back when it was smaller, do this\n\n\tgit init x &&\n cd x &&\n git fetch $that_repository $that_tag:refs/tags/back_then_i_was_small\n\nto prime the object store of a temporary repository 'x' with a\nhopefully smaller transfer, and then use it as a \"--reference\"\nrepository to the real clone.\n"},{"id":"226279","messageId":"201308300358.33272.vkrishn4@gmail.com","threadId":"34801","inReplyTo":"xmqqvc2o16pn.fsf@gitster.dls.corp.google.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"V.Krishn","fromEmail":"vkrishn4@gmail.com","sentAt":"2013-08-29T22:28:33Z","receivedAt":"2013-08-29T22:28:33Z","isPatch":false,"sender":{"key":"vkrishn4@gmail.com","avatar":null},"body":"On Friday, August 30, 2013 03:48:44 AM you wrote:\n> \"V.Krishn\" writes:\n> > On Friday, August 30, 2013 02:40:34 AM you wrote:\n> >> V.Krishn wrote:\n> >> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c\n> >> > cleans what been downloaded, and process needs re-start.\n> >> > \n> >> > Is there a way to recover or continue from already downloaded files\n> >> > during cloning ?\n> >> \n> >> No, sadly. The pack sent for a clone is generated dynamically, so\n> >> there's no easy way to support the equivalent of an HTTP Range request\n> >> to resume. Someone might implement an appropriate protocol extension\n> >> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\n> >> but for now it doesn't exist.\n> > \n> > This is what I tried but then realized something more is needed:\n> > \n> > During stalled clone avoid Ctrl+c.\n> > 1. Copy the content .i.e .git folder some other place.\n> > 2. cd \n> > 3. git config fetch.unpackLimit 999999\n> > 4. git config transfer.unpackLimit 999999\n> \n> These two steps will not help, as negotiation between the sender and\n> the receiver is based on the commits that are known to be complete,\n> and an earlier failed \"fetch\" will not (and should not) update refs\n> on the receiver's side.\n> \n> >> What you *can* do today is create a bundle from the large repo\n> >> somewhere with a reliable connection and then grab that using a\n> >> resumable transport such as HTTP.\n> \n> Yes.\n> \n> Another possibility is, if the project being cloned has a tag (or a\n> branch) that points at a commit back when it was smaller, do this\n> \n> \tgit init x &&\n> cd x &&\n> git fetch $that_repository\n> $that_tag:refs/tags/back_then_i_was_small\n> \n> to prime the object store of a temporary repository 'x' with a\n> hopefully smaller transfer, and then use it as a \"--reference\"\n> repository to the real clone.\n\nWould be nice if,\n1. the clone process downloaded all files in .git before the blobs or packing \nprocess and added a lock file like .clone and then started the packing \nprocess.\n2. Any interrupt((ctrl+c) should not delete the already dowloaded files but on \nre-clone process it should check .clone file and resume cloning.\n3. Upon finishing cloning delete .clone file.\n\n-- \nRegards.\nV.Krishn\n"},{"id":"226338","messageId":"CACsJy8CxeZGu-nudeGtAFJdzsYMjyNX01yeVOAcRV1kOjA+dkg@mail.gmail.com","threadId":"34801","inReplyTo":"20130829211034.GB4110@google.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-30T12:17:03Z","receivedAt":"2013-08-30T12:17:03Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Aug 30, 2013 at 4:10 AM, Jonathan Nieder wrote:\n> V.Krishn wrote:\n>\n>> Quite sometimes when cloning a large repo stalls, hitting Ctrl+c cleans what\n>> been downloaded, and process needs re-start.\n>>\n>> Is there a way to recover or continue from already downloaded files during\n>> cloning ?\n>\n> No, sadly. The pack sent for a clone is generated dynamically, so\n> there's no easy way to support the equivalent of an HTTP Range request\n> to resume. Someone might implement an appropriate protocol extension\n> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\n> but for now it doesn't exist.\n\nOK how about a new capability \"resume\" to upload-pack. fetch-pack can\nthen send capability \"resume[=,]\" to upload-pack. The\nfirst time it sends \"resume\" without parameters, and upload-pack will\nsend back an SHA-1 to identify the pack being transferred together\nwith a full pack as usual. When early disconnection happens, it sends\nthe received SHA-1 and the received pack's size so far. It either\nreceives the remaining part, or a full pack.\n\nWhen upload-pack gets \"resume\", it calculates a checksum of all input\nthat may impact pack generation. If the checksum matches the SHA-1\nfrom fetch-pack, it'll continue to generate the pack as usual, but\nwill skip sending the first bytes (maybe with a fake header so\nthat fetch-pack realizes this is a partial pack). If the checksum does\nnot match, it sends full pack again. I count on index-pack to spot\ncorrupt resumed pack due to bugs.\n\nThe input to calculate SHA-1 checksum includes:\n\n - the result SHA-1 list from rev-list\n - git version string\n - .git/shallow\n - replace object database\n - pack.* config\n - maybe some other variables (I haven't checked pack-objects)\n\nAnother Git implementation can generate this SHA-1 in a totally\ndifferent way and may even cache the generated pack.\n\nIf at resume time, the load balancer directs the request to another\nupload-pack that generates this SHA-1 differently, ok this won't work\n(i.e. full pack is returned). In a busy repository, some refs may have\nmoved so rev-list result at the resume time won't match any more, but\nwe can deal with that later by relaxing to allow \"want \" lines with\nSHA-1 that are reachable from current refs, not just one of the refs\n(pack v4 or reachability bitmaps help).\n-- \nDuy\n"},{"id":"226340","messageId":"CACsJy8DntFs-exeQ28=tqF3963zd1szTghZybmC6DrTNNF8Lfg@mail.gmail.com","threadId":"34801","inReplyTo":"CACsJy8CxeZGu-nudeGtAFJdzsYMjyNX01yeVOAcRV1kOjA+dkg@mail.gmail.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-30T12:41:45Z","receivedAt":"2013-08-30T12:41:45Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Aug 30, 2013 at 7:17 PM, Duy Nguyen wrote:\n> OK how about a new capability \"resume\" to upload-pack. fetch-pack can\n> then send capability \"resume[=,]\" to upload-pack. The\n> first time it sends \"resume\" without parameters, and upload-pack will\n> send back an SHA-1 to identify the pack being transferred together\n> with a full pack as usual. When early disconnection happens, it sends\n> the received SHA-1 and the received pack's size so far. It either\n> receives the remaining part, or a full pack.\n>\n> When upload-pack gets \"resume\", it calculates a checksum of all input\n> that may impact pack generation. If the checksum matches the SHA-1\n> from fetch-pack, it'll continue to generate the pack as usual, but\n> will skip sending the first bytes (maybe with a fake header so\n> that fetch-pack realizes this is a partial pack). If the checksum does\n> not match, it sends full pack again. I count on index-pack to spot\n> corrupt resumed pack due to bugs.\n>\n> The input to calculate SHA-1 checksum includes:\n>\n> - the result SHA-1 list from rev-list\n> - git version string\n> - .git/shallow\n> - replace object database\n> - pack.* config\n> - maybe some other variables (I haven't checked pack-objects)\n\nshould have tested something first before I wrote. --threads adds some\nrandomness to pack generation so it has to be --threads=1. Not sure if\ngit repository hosts are happy with it..\n\n> Another Git implementation can generate this SHA-1 in a totally\n> different way and may even cache the generated pack.\n>\n> If at resume time, the load balancer directs the request to another\n> upload-pack that generates this SHA-1 differently, ok this won't work\n> (i.e. full pack is returned). In a busy repository, some refs may have\n> moved so rev-list result at the resume time won't match any more, but\n> we can deal with that later by relaxing to allow \"want \" lines with\n> SHA-1 that are reachable from current refs, not just one of the refs\n> (pack v4 or reachability bitmaps help).\n> --\n> Duy\n\n\n\n-- \nDuy\n"},{"id":"226694","messageId":"201309040636.48290.vkrishn4@gmail.com","threadId":"34801","inReplyTo":"xmqqvc2o16pn.fsf@gitster.dls.corp.google.com","subject":"Re: Stalled git cloning and possible solutions","fromName":"V.Krishn","fromEmail":"vkrishn4@gmail.com","sentAt":"2013-09-04T01:06:48Z","receivedAt":"2013-09-04T01:06:48Z","isPatch":false,"sender":{"key":"vkrishn4@gmail.com","avatar":null},"body":"On Friday, August 30, 2013 03:48:44 AM you wrote:\n> \"V.Krishn\" writes:\n> > On Friday, August 30, 2013 02:40:34 AM you wrote:\n> >> V.Krishn wrote:\n> >> > Quite sometimes when cloning a large repo stalls, hitting Ctrl+c\n> >> > cleans what been downloaded, and process needs re-start.\n> >> > \n> >> > Is there a way to recover or continue from already downloaded files\n> >> > during cloning ?\n> >> \n> >> No, sadly. The pack sent for a clone is generated dynamically, so\n> >> there's no easy way to support the equivalent of an HTTP Range request\n> >> to resume. Someone might implement an appropriate protocol extension\n> >> to tackle this (e.g., peff's seed-with-clone.bundle hack) some day,\n> >> but for now it doesn't exist.\n> > \n> > This is what I tried but then realized something more is needed:\n> > \n> > During stalled clone avoid Ctrl+c.\n> > 1. Copy the content .i.e .git folder some other place.\n> > 2. cd \n> > 3. git config fetch.unpackLimit 999999\n> > 4. git config transfer.unpackLimit 999999\n> \n> These two steps will not help, as negotiation between the sender and\n> the receiver is based on the commits that are known to be complete,\n> and an earlier failed \"fetch\" will not (and should not) update refs\n> on the receiver's side.\n> \n> >> What you *can* do today is create a bundle from the large repo\n> >> somewhere with a reliable connection and then grab that using a\n> >> resumable transport such as HTTP.\n> \n> Yes.\n> \n> Another possibility is, if the project being cloned has a tag (or a\n> branch) that points at a commit back when it was smaller, do this\n> \n> \tgit init x &&\n> cd x &&\n> git fetch $that_repository\n> $that_tag:refs/tags/back_then_i_was_small\n> \n> to prime the object store of a temporary repository 'x' with a\n> hopefully smaller transfer, and then use it as a \"--reference\"\n> repository to the real clone.\n\nWhat more files/info would be needed.\nI noticed the tmp_pack_xxxxxx may not have object type commit/tree.\nDo I need to manually create .git/refs..\n\nI was wondering the following would further help in recovering.\n\nA\n1. If pack file was created in sequence to commit history(date), i.e \nblob+commit+tree....tags...+blob+commit+tree. \nalso if in parallel idx was also created or atleast a tmp idx.\n2. Update other files in .git dir before pack process.\n (as stated in previous email).\n3. Objects are named like datestamp(epoch)+sha1 \n and stored in epoch directory. (date fmt can be yymmdd).\n (this might break back-compat)\n4. Add \"git fsck --defrag [1..4]\" \n #this can take another parameter like level, \n applying various heuristic optimization.\n\nB\nAnother option would be:\ngit clone --use-method=rsync\nthis would transfer files as is in .git dir (ones necessary).\nAnd run `git gc` or any other housekeeping upon completion.\nThis method would allow resuming.\nCons:\n Any change in pack file on server during download becomes a potential issue.\n\nThe clone resume may not be a priority but if a minor changes can help in \nrecovery, this would be nice. \n\nI still like the bundle method if git services made this easy.\n\n-- \nRegards.\nV.Krishn\n"}]}