{"thread":{"id":"23357","subject":"git clone: very long \"resolving deltas\" phase","startedAt":"2010-04-06T14:18:53Z","lastAt":"2010-04-12T15:31:47Z","messageCount":39,"participants":["Vitaly Berov","Matthieu Moy","Vitaly","Andreas Ericsson","Nicolas Pitre","Ilari Liusvaara","Marat Radchenko","Jakub Narebski","Sverre Rabbelier"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"138752","messageId":"loom.20100406T161552-321@post.gmane.org","threadId":"23357","inReplyTo":null,"subject":"git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-06T14:18:53Z","receivedAt":"2010-04-06T14:18:53Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"We have quite a large repository and \"git clone\" takes about 6 hours. Herewith \n\"resolving deltas\" takes most of the time.\nWhat git does at this stage and how can we optimize it?\n"},{"id":"138754","messageId":"vpqljd062xb.fsf@bauges.imag.fr","threadId":"23357","inReplyTo":"loom.20100406T161552-321@post.gmane.org","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-04-06T15:01:04Z","receivedAt":"2010-04-06T15:01:04Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Vitaly Berov writes:\n\n> We have quite a large repository and \"git clone\" takes about 6 hours. Herewith \n> \"resolving deltas\" takes most of the time.\n> What git does at this stage and how can we optimize it?\n\nDoes running \"git gc\" (long, but done once and for all) on the server\nhelp?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"138756","messageId":"4BBB5304.1090709@gmail.com","threadId":"23357","inReplyTo":"vpqljd062xb.fsf@bauges.imag.fr","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-06T15:28:04Z","receivedAt":"2010-04-06T15:28:04Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/06/2010 07:01 PM, Matthieu Moy wrote:\n> Vitaly Berov writes:\n>\n>> We have quite a large repository and \"git clone\" takes about 6 hours. Herewith\n>> \"resolving deltas\" takes most of the time.\n>> What git does at this stage and how can we optimize it?\n>\n> Does running \"git gc\" (long, but done once and for all) on the server\n> help?\n>\nDidn't try this one, but I'll give it a try, thanks.\n\nAnd what does this stage do?\n"},{"id":"138757","messageId":"4BBB5346.6070607@gmail.com","threadId":"23357","inReplyTo":"vpqljd062xb.fsf@bauges.imag.fr","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-06T15:29:10Z","receivedAt":"2010-04-06T15:29:10Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"I didn't try this, but I'll give it a try, thanks.\n\nAnd what does this stage mean?\n\nOn 04/06/2010 07:01 PM, Matthieu Moy wrote:\n> Vitaly Berov writes:\n>\n> \n>> We have quite a large repository and \"git clone\" takes about 6 hours. Herewith\n>> \"resolving deltas\" takes most of the time.\n>> What git does at this stage and how can we optimize it?\n>> \n> Does running \"git gc\" (long, but done once and for all) on the server\n> help?\n>\n> \n"},{"id":"138758","messageId":"4BBB53F5.9010106@op5.se","threadId":"23357","inReplyTo":"4BBB5346.6070607@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2010-04-06T15:32:05Z","receivedAt":"2010-04-06T15:32:05Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 04/06/2010 05:29 PM, Vitaly wrote:\n> I didn't try this, but I'll give it a try, thanks.\n> \n> And what does this stage mean?\n> \n\nIt means the server is busy creating a packfile to send\nover the wire. If you pack the repository before cloning\nfrom it, deltas from the packfile will simply be copied\ninto the new pack. This will provide a huge speedboost,\nso make sure to repack the repository on the server every\nonce in a while.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"138761","messageId":"v2qecffb261004060856qcb769a06vf40ad2a75a3aa4fd@mail.gmail.com","threadId":"23357","inReplyTo":"q2mec874dac1004060850r5eaa41fak2ba9889d07794651@mail.gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-06T15:56:36Z","receivedAt":"2010-04-06T15:56:36Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Why does git compute checksums on the client side? Isn't it already\ncalculated on the \"server\" side?\n\n2010/4/6 Shawn Pearce :\n> Nope, the resolving deltas phase is about computing the checksums of each\n> object on the client side of the connection.  Repacking the server might\n> have little impact on this phase, other than maybe to reduce the size and\n> thus the disk io required to scan the entire pack.\n>\n> On Apr 6, 2010 9:32 AM, \"Andreas Ericsson\" wrote:\n>> On 04/06/2010 05:29 PM, Vitaly wrote:\n>>> I didn't try this, but I'll give it a try, thanks.\n>>>\n>>> And what does this stage mean?\n>>>\n>>\n>> It means the server is busy creating a packfile to send\n>> over the wire. If you pack the repository before cloning\n>> from it, deltas from the packfile will simply be copied\n>> into the new pack. This will provide a huge speedboost,\n>> so make sure to repack the repository on the server every\n>> once in a while.\n>>\n>> --\n>> Andreas Ericsson andreas.ericsson@op5.se\n>> OP5 AB www.op5.se\n>> Tel: +46 8-230225 Fax: +46 8-230231\n>>\n>> Considering the successes of the wars on alcohol, poverty, drugs and\n>> terror, I think we should give some serious thought to declaring war\n>> on peace.\n>> --\n>> To unsubscribe from this list: send the line \"unsubscribe git\" in\n>> the body of a message to majordomo@vger.kernel.org\n>> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n"},{"id":"138777","messageId":"alpine.LFD.2.00.1004061701250.7232@xanadu.home","threadId":"23357","inReplyTo":"vpqljd062xb.fsf@bauges.imag.fr","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-06T21:01:40Z","receivedAt":"2010-04-06T21:01:40Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 6 Apr 2010, Matthieu Moy wrote:\n\n> Vitaly Berov writes:\n> \n> > We have quite a large repository and \"git clone\" takes about 6 hours. Herewith \n> > \"resolving deltas\" takes most of the time.\n> > What git does at this stage and how can we optimize it?\n> \n> Does running \"git gc\" (long, but done once and for all) on the server\n> help?\n\nNo, that won't help.\n\n\nNicolas\n"},{"id":"138778","messageId":"alpine.LFD.2.00.1004061702340.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBB53F5.9010106@op5.se","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-06T21:05:28Z","receivedAt":"2010-04-06T21:05:28Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 6 Apr 2010, Andreas Ericsson wrote:\n\n> On 04/06/2010 05:29 PM, Vitaly wrote:\n> > I didn't try this, but I'll give it a try, thanks.\n> > \n> > And what does this stage mean?\n> > \n> \n> It means the server is busy creating a packfile to send\n> over the wire.\n\nNo.\n\nThe \"Resolving deltas\" is performed locally, when Git is actually \nexpanding all the deltas in the received pack to find the actual SHA1 of \nthe resulting object in order to create the pack index.\n\n\nNicolas\n"},{"id":"138779","messageId":"alpine.LFD.2.00.1004061705390.7232@xanadu.home","threadId":"23357","inReplyTo":"v2qecffb261004060856qcb769a06vf40ad2a75a3aa4fd@mail.gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-06T21:09:04Z","receivedAt":"2010-04-06T21:09:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 6 Apr 2010, Vitaly Berov wrote:\n\n> Why does git compute checksums on the client side? Isn't it already\n> calculated on the \"server\" side?\n\nYes. But Git clients can't trust the server like that.\n\nThe only way to make sure the server didn't send you crap data, or worse \nmaliciously altered data, is actually to not transfer any checksum data \nbut to recompute and validate the received payload locally.\n\nThis being said, you should never have to wait 6 hours for that phase to \ncomplete. It is typically a matter of minutes if not seconds.\n\n\nNicolas\n"},{"id":"138781","messageId":"alpine.LFD.2.00.1004061709360.7232@xanadu.home","threadId":"23357","inReplyTo":"loom.20100406T161552-321@post.gmane.org","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-06T21:10:39Z","receivedAt":"2010-04-06T21:10:39Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 6 Apr 2010, Vitaly Berov wrote:\n\n> We have quite a large repository and \"git clone\" takes about 6 hours. Herewith \n> \"resolving deltas\" takes most of the time.\n\nThis simply makes no sense.\n\nIs this repository publicly clonable?\n\n\nNicolas\n"},{"id":"138805","messageId":"4BBC1E15.9040403@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004061705390.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T05:54:29Z","receivedAt":"2010-04-07T05:54:29Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"I suspected the security reasons.\n\nOk, we work in trusted environment. How can we turn this behavior off?\n\nVitaly\n\nOn 04/07/2010 01:09 AM, Nicolas Pitre wrote:\n> On Tue, 6 Apr 2010, Vitaly Berov wrote:\n>\n>> Why does git compute checksums on the client side? Isn't it already\n>> calculated on the \"server\" side?\n>\n> Yes. But Git clients can't trust the server like that.\n>\n> The only way to make sure the server didn't send you crap data, or worse\n> maliciously altered data, is actually to not transfer any checksum data\n> but to recompute and validate the received payload locally.\n>\n> This being said, you should never have to wait 6 hours for that phase to\n> complete. It is typically a matter of minutes if not seconds.\n>\n>\n> Nicolas\n"},{"id":"138806","messageId":"4BBC1E47.80308@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004061705390.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T05:55:19Z","receivedAt":"2010-04-07T05:55:19Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"I suspected the security reasons.\n\nOk, we work in trusted environment. How can we turn this behavior off?\n\nOn 04/07/2010 01:09 AM, Nicolas Pitre wrote:\n> On Tue, 6 Apr 2010, Vitaly Berov wrote:\n>\n> \n>> Why does git compute checksums on the client side? Isn't it already\n>> calculated on the \"server\" side?\n>> \n> Yes. But Git clients can't trust the server like that.\n>\n> The only way to make sure the server didn't send you crap data, or worse\n> maliciously altered data, is actually to not transfer any checksum data\n> but to recompute and validate the received payload locally.\n>\n> This being said, you should never have to wait 6 hours for that phase to\n> complete. It is typically a matter of minutes if not seconds.\n>\n>\n> Nicolas\n>\n> \n"},{"id":"138808","messageId":"4BBC1ED3.6020008@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004061709360.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T05:57:39Z","receivedAt":"2010-04-07T05:57:39Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Hmm, what does it mean - \"makes no sense\"? It works as it works - for \nseveral hours.\n\nNo, we work in a trusted environment. Our repository isn't open for \nexternal people.\n\nOn 04/07/2010 01:10 AM, Nicolas Pitre wrote:\n> On Tue, 6 Apr 2010, Vitaly Berov wrote:\n>\n> \n>> We have quite a large repository and \"git clone\" takes about 6 hours. Herewith\n>> \"resolving deltas\" takes most of the time.\n>> \n> This simply makes no sense.\n>\n> Is this repository publicly clonable?\n>\n>\n> Nicolas\n>\n> \n"},{"id":"138811","messageId":"20100407080049.GA14041@LK-Perkele-V2.elisa-laajakaista.fi","threadId":"23357","inReplyTo":"4BBC1E15.9040403@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2010-04-07T08:00:50Z","receivedAt":"2010-04-07T08:00:50Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Wed, Apr 07, 2010 at 09:54:29AM +0400, Vitaly Berov wrote:\n> I suspected the security reasons.\n> \n> Ok, we work in trusted environment. How can we turn this behavior off?\n \nIt can't be turned off. Protocol requires client to recompute hashes\nas they are not explicitly available in transport stream (must be inferred\ninstead).\n\n> >This being said, you should never have to wait 6 hours for that phase to\n> >complete. It is typically a matter of minutes if not seconds.\n\nThe reasons why it might take 6 hours (offhand from memory):\n\n- Extremely large repo\n- Very large files in repo pushing client into swap.\n\n-Ilari\n"},{"id":"138812","messageId":"4BBC3EEC.6010702@gmail.com","threadId":"23357","inReplyTo":"20100407080049.GA14041@LK-Perkele-V2.elisa-laajakaista.fi","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T08:14:36Z","receivedAt":"2010-04-07T08:14:36Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Too bad..\nYes, we really have a very large repo with binary files.\n\nSo, as far as I understand, the fastest way is to use rsync or smth like \nthis instead of \"git clone\".\n\nP.S. Btw, how can I ask for a feature of incorporating hashes into \ntransport stream in trusted environments?\n\nOn 04/07/2010 12:00 PM, Ilari Liusvaara wrote:\n> On Wed, Apr 07, 2010 at 09:54:29AM +0400, Vitaly Berov wrote:\n> \n>> I suspected the security reasons.\n>>\n>> Ok, we work in trusted environment. How can we turn this behavior off?\n>> \n>\n> It can't be turned off. Protocol requires client to recompute hashes\n> as they are not explicitly available in transport stream (must be inferred\n> instead).\n>\n> \n>>> This being said, you should never have to wait 6 hours for that phase to\n>>> complete. It is typically a matter of minutes if not seconds.\n>>> \n> The reasons why it might take 6 hours (offhand from memory):\n>\n> - Extremely large repo\n> - Very large files in repo pushing client into swap.\n>\n> -Ilari\n>\n> \n"},{"id":"138815","messageId":"20100407090016.GA14340@LK-Perkele-V2.elisa-laajakaista.fi","threadId":"23357","inReplyTo":"4BBC3EEC.6010702@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2010-04-07T09:00:16Z","receivedAt":"2010-04-07T09:00:16Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Wed, Apr 07, 2010 at 12:14:36PM +0400, Vitaly wrote:\n> Too bad..\n> Yes, we really have a very large repo with binary files.\n\nLarge binary files are the worst. I think that disabling deltification\n('-delta' as attribute[*]) on them might actually help somewhat...\n\n> P.S. Btw, how can I ask for a feature of incorporating hashes into\n> transport stream in trusted environments?\n\nOn this mailing list. But as a tip: don't bother: It is by far too\nlarge change relative to any possible benefit. \n\n[*] I think 'info/attributes' on server influences wheither\nthose objects are attempted to be deltified or not.\n\n-Ilari\n"},{"id":"138816","messageId":"loom.20100407T112028-388@post.gmane.org","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004061702340.7232@xanadu.home","subject":"Re: git clone: very long "resolving deltas" phase","fromName":"Marat Radchenko","fromEmail":"marat@slonopotamus.org","sentAt":"2010-04-07T09:22:37Z","receivedAt":"2010-04-07T09:22:37Z","isPatch":false,"sender":{"key":"marat@slonopotamus.org","avatar":"https://avatars.githubusercontent.com/u/92637?v=4"},"body":"Nicolas Pitre fluxnic.net> writes:\n> The \"Resolving deltas\" is performed locally, when Git is actually \n> expanding all the deltas in the received pack to find the actual SHA1 of \n> the resulting object in order to create the pack index.\nIs there any technical limitation why it cannot be done simultaniously with\nfetch (piped or whatever), instead of a separate step after fetch?\n"},{"id":"138817","messageId":"m31veriozm.fsf@localhost.localdomain","threadId":"23357","inReplyTo":"4BBC3EEC.6010702@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-04-07T09:37:20Z","receivedAt":"2010-04-07T09:37:20Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Please do not toppost.\n\nVitaly writes:\n\n> Too bad..\n> Yes, we really have a very large repo with binary files.\n> \n> So, as far as I understand, the fastest way is to use rsync or smth\n> like this instead of \"git clone\".\n> \n> P.S. Btw, how can I ask for a feature of incorporating hashes into\n> transport stream in trusted environments?\n\nIf you have very large binary files, perhaps git-bigfiles fork would\nhelp you: http://caca.zoy.org/wiki/git-bigfiles\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"138823","messageId":"alpine.LFD.2.00.1004070842040.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBC1E47.80308@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T12:42:53Z","receivedAt":"2010-04-07T12:42:53Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Vitaly wrote:\n\n> I suspected the security reasons.\n> \n> Ok, we work in trusted environment. How can we turn this behavior off?\n\nyou can't. This is fundamental to the Git native protocol.\n\n\nNicolas\n"},{"id":"138824","messageId":"alpine.LFD.2.00.1004070842580.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBC1ED3.6020008@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T12:55:09Z","receivedAt":"2010-04-07T12:55:09Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Vitaly wrote:\n\n> Hmm, what does it mean - \"makes no sense\"? It works as it works - for several\n> hours.\n\nAs I said, several hours for this operation makes no sense. This should \ntake minutes if no seconds. *This* is what needs fixing.\n\n> No, we work in a trusted environment. Our repository isn't open for external\n> people.\n\nI was asking that because that would have helped me (or any other Git \ndeveloper) analyse the issue and provide a fix.\n\nOK then. What happens if you do the following on the server machine \nwhere the repository is stored:\n\n\tgit repack -a -f -d\n\nHow long does this take?\n\nHow long does the \"Resolving deltas\" take when cloning this repacked \nrepository? (don't wait more than 10 minutes for it).\n\nIf the \"Resolving deltas\" takes more than 10 minutes, could you capture \na strace dump from that process during a minute or so and post it here?\n\nHmmm. Is this on Linux or Windows?\n\n\nNicolas\n"},{"id":"138826","messageId":"alpine.LFD.2.00.1004070952290.7232@xanadu.home","threadId":"23357","inReplyTo":"20100407080049.GA14041@LK-Perkele-V2.elisa-laajakaista.fi","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T14:08:55Z","receivedAt":"2010-04-07T14:08:55Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Ilari Liusvaara wrote:\n\n> On Wed, Apr 07, 2010 at 09:54:29AM +0400, Vitaly Berov wrote:\n> > I suspected the security reasons.\n> > \n> > Ok, we work in trusted environment. How can we turn this behavior off?\n> \n> It can't be turned off. Protocol requires client to recompute hashes\n> as they are not explicitly available in transport stream (must be inferred\n> instead).\n> \n> > >This being said, you should never have to wait 6 hours for that phase to\n> > >complete. It is typically a matter of minutes if not seconds.\n> \n> The reasons why it might take 6 hours (offhand from memory):\n> \n> - Extremely large repo\n\nSix hours is still way out of the expected computational requirement. \nThat's an expected time for an aggressive repack for example, where \n_each_ delta is attempted against a different base up to 250 times. \nBut when indexing a fetched pack, each delta is expected to be computed \nonly once.\n\n> - Very large files in repo pushing client into swap.\n\nThis shouldn't happen since commit 92392b4a which provide a cap on \nmemory usage during the delta resolution process.\n\nSo without a look at the actual repository causing this pathological \nbehavior it is hard to guess what the issue and the required fix might \nbe.\n\n\nNicolas\n"},{"id":"138827","messageId":"alpine.LFD.2.00.1004071009180.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBC3EEC.6010702@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T14:20:41Z","receivedAt":"2010-04-07T14:20:41Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Vitaly wrote:\n\n> Too bad..\n> Yes, we really have a very large repo with binary files.\n> \n> So, as far as I understand, the fastest way is to use rsync or smth like this\n> instead of \"git clone\".\n\nYou should still be able to use 'git clone' with a rsync:// style URL.\n\n> P.S. Btw, how can I ask for a feature of incorporating hashes into transport\n> stream in trusted environments?\n\nAs I'm trying to make you understand repeatedly now, this shouldn't be \nneeded. A real fix for the bad behavior would be in order before \npapering over it.\n\nIf the large binary blobs are the source of the clone problem, then they \nwill cause the same problems with other commands such as 'git diff' or \neven 'git checkout' later on. So that \"feature\" you're asking for is \nmisguided.\n\nWhat you might try on your client machines is this:\n\n\tgit config --global core.deltaBaseCacheLimit 256m\n\nbefore doing a clone.\n\n\nNicolas\n"},{"id":"138829","messageId":"s2kfabb9a1e1004070729j617d1f98r295723a54a30cd6d@mail.gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004070952290.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2010-04-07T14:29:31Z","receivedAt":"2010-04-07T14:29:31Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Wed, Apr 7, 2010 at 09:08, Nicolas Pitre wrote:\n> This shouldn't happen since commit 92392b4a which provide a cap on\n> memory usage during the delta resolution process.\n\nWhich made me think of asking:\n\nVitaly, what version of git are you running? Both client and server side please.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"138830","messageId":"4BBC981F.1000802@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004071009180.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T14:35:11Z","receivedAt":"2010-04-07T14:35:11Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/07/2010 06:20 PM, Nicolas Pitre wrote:\n>> P.S. Btw, how can I ask for a feature of incorporating hashes into transport\n>> stream in trusted environments?\n>> \n> As I'm trying to make you understand repeatedly now, this shouldn't be\n> needed. A real fix for the bad behavior would be in order before\n> papering over it.\n> \nNicolas, my post have been written before I received your message about \nreproducing and \"stracing\" the problem. I caught your idea and now \nreproducing the problem.\nMy estimate is tomorrow (repack takes quite a long time).\n\nVitaly\n"},{"id":"138833","messageId":"4BBC988C.2030606@gmail.com","threadId":"23357","inReplyTo":"s2kfabb9a1e1004070729j617d1f98r295723a54a30cd6d@mail.gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-07T14:37:00Z","receivedAt":"2010-04-07T14:37:00Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/07/2010 06:29 PM, Sverre Rabbelier wrote:\n> Heya,\n>\n> On Wed, Apr 7, 2010 at 09:08, Nicolas Pitre wrote:\n> \n>> This shouldn't happen since commit 92392b4a which provide a cap on\n>> memory usage during the delta resolution process.\n>> \n> Which made me think of asking:\n>\n> Vitaly, what version of git are you running? Both client and server side please.\n> \ngit version 1.7.0.4, both sides.\n"},{"id":"138834","messageId":"alpine.LFD.2.00.1004071022140.7232@xanadu.home","threadId":"23357","inReplyTo":"loom.20100407T112028-388@post.gmane.org","subject":"Re: git clone: very long "resolving deltas" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T14:40:28Z","receivedAt":"2010-04-07T14:40:28Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Marat Radchenko wrote:\n\n> Nicolas Pitre fluxnic.net> writes:\n> > The \"Resolving deltas\" is performed locally, when Git is actually \n> > expanding all the deltas in the received pack to find the actual SHA1 of \n> > the resulting object in order to create the pack index.\n> Is there any technical limitation why it cannot be done simultaniously with\n> fetch (piped or whatever), instead of a separate step after fetch?\n\nThe non delta compressed objects are indexed simultaneously as they're \nreceived on the wire. However this is way suboptimal to do that for \ndelta objects because\n\n1) The base object needed to resolve a given delta object might not have \n been received yet. This means in this case that the delta will have \n to be resolved later anyway, and finding out if a just received \n object might be a base object for previously received objects is \n rather costly, and even impossible if that potential base object is \n itself a delta. So it is best to figure out the delta dependencies \n only once at the end of the transfer.\n\n2) When resolving deep delta chains, it is best to start from the root \n i.e. create the result from a delta object and resolve all deltas \n with this result for base recursively, not to expand deltas \n repeatedly which would turn this process into exponential CPU usage. \n Again this can be done only when all delta objects have been \n received.\n\n\nNicolas\n"},{"id":"138838","messageId":"alpine.LFD.2.00.1004071048340.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBC981F.1000802@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-07T14:55:35Z","receivedAt":"2010-04-07T14:55:35Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 7 Apr 2010, Vitaly wrote:\n\n> Nicolas, my post have been written before I received your message about\n> reproducing and \"stracing\" the problem. I caught your idea and now\n> reproducing the problem.\n\nNo problem.\n\n> My estimate is tomorrow (repack takes quite a long time).\n\nThe repack isn't so important. If it takes that long you might simply \ninterrupt it and strace the client when \"resolving deltas\" is looking to \nbe insanely long. In reality it is best if you don't repack as the \nclient needs to cope with whatever the server throws at it and repacking \nyour repo might hide the client issue.\n\nThen playing with core.deltaBaseCacheLimit instead would be quite \ninteresting.\n\n\nNicolas\n"},{"id":"139016","messageId":"4BBECD31.9080307@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004071048340.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-09T06:46:09Z","receivedAt":"2010-04-09T06:46:09Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Hi,\n\nOn 04/07/2010 06:55 PM, Nicolas Pitre wrote:\n>\n> Then playing with core.deltaBaseCacheLimit instead would be quite\n> interesting.\nIt's difficult to play with parameters because only receiving objects \nphase takes 1.5-2 hours. But I'll try \"git config --global \ncore.deltaBaseCacheLimit 256m\" as you recommended.\n\nVitaly\n"},{"id":"139017","messageId":"4BBECE44.4040608@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004070842580.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-09T06:50:44Z","receivedAt":"2010-04-09T06:50:44Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Hi,\n\nOn 04/07/2010 04:55 PM, Nicolas Pitre wrote:\n>\n> I was asking that because that would have helped me (or any other Git\n> developer) analyse the issue and provide a fix.\n>\n> OK then. What happens if you do the following on the server machine\n> where the repository is stored:\n>\n> \tgit repack -a -f -d\n>\n> How long does this take?\n>\n> How long does the \"Resolving deltas\" take when cloning this repacked\n> repository? (don't wait more than 10 minutes for it).\nNicolas, we haven't stopped the process as you recommended, sorry for that.\n\nSo, the results: it took 37 hours. 20 hours is compressing objects \n(delta compression using up to 4 threads), 17 hours is writing objects. \nAlmost all of the time the bottleneck was a CPU.\n\nObjects amount: 3997548.\nSize of the repository: ~57Gb.\n\n> If the \"Resolving deltas\" takes more than 10 minutes, could you capture\n> a strace dump from that process during a minute or so and post it here?\nI'll capture strace later.\n\n> Hmmm. Is this on Linux or Windows?\nShort spec: Ubuntu 9.04 (64 bit), Intel(R) Core(TM)2 Quad CPU Q8400 \n2.66GHz, 8 GB of memory\n\nBy the way, we have a large amount of binary files in our rep.\n\nVitaly\n"},{"id":"139022","messageId":"vpqr5mp6o1q.fsf@bauges.imag.fr","threadId":"23357","inReplyTo":"4BBECE44.4040608@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2010-04-09T08:13:53Z","receivedAt":"2010-04-09T08:13:53Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Vitaly Berov writes:\n\n> Objects amount: 3997548.\n> Size of the repository: ~57Gb.\n[...]\n> By the way, we have a large amount of binary files in our rep.\n\nThis is clearly not the kind of repositories Git is good at. I\nencourage you to continue this discussion, and try to find a way to\nget it working, but the standard approach (probably a \"my 2 cents\"\nkind of advices, but ...) would be:\n\n* Split your repo into smaller ones (submodules ...)\n\n* Avoid versionning binary files\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"139063","messageId":"alpine.LFD.2.00.1004091508340.7232@xanadu.home","threadId":"23357","inReplyTo":"vpqr5mp6o1q.fsf@bauges.imag.fr","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-09T19:18:27Z","receivedAt":"2010-04-09T19:18:27Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 Apr 2010, Matthieu Moy wrote:\n\n> Vitaly Berov writes:\n> \n> > Objects amount: 3997548.\n> > Size of the repository: ~57Gb.\n> [...]\n> > By the way, we have a large amount of binary files in our rep.\n> \n> This is clearly not the kind of repositories Git is good at. I\n> encourage you to continue this discussion, and try to find a way to\n> get it working, but the standard approach (probably a \"my 2 cents\"\n> kind of advices, but ...) would be:\n> \n> * Split your repo into smaller ones (submodules ...)\n> \n> * Avoid versionning binary files\n\nI still think that Git ought to \"just work\" with such a repository.\nThere are things that should be done for that, like applying the \ncore.bigFileThreshold configuration variable to more places, such as \ndelta compression, object creation, diff generation, etc.\n\nOf course Git won't be as good at saving disk space in that case, but \nwhen your repo is 57GB you probably don't care much if it grows to 80GB \nbut cloning it is twice as fast.\n\nYet, I still don't think the current issue with the receiving end of a \nclone taking 6 hours in \"Resolving deltas\" is normal, independently of \ncore.bigFileThreshold.\n\n\nNicolas\n"},{"id":"139065","messageId":"alpine.LFD.2.00.1004091518500.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBECE44.4040608@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-09T19:25:21Z","receivedAt":"2010-04-09T19:25:21Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 Apr 2010, Vitaly Berov wrote:\n\n> > OK then. What happens if you do the following on the server machine\n> > where the repository is stored:\n> > \n> > \tgit repack -a -f -d\n> > \n> > How long does this take?\n> \n> So, the results: it took 37 hours. 20 hours is compressing objects (delta\n> compression using up to 4 threads), 17 hours is writing objects. Almost all of\n> the time the bottleneck was a CPU.\n> \n> Objects amount: 3997548.\n> Size of the repository: ~57Gb.\n\nOK. You probably have a size record. :-)\n\nHow big is the .pack file in .git/objects/pack/ ?\n\n> By the way, we have a large amount of binary files in our rep.\n\nHow many? How big?\n\n\nNicolas\n"},{"id":"139066","messageId":"alpine.LFD.2.00.1004091525480.7232@xanadu.home","threadId":"23357","inReplyTo":"4BBECD31.9080307@gmail.com","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-09T19:30:19Z","receivedAt":"2010-04-09T19:30:19Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 9 Apr 2010, Vitaly Berov wrote:\n\n> Hi,\n> \n> On 04/07/2010 06:55 PM, Nicolas Pitre wrote:\n> > \n> > Then playing with core.deltaBaseCacheLimit instead would be quite\n> > interesting.\n> It's difficult to play with parameters because only receiving objects phase\n> takes 1.5-2 hours.\n\nHuh... I guess that's over 100Mbps ethernet?\n\n57GB / 1.5h -> approx 10MB/s\n\n\nNicolas\n"},{"id":"139123","messageId":"4BC01B97.5080601@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004091525480.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-10T06:32:55Z","receivedAt":"2010-04-10T06:32:55Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/09/2010 11:30 PM, Nicolas Pitre wrote:\n> On Fri, 9 Apr 2010, Vitaly Berov wrote:\n>\n>> Hi,\n>>\n>> On 04/07/2010 06:55 PM, Nicolas Pitre wrote:\n>>>\n>>> Then playing with core.deltaBaseCacheLimit instead would be quite\n>>> interesting.\n>> It's difficult to play with parameters because only receiving objects phase\n>> takes 1.5-2 hours.\n>\n> Huh... I guess that's over 100Mbps ethernet?\n>\n> 57GB / 1.5h -> approx 10MB/s\n\nYes\n\nVitaly\n"},{"id":"139136","messageId":"hppb4i$a8m$1@dough.gmane.org","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004091518500.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-10T07:58:44Z","receivedAt":"2010-04-10T07:58:44Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/09/2010 11:25 PM, Nicolas Pitre wrote:\n>>\n>> Objects amount: 3997548.\n>> Size of the repository: ~57Gb.\n>\n> OK. You probably have a size record. :-)\nThat's game development. We have ~100 artists who produce text and \nbinary files as \"sources\". FYI, the \"end client version\" is ~2.5GB.\n\n> How big is the .pack file in .git/objects/pack/ ?\n~56Gb\n\n>\n>> By the way, we have a large amount of binary files in our rep.\n>\n> How many? How big?\n\nTotal amount of files ~400000, amount of binaries ~200000.\nDistribution of sizes: 5% of 4M - 32K, 5% of 32K - 12K, 12K - 6K, 6K - \n4K, 4K - 2.5K, 2.5K - 2.3K, 2.3K - 2K, the rest\n\nVitaly\n\nP.S. By the way, msysgit can't handle this repository, blocker bug is:\nhttp://code.google.com/p/msysgit/issues/detail?id=365&q=mmap&colspec=ID%20Type%20Status%20Priority%20Component%20Owner%20Summary.\nSo I thinking about stopping the evaluation, though I like git \n(especially after a long subversion experience :))\n"},{"id":"139137","messageId":"hppbhu$bc7$1@dough.gmane.org","threadId":"23357","inReplyTo":"vpqr5mp6o1q.fsf@bauges.imag.fr","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-10T08:05:54Z","receivedAt":"2010-04-10T08:05:54Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"On 04/09/2010 12:13 PM, Matthieu Moy wrote:\n> Vitaly Berov writes:\n>\n>> Objects amount: 3997548.\n>> Size of the repository: ~57Gb.\n> [...]\n>> By the way, we have a large amount of binary files in our rep.\n>\n> This is clearly not the kind of repositories Git is good at.\nHmm.. I'm looking for a good repository because I'm tired of subversion, \nPerforce isn't an option to (very expensive and even more \nuncomfortable). It seems like there only Git/Mercurial are good options. \nCan you recommend some other scms?\n\n> I encourage you to continue this discussion, and try to find a way to\n> get it working, but the standard approach (probably a \"my 2 cents\"\n> kind of advices, but ...) would be:\n>\n> * Split your repo into smaller ones (submodules ...)\n>\n> * Avoid versionning binary files\n\nI can't get rid of binary files because they are the \"sources\" of our \nartists work (the develop a game).\nSplitting a repo can be an option, but it's very inconvenient for us.\n\nVitaly\n"},{"id":"139148","messageId":"hppu9a$ugq$1@dough.gmane.org","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004070842580.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly Berov","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-10T13:25:29Z","receivedAt":"2010-04-10T13:25:29Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Hi,\n\nOn 04/07/2010 04:55 PM, Nicolas Pitre wrote:\n> On Wed, 7 Apr 2010, Vitaly wrote:\n>\n>\n> OK then. What happens if you do the following on the server machine\n> where the repository is stored:\n>\n> \tgit repack -a -f -d\n>\n> How long does this take?\n>\n> If the \"Resolving deltas\" takes more than 10 minutes, could you capture\n> a strace dump from that process during a minute or so and post it here?\n\nNicolas, I took strace and sent it to you personally.\n\nHere is the extract (99% of strace is the same):\n--------------\naccess(\"/home/vitaly/Projects/test/a1/.git/objects/0f/9a3d28766f8b767fb64166139dd65c079512de\", \nF_OK) = -1 ENOENT (No such file or directory)\npread(5, \n\"x\\234\\324\\275y\\\\Ni\\374\\377\\177\\256\\323]\\335Q\\271S\\332\\220\\\"\\n\\241\\10Q\\10!$!d/\\262\"..., \n214850, 8944159649) = 214850\naccess(\"/home/vitaly/Projects/test/a1/.git/objects/a5/5430cbc6674b56d7c2d2d81ef5b7d5c8ebdec8\", \nF_OK) = -1 ENOENT (No such file or directory)\npread(5, \"x\\234\\354\\275\\vT\\224U\\0270<\\347\\231\\v\\363\\250\\244#\\f0 \n\\\"\\\"\\\"\\312ETD\\300af\"..., 159502, 8944374506) = 159502\naccess(\"/home/vitaly/Projects/test/a1/.git/objects/e5/02b7d050d1b81ebc256234e303eac17116c9fb\", \nF_OK) = -1 ENOENT (No such file or directory)\npread(5, \n\"x\\234\\324\\274yX\\24G\\3607>\\3353\\263,\\310\\342\\\"7\\2.\\202\\342\\1\\10\\212\\210\\212\\10\\236x\\341\"..., \n61131, 8944534014) = 61131\naccess(\"/home/vitaly/Projects/test/a1/.git/objects/5b/6bdba61771e5ba63ba8b43659db1612345c2eb\", \nF_OK) = -1 ENOENT (No such file or directory)\npread(5, \n\"x\\234\\324\\275\\tX\\216Y\\3747~\\237\\323\\366DOJiyR\\236\\210B*$!\\311\\276\\223=[\\n\"..., \n236685, 8944595152) = 236685\n-----------------\nAs for me, it looks very suspicious.\n\nVitaly\n"},{"id":"139224","messageId":"alpine.LFD.2.00.1004101617460.7232@xanadu.home","threadId":"23357","inReplyTo":"hppu9a$ugq$1@dough.gmane.org","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-04-11T00:50:42Z","receivedAt":"2010-04-11T00:50:42Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 10 Apr 2010, Vitaly Berov wrote:\n\n> Hi,\n> \n> On 04/07/2010 04:55 PM, Nicolas Pitre wrote:\n> > On Wed, 7 Apr 2010, Vitaly wrote:\n> > \n> > \n> > OK then. What happens if you do the following on the server machine\n> > where the repository is stored:\n> > \n> > \tgit repack -a -f -d\n> > \n> > How long does this take?\n> > \n> > If the \"Resolving deltas\" takes more than 10 minutes, could you capture\n> > a strace dump from that process during a minute or so and post it here?\n> \n> Nicolas, I took strace and sent it to you personally.\n> \n> Here is the extract (99% of strace is the same):\n> --------------\n> access(\"/home/vitaly/Projects/test/a1/.git/objects/0f/9a3d28766f8b767fb64166139dd65c079512de\", F_OK) = -1 ENOENT (No such file or directory)\n> pread(5, \"x\\234\\324\\275y\\\\Ni\\374\\377\\177\\256\\323]\\335Q\\271S\\332\\220\\\"\\n\\241\\10Q\\10!$!d/\\262\"..., 214850, 8944159649) = 214850\n> access(\"/home/vitaly/Projects/test/a1/.git/objects/a5/5430cbc6674b56d7c2d2d81ef5b7d5c8ebdec8\", F_OK) = -1 ENOENT (No such file or directory)\n> pread(5, \"x\\234\\354\\275\\vT\\224U\\0270<\\347\\231\\v\\363\\250\\244#\\f0\\\"\\\"\\\"\\312ETD\\300af\"..., 159502, 8944374506) = 159502\n> access(\"/home/vitaly/Projects/test/a1/.git/objects/e5/02b7d050d1b81ebc256234e303eac17116c9fb\", F_OK) = -1 ENOENT (No such file or directory)\n> pread(5, \"x\\234\\324\\274yX\\24G\\3607>\\3353\\263,\\310\\342\\\"7\\2.\\202\\342\\1\\10\\212\\210\\212\\10\\236x\\341\"..., 61131, 8944534014) = 61131\n> access(\"/home/vitaly/Projects/test/a1/.git/objects/5b/6bdba61771e5ba63ba8b43659db1612345c2eb\", F_OK) = -1 ENOENT (No such file or directory)\n> pread(5, \"x\\234\\324\\275\\tX\\216Y\\3747~\\237\\323\\366DOJiyR\\236\\210B*$!\\311\\276\\223=[\\n\"..., 236685, 8944595152) = 236685\n> -----------------\n> As for me, it looks very suspicious.\n\nIt isn't. The pread() is performed for each delta object within the \nreceived pack to be resolved, and then the access() is performed to make \nsure the resolved delta doesn't match an object in loose form with the \nsame hash. Of course deltas are recursive, meaning that a delta might \nrefer to a base object which is itself a delta, and so on. And yet a \nbase object might have many delta objects referring to it. So without a \nsmart delta resolution ordering and caching, we'd end up with an \nexponential number of pread calls. However the cache size is limited to \navoid memory exhaustion with deep and wide delta trees (that's the \ncore.deltaBaseCacheLimit config variable).\n\nSo from that strace capture you sent me, we can get:\n\n$ grep access strace.txt | wc -l\n3925\n\n$ grep pread strace.txt | wc -l\n4095\n\n$ grep pread strace.txt | sort | uniq -d | wc -l\n75\n\nSo, given 3925 deltas to process, only 4095 objects were read, which is \nnot too bad. Still, 75 of them were read more than once, which means \nthey were evicted from the cache while they were still needed. The \ncore.deltaBaseCacheLimit could be increased to avoid those 75 \nduplicates. Let's have a look at a few of them:\n\n$ grep pread strace.txt | sort | uniq -cd | sort -nr\n 20 pread(5, \"x\\234\\354\\275w\\234\\34\\305\\3210<;\\263;\\233\\357nf\\357v/I{\\312\\243\\333=\\235t\\247p'\"..., 1265653, 504922895) = 1265653\n 20 pread(5, \"x\\234\\254}\\7|\\24E\\373\\377\\315\\354f\\357r\\227v\\227\\344.\\275A`\\271\\\\\\200\\20:\\204^\\245\\203\"..., 264956, 506188555) = 264956\n 6 pread(5, \"x\\234\\274}\\7xT\\305\\366\\370\\336\\231\\335\\273\\273\\251\\354f\\263\\233\\36:\\227d\\3\\201@ \\224PB\\257\"..., 253102, 49016172335) = 253102\n 6 pread(5, \"x\\234\\274\\275\\7\\224\\34\\305\\3618<;\\263;\\263\\341\\322\\356\\336\\355^\\222N\\361\\30\\335\\355)\\235\\20w\\2\"..., 506683, 48982212429) = 506683\n 6 pread(5, \"x\\234\\254}\\7x\\34\\305\\25\\360\\336\\336\\335^Q\\275;\\351N\\262d\\313M\\362\\372tr\\2231\\226\\1\\27\"..., 402609, 49245906707) = 402609\n 6 pread(5, \"x\\234\\254\\275\\t|\\24\\305\\3628\\2763\\323;\\273I6\\t\\331lvs\\21B\\270\\206\\315&\\1\\2H@\"..., 176754, 49246749832) = 176754\n 6 pread(5, \"x\\234\\234}\\7|T\\305\\366\\377\\336\\331\\315\\246P\\23H\\205$t\\226$t)\\t\\322\\244\\367\\5\\5\\5\\244\"..., 236257, 49246513568) = 236257\n 6 pread(5, \"x\\234\\224}\\7|TE\\360\\377\\355\\356\\225t\\270\\224K\\207$@r\\244\\2\\241\\205\\320{\\21\\10 \\35\\351\"..., 204238, 49246309323) = 204238\n 5 pread(5, \"x\\234\\264\\275\\7xT\\305\\0277\\274\\367\\316\\335\\273%u7\\311n*!\\204rI6\\t$\\20\\10\\275\\211\"..., 233622, 49247108828) = 233622\n 5 pread(5, \"x\\234\\254\\275\\7|T\\305\\363\\0~\\357\\275\\275w\\227\\334%\\341\\222\\334]zB \\341\\270\\\\\\n\\204\\26:\"..., 182228, 49246926593) = 182228\n 5 pread(5, \"x\\234\\234\\274\\5x\\24\\327\\32\\377\\177\\316\\314fvf\\26\\22\\226@\\2\\4\\222l\\4\\\"\\33 \\4\\v\\20\\10\"..., 70234, 49247342456) = 70234\n 4 pread(5, \"x\\234m{\\t\\\\TU\\373\\377\\3343\\\\\\34\\206a\\270s/\\273\\10\\303* \\340 \\240\\250\\250\\203\\\"\\232\"..., 9345, 49425395631) = 9345\n 4 pread(5, \"x\\234-\\326\\177P\\323e\\34\\300\\3619`\\342T\\4\\324)\\23\\1'\\242\\241\\233c\\300D:)~\\232\\250\"..., 1211, 49248626398) = 1211\n 4 pread(5, \"x\\234\\314\\275w\\234\\24E\\363\\7\\274\\323\\263;\\273\\227o\\367\\356v\\357\\270\\304\\35\\341\\3662p\\204;r\\16\"..., 149400, 49425246225) = 149400\n 4 pread(5, \"x\\234\\274\\275\\7|\\34\\305\\3658\\276\\345n\\257J\\362\\355Iw\\262e[r_K'7al\\31\\343\\2\"..., 549072, 38602368468) = 549072\n\nSo... the first two objects are clearly a problem as they are re-loaded \nover and over. Given that their offset is far away from the others i.e. \nrelatively at the beginning of the pack, they probably are quite high in \ntheir delta hierarchy. And what's really bad is to see those at the \nbeginning of 10 pread() calls in a row meaning that an entire delta \nstring has to be replayed in order to get back all those base objects \nthat were evicted from the cache. That's clearly wasted CPU cycles and \nthat shouldn't happen with a large enough value for \ncore.deltaBaseCacheLimit. Given that your files are \"relatively\" small \ni.e. in the 4MB range max, then the cache should be able to hold quite \nmany of them. At the moment with its 16MB limit, only a few of those \nobjects would evict many objects from the cache quickly.\n\nIf this is still not good enough, then you could add a negative delta \nattribute to those large binary files (see \nhttp://www.kernel.org/pub/software/scm/git/docs/gitattributes.html)\nand repack the repository on the server. Of course that will make the \nrepository larger and the data transfer longer when cloning, but the \n\"resolving deltas\" will be much faster. This is therefore a tradeoff.\n\nAnother solution which might be way more practical for users of such a \nhuge repository is simply to use a shallow clone. Surely those people \ncloning this repository might not need the full history of the \nrepository. So you could simply use:\n\n\tgit clone --depth=10 ...\n\nand have only the last 10 revisions transferred. Later on the \nrepository can be deepened by passing the --depth argument with a larger \nvalue to the fetch command if need be.\n\n\nNicolas\n"},{"id":"139345","messageId":"4BC33CE3.4030103@gmail.com","threadId":"23357","inReplyTo":"alpine.LFD.2.00.1004101617460.7232@xanadu.home","subject":"Re: git clone: very long \"resolving deltas\" phase","fromName":"Vitaly","fromEmail":"vitaly.berov@gmail.com","sentAt":"2010-04-12T15:31:47Z","receivedAt":"2010-04-12T15:31:47Z","isPatch":false,"sender":{"key":"vitaly.berov@gmail.com","avatar":null},"body":"Hello,\n\nOn 04/11/2010 04:50 AM, Nicolas Pitre wrote:\n> core.deltaBaseCacheLimit. Given that your files are \"relatively\" small\n> i.e. in the 4MB range max, then the cache should be able to hold quite\n> many of them. At the moment with its 16MB limit, only a few of those\n> objects would evict many objects from the cache quickly.\n>\n> If this is still not good enough, then you could add a negative delta\n> attribute to those large binary files (see\n> http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html)\n> and repack the repository on the server. Of course that will make the\n> repository larger and the data transfer longer when cloning, but the\n> \"resolving deltas\" will be much faster. This is therefore a tradeoff.\n>\n> Another solution which might be way more practical for users of such a\n> huge repository is simply to use a shallow clone. Surely those people\n> cloning this repository might not need the full history of the\n> repository. So you could simply use:\n>\n> \tgit clone --depth=10 ...\n>\n> and have only the last 10 revisions transferred. Later on the\n> repository can be deepened by passing the --depth argument with a larger\n> value to the fetch command if need be.\n>\n>\n> Nicolas\n>\n> \nThanks for comprehensive answer, Nicolas. Now I see 3 directions to work \non: cacheLimit, negative delta attributes and shortening the history \n(actually, I don't think \"clone --depth\" is feasible in our environment, \nbut we can try to backup and just purge the history).\n"}]}