{"thread":{"id":"4236","subject":"Local clone/fetch with cogito is glacial","startedAt":"2006-05-21T23:47:45Z","lastAt":"2006-05-22T23:24:37Z","messageCount":11,"participants":["H. Peter Anvin","Sean","Linus Torvalds","Petr Baudis"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"20408","messageId":"4470FC21.6010104@zytor.com","threadId":"4236","inReplyTo":null,"subject":"Local clone/fetch with cogito is glacial","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2006-05-21T23:47:45Z","receivedAt":"2006-05-21T23:47:45Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"It appears that doing a *local* -- meaning using a file path or file URL \n-- clone or fetch with cogito is just glacial when the repository has an \neven moderate number of tags (and it's fetching the tags that takes all \nthe time.)  That's a really serious problem for me.\n\n\t-hpa\n"},{"id":"20416","messageId":"BAYC1-PASMTP11FDE05B530CFF43C043E5AE9A0@CEZ.ICE","threadId":"4236","inReplyTo":"4470FC21.6010104@zytor.com","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Sean","fromEmail":"seanlkml@sympatico.ca","sentAt":"2006-05-22T01:20:26Z","receivedAt":"2006-05-22T01:20:26Z","isPatch":false,"sender":{"key":"seanlkml@sympatico.ca","avatar":"https://gravatar.com/avatar/f92923f54fc08c401fc59b71829d4b89e9b8087fbba45ff87c82e6a83aee02ae?d=mp&s=160"},"body":"On Sun, 21 May 2006 16:47:45 -0700\n\"H. Peter Anvin\" <hpa@zytor.com> wrote:\n\n> It appears that doing a *local* -- meaning using a file path or file URL \n> -- clone or fetch with cogito is just glacial when the repository has an \n> even moderate number of tags (and it's fetching the tags that takes all \n> the time.)  That's a really serious problem for me.\n> \n\nPeter, does git clone work acceptably for you?\n\nSean\n"},{"id":"20418","messageId":"Pine.LNX.4.64.0605211840210.3649@g5.osdl.org","threadId":"4236","inReplyTo":"4470FC21.6010104@zytor.com","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-05-22T01:40:41Z","receivedAt":"2006-05-22T01:40:41Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 21 May 2006, H. Peter Anvin wrote:\n>\n> It appears that doing a *local* -- meaning using a file path or file URL -- \n> clone or fetch with cogito is just glacial when the repository has an even\n> moderate number of tags (and it's fetching the tags that takes all the time.)\n> That's a really serious problem for me.\n\nI think this is purely a cogito problem. \n\nUse \"git clone\" instead.\n\n\t\tLinus\n"},{"id":"20477","messageId":"44722A8F.9020609@zytor.com","threadId":"4236","inReplyTo":"BAYC1-PASMTP11FDE05B530CFF43C043E5AE9A0@CEZ.ICE","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2006-05-22T21:18:07Z","receivedAt":"2006-05-22T21:18:07Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Sean wrote:\n> On Sun, 21 May 2006 16:47:45 -0700\n> \"H. Peter Anvin\" <hpa@zytor.com> wrote:\n> \n>> It appears that doing a *local* -- meaning using a file path or file URL \n>> -- clone or fetch with cogito is just glacial when the repository has an \n>> even moderate number of tags (and it's fetching the tags that takes all \n>> the time.)  That's a really serious problem for me.\n>>\n> \n> Peter, does git clone work acceptably for you?\n> \n\nWell, it does, except it doesn't set up the cogito branches (which one can of course copy \nmanually.)\n\ncg-clone probably should be rewritten as a thin wrapper around git-clone.\n\n\t-hpa\n"},{"id":"20480","messageId":"20060522220206.GA10488@pasky.or.cz","threadId":"4236","inReplyTo":"44722A8F.9020609@zytor.com","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2006-05-22T22:02:06Z","receivedAt":"2006-05-22T22:02:06Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, May 22, 2006 at 11:18:07PM CEST, I got a letter\nwhere \"H. Peter Anvin\" <hpa@zytor.com> said that...\n> Sean wrote:\n> >On Sun, 21 May 2006 16:47:45 -0700\n> >\"H. Peter Anvin\" <hpa@zytor.com> wrote:\n> >\n> >>It appears that doing a *local* -- meaning using a file path or file URL \n> >>-- clone or fetch with cogito is just glacial when the repository has an \n> >>even moderate number of tags (and it's fetching the tags that takes all \n> >>the time.)  That's a really serious problem for me.\n> >>\n> >\n> >Peter, does git clone work acceptably for you?\n> >\n> \n> Well, it does, except it doesn't set up the cogito branches (which one can \n> of course copy manually.)\n\nWhat about incremental fetches using git-fetch? From a quick scan of the\ngit-fetch automagic tags following code, it seems to be even\nsignificantly more expensive than Cogito's (in terms of number of\nforks).\n\ngit-clone has an advantage here since it clones _everything_ while\nCogito fetches only stuff related to the branch you are cloning, and\nverifying if what it fetches is sensible for you unfortunately takes a\nlot of time. :/ I guess there is no way to verify presence of multiple\nobjects at once and there is also no way to order local fetch of\nmultiple objects at once.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nRight now I am having amnesia and deja-vu at the same time.  I think\nI have forgotten this before.\n"},{"id":"20484","messageId":"447239F0.9030705@zytor.com","threadId":"4236","inReplyTo":"20060522220206.GA10488@pasky.or.cz","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2006-05-22T22:23:44Z","receivedAt":"2006-05-22T22:23:44Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Petr Baudis wrote:\n> \n> What about incremental fetches using git-fetch? From a quick scan of the\n> git-fetch automagic tags following code, it seems to be even\n> significantly more expensive than Cogito's (in terms of number of\n> forks).\n> \n\nWell, I haven't used git-fetch, so I can't comment on that one.\n\n> git-clone has an advantage here since it clones _everything_ while\n> Cogito fetches only stuff related to the branch you are cloning, and\n> verifying if what it fetches is sensible for you unfortunately takes a\n> lot of time. :/ I guess there is no way to verify presence of multiple\n> objects at once and there is also no way to order local fetch of\n> multiple objects at once.\n\nNote that non-local cg-clones are at least an order of magnitude faster, even when the \nnonlocal is just git+ssh:.  One could presumably do the same thing over a pipe.\n\n\t-hpa\n"},{"id":"20486","messageId":"20060522225054.GL11941@pasky.or.cz","threadId":"4236","inReplyTo":"447239F0.9030705@zytor.com","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2006-05-22T22:50:54Z","receivedAt":"2006-05-22T22:50:54Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Tue, May 23, 2006 at 12:23:44AM CEST, I got a letter\nwhere \"H. Peter Anvin\" <hpa@zytor.com> said that...\n> Petr Baudis wrote:\n> >git-clone has an advantage here since it clones _everything_ while\n> >Cogito fetches only stuff related to the branch you are cloning, and\n> >verifying if what it fetches is sensible for you unfortunately takes a\n> >lot of time. :/ I guess there is no way to verify presence of multiple\n> >objects at once and there is also no way to order local fetch of\n> >multiple objects at once.\n> \n> Note that non-local cg-clones are at least an order of magnitude faster, \n> even when the nonlocal is just git+ssh:.  One could presumably do the same \n> thing over a pipe.\n\nEven rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an\nalmost entirely different code patch and it's much more efficient since\nI just accumulate the tag object ids I want to check and then pour them\nto git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nRight now I am having amnesia and deja-vu at the same time.  I think\nI have forgotten this before.\n"},{"id":"20487","messageId":"4472432A.8010002@zytor.com","threadId":"4236","inReplyTo":"20060522225054.GL11941@pasky.or.cz","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2006-05-22T23:03:06Z","receivedAt":"2006-05-22T23:03:06Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Petr Baudis wrote:\n> \n> Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an\n> almost entirely different code patch and it's much more efficient since\n> I just accumulate the tag object ids I want to check and then pour them\n> to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(\n> \n\nNo, but git-fetch-pack could operate over a local pipe just fine (after all, all it does \nis ssh an \"git-send-pack\" command to the other side.)\n\n\t-hpa\n"},{"id":"20488","messageId":"20060522230841.GM11941@pasky.or.cz","threadId":"4236","inReplyTo":"4472432A.8010002@zytor.com","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2006-05-22T23:08:41Z","receivedAt":"2006-05-22T23:08:41Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Tue, May 23, 2006 at 01:03:06AM CEST, I got a letter\nwhere \"H. Peter Anvin\" <hpa@zytor.com> said that...\n> Petr Baudis wrote:\n> >\n> >Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an\n> >almost entirely different code patch and it's much more efficient since\n> >I just accumulate the tag object ids I want to check and then pour them\n> >to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(\n> >\n> \n> No, but git-fetch-pack could operate over a local pipe just fine (after \n> all, all it does is ssh an \"git-send-pack\" command to the other side.)\n\nYes, but in that case it couldn't hardlink the objects so you would see\nquite a big bump in disk usage if you have many local clones of the same\nrepo.\n\nThat said, hardlinking is probably not all that big an advantage if you\nrepack often, repack everywhere, and in the many-repositories cases it\nmight be more sensible to use alternates (which is what cg-clone -l\nshould really do instead of symlinking), so it might be well worth\nthe sacrifice.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nRight now I am having amnesia and deja-vu at the same time.  I think\nI have forgotten this before.\n"},{"id":"20493","messageId":"Pine.LNX.4.64.0605221615030.3697@g5.osdl.org","threadId":"4236","inReplyTo":"20060522225054.GL11941@pasky.or.cz","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-05-22T23:18:48Z","receivedAt":"2006-05-22T23:18:48Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 23 May 2006, Petr Baudis wrote:\n> \n> Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an\n> almost entirely different code patch and it's much more efficient since\n> I just accumulate the tag object ids I want to check and then pour them\n> to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(\n\nSure you can. Well, not to http-fetch, but git-fetch-pack should work fine \nfor a local repo.\n\nThe git-clone script will literally special-case rsync:// and http://. \nEverything else should work fine with git-fetch-pack.\n\n\t\tLinus\n"},{"id":"20495","messageId":"20060522232437.GN11941@pasky.or.cz","threadId":"4236","inReplyTo":"Pine.LNX.4.64.0605221615030.3697@g5.osdl.org","subject":"Re: Local clone/fetch with cogito is glacial","fromName":"Petr Baudis","fromEmail":"pasky@ucw.cz","sentAt":"2006-05-22T23:24:37Z","receivedAt":"2006-05-22T23:24:37Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"> The git-clone script will literally special-case rsync:// and http://. \n> Everything else should work fine with git-fetch-pack.\n\nAha, I overlooked that what I described goes on in git-clone happens\nonly with git-clone -l, otherwise it indeed seems to use git-fetch-pack.\nSorry about the confusion.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nRight now I am having amnesia and deja-vu at the same time.  I think\nI have forgotten this before.\n"}]}