{"thread":{"id":"1357","subject":"Dump http servers still slow?","startedAt":"2005-07-28T21:00:23Z","lastAt":"2005-08-01T14:03:57Z","messageCount":10,"participants":["Darrin Thompson","Junio C Hamano","Ryan Anderson","Radoslaw AstralStorm Szkodzinski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"6531","messageId":"1122584423.12374.11.camel@localhost.localdomain","threadId":"1357","inReplyTo":null,"subject":"Dump http servers still slow?","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-28T21:00:23Z","receivedAt":"2005-07-28T21:00:23Z","isPatch":false,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"Junio,\n\nI just ran git clone against the mainline git repository using both http\nand rsync. http was still quite slow compared to rsync. I expected that\nthe http time would be much faster than in the past due to the pack\nfile.\n\nIs there something simple I'm missing?\n\n--\nDarrin\n"},{"id":"6542","messageId":"7vy87qpcwg.fsf@assigned-by-dhcp.cox.net","threadId":"1357","inReplyTo":"1122584423.12374.11.camel@localhost.localdomain","subject":"Re: Dump http servers still slow?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-29T02:24:47Z","receivedAt":"2005-07-29T02:24:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Darrin Thompson writes:\n\n> I just ran git clone against the mainline git repository using both http\n> and rsync. http was still quite slow compared to rsync. I expected that\n> the http time would be much faster than in the past due to the pack\n> file.\n>\n> Is there something simple I'm missing?\n\nNo, the only thing you missed was that I did not write it to\nmake it fast, but just to make it work ;-). The commit walker\nsimply does not work against a dumb http server repository that\nis packed and prune-packed, which is already the case for both\nkernel and git repositories.\n\nThe thing is, the base pack for the git repository is 1.8MB\ncurrently containing 4500+ objects, while we accumulated 600+\nunpacked objects since then which is about ~5MB. The commit\nwalker needs to fetched the latter one by one in the old way.\n\nWhen packed incrementally on top of the base pack, these 600+\nunpacked objects compress down to something like 400KB, and I\nwas hoping we could wait until we accumulate enough to produce\nan incremental about a meg or so ...\n"},{"id":"6588","messageId":"1122645821.4263.6.camel@localhost.localdomain","threadId":"1357","inReplyTo":"7vy87qpcwg.fsf@assigned-by-dhcp.cox.net","subject":"Re: Dump http servers still slow?","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-29T14:03:41Z","receivedAt":"2005-07-29T14:03:41Z","isPatch":false,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"On Thu, 2005-07-28 at 19:24 -0700, Junio C Hamano wrote:\n> The thing is, the base pack for the git repository is 1.8MB\n> currently containing 4500+ objects, while we accumulated 600+\n> unpacked objects since then which is about ~5MB. The commit\n> walker needs to fetched the latter one by one in the old way.\n> \n> When packed incrementally on top of the base pack, these 600+\n> unpacked objects compress down to something like 400KB, and I\n> was hoping we could wait until we accumulate enough to produce\n> an incremental about a meg or so ...\n\nOk... so lets check my assumptions:\n\n1. Pack files should reduce the number of http round trips.\n2. What I'm seeing when I check out mainline git is the acquisition of a\nsingle large pack, then 600+ more recent objects. Better than before,\nbut still hundreds of round trips.\n3. If I wanted to further speed up the initial checkout on my own\nrepositories I could frequently repack my most recent few hundred\nobjects.\n4. If curl had pipelining then less pack management would be needed.\n\nWhere is the code for gitweb? (i.e. http://kernel.org/git ) Seems like\nit could benefit from some git-send-pack superpowers.\n\n--\nDarrin\n"},{"id":"6591","messageId":"20050729144802.GA2280@mythryan2.michonline.com","threadId":"1357","inReplyTo":"1122645821.4263.6.camel@localhost.localdomain","subject":"Re: Dump http servers still slow?","fromName":"Ryan Anderson","fromEmail":"ryan@michonline.com","sentAt":"2005-07-29T14:48:02Z","receivedAt":"2005-07-29T14:48:02Z","isPatch":false,"sender":{"key":"ryan@michonline.com","avatar":null},"body":"On Fri, Jul 29, 2005 at 09:03:41AM -0500, Darrin Thompson wrote:\n> \n> Where is the code for gitweb? (i.e. http://kernel.org/git ) Seems like\n> it could benefit from some git-send-pack superpowers.\n\nhttp://www.kernel.org/pub/software/scm/gitweb/\n\nIt occurs to me that pulling this into the main git repository might not\nbe a bad idea, since it is currently living outside any revision\ntracking at the moment.\n\n-- \n\nRyan Anderson\n sometimes Pug Majere\n"},{"id":"6593","messageId":"1122649056.4263.13.camel@localhost.localdomain","threadId":"1357","inReplyTo":"20050729144802.GA2280@mythryan2.michonline.com","subject":"Re: Dump http servers still slow?","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-29T14:57:36Z","receivedAt":"2005-07-29T14:57:36Z","isPatch":false,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"On Fri, 2005-07-29 at 10:48 -0400, Ryan Anderson wrote:\n> On Fri, Jul 29, 2005 at 09:03:41AM -0500, Darrin Thompson wrote:\n> > \n> > Where is the code for gitweb? (i.e. http://kernel.org/git ) Seems like\n> > it could benefit from some git-send-pack superpowers.\n> \n> http://www.kernel.org/pub/software/scm/gitweb/\n> \n> It occurs to me that pulling this into the main git repository might not\n> be a bad idea, since it is currently living outside any revision\n> tracking at the moment.\n> \n\nCan't see the code.\n\nhttp://www.kernel.org/pub/software/scm/gitweb/gitweb.cgi\n\nInternal Server Error\n\n--\nDarrin\n"},{"id":"6594","messageId":"20050729170824.5a946769.astralstorm@gorzow.mm.pl","threadId":"1357","inReplyTo":"1122649056.4263.13.camel@localhost.localdomain","subject":"Re: Dump http servers still slow?","fromName":"Radoslaw AstralStorm Szkodzinski","fromEmail":"astralstorm@gorzow.mm.pl","sentAt":"2005-07-29T15:08:24Z","receivedAt":"2005-07-29T15:08:24Z","isPatch":false,"sender":{"key":"astralstorm@gorzow.mm.pl","avatar":null},"body":"On Fri, 29 Jul 2005 09:57:36 -0500\nDarrin Thompson wrote:\n\n> Can't see the code.\n> \n> http://www.kernel.org/pub/software/scm/gitweb/gitweb.cgi\n> \n> Internal Server Error\n> \n\nUse FTP.\n\n-- \nAstralStorm\n\nGPG Key ID = 0xD1F10BA2\nGPG Key fingerprint = 96E2 304A B9C4 949A 10A0 9105 9543 0453 D1F1 0BA2\nPlease encrypt if you can.\n"},{"id":"6595","messageId":"1122650784.4263.15.camel@localhost.localdomain","threadId":"1357","inReplyTo":"20050729170824.5a946769.astralstorm@gorzow.mm.pl","subject":"Re: Dump http servers still slow?","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-07-29T15:26:24Z","receivedAt":"2005-07-29T15:26:24Z","isPatch":false,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"On Fri, 2005-07-29 at 17:08 +0200, Radoslaw AstralStorm Szkodzinski\nwrote:\n> On Fri, 29 Jul 2005 09:57:36 -0500\n> Darrin Thompson wrote:\n> \n> > Can't see the code.\n> > \n> > http://www.kernel.org/pub/software/scm/gitweb/gitweb.cgi\n> > \n> > Internal Server Error\n> > \n> \n> Use FTP.\n> \n\nDuh. Thanks.\n\n--\nDarrin\n"},{"id":"6617","messageId":"7vmzo56o0u.fsf@assigned-by-dhcp.cox.net","threadId":"1357","inReplyTo":"1122645821.4263.6.camel@localhost.localdomain","subject":"Re: Dump http servers still slow?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-30T02:11:45Z","receivedAt":"2005-07-30T02:11:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Darrin Thompson writes:\n\n> Ok... so lets check my assumptions:\n>\n> 1. Pack files should reduce the number of http round trips.\n> 2. What I'm seeing when I check out mainline git is the acquisition of a\n> single large pack, then 600+ more recent objects. Better than before,\n> but still hundreds of round trips.\n> 3. If I wanted to further speed up the initial checkout on my own\n> repositories I could frequently repack my most recent few hundred\n> objects.\n> 4. If curl had pipelining then less pack management would be needed.\n\nAll true. Another possibility is to make multiple requests in\nparallel; if curl does not do pipelining, either switch to\nsomething that does, or have more then one process using curl.\n\nThe dumb server preparation creates three files, two of which is\ncurrently used by clone (one is list of packs, the other is list\nof branches and tags). The third one is commit ancestry\ninformation. The commit walker could be taught to read it to\nfigure out what commits it still needs to fetch without waiting\nfor the commit being retrieved to be parsed.\n\nSorry, I am not planning to write that part myself.\n\nOne potential low hanging fruit is that even for cloning via\ngit:// URL we _might_ be better off starting with the dumb\nserver protocol; get the list of statically prepared packs and\nobtain them upfront before starting the clone-pack/upload-pack\nprotocol pair.\n"},{"id":"6657","messageId":"7v1x5ftqn5.fsf@assigned-by-dhcp.cox.net","threadId":"1357","inReplyTo":"1122645821.4263.6.camel@localhost.localdomain","subject":"Re: Dump http servers still slow?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-31T06:51:10Z","receivedAt":"2005-07-31T06:51:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Darrin Thompson writes:\n\n> 1. Pack files should reduce the number of http round trips.\n> 2. What I'm seeing when I check out mainline git is the acquisition of a\n> single large pack, then 600+ more recent objects. Better than before,\n> but still hundreds of round trips.\n\nI've packed the git.git repository, by the way. It has 43\nunpacked objects totalling 224 kilobytes, so cloning over dumb\nhttp should go a lot faster until we accumulate more unpacked\nobjects.\n\nSome of you may have noticed that in the proposed updates queue\n(\"pu\" branch) I have a couple of commits related to pulling from\na packed dumb http server. There are two \"git fetch http://\"\ncommits to let you pull from such, and another stupid \"count\nobjects\" script that you can use to see how many unpacked\nobjects you have in your repository; the latter is to help\nyou decide when to repack.\n\nBrave souls may want to try out the dumb http fetch. For\nexample, it _should_ do the right thing even if you do the\nfollowing:\n\n $ git clone http://www.kernel.org/pub/scm/git/git.git/ newdir\n $ cd newdir\n $ mv .git/objects/pack/pack-* . ;# even if you unpack packs on your\n $ rm -f pack-*.idx\t\t ;# end, it should do the right thing.\n $ for pack in pack-*.pack; do\n git-unpack-objects <$pack\n rm -f \"$pack\"\n done\n $ rm -f .git/refs/heads/pu\n $ git prune ;# lose objects in \"pu\" but still not in \"master\"\n $ git pull origin pu\n $ git ls-remote origin |\n while read sha1 refname\n do\n case \"$refname\" in\n refs/heads/master) echo $sha1 >\".git/$refname\" ;;\n esac\n done ;# revert master to upstream master\n $ old=$(git-rev-parse master^^^^^^^^^^)\n $ echo \"$old\" >.git/refs/heads/master ;# rewind further\n $ git checkout -f master\n $ git prune ;# try losing a bit more objects.\n $ git pull origin master\n $ git ls-remote ./.\t\t;# show me my refs\n $ git ls-remote origin\t\t;# show me his refs\n\nUnlike my other shell scripts I usually write in my e-mail\nbuffer, I have actually run the above ;-).\n\n-jc\n"},{"id":"6708","messageId":"1122905037.4042.1.camel@localhost.localdomain","threadId":"1357","inReplyTo":"7v1x5ftqn5.fsf@assigned-by-dhcp.cox.net","subject":"Re: Dump http servers still slow?","fromName":"Darrin Thompson","fromEmail":"darrint@progeny.com","sentAt":"2005-08-01T14:03:57Z","receivedAt":"2005-08-01T14:03:57Z","isPatch":false,"sender":{"key":"darrint@progeny.com","avatar":null},"body":"On Sat, 2005-07-30 at 23:51 -0700, Junio C Hamano wrote:\n> Darrin Thompson writes:\n> \n> > 1. Pack files should reduce the number of http round trips.\n> > 2. What I'm seeing when I check out mainline git is the acquisition of a\n> > single large pack, then 600+ more recent objects. Better than before,\n> > but still hundreds of round trips.\n> \n> I've packed the git.git repository, by the way. It has 43\n> unpacked objects totalling 224 kilobytes, so cloning over dumb\n> http should go a lot faster until we accumulate more unpacked\n> objects.\n\nI did a pull from the office and the times were 27 sec for http and 17\nsec for rsync. So the moral of the story should be that frequent repacks\nare sufficient for decent http performance.\n\n--\nDarrin\n"}]}