{"thread":{"id":"6145","subject":"read-for-fill and caching in gitweb (Re: kernel.org mirroring)","startedAt":"2006-12-28T20:45:07Z","lastAt":"2006-12-29T19:31:56Z","messageCount":6,"participants":["Martin Langhoff","Robert Fitzsimons","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"30442","messageId":"46a038f90612281245s52bdd868h8c421951c7abeb84@mail.gmail.com","threadId":"6145","inReplyTo":null,"subject":"read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-12-28T20:45:07Z","receivedAt":"2006-12-28T20:45:07Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 12/9/06, Linus Torvalds wrote:\n> Actually, just looking at the examples, it looks like memcached is\n> fundamentally flawed, exactly the same way Apache mod_cache is\n> fundamentally flawed.\n\nmemcached is really fast internally, but can be rather slow from the\nPOV of the client code, as it forces a costly\nmarshalling/unmarshalling of data. For perl-only situations where it\nis OK to have per-server caches, I have been looking at\nCache::FastMmap. I will probably try to implement caching for the\nprojects, summary & log/shortlog pages using Cache::FastMap\n\nAnd I'll do read-for-fill for it, and see how that goes.\n\n(BTW, in the last week I've had to implement a similar\nanti-thundering-herds cache in PHP using memcached and/or eaccelerator\n-- a shmem cache -- and I've done a read-for-fill for both of them\nthat works reasonably well.)\n\ncheers,\n\n\nmartin\n"},{"id":"30458","messageId":"20061229032126.GE6558@localhost","threadId":"6145","inReplyTo":"46a038f90612281245s52bdd868h8c421951c7abeb84@mail.gmail.com","subject":"Re: read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Robert Fitzsimons","fromEmail":"robfitz@273k.net","sentAt":"2006-12-29T03:21:26Z","receivedAt":"2006-12-29T03:21:26Z","isPatch":false,"sender":{"key":"robfitz@273k.net","avatar":null},"body":"> I will probably try to implement caching for the\n> projects, summary & log/shortlog pages using Cache::FastMap\n\nHere are the mean (and standard deviation) in milliseconds for those\npages using a few different versions of gitweb.\n\n project_list summary shortlog log\nv267 173 1.6 1141 8.8 795 5.0 919 1.9\n1.4.4.3 220 2.3 397 2.4 930 4.2 1113 56.9\n1.5.0.rc0.g4a4d 226 1.9 292 1.7 352 4.0 491 6.7\n1.5.0.rc0.g4a4d 60 1.0 131 0.7 195 1.2 347 3.7\n(mod_perl)\n \nI think there would be a benefit in deploying a more recent version of\ngitweb on kernel.org and and even bigger benefit if it use mod_perl. I\nwould be happy to help, if I can.\n\nI'll look into the increase in time for the project_list in more recent\nversions of gitweb, tomorrow.\n\nRobert\n"},{"id":"30469","messageId":"200612291140.46909.jnareb@gmail.com","threadId":"6145","inReplyTo":"20061229032126.GE6558@localhost","subject":"Re: read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-29T10:40:45Z","receivedAt":"2006-12-29T10:40:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Robert Fitzsimons wrote:\n\n> Here are the mean (and standard deviation) in milliseconds for those\n> pages using a few different versions of gitweb.\n> \n> project_list summary shortlog log\n> v267 173 1.6 1141 8.8 795 5.0 919 1.9\n> 1.4.4.3 220 2.3 397 2.4 930 4.2 1113 56.9\n> 1.5.0.rc0.g4a4d 226 1.9 292 1.7 352 4.0 491 6.7\n> 1.5.0.rc0.g4a4d 60 1.0 131 0.7 195 1.2 347 3.7\n> (mod_perl)\n\n> I'll look into the increase in time for the project_list in more recent\n> versions of gitweb, tomorrow.\n\nIt is simply the case that new features cost more. Namely in earlier\nversions of gitweb Last Change time was taken from HEAD (from current\nbranch), in newer we check all branches (using git-for-each-ref).\nFor published public repository it migh make sense to pack also heads\n(make them packed refs).\n\nI was thinking about making this a gitweb %feature, allowing gitweb\nadministrator to chose if Last Change is taken from all branches\n(as it is now), from HEAD (as it was before), or from given branch\n(for example master).\n\nAnother thing that might made small increase in time is checking\nif project is to be visible to gitweb ($export_ok and $strict_export).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"30470","messageId":"46a038f90612290346n35386e14g922465d66beaf5ab@mail.gmail.com","threadId":"6145","inReplyTo":"200612291140.46909.jnareb@gmail.com","subject":"Re: read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-12-29T11:46:15Z","receivedAt":"2006-12-29T11:46:15Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 12/29/06, Jakub Narebski wrote:\n> It is simply the case that new features cost more. Namely in earlier\n> versions of gitweb Last Change time was taken from HEAD (from current\n> branch), in newer we check all branches (using git-for-each-ref).\n> For published public repository it migh make sense to pack also heads\n> (make them packed refs).\n\nI haven't been using packed refs at all, but it sounds like it's a\nsingle file. So we can stat just that file rather than ask questions\nabout the heads themselves. That makes checking for if-modified-since\ncheap as well.\n\n> I was thinking about making this a gitweb %feature, allowing gitweb\n> administrator to chose if Last Change is taken from all branches\n> (as it is now), from HEAD (as it was before), or from given branch\n> (for example master).\n\nI think the natural thing is to check all heads (doing it on the cheap\non packed-refs repos) and provide tuning tips. in this case \"use\npacked refs\" which I guess will become the default eventually.\n\ncheers,\n\n\nmartin\n"},{"id":"30473","messageId":"200612291347.10805.jnareb@gmail.com","threadId":"6145","inReplyTo":"46a038f90612290346n35386e14g922465d66beaf5ab@mail.gmail.com","subject":"Re: read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-29T12:47:09Z","receivedAt":"2006-12-29T12:47:09Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Martin Langhoff wrote:\n> On 12/29/06, Jakub Narebski wrote:\n>> It is simply the case that new features cost more. Namely in earlier\n>> versions of gitweb Last Change time was taken from HEAD (from current\n>> branch), in newer we check all branches (using git-for-each-ref).\n>> For published public repository it migh make sense to pack also heads\n>> (make them packed refs).\n> \n> I haven't been using packed refs at all, but it sounds like it's a\n> single file. So we can stat just that file rather than ask questions\n> about the heads themselves. That makes checking for if-modified-since\n> cheap as well.\n\nThat I think would work _only_ for the working repository. For \npublishing bare repository you push into (or which is a mirror of some \nother repository) I think stat $GIT_DIR/packed-refs would return date\nof last push (last mirror), not when repository was last committed to...\n \n>> I was thinking about making this a gitweb %feature, allowing gitweb\n>> administrator to chose if Last Change is taken from all branches\n>> (as it is now), from HEAD (as it was before), or from given branch\n>> (for example master).\n> \n> I think the natural thing is to check all heads (doing it on the cheap\n> on packed-refs repos) and provide tuning tips. in this case \"use\n> packed refs\" which I guess will become the default eventually.\n\n...but this could be included in above %feature.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"30486","messageId":"20061229193156.GG6558@localhost","threadId":"6145","inReplyTo":"200612291140.46909.jnareb@gmail.com","subject":"Re: read-for-fill and caching in gitweb (Re: kernel.org mirroring)","fromName":"Robert Fitzsimons","fromEmail":"robfitz@273k.net","sentAt":"2006-12-29T19:31:56Z","receivedAt":"2006-12-29T19:31:56Z","isPatch":false,"sender":{"key":"robfitz@273k.net","avatar":null},"body":"> > project_list summary shortlog log\n> > v267 173 1.6 1141 8.8 795 5.0 919 1.9\n> > 1.4.4.3 220 2.3 397 2.4 930 4.2 1113 56.9\n> > 1.5.0.rc0.g4a4d 226 1.9 292 1.7 352 4.0 491 6.7\n> > 1.5.0.rc0.g4a4d 60 1.0 131 0.7 195 1.2 347 3.7\n> > (mod_perl)\n\n> It is simply the case that new features cost more. Namely in earlier\n> versions of gitweb Last Change time was taken from HEAD (from current\n> branch), in newer we check all branches (using git-for-each-ref).\n> For published public repository it migh make sense to pack also heads\n> (make them packed refs).\n>\n> I was thinking about making this a gitweb %feature, allowing gitweb\n> administrator to chose if Last Change is taken from all branches\n> (as it is now), from HEAD (as it was before), or from given branch\n> (for example master).\n\nI've sent a separate email with a patch to add this feature.\n(\"[PATCH] gitweb: New feature last_modified_ref.\"\n<20061229185805.GF6558@localhost>).\n\nHere are the new numbers. Notes: I've only got 3 projects in my project\nlist and I did a 'git gc' on them since yesterday.\n\n project_list summary shortlog log\n\nv267 174 1.1 286 2.1 794 3.4 921 3.2\n1.4.4.3 207 1.7 383 2.0 921 5.2 1082 3.8\ng04509 + patch 213 1.6 297 68.9 341 3.9 484 5.0\ng04509 + patch 71 69.9 117 2.5 190 2.1 341 2.7\n(mod_perl)\ng04509 + patch 209 1.0 276 1.5 342 3.3 483 6.3\n(HEAD)\ng04509 + patch 66 70.1 117 2.6 189 3.4 341 3.8\n(HEAD, mod_perl)\n\nThe v267 summary time is wrong, that version of gitweb is not\npacked-refs aware.\n\nI think I need a more consistent test setup I'm seeing some weird\ndeviations.\n\nRobert\n"}]}