{"thread":{"id":"6499","subject":".git/info/refs","startedAt":"2007-01-24T07:38:46Z","lastAt":"2007-02-01T17:51:51Z","messageCount":39,"participants":["H. Peter Anvin","Jakub Narebski","Johannes Schindelin","hpa@zytor.com","Junio C Hamano","Santi Béjar","Matthias Lederhofer"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"32493","messageId":"45B70D06.3050506@zytor.com","threadId":"6499","inReplyTo":null,"subject":".git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T07:38:46Z","receivedAt":"2007-01-24T07:38:46Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Would it be an incompatible change to add the commit date (and perhaps \nthe author date) to .git/info/refs?  I believe that would make it \npossible to dramatically (orders of magnitude) speed up the generation \nof the gitweb index page, which is easily the most expensive gitweb page \nto generate.\n\n\t-=hpa\n"},{"id":"32494","messageId":"ep78r0$h2u$1@sea.gmane.org","threadId":"6499","inReplyTo":"45B70D06.3050506@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-24T09:28:56Z","receivedAt":"2007-01-24T09:28:56Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin wrote:\n\n> Would it be an incompatible change to add the commit date (and perhaps \n> the author date) to .git/info/refs?  I believe that would make it \n> possible to dramatically (orders of magnitude) speed up the generation \n> of the gitweb index page, which is easily the most expensive gitweb page \n> to generate.\n\nWith new gitweb and new git it is not that expensive. It is now one call\nto git-for-each-ref per repository.\n\nBesides, we can't rely that .git/info/refs is up to date, or even exists.\nIt is for dumb protocols, not for gitweb.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"32527","messageId":"45B7818F.6020805@zytor.com","threadId":"6499","inReplyTo":"ep78r0$h2u$1@sea.gmane.org","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T15:55:59Z","receivedAt":"2007-01-24T15:55:59Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Jakub Narebski wrote:\n> H. Peter Anvin wrote:\n> \n>> Would it be an incompatible change to add the commit date (and perhaps \n>> the author date) to .git/info/refs?  I believe that would make it \n>> possible to dramatically (orders of magnitude) speed up the generation \n>> of the gitweb index page, which is easily the most expensive gitweb page \n>> to generate.\n> \n> With new gitweb and new git it is not that expensive. It is now one call\n> to git-for-each-ref per repository.\n\nThat IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n\n> Besides, we can't rely that .git/info/refs is up to date, or even exists.\n> It is for dumb protocols, not for gitweb.\n\nWell, SOMETHING needs to be done for this page, since it can take 15 \nminutes or more to generate.  Caching doesn't help one iota, since it's \nstale before being generated.\n\n\t-hpa\n"},{"id":"32528","messageId":"Pine.LNX.4.63.0701241658490.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45B7818F.6020805@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-24T16:02:23Z","receivedAt":"2007-01-24T16:02:23Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 24 Jan 2007, H. Peter Anvin wrote:\n\n> Jakub Narebski wrote:\n> > H. Peter Anvin wrote:\n> > \n> > > Would it be an incompatible change to add the commit date (and perhaps the\n> > > author date) to .git/info/refs?  I believe that would make it possible to\n> > > dramatically (orders of magnitude) speed up the generation of the gitweb\n> > > index page, which is easily the most expensive gitweb page to generate.\n> > \n> > With new gitweb and new git it is not that expensive. It is now one call\n> > to git-for-each-ref per repository.\n> \n> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n> \n> > Besides, we can't rely that .git/info/refs is up to date, or even exists.\n> > It is for dumb protocols, not for gitweb.\n> \n> Well, SOMETHING needs to be done for this page, since it can take 15 \n> minutes or more to generate.  Caching doesn't help one iota, since it's \n> stale before being generated.\n\nTo me, it seems like all boils down to caching parsed data structures. \nI.e. parse the config, then serialize the parsed data to a file. Don't \nreparse the config unless it is 1 hour older than the config.\n\nLikewise, run for-each-ref, and serialize the parsed data into a file. \nDon't rerun for-each-ref if that file is younger than 15 minutes.\n\nMaybe the same for the first 200 commits of each branch.\n\n(I made those times up, but you get the idea.)\n\nCiao,\nDscho\n"},{"id":"32530","messageId":"45B78836.5080508@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701241658490.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T16:24:22Z","receivedAt":"2007-01-24T16:24:22Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> \n> To me, it seems like all boils down to caching parsed data structures. \n> I.e. parse the config, then serialize the parsed data to a file. Don't \n> reparse the config unless it is 1 hour older than the config.\n> \n> Likewise, run for-each-ref, and serialize the parsed data into a file. \n> Don't rerun for-each-ref if that file is younger than 15 minutes.\n> \n> Maybe the same for the first 200 commits of each branch.\n> \n> (I made those times up, but you get the idea.)\n> \n\nA much better idea is to have that data structure updated on repository \nupdates, which is the whole point behind .git/info/refs.  On kernel.org, \nat least, if you don't keep .git/info/refs up to date you need to get \nyour fingers whacked anyway, since it damages usability for one \nparticular class of users.\n\n\t-hpa\n"},{"id":"32532","messageId":"Pine.LNX.4.63.0701241731400.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45B78836.5080508@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-24T16:38:00Z","receivedAt":"2007-01-24T16:38:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 24 Jan 2007, H. Peter Anvin wrote:\n\n> Johannes Schindelin wrote:\n> > \n> > To me, it seems like all boils down to caching parsed data structures. I.e.\n> > parse the config, then serialize the parsed data to a file. Don't reparse\n> > the config unless it is 1 hour older than the config.\n> > \n> > Likewise, run for-each-ref, and serialize the parsed data into a file. Don't\n> > rerun for-each-ref if that file is younger than 15 minutes.\n> > \n> > Maybe the same for the first 200 commits of each branch.\n> > \n> > (I made those times up, but you get the idea.)\n> > \n> \n> A much better idea is to have that data structure updated on repository \n> updates, which is the whole point behind .git/info/refs.  On kernel.org, \n> at least, if you don't keep .git/info/refs up to date you need to get \n> your fingers whacked anyway, since it damages usability for one \n> particular class of users.\n\nGranted, for some things this might work. However, I would not wreak havoc \nby changing the format of .git/info/refs, rather put the details you \nwanted into .git/info/refs-details.\n\nHowever, for other things (like showing a certain number of commits), it \n_might_ make sense to cache them (e.g. when literally thousands of people \nlook at the 100 last commits of linux-2.6.git), but not for others (e.g. \nthe 100th last to the 200th last commit of git-tools.git).\n\nHaving said that, it should be relatively easy to store the (parsed, or at \nleast easily parseable) 500 last commits of a branch into \n.git/info/commits-<branch>.\n\nThis would put the burden of publishing a branch higher, easening the \noverall load on the server.\n\nJakub?\n\nCiao,\nDscho\n"},{"id":"32533","messageId":"45B78C55.2030204@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701241731400.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T16:41:57Z","receivedAt":"2007-01-24T16:41:57Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> \n> Granted, for some things this might work. However, I would not wreak havoc \n> by changing the format of .git/info/refs, rather put the details you \n> wanted into .git/info/refs-details.\n> \n\nIt's not clear to me if it would be wrecking havoc.  After all, if a \nformat can't be expanded *at all*, there is something wrong, and adding \nthings to the end of a line is a common structured way of expansion. \nHence the original query\n\n> However, for other things (like showing a certain number of commits), it \n> _might_ make sense to cache them (e.g. when literally thousands of people \n> look at the 100 last commits of linux-2.6.git), but not for others (e.g. \n> the 100th last to the 200th last commit of git-tools.git).\n\nAny query that's within a repository is fairly easily cachable \npost-generation.  The front page (and its RSS variant) is a bit of an \nexception, because it involves all repositories at once.\n\nDoesn't mean we couldn't do better, but...\n\n\t-hpa\n"},{"id":"32536","messageId":"Pine.LNX.4.63.0701241746390.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45B78C55.2030204@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-24T16:52:05Z","receivedAt":"2007-01-24T16:52:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 24 Jan 2007, H. Peter Anvin wrote:\n\n> Johannes Schindelin wrote:\n> > \n> > Granted, for some things this might work. However, I would not wreak \n> > havoc by changing the format of .git/info/refs, rather put the details \n> > you wanted into .git/info/refs-details.\n> > \n> \n> It's not clear to me if it would be wrecking havoc.  After all, if a \n> format can't be expanded *at all*, there is something wrong, and adding \n> things to the end of a line is a common structured way of expansion. \n> Hence the original query\n\nThe idea of .git/info/refs is to enable dumb transports to fetch something \nakin to intelligently. They don't need that information, and frankly, I \ndon't think they should need to understand it.\n\nI also expect that they interpret everything after the sha1 as refname, \nwhat with our having become quite liberal with refnames (they can contain \nspaces, tabs, and even a small amount of special K). So I don't see a way \nto upgrade the file format.\n\nBut as should be clear by now, I'd prefer additional information -- that \nis of no interest to dumb transports anyway -- to be put in an own file.\n\nThat also opens the possibility of, say .git/info/perl/, which contains \n_only_ serialized perl objects! I imagine this could be a performance \nbooster.\n\n> > However, for other things (like showing a certain number of commits), \n> > it _might_ make sense to cache them (e.g. when literally thousands of \n> > people look at the 100 last commits of linux-2.6.git), but not for \n> > others (e.g. the 100th last to the 200th last commit of \n> > git-tools.git).\n> \n> Any query that's within a repository is fairly easily cachable \n> post-generation.  The front page (and its RSS variant) is a bit of an \n> exception, because it involves all repositories at once.\n\n... and here we have a problem, right? No single update hook can update \nthe _whole_ information.\n\nCiao,\nDscho\n"},{"id":"32539","messageId":"45B79208.2080502@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701241746390.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T17:06:16Z","receivedAt":"2007-01-24T17:06:16Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 24 Jan 2007, H. Peter Anvin wrote:\n> \n>> Johannes Schindelin wrote:\n>>> Granted, for some things this might work. However, I would not wreak \n>>> havoc by changing the format of .git/info/refs, rather put the details \n>>> you wanted into .git/info/refs-details.\n>>>\n>> It's not clear to me if it would be wrecking havoc.  After all, if a \n>> format can't be expanded *at all*, there is something wrong, and adding \n>> things to the end of a line is a common structured way of expansion. \n>> Hence the original query\n> \n> The idea of .git/info/refs is to enable dumb transports to fetch something \n> akin to intelligently. They don't need that information, and frankly, I \n> don't think they should need to understand it.\n\nI don't think adding 10 digits to each line is going to be a sizable \nimpact on anything.\n\n> I also expect that they interpret everything after the sha1 as refname, \n> what with our having become quite liberal with refnames (they can contain \n> spaces, tabs, and even a small amount of special K). So I don't see a way \n> to upgrade the file format.\n\nThey can also contain newlines, probably, so escaping is obligatory anyway.\n\n> But as should be clear by now, I'd prefer additional information -- that \n> is of no interest to dumb transports anyway -- to be put in an own file.\n\nYes, but the argument seems to be philosophical.\n\n> That also opens the possibility of, say .git/info/perl/, which contains \n> _only_ serialized perl objects! I imagine this could be a performance \n> booster.\n\nFor certain things, I'm sure.\n\n>>> However, for other things (like showing a certain number of commits), \n>>> it _might_ make sense to cache them (e.g. when literally thousands of \n>>> people look at the 100 last commits of linux-2.6.git), but not for \n>>> others (e.g. the 100th last to the 200th last commit of \n>>> git-tools.git).\n>> Any query that's within a repository is fairly easily cachable \n>> post-generation.  The front page (and its RSS variant) is a bit of an \n>> exception, because it involves all repositories at once.\n> \n> ... and here we have a problem, right? No single update hook can update \n> the _whole_ information.\n\nI don't see a problem.\n\n\t-hpa\n"},{"id":"32540","messageId":"ep83m2$mts$1@sea.gmane.org","threadId":"6499","inReplyTo":"45B78C55.2030204@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-24T17:10:03Z","receivedAt":"2007-01-24T17:10:03Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin wrote:\n\n> Johannes Schindelin wrote:\n>> \n>> Granted, for some things this might work. However, I would not wreak havoc \n>> by changing the format of .git/info/refs, rather put the details you \n>> wanted into .git/info/refs-details.\n> \n> It's not clear to me if it would be wrecking havoc.  After all, if a \n> format can't be expanded *at all*, there is something wrong, and adding \n> things to the end of a line is a common structured way of expansion. \n> Hence the original query\n\nI don't think it can be easily expanded. .git/info/refs is meant for\nhttp-fetch, and it mimics git-ls-remote / git-peek-remote output.\n\nBTW. putting the info of git-for-each-ref into .git/info/refs-details\nwould mean that instead of \"24175 calls to git\" one would need to\nread 24175 files. Perhaps the whole info needed to generate projects\nindex page should be pre-generated on push (update), instead of per\nproject (per repository) .git/info/refs-details\n\n>> However, for other things (like showing a certain number of commits), it \n>> _might_ make sense to cache them (e.g. when literally thousands of people \n>> look at the 100 last commits of linux-2.6.git), but not for others (e.g. \n>> the 100th last to the 200th last commit of git-tools.git).\n> \n> Any query that's within a repository is fairly easily cachable \n> post-generation.  The front page (and its RSS variant) is a bit of an \n> exception, because it involves all repositories at once.\n\nActually \"RSS\", or to be more exact OPML variant of front page in its\ncurrent invocation is equivalent of project_index page, and it can be\ngenerated once (well, once per adding / removing / renaming a repository).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"32541","messageId":"Pine.LNX.4.63.0701241817240.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"ep83m2$mts$1@sea.gmane.org","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-24T17:20:37Z","receivedAt":"2007-01-24T17:20:37Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 24 Jan 2007, Jakub Narebski wrote:\n\n> H. Peter Anvin wrote:\n> \n> > Johannes Schindelin wrote:\n> >> \n> >> Granted, for some things this might work. However, I would not wreak havoc \n> >> by changing the format of .git/info/refs, rather put the details you \n> >> wanted into .git/info/refs-details.\n> > \n> > It's not clear to me if it would be wrecking havoc.  After all, if a \n> > format can't be expanded *at all*, there is something wrong, and adding \n> > things to the end of a line is a common structured way of expansion. \n> > Hence the original query\n> \n> I don't think it can be easily expanded. .git/info/refs is meant for\n> http-fetch, and it mimics git-ls-remote / git-peek-remote output.\n\nExactly.\n\n> BTW. putting the info of git-for-each-ref into .git/info/refs-details\n> would mean that instead of \"24175 calls to git\" one would need to\n> read 24175 files. Perhaps the whole info needed to generate projects\n> index page should be pre-generated on push (update), instead of per\n> project (per repository) .git/info/refs-details\n\nYou completely lost me there. A push (update) is done as a specific user, \nwho should not be able to write to a _global_ file!\n\nNevertheless, \"24175 calls to git\" is sure as hell more expensive than \n\"reading 24175 files\".\n\nPlus, if we integrate the functionality to write .git/info/refs-details \ninto update-server-info, you can reduce that further: it is no longer \nper-branch, but per-repo.\n\nCiao,\nDscho\n"},{"id":"32543","messageId":"ep83ur$p3i$2@sea.gmane.org","threadId":"6499","inReplyTo":"45B79208.2080502@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-24T17:25:38Z","receivedAt":"2007-01-24T17:25:38Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin wrote:\n\n>> I also expect that they interpret everything after the sha1 as refname, \n>> what with our having become quite liberal with refnames (they can contain \n>> spaces, tabs, and even a small amount of special K). So I don't see a way \n>> to upgrade the file format.\n> \n> They can also contain newlines, probably, so escaping is obligatory\n> anyway. \n\nNo, refnames can not contain newlines.\n\nAnd .git/info/refs mimics git-ls-remote / git-peek-remote output,\nand is meant for dumb transports.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"32555","messageId":"200701242140.07459.jnareb@gmail.com","threadId":"6499","inReplyTo":"45B7818F.6020805@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-24T20:40:06Z","receivedAt":"2007-01-24T20:40:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin wrote:\n> Jakub Narebski wrote:\n\n>> Besides, we can't rely that .git/info/refs is up to date, or even exists.\n>> It is for dumb protocols, not for gitweb.\n> \n> Well, SOMETHING needs to be done for this page, since it can take 15 \n> minutes or more to generate.  Caching doesn't help one iota, since it's \n> stale before being generated.\n\nThe simple and fast solution would be to make post-update hook contain\nthe git-for-each-ref with parameters like in git_get_last_activity,\nsaving e.g. to .git/info/last-committer, and in gitweb read this file\nif it exist, run git-for-each-ref otherwise (similar to what we used to\ndo with .git/info/refs and git-peek-remote in gitweb).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"32559","messageId":"2197.69.104.58.50.1169671494.squirrel@www.zytor.com","threadId":"6499","inReplyTo":"200701242140.07459.jnareb@gmail.com","subject":"Re: .git/info/refs","fromName":"","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T20:44:54Z","receivedAt":"2007-01-24T20:44:54Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"> H. Peter Anvin wrote:\n>\n>>> Besides, we can't rely that .git/info/refs is up to date, or even\n>>> exists.\n>>> It is for dumb protocols, not for gitweb.\n>>\n>> Well, SOMETHING needs to be done for this page, since it can take 15\n>> minutes or more to generate.  Caching doesn't help one iota, since it's\n>> stale before being generated.\n>\n> The simple and fast solution would be to make post-update hook contain\n> the git-for-each-ref with parameters like in git_get_last_activity,\n> saving e.g. to .git/info/last-committer, and in gitweb read this file\n> if it exist, run git-for-each-ref otherwise (similar to what we used to\n> do with .git/info/refs and git-peek-remote in gitweb).\n>\n\nRight, this is basically what I'm saying; the question is only whether or\nnot this fits into .git/info/refs or should be a separate file.\n\nEither way, I think git-update-server-info should generate all these files.\n\n    -hpa\n"},{"id":"32560","messageId":"2198.69.104.58.50.1169671500.squirrel@www.zytor.com","threadId":"6499","inReplyTo":"200701242140.07459.jnareb@gmail.com","subject":"Re: .git/info/refs","fromName":"","fromEmail":"hpa@zytor.com","sentAt":"2007-01-24T20:45:00Z","receivedAt":"2007-01-24T20:45:00Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"> H. Peter Anvin wrote:\n>\n>>> Besides, we can't rely that .git/info/refs is up to date, or even\n>>> exists.\n>>> It is for dumb protocols, not for gitweb.\n>>\n>> Well, SOMETHING needs to be done for this page, since it can take 15\n>> minutes or more to generate.  Caching doesn't help one iota, since it's\n>> stale before being generated.\n>\n> The simple and fast solution would be to make post-update hook contain\n> the git-for-each-ref with parameters like in git_get_last_activity,\n> saving e.g. to .git/info/last-committer, and in gitweb read this file\n> if it exist, run git-for-each-ref otherwise (similar to what we used to\n> do with .git/info/refs and git-peek-remote in gitweb).\n>\n\nRight, this is basically what I'm saying; the question is only whether or\nnot this fits into .git/info/refs or should be a separate file.\n\nEither way, I think git-update-server-info should generate all these files.\n\n    -hpa\n"},{"id":"32591","messageId":"Pine.LNX.4.63.0701250900380.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"2197.69.104.58.50.1169671494.squirrel@www.zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-25T08:14:05Z","receivedAt":"2007-01-25T08:14:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 24 Jan 2007, hpa@zytor.com wrote:\n\n> > H. Peter Anvin wrote:\n> >\n> >>> Besides, we can't rely that .git/info/refs is up to date, or even\n> >>> exists.\n> >>> It is for dumb protocols, not for gitweb.\n> >>\n> >> Well, SOMETHING needs to be done for this page, since it can take 15\n> >> minutes or more to generate.  Caching doesn't help one iota, since it's\n> >> stale before being generated.\n> >\n> > The simple and fast solution would be to make post-update hook contain\n> > the git-for-each-ref with parameters like in git_get_last_activity,\n> > saving e.g. to .git/info/last-committer, and in gitweb read this file\n> > if it exist, run git-for-each-ref otherwise (similar to what we used to\n> > do with .git/info/refs and git-peek-remote in gitweb).\n> >\n> \n> Right, this is basically what I'm saying; the question is only whether or\n> not this fits into .git/info/refs or should be a separate file.\n> \n> Either way, I think git-update-server-info should generate all these files.\n\nWell, no. At least not per default. What you want is _very_ special to \ngitweb. It is _only_ needed by gitweb. And .git/info/refs is for _dumb \ntransports_, _not_ for gitweb.\n\nThat said, I think it makes sense _in your setup_ to trigger updating \n_another_ file for use in gitweb.\n\nRemember, this is all very, very special for gitweb. So let's separate it \ncleanly from all which is not special for gitweb.\n\nI hope I have made it clear why (at least IMHO) it would be wrong, wrong, \nwrong to change the format of .git/info/refs _only_ for gitweb, which it \nis not meant for to begin with.\n\nSo let's introduce another file in .git/info/ especially dedicated to \ngitweb.\n\nThen we are free to introduce real cool performance hacks, like using \nStorable to store the parsed data structures (I was alluding to this in an \nearlier reply, as \"serializing\"). Then you just retrieve the file -- if it \nexists -- or call for-each-ref (like Jakub said).\n\nBy separating this gitweb-special thing cleanly, maybe into a hook, we can \nhave a perl script which writes this file. We can write a simple hash, \nwhich may or may not contain keys, thus being of \"extensible format\".\n\nBy having this perl script, you can -- as root -- run it as the \nappropriate user for each repository where it does not exist yet.\n\nRemains the problem: how do we _force_ this hook enabled site-wide, i.e. \nin _all_ repos?\n\nBut that is too easy: just edit the existing template, and then replace \nthe update hooks in all repos (possibly verifying that the existing update \nhook indeed matches the old template).\n\nSo what problems remain with this approach?\n\nCiao,\nDscho\n"},{"id":"32615","messageId":"45B8D704.9040905@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701250900380.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-25T16:12:52Z","receivedAt":"2007-01-25T16:12:52Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n>>\n>> Either way, I think git-update-server-info should generate all these files.\n> \n> Well, no. At least not per default. What you want is _very_ special to \n> gitweb. It is _only_ needed by gitweb. And .git/info/refs is for _dumb \n> transports_, _not_ for gitweb.\n> \n> That said, I think it makes sense _in your setup_ to trigger updating \n> _another_ file for use in gitweb.\n> \n> Remember, this is all very, very special for gitweb. So let's separate it \n> cleanly from all which is not special for gitweb.\n> \n> I hope I have made it clear why (at least IMHO) it would be wrong, wrong, \n> wrong to change the format of .git/info/refs _only_ for gitweb, which it \n> is not meant for to begin with.\n\nNo, you keep using circular reasoning.\n\n\t-hpa\n"},{"id":"32620","messageId":"Pine.LNX.4.63.0701251749400.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45B8D704.9040905@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-25T16:50:53Z","receivedAt":"2007-01-25T16:50:53Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 25 Jan 2007, H. Peter Anvin wrote:\n\n> > I hope I have made it clear why (at least IMHO) it would be wrong, \n> > wrong, wrong to change the format of .git/info/refs _only_ for gitweb, \n> > which it is not meant for to begin with.\n> \n> No, you keep using circular reasoning.\n\nNo. Once again, .git/info/refs is _not_ for gitweb. But I will stop \narguing about that topic, because I don't have enough time for that.\n\nCiao,\nDscho\n"},{"id":"32624","messageId":"45B8E551.9020808@zytor.com","threadId":"6499","inReplyTo":"ep83m2$mts$1@sea.gmane.org","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-25T17:13:53Z","receivedAt":"2007-01-25T17:13:53Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Jakub Narebski wrote:\n> \n> I don't think it can be easily expanded. .git/info/refs is meant for\n> http-fetch, and it mimics git-ls-remote / git-peek-remote output.\n\nFor heaven's sake, in computer science we can *NEVER* use the same \nfeature for *MORE THAN ONE THING*.  If it doesn't work format-wise \nthat's fine, but \"it's only supposed to be used by dumb transports\" is \nridiculous.\n\n> BTW. putting the info of git-for-each-ref into .git/info/refs-details\n> would mean that instead of \"24175 calls to git\" one would need to\n> read 24175 files. Perhaps the whole info needed to generate projects\n> index page should be pre-generated on push (update), instead of per\n> project (per repository) .git/info/refs-details\n\nNo, it should be one file per repository, not one file per ref.  Why? \nObviously we don't want 24175 files to be accessed.  However, a push can \nonly affect files for which the repository owner has permission and \nwhich resides in the repository filespace, so it should stay inside that \n  space.\n\nOn kernel.org, this would reduce the load from 24175 calls to git to \nreading 250 files.  Although the latter is still expensive (and will \nprobably need post-generation caching) the files should be small and \ncacheable by the kernel, and the resulting I/O load should be quite small.\n\nAnyway, as far as git-update-server-index is concerned, I'm *very* \nconcerned that there be a single command that updates all the cached \ninformation across the repository.  Telling everyone to update their \nhooks every time we want to add cached information is silly.  Right now, \ngit-update-server-index is the command to update cached information, and \nfor usability reasons there should be a single entry point.\n\n\t-hpa\n"},{"id":"32640","messageId":"7vireuaj9d.fsf@assigned-by-dhcp.cox.net","threadId":"6499","inReplyTo":"45B7818F.6020805@zytor.com","subject":"Re: .git/info/refs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-25T21:28:30Z","receivedAt":"2007-01-25T21:28:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"H. Peter Anvin\" <hpa@zytor.com> writes:\n\n> Jakub Narebski wrote:\n>> H. Peter Anvin wrote:\n>>\n>>> Would it be an incompatible change to add the commit date (and\n>>> perhaps the author date) to .git/info/refs?  I believe that would\n>>> make it possible to dramatically (orders of magnitude) speed up the\n>>> generation of the gitweb index page, which is easily the most\n>>> expensive gitweb page to generate.\n>>\n>> With new gitweb and new git it is not that expensive. It is now one call\n>> to git-for-each-ref per repository.\n>\n> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n\nDo you mean you have 24k _REPOSITORIES_ served by gitweb on\nkernel.org?\n"},{"id":"32643","messageId":"45B92332.5060206@zytor.com","threadId":"6499","inReplyTo":"7vireuaj9d.fsf@assigned-by-dhcp.cox.net","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-25T21:37:54Z","receivedAt":"2007-01-25T21:37:54Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Junio C Hamano wrote:\n>>>\n>>>> Would it be an incompatible change to add the commit date (and\n>>>> perhaps the author date) to .git/info/refs?  I believe that would\n>>>> make it possible to dramatically (orders of magnitude) speed up the\n>>>> generation of the gitweb index page, which is easily the most\n>>>> expensive gitweb page to generate.\n>>> With new gitweb and new git it is not that expensive. It is now one call\n>>> to git-for-each-ref per repository.\n>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n> \n> Do you mean you have 24k _REPOSITORIES_ served by gitweb on\n> kernel.org?\n> \n\nNo, we currently have 250 repositories with a total of 24175 refs.\n\n\t-hpa\n"},{"id":"32646","messageId":"7v3b5yai6c.fsf@assigned-by-dhcp.cox.net","threadId":"6499","inReplyTo":"45B92332.5060206@zytor.com","subject":"Re: .git/info/refs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-25T21:51:55Z","receivedAt":"2007-01-25T21:51:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"H. Peter Anvin\" <hpa@zytor.com> writes:\n\n>>>> With new gitweb and new git it is not that expensive. It is now one call\n>>>> to git-for-each-ref per repository.\n>>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n>>\n>> Do you mean you have 24k _REPOSITORIES_ served by gitweb on\n>> kernel.org?\n>\n> No, we currently have 250 repositories with a total of 24175 refs.\n\nThen that would mean 250 calls to git-for-each-ref, wouldn't it?\n"},{"id":"32648","messageId":"45B928AD.50508@zytor.com","threadId":"6499","inReplyTo":"7v3b5yai6c.fsf@assigned-by-dhcp.cox.net","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-25T22:01:17Z","receivedAt":"2007-01-25T22:01:17Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Junio C Hamano wrote:\n> \"H. Peter Anvin\" <hpa@zytor.com> writes:\n> \n>>>>> With new gitweb and new git it is not that expensive. It is now one call\n>>>>> to git-for-each-ref per repository.\n>>>> That IS hugely expensive.  On kernel.org, that is 24175 calls to git.\n>>> Do you mean you have 24k _REPOSITORIES_ served by gitweb on\n>>> kernel.org?\n>> No, we currently have 250 repositories with a total of 24175 refs.\n> \n> Then that would mean 250 calls to git-for-each-ref, wouldn't it?\n> \n\nWell, I think it was Johannes that said once for each ref.  But either \nwhich way, it's a totally unacceptable load with resulting unacceptable \nlatency.\n\n\t-hpa\n"},{"id":"32660","messageId":"Pine.LNX.4.63.0701260029580.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45B928AD.50508@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-25T23:33:51Z","receivedAt":"2007-01-25T23:33:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 25 Jan 2007, H. Peter Anvin wrote:\n\n> Junio C Hamano wrote:\n>\n> > Then that would mean 250 calls to git-for-each-ref, wouldn't it?\n> \n> Well, I think it was Johannes that said once for each ref.  But either \n> which way, it's a totally unacceptable load with resulting unacceptable \n> latency.\n\nNo. I would never say that you have to run for-each-ref for each ref. \nThat's plain stupid.\n\nBTW I take some satisfaction in that you finally agreed (in another email) \nthat some post-creation caching is necessary.\n\nI would be even more satisfied if you finally agreed that it is a good \npractice to separate conceptually different things, and not continued ad \ninfinitum (and ad nauseam) arguing that .git/info/refs should serve dumb \ntransports, and gitweb, and eventually bring peace to everybody on this \nplanet.\n\nCiao,\nDscho\n"},{"id":"32721","messageId":"200701261222.50725.jnareb@gmail.com","threadId":"6499","inReplyTo":"45B8E551.9020808@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-26T11:22:49Z","receivedAt":"2007-01-26T11:22:49Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin <hpa@zytor.com> wrote:\n> Jakub Narebski wrote:\n>> \n>> I don't think it can be easily expanded. .git/info/refs is meant for\n>> http-fetch, and it mimics git-ls-remote / git-peek-remote output.\n> \n> For heaven's sake, in computer science we can *NEVER* use the same \n> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise \n> that's fine, but \"it's only supposed to be used by dumb transports\" is \n> ridiculous.\n\n.git/info/refs is for dumb transports, so if we follow \"do not use\nthe same feature for more than one thing\" principle we should not\nchange its format for gitweb.\n\n.git/info/refs is one of auxiliary info files to help dumb servers,\n(servers that does not do on-the-fly pack generation), to help\nclients discover what references server has.  The second auxiliary\ninfo file is .git/objects/info/packs.  Both are generated by\ngit-update-server-info command, usually run from post-update hook.\n\nBecause .git/info/refs format is the same as git-ls-remote output\n(AFAIK smart servers use git-ls-remote or git-peek-remote; dumb\nservers use .git/info/refs) we used and can use it as ''cached''\n\"git ls-remote .\" / \"git peek-remote .\" / \"git show-ref --dereference\"\noutput. For bare repositories where new data arrives only via\n'update' (via push or fetch) and always trigger post-update hook,\nand not for example via git-commit which does not invoke post-update\nhook, the information in .git/info/refs is always fresh.\n\nWhat I propose as quick solution is to add new (perhaps local)\ngit-update-gitweb-info command which is to be used in post-update\n(and perhaps post-commit for non-bare repos) hook, and which results\nwe would use in gitweb.  See patch at the bottom.\n\n>> BTW. putting the info of git-for-each-ref into .git/info/refs-details\n>> would mean that instead of \"24175 calls to git\" one would need to\n>> read 24175 files. Perhaps the whole info needed to generate projects\n>> index page should be pre-generated on push (update), instead of per\n>> project (per repository) .git/info/refs-details\n> \n> No, it should be one file per repository, not one file per ref.  Why? \n> Obviously we don't want 24175 files to be accessed.  However, a push can \n> only affect files for which the repository owner has permission and \n> which resides in the repository filespace, so it should stay inside that \n> space.\n\nGitweb _newer_ did one call to git _per ref_, but always one call to git\n_per repository_!  Old git always used HEAD ref to get \"Last Change\" info\nand used one call to git-rev-list (if I remember correctly), new git\nchecks all refs to get \"Last Change\" info but uses _one_ call to \ngit-for-each-ref.  Because we did not want to affect gitweb performance\nbadly we waited for changing \"Last Change\" to check all refs and not\nonly HEAD to have git-for-each-ref to use one call to git command for that.\nHistorically it was first use of git-for-each-ref in gitweb.\n\nSidenote: I planned to add new %feature to gitweb to allow to chose\nif to use all refs for \"Last Change\" info, HEAD ref, or some given ref\n(for example \"master\").  But that would perhaps wait for .git/config\nparser in Perl.\n\n> On kernel.org, this would reduce the load from 24175 calls to git to \n> reading 250 files.  Although the latter is still expensive (and will \n> probably need post-generation caching) the files should be small and \n> cacheable by the kernel, and the resulting I/O load should be quite small.\n\nOh, so there are around 250 projects, and around 24175 references\ntogether in those projects on kernel.org?  I thought it were 24175\n_projects_ (repositories)...\n\nCurrently, it is 250 calls to git, reading 24175 files (unless refs\nare packed, then it would be reading 250 files) to get refs (heads)\ninfo, and reading around 2*250 files (packs + index) to get last\nchange info.  Not \"24175 calls to git\".\n\n> Anyway, as far as git-update-server-info is concerned, I'm *very* \n> concerned that there be a single command that updates all the cached \n> information across the repository.  Telling everyone to update their \n> hooks every time we want to add cached information is silly.  Right now, \n> git-update-server-info is the command to update cached information, and \n> for usability reasons there should be a single entry point.\n\ngit-update-server-info is to \"update auxiliary info file to help dumb\nservers\". I propose to use (new) git-update-gitweb-info to help gitweb.\nOne command for one feature.  This would mean unfortunately adding\n\"exec git-update-gitweb-info\" line (if it does not exist) to existing\nprojects post-update hooks; for new projects it would be I think enough\nto modify post-update template (templates/hooks--post-update or\n/usr/share/git-core/templates/hooks/post-update).\n\n\nBelow the patches of how it can be done.  Does not include corrections\nto Makefile to install git-update-gitweb-info.  NOT TESTED!\n\nBTW final version of git-update-gitweb-info probably should be a built-in\ncommand, like git-update-server-info, not a script.\n\n\ndiff --git a/git-update-gitweb-info.sh b/git-update-gitweb-info.sh\nnew file mode 100755\nindex 0000000..5bb44df\n--- /dev/null\n+++ b/git-update-gitweb-info.sh\n@@ -0,0 +1,7 @@\n+#!/bin/sh\n+\n+. git-sh-setup\n+test -w \"$GIT_DIR/info/last-changed\" &&\n+git-for-each-ref \\\n+\t--format='%(committer)' --sort=-committerdate --count=1 refs/heads \\\n+\t> \"$GIT_DIR/info/last-changed\"\ndiff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl\nindex 88af2e6..e7874a6 100755\n--- a/gitweb/gitweb.perl\n+++ b/gitweb/gitweb.perl\n@@ -1150,12 +1150,16 @@ sub git_get_last_activity {\n \tmy ($path) = @_;\n \tmy $fd;\n \n-\t$git_dir = \"$projectroot/$path\";\n-\topen($fd, \"-|\", git_cmd(), 'for-each-ref',\n-\t     '--format=%(committer)',\n-\t     '--sort=-committerdate',\n-\t     '--count=1',\n-\t     'refs/heads') or return;\n+\tif (-r \"$projectroot/$path/info/last-changed\") {\n+\t\topen $fd, \"$projectroot/$path/info/last-changed\";\n+\t} else {\n+\t\t$git_dir = \"$projectroot/$path\";\n+\t\topen($fd, \"-|\", git_cmd(), 'for-each-ref',\n+\t\t     '--format=%(committer)',\n+\t\t     '--sort=-committerdate',\n+\t\t     '--count=1',\n+\t\t     'refs/heads') or return;\n+\t}\n \tmy $most_recent = <$fd>;\n \tclose $fd or return;\n \tif ($most_recent =~ / (\\d+) [-+][01]\\d\\d\\d$/) {\ndiff --git a/templates/hooks--post-update b/templates/hooks--post-update\nold mode 100644\nnew mode 100755\nindex bcba893..b119224\n--- a/templates/hooks--post-update\n+++ b/templates/hooks--post-update\n@@ -6,3 +6,4 @@\n # To enable this hook, make this file executable by \"chmod +x post-update\".\n \n exec git-update-server-info\n+exec git-update-gitweb-info\n\n-- \nJakub Narebski\nPoland\n"},{"id":"32725","messageId":"7vireuxbel.fsf@assigned-by-dhcp.cox.net","threadId":"6499","inReplyTo":"45B8E551.9020808@zytor.com","subject":"Re: .git/info/refs","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-26T11:41:54Z","receivedAt":"2007-01-26T11:41:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"H. Peter Anvin\" <hpa@zytor.com> writes:\n\n> For heaven's sake, in computer science we can *NEVER* use the same\n> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise\n> that's fine, but \"it's only supposed to be used by dumb transports\" is\n> ridiculous.\n\nHmmmm... I am lost here....\n\n> Right\n> now, git-update-server-index is the command to update cached\n> information, and for usability reasons there should be a single entry\n> point.\n\nModulo s/-index/-info/, I agree that would be a very sensible\nposition, as long as the cost to generate additional cached\ninformation necessary to help gitweb is reasonably small, I am\nnot opposed to have it generate another file [*1*].\n\n\n[*1*]\n\nI've been looking for backward-compatible holes in ls-remote and\nits users, hoping we somehow could shoehorn this information in\ninfo/refs, as I do not think its file format is sacred, nor the\nfile is there _only_ to help dumb transports.  As long as the\npublished way to access that information stays consistent, the\nunderlying file format is a fair game.  However, I do not think\nthe ls-remote command implementations in the wild has such a\nhole I can exploit.\n"},{"id":"32753","messageId":"45BA2ED2.7080807@zytor.com","threadId":"6499","inReplyTo":"7vireuxbel.fsf@assigned-by-dhcp.cox.net","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-26T16:39:46Z","receivedAt":"2007-01-26T16:39:46Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Junio C Hamano wrote:\n> \"H. Peter Anvin\" <hpa@zytor.com> writes:\n> \n>> For heaven's sake, in computer science we can *NEVER* use the same\n>> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise\n>> that's fine, but \"it's only supposed to be used by dumb transports\" is\n>> ridiculous.\n> \n> Hmmmm... I am lost here....\n> \n\nJakub and Johannes seems to have been arguing that \"info/refs is for \ndumb transports, therefore it cannot be used for any other purpose.\"  I \nfind this argument utterly bizarre, since in general, in computer \nscience, you try to be multipurpose whenever practical.\n\n\t-hpa\n"},{"id":"32755","messageId":"200701261806.37488.jnareb@gmail.com","threadId":"6499","inReplyTo":"45BA2ED2.7080807@zytor.com","subject":"Re: .git/info/refs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-01-26T17:06:36Z","receivedAt":"2007-01-26T17:06:36Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"H. Peter Anvin wrote:\n> Junio C Hamano wrote:\n>> \"H. Peter Anvin\" <hpa@zytor.com> writes:\n>> \n>>> For heaven's sake, in computer science we can *NEVER* use the same\n>>> feature for *MORE THAN ONE THING*.  If it doesn't work format-wise\n>>> that's fine, but \"it's only supposed to be used by dumb transports\" is\n>>> ridiculous.\n\nPlease, for the future, mark irony if it might be mistaken...\n \n>> Hmmmm... I am lost here....\n>\n> Jakub and Johannes seems to have been arguing that \"info/refs is for \n> dumb transports, therefore it cannot be used for any other purpose.\"  I \n> find this argument utterly bizarre, since in general, in computer \n> science, you try to be multipurpose whenever practical.\n\nFirst, changing info/refs format _might_ break fetch related scripts,\nwhich rely on git-peek-remote / git-ls-remote / info/refs format.\n\nSecond, it is a bit impractical because info/refs contain (and must\ncontain) also _tags_ information (which is not needed for gitweb\n\"Last Change\" field in projects list) and referenced object for\nthose tags. Tags need not to point to commits, nor dereference\nto commits: for example in git.git tags v1.0rc1 to v1.0rc6 points\nto other tags, and junio-gpg-pub point to out-of-tree blob (which\ndoes not have any \"commit time\" associated). So what to write there\nin the \"commit time\" field? What to write in \"commit time\" for tags?\n-- \nJakub Narebski\nPoland\n"},{"id":"32763","messageId":"Pine.LNX.4.63.0701262200131.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45BA2ED2.7080807@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-01-26T21:09:16Z","receivedAt":"2007-01-26T21:09:16Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 26 Jan 2007, H. Peter Anvin wrote:\n\n> Junio C Hamano wrote:\n> > \"H. Peter Anvin\" <hpa@zytor.com> writes:\n> > \n> > > For heaven's sake, in computer science we can *NEVER* use the same\n> > > feature for *MORE THAN ONE THING*.  If it doesn't work format-wise\n> > > that's fine, but \"it's only supposed to be used by dumb transports\" is\n> > > ridiculous.\n> > \n> > Hmmmm... I am lost here....\n> > \n> \n> Jakub and Johannes seems to have been arguing that \"info/refs is for dumb\n> transports, therefore it cannot be used for any other purpose.\"  I find this\n> argument utterly bizarre, since in general, in computer science, you try to be\n> multipurpose whenever practical.\n\nYou keep on harping on that issue. I really get tired of it.\n\nYou seem to propose that we should stuff things into .git/info/refs, just \nbecause it is already there.\n\nYou seem to suggest that computer science is all about breaking things, to \nmuddle waters by mixing things which are clearly different kinds of \nkettle, to \"just\" add a small thing here and there, all under the guise of \nmulti-purposity or whatever.\n\nYou seem to reason that practicality is more important than good style.\n\nYou know, this reasoning brought to us that big crap sh*tpile called \nWindows. They also thought: it's not a big deal, let's introduce just a \nlittle thing here and there, and a direct call from this component to that \ncomponent cannot hurt, cannit?\n\nI am really, really getting tired of that kind of reasoning.\n\nIf you don't see how UNELEGANT it is to force dumb transports to download \nthings MEANT FOR GITWEB, and how much NICER it would be to have a file \nWHICH IS ONLY MEANT FOR GITWEB TO BEGIN WITH, and which is easily \nEXTENSIBLE, since we don't have to CARE about DUMB TRANSPORTS, because \ngitweb data is a PURELY LOCAL thing, while dumb transports are NOT, I will \njust start to ignore you.\n\nCiao,\nDscho\n"},{"id":"32765","messageId":"45BA7365.1090302@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701262200131.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-26T21:32:21Z","receivedAt":"2007-01-26T21:32:21Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n[... stuff ...]\n\nI really could care less, as long as a single invocation is used to \nupdate the cache information.\n\nI disagree with your aesthetic argument, but it doesn't matter much to \nme either way.\n\n\t-hpa\n"},{"id":"32766","messageId":"45BA7878.9000300@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701262200131.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-26T21:54:00Z","receivedAt":"2007-01-26T21:54:00Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"By the way, let me be the first to apologize for the emotional \nescalation.  What matters to me is that the information is cached and \nupdated by a common cached information entry point (the existing \ngit-update-server-index would be preferred, obviously), not where the \ninformation ends up.\n\n\t-hpa\n"},{"id":"32884","messageId":"45BBCD27.5050408@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0701260029580.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-01-27T22:07:35Z","receivedAt":"2007-01-27T22:07:35Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> No. I would never say that you have to run for-each-ref for each ref. \n> That's plain stupid.\n\nI went back and looked at the thread, and I had indeed misread the \noriginal message, which was from Jakub, not you.  I think I got in the \n\"this is surreal\" mode as a result of that (invoking for-each-ref 250 \ntimes is bad enough, obviously.)\n\n> BTW I take some satisfaction in that you finally agreed (in another email) \n> that some post-creation caching is necessary.\n\nI don't believe I have ever disputed that (in fact, I have pushed very \nhard for gitweb to do post-creation caching.)\n\n> I would be even more satisfied if you finally agreed that it is a good \n> practice to separate conceptually different things, and not continued ad \n> infinitum (and ad nauseam) arguing that .git/info/refs should serve dumb \n> transports, and gitweb, and eventually bring peace to everybody on this \n> planet.\n\nI've already said I think it's an aesthetic argument, but I don't really \ncare either way, as long as there is only one hook that updates all the \ncaches.  I don't want the user to have to juggle an arbitrary and \nincreasing number of hooks.\n\nFair?\n\n\t-hpa\n"},{"id":"33142","messageId":"8aa486160701310738q31fb4fd2g18e252959f4e5c6f@mail.gmail.com","threadId":"6499","inReplyTo":"45BBCD27.5050408@zytor.com","subject":"Re: .git/info/refs","fromName":"Santi Béjar","fromEmail":"sbejar@gmail.com","sentAt":"2007-01-31T15:38:17Z","receivedAt":"2007-01-31T15:38:17Z","isPatch":false,"sender":{"key":"santi@agolina.net","avatar":null},"body":"On 1/27/07, H. Peter Anvin <hpa@zytor.com> wrote:\n> Johannes Schindelin wrote:\n> > No. I would never say that you have to run for-each-ref for each ref.\n> > That's plain stupid.\n>\n> I went back and looked at the thread, and I had indeed misread the\n> original message, which was from Jakub, not you.  I think I got in the\n> \"this is surreal\" mode as a result of that (invoking for-each-ref 250\n> times is bad enough, obviously.)\n>\n\nNormally I'm not interested in the \"Last Change\" column, I just want\nto go to the project summary page, and normally I'm not interested in\nthe last 16 tags (the last three are just enough). For me they should\nbe show only when explicitly asked.\n\nSanti\n"},{"id":"33235","messageId":"Pine.LNX.4.63.0702011501250.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45BBCD27.5050408@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-01T14:03:01Z","receivedAt":"2007-02-01T14:03:01Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nI just had another idea: why not generate the content of the \"cover page\" \nin a cron job, every minute or so, and save it into a static index.html? \nThis should take quite a load from the server, since not even Perl has to \nbe started to serve that page.\n\nCiao,\nDscho\n"},{"id":"33249","messageId":"45C2124C.5070308@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0702011501250.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-02-01T16:16:12Z","receivedAt":"2007-02-01T16:16:12Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> I just had another idea: why not generate the content of the \"cover page\" \n> in a cron job, every minute or so, and save it into a static index.html? \n> This should take quite a load from the server, since not even Perl has to \n> be started to serve that page.\n> \n\nEhm... because it often takes longer than that to generate the page?\n\nWe can pre-generate the page before the first hit, but that's not a \nreplacement for update-time caching.\n\n\t-hpa\n"},{"id":"33253","messageId":"Pine.LNX.4.63.0702011749100.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6499","inReplyTo":"45C2124C.5070308@zytor.com","subject":"Re: .git/info/refs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-01T16:52:46Z","receivedAt":"2007-02-01T16:52:46Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 1 Feb 2007, H. Peter Anvin wrote:\n\n> Johannes Schindelin wrote:\n> > Hi,\n> > \n> > I just had another idea: why not generate the content of the \"cover page\" in\n> > a cron job, every minute or so, and save it into a static index.html? This\n> > should take quite a load from the server, since not even Perl has to be\n> > started to serve that page.\n> > \n> \n> Ehm... because it often takes longer than that to generate the page?\n\nSorry, I should have been clearer. Plan:\n\n1. echo \"Generating\" > /htdocs/git/index.html\n2. edit crontab to do this every minute:\n2.1 gitweb is called directly_, to generate /htdocs/git/index.html.new\n2.2 /htdocs/git/index.html.new is _moved_ into /htdocs/git/index.html, \n    overwriting the existing one.\n\nYes, there could be two instances of this task concurrently. No, it does \nnot matter.\n\n> We can pre-generate the page before the first hit, but that's not a \n> replacement for update-time caching.\n\nIt was only meant as a quick fix for the horrible workload.\n\nJust a thought, feel free to ignore me,\nDscho\n"},{"id":"33254","messageId":"45C21BA0.4080700@zytor.com","threadId":"6499","inReplyTo":"Pine.LNX.4.63.0702011749100.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-02-01T16:56:00Z","receivedAt":"2007-02-01T16:56:00Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Johannes Schindelin wrote:\n> \n> Sorry, I should have been clearer. Plan:\n> \n> 1. echo \"Generating\" > /htdocs/git/index.html\n> 2. edit crontab to do this every minute:\n> 2.1 gitweb is called directly_, to generate /htdocs/git/index.html.new\n> 2.2 /htdocs/git/index.html.new is _moved_ into /htdocs/git/index.html, \n>     overwriting the existing one.\n> \n> Yes, there could be two instances of this task concurrently. No, it does \n> not matter.\n> \n\nYes, it does matter, because it drives the load up further.  If you \nstart having this going on in overlapping instances, then you're soon on \nthe downhill slope of a cascading failure.\n\n>> We can pre-generate the page before the first hit, but that's not a \n>> replacement for update-time caching.\n> \n> It was only meant as a quick fix for the horrible workload.\n\nAnd we have already experimented with it.  It unfortunately doesn't help \nmuch, it only makes matters worse.\n\n\t-hpa\n"},{"id":"33259","messageId":"20070201173234.GA29170@moooo.ath.cx","threadId":"6499","inReplyTo":"45C21BA0.4080700@zytor.com","subject":"Re: .git/info/refs","fromName":"Matthias Lederhofer","fromEmail":"matled@gmx.net","sentAt":"2007-02-01T17:32:34Z","receivedAt":"2007-02-01T17:32:34Z","isPatch":false,"sender":{"key":"matled@gmx.net","avatar":null},"body":"H. Peter Anvin <hpa@zytor.com> wrote:\n> Yes, it does matter, because it drives the load up further.  If you \n> start having this going on in overlapping instances, then you're soon on \n> the downhill slope of a cascading failure.\nAdd some other locking mechanism.\n\n> And we have already experimented with it.  It unfortunately doesn't help \n> much, it only makes matters worse.\nThe gitweb overview page has less than one hit per minute?  Otherwise\nthis should help.\n"},{"id":"33265","messageId":"45C228B7.7080301@zytor.com","threadId":"6499","inReplyTo":"20070201173234.GA29170@moooo.ath.cx","subject":"Re: .git/info/refs","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-02-01T17:51:51Z","receivedAt":"2007-02-01T17:51:51Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"Matthias Lederhofer wrote:\n> H. Peter Anvin <hpa@zytor.com> wrote:\n>> Yes, it does matter, because it drives the load up further.  If you \n>> start having this going on in overlapping instances, then you're soon on \n>> the downhill slope of a cascading failure.\n> Add some other locking mechanism.\n> \n>> And we have already experimented with it.  It unfortunately doesn't help \n>> much, it only makes matters worse.\n> The gitweb overview page has less than one hit per minute?  Otherwise\n> this should help.\n\nWe already cache it with a forced duration of some 15 minutes.  The end \nresult is exactly the same.\n\n\t-hpa\n"}]}