{"thread":{"id":"1792","subject":"dumb transports not being welcomed..","startedAt":"2005-09-13T21:07:58Z","lastAt":"2005-09-15T09:17:25Z","messageCount":25,"participants":["Junio C Hamano","Sam Ravnborg","Jeff Garzik","Linus Torvalds","Johannes Schindelin","Kay Sievers","Sven Verdoolaege","Jon Loeliger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"8480","messageId":"7vek7s1xsh.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":null,"subject":"dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T21:07:58Z","receivedAt":"2005-09-13T21:07:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"I've looked at ~80 git repositories publicly available at\nkernel.org and noticed only 23 are prepared to handle dumb\ntransports. That probably means either most of these trees are\nnot pulled by people without kernel.org accounts, or public are\nusing rsync or cogito to pull from these trees. I somehow find\nthis number very discouraging ...\n"},{"id":"8481","messageId":"20050913211444.GA27029@mars.ravnborg.org","threadId":"1792","inReplyTo":"7vek7s1xsh.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Sam Ravnborg","fromEmail":"sam@ravnborg.org","sentAt":"2005-09-13T21:14:44Z","receivedAt":"2005-09-13T21:14:44Z","isPatch":false,"sender":{"key":"sam@ravnborg.org","avatar":"https://gravatar.com/avatar/168a912606ed0742d840bb365e3cc21db390c36531a58341dc7a069cc1f15f62?d=mp&s=160"},"body":"On Tue, Sep 13, 2005 at 02:07:58PM -0700, Junio C Hamano wrote:\n> I've looked at ~80 git repositories publicly available at\n> kernel.org and noticed only 23 are prepared to handle dumb\n> transports. That probably means either most of these trees are\n> not pulled by people without kernel.org accounts, or public are\n> using rsync or cogito to pull from these trees. I somehow find\n> this number very discouraging ...\n\nWhats wrong using cogito?\nIn other words. Why does you feel like that when we use cogito to do\ncg-update.\n\n\tSam\n"},{"id":"8482","messageId":"7vacig1wrb.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"20050913211444.GA27029@mars.ravnborg.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T21:30:16Z","receivedAt":"2005-09-13T21:30:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sam Ravnborg writes:\n\n> Whats wrong using cogito?\n> In other words. Why does you feel like that when we use cogito to do\n> cg-update.\n\nUsing cogito is not a problem at all. The mechanism to prepare\ntrees to serve wider audience not being used widely is.\n"},{"id":"8485","messageId":"20050913214213.GA2549@mars.ravnborg.org","threadId":"1792","inReplyTo":"7vacig1wrb.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Sam Ravnborg","fromEmail":"sam@ravnborg.org","sentAt":"2005-09-13T21:42:13Z","receivedAt":"2005-09-13T21:42:13Z","isPatch":false,"sender":{"key":"sam@ravnborg.org","avatar":"https://gravatar.com/avatar/168a912606ed0742d840bb365e3cc21db390c36531a58341dc7a069cc1f15f62?d=mp&s=160"},"body":"On Tue, Sep 13, 2005 at 02:30:16PM -0700, Junio C Hamano wrote:\n> Sam Ravnborg writes:\n> \n> > Whats wrong using cogito?\n> > In other words. Why does you feel like that when we use cogito to do\n> > cg-update.\n> \n> Using cogito is not a problem at all. The mechanism to prepare\n> trees to serve wider audience not being used widely is.\n\nWhat is the right method to use at kernel.org then?\n\nI did:\n\nrm -rf kbuild.git\ncp -al ../linus/linus-2.6.git kbuild.get\necho ... > ---/alternates\nGIT_DIR=kbuild.dir git-prune-cache\n\nWorked like a charm although a bit hacky...\nAnd I had a tree to start out from.\n\n\tSam\n"},{"id":"8492","messageId":"7vpsrcwrc1.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"7vacig1wrb.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T22:11:42Z","receivedAt":"2005-09-13T22:11:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Sam Ravnborg writes:\n>\n>> Whats wrong using cogito?\n>> In other words. Why does you feel like that when we use cogito to do\n>> cg-update.\n>\n> Using cogito is not a problem at all. The mechanism to prepare\n> trees to serve wider audience not being used widely is.\n\nI need to clarify what I meant by 'not welcoming dumb transport'\na bit better. Namely, those (~80 - 23) = ~57 repositories lack\nsupport for 'git ls-remote' over http, which means you cannot\ndiscover what refs the repository has.\n\nSome people argued that it can be done via recursive wget on\nrefs/ hierarchy. Here is what you would get if you do that\nagainst kernel.org:\n\n $ wget -r -np -nH --cut-dirs=4 http://kernel.org/pub/scm/git/git.git/refs/.\n $ ls -R refs\n refs:\n ./ index.html index.html?C=N;O=A index.html?C=S;O=D\n ../ index.html?C=M;O=A index.html?C=N;O=D tags/\n heads/ index.html?C=M;O=D index.html?C=S;O=A\n\n refs/heads:\n ./ index.html?C=M;O=A index.html?C=N;O=D master todo\n ../ index.html?C=M;O=D index.html?C=S;O=A pu\n index.html index.html?C=N;O=A index.html?C=S;O=D rc\n\n refs/tags:\n ./ index.html?C=M;O=D index.html?C=S;O=D v0.99.2 v0.99.6\n ../ index.html?C=N;O=A junio-gpg-pub v0.99.3\n index.html index.html?C=N;O=D v0.99 v0.99.4\n index.html?C=M;O=A index.html?C=S;O=A v0.99.1 v0.99.5\n\nOf course, I do not have a branch called index.html there, and\nthis also means I will not be able to have a branch with that\nname even if I wanted to.\n\nAlso some webservers are configured not to even allow directory\nindex, and they may use different formatting for directory index\neven when they do support it, so excluding anything that matches\nindex.html* would work well but that is only heuristics.\n\nThe file $GIT_DIR/info/refs was introduced to solve this by\nlisting the available refs for discovery, and hooks/post-update,\nwhen enabled, runs update-server-info to update the file (among\nother things) whenever you push into the repository. info/refs\nis not strictly necessary for repositories at kernel.org because\npeople tend to know what refs are available for pulling and you\ncan always visit there via gitweb to find it out.\n\nI just felt that it is a good habit to get into to prepare your\nrepositories in a shape usable even when served by an HTTP\nserver that is less forgiving than what kernel.org runs -- that\nwas what I felt \"discouraging\" about.\n\nAnother thing is that the missing info/refs file means the\nrepository is not prepared with update-server-info, so it is\nlikely that it lacks objects/info/packs to describe what packs\nare in the object database. I believe cogito uses git-http-pull\nafter you tell which ref to pull, and this step would break if\nthe repository is packed, objects/info/packs is not available,\nand if the downloader does not have an object that is already\nprune-packed in the repository. This means either people are\nnot packing their repository (hence nobody complained), or\npublic are pulling over rsync transport (which slurps everything\nin sight). Both are good reasons to feel discouraged about.\n"},{"id":"8493","messageId":"432750E1.3020508@pobox.com","threadId":"1792","inReplyTo":"7vpsrcwrc1.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-09-13T22:21:21Z","receivedAt":"2005-09-13T22:21:21Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n> The file $GIT_DIR/info/refs was introduced to solve this by\n> listing the available refs for discovery, and hooks/post-update,\n> when enabled, runs update-server-info to update the file (among\n> other things) whenever you push into the repository.\n\nThis is helpful. I'll run git-update-server-info before each push, now.\n\n\tJeff\n"},{"id":"8495","messageId":"7v7jdkwqj0.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"432750E1.3020508@pobox.com","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T22:29:07Z","receivedAt":"2005-09-13T22:29:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Garzik writes:\n\n> Junio C Hamano wrote:\n>> The file $GIT_DIR/info/refs was introduced to solve this by\n>> listing the available refs for discovery, and hooks/post-update,\n>> when enabled, runs update-server-info to update the file (among\n>> other things) whenever you push into the repository.\n>\n> This is helpful. I'll run git-update-server-info before each push, now.\n\nJust to make sure I am not misunderstanding you, is your \"publish\"\nworkflow like this?\n\n - do your work on your private machine\n - git-update-server-info in your private repository\n - rsync that out to master.kernel.org\n\nIf so, then \"before each push\" makes sense. \n\nI know you understand the following but this is for other people\non the list.\n\nThe hooks/post-update I was talking about assumes a different\n\"publish\" workflow:\n\n - do your work on your private machine\n - git push master.kernel.org:/pub/scm/...\n\nand you have hooks/post-update enabled in the repository on\nmaster.kernel.org; the counterpart program for 'git push' which\nruns on master.kernel.org updates the info/refs and objects/info/packs\nfile over there once your push is done.\n"},{"id":"8494","messageId":"Pine.LNX.4.58.0509131525250.26803@g5.osdl.org","threadId":"1792","inReplyTo":"7vpsrcwrc1.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-13T22:29:35Z","receivedAt":"2005-09-13T22:29:35Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 13 Sep 2005, Junio C Hamano wrote:\n> \n> I need to clarify what I meant by 'not welcoming dumb transport'\n> a bit better. Namely, those (~80 - 23) = ~57 repositories lack\n> support for 'git ls-remote' over http, which means you cannot\n> discover what refs the repository has.\n\nYou do realize that up until a week ago (six days, to be exact),\nkernel.org was running git-0.99.4, which I don't think actually\nimplemented any of the info stuff?\n\nSo out of the 57 repositories, how many haven't been updated in a week?\n\nI suspect that explains a large portion of it.\n\nAlso, I really do think that the dumb transports are oversold, and \ngit-daemon is undersold. I know all about firewalls, but I also think that \nif people used the smart protocols more, that's a problem that would \nlargely solve itself. \n\nDumb protocols can never do really well. That's just very fundamental. \n\n\t\tLinus\n"},{"id":"8497","messageId":"7vwtlkvbk0.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131525250.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T22:37:51Z","receivedAt":"2005-09-13T22:37:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> You do realize that up until a week ago (six days, to be exact),\n> kernel.org was running git-0.99.4, which I don't think actually\n> implemented any of the info stuff?\n\nAh, no I didn't. For future reference, how can I find that kind\nof thing myself (not \"what was not in 0.99.4\", but \"what did\nkernel.org run a week ago\")?\n\n> Dumb protocols can never do really well. That's just very fundamental. \n\nI agree. I am waiting for git-deamon to happen on kernel.org,\n\nI am hoping there won't be much problems but am somewhat worried\nthat customized packing for each client might turn out to be too\nmuch load.\n"},{"id":"8498","messageId":"Pine.LNX.4.58.0509131554060.26803@g5.osdl.org","threadId":"1792","inReplyTo":"7vwtlkvbk0.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-13T22:55:54Z","receivedAt":"2005-09-13T22:55:54Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 13 Sep 2005, Junio C Hamano wrote:\n> \n> Ah, no I didn't. For future reference, how can I find that kind\n> of thing myself (not \"what was not in 0.99.4\", but \"what did\n> kernel.org run a week ago\")?\n\nI don't know. Hpa did an announcement when kernel.org switched to 0.99.4, \nbut I never saw any announcement of upgrades (I only noticed that the date \non /usr/bin/git is now Sep 7 a coupld of days ago - no proof of upgrade, \nbut _something_ happened six days ago ;)\n\nI personally run my own private set of binaries anyway, and I upgrade \npretty randomly, so what the official kernel.org installation is doesnt' \naffect me personally ;)\n\n\t\tLinus\n"},{"id":"8499","messageId":"7vpsrcvafm.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131554060.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-13T23:02:05Z","receivedAt":"2005-09-13T23:02:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Tue, 13 Sep 2005, Junio C Hamano wrote:\n>> \n>> Ah, no I didn't. For future reference, how can I find that kind\n>> of thing myself (not \"what was not in 0.99.4\", but \"what did\n>> kernel.org run a week ago\")?\n>\n> I don't know. Hpa did an announcement when kernel.org switched to 0.99.4, \n> but I never saw any announcement of upgrades (I only noticed that the date \n> on /usr/bin/git is now Sep 7 a coupld of days ago - no proof of upgrade, \n> but _something_ happened six days ago ;)\n\nThanks, that much I could have figured out myself.\n\n> I personally run my own private set of binaries anyway, and I upgrade \n> pretty randomly, so what the official kernel.org installation is doesnt' \n> affect me personally ;)\n\nSame here.\n"},{"id":"8500","messageId":"Pine.LNX.4.63.0509140152160.24606@wgmdd8.biozentrum.uni-wuerzburg.de","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131525250.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-09-14T00:01:04Z","receivedAt":"2005-09-14T00:01:04Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 13 Sep 2005, Linus Torvalds wrote:\n\n> \n> \n> Also, I really do think that the dumb transports are oversold, and \n> git-daemon is undersold.\n\nMy tests confirm that a single git-pull via git-daemon brings a small \nmachine to its knees. Which means that multiple git-pull's bring a nice \nbig machine like kernel.org to its knees.\n\nIMHO the culprit is git-rev-list, which takes ages and ages for big \nrepositories (beware: this could be my Darwin client which might be \nincapable to stop the rev enumeration in time; but if that can be done \nunintentionally, this can be intentionally, too!).\n\nDid anybody think about using the information which helps the dumb \ntransports for intelligent transports, too? (A sort of cache for \ngit-rev-list would do wonders...) This could at least help the CPU load on \nthe server.\n\n(In retrospect it might have been a mistake to make the call to \ngit-update-server-info optional: maybe an environment variable should be \nset to _inhibit_ the behaviour for those which absolutely cannot live with \nthe performance hit.)\n\nCiao,\nDscho\n"},{"id":"8501","messageId":"Pine.LNX.4.58.0509131742240.26803@g5.osdl.org","threadId":"1792","inReplyTo":"Pine.LNX.4.63.0509140152160.24606@wgmdd8.biozentrum.uni-wuerzburg.de","subject":"Re: dumb transports not being welcomed..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-14T00:57:22Z","receivedAt":"2005-09-14T00:57:22Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 14 Sep 2005, Johannes Schindelin wrote:\n> \n> IMHO the culprit is git-rev-list, which takes ages and ages for big \n> repositories (beware: this could be my Darwin client which might be \n> incapable to stop the rev enumeration in time; but if that can be done \n> unintentionally, this can be intentionally, too!).\n\nPacked too?\n\ngit-rev-list will take a long time if the tree is unpacked and not in the \ncache. It's all disk seeks. That's _especially_ true of a full clone \n(which will walk the whole way down).\n\nBut I have tons of memory in my machines, and I haven't looked at how \nbadly it does if you don't have that. I know that master.kernel.org is \ncertainly not having any trouble at all with me pulling from lots of \ntrees.. Maybe git-rev-list uses up lots of your memory.\n\nI'm seeing 14 seconds of CPU-time for a _full_ kernel history, with\n\"--objects\". Yes, it's not exactly cheap, and maybe I should optimize it\n(it's all in the \"--objects\" handling and probably a large portion of it\nis because trees actually pack very well indeed, so it's actually\nunpacking a lot of trees), but considering that that is preparing the\nmetadata for pulling down a hundred megs of stuff..\n\nThat said, I do think that --objects handling is _very_ CPU-hungry. The \noffender is this old commit of mine:\n\n\t4311d328fee11fbd80862e3c5de06a26a0e80046\n\tAuthor: Linus Torvalds \n\tDate: Sat Jul 23 10:01:49 2005 -0700\n\n\t Be more aggressive about marking trees uninteresting\n\t...\n\nwhich is much better about avoiding objects in old trees, but it does so \nat the expense of being _horribly_ CPU-inefficient. It will walk through \nevery tree of every commit that we decided was uninteresting.\n\nYou can try to just undo that one commit - it will make pack-files have a \nfew extraneous objects, but I think it will make a huge difference in the \nCPU cost of \"small pulls\" (it won't matter at all for the \"git clone\" \ncase: for that case we just always have to walk the whole object tree).\n\n\t\tLinus\n"},{"id":"8502","messageId":"Pine.LNX.4.58.0509131819310.26803@g5.osdl.org","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131742240.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-14T01:42:05Z","receivedAt":"2005-09-14T01:42:05Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 13 Sep 2005, Linus Torvalds wrote:\n> \n> That said, I do think that --objects handling is _very_ CPU-hungry. The \n> offender is this old commit of mine:\n\nNo, never mind. Even without that, we end up walking a _lot_ of really\nuninterestng \"internal\" trees (ie trees where all parents were\nuninteresting, and they were parsed just because we had to parse a lot of \ncommits to determine what they reached). \n\nTo explain it a bit better, let's see a common case:\n\n\nHEAD:\t\ta\n\t / \\\n\t b \\\n\t / \\ \\\n\t c d \\\n\t / / \\ \\\n\t e f g x\n\t \\ / / /\n\t h i /\n\t \\ / /\n\t j /\n\t \\ /\nOld history: k\n\n\nNow, imagine that we do \n\n\tgit-rev-list b..a\n\nwhich results in just two commits: 'x' and 'a' (everything else is\nreachable from 'b'). This is actually not that uncommon. However, in order \nto realize that, we had to walk through _all_ of a..k and x before we saw \nthat 'b'..'k' were all uninteresting, and there was nothing else reachable \nthat migt be interesting.\n\nNow, that's pretty cheap per se. git-rev-list is optimized for this case, \nand hey, it's usually just a few hundred objects. Not a big deal - \ngenerating the commit list takes a small fraction of a second.\n\nHowever, now the true cost of \"--objects\" is clear: we will walk the two\n\"positive\" trees ('a' and 'x') and look up all their objects (about 35,000\nof them) interesting. So far so good. Just another fraction of a second. \n\nHOWEVER, then we walk _every_single_uninteresting_commit_ and walk _their_\nobjects to say \"we've got this already\". And the uninteresting commits are\noften many more than the interesting ones - we might have had to go\nseveral weeks back to list them all. The above example is not at all\nextreme: we might have something like 20 interesting commits, and several\nhundreds of the uninteresting ones.\n\nNow, the way to optimize things is to realize that there are two \"classes\" \nof uninteresting commits. There are the uninteresting commits that are \nadjacent to an interesting one (in the above example, they are \"b\" and \n\"k\"), and there are the uninteresting commits that are only reachable from \n-other- uninteresting commits ('c'..'j'). Let's call the latter class \n\"doubly uninteresting commits\", and the former class \"uninteresting edge \ncommits\".\n\nAnd we really don't need to walk the \"doubly uninteresting\" trees. But we\ndo. Because we don't have another phase to discover the edge (we can't do\nthat during the initial discovery phase, because we don't know if a commit\nis going to end up interesting in the end - we migth have another commit\nthat we haven't seen yet that might be the parent of a commit that _looks_\ninteresting right now, but ends up being uninteresting because that\neventually seen parent ended up being uninteresting).\n\nIn other words: I bet I could make \"git-rev-list --objects\" go from ten\nseconds to a single second if I did that edge discovery for most small\nincremental updates. Instead, I'm lazy, and I'm describing the problem on \nthe list as an \"educational experience\", and am callously hoping that \nsomebody will see it as an interesting challenge ;)\n\nBtw, the above is definitely not made up. If I did my statistics right,\ndoing \"git-rev-list v2.6.14-rc1..\" with the current tree results in 178\n\"interesting\" commits, and 6251 \"uninteresting\" ones. And I bet 99% of\nthose uninteresting ones are \"doubly uninteresting\" - and we're just\nwasting CPU time looking at what objects are reachable from them..\n\n\t\tLinus\n"},{"id":"8503","messageId":"20050914022516.GA3379@vrfy.org","threadId":"1792","inReplyTo":"7vpsrcvafm.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Kay Sievers","fromEmail":"kay.sievers@vrfy.org","sentAt":"2005-09-14T02:25:16Z","receivedAt":"2005-09-14T02:25:16Z","isPatch":false,"sender":{"key":"kay.sievers@vrfy.org","avatar":null},"body":"On Tue, Sep 13, 2005 at 04:02:05PM -0700, Junio C Hamano wrote:\n> Linus Torvalds writes:\n> \n> > On Tue, 13 Sep 2005, Junio C Hamano wrote:\n> >> \n> >> Ah, no I didn't. For future reference, how can I find that kind\n> >> of thing myself (not \"what was not in 0.99.4\", but \"what did\n> >> kernel.org run a week ago\")?\n> >\n> > I don't know. Hpa did an announcement when kernel.org switched to 0.99.4, \n> > but I never saw any announcement of upgrades (I only noticed that the date \n> > on /usr/bin/git is now Sep 7 a coupld of days ago - no proof of upgrade, \n> > but _something_ happened six days ago ;)\n> \n> Thanks, that much I could have figured out myself.\n\nThat's easier:\n\n $ rpm -q git-core\n git-core-0.99.6-1\n\n $ rpm -q cogito\n cogito-0.14-1\n\nKay\n"},{"id":"8504","messageId":"7vbr2wuxx3.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"20050914022516.GA3379@vrfy.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-14T03:32:24Z","receivedAt":"2005-09-14T03:32:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Kay Sievers writes:\n\n> That's easier:\n>\n> $ rpm -q git-core\n> git-core-0.99.6-1\n>\n> $ rpm -q cogito\n> cogito-0.14-1\n\nSorry, I fail to see how it answers my question: \"what did\nkernel.org run a week ago?\"\n"},{"id":"8507","messageId":"Pine.LNX.4.63.0509141014580.30708@wgmdd8.biozentrum.uni-wuerzburg.de","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131742240.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2005-09-14T08:38:58Z","receivedAt":"2005-09-14T08:38:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 13 Sep 2005, Linus Torvalds wrote:\n> \n> On Wed, 14 Sep 2005, Johannes Schindelin wrote:\n> > \n> > IMHO the culprit is git-rev-list, which takes ages and ages for big \n> > repositories (beware: this could be my Darwin client which might be \n> > incapable to stop the rev enumeration in time; but if that can be done \n> > unintentionally, this can be intentionally, too!).\n> \n> Packed too?\n\nYes. Almost all of it.\n\n> git-rev-list will take a long time if the tree is unpacked and not in the \n> cache. It's all disk seeks. That's _especially_ true of a full clone \n> (which will walk the whole way down).\n\nThat could be the case, but my test case is a CVS project I track on one \nside, and I fetch on the other side. Therefore, your diagram from your \nother mail does not really apply. My history looks more or less like this:\n\na b c d origin\n |\n |\n |\n |\n\\ \\ \\ \\ |\n ---------|\n\nSo, the origin is a linear CVS project. With many, many, many commits. At \none stage I broke off new git branches. I kept tracking the CVS project, \nthough.\n\nWhat I see when fetching all heads (thanks to Junio, this is one call to \ngit-fetch now), where all but origin are up to date, is that it takes a \nvery long time. Swapping kicks in, and top tells me that 26.6% of the \nmemory is occupied by git-rev-list (The server has 128M, with 1G swap, and \nI am unfortunately not the only user of this machine).\n\nI fail to see why it should need those amounts of memory. (I tested this \nover the ssh protocol, which should essentially do the same as git-daemon, \nright?) After all, the merge point between the branches should be marked \nuninteresting after one single step from each of my private branches.\n\n> But I have tons of memory in my machines, and I haven't looked at how \n> badly it does if you don't have that. I know that master.kernel.org is \n> certainly not having any trouble at all with me pulling from lots of \n> trees.. Maybe git-rev-list uses up lots of your memory.\n\nThat certainly is the case.\n\nAs for master.kernel.org: Unfortunately, you will not be the only puller. \nAnd if your process needs just 5% of the RAM, then 21 pullers will be too \nmany.\n\n> That said, I do think that --objects handling is _very_ CPU-hungry.\n\nIn my experience, before the swapping started, the process did not get \nmore than 20% CPU.\n\nNevertheless, I still think that it would be a good idea to reuse the \nfiles created for the dumb transport for the intelligent transport. \nEspecially for a project which is more often fetched than uploaded.\n\nI also see other strange things like packing 0 objects, and packing >0 \nobjects after just having fetched from that repository. Hopefully I will \nhave time to look into that (and understand the code to begin with).\n\nCiao,\nDscho\n"},{"id":"8511","messageId":"20050914104539.GP15165MdfPADPa@greensroom.kotnet.org","threadId":"1792","inReplyTo":"7vpsrcwrc1.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Sven Verdoolaege","fromEmail":"skimo@kotnet.org","sentAt":"2005-09-14T10:45:39Z","receivedAt":"2005-09-14T10:45:39Z","isPatch":false,"sender":{"key":"skimo@kotnet.org","avatar":null},"body":"On Tue, Sep 13, 2005 at 03:11:42PM -0700, Junio C Hamano wrote:\n> The file $GIT_DIR/info/refs was introduced to solve this by\n> listing the available refs for discovery, and hooks/post-update,\n> when enabled, runs update-server-info to update the file (among\n> other things) whenever you push into the repository. \n\nIt doesn't help that update-server-info crashes if you run\nit for the first time on an old repo.\nMaybe it should create the appropriate directory structure on the fly,\nbut the patch below at least checks whether new rev-cache could\nbe created.\n\nskimo\n--\n\nwrite_rev_cache: check whether new cache could be created.\n\n---\ncommit d30b87459c690ff68e65dfe8ecdc585dab64323a\ntree 51127c1af00f8fd63e7b996384e86d7d31ad5562\nparent 2ba6c47be1762726ad0c1d5779064c489150d789\nauthor Sven Verdoolaege Wed, 14 Sep 2005 12:40:28 +0200\ncommitter Sven Verdoolaege Wed, 14 Sep 2005 12:40:28 +0200\n\n rev-cache.c | 8 +++++++-\n rev-cache.h | 2 +-\n server-info.c | 7 ++++---\n 3 files changed, 12 insertions(+), 5 deletions(-)\n\ndiff --git a/rev-cache.c b/rev-cache.c\n--- a/rev-cache.c\n+++ b/rev-cache.c\n@@ -103,7 +103,7 @@ static void write_one_rev_cache(FILE *re\n \t\twrite_one_rev_cache(rev_cache_file, rle->ri);\n }\n \n-void write_rev_cache(const char *newpath, const char *oldpath)\n+int write_rev_cache(const char *newpath, const char *oldpath)\n {\n \t/* write the following commit ancestry information in\n \t * $GIT_DIR/info/rev-cache.\n@@ -131,6 +131,11 @@ void write_rev_cache(const char *newpath\n \t\tsize_t sz;\n \t\tFILE *oldfp = fopen(oldpath, \"r\");\n \t\trev_cache_file = fopen(newpath, \"w\");\n+\t\tif (!rev_cache_file) {\n+\t\t\tif (oldfp)\n+\t\t\t\tfclose(oldfp);\n+\t\t\treturn error(\"cannot open %s\", newpath);\n+\t\t}\n \t\tif (oldfp) {\n \t\t\twhile (1) {\n \t\t\t\tsz = fread(buf, 1, sizeof(buf), oldfp);\n@@ -161,6 +166,7 @@ void write_rev_cache(const char *newpath\n \t\twrite_one_rev_cache(rev_cache_file, ri);\n \t}\n \tfclose(rev_cache_file);\n+\treturn 0;\n }\n \n static void add_parent(struct rev_cache *child,\ndiff --git a/rev-cache.h b/rev-cache.h\n--- a/rev-cache.h\n+++ b/rev-cache.h\n@@ -24,6 +24,6 @@ struct rev_list_elem {\n extern int find_rev_cache(const unsigned char *);\n extern int read_rev_cache(const char *, FILE *, int);\n extern int record_rev_cache(const unsigned char *, FILE *);\n-extern void write_rev_cache(const char *new, const char *old);\n+extern int write_rev_cache(const char *new, const char *old);\n \n #endif\ndiff --git a/server-info.c b/server-info.c\n--- a/server-info.c\n+++ b/server-info.c\n@@ -536,6 +536,7 @@ static int update_info_revs(int force)\n \tchar *path0 = strdup(git_path(\"info/rev-cache\"));\n \tint len = strlen(path0);\n \tchar *path1 = xmalloc(len + 2);\n+\tint errs = 0;\n \n \tstrcpy(path1, path0);\n \tstrcpy(path1 + len, \"+\");\n@@ -548,11 +549,11 @@ static int update_info_revs(int force)\n \tfor_each_ref(record_rev_cache_ref);\n \n \t/* update the rev-cache database */\n-\twrite_rev_cache(path1, force ? \"/dev/null\" : path0);\n-\trename(path1, path0);\n+\terrs = errs || write_rev_cache(path1, force ? \"/dev/null\" : path0);\n+\terrs = errs || rename(path1, path0);\n \tfree(path1);\n \tfree(path0);\n-\treturn 0;\n+\treturn errs;\n }\n \n /* public */\n"},{"id":"8519","messageId":"432823BC.30305@pobox.com","threadId":"1792","inReplyTo":"7v7jdkwqj0.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-09-14T13:21:00Z","receivedAt":"2005-09-14T13:21:00Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n> Jeff Garzik writes:\n> \n> \n>>Junio C Hamano wrote:\n>>\n>>>The file $GIT_DIR/info/refs was introduced to solve this by\n>>>listing the available refs for discovery, and hooks/post-update,\n>>>when enabled, runs update-server-info to update the file (among\n>>>other things) whenever you push into the repository.\n>>\n>>This is helpful. I'll run git-update-server-info before each push, now.\n> \n> \n> Just to make sure I am not misunderstanding you, is your \"publish\"\n> workflow like this?\n> \n> - do your work on your private machine\n> - git-update-server-info in your private repository\n> - rsync that out to master.kernel.org\n> \n> If so, then \"before each push\" makes sense. \n\nCorrect.\n\n\tJeff\n"},{"id":"8520","messageId":"1126707016.14036.14.camel@cashmere.sps.mot.com","threadId":"1792","inReplyTo":"7vpsrcwrc1.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Jon Loeliger","fromEmail":"jdl@freescale.com","sentAt":"2005-09-14T14:10:16Z","receivedAt":"2005-09-14T14:10:16Z","isPatch":false,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"On Tue, 2005-09-13 at 17:11, Junio C Hamano wrote:\n\n> I just felt that it is a good habit to get into to prepare your\n> repositories in a shape usable even when served by an HTTP\n> server that is less forgiving than what kernel.org runs -- that\n> was what I felt \"discouraging\" about.\n\nWell, that is sort of just it, too. Why not make the\ndefault, obvious, common repo prep mechanism do all\nthe necessary steps for proper presentation? Having\nto remember to do 6 steps just begs for an additional\nlayer of scripting.\n\n> This means either people are\n> not packing their repository (hence nobody complained), or\n> public are pulling over rsync transport (which slurps everything\n> in sight). Both are good reasons to feel discouraged about.\n\nI confess, I've been using rsync as it is what appears\nto be able to reliably get a repository that works.\n\njdl\n"},{"id":"8524","messageId":"Pine.LNX.4.58.0509140759030.26803@g5.osdl.org","threadId":"1792","inReplyTo":"Pine.LNX.4.63.0509141014580.30708@wgmdd8.biozentrum.uni-wuerzburg.de","subject":"Re: dumb transports not being welcomed..","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-09-14T15:07:24Z","receivedAt":"2005-09-14T15:07:24Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 14 Sep 2005, Johannes Schindelin wrote:\n> \n> What I see when fetching all heads (thanks to Junio, this is one call to \n> git-fetch now), where all but origin are up to date, is that it takes a \n> very long time. Swapping kicks in, and top tells me that 26.6% of the \n> memory is occupied by git-rev-list (The server has 128M, with 1G swap, and \n> I am unfortunately not the only user of this machine).\n\nOk. As mentioned, I've not looked at memory usage. The machines I play \nwith tend to have 2GB or more, simply because bk needed at least 1GB to be \nnice and cached on the kernel ;)\n\nGit has needed less than bk, so I've not cared ;)\n\n> I fail to see why it should need those amounts of memory. (I tested this \n> over the ssh protocol, which should essentially do the same as git-daemon, \n> right?) After all, the merge point between the branches should be marked \n> uninteresting after one single step from each of my private branches.\n\nOne of the issues is that git-rev-list will (for example) keep track of \nthe commit messages too for every commit. That in itself can be a lot of \nstuff, depending on how active the tree is and how large the messages are.\n\nNow, that should be easy enough to fix (parse_commit() normally saves the \nbuffer it parses into \"commit->buffer\", so we'd just need to do something \nlike\n\n\tif (!verbose_header && commit->buffer) {\n\t\tfree(commit->buffer);\n\t\tcommit->buffer = NULL;\n\t}\n\nfor each commit.\n\nBut for --objects, the bigger memory pressure is that it needs to track \nthe \"struct object\" for every single object when it generates the \nreference tracking. And THAT tends to be expensive. The object lists are \nalso not very space-efficient (ie one small allocation for each list \nentry).\n\nWe could probably make objects/lists more space-efficient.\n\n> I also see other strange things like packing 0 objects, and packing >0 \n> objects after just having fetched from that repository. Hopefully I will \n> have time to look into that (and understand the code to begin with).\n\nWell, the \"packing 0 objects\" should be normal. I'm surprised at the \">0\" \ncase after a fetch: the packign is _not_ guaranteed to be exact, but if \nyou have the exact same state as (or a superset of) the other end, you \nshould always see a zero.\n\n\t\tLinus\n"},{"id":"8531","messageId":"7vk6hjpqxu.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"20050914104539.GP15165MdfPADPa@greensroom.kotnet.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-14T16:14:21Z","receivedAt":"2005-09-14T16:14:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sven Verdoolaege writes:\n\n> It doesn't help that update-server-info crashes if you run\n> it for the first time on an old repo.\n> Maybe it should create the appropriate directory structure on the fly,\n> but the patch below at least checks whether new rev-cache could\n> be created.\n\nTrue; thanks for the patch. However, since nobody seems to use\nrev-cache, it _might_ make sense to just yank it out. If it\nturns out to be useful later we could always resurrect it.\n"},{"id":"8549","messageId":"7vk6hjiiew.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"1126707016.14036.14.camel@cashmere.sps.mot.com","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-14T19:00:23Z","receivedAt":"2005-09-14T19:00:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jon Loeliger writes:\n\n> Well, that is sort of just it, too. Why not make the\n> default, obvious, common repo prep mechanism do all\n> the necessary steps for proper presentation? Having\n> to remember to do 6 steps just begs for an additional\n> layer of scripting.\n\nFair enough. My excuse is that Linus did not want the\nupdate-server-info hook enabled by default. He does not believe\nin dumb transports anyway, but aside from that, it still is a\nvalid attitude because it is not necessary when you do not\nintend to publish your repository over dumb transport at all but\nstill want to push into it. And another excuse is I do not\nin general think enabling hooks by default is a good idea.\n\nEven if you built your repository with older git tools, you\nshould be always able to say 'GIT_DIR=that-repository\ngit-init-db' without damaging its existing contents to install\nthe disabled hooks in its hooks/ directory.\n\n> I confess, I've been using rsync as it is what appears\n> to be able to reliably get a repository that works.\n\nAnd I thought rsync was a reliable way too, until I saw a\nmessage from Tony Luck this morning X-<.\n"},{"id":"8550","messageId":"1126725206.14036.22.camel@cashmere.sps.mot.com","threadId":"1792","inReplyTo":"7vk6hjiiew.fsf@assigned-by-dhcp.cox.net","subject":"Re: dumb transports not being welcomed..","fromName":"Jon Loeliger","fromEmail":"jdl@freescale.com","sentAt":"2005-09-14T19:13:26Z","receivedAt":"2005-09-14T19:13:26Z","isPatch":false,"sender":{"key":"jdl@jdl.com","avatar":"https://gravatar.com/avatar/75ce9a10b151acd2c28ec4ab2136dba7b2ff1634530bd04b155981a749d08a64?d=mp&s=160"},"body":"On Wed, 2005-09-14 at 14:00, Junio C Hamano wrote:\n\n> Fair enough. My excuse is that Linus did not want the\n> update-server-info hook enabled by default. He does not believe\n> in dumb transports anyway, but aside from that, it still is a\n> valid attitude because it is not necessary when you do not\n> intend to publish your repository over dumb transport at all but\n> still want to push into it. And another excuse is I do not\n> in general think enabling hooks by default is a good idea.\n> \n> Even if you built your repository with older git tools, you\n> should be always able to say 'GIT_DIR=that-repository\n> git-init-db' without damaging its existing contents to install\n> the disabled hooks in its hooks/ directory.\n\nHmmm. Maybe this is all begging a form of documentation\ndown the \"Best Practices\" line, or \"Tips I Learned\nBy Reading Junio and Linus Postings on Git\". :-)\n\n> And I thought rsync was a reliable way too, until I saw a\n> message from Tony Luck this morning X-<.\n\nHeh. I read his problem description too, and it sounded\nremarkably close to the HTTP pull problems that I had\nbeen victimized by, thus converting me to rsync (for now).\n\nI have recloned entire repos due to the aborted pulls\nbeing left in an inconsistent state. In fact, that is\nwhat lead me to find git-fsck-cache, which I was expecting\nto sort out the missing pieces and detect an incomplete\nclone. Perhaps marking it as \"needing completion\" via\nanother clone/pull effort. Dunno.\n\njdl\n"},{"id":"8594","messageId":"7v1x3q4rmi.fsf@assigned-by-dhcp.cox.net","threadId":"1792","inReplyTo":"Pine.LNX.4.58.0509131525250.26803@g5.osdl.org","subject":"Re: dumb transports not being welcomed..","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-15T09:17:25Z","receivedAt":"2005-09-15T09:17:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> Also, I really do think that the dumb transports are oversold, and \n> git-daemon is undersold. I know all about firewalls, but I also think that \n> if people used the smart protocols more, that's a problem that would \n> largely solve itself. \n\nThis reminds me of one thing I've been wanting to have on the\nclient side (git-fetch-pack): HTTP CONNECT passthru ala\n\"tn-gw-nav -H\", which is one of the ways many people ssh out\nover firewalls if I understand correctly. CVS pserver protocol\nseems to do the same using `proxy' connection option.\n\nAnybody interested?\n"}]}