{"thread":{"id":"7528","subject":"kde.git is now online","startedAt":"2007-04-05T17:03:49Z","lastAt":"2007-04-06T12:59:00Z","messageCount":11,"participants":["H. Peter Anvin","Linus Torvalds","Nicolas Pitre","Chris Lee","Junio C Hamano","Geert Bosch"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"38721","messageId":"46152BF5.3050502@zytor.com","threadId":"7528","inReplyTo":null,"subject":"kde.git is now online","fromName":"H. Peter Anvin","fromEmail":"hpa@zytor.com","sentAt":"2007-04-05T17:03:49Z","receivedAt":"2007-04-05T17:03:49Z","isPatch":false,"sender":{"key":"hpa@zytor.com","avatar":null},"body":"I received the DVD from Chris Lee with a test conversion of KDE's \nSubversion repository to git.\n\nI have uploaded it to:\n\nhttp://userweb.kernel.org/~hpa/kdegit/\n\nIt's available both as a tarball and as an expanded tree.\n\n\t-hpa\n\nP.S. I still want Kcharselect to display the Unicode names of the \ncharacters.\n"},{"id":"38723","messageId":"Pine.LNX.4.64.0704051029240.6730@woody.linux-foundation.org","threadId":"7528","inReplyTo":"46152BF5.3050502@zytor.com","subject":"Re: kde.git is now online","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-04-05T17:30:29Z","receivedAt":"2007-04-05T17:30:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 5 Apr 2007, H. Peter Anvin wrote:\n> \n> http://userweb.kernel.org/~hpa/kdegit/\n> \n> It's available both as a tarball and as an expanded tree.\n\nThanks. Am downloading it right now (\"0% 146.99kB/s\" - it will take quite \nsome time ;)\n\n\t\tLinus\n"},{"id":"38724","messageId":"alpine.LFD.0.98.0704051334590.28181@xanadu.home","threadId":"7528","inReplyTo":"Pine.LNX.4.64.0704051029240.6730@woody.linux-foundation.org","subject":"Re: kde.git is now online","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-05T17:38:00Z","receivedAt":"2007-04-05T17:38:00Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 5 Apr 2007, Linus Torvalds wrote:\n\n> \n> \n> On Thu, 5 Apr 2007, H. Peter Anvin wrote:\n> > \n> > http://userweb.kernel.org/~hpa/kdegit/\n> > \n> > It's available both as a tarball and as an expanded tree.\n> \n> Thanks. Am downloading it right now (\"0% 146.99kB/s\" - it will take quite \n> some time ;)\n\nI'm downloading it too (794.3 KB/s).\n\n\nNicolas\n"},{"id":"38726","messageId":"db69205d0704051103u1b3e18f8o37806086a3f01c2b@mail.gmail.com","threadId":"7528","inReplyTo":"Pine.LNX.4.64.0704051029240.6730@woody.linux-foundation.org","subject":"Re: kde.git is now online","fromName":"Chris Lee","fromEmail":"clee@kde.org","sentAt":"2007-04-05T18:03:37Z","receivedAt":"2007-04-05T18:03:37Z","isPatch":false,"sender":{"key":"clee@kde.org","avatar":"https://gravatar.com/avatar/c930bdc8cc6465094a5722188409ecb8955e0da2b188d7340137074b08f857e3?d=mp&s=160"},"body":"On 4/5/07, Linus Torvalds wrote:\n>\n> On Thu, 5 Apr 2007, H. Peter Anvin wrote:\n> >\n> > http://userweb.kernel.org/~hpa/kdegit/\n> >\n> > It's available both as a tarball and as an expanded tree.\n>\n> Thanks. Am downloading it right now (\"0% 146.99kB/s\" - it will take quite\n> some time ;)\n\nImagine how much longer it'd take if it were being served up from my\nhome connection. :)\n\nMany thanks to hpa for putting this up!\n\n-clee\n"},{"id":"38728","messageId":"alpine.LFD.0.98.0704051532240.28181@xanadu.home","threadId":"7528","inReplyTo":"alpine.LFD.0.98.0704051334590.28181@xanadu.home","subject":"Re: kde.git is now online","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-05T19:45:59Z","receivedAt":"2007-04-05T19:45:59Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 5 Apr 2007, Nicolas Pitre wrote:\n\n> On Thu, 5 Apr 2007, Linus Torvalds wrote:\n> \n> > \n> > \n> > On Thu, 5 Apr 2007, H. Peter Anvin wrote:\n> > > \n> > > http://userweb.kernel.org/~hpa/kdegit/\n> > > \n> > > It's available both as a tarball and as an expanded tree.\n> > \n> > Thanks. Am downloading it right now (\"0% 146.99kB/s\" - it will take quite \n> > some time ;)\n> \n> I'm downloading it too (794.3 KB/s).\n\nOK this is a really nice test repo. I have only 1 GB of ram, and \nalthough basic operations appear to work just fine, this data set shows \nits weight in some ways.\n\nFor example I think there might be ways to improve the pack mmap \nwindowing, or git-fsck's IO patterns. For example, git-fsck --full \nspend 96% of the time waiting for IO completion and only 4% actually \nperforming some work according to top. At that rate that makes fsck \n--full rather unusable on this repo. Without --full then fsck completes \nin less than 2 seconds.\n\n\nNicolas\n"},{"id":"38732","messageId":"Pine.LNX.4.64.0704051338290.6730@woody.linux-foundation.org","threadId":"7528","inReplyTo":"alpine.LFD.0.98.0704051532240.28181@xanadu.home","subject":"Re: kde.git is now online","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-04-05T20:51:35Z","receivedAt":"2007-04-05T20:51:35Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 5 Apr 2007, Nicolas Pitre wrote:\n> \n> For example I think there might be ways to improve the pack mmap \n> windowing, or git-fsck's IO patterns. For example, git-fsck --full \n> spend 96% of the time waiting for IO completion and only 4% actually \n> performing some work according to top. At that rate that makes fsck \n> --full rather unusable on this repo. Without --full then fsck completes \n> in less than 2 seconds.\n\nWithout \"--full\", it doesn't actually really do anything much, since it \nwill basically ignore objects that are in the pack.\n\nWith --full, there are certainly things that we could improve upon. We \ncurrently tend to walk things a few times for pack contents: \n - first we do the SHA1 of the full pack\n - then we go back, and unpack and fsck each entry in the pack.\n\nSo if the pack-file is too big to fit in memory, we'll basically always \nread it at least twice (and that's ignoring the fact that delta lookup \nwill obviously seek back and forth, which makes access patterns worse).\n\nOn the other hand, there's a perfectly good reason why we don't actually \nfsck pack-files by default. They're \"stable storage\". You don't normally \nneed to. So I'd not worry too much about fsck performance. I suspect \nyou'll find that with 1GB or RAM you'll have other performance problems \nthat are more pressing (\"git clone\" comes to mind ;)\n\nMe, I'm just 53% done with the download, so I probably won't be looking at \nthis today ;)\n\n\t\t\tLinus\n"},{"id":"38736","messageId":"7vk5wqo6ms.fsf@assigned-by-dhcp.cox.net","threadId":"7528","inReplyTo":"46152BF5.3050502@zytor.com","subject":"Re: kde.git is now online","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-04-05T21:26:03Z","receivedAt":"2007-04-05T21:26:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks. Slurping now.\n"},{"id":"38740","messageId":"alpine.LFD.0.98.0704051703140.28181@xanadu.home","threadId":"7528","inReplyTo":"Pine.LNX.4.64.0704051338290.6730@woody.linux-foundation.org","subject":"Re: kde.git is now online","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-05T22:00:11Z","receivedAt":"2007-04-05T22:00:11Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 5 Apr 2007, Linus Torvalds wrote:\n\n> Without \"--full\", it doesn't actually really do anything much, since it \n> will basically ignore objects that are in the pack.\n> \n> With --full, there are certainly things that we could improve upon. We \n> currently tend to walk things a few times for pack contents: \n> - first we do the SHA1 of the full pack\n> - then we go back, and unpack and fsck each entry in the pack.\n> \n> So if the pack-file is too big to fit in memory, we'll basically always \n> read it at least twice (and that's ignoring the fact that delta lookup \n> will obviously seek back and forth, which makes access patterns worse).\n> \n> On the other hand, there's a perfectly good reason why we don't actually \n> fsck pack-files by default. They're \"stable storage\". You don't normally \n> need to. So I'd not worry too much about fsck performance.\n\nWell.... still it certainly can be helped a bit. I wouldn't mind it \nspending half an hour of CPU if it needs to. But I just interrupted it\nwith ^C with the following result so far:\n\nreal 75m44.374s\nuser 2m5.318s\nsys 0m54.059s\n\n(I should have used /usr/bin/time to see the number of page faults).\n\n> I suspect you'll find that with 1GB or RAM you'll have other \n> performance problems that are more pressing (\"git clone\" comes to mind \n> ;)\n\nWell... same issue actually. git-pack-objects spent about 40 secs \nfirmly at 100% CPU usage counting objects.\n\nThen it got stuck on:\n\n\tremote: Done counting 4111366 objects.\n\nagain spending 3% CPU and the rest waiting for IO with the disk \ndefinitely trashing. It didn't allocate more than 47% of memory during \nthat phase which lasted a few minutes.\n\nThen, the \"Indexing 4111366 objects.\" message appeared and CPU usage \nwent up to 6% CPU with 67% memory for pack-objects and 30% CPU and 7% \nmemory for index-pack while the rest was spent waiting for IO. This \nalso took maybe two minutes.\n\nAnd now it reached the \"Resolving 3305158 deltas.\" phase with only \nindex-pack on the radar with approx 10% CPU and 19% memory, and the rest \nof the time waiting for IO again.\n\nIt has been probably half an our now and the thing is at:\n\n\t 21% (710502/3305158) done\n\nSo it will work and eventually complete. And the good news is that the \nworst part performance wise is on the client side. But it looks like \nwe're definitely trashing the kernel buffer cache.\n\n\nNicolas\n"},{"id":"38771","messageId":"Pine.LNX.4.64.0704051730460.6730@woody.linux-foundation.org","threadId":"7528","inReplyTo":"alpine.LFD.0.98.0704051703140.28181@xanadu.home","subject":"Re: kde.git is now online","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-04-06T01:24:12Z","receivedAt":"2007-04-06T01:24:12Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 5 Apr 2007, Nicolas Pitre wrote:\n> \n> Well.... still it certainly can be helped a bit. I wouldn't mind it \n> spending half an hour of CPU if it needs to. But I just interrupted it\n> with ^C with the following result so far:\n> \n> real 75m44.374s\n> user 2m5.318s\n> sys 0m54.059s\n\nWell, the thing is, this is \"normal\", and doesn't really have a lot to do \nwith git.\n\nIf the actual working set is larger than available memory, ~5% CPU time is \nactually pretty good. \n\nThe only way to improve on it is to try to make the working set smaller. \nSadly, that's often a really difficult thing to do ;(\n\n> > I suspect you'll find that with 1GB or RAM you'll have other \n> > performance problems that are more pressing (\"git clone\" comes to mind \n> > ;)\n> \n> Well... same issue actually. git-pack-objects spent about 40 secs \n> firmly at 100% CPU usage counting objects.\n> \n> Then it got stuck on:\n> \n> \tremote: Done counting 4111366 objects.\n> \n> again spending 3% CPU and the rest waiting for IO with the disk \n> definitely trashing.\n\nWell, I seriously doubt it's the \"same issue\" except in the sense that \nyes, if you work with all objects, you are going to have a big working \nset.\n\nNote that \"working set\" is different from \"memory footprint\". If you have \ngood locality, the working set can be a *lot* smaller than the memory \nfootprint, and that tends to be the best/only way to improve the working \nset: trying to not jump back-and-forth between different things.\n\nOne example of that kind of shrinkage of the working set was Junios commit \n57584d9eddc3482c5db0308203b9df50dc62109c to \"git blame\": by comparing the \n*pointers* rather than what they pointed to, you avoid having to follow \nthe pointer all the way down.\n\nHowever, doing that in general tends to be very difficult. We use hashes \nextensively (not just the obvious SHA1 hashes, but the object lookup \nitself is based on hash tables etc), and while they are nice and fast O(1) \nwhen you have enough memory, they do tend to spread things out so that you \nare using your memory potentially very sparsely, which is the last thing \nyou want to do if you are paging.\n\nSide note: I finally got the thing downloaded, and so I did a\n\n\tgit checkout -f\n\nand the trace is pretty horrid. It looks something like this:\n\n\t...\n\tlstat(\"kdeaccessibility/IconThemes/mono/scalable/apps/kimagemapeditor.svgz\", 0x7fff6f8d29f0) = -1 ENOENT (No such file or directory)\n\tmkdir(\"kdeaccessibility\", 0777) = -1 EEXIST (File exists)\n\tunlink(\"kdeaccessibility\") = -1 EISDIR (Is a directory)\n\tstat(\"kdeaccessibility\", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0\n\tmkdir(\"kdeaccessibility/IconThemes\", 0777) = -1 EEXIST (File exists)\n\tunlink(\"kdeaccessibility/IconThemes\") = -1 EISDIR (Is a directory)\n\tstat(\"kdeaccessibility/IconThemes\", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0\n\tmkdir(\"kdeaccessibility/IconThemes/mono\", 0777) = -1 EEXIST (File exists)\n\tunlink(\"kdeaccessibility/IconThemes/mono\") = -1 EISDIR (Is a directory)\n\tstat(\"kdeaccessibility/IconThemes/mono\", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0\n\tmkdir(\"kdeaccessibility/IconThemes/mono/scalable\", 0777) = -1 EEXIST (File exists)\n\tunlink(\"kdeaccessibility/IconThemes/mono/scalable\") = -1 EISDIR (Is a directory)\n\tstat(\"kdeaccessibility/IconThemes/mono/scalable\", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0\n\tmkdir(\"kdeaccessibility/IconThemes/mono/scalable/apps\", 0777) = -1 EEXIST (File exists)\n\tunlink(\"kdeaccessibility/IconThemes/mono/scalable/apps\") = -1 EISDIR (Is a directory)\n\tstat(\"kdeaccessibility/IconThemes/mono/scalable/apps\", {st_mode=S_IFDIR|0775, st_size=12288, ...}) = 0\n\topen(\"kdeaccessibility/IconThemes/mono/scalable/apps/kimagemapeditor.svgz\", O_WRONLY|O_CREAT|O_EXCL, 0666) = 5\n\twrite(5, \"\\37\\213\\10\\10\\205\\3\\263A\\0\\3kimagemapeditor.svg\\0\\344Z\"..., 10112) = 10112\n\tclose(5) = 0\n\tlstat(\"kdeaccessibility/IconThemes/mono/scalable/apps/kimagemapeditor.svgz\", {st_mode=S_IFREG|0664, st_size=10112, ...}) = 0\n\t...\n\nand that repeats for every single file. There's 233,902 of them. Oops.\n\nOn the other hand, we do certain things pretty well. A \"git diff\", with\nenough memory, takes 0.65s. That's just over *half*a*second* for 233\n*thousand* files. I'd want to have tons of memory to work with this\nrepository, but if I did, I'd still think git is the best thing since\nsliced bread. \n\nAnd doing ops like \"git blame\" on some random file I looked at was\nactually instantaneous. I probably happened to pick a new file just by\nluck, but still.. Most things definitely work pretty damn well. \n\n(Update: I did a\n\n\tgit log --raw -r |\n\t\tgrep '^:100644.*M' |\n\t\tcut -f2 |\n\t\tsort |\n\t\tuniq -c |\n\t\tsort -n\n\nto see the file that was updated the most, to get some kind of\nworst-case for \"git blame\". The list looks like:\n\n ...\n 1091 koffice/kword/kwview.cc\n 1099 kdelibs/khtml/khtml_part.cpp\n 1116 koffice/kpresenter/kpresenter_view.cc\n 1171 kdevelop/ChangeLog\n 1667 kde-common/accounts\n\nand while \"git blame\" is slow on them, it's not *painfully* so. It took\n13s to get the kdevelop/ChangeLog blame, and 31s (probably because the\ndiffs are much more interesting) to get the kpresenter_view.cc blame. \nToo slow, but still usable, and \"git gui\" again made it more interesting\nto wait for it.. \n\nThat said, the more I look at this, the more I think that this is *the*\nperfect example of why you shouldn't put everything in one big\nrepository. Git should be able to handle it, but nobody should really\ndo things like that. It's just stupid.\n\nI will think hard about submodules.\n\n\t\t\tLinus\n"},{"id":"38787","messageId":"43336D7E-AFE7-4BAF-93C7-B302933363B5@adacore.com","threadId":"7528","inReplyTo":"7vk5wqo6ms.fsf@assigned-by-dhcp.cox.net","subject":"Re: kde.git is now online","fromName":"Geert Bosch","fromEmail":"bosch@adacore.com","sentAt":"2007-04-06T11:32:53Z","receivedAt":"2007-04-06T11:32:53Z","isPatch":false,"sender":{"key":"bosch@adacore.com","avatar":null},"body":"On my Mac OS X system, cloning this fails with:\n\npotomac:~/kde%git clone http://userweb.kernel.org/~hpa/kdegit/kde.git\nInitialized empty Git repository in /Users/bosch/kde/kde/.git/\nGetting alternates list for http://userweb.kernel.org/~hpa/kdegit/ \nkde.git/\nGetting pack list for http://userweb.kernel.org/~hpa/kdegit/kde.git/\nGetting index for pack c3df59bc67f69b3861ebef8de308156f1c5fe017\nGetting pack c3df59bc67f69b3861ebef8de308156f1c5fe017\nwhich contains ca908d2d51f154aab9f5727c1e57fb23a2942485\nfatal: packfile /Users/bosch/kde/kde/.git/objects/pack/pack- \nc3df59bc67f69b3861ebef8de308156f1c5fe017.pack cannot be mapped.\n\nEven worse, al files seem to have been deleted, so I have to\ndownload this again. I guess, I shouldn't have used git clone...\n\n -Geert\n"},{"id":"38789","messageId":"alpine.LFD.0.98.0704060857240.28181@xanadu.home","threadId":"7528","inReplyTo":"43336D7E-AFE7-4BAF-93C7-B302933363B5@adacore.com","subject":"Re: kde.git is now online","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-04-06T12:59:00Z","receivedAt":"2007-04-06T12:59:00Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 6 Apr 2007, Geert Bosch wrote:\n\n> On my Mac OS X system, cloning this fails with:\n> \n> potomac:~/kde%git clone http://userweb.kernel.org/~hpa/kdegit/kde.git\n> Initialized empty Git repository in /Users/bosch/kde/kde/.git/\n> Getting alternates list for http://userweb.kernel.org/~hpa/kdegit/kde.git/\n> Getting pack list for http://userweb.kernel.org/~hpa/kdegit/kde.git/\n> Getting index for pack c3df59bc67f69b3861ebef8de308156f1c5fe017\n> Getting pack c3df59bc67f69b3861ebef8de308156f1c5fe017\n> which contains ca908d2d51f154aab9f5727c1e57fb23a2942485\n> fatal: packfile\n> /Users/bosch/kde/kde/.git/objects/pack/pack-c3df59bc67f69b3861ebef8de308156f1c5fe017.pack\n> cannot be mapped.\n> \n> Even worse, al files seem to have been deleted, so I have to\n> download this again. I guess, I shouldn't have used git clone...\n\nGiven that this repo is pushing it to the limits, you better download \nthe .tar.bz2 archive and play with it locally first.\n\n\nNicolas\n"}]}