{"thread":{"id":"19082","subject":"Eric Sink's blog - notes on git, dscms and a \"whole product\" approach","startedAt":"2009-04-27T08:55:55Z","lastAt":"2009-05-04T08:01:57Z","messageCount":39,"participants":["Martin Langhoff","Jakub Narebski","Robin Rosenberg","Jeff King","Sitaram Chamarty","Markus Heidelberg","Michael Witten","Shawn O. Pearce","Nicolas Pitre","Alex Riesen","Kjetil Barvik","Steven Noonan","James Pickens","Dmitry Potapov","Mike Hommey","Tony Finch","Linus Torvalds","david@lang.hm","Daniel Barkalow","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"112405","messageId":"46a038f90904270155i6c802fceoffc73eb5ab57130e@mail.gmail.com","threadId":"19082","inReplyTo":null,"subject":"Eric Sink's blog - notes on git, dscms and a \"whole product\" approach","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-04-27T08:55:55Z","receivedAt":"2009-04-27T08:55:55Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"Eric Sink hs been working on the (commercial, proprietary) centralised\nSCM Vault for a while. He's written recently about his explorations\naround the new crop of DSCMs, and I think it's quite interesting. A\nquick search of the list archives makes me thing it wasn't discussed\nbefore.\n\nThe guy is knowledgeable, and writes quite witty posts -- naturally,\nthere's plenty to disagree on, but I'd like to encourage readers not\nto nitpick or focus on where Eric is wrong. It is interesting to read\nwhere he thinks git and other DSCMs are missing the mark.\n\n Maybe he's right, maybe he's wrong, but damn he's interesting :-)\n\nSo here's the blog - http://www.ericsink.com/\n\nThese are the best entry points\n http://www.ericsink.com/entries/quirky.html\n http://www.ericsink.com/entries/hg_denzel.html\n\nTo be frank, I think he's wrong in some details (as he's admittedly\nonly spent limited time with it) but right on the larger-picture\n(large userbases want it integrated and foolproof, bugtracking needs\nto go distributed alongside the code, git is as powerful^Wdangerous as\nC).\n\ncheers,\n\n\n\nmartin\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"112509","messageId":"m3r5zdnhqu.fsf@localhost.localdomain","threadId":"19082","inReplyTo":"46a038f90904270155i6c802fceoffc73eb5ab57130e@mail.gmail.com","subject":"Cross-Platform Version Control (was: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-28T11:24:31Z","receivedAt":"2009-04-28T11:24:31Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Martin Langhoff writes:\n\n> Eric Sink hs been working on the (commercial, proprietary) centralised\n> SCM Vault for a while. He's written recently about his explorations\n> around the new crop of DSCMs, and I think it's quite interesting. A\n> quick search of the list archives makes me thing it wasn't discussed\n> before.\n> \n> The guy is knowledgeable, and writes quite witty posts -- naturally,\n> there's plenty to disagree on, but I'd like to encourage readers not\n> to nitpick or focus on where Eric is wrong. It is interesting to read\n> where he thinks git and other DSCMs are missing the mark.\n> \n> Maybe he's right, maybe he's wrong, but damn he's interesting :-)\n> \n> So here's the blog - http://www.ericsink.com/\n\n\"Here's a blog\"... and therefore my dilemma. Should I post my reply\nas a comment to this blog, or should I reply here on git mailing list?\n \n> These are the best entry points\n\nBecause those two entries are quite different, I'll reply separately\n\n1. \"Ten Quirky Issues with Cross-Platform Version Control\"\n> http://www.ericsink.com/entries/quirky.html\n\nwhich is generic comment about (mainly) using version control\nin heterogenic environment, where different machines have different\nfilesystem limitations. I'll concentrate here on that issue.\n\n2. \"Mercurial, Subversion, and Wesley Snipes\"\n> http://www.ericsink.com/entries/hg_denzel.html\n\nwhere, paraphrasing, Eric Sink says that he doesn't write about\nMercurial and Subversion because they are perfect. Or at least not\nas controversial (and controversial means interesting).\n\n> \n> To be frank, I think he's wrong in some details (as he's admittedly\n> only spent limited time with it) but right on the larger-picture\n> (large userbases want it integrated and foolproof, bugtracking needs\n> to go distributed alongside the code, git is as powerful^Wdangerous as\n> C).\n\nNeither of mentioned above blog posts touches those issues, BTW...\n\n----------------------------------------------------------------------\nAd 1. \"Ten Quirky Issues with Cross-Platform Version Control\"\n\nActually those are two issues: troubles with different limitations of\ndifferent filesystems, and different dealing with line endings in text\nfiles on different platforms.\n\n\nLine endings (issue 8.) is in theory and in practice (at least for\nGit) a non-issue. \n\nIn theory you should use project's convention for end of line\ncharacter in text files, and use smart editor that can deal (or can be\nconfigured to deal) with this issue correctly.\n\nIn practice this is a matter of correctly setting up core.autocrlf\n(and in more complicated cases, where more complicated means for git\nvery very rare, configuring which files are text and which are not).\n\n\nThere are a few classes of troubles with filesystems (with filenames).\n\n1. Different limitations on file names (e.g. pathname length),\n different special characters, different special filenames (if any).\n Those are issues 2. (special basename PRN on MS Windows), \n issue 3. (trailing dot, trailing whitespace), issue 4. (pathname\n and filename length limit), issue 6. (special characters, in this\n case colon being path element delimiter on MacOS, but it is also\n about special characters like colon, asterisk and question mark\n on MS Windows) and also issue 7. (name that begins with dash)\n in Eric Sink article.\n\n The answer is convention for filenames in a project. Simply DON'T\n use filenames which can cause problems. There is no way to simply\n solve this problem in version control system, although I think if\n you really, really, really need it you should be able to cobble\n something together using low-level git tools to have different name\n for filename in working directory from the one used in repository\n (and index).\n\n See also David A. Wheeler essay \"Fixing Unix/Linux/POSIX Filenames:\n Control Characters (such as Newline), Leading Dashes, and Other Problems\" \n http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n\n DON'T DO THAT.\n\n\n2. \"Case-insensitive\" but \"case-preserving\" filesystems; the case\n where some different filenames are equivalent (like 'README' and\n 'readme' on case-insensitive filesystem), but are returned as you\n created them (so if you created 'README', you would get 'README' in\n directory listing, but filesystem would return that 'readme' exists\n too). This is issue 1. ('README' and 'readme' in the same\n directory) in Eric Sink article.\n\n The answer is like for previous issue: don't. Simply DO NOT create\n files with filenames which differ only in case (like unfortunate\n ct_conntrack.h and cn_CONNTRACK.h or similar in Linux kernel).\n\n But I think that even in case where such unfortunate incident (two\n filenames differing only in case) occur, you can deal with it in\n Git by using lower level tools (and editing only one of two such\n files at once). You would get spurious info about modified files\n in git-status, though... perhaps that could be improved using\n infrastructure created (IIRC) by Linus for dealing with 'insane'\n filesystems.\n\n DON'T DO THAT, SOLVABLE.\n\n\n3. Non \"Case-preserving\" filesystems, where filename as sequence of\n bytes differ between what you created, and what you get from\n filesystem. An example here is MacOS X filesystem, which accepts\n filenames in NFC composed normalized form of Unicode, but stores\n them internally and returns them in NFD decomposed form. This is\n issue 9. (Español being \"Espa\\u00f1ol\" in NFC, but \"Espan\\u0303ol\"\n in NFD).\n\n In this case 'don't do this' might be not acceptable answer.\n Perhaps you need non-ASCII characters in filenames. Not always can\n you use filesystem or specify mount point option that makes it not\n a problem.\n\n I remember that this issue was discussed extensively on git mailing\n list, but I don't remember what was the conclusion (beside agreeing\n that filesystem that is not \"*-preserving\" is not sane filesystem ;).\n In particular I do not remember if Git can deal with this issue\n sanely (I remember Linus adding infrastructure for that, but did it\n solve this problem...).\n\n PROBABLY SOLVED.\n\n\n4. Filesystems which cannot store all SCM-sane metainfo, for example\n filesystems without support for symbolic links, or without support\n for executable permission (executable bit). This is extension of\n issue 10. (which is limited to symbolic links) in Eric Sink\n article.\n\n In Git you have core.fileMode to ignore executable bit differences\n (you would need to use SCM tools and not filesystem tools to\n maniulate it), and core.symlinks to be able to checkout symlinks as\n plain text files (again using SCM tools to manipulate).\n\n SOLVED.\n\n\nThere is also mistaken implicit assumption that version control\nsystems have (and should) preserve all metadata.\n\n5. The issue of extra metadata that is not SCM-sane, and which\n different filesystems can or cannot store. Examples include full\n Unix permissions, Unix ownership (and groups file belongs to),\n other permission-related metadata such as ACL, extra resources tied\n to file such as EA (extended attributes) for some Linux filesystems\n or (in)famous resource form in MacOS. This is issue 5. (resource\n fork on MacOS vs. xattrs on Linux) in Eric Sink article.\n\n This is not an issue for SCM: _source_ code management system\n to solve. Preserving extra metadata indiscrimitedly can cause\n problems, like e.g. full permissions and ownership. Therefore\n SCM preserve only limited SCM-sane subset of metadata. If you\n need to preserve extra metadata, you can use (in good SCMs) hooks\n for that, like e.g. etckeeper uses metastore (in Git).\n\n NOT A PROBLEM.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"112546","messageId":"m3ocugod96.fsf@localhost.localdomain","threadId":"19082","inReplyTo":"46a038f90904270155i6c802fceoffc73eb5ab57130e@mail.gmail.com","subject":"Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-28T18:16:07Z","receivedAt":"2009-04-28T18:16:07Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Martin Langhoff writes:\n\n> Eric Sink hs been working on the (commercial, proprietary) centralised\n> SCM Vault for a while. He's written recently about his explorations\n> around the new crop of DSCMs, and I think it's quite interesting. A\n> quick search of the list archives makes me thing it wasn't discussed\n> before.\n> \n> The guy is knowledgeable, and writes quite witty posts -- naturally,\n> there's plenty to disagree on, but I'd like to encourage readers not\n> to nitpick or focus on where Eric is wrong. It is interesting to read\n> where he thinks git and other DSCMs are missing the mark.\n> \n> Maybe he's right, maybe he's wrong, but damn he's interesting :-)\n> \n> So here's the blog - http://www.ericsink.com/\n\n\"Here's a blog\"... and therefore my dilemma. Should I post my reply\nas a comment to this blog, or should I reply here on git mailing list?\n\nI think I will just add link to this thread in GMane mailing list\narchive for git mailing list...\n \n> These are the best entry points\n* \"Ten Quirky Issues with Cross-Platform Version Control\"\n> http://www.ericsink.com/entries/quirky.html\n\nwhich I have answered in separate post in this thread\n\n* \"Mercurial, Subversion, and Wesley Snipes\"\n> http://www.ericsink.com/entries/hg_denzel.html\n\nwhich I will comment now. The 'ES>' prefix means quoting above blog\npost.\n\n\nFirst there is a list of earlier blog post, with links, which makes\narticle in question a good staring point.\n\nES> As part of that effort, I have undertaken an exploration of the\nES> DVCS world. Several weeks ago I started writing one blog entry\nES> every week, mostly focused on DVCS topics. In chronological\nES> order, here they are:\nES>\nES> * The one where I gripe about Git's index\n\nwhere Eric complains that \"git add -p\" allows for committing untested\nchanges... not knowing about \"git stash --keep-index\", and not\nunderstanding that comitting is (usually) separate from publishing in\ndistributed version control systems (so you can check after commit,\nand amend commit if it does not pass test).\n\nES> * The one where I whine about the way Git allows developers to\nES> rearrange the DAG\n\nwhere Eric seems to not notice that you are strongly encouraged to do\n'rearranging the DAG' (rewriting the history) _only_ in unpublished\n(not made public) part of history.\n\nES> * The one where it looks like I am against DAG-based version\nES> control but I'm really not\n\nwhere Eric conflates linear versus merge workflows with\nupdate-before-commit versus commit-then-merge paradigm, not noticing\nthat you can have linear history using sane commit-update-rebase\nrather than unsafe update-before-commit.\n\nES> * The one where I fuss about DVCSes that try to act like\nES> centralized tools\n\nwhere DVCS in question that behaves this way is Bazaar (if I\nunderstood this correctly).\n\nES> * The one where I complain that DVCSes have a lousy story when it\nES> comes to bug-tracking\n\nwhere Eric correctly notice that distributed version control would not\nhelp much if you use centralized bugtracker, and speculates about\nrequired features that distributed bugtracker should have. Very nice\npost in my opinion.\n\nES> * The one where I lament that I want to like Darcs but I can't\n\nwhere Eric talks about difference between parentage in merge commit\n(which is needed for good merging) and \"parentage\"/weak link in\ncherry-picked commit; Git uses weak link = no link.\n\nES> * The one where I speculate cluelessly about why Git is so fast\n\nwhere Eric guesses instead of asking on git mailing list or #git\nchannel... ;-)\n\nES> Along the way, I've been spending some time getting hands-on\nES> experience with these tools. I've been using Bazaar for several\nES> months. I don't like it very much. I am currently in the process\nES> of switching to Git, but I don't expect to like it very much\nES> either.\n\nAaaargh... if you expect to not like it very much, I would be very\nsuprised if you find it to your liking...\n\nES> So why don't I write about Mercurial? Because I'm pretty sure I\nES> would like it.\nES>\nES> I chose Bazaar and Git for the experience. But if I were choosing\nES> a DVCS as a regular user, I would choose Mercurial. I've used it\nES> some, and found it to be incredibly pleasant. It seems like the\nES> DVCS that got everything just about right. That's great if you're\nES> a user, but for a writer, what's interesting about that?\n\nWell, Mercurial IMHO didn't get everything right. Not mentioning\nimplementation issues, like dealing with copies, binary files, and\nlarge files, it got IMHO wrong:\n * branching in multiple branches per repository\n * tags which should be transferrable but non-versioned\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"112575","messageId":"200904282300.57087.robin.rosenberg.lists@dewire.com","threadId":"19082","inReplyTo":"m3r5zdnhqu.fsf@localhost.localdomain","subject":"Re: Cross-Platform Version Control (was: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-04-28T21:00:56Z","receivedAt":"2009-04-28T21:00:56Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"tisdag 28 april 2009 13:24:31 skrev Jakub Narebski :\n> Line endings (issue 8.) is in theory and in practice (at least for\n> Git) a non-issue. \n> \n> In theory you should use project's convention for end of line\n> character in text files, and use smart editor that can deal (or can be\n> configured to deal) with this issue correctly.\nWindows people will disagree.\n\n> In practice this is a matter of correctly setting up core.autocrlf\n> (and in more complicated cases, where more complicated means for git\n> very very rare, configuring which files are text and which are not).\n\nWhich proves it is an issue or we wouldn't need to tune settings\nto make it work right. A non-issue is something that \"just works\"\nwithout turning knobs. I had had to think more than once on\nwhat the issue was and the right way to solve these issues. It\ncan be considered wierd, because Eclipse on Linux generated files\nwith CRLF which I happily committed and Git on Windows happily\nconverted to LF and determined that the HEAD and index was out\nof sync, but refuesed to commit the CRLF>LF change becuase there\nwas no \"diff\".. You know the fix, but don't tell me it's not an issue.\n\n-- robin\n"},{"id":"112603","messageId":"46a038f90904282355g43bf0cv909905f6028f054f@mail.gmail.com","threadId":"19082","inReplyTo":"m3r5zdnhqu.fsf@localhost.localdomain","subject":"Re: Cross-Platform Version Control (was: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-04-29T06:55:29Z","receivedAt":"2009-04-29T06:55:29Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Tue, Apr 28, 2009 at 1:24 PM, Jakub Narebski wrote:\n>   DON'T DO THAT.\n>   DON'T DO THAT, SOLVABLE.\n\nAs I mentioned, Eric is taking the perspective of offering a supported\nSCM to a large and diverse audience. As such, his notes are\ninteresting not because he's right or he's wrong.\n\nWe can be \"right\" and say \"don't do that\" if we shrink our audience so\nthat it looks a lot like us. There, fixed.\n\nBut something tells me that successful tools are -- by definition --\ntools that grow past their creators use.\n\nSo from Eric's perspective, it is worthwhile to work on all those\nissues, and get the right for the end user -- support things we don't\nlike, offer foolproof catches and warnings that prevent the user from\nshooting their lovely toes off to mars, etc.\n\nHis perspective is one of commercial licensing, but even if we aren't\ndriven by the \"each new user is a new dollar\" bit, the long term hopes\nfor git might also be to be widely used and to improve the version\ncontrol life of many unsuspecting users.\n\nTo get there, I suspect we have to understand more of Eric's perspective.\n\nthat's my 2c.\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"112604","messageId":"20090429072105.GD22593@coredump.intra.peff.net","threadId":"19082","inReplyTo":"46a038f90904282355g43bf0cv909905f6028f054f@mail.gmail.com","subject":"Re: Cross-Platform Version Control (was: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-04-29T07:21:05Z","receivedAt":"2009-04-29T07:21:05Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 29, 2009 at 08:55:29AM +0200, Martin Langhoff wrote:\n\n> So from Eric's perspective, it is worthwhile to work on all those\n> issues, and get the right for the end user -- support things we don't\n> like, offer foolproof catches and warnings that prevent the user from\n> shooting their lovely toes off to mars, etc.\n\nI read a few of his blog postings. He kept complaining about the\nfeatures of git that I like the most. :)\n\nSo one thing I took away from it is that there probably isn't _one_\ninterface that works for everybody. I can see his arguments about how\n\"add -p\" can be dangerous, and how history rewriting can be dangerous.\nSo for some users, blocking those features makes sense.\n\nBut for other users (myself included), those are critical features that\nmake me _way_ more productive. And I manage the risk that comes from\nusing them as part of my workflow, and it isn't a problem in practice.\n\nWhile part of me is happy that cogito is now dead (not because I didn't\nthink it was good, but because having two sets of tools just seemed to\ncreate maintenance and staleness headaches), I do sometimes wonder if we\nwould be better off with several \"from scratch\" git interfaces based\naround the plumbing (or even a C library). And I don't just mean simple\nwrappers around git commands, but whole new interfaces which make\ndecisions like \"no history rewriting at all\", and try to provide a safer\ninterface based on that.\n\nOf course, _I_ wouldn't want to use such an interface. But in theory I\ncould seamlessly interoperate with people who did.\n\n-Peff\n"},{"id":"112609","messageId":"200904290952.17789.jnareb@gmail.com","threadId":"19082","inReplyTo":"46a038f90904282355g43bf0cv909905f6028f054f@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-29T07:52:16Z","receivedAt":"2009-04-29T07:52:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Wed, 29 April 2009, Martin Langhoff wrote:\n> On Tue, Apr 28, 2009 at 1:24 PM, Jakub Narebski \n> wrote: \n\n[I think you cut out a bit too much. Here I resurrected it]\n\nJN> 1. Different limitations on file names (e.g. pathname length),\nJN> different special characters, different special filenames\nJN> (if any).\n[...]\nJN> The answer is convention for filenames in a project. Simply\nJN> DON'T use filenames which can cause problems.\n[...]\n\n> >   DON'T DO THAT.\n\nWhat could be proper solution to that, if you do not accept social \nrather than technical restriction? We can have pre-commit hook that \nchecks for portability for filenames (which is deployment specific,\nand shouldn't be part of SCM perhaps with an exception of being example \nhook) but it wouldn't help dealing with non-portable filenames on \nfilesystem that cannot represent them that are there.\n\nIf I remember correctly Git for some time has layer which can translate \nbetween filenames in repository and filenames on filesystem, but I'm \nnot sure if it is generic enough for it to be a solution to this \nproblem, and currently there is no way to manipulate this mapping, I \nthink.\n\n\nJN> 2. \"Case-insensitive\" but \"case-preserving\" filesystems. [...]\nJN>\nJN> The answer is like for previous issue: don't. Simply DO NOT\nJN> create files with filenames which differ only in case [...]\n\n> >   DON'T DO THAT, SOLVABLE.\n\nBy 'solvable' here I mean that you should be able to modify only one of \nclashing files at once (checkout 'README', modify, add to index, remove \nfrom filesystem, checkout 'readme', modify, etc.), and deal with \nannoyances in git-status output. It can be done in Git, with medium \namount of hacking. I don't think any other SCM can do even this, and\nI cannot think of a better, automatic solution that would somehow deal \nwith case-clashing.\n\nNote that all deals are off in case-insensitive and not preserving \nfilesystem.\n\nBy the way, wouldn't be a better solution to use sane filesystem, rather \nthan complicating SCM? ;-)\n\n> \n> As I mentioned, Eric is taking the perspective of offering a supported\n> SCM to a large and diverse audience. As such, his notes are\n> interesting not because he's right or he's wrong.\n> \n> We can be \"right\" and say \"don't do that\" if we shrink our audience so\n> that it looks a lot like us. There, fixed.\n\n\n [...] the attitude of the knife — chopping off what's incomplete and\n saying: \"Now it's complete because it's ended here.\"\n\n\nI could not resist posting this quote :-P\n\n> \n> But something tells me that successful tools are -- by definition --\n> tools that grow past their creators use.\n> \n> So from Eric's perspective, it is worthwhile to work on all those\n> issues, and get the right for the end user -- support things we don't\n> like, offer foolproof catches and warnings that prevent the user from\n> shooting their lovely toes off to mars, etc.\n\nWarnings and catches I can accept; adding complications and corner cases \nfor situations which can be trivially avoided with a bit of social \nengineering aka. project guidelines... not so much.\n\nI simply cannot see the situation where you _must_ have dangerously \nunportable file names (trailing dot, trailing whitespace) and \ncase-clashing files...\n\n> \n> His perspective is one of commercial licensing, but even if we aren't\n> driven by the \"each new user is a new dollar\" bit, the long term hopes\n> for git might also be to be widely used and to improve the version\n> control life of many unsuspecting users.\n> \n> To get there, I suspect we have to understand more of Eric's\n> perspective. \n> \n> that's my 2c.\n\nBy the way, I think that the article on cross-platform version control \n(version control in heterogenic environment) is quite good article.\nI don't quite like the \"10 Issues\"/\"Top 10\" way of writing, but the \narticle examines different ways that heterogenic environment can trip \nSCM. \n\nIn my opinion Git does quite good here, where it can, and where the \nissue is to be solved by SCM and not otherwise (extra metadata like \nresource fork).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"112608","messageId":"slrngvg1u9.65c.sitaramc@sitaramc.homelinux.net","threadId":"19082","inReplyTo":"m3ocugod96.fsf@localhost.localdomain","subject":"Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2009-04-29T07:54:50Z","receivedAt":"2009-04-29T07:54:50Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On 2009-04-28, Jakub Narebski wrote:\n\n> ES> * The one where I lament that I want to like Darcs but I can't\n>\n> where Eric talks about difference between parentage in merge commit\n> (which is needed for good merging) and \"parentage\"/weak link in\n> cherry-picked commit; Git uses weak link = no link.\n\nWell the patch-id is a sort of \"compute on demand\" link, so\nit would qualify as a weak link, especially because git\nmanages to use it during a rebase.\n\nI wanted to point that out but I didn't see a link to post\ncomments so I didn't bother.\n"},{"id":"112613","messageId":"46a038f90904290125n11476cf3icbacab4f6d8a5f5a@mail.gmail.com","threadId":"19082","inReplyTo":"200904290952.17789.jnareb@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-04-29T08:25:56Z","receivedAt":"2009-04-29T08:25:56Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Apr 29, 2009 at 9:52 AM, Jakub Narebski wrote:\n>> >   DON'T DO THAT.\n>\n> What could be proper solution to that, if you do not accept social\n> rather than technical restriction?\n\nLet's say strong checks for case sensitivity clashes, leading/trailing\ndots, utf-8 encoding maladies, etc switched on by default. And note\nthat to be user-friendly you want most of those checks at 'add' time.\n\n If we don't like a particular FS, or we think it is messing up our\nutf-8 filenames, say it up-front, at clone and checkout time. For\nexample, if the checkout has files with interesting utf-8 names, it'd\nbe reasonable to check for filename mangling.\n\nSome things are hard or impossible to prevent - the utf-8 encoding\nmaladies of OSX for example. But it may be detectable on checkout.\n\nIn short, play on the defensive, for the benefit of users who are not\nkernel developers.\n\nIt will piss off kernel & git developers and slow some operations\nsomewhat. It will piss off oldtimers like me. But I'll say git config\n--global core.trainingwheels no and life will be good.\n\nIt may be - as Jeff King points out - a matter of a polished git\nporcelain. We've seen lots of porcelains, but no smooth user-targetted\nporcelain yet.\n\ncheers,\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"112679","messageId":"200904292205.38530.markus.heidelberg@web.de","threadId":"19082","inReplyTo":"20090429072105.GD22593@coredump.intra.peff.net","subject":"Re: Cross-Platform Version Control (was: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Markus Heidelberg","fromEmail":"markus.heidelberg@web.de","sentAt":"2009-04-29T20:05:37Z","receivedAt":"2009-04-29T20:05:37Z","isPatch":false,"sender":{"key":"markus.heidelberg@web.de","avatar":"https://avatars.githubusercontent.com/u/6334512?v=4"},"body":"Jeff King, 29.04.2009:\n> On Wed, Apr 29, 2009 at 08:55:29AM +0200, Martin Langhoff wrote:\n> \n> > So from Eric's perspective, it is worthwhile to work on all those\n> > issues, and get the right for the end user -- support things we don't\n> > like, offer foolproof catches and warnings that prevent the user from\n> > shooting their lovely toes off to mars, etc.\n> \n> I read a few of his blog postings. He kept complaining about the\n> features of git that I like the most. :)\n> \n> I can see his arguments about how\n> \"add -p\" can be dangerous\n\nActually, I don't see a very special case here with committing a never\ncompiled/tested worktree state. You can do this with every VCS (without\nan index like git) with just selectively committing files instead of the\nwhole current worktree.\n\nMarkus\n"},{"id":"112758","messageId":"m3fxfqnxn5.fsf_-_@localhost.localdomain","threadId":"19082","inReplyTo":"m3ocugod96.fsf@localhost.localdomain","subject":"Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-30T12:17:58Z","receivedAt":"2009-04-30T12:17:58Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski writes:\n\n> Martin Langhoff writes:\n> \n> > Eric Sink hs been working on the (commercial, proprietary) centralised\n> > SCM Vault for a while. He's written recently about his explorations\n> > around the new crop of DSCMs, and I think it's quite interesting.\n\n[...]\n\n> > So here's the blog - http://www.ericsink.com/\n\n[...]\n> * \"Mercurial, Subversion, and Wesley Snipes\"\n> > http://www.ericsink.com/entries/hg_denzel.html\n> \n> which I will comment now. The 'ES>' prefix means quoting above blog\n> post.\n\n[...]\n> ES> * The one where I speculate cluelessly about why Git is so fast\n> \n> where Eric guesses instead of asking on git mailing list or #git\n> channel... ;-)\n\nThis issue is interesting: what features and what design decision\nmake Git fast? One of the goals of Git was good performance; are\nwe there?\n\nAll quotes marked 'es> ' below are from \"Why is Git so Fast?\" post\nhttp://www.ericsink.com/entries/why_is_git_fast.html\n\nes> One: Maybe Git is fast simply because it's a DVCS.\nes>\nes> There's probably some truth here. One of the main benefits touted\nes> by the DVCS fanatics is the extra performance you get when\nes> everything is \"local\".\n\nThis is I think quite obvious. Accessing memory is faster than\nacessing disk, which in turn is faster than accessing network. So if\ncommit and (change)log does not require access to server via network,\nthey are so much faster.\n\nBTW. that is why Subversion stores along working copy 'pristine'\nversions of files: to make status and diff fast enough to be usable.\nWhich in turn might make SVN checkout to be larger than full Git\nclone ;-)\n\nes>\nes> But this answer isn't enough. Maybe it explains why Git is faster\nes> than Subversion, but it doesn't explain why Git is so often\nes> described as being faster than the other DVCSs.\n\nNot only described; see http://git.or.cz/gitwiki/GitBenchmarks\n(although some, if not most of those benchmarks are dated,\nand e.g. Bazaar claims to have much better performance now).\n\nes>\nes> Two: Maybe Git is fast because Linus Torvalds is so smart.\n\n[non answer; the details are important]\n\nes> Three: Maybe Git is fast because it's written in C instead of one\nes> of those newfangled higher-level languages.\nes>\nes> Nah, probably not. Lots of people have written fast software in\nes> C#, Java or Python.\nes>\nes> And lots of people have written really slow software in\nes> traditional native languages like C/C++. [...]\n\nWell, I guess that access to low-level optimization techniques like\nmmap are important for performance. But here I am guessing and\nspeculating like Eric did; well, I am asking on a proper forum ;-)\n\nWe have some anegdotical evidence supporting this possibility (which\nEric dismisses), namely the fact that pure-Python Bazaar is slowest of\nthree most common open source DVCS (Git, Mercurial, bazaar) and the\nfact that parts of Mercurial were written in C for better performance.\n\nWe can also compare implementations of Git in other, higher level\nlanguages, with reference implementation in C (and shell scripts, and\nPerl ;-)). For example most complete I think but still not fully\ncomplete Java implementation: JGit. I hope that JGit developers can\ntell us whether using higher level language affects performance, how\nmuch, and what features of higher-level language are causing decrease\nin performance. Of course we have to take into account the\npossibility that JGit isn't simply as well optimized because of less\nmanpower.\n\nes>\nes> Four: Maybe Git is fast because being fast is the primary goal for\nes> Git.\n\n[non answer; the details are important]\n\nes>\nes> Five: Maybe Git is fast because it does less.\nes>\nes> One of my favorite recent blog entries is this piece[1] which\nes> claims that the way to make code faster is to have it do less.\nes>\nes> [1] \"How to write fast code\" by Kas Thomas\nes> http://asserttrue.blogspot.com/2009/03/how-to-write-fast-code.html\n[...]\n\nes>\nes> For example, the way you get something in the Git index is you use\nes> the \"git add\" command. Git doesn't scan your working copy for\nes> changed files unless you explicitly tell it to. This can be a\nes> pretty big performance win for huge trees. Even when you use the\nes> \"remember the timestamp\" trick, detecting modified files in a\nes> really big tree can take a noticeable amount of time.\n\nThat of course depends on how you compare performance of different\nversion control systems (to not compare apples with oranges). But if\nyou compare e.g. \" commit\" with Git equivalent \"git commit -a\"\nthe above is simply not true.\n\nBTW. when doing comparison you have to take care of the reverse,\ne.g. git doing more like calculating and dislaying diffstat by default\nfor merges/pulls.\n\nes>\nes> Or maybe Git's shortcut for handling renames is faster than doing\nes> them more correctly[2] like Bazaar does.\nes>\nes> [2] \"Renaming is the killer app of distributed version control\"\nes> http://www.markshuttleworth.com/archives/123\n\nErrr... what?\n\n\nes> Six: Maybe Git is fast because it doesn't use much external code.\nes>\nes> Very often, when you are facing a decision to use somebody else's\nes> code or write it yourself, there is a performance tradeoff. Not\nes> always, but often. Maybe the third party code is just slower than\nes> the code you could write yourself if you had time to do it. Or\nes> maybe there is an impedance mismatch between the API of the\nes> external library and your own architecture.\nes>\nes> This can happen even when the library is very high quality. For\nes> example, consider libcurl. This is a great library. Tons of\nes> people use it. But it does have one problem that will cause\nes> performance problems for some users: When using libcurl to fetch\nes> an object, it wants to own the buffer. In some situations, this\nes> can end up forcing you to use extra memcpys or temporary files.\nes> The reason all the low level calls like send() and recv() allow\nes> the caller to own the loop and the buffer is because this is the\nes> best way to avoid the need to make extra copies of the data on\nes> disk or in memory.\n[...]\n\nes>\nes> Maybe Git is fast because every time they faced one of these \"buy\nes> vs. build\" choices, they decided to just write it themselves.\n\nI don't think so. Rather the opposite is true. Git uses libcurl for\nHTTP transport. Git uses zlib for compression. Git uses SHA-1 from\nOpenSSL or from Mozilla. Git uses (modified, internal) LibXDiff for\n(binary) deltaifying, for diffs and for merges.\n\nOTOH Git includes several own micro-libraries: parseopt, strbuf,\nALLOC_GROW, etc. NIH syndrome? I don't think so; rather avoiding\nextra dependencies (bstring vs strbuf), and existing solutions not\nfitting all needs (popt/argp/getopt vs parse-options).\n\nes> Seven: Maybe Git isn't really that fast.\nes>\nes> If there is one thing I've learned about version control it's that\nes> everybody's situation is different. It is quite likely that Git\nes> is a lot faster for some scenarios than it is for others.\nes>\nes> How does Git handle really large trees? Git was designed primary\nes> to support the efforts of the Linux kernel developers. A lot of\nes> people think the Linux kernel is a large tree, but it's really\nes> not. Many enterprise configuration management repositories are\nes> FAR bigger than the Linux kernel.\n\nc.f. \"Why Perforce is more scalable than Git\" by Steve Hanov\n http://gandolf.homelinux.org/blog/index.php?id=50\n\nI don't really know about this.\n\nBut there is one issue Eric Sink didn't think about:\n\nEight: Git seems fast.\n======================\n\nHere I mean concentaring on low _latency_, which means that when git\nproduces more than one page of output (for example \"git log\"), it tries to output the first page as fast as possible; which means that first page e.g.\n\"git | head -25 >/dev/null\" has to be fast, and not \n\"git >/dev/null\" itself.\n\nHaving progress indicator appearing whenever is longer wait (quite\nfresh feature) also help impression of being fast...\n\n\nAnd what do you think about this?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"112761","messageId":"b4087cc50904300556s359c91dfu444fa40ea85bd66e@mail.gmail.com","threadId":"19082","inReplyTo":"m3fxfqnxn5.fsf_-_@localhost.localdomain","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Michael Witten","fromEmail":"mfwitten@gmail.com","sentAt":"2009-04-30T12:56:35Z","receivedAt":"2009-04-30T12:56:35Z","isPatch":false,"sender":{"key":"mfwitten@gmail.com","avatar":"https://avatars.githubusercontent.com/u/597101?v=4"},"body":"On Thu, Apr 30, 2009 at 07:17, Jakub Narebski wrote:\n> I hope that JGit developers can\n> tell us whether using higher level language affects performance, how\n> much, and what features of higher-level language are causing decrease\n> in performance.\n\nJava is definitely higher than C, but you can do some pretty low-level\noperations on bits and bytes and the like, not to mention the presence\nof a JIT.\n\nMy point: I don't think that Java can tell us anything special in this regard.\n"},{"id":"112762","messageId":"20090430142244.GA23550@coredump.intra.peff.net","threadId":"19082","inReplyTo":"m3fxfqnxn5.fsf_-_@localhost.localdomain","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-04-30T14:22:44Z","receivedAt":"2009-04-30T14:22:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Apr 30, 2009 at 05:17:58AM -0700, Jakub Narebski wrote:\n\n> This is I think quite obvious. Accessing memory is faster than\n> acessing disk, which in turn is faster than accessing network. So if\n> commit and (change)log does not require access to server via network,\n> they are so much faster.\n\nLike all generalizations, this is only mostly true. Fast network servers\nwith big caches can outperform disks for some loads. And in many cases\nwith a VCS, you are performing a query that might look over the whole\ndataset, but return only a small fraction of data.\n\nSo I wouldn't rule out the possibility of a pleasant VCS experience on a\nnetwork-optimized system backed by beefy servers on a local network. I\nhave never used perforce, but I get the impression that it is more\noptimized for such a situation. Git is really optimized for open source\nprojects: slow servers across high-latency, low-bandwidth links.\n\n> es> Nah, probably not. Lots of people have written fast software in\n> es> C#, Java or Python.\n> es>\n> es> And lots of people have written really slow software in\n> es> traditional native languages like C/C++. [...]\n> \n> Well, I guess that access to low-level optimization techniques like\n> mmap are important for performance. But here I am guessing and\n> speculating like Eric did; well, I am asking on a proper forum ;-)\n\nCertainly there's algorithmic fastness that you can do in any language,\nand I think git does well at that. Most operations are independent of\nthe total size of history (e.g., branching is O(1) and commit is\nO(changed files), diff looks only at endpoints, etc). Operations which\ndeal only with history are independent of the size of the tree (e.g.,\n\"git log\" and the history graph in gitk look only at commits, never at\nthe tree). And when we do have to look at the tree, we can drastically\nreduce our I/O by comparing hashes instead of full files.\n\nBut there are also some micro-optimizations that make a big difference\nin practice. Some of them can be done in any language. For example, the\npackfiles are ordered by type so that all of the commits have a nice I/O\npattern when doing a history walk.\n\nSome other micro-optimizations are really language-specific, though. I\ndon't recall the numbers, but I think Linus got measurable speedups from\ncutting the memory footprint of the object and commit structs (which\ngave better cache usage patterns). Git uses some variable-length fields\ninside structs instead of a pointer to a separate allocated string to\ngive better memory access patterns. Tricks like that won't give the\norder-of-magnitude speedups that algorithmic optimizations will, but 10%\nhere and 20% there means you can get a system that is a few times faster\nthan the competition. For an operation that takes 0.1s anyway, that\ndoesn't matter. But with current hardware and current project size, you\nare often talking about dropping a 3-second operation down to 1s or\n0.5s, which just feels a lot snappier.\n\nAnd finally, git tries to do as little work as possible when starting a\nnew command, and streams output as soon as possible. Which means that in\na command-line setting, git can _feel_ snappier, because it starts\noutput immediately. Higher-level languages can often have a much longer\nstartup time, especially if they have a lot of modules to load. E.g.,:\n\n # does enough work to easily fill your pager\n $ time git log -100 >/dev/null\n real 0m0.011s\n user 0m0.008s\n sys 0m0.004s\n\n # does nothing, just starts perl and aborts with usage\n $ time git send-email >/dev/null\n real 0m0.150s\n user 0m0.104s\n sys 0m0.048s\n\nBoth are warm-cache times. C git gives you output almost instaneously,\nwhereas just loading perl with a modest set of modules introduces a\nnoticeable pause before any work is actually done. In the grand scheme\nof things, .1s probably isn't relevant, but I think avoiding that delay\nadds to the perception of git as fast.\n\n> es> Or maybe Git's shortcut for handling renames is faster than doing\n> es> them more correctly[2] like Bazaar does.\n> es>\n> es> [2] \"Renaming is the killer app of distributed version control\"\n> es> http://www.markshuttleworth.com/archives/123\n> \n> Errr... what?\n\nYeah, I had the same thought. Git's rename handling is _much_ more\ncomputationally intensive than other systems. In fact, it is one of only\ntwo places where I have ever wanted git to be any faster (the other\nbeing repacking of large repos).\n\n> Eight: Git seems fast.\n> ======================\n> \n> Here I mean concentaring on low _latency_, which means that when git\n\nI do think this helps (see above), but I wanted to note that it is more\nthan just \"streaming\"; I think other systems stream, as well. For\nexample, I am pretty sure that \"cvs log\" streamed (but thank god it has\nbeen so long since I touched CVS that I can't really remember), but it\n_still_ felt awfully slow.\n\nSo it is also about keeping start times low and having your data in a\nformat that is ready to use.\n\n-Peff\n"},{"id":"112770","messageId":"200904301728.06989.jnareb@gmail.com","threadId":"19082","inReplyTo":"b4087cc50904300556s359c91dfu444fa40ea85bd66e@mail.gmail.com","subject":"Re: Why Git is so fast","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-30T15:28:04Z","receivedAt":"2009-04-30T15:28:04Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 30 Apr 2009, Michael Witten wrote:\n> On Thu, Apr 30, 2009 at 07:17, Jakub Narebski wrote:\n\n> > I hope that JGit developers can\n> > tell us whether using higher level language affects performance, how\n> > much, and what features of higher-level language are causing decrease\n> > in performance.\n> \n> Java is definitely higher than C, but you can do some pretty low-level\n> operations on bits and bytes and the like, not to mention the presence\n> of a JIT.\n> \n> My point: I don't think that Java can tell us anything special in this regard.\n\nLet's rephrase question a bit then: what low-level operation were needed\nfor good performance in JGit? \n\n-- \nJakub Narebski\nPoland\n"},{"id":"112779","messageId":"20090430184319.GP23604@spearce.org","threadId":"19082","inReplyTo":"b4087cc50904300556s359c91dfu444fa40ea85bd66e@mail.gmail.com","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-30T18:43:19Z","receivedAt":"2009-04-30T18:43:19Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Michael Witten wrote:\n> On Thu, Apr 30, 2009 at 07:17, Jakub Narebski wrote:\n> > I hope that JGit developers can\n> > tell us whether using higher level language affects performance, how\n> > much, and what features of higher-level language are causing decrease\n> > in performance.\n> \n> Java is definitely higher than C, but you can do some pretty low-level\n> operations on bits and bytes and the like, not to mention the presence\n> of a JIT.\n\nBut its still costly compared to C.\n \n> My point: I don't think that Java can tell us anything special in this regard.\n\nSure it can.\n\nPeff I think made a good point here, that we rely on a lot of small\ntweaks in the C git code to get *really* good performance. 5% here,\n10% there, and suddenly you are 60% faster than you were before.\nNico, Linus, Junio, they have all spent some time over the past\n3 or 4 years trying to tune various parts of Git to just flat out\nrun fast.\n\nHigher level languages hide enough of the machine that we can't\nmake all of these optimizations.\n\nJGit struggles with not having mmap(), or when you do use Java NIO\nMappedByteBuffer, we still have to copy to a temporary byte[] in\norder to do any real processing. C Git avoids that copy. Sure,\nother higher level langauges may offer a better mmap facility,\nbut they also tend to offer garbage collection and most try to tie\nthe mmap management into the GC \"for safety and ease of use\".\n\nJGit struggles with not having unsigned types in Java. There are\nmany locations in JGit where we really need \"unsigned int32_t\" or\n\"unsigned long\" (largest machine word available) or \"unsigned char\"\nbut these types just don't exist in Java. Converting a byte up to\nan int just to treat it as an unsigned requires an extra \" & 0xFF\"\noperation to remove the sign extension.\n\nJGit struggles with not having an efficient way to represent a SHA-1.\nC can just say \"unsigned char[20]\" and have it inline into the\ncontainer's memory allocation. A byte[20] in Java will cost an\n*additional* 16 bytes of memory, and be slower to access because\nthe bytes themselves are in a different area of memory from the\ncontainer object. We try to work around it by converting from a\nbyte[20] to 5 ints, but that costs us machine instructions.\n\nC Git takes for granted that memcpy(a, b, 20) is dirt cheap when\ndoing a copy from an inflated tree into a struct object. JGit has\nto pay a huge penalty to copy that 20 byte region out into 5 ints,\nbecause later on, those 5 ints are cheaper.\n\nOther higher level languages also lack the ability to mark a\ntype unsigned. Or face similiar penalties with storing a 20 byte\nbinary region.\n\nNative Java collection types have been a snare for us in JGit.\nWe've used java.util.* types when they seem to be handy and already\nsolve the data structure problem at hand, but they tend to preform\na lot worse than writing a specialized data structure.\n\nFor example, we have ObjectIdSubclassMap for what should be\nMap. Only it requires that the Object type you\nuse as the \"value\" entry in the map extend from ObjectId, as the\ninstance serves as both key *and* value. But it screams when\ncompared to HashMap. (For those who don't know,\nObjectId is JGit's \"unsigned char[20]\" for a SHA-1.)\n\nJust a day or so ago I wrote LongMap, a faster HashMap,\nfor hashing objects by indexes in a pack file. Again, the boxing\ncosts in Java to convert a \"long\" (largest integer type) into an\nObject that the standard HashMap type would accept was rather high.\n\nRight now, JGit is still paying dearly when it comes to ripping\napart a commit or a tree object to follow the object links. Or when\ninvoking inflate(). We spend a lot more time doing this sort of work\nthan C git does, and yet we're trying to be as close to the machine\nas we can go by using byte[] whenever possible, by avoiding copying\nwhenever possible, and avoiding memory allocation when possible.\n\nNotably, `rev-list --objects --all` takes about 2x as long in\nJGit as it does in C Git on a project like the linux kernel, and\n`index-pack` for the full ~270M pack file takes about 2x as long.\n\nBoth parts of JGit are about as good as I know how to make them,\nbut we're really at the mercy of the JIT, and changes in the JIT\ncan cause us to perform worse (or better) than before. Unlike in\nC Git where Linus has done assembler dumps of sections of code and\ntried to determine better approaches. :-)\n\nSo. Yes, its practical to build Git in a higher level language, but\nyou just can't get the same performance, or tight memory utilization,\nthat C Git gets. That's what that higher level language abstraction\ncosts you. But, JGit performs reasonably well; well enough that\nwe use internally at Google as a git server.\n\n-- \nShawn.\n"},{"id":"112780","messageId":"20090430185244.GR23604@spearce.org","threadId":"19082","inReplyTo":"200904301728.06989.jnareb@gmail.com","subject":"Re: Why Git is so fast","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-30T18:52:44Z","receivedAt":"2009-04-30T18:52:44Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> Let's rephrase question a bit then: what low-level operation were needed\n> for good performance in JGit? \n\nAside from the message I just posted:\n\n- Avoid String, its too expensive most of the time. Stick with\n byte[], and better, stick with data that is a triplet of (byte[],\n int start, int end) to define a region of data. Yes, its annoying,\n as its 3 values you need to pass around instead of just 1, but\n its makes a big difference in running time.\n\n- Avoid allocating byte[] for SHA-1s, instead we convert to 5 ints,\n which can be inlined into an object allocation.\n\n- Subclass instead of contain references. We extend ObjectId to\n attach application data, rather than contain a reference to an\n ObjectId. Classical Java programming techniques would say this\n is a violation of encapsulatio. But it gets us the same memory\n impact that C Git gets by saying:\n\n struct appdata {\n unsigned char[20] sha1;\n ....\n\t}\n\n- We're hurting dearly for not having more efficient access to the\n pack-*.pack file data. mmap in Java is crap. We implement our\n own page buffer, reading in blocks of 8192 bytes at a time and\n holding them in our own cache.\n\n Really, we should write our own mmap library as an optional JNI\n thing, and tie it into libz so we can efficiently run inflate()\n off the pack data directly.\n\n- We're hurting dearly for not having more efficient access to the\n pack-*.idx files. Again, with no mmap we read the entire bloody\n index into memory. But since you won't touch most of it we keep\n it in large byte[], but since you are searching with an ObjectId\n (5 ints) we pay a conversion price on every search step where\n we have to copy from the large byte[] to 5 local variable ints,\n and then compare to the ObjectId. Its an overhead C git doesn't\n have to deal with.\n\nAnyway.\n\nI'm still just amazed at how well JGit runs given these limitations.\nI guess that's Moore's Law for you. 10 years ago, JGit wouldn't\nhave been practical.\n\n-- \nShawn.\n"},{"id":"112781","messageId":"alpine.LFD.2.00.0904301401120.6741@xanadu.home","threadId":"19082","inReplyTo":"m3fxfqnxn5.fsf_-_@localhost.localdomain","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-04-30T18:56:23Z","receivedAt":"2009-04-30T18:56:23Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 30 Apr 2009, Jakub Narebski wrote:\n\n> Jakub Narebski writes:\n> \n> es> Two: Maybe Git is fast because Linus Torvalds is so smart.\n> \n> [non answer; the details are important]\n\nI think Linus is certainly responsible for a big part of Git's speed. \nHe came with the basic data structure used by git which has lots to do \nwith that. Also, he designed Git specifically to fulfill a need for \nwhich none of the alternatives were fast enough. Hence Git was designed \nfrom the ground up with speed as one of the primary design goals, such \nas being able to create multiple commits per second instead of the other \nway around (several seconds per commit). And yes, Linus is usually smart \nenough with the proper mindset to achieve such goals.\n\n> es> Three: Maybe Git is fast because it's written in C instead of one\n> es> of those newfangled higher-level languages.\n> es>\n> es> Nah, probably not. Lots of people have written fast software in\n> es> C#, Java or Python.\n> es>\n> es> And lots of people have written really slow software in\n> es> traditional native languages like C/C++. [...]\n> \n> Well, I guess that access to low-level optimization techniques like\n> mmap are important for performance. But here I am guessing and\n> speculating like Eric did; well, I am asking on a proper forum ;-)\n> \n> We have some anegdotical evidence supporting this possibility (which\n> Eric dismisses), namely the fact that pure-Python Bazaar is slowest of\n> three most common open source DVCS (Git, Mercurial, bazaar) and the\n> fact that parts of Mercurial were written in C for better performance.\n> \n> We can also compare implementations of Git in other, higher level\n> languages, with reference implementation in C (and shell scripts, and\n> Perl ;-)). For example most complete I think but still not fully\n> complete Java implementation: JGit. I hope that JGit developers can\n> tell us whether using higher level language affects performance, how\n> much, and what features of higher-level language are causing decrease\n> in performance. Of course we have to take into account the\n> possibility that JGit isn't simply as well optimized because of less\n> manpower.\n\nOne of the main JGit developers is Shawn Pearce. If you look at Shawn's \ncontribution to C git, they mostly are all related to performance \nissues. Amongst other things, he is the author of git-fast-import, he \ncontributed the pack access windowing code, and he was also involved in \nthe initial design of pack v4. Hence Shawn is a smart guy who certainly \nknows one or two things about performance optimizations. Yet he \nreported on this list that his efforts to make JGit faster were not much \nsuccessful anymore, most probably due to the language overhead.\n\n> es> Four: Maybe Git is fast because being fast is the primary goal for\n> es> Git.\n> \n> [non answer; the details are important]\n\nStill, this is actually true (see about Linus above). Without such a \ngoal, you quickly lose sight of performance regressions.\n\n> es> Maybe Git is fast because every time they faced one of these \"buy\n> es> vs. build\" choices, they decided to just write it themselves.\n> \n> I don't think so. Rather the opposite is true. Git uses libcurl for\n> HTTP transport. Git uses zlib for compression. Git uses SHA-1 from\n> OpenSSL or from Mozilla. Git uses (modified, internal) LibXDiff for\n> (binary) deltaifying, for diffs and for merges.\n\nWell, I think he's right on this point as well. libcurl is not so \nrelevant since it is rarely the bottleneck (the network bandwidth itself \nusually is). zlib is already as fast as it can be as multiple attempts \nto make it faster didn't succeed. Git already carries its own version \nof SHA-1 code for ARM and PPC because the alternatives were slower. \nThe fact that libxdiff was made internal is indeed to have a better \nimpedance matching with the core code, otherwise it could have remained \nfully external just like zlib. And the binary delta code is not \nlibxdiff anymore but a much smaller, straight forward, and optimized to \ndeath version to achieve speed over versatility (no need to be versatile \nwhen strictly dealing with Git's needs only).\n\n> es> Seven: Maybe Git isn't really that fast.\n> es>\n> es> If there is one thing I've learned about version control it's that\n> es> everybody's situation is different. It is quite likely that Git\n> es> is a lot faster for some scenarios than it is for others.\n> es>\n> es> How does Git handle really large trees? Git was designed primary\n> es> to support the efforts of the Linux kernel developers. A lot of\n> es> people think the Linux kernel is a large tree, but it's really\n> es> not. Many enterprise configuration management repositories are\n> es> FAR bigger than the Linux kernel.\n> \n> c.f. \"Why Perforce is more scalable than Git\" by Steve Hanov\n> http://gandolf.homelinux.org/blog/index.php?id=50\n> \n> I don't really know about this.\n\nGit certainly sucks big time with large files.\n\nGit also sucks to a lesser extent (but still) with very large \nrepositories.\n\nBut large trees? I don't think Git is worse than anything out there \nwith a large tree of average size files.\n\nYet, this point is misleading because when people gives to Git the \nreputation of being faster, this is certainly from comparison of \noperations performed on the same source tree. Who cares about scenarios \nfor which the tool was not designed? Those \"enterprise configuration \nmanagement repositories\" are not what Git was designed for indeed, but \nneither was Mercurial nor Bazaar, or any other contender to which Git is \nusually compared.\n\n\nNicolas\n"},{"id":"112782","messageId":"81b0412b0904301216j7ef73870y775cf6d89b5aa71e@mail.gmail.com","threadId":"19082","inReplyTo":"alpine.LFD.2.00.0904301401120.6741@xanadu.home","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-04-30T19:16:59Z","receivedAt":"2009-04-30T19:16:59Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/4/30 Nicolas Pitre :\n> Yet, this point is misleading because when people gives to Git the\n> reputation of being faster, this is certainly from comparison of\n> operations performed on the same source tree.  Who cares about scenarios\n> for which the tool was not designed?  Those \"enterprise configuration\n> management repositories\" are not what Git was designed for indeed, but\n\nEspecially when no sane developer will put in his repository the toolchain\n(pre-compiled. For all supported platforms!), all the supporting tools\n(like grep,\nfind, etc.Pre-compiled _and_ source), the in-house framework (pre-compiled\nand source, again), firmware (pre-compiled and put in the repository weekly),\nand operating system code (pre-compiled, with firmware-specific drivers,\nupdated, you guessed it, weekly), and well, there is the project itself (Java or\nC++, and documentation in .doc and .xls)...\nNow, what kind of self-hating idiot will design a system for that kind of abuse?\n(And if someone says that's is not true in the most enterprise\nf$%cking configurations,\nhe definitely hasn't had to live through big enough number of them).\n"},{"id":"112786","messageId":"200904302134.00569.jnareb@gmail.com","threadId":"19082","inReplyTo":"alpine.LFD.2.00.0904301401120.6741@xanadu.home","subject":"Re: Why Git is so fast","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-04-30T19:33:59Z","receivedAt":"2009-04-30T19:33:59Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 30 Apr 2009, Nicolas Pitre wrote:\n> On Thu, 30 Apr 2009, Jakub Narebski wrote:\n> > Jakub Narebski writes:\n\n> > es> Maybe Git is fast because every time they faced one of these \"buy\n> > es> vs. build\" choices, they decided to just write it themselves.\n> > \n> > I don't think so. Rather the opposite is true. Git uses libcurl for\n> > HTTP transport. Git uses zlib for compression. Git uses SHA-1 from\n> > OpenSSL or from Mozilla. Git uses (modified, internal) LibXDiff for\n> > (binary) deltaifying, for diffs and for merges.\n> \n> Well, I think he's right on this point as well. [...]\n> The fact that libxdiff was made internal is indeed to have a better \n> impedance matching with the core code, otherwise it could have remained \n> fully external just like zlib. And the binary delta code is not \n> libxdiff anymore but a much smaller, straight forward, and optimized to \n> death version to achieve speed over versatility (no need to be versatile \n> when strictly dealing with Git's needs only).\n\nHrmmmm... I have thought that LibXDiff was internalized mainly for ease\nof modification, as my impression is that LibXDiff is single developer\neffort, while Git from beginning have many contributors (and submodules\ndidn't exist then). If I remember correctly the rcsmerge/diff3 algorithm\nwas added first in internalized git's xdiff... was it added to LibXDiff\nproper, anyway?\n\nBTW. I wonder what other F/OSS version control systems: Bazaar,\nMercurial, Darcs, Monotone use for binary deltas, for diff engine,\nand for textual three-way merge engine. Hmmm... perhaps I'll ask\non #revctrl\n\n-- \nJakub Narebski\nPoland\n"},{"id":"112789","messageId":"86iqkllw0c.fsf@broadpark.no","threadId":"19082","inReplyTo":"20090430185244.GR23604@spearce.org","subject":"Re: Why Git is so fast","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-04-30T20:36:03Z","receivedAt":"2009-04-30T20:36:03Z","isPatch":false,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"* \"Shawn O. Pearce\" writes:\n \n| - Avoid allocating byte[] for SHA-1s, instead we convert to 5 ints,\n| which can be inlined into an object allocation.\n\n What to pepole think about doing something simmilar in C GIT?\n\n That is, convert the current internal representation of the SHA-1 from\n \"unsigned char sha1[20]\" to \"unsigned long sha1[5]\"?\n\n Ok, I currently see 2 problems with it:\n\n 1) Will the type \"unsigned long\" always be unsigned 32 bit on all\n platforms on all computers? do we need an \"unit_32_t\" thing?\n\n 2) Can we get in truble because of differences between litle- and\n big-endian machines?\n\n And, simmilar I can see or guess the following would be positive with\n this change:\n\n 3) From a SHA1 library I worked with some time ago, I noticed that\n it internaly used the type \"unsigned long arr[5]\", so it can\n mabye be possible to get some shurtcuts or maybe speedups here,\n if we want to do it.\n\n 4) The \"static inline void hashcpy(....)\" in cache.h could then\n maybe be written like this:\n\n static inline void hashcpy(unsigned long sha_dst[5], const unsigned long sha_src[5])\n {\n sha_dst[0] = sha_src[0];\n sha_dst[1] = sha_src[1];\n sha_dst[2] = sha_src[2];\n sha_dst[3] = sha_src[3];\n sha_dst[4] = sha_src[4];\n }\n\n And hopefully will be compiled to just 5 store/more\n instructions, or at least hopefully be faster than the currently\n memcpy() call. But mabye we get more compiled instructions compared\n to a single call to memcpy()?\n\n 5) Simmilar as 4) for the other SHA1 realted hash functions nearby\n hashcpy() in cache.h\n\n OK, just some thought's. Sorry if this allready has been discussed\n but could not find something abouth it after a simple google search.\n\n -- kjetil\n"},{"id":"112790","messageId":"20090430204033.GV23604@spearce.org","threadId":"19082","inReplyTo":"86iqkllw0c.fsf@broadpark.no","subject":"Re: Why Git is so fast","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-04-30T20:40:33Z","receivedAt":"2009-04-30T20:40:33Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Kjetil Barvik wrote:\n> * \"Shawn O. Pearce\" writes:\n> \n> | - Avoid allocating byte[] for SHA-1s, instead we convert to 5 ints,\n> | which can be inlined into an object allocation.\n> \n> What to pepole think about doing something simmilar in C GIT?\n> \n> That is, convert the current internal representation of the SHA-1 from\n> \"unsigned char sha1[20]\" to \"unsigned long sha1[5]\"?\n\nIts not worth the code churn.\n \n> Ok, I currently see 2 problems with it:\n> \n> 1) Will the type \"unsigned long\" always be unsigned 32 bit on all\n> platforms on all computers? do we need an \"unit_32_t\" thing?\n\nYea, \"unsigned long\" isn't always 32 bits. So we'd need to use\nuint32_t. Which we already use elsewhere, but still.\n \n> 2) Can we get in truble because of differences between litle- and\n> big-endian machines?\n\nYes, especially if compare was implemented using native uint32_t\ncompare and the processor was little-endian.\n\n> 4) The \"static inline void hashcpy(....)\" in cache.h could then\n> maybe be written like this:\n\nIts already done as \"memcpy(a, b, 20)\" which most compilers will\ninline and probably reduce to 5 word moves anyway. That's why\nhashcpy() itself is inline.\n \n-- \nShawn.\n"},{"id":"112791","messageId":"8663gllt88.fsf@broadpark.no","threadId":"19082","inReplyTo":"20090430204033.GV23604@spearce.org","subject":"Re: Why Git is so fast","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-04-30T21:36:07Z","receivedAt":"2009-04-30T21:36:07Z","isPatch":false,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"* \"Shawn O. Pearce\" writes:\n|> 4) The \"static inline void hashcpy(....)\" in cache.h could then\n|> maybe be written like this:\n|\n| Its already done as \"memcpy(a, b, 20)\" which most compilers will\n| inline and probably reduce to 5 word moves anyway. That's why\n| hashcpy() itself is inline.\n\n But would the compiler be able to trust that the hashcpy() is always\n called with correct word alignment on variables a and b?\n\n I made a test and compiled git with:\n\n make USE_NSEC=1 CFLAGS=\"-march=core2 -mtune=core2 -O2 -g2 -fno-stack-protector\" clean all\n\n compiler: gcc (Gentoo 4.3.3-r2 p1.1, pie-10.1.5) 4.3.3\n CPU: Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz GenuineIntel\n\n Then used gdb to get the following:\n\n(gdb) disassemble write_sha1_file\nDump of assembler code for function write_sha1_file:\n0x080e3830 :\tpush %ebp\n0x080e3831 :\tmov %esp,%ebp\n0x080e3833 :\tsub $0x58,%esp\n0x080e3836 :\tlea -0x10(%ebp),%eax\n0x080e3839 :\tmov %ebx,-0xc(%ebp)\n0x080e383c :\tmov %esi,-0x8(%ebp)\n0x080e383f :\tmov %edi,-0x4(%ebp)\n0x080e3842 :\tmov 0x14(%ebp),%ebx\n0x080e3845 :\tmov %eax,0x8(%esp)\n0x080e3849 :\tlea -0x44(%ebp),%edi\n0x080e384c :\tlea -0x24(%ebp),%esi\n0x080e384f :\tmov %edi,0x4(%esp)\n0x080e3853 :\tmov %esi,(%esp)\n0x080e3856 :\tmov 0x10(%ebp),%ecx\n0x080e3859 :\tmov 0xc(%ebp),%edx\n0x080e385c :\tmov 0x8(%ebp),%eax\n0x080e385f :\tcall 0x80e0350 \n0x080e3864 :\ttest %ebx,%ebx\n0x080e3866 :\tje 0x80e3885 \n\n0x080e3868 :\tmov -0x24(%ebp),%eax\n0x080e386b :\tmov %eax,(%ebx)\n0x080e386d :\tmov -0x20(%ebp),%eax\n0x080e3870 :\tmov %eax,0x4(%ebx)\n0x080e3873 :\tmov -0x1c(%ebp),%eax\n0x080e3876 :\tmov %eax,0x8(%ebx)\n0x080e3879 :\tmov -0x18(%ebp),%eax\n0x080e387c :\tmov %eax,0xc(%ebx)\n0x080e387f :\tmov -0x14(%ebp),%eax\n0x080e3882 :\tmov %eax,0x10(%ebx)\n\n I admit that I am not particular familar with intel machine\n instructions, but I guess that the above 10 mov instructions is the\n result for the compiled inline hashcpy() in the write_sha1_file()\n function in sha1_file.c\n\n Question: would it be possible for the compiler to compile it down to\n just 5 mov instructions if we had used unsigned 32 bits type? Or is\n this the best we can reasonable hope for inside the write_sha1_file()\n function?\n\n I checked 3 other output of \"disassemble function_foo\", and it seems\n that those 3 functions I checked got 10 mov instructions for the\n inline hashcpy(), as far as I can tell.\n\n0x080e3885 :\tmov %esi,(%esp)\n0x080e3888 :\tcall 0x80e3800 \n0x080e388d :\txor %edx,%edx\n0x080e388f :\ttest %eax,%eax\n0x080e3891 :\tjne 0x80e38b6 \n0x080e3893 :\tmov 0xc(%ebp),%eax\n0x080e3896 :\tmov %edi,%edx\n0x080e3898 :\tmov %eax,0x4(%esp)\n0x080e389c :\tmov -0x10(%ebp),%ecx\n0x080e389f :\tmov 0x8(%ebp),%eax\n0x080e38a2 :\tmovl $0x0,0x8(%esp)\n0x080e38aa :\tmov %eax,(%esp)\n0x080e38ad :\tmov %esi,%eax\n0x080e38af :\tcall 0x80e1e40 \n0x080e38b4 :\tmov %eax,%edx\n0x080e38b6 :\tmov %edx,%eax\n0x080e38b8 :\tmov -0xc(%ebp),%ebx\n0x080e38bb :\tmov -0x8(%ebp),%esi\n0x080e38be :\tmov -0x4(%ebp),%edi\n0x080e38c1 :\tleave \n0x080e38c2 :\tret \nEnd of assembler dump.\n(gdb) \n\n So, maybe the compiler is doing the right thing after all?\n\n -- kjetil\n"},{"id":"112797","messageId":"f488382f0904301723i37ef03d9w4e93848e603ed28b@mail.gmail.com","threadId":"19082","inReplyTo":"8663gllt88.fsf@broadpark.no","subject":"Re: Why Git is so fast","fromName":"Steven Noonan","fromEmail":"steven@uplinklabs.net","sentAt":"2009-05-01T00:23:57Z","receivedAt":"2009-05-01T00:23:57Z","isPatch":false,"sender":{"key":"steven@uplinklabs.net","avatar":"https://gravatar.com/avatar/b0cd397a10638433f76e084531aa0af3bef85f8fdb59b1ebe2ddaf168cd100e9?d=mp&s=160"},"body":"On Thu, Apr 30, 2009 at 2:36 PM, Kjetil Barvik wrote:\n> * \"Shawn O. Pearce\" writes:\n> |>      4) The \"static inline void hashcpy(....)\" in cache.h could then\n> |>         maybe be written like this:\n> |\n> | Its already done as \"memcpy(a, b, 20)\" which most compilers will\n> | inline and probably reduce to 5 word moves anyway.  That's why\n> | hashcpy() itself is inline.\n>\n>  But would the compiler be able to trust that the hashcpy() is always\n>  called with correct word alignment on variables a and b?\n>\n>  I made a test and compiled git with:\n>\n>    make USE_NSEC=1 CFLAGS=\"-march=core2 -mtune=core2 -O2 -g2 -fno-stack-protector\" clean all\n>\n>  compiler: gcc (Gentoo 4.3.3-r2 p1.1, pie-10.1.5) 4.3.3\n>  CPU: Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz GenuineIntel\n>\n>  Then used gdb to get the following:\n>\n> (gdb) disassemble write_sha1_file\n> Dump of assembler code for function write_sha1_file:\n> 0x080e3830 : push   %ebp\n> 0x080e3831 : mov    %esp,%ebp\n> 0x080e3833 : sub    $0x58,%esp\n> 0x080e3836 : lea    -0x10(%ebp),%eax\n> 0x080e3839 : mov    %ebx,-0xc(%ebp)\n> 0x080e383c :        mov    %esi,-0x8(%ebp)\n> 0x080e383f :        mov    %edi,-0x4(%ebp)\n> 0x080e3842 :        mov    0x14(%ebp),%ebx\n> 0x080e3845 :        mov    %eax,0x8(%esp)\n> 0x080e3849 :        lea    -0x44(%ebp),%edi\n> 0x080e384c :        lea    -0x24(%ebp),%esi\n> 0x080e384f :        mov    %edi,0x4(%esp)\n> 0x080e3853 :        mov    %esi,(%esp)\n> 0x080e3856 :        mov    0x10(%ebp),%ecx\n> 0x080e3859 :        mov    0xc(%ebp),%edx\n> 0x080e385c :        mov    0x8(%ebp),%eax\n> 0x080e385f :        call   0x80e0350 \n> 0x080e3864 :        test   %ebx,%ebx\n> 0x080e3866 :        je     0x80e3885 \n>\n> 0x080e3868 :        mov    -0x24(%ebp),%eax\n> 0x080e386b :        mov    %eax,(%ebx)\n> 0x080e386d :        mov    -0x20(%ebp),%eax\n> 0x080e3870 :        mov    %eax,0x4(%ebx)\n> 0x080e3873 :        mov    -0x1c(%ebp),%eax\n> 0x080e3876 :        mov    %eax,0x8(%ebx)\n> 0x080e3879 :        mov    -0x18(%ebp),%eax\n> 0x080e387c :        mov    %eax,0xc(%ebx)\n> 0x080e387f :        mov    -0x14(%ebp),%eax\n> 0x080e3882 :        mov    %eax,0x10(%ebx)\n>\n>  I admit that I am not particular familar with intel machine\n>  instructions, but I guess that the above 10 mov instructions is the\n>  result for the compiled inline hashcpy() in the write_sha1_file()\n>  function in sha1_file.c\n>\n>  Question: would it be possible for the compiler to compile it down to\n>  just 5 mov instructions if we had used unsigned 32 bits type?  Or is\n>  this the best we can reasonable hope for inside the write_sha1_file()\n>  function?\n>\n>  I checked 3 other output of \"disassemble function_foo\", and it seems\n>  that those 3 functions I checked got 10 mov instructions for the\n>  inline hashcpy(), as far as I can tell.\n>\n> 0x080e3885 :        mov    %esi,(%esp)\n> 0x080e3888 :        call   0x80e3800 \n> 0x080e388d :        xor    %edx,%edx\n> 0x080e388f :        test   %eax,%eax\n> 0x080e3891 :        jne    0x80e38b6 \n> 0x080e3893 :        mov    0xc(%ebp),%eax\n> 0x080e3896 :       mov    %edi,%edx\n> 0x080e3898 :       mov    %eax,0x4(%esp)\n> 0x080e389c :       mov    -0x10(%ebp),%ecx\n> 0x080e389f :       mov    0x8(%ebp),%eax\n> 0x080e38a2 :       movl   $0x0,0x8(%esp)\n> 0x080e38aa :       mov    %eax,(%esp)\n> 0x080e38ad :       mov    %esi,%eax\n> 0x080e38af :       call   0x80e1e40 \n> 0x080e38b4 :       mov    %eax,%edx\n> 0x080e38b6 :       mov    %edx,%eax\n> 0x080e38b8 :       mov    -0xc(%ebp),%ebx\n> 0x080e38bb :       mov    -0x8(%ebp),%esi\n> 0x080e38be :       mov    -0x4(%ebp),%edi\n> 0x080e38c1 :       leave\n> 0x080e38c2 :       ret\n> End of assembler dump.\n> (gdb)\n>\n>  So, maybe the compiler is doing the right thing after all?\n>\n\nWell, I just tested this with GCC myself. I used this segment of code:\n\n #include \n void hashcpy(unsigned char *sha_dst, const unsigned char *sha_src)\n {\n memcpy(sha_dst, sha_src, 20);\n }\n\nI compiled using Apple's GCC 4.0.1 (note that GCC 4.3 and 4.4 vanilla\nyield the same code) with these parameters to get Intel assembly:\n gcc -O2 -arch i386 -march=pentium3 -mtune=pentium3\n-fomit-frame-pointer -fno-strict-aliasing -S test.c\nand these parameters to get the equivalent PowerPC code:\n gcc -O2 -mcpu=G5 -arch ppc -fomit-frame-pointer\n-fno-strict-aliasing -S test.c\n\nIntel code:\n .text\n .align 4,0x90\n.globl _hashcpy\n_hashcpy:\n subl $12, %esp\n movl 20(%esp), %edx\n movl 16(%esp), %ecx\n movl (%edx), %eax\n movl %eax, (%ecx)\n movl 4(%edx), %eax\n movl %eax, 4(%ecx)\n movl 8(%edx), %eax\n movl %eax, 8(%ecx)\n movl 12(%edx), %eax\n movl %eax, 12(%ecx)\n movl 16(%edx), %eax\n movl %eax, 16(%ecx)\n addl $12, %esp\n ret\n .subsections_via_symbols\n\n\nand the PowerPC code:\n\n .section __TEXT,__text,regular,pure_instructions\n .section __TEXT,__picsymbolstub1,symbol_stubs,pure_instructions,32\n .machine ppc970\n .text\n .align 2\n .p2align 4,,15\n .globl _hashcpy\n_hashcpy:\n lwz r0,0(r4)\n lwz r2,4(r4)\n lwz r9,8(r4)\n lwz r11,12(r4)\n stw r0,0(r3)\n stw r2,4(r3)\n stw r9,8(r3)\n stw r11,12(r3)\n lwz r0,16(r4)\n stw r0,16(r3)\n blr\n .subsections_via_symbols\n\n\nSo it does look like GCC does what it should and it inlines the memcpy.\n\nA bit off topic, but the results are rather interesting to me, and I\nthink I see a weakness in how GCC is doing this on Intel. Someone\nplease correct me if I'm wrong, but the PowerPC code seems much better\nbecause it can yield very high instruction-level parallelism. It does\n5 loads and then 5 stores, using 4 registers for temporary storage and\n2 registers for pointers.\n\nI realize the Intel x86 architecture is quite constrained in that it\nhas so few general purpose registers, but there has to be better code\nthan what GCC emitted above. It seems like the processor would stall\nbecause of the quantity of sequential inter-dependent instructions\nthat can't be done in parallel (mov to memory that depends on a mov to\neax, etc).\n\nI suppose the code might not be stalling if it's using the maximum\nnumber of registers and doing as many memory accesses that it can per\nclock, but based on known details about the architecture, does it seem\nto be doing that?\n\n- Steven\n"},{"id":"112798","messageId":"885649360904301825i40b6b7b7o9874ee3df2809a21@mail.gmail.com","threadId":"19082","inReplyTo":"f488382f0904301723i37ef03d9w4e93848e603ed28b@mail.gmail.com","subject":"Re: Why Git is so fast","fromName":"James Pickens","fromEmail":"jepicken@gmail.com","sentAt":"2009-05-01T01:25:21Z","receivedAt":"2009-05-01T01:25:21Z","isPatch":false,"sender":{"key":"jepicken@gmail.com","avatar":null},"body":"On Thu, Apr 30, 2009, Steven Noonan wrote:\n> A bit off topic, but the results are rather interesting to me, and I\n> think I see a weakness in how GCC is doing this on Intel. Someone\n> please correct me if I'm wrong, but the PowerPC code seems much better\n> because it can yield very high instruction-level parallelism. It does\n> 5 loads and then 5 stores, using 4 registers for temporary storage and\n> 2 registers for pointers.\n>\n> I realize the Intel x86 architecture is quite constrained in that it\n> has so few general purpose registers, but there has to be better code\n> than what GCC emitted above. It seems like the processor would stall\n> because of the quantity of sequential inter-dependent instructions\n> that can't be done in parallel (mov to memory that depends on a mov to\n> eax, etc).\n\nThere aren't any unnecessary dependencies. Take this sequence:\n\n1: movl (%edx), %eax\n2: movl %eax, (%ecx)\n3: movl 4(%edx), %eax\n4: movl %eax, 4(%ecx)\n\nThere are two unavoidable dependencies - #2 depends on #1, and #4\ndepends on #3. #3 does not depend on #2, even though they both\nuse %eax, because #3 is a write to %eax. So whatever was in %eax\nbefore #3 is irrelevant. The processor knows this and will use\nregister renaming to execute #1 and #3 in parallel, and #2 and #4\nin parallel.\n\nJames\n"},{"id":"112801","messageId":"20090501052434.GA4750@dpotapov.dyndns.org","threadId":"19082","inReplyTo":"86iqkllw0c.fsf@broadpark.no","subject":"Re: Why Git is so fast","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-05-01T05:24:34Z","receivedAt":"2009-05-01T05:24:34Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Thu, Apr 30, 2009 at 10:36:03PM +0200, Kjetil Barvik wrote:\n> 4) The \"static inline void hashcpy(....)\" in cache.h could then\n> maybe be written like this:\n> \n> static inline void hashcpy(unsigned long sha_dst[5], const unsigned long sha_src[5])\n> {\n> sha_dst[0] = sha_src[0];\n> sha_dst[1] = sha_src[1];\n> sha_dst[2] = sha_src[2];\n> sha_dst[3] = sha_src[3];\n> sha_dst[4] = sha_src[4];\n> }\n> \n> And hopefully will be compiled to just 5 store/more\n> instructions, or at least hopefully be faster than the currently\n> memcpy() call. But mabye we get more compiled instructions compared\n> to a single call to memcpy()?\n\nGood compilers can inline memcpy and should produce more efficient code\nfor the target architecture, which can be faster than manually written.\nOn x86_64, memcpy() requires only 3 load/store operations to copy SHA-1\nwhile the above code requires 5 operations.\n\nDmitry\n"},{"id":"112812","messageId":"864ow59o53.fsf@broadpark.no","threadId":"19082","inReplyTo":"f488382f0904301723i37ef03d9w4e93848e603ed28b@mail.gmail.com","subject":"Re: Why Git is so fast","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-05-01T09:19:04Z","receivedAt":"2009-05-01T09:19:04Z","isPatch":false,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"* Steven Noonan writes:\n| On Thu, Apr 30, 2009 at 2:36 PM, Kjetil Barvik wrote:\n|> * \"Shawn O. Pearce\" writes:\n|> |> 4) The \"static inline void hashcpy(....)\" in cache.h could then\n|> |> maybe be written like this:\n|> |\n|> | Its already done as \"memcpy(a, b, 20)\" which most compilers will\n|> | inline and probably reduce to 5 word moves anyway. That's why\n|> | hashcpy() itself is inline.\n|>\n|> But would the compiler be able to trust that the hashcpy() is always\n|> called with correct word alignment on variables a and b?\n\n \n\n| Well, I just tested this with GCC myself. I used this segment of code:\n|\n| #include \n| void hashcpy(unsigned char *sha_dst, const unsigned char *sha_src)\n| {\n| memcpy(sha_dst, sha_src, 20);\n| }\n\n OK, here is a smal test, which maybe shows at least one difference\n between using \"unsigned char sha1[20]\" and \"unsigned long sha1[5]\".\n Given the following file, memcpy_test.c:\n\n#include \nextern void hashcpy_uchar(unsigned char *sha_dst, const unsigned char *sha_src);\nvoid hashcpy_uchar(unsigned char *sha_dst, const unsigned char *sha_src)\n{\n memcpy(sha_dst, sha_src, 20);\n}\nextern void hashcpy_ulong(unsigned long *sha_dst, const unsigned long *sha_src);\nvoid hashcpy_ulong(unsigned long *sha_dst, const unsigned long *sha_src)\n{\n memcpy(sha_dst, sha_src, 5);\n}\n\n And, compiled with the following:\n\n gcc -O2 -mtune=core2 -march=core2 -S -fomit-frame-pointer memcpy_test.c\n\n It produced the following memcpy_test.s file:\n\n .file \"memcpy_test.c\"\n .text\n .p2align 4,,15\n.globl hashcpy_ulong\n .type hashcpy_ulong, @function\nhashcpy_ulong:\n movl 8(%esp), %edx\n movl 4(%esp), %ecx\n movl (%edx), %eax\n movl %eax, (%ecx)\n movzbl 4(%edx), %eax\n movb %al, 4(%ecx)\n ret\n .size hashcpy_ulong, .-hashcpy_ulong\n .p2align 4,,15\n.globl hashcpy_uchar\n .type hashcpy_uchar, @function\nhashcpy_uchar:\n movl 8(%esp), %edx\n movl 4(%esp), %ecx\n movl (%edx), %eax\n movl %eax, (%ecx)\n movl 4(%edx), %eax\n movl %eax, 4(%ecx)\n movl 8(%edx), %eax\n movl %eax, 8(%ecx)\n movl 12(%edx), %eax\n movl %eax, 12(%ecx)\n movl 16(%edx), %eax\n movl %eax, 16(%ecx)\n ret\n .size hashcpy_uchar, .-hashcpy_uchar\n .ident \"GCC: (Gentoo 4.3.3-r2 p1.1, pie-10.1.5) 4.3.3\"\n .section .note.GNU-stack,\"\",@progbits\n\n So, the \"unsigned long\" type hashcpy() used 7 instructions, compared\n to 13 for the \"unsigned char\" type hascpy().\n\n Would I guess correct if the hashcpy_ulong() function will also use\n less CPU cycles, and then would be faster than hashcpy_uchar()?\n\n -- kjetil\n"},{"id":"112813","messageId":"20090501093427.GA13264@glandium.org","threadId":"19082","inReplyTo":"864ow59o53.fsf@broadpark.no","subject":"Re: Why Git is so fast","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2009-05-01T09:34:27Z","receivedAt":"2009-05-01T09:34:27Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, May 01, 2009 at 11:19:04AM +0200, Kjetil Barvik wrote:\n> * Steven Noonan writes:\n> | On Thu, Apr 30, 2009 at 2:36 PM, Kjetil Barvik wrote:\n> |> * \"Shawn O. Pearce\" writes:\n> |> |> 4) The \"static inline void hashcpy(....)\" in cache.h could then\n> |> |> maybe be written like this:\n> |> |\n> |> | Its already done as \"memcpy(a, b, 20)\" which most compilers will\n> |> | inline and probably reduce to 5 word moves anyway. That's why\n> |> | hashcpy() itself is inline.\n> |>\n> |> But would the compiler be able to trust that the hashcpy() is always\n> |> called with correct word alignment on variables a and b?\n> \n> \n> \n> | Well, I just tested this with GCC myself. I used this segment of code:\n> |\n> | #include \n> | void hashcpy(unsigned char *sha_dst, const unsigned char *sha_src)\n> | {\n> | memcpy(sha_dst, sha_src, 20);\n> | }\n> \n> OK, here is a smal test, which maybe shows at least one difference\n> between using \"unsigned char sha1[20]\" and \"unsigned long sha1[5]\".\n> Given the following file, memcpy_test.c:\n> \n> #include \n> extern void hashcpy_uchar(unsigned char *sha_dst, const unsigned char *sha_src);\n> void hashcpy_uchar(unsigned char *sha_dst, const unsigned char *sha_src)\n> {\n> memcpy(sha_dst, sha_src, 20);\n> }\n> extern void hashcpy_ulong(unsigned long *sha_dst, const unsigned long *sha_src);\n> void hashcpy_ulong(unsigned long *sha_dst, const unsigned long *sha_src)\n> {\n> memcpy(sha_dst, sha_src, 5);\n> }\n> \n> And, compiled with the following:\n> \n> gcc -O2 -mtune=core2 -march=core2 -S -fomit-frame-pointer memcpy_test.c\n> \n> It produced the following memcpy_test.s file:\n> \n> .file \"memcpy_test.c\"\n> .text\n> .p2align 4,,15\n> .globl hashcpy_ulong\n> .type hashcpy_ulong, @function\n> hashcpy_ulong:\n> movl 8(%esp), %edx\n> movl 4(%esp), %ecx\n> movl (%edx), %eax\n> movl %eax, (%ecx)\n> movzbl 4(%edx), %eax\n> movb %al, 4(%ecx)\n> ret\n> .size hashcpy_ulong, .-hashcpy_ulong\n> .p2align 4,,15\n> .globl hashcpy_uchar\n> .type hashcpy_uchar, @function\n> hashcpy_uchar:\n> movl 8(%esp), %edx\n> movl 4(%esp), %ecx\n> movl (%edx), %eax\n> movl %eax, (%ecx)\n> movl 4(%edx), %eax\n> movl %eax, 4(%ecx)\n> movl 8(%edx), %eax\n> movl %eax, 8(%ecx)\n> movl 12(%edx), %eax\n> movl %eax, 12(%ecx)\n> movl 16(%edx), %eax\n> movl %eax, 16(%ecx)\n> ret\n> .size hashcpy_uchar, .-hashcpy_uchar\n> .ident \"GCC: (Gentoo 4.3.3-r2 p1.1, pie-10.1.5) 4.3.3\"\n> .section .note.GNU-stack,\"\",@progbits\n> \n> So, the \"unsigned long\" type hashcpy() used 7 instructions, compared\n> to 13 for the \"unsigned char\" type hascpy().\n\nBut your \"unsigned long\" version only copies 5 bytes...\n\nMike\n"},{"id":"112815","messageId":"86zldx88ia.fsf@broadpark.no","threadId":"19082","inReplyTo":"20090501093427.GA13264@glandium.org","subject":"Re: Why Git is so fast","fromName":"Kjetil Barvik","fromEmail":"barvik@broadpark.no","sentAt":"2009-05-01T09:42:05Z","receivedAt":"2009-05-01T09:42:05Z","isPatch":false,"sender":{"key":"barvik@broadpark.no","avatar":null},"body":"* Mike Hommey writes:\n \n| But your \"unsigned long\" version only copies 5 bytes...\n\n Yes, that is true... OK, same result for hashcpy_uchar() and\n hashcpy_ulong() when corrected for this.\n\n --kjetil, with a brown paper bag\n"},{"id":"112814","messageId":"20090501094221.GB13264@glandium.org","threadId":"19082","inReplyTo":"20090501052434.GA4750@dpotapov.dyndns.org","subject":"Re: Why Git is so fast","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2009-05-01T09:42:21Z","receivedAt":"2009-05-01T09:42:21Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Fri, May 01, 2009 at 09:24:34AM +0400, Dmitry Potapov wrote:\n> On Thu, Apr 30, 2009 at 10:36:03PM +0200, Kjetil Barvik wrote:\n> > 4) The \"static inline void hashcpy(....)\" in cache.h could then\n> > maybe be written like this:\n> > \n> > static inline void hashcpy(unsigned long sha_dst[5], const unsigned long sha_src[5])\n> > {\n> > sha_dst[0] = sha_src[0];\n> > sha_dst[1] = sha_src[1];\n> > sha_dst[2] = sha_src[2];\n> > sha_dst[3] = sha_src[3];\n> > sha_dst[4] = sha_src[4];\n> > }\n> > \n> > And hopefully will be compiled to just 5 store/more\n> > instructions, or at least hopefully be faster than the currently\n> > memcpy() call. But mabye we get more compiled instructions compared\n> > to a single call to memcpy()?\n> \n> Good compilers can inline memcpy and should produce more efficient code\n> for the target architecture, which can be faster than manually written.\n> On x86_64, memcpy() requires only 3 load/store operations to copy SHA-1\n> while the above code requires 5 operations.\n\nI guess, though, that some enforced alignment could help produce\nslightly more efficient code on some architectures (most notably sparc,\nwhich really doesn't like to deal with unaligned words).\n\nMike\n"},{"id":"112819","messageId":"20090501104655.GC4863@dpotapov.dyndns.org","threadId":"19082","inReplyTo":"20090501094221.GB13264@glandium.org","subject":"Re: Why Git is so fast","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-05-01T10:46:55Z","receivedAt":"2009-05-01T10:46:55Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Fri, May 01, 2009 at 11:42:21AM +0200, Mike Hommey wrote:\n> On Fri, May 01, 2009 at 09:24:34AM +0400, Dmitry Potapov wrote:\n> > \n> > Good compilers can inline memcpy and should produce more efficient code\n> > for the target architecture, which can be faster than manually written.\n> > On x86_64, memcpy() requires only 3 load/store operations to copy SHA-1\n> > while the above code requires 5 operations.\n> \n> I guess, though, that some enforced alignment could help produce\n> slightly more efficient code on some architectures (most notably sparc,\n> which really doesn't like to deal with unaligned words).\n\nAgreed. Enforcing good alignment may be useful. My point was that avoiding\nmemcpy with modern compilers is rather pointless or even harmful because the\ncompiler know more about the target architecture than the author of the code.\n\nDmitry\n"},{"id":"112828","messageId":"alpine.LSU.2.00.0905011840260.28199@hermes-2.csi.cam.ac.uk","threadId":"19082","inReplyTo":"8663gllt88.fsf@broadpark.no","subject":"Re: Why Git is so fast","fromName":"Tony Finch","fromEmail":"dot@dotat.at","sentAt":"2009-05-01T17:42:38Z","receivedAt":"2009-05-01T17:42:38Z","isPatch":false,"sender":{"key":"dot@dotat.at","avatar":"https://avatars.githubusercontent.com/u/68429?v=4"},"body":"On Thu, 30 Apr 2009, Kjetil Barvik wrote:\n>\n> I admit that I am not particular familar with intel machine\n> instructions, but I guess that the above 10 mov instructions is the\n> result for the compiled inline hashcpy() in the write_sha1_file()\n> function in sha1_file.c\n>\n> Question: would it be possible for the compiler to compile it down to\n> just 5 mov instructions if we had used unsigned 32 bits type?\n\nNo, because the x86 can't do direct memory-to-memory moves.\n\nTony.\n-- \nf.anthony.n.finch http://dotat.at/\nGERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.\nMODERATE OR GOOD.\n"},{"id":"112830","messageId":"alpine.LFD.2.00.0905011431460.5379@localhost.localdomain","threadId":"19082","inReplyTo":"20090430142244.GA23550@coredump.intra.peff.net","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-01T18:43:49Z","receivedAt":"2009-05-01T18:43:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 30 Apr 2009, Jeff King wrote:\n> \n> Like all generalizations, this is only mostly true. Fast network servers\n> with big caches can outperform disks for some loads.\n\nThat's _very_ few loads.\n\nIt doesn't matter how good a server you have, network filesystems \ninvariably suck.\n\nWhy? It's not that the network or the server sucks - you can easily find \nbeefy NAS setups that have big raids etc and are much faster than most \nlocal disks.\n\nAnd they _still_ suck.\n\nSimple reason: caching. It's a lot easier to cache local filesystems. Even \nmodern networked filesystems (ie NFSv4), that do a pretty good job on a \nfile-per-file basis with delegations etc, and they still tend to suck \nhorribly at metadata.\n\nIn contrast, a workstation with local filesystems and enough memory to \ncache it well will just be a lot nicer.\n\n> So I wouldn't rule out the possibility of a pleasant VCS experience on a\n> network-optimized system backed by beefy servers on a local network.\n\nHey, you can always throw resources at it.\n\nBut no:\n\n> I have never used perforce, but I get the impression that it is more \n> optimized for such a situation.\n\nI doubt it. I suspect git will outperform pretty much anything else in \nthat kind of situation too.\n\nOne thing that git does - and some other VCS's avoid - is to actually \nstat() the whole working tree in order to not need special per-file \"I use \nthis file\" locking semantics. That can in theory make git slower over a \nnetwork filesystem than such (very broken) alternatives.\n\nIf your VCS requires that you mark all files for editing somehow (ie you \ncan't just use your favourite editor or scripting to modify files, but \nhave to use \"p4 edit\" to say that you're going to write to the file, and \nthe file is otherwise read-only), then such a VCS can - by being annoying \nand in your way - do some things faster than git can.\n\nAnd yes, perforce does that (the \"p4 edit\" command is real, and exists).\n\nAnd yes, in theory that can probably mean that perforce doesn't care so \nmuch about the metadata caching problem on network filesystems - because \np4 will maintain some file of its own that contains the metadata.\n\nBut I suspect that the git \"async stat\" (\"core.preloadindex\") thing means \nthat git will kick p4 *ss even on that benchmark, and be a whole lot more \npleasant to use. Even on networked filesystems.\n\n\t\t\tLinus\n"},{"id":"112831","messageId":"20090501190854.GA13770@coredump.intra.peff.net","threadId":"19082","inReplyTo":"alpine.LFD.2.00.0905011431460.5379@localhost.localdomain","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-01T19:08:54Z","receivedAt":"2009-05-01T19:08:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, May 01, 2009 at 02:43:49PM -0400, Linus Torvalds wrote:\n\n> > Like all generalizations, this is only mostly true. Fast network servers\n> > with big caches can outperform disks for some loads.\n> [...]\n> In contrast, a workstation with local filesystems and enough memory to \n> cache it well will just be a lot nicer.\n> [...]\n> > I have never used perforce, but I get the impression that it is more \n> > optimized for such a situation.\n> \n> I doubt it. I suspect git will outperform pretty much anything else in \n> that kind of situation too.\n\nThanks for the analysis; what you said makes sense to me. However, there\nis at least one case of somebody complaining that git doesn't scale as\nwell as perforce for their load:\n\n http://gandolf.homelinux.org/blog/index.php?id=50\n\nPart of his issue is with git-p4 sucking, which it probably does. But\npart of it sounds like he has a gigantic workload (the description of\nwhich sounds silly to me, but I respect the fact that he is probably\ndescribing standard practice among some companies), and that workload is\njust a little too gigantic for the workstations to handle. I.e., by\nthrowing resources at the central server they can avoid throwing as many\nat each workstation.\n\nBut there are so few details it's hard to say whether he's doing\nsomething else wrong or suboptimally. He does mention Windows, which\nIIRC has horrific stat performance.\n\n-Peff\n"},{"id":"112833","messageId":"alpine.DEB.1.10.0905011211080.15782@asgard","threadId":"19082","inReplyTo":"20090501190854.GA13770@coredump.intra.peff.net","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-05-01T19:13:50Z","receivedAt":"2009-05-01T19:13:50Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 1 May 2009, Jeff King wrote:\n\n> On Fri, May 01, 2009 at 02:43:49PM -0400, Linus Torvalds wrote:\n>\n>>> Like all generalizations, this is only mostly true. Fast network servers\n>>> with big caches can outperform disks for some loads.\n>> [...]\n>> In contrast, a workstation with local filesystems and enough memory to\n>> cache it well will just be a lot nicer.\n>> [...]\n>>> I have never used perforce, but I get the impression that it is more\n>>> optimized for such a situation.\n>>\n>> I doubt it. I suspect git will outperform pretty much anything else in\n>> that kind of situation too.\n>\n> Thanks for the analysis; what you said makes sense to me. However, there\n> is at least one case of somebody complaining that git doesn't scale as\n> well as perforce for their load:\n>\n> http://gandolf.homelinux.org/blog/index.php?id=50\n>\n> Part of his issue is with git-p4 sucking, which it probably does. But\n> part of it sounds like he has a gigantic workload (the description of\n> which sounds silly to me, but I respect the fact that he is probably\n> describing standard practice among some companies), and that workload is\n> just a little too gigantic for the workstations to handle. I.e., by\n> throwing resources at the central server they can avoid throwing as many\n> at each workstation.\n>\n> But there are so few details it's hard to say whether he's doing\n> something else wrong or suboptimally. He does mention Windows, which\n> IIRC has horrific stat performance.\n\nthe key thing for his problem is the support for large binary objects. \nthere was discussion here a few weeks ago about ways to handle such things \nwithout trying to pull them into packs. I suspect that solving those sorts \nof issues would go a long way towards closing the gap on this workload.\n\nthere may be issues in doing a clone for repositories that large, I don't \nremember exactly what happens when you have something larger than 4G to \nsend in a clone.\n\nDavid Lang\n"},{"id":"112836","messageId":"alpine.LFD.2.00.0905011522580.6741@xanadu.home","threadId":"19082","inReplyTo":"alpine.DEB.1.10.0905011211080.15782@asgard","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-01T19:32:18Z","receivedAt":"2009-05-01T19:32:18Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 1 May 2009, david@lang.hm wrote:\n\n> the key thing for his problem is the support for large binary objects. there\n> was discussion here a few weeks ago about ways to handle such things without\n> trying to pull them into packs. I suspect that solving those sorts of issues\n> would go a long way towards closing the gap on this workload.\n> \n> there may be issues in doing a clone for repositories that large, I don't\n> remember exactly what happens when you have something larger than 4G to send\n> in a clone.\n\nIf you have files larger than 4G then you definitively need a 64-bit \nmachine with plenty of RAM for git to at least be able to cope at the \nmoment.\n\nThat should be easy to add a config option to determine how big is a big \nfile, and store those big files directly in a pack of their own instead \nof a loose object (for easy pack reuse during a further repack), and \nnever attempt to deltify them, etc. etc. At which point git will handle \nbig files just fine even on a 32-bit machine but it won't do more than \ncopying them in and out, and possibly deflating/inflating them while at \nit, but nothing fancier.\n\n\nNicolas\n"},{"id":"112840","messageId":"alpine.LNX.2.00.0905011618370.2147@iabervon.org","threadId":"19082","inReplyTo":"20090501190854.GA13770@coredump.intra.peff.net","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-05-01T21:17:31Z","receivedAt":"2009-05-01T21:17:31Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Fri, 1 May 2009, Jeff King wrote:\n\n> On Fri, May 01, 2009 at 02:43:49PM -0400, Linus Torvalds wrote:\n> \n> > > Like all generalizations, this is only mostly true. Fast network servers\n> > > with big caches can outperform disks for some loads.\n> > [...]\n> > In contrast, a workstation with local filesystems and enough memory to \n> > cache it well will just be a lot nicer.\n> > [...]\n> > > I have never used perforce, but I get the impression that it is more \n> > > optimized for such a situation.\n> > \n> > I doubt it. I suspect git will outperform pretty much anything else in \n> > that kind of situation too.\n> \n> Thanks for the analysis; what you said makes sense to me. However, there\n> is at least one case of somebody complaining that git doesn't scale as\n> well as perforce for their load:\n> \n> http://gandolf.homelinux.org/blog/index.php?id=50\n> \n> Part of his issue is with git-p4 sucking, which it probably does. But\n> part of it sounds like he has a gigantic workload (the description of\n> which sounds silly to me, but I respect the fact that he is probably\n> describing standard practice among some companies), and that workload is\n> just a little too gigantic for the workstations to handle. I.e., by\n> throwing resources at the central server they can avoid throwing as many\n> at each workstation.\n\nI think his problem is that he's trying to replace his p4 repository with \na git repository, which is a bit like trying to download github, rather \nthan a project from github. Perforce is good at dealing with the case \nwhere people check in a vast quantity of junk that you don't check out.\n\nThat is, you can back up your workstation into Perforce, and it won't \naffect anyone's performance if you use a path that's not in the range that \nanybody else checks out. And people actually do that. And Perforce doesn't \nmake a distinction between different projects and different branches of \nthe same project and different subdirectories of a branch of the same \nproject, so it's impossible to tease apart except by company policy.\n\nGit doesn't scale in that it can't do the extremely narrow checkouts you \nneed if your repository root directory contains thousands of complete \nunrelated projects with each branch of each project getting a \nsubdirectory. On the other hand, it does a great job when the data is \nalready partitioned into useful repositories.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"112841","messageId":"alpine.LFD.2.00.0905011420580.5379@localhost.localdomain","threadId":"19082","inReplyTo":"20090501190854.GA13770@coredump.intra.peff.net","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-01T21:37:28Z","receivedAt":"2009-05-01T21:37:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 1 May 2009, Jeff King wrote:\n> \n> Thanks for the analysis; what you said makes sense to me. However, there\n> is at least one case of somebody complaining that git doesn't scale as\n> well as perforce for their load:\n\nSo we definitely do have scaling issues, there's no question about that. I \njust don't think they are about enterprise network servers vs the more \nworkstation-oriented OSS world..\n\nI think they're likely about the whole git mentality of looking at the big \npicture, and then getting swamped by just how _huge_ that picture can be \nif somebody just put the whole world in a single repository..\n\nWith perforce, repository maintenance is such a central issue that the \nwhole p4 mentality seems to _encourage_ everybody to put everything into \nbasically one single p4 repository. And afaik, p4 basically works mostly \nlike CVS, ie it really ends up being pretty much oriented to a \"one file \nat a time\" model.\n\nWhich is nice in that you can have a million files, and then only check \nout a few of them - you'll never even _see_ the impact of the other \n999,995 files.\n\nAnd git obviously doesn't have that kind of model at all. Git \nfundamnetally never really looks at less than the whole repo. Even if you \nlimit things a bit (ie check out just a portion, or have the history go \nback just a bit), git ends up still always caring about the whole thing, \nand carrying the knowledge around.\n\nSo git scales really badly if you force it to look at everything as one \n_huge_ repository. I don't think that part is really fixable, although we \ncan probably improve on it.\n\nAnd yes, then there's the \"big file\" issues. I really don't know what to \ndo about huge files. We suck at them, I know. There are work-arounds (like \nnot deltaing big objects at all), but they aren't necessarily that great \neither.\n\nI bet we could probably improve git large-file behavior for many common \ncases. Do we have a good test-case of some particular suckiness that is \nactually relevant enough that people might decide to look at it (and by \n\"people\", I do mean myself too - but I'd need to be somewhat motivated by \nit. A usage case that we suck at and that is available and relevant).\n\n\t\t\tLinus\n"},{"id":"112845","messageId":"alpine.DEB.1.10.0905011503400.15782@asgard","threadId":"19082","inReplyTo":"alpine.LFD.2.00.0905011420580.5379@localhost.localdomain","subject":"Re: Why Git is so fast (was: Re: Eric Sink's blog - notes on git, dscms and a \"whole product\" approach)","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-05-01T22:11:03Z","receivedAt":"2009-05-01T22:11:03Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Fri, 1 May 2009, Linus Torvalds wrote:\n\n> I bet we could probably improve git large-file behavior for many common\n> cases. Do we have a good test-case of some particular suckiness that is\n> actually relevant enough that people might decide to look at it (and by\n> \"people\", I do mean myself too - but I'd need to be somewhat motivated by\n> it. A usage case that we suck at and that is available and relevant).\n\nI think that a sane use case that would make sense to people is based on \nthe 'game developer' example\n\nthey have source code, but they also have large images (and sometimes \nmovie clips), where a particular release of the game needs a particular \nset of the images. during development you may change images frequently \n(although most changesets probably only change a few, if any of the \nimages)\n\nthe images can be large (movies can be very large), and since they are \nalready compressed they don't diff or compress well.\n\nDavid Lang\n"},{"id":"112963","messageId":"49FEA0F5.2050800@op5.se","threadId":"19082","inReplyTo":"81b0412b0904301216j7ef73870y775cf6d89b5aa71e@mail.gmail.com","subject":"Re: Why Git is so fast","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-04T08:01:57Z","receivedAt":"2009-05-04T08:01:57Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Alex Riesen wrote:\n> 2009/4/30 Nicolas Pitre :\n>> Yet, this point is misleading because when people gives to Git the\n>> reputation of being faster, this is certainly from comparison of\n>> operations performed on the same source tree. Who cares about scenarios\n>> for which the tool was not designed? Those \"enterprise configuration\n>> management repositories\" are not what Git was designed for indeed, but\n> \n> Especially when no sane developer will put in his repository the toolchain\n> (pre-compiled. For all supported platforms!), all the supporting tools\n> (like grep,\n> find, etc.Pre-compiled _and_ source), the in-house framework (pre-compiled\n> and source, again), firmware (pre-compiled and put in the repository weekly),\n> and operating system code (pre-compiled, with firmware-specific drivers,\n> updated, you guessed it, weekly), and well, there is the project itself (Java or\n> C++, and documentation in .doc and .xls)...\n\nWell, git could actually handle that just fine if the toolchain was in a\nsubmodule or even in a separate repository that developers never had to\nworry about. Then you'd design a little tool that said \"re-create build 8149\"\nand it would pull the tools used to do that, and the code and the artwork,\nand then set to work. It'd be an overnight (or over-weekend) job, but no\nman-hours would be spent on it. That's how I'd do it anyways, probably\nwith the \"build\" repository as a master repo with \"tools\", \"code\" and\n\"artwork\" as submodules to it.\n\n> Now, what kind of self-hating idiot will design a system for that kind of abuse?\n\nNoone, naturally, but one might design a system where each folder\nin the repository root is considered a repository in its own right,\nand then get that more or less for free.\n\nThe problem with git for such scenarios is that you have to think\n*before* creating the repository, or play silly buggers when importing\nwhich makes it hard to see how the pieces fit together afterwards.\n\nA tool that could take a repository from a different scm, create a\nmaster repository and several submodule repositories from it would\nprobably solve many of the issues gaming companies have if they want\nto switch to using git. Not least because it would open their eyes\nto how that sort of separation can be done in git, and why it's\nuseful. The binary repos can then turn off delta-compression (and\nzlib compression) for all its blobs using a .gitattributes file,\nand things would be several orders of magnitudes faster.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"}]}