{"thread":{"id":"17510","subject":"Comments on \"Understanding Version Control\" by Eric S. Raymond","startedAt":"2009-02-02T18:48:53Z","lastAt":"2009-02-10T01:20:06Z","messageCount":21,"participants":["Jakub Narebski","Theodore Tso","Eric S. Raymond","Junio C Hamano","Miles Bader"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"102879","messageId":"200902021948.54700.jnareb@gmail.com","threadId":"17510","inReplyTo":null,"subject":"Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-02T18:48:53Z","receivedAt":"2009-02-02T18:48:53Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Some time ago I have found \"Understanding Version-Control Systems\"\n(http://www.catb.org/esr/writings/version-control/version-control.html)\npaper (in draft version) by Eric S. Raymond, via Joel Spolsky\nrecommendation in 'Podcast #37' entry in Stackoverflow Blog\n(http://blog.stackoverflow.com/2009/01/podcast-37/).\n\nThis is a very nice _survey paper_, currently in development, so we\nnow have time for comments, corrections and suggestions.  It is\ncurrently on hiatus (it seems that last changes are from the Jan 2008,\nslightly more than a year ago), but it is not abandoned.  ESR plans to\nget back to it after Wesnoth and gpsd do their upcoming releases.\n\nIn my opinion the most important issue is concentrating on \"container\nidentity\" instead of on the underlying issue of renames in version\ncontrol, which includes intelligent, rename-aware merge; talk about\nissues and not about possible solution.  I will concentrate on this\nissue for now, and leave for example issue of workflows, and of VCS\nhistory for possible later posts (it is long enough as is).\n\n\nBelow you can find my comments; quoted fragments of \"Understanding\nVersion Control' essay are prefixed with 'UVC> '.  'TODO' refers to\nhttp://www.catb.org/esr/writings/version-control/TODO.html\n\nPlease do participate in this discussion, especially if you have\nsomething to say with respect to rename detection versus rename\ntracking issue.  Thanks in advance.\n\n----------------------------------------------------------------------\nhttp://www.catb.org/esr/writings/version-control/version-control.html\n\n(Independent comment)\n\n[...]\nUVC> This leads to an awkward case called a cross-merge which tends to\nUVC> confuse history-aware merging tools.\n\nFirst, please remember that history-unaware merging tools, such as CVS\nmerge operation (unless you manually specified points in history) have\ntrouble with much simpler case of repeated merging into the same\nbranch.\n\nSecond, I think that currently DVCS can deal with criss-cross merges\ncorrectly; for example git with its recursive merge strategy, or so\ncalled MarkMerge from revctrl.org [TODO: probably needs survey how\nother modern merging DVCS, such as Mercurial, Bazaar or Monotone, deal\nwith criss-cross merges].\n\n......................................................................\n\nUVC> == Container identity ==\nUVC> \nUVC> In a VCS with container identity, files and directories have a\nUVC> stable internal identity, initialized when the file is added to\nUVC> the repository. This identity follows them through renames and\nUVC> moves. This means that filenames and directories are versioned,\nUVC> so that it is possible (for example) for the VCS to notice while\nUVC> doing a merge between branches that two files with different\nUVC> names are descendants of the same file and do a smarter merge of\nUVC> their contents.\n\nThis whole subsection confuses in my opinion the goal (rename-aware\nmerge) with means (container identity).  It should concentrate on\nintelligent merging after renames or moves, not on the possible ways\nto do this.  It should list what is expected of modern VCS to do,\n_then_ enumerate possible ways to do this.\n\nBut the most important thing is that \"container identity\" is a wrong\nidea to have: wholesame rename (or copy) of a file is just special\ncase of more generic code movement and copying.\n\nUVC> \nUVC> Container identity can be implemented by giving each file in the\nUVC> repository a 'true name' analogous to a Unix inode,\nUVC> Alternatively, it can be implemented implicitly by keeping all\nUVC> records of renames in history and chasing through them each time\nUVC> the VCS needs to check what file X was called in revision Y.\n\nIf I understand correctly 'file-ids' is the way original Arch (and its\ndescendant) did this, and how Bazaar deal with file and directory\nrenames, while 'tracking renames' is what Mercurial doea (a bit\nincompletly).\n\nBut there is _another_ way to have rename-aware merge, and it is how\nGit does it.  Git uses heuristic similarity based _rename detection_\nto find how to do a merge in presence of renames.  This works quite\nwell _in practice_ (but it makes it harder to _test_ this solution).\n\n\nThose three ways of dealing with renames have the following advantages\nand disadvantages:\n\n1. Need to use VCS to do file and directory renames and copies.\n\n   Both 'file-ids' and 'rename tracking' have this limitation.\n\n   This means that you cannot use your favorite filemanager, or IDE\n   if it doesn't have support for your VCS, or patches to move and\n   copy files.  This might be migitated a bit by doing rename\n   detection at commit time (Mercurial has this ability wia\n   hg-addremove, but it is not automatic), but that means freezing\n   current algorithm results; with rename detection you can always\n   take advantage of improved rename detection.\n\n2. Ability to improve and correct errors in rename information.\n\n   Both 'file-ids' and 'rename tracking' make it hard to correct\n   errors in rename info, for example if you incorrectly marked file\n   as copy or rename, or forgot to mark it as copy or rename.  If\n   'file-ids' info is stored separately from history, it is less of an\n   issue, but I don't see how you can correct errorneous rename\n   information in 'rename tracking' solution short of rewriting\n   history.\n\n   In the case of 'rename detection' solution Git uses it is\n   non-issue, as no rename information is stored.\n\n3. Dealing with independently (in separate branches) added files.\n   (this issue can be found in \"Test suite\" section of TODO).\n\n   Most 'file-ids' solution have the problem if you add the same file\n   independently on different branches (e.g. adding file via patch).\n   First, one of histories must vanish; second, you have to repeat\n   resolution if file-id conflict for every repeated merge.\n\n   From TODO 'rename tracking' solution used in Mercurial doesn't have\n   this problem, and of course Git's 'rename detection' doesn't have\n   it either.\n\n   Note that there are different versions of this issue: same name\n   and same contents, different name same contents (rename without\n   ancestor), and same name different contents.  The last case is of\n   issue with automatic file-ids if they are dependent on file name;\n   first case is of issue with automatic file-ids if they depend on\n   branch/comitter/time, i.e. if file-ids differ on different\n   branches.  Note: same contents here migh mean _almost_ the same\n   contents.\n\n4. Creating new files in renamed directories.\n   \n   Usually called support for directory renames; the problem is if one\n   side (on one branch) renames some directory, and other side (other\n   branch) creates new files in the old-name directory.\n\n   For 'file-ids' solution this means that there have to be 'file-ids'\n   ('inodes') also for directories; Bazaar has this feature.  For\n   'rename tracking' solution this means that rename information for\n   directories has to be stored somewhere; from what I understand\n   Mercurial with its filename-hashed storage and per-file stored\n   rename information doesn't have this feature.  For both 'file-id'\n   and 'rename tracking' ('rename info') solutions this I think\n   usually mean that you have to do directory renames using VCS\n   tools.\n\n   Whle on first glance it might seem that 'rename detection' solution\n   (like the one used in Git) cannot deal with this problem it is not\n   true.  VCS employing similarity based rename detection can detect\n   wholesame directory rename based on pattern of file renames, and\n   can put new file in new-name of directory.  Moreover it should be\n   able to deal quite sanely with more complicated cases like\n   splitting or merging of directories.  _However_ Git currently does\n   not support this case (but see the note below), although there were\n   some preliminary patches adding 'wholesame directory rename\n   detection', so it is not purely theoretical.\n\n   Note: usually you cannot simply move file to new directory without\n   any other changes, so automatic creating new file (from the point\n   of view of branch we merge into) in new-name directory instead of\n   old-name directory might be not a good idea without stopping for\n   manual merge conflict resolution, as it would result in semantic\n   conflict.\n   \n5. Misdetection of file renames, and remembering manual corrections.\n   (this problem concerns only 'rename detection' solution).\n\n   Of course 'rename detection' algorithm is not perfect.  It can find\n   rename when there isn't any if many files consist mainly from\n   identical boilerplate (e.g. copyright, license, etc.).  It can fail\n   to detect rename if files differ too much (usually then files\n   cannot be merged automatically anyway then), or if files are too\n   small (like usually in simple test cases used).  In the presence of\n   multiple file renames and copies it can assign move or copy source\n   to the wrong file.\n\n   Closely related to this issue is a problem of remembering manual\n   corrections to rename detection algorithm, and manual hints (like\n   inn 'rename tracking' solution).  Both were discussed on git\n   mailing list, the former under the name of having git-rerere2,\n   remembering (and reusing) of recorded resolutions of tree-level\n   conflicted merge, but there were (if I remember correctly) no\n   conclusion and no patches.\n   \n6. Performance bottlenecks in managing renames during merge\n\n   I don't know if it is a problem for 'file-id' solution (Bazaar), or\n   for 'rename tracking' solution (Mercurial), but with 'rename\n   detection' (Git) if there were a lot of reorganization of directory\n   hierarchy between merge points then merge can take a lot of time.\n\n\nUVC> \nUVC> Absence of container identity has the symptom that file\nUVC> rename/move operations have to be modeled as a file add followed\nUVC> by a delete, with the deleted file's history magically copied\nUVC> during the add. \n\nI don't know if this paragraph was meant to be about another issue\nwith renames in VCS, namely dealing with renames and copise during\nhistory browsing (which includes both log of changes, and tracking\nline-wise file history aka. annotate/blame/praise).\n\n\n1. Rename-aware changelog (commit logs).\n\n1.1. Showing renames in \"<scm> log\", i.e. in whole project history.\n\n     The only complication with rename detection here is choosing a\n     level of rename and copy detection, as it might be CPU\n     intensive.  In Git those are 'detect renames', 'detect copies as\n     well as renames' and 'find copies harder'. That can, of course,\n     be configured.\n\n1.2. Following renames in \"<scm> log <filename>\", i.e. in single file\n     history.\n\n     Currently Git does not support it very well.  There is '--follow'\n     option for git-log, but it is more of a hack to have something\n     similar to what for example Subversion provides, than a full\n     solution.  It works for simple histories, and might fail for more\n     complicated ones; it is not however fundamental limitation.  On\n     can always use \"git log -- <old name> <new name>\"...\n\n     On the other hand single file history should be for developers\n     second-class citizen in VCS supporting changesets; full history\n     is _more_ than sum of single-file histories.\n\n\n2. Rename-aware line-wise file history (which means annotating file\n   with history of each line, something like \"cvs annotate\").\n\n   Note that line-wise file history cannot deal with _deletions_;\n   it is a known limitation of this tool.\n\n2.1. Following wholesame file renames and copying.\n\n     The issue is not stopping at file rename when tracking where\n     given line came from... and of course representing this\n     information in blame/annotate output.  git-blame supports this,\n     and I think other VCS (Bazaar, Mercurial) also does.\n     [TODO: check this]\n\n2.2. Following code movement and copying.\n\n     You can request that git-blame detect moving lines in the file\n     (e.g. moving around code), and detect lines copied from other\n     files (code moved across files).  I think it is (together with\n     ability to ignore changes in whitespace in blame/annotate)\n     currently (?) feature unique to Git; it also shows that idea of\n     \"container identities\" is limited and narrow-minded, and that one\n     should think of file renames as of special case of code\n     movement. \n     [TODO: perhaps sreenshot of \"git gui blame\" or equivalent?]\n     [TODO: some stats about file renames and other code movement]\n\nUVC> \nUVC> Usually VCSes that lack container identity also create parent\nUVC> directories on the fly whenever a file is added or checked out, and\nUVC> you cannot actually have an empty directory in a repository.\n \nIn my opinion this is an fundamentally unrelated issue.  The question\nif directories are created and deleted on demand is behavioral issue;\nnote that _not_ having directories deleted on the fly result in higher\nprobability of file/directory conflict.\n\nGit is a bit peculiar in this case: empty directories _can_ be\nrepresented in repository, currently cannot be represented in (flat)\nstaging area aka. index although it shouldn't be too hard to add, and\nare removed on the fly which again shouldn't be too hard to change.\n\nNote that you can track empty directories in Git (even if VCS adds\nthem and removes them on the fly) by trick of having for example empty\n'.gitignore' file in it.\n\n\nUVC> == Snapshots vs. changesets ==\nUVC> \nUVC> There are two ways to model the history of a line of\nUVC> development. One is as a series of snapshots of an evolving tree\nUVC> of files. The other is as a series of changesets transforming\nUVC> that tree from a root state (usually empty) to a tip state.\n\nGit is snapshot-based (although packed format uses [binary] delta\ncompression).  Mercurial is changeset-based.  Bazaar uses different\nrepresentation altogether, I think[1] (used to use weave); wiki\nsays that it is snapshot-oriented, but it has file-ids.\n\n[1] http://bazaar-vcs.org/BazaarFormats is a bit lacking in details\n\n[...]\nUVC> Changeset-based systems have some further distinctions based on\nUVC> what kinds of data a changeset carries. At minimum, a changeset\nUVC> is a group of deltas to individual files, but there are\nUVC> variations in what kind of file-tree operations are represented\nUVC> in changesets.  \n\nI think most famous _example_ here is Darcs, with _idea_ of other\noperations than simple text delta, like \"rename this variable to that\nname\" example in documentation, but I don't know if it was actually\nwent beyound hand-waving.\n\nNote that patch commutation algebra of Darcs (most pure-changeset VCS\nthere, in my opinion) might look like a neat idea, but please remember\nthan Darcs had (and perhaps has still) exponential bad performance of\nmerging in some cases.\n\nUVC> \nUVC> Changesets which include an explicit representation of\nUVC> file/directory moves and renames make it easy to implement\nUVC> container identity. (Container identity could also be implemented\nUVC> as a separate sequence of transaction records running parallel to\nUVC> a snapshot-sequence representation, but I know of no VCS that\nUVC> actually does this.) \n\n[TODO: check how Bazaar actually does this, as it is example of VCS\nwith 'file-ids', also for directories; check where Mercurial stores\n'rename tracking' information].\n\n[...]\nUVC> Snapshot and changesets are not perfectly dual\nUVC> representations. It took a long time for VCS designers to notice\nUVC> this; the broken symmetry was at the core of a well-known\nUVC> argument between the designers of Arch and Subversion in 2003,\nUVC> and did not begin to become widely understood until after Martin\nUVC> Pool's 2004 essay \"Integrals and Derivatives\"[2]. Pool, a\nUVC> co-author of bzr, correctly noted that attempts to stick with the\nUVC> more intuitive sequence-of-snapshots representation have several\nUVC> troubling consequences, including making container identity and\nUVC> past merges between branches more difficult to track. \nUVC> \nUVC> [2] http://sourcefrog.net/weblog/software/vc/derivatives.html\n\nBelow there is excerpt from OLD \"Integrals and Derivatives\" essay\n(I hope I have choosen relevant part of this essay).\n\nID> Working in terms of changesets, or at least having the option to\nID> do so allows more powerful operation. \nID>\nID> For example, consider repeated merges among a related set of\nID> trees. Arch and Darcs [which work primarily in the changeset\nID> domain] handle this well, because they can easily remember which\nID> changesets have already come across. Subversion and CVS tend to\nID> handle it poorly, because merely tracking which version from the\nID> other tree has merged doesn't really capture the right\nID> information.\n\nThis is, as we now know, _wrong_.  Subversion and CVS handle this\npoorly not because they are snapshot based (CVS most certainly is not:\nit is file-level delta based), but because they do not store merge\ninformation (which revisions were merged to form a merge commit,\ni.e. all parents of a merge commit): they lack merge tracking.\nTherefore it is not possible to find common ancestors (merge bases),\nbecause there is no enough information.  _Not_ because being\nsnapshot-based.\n\nAdditionally it is now I think universally acknowledged that three-way\nmerge (with a bit extra to deal with possibly multiple merge-bases,\ne.g. in presence of criss-cross merge) is superior merge algorithm; it\nis certainly superior to \"reapply patches skipping already present\"\nalgorithm that quoted example seems to imply that Arch and Darcs use\n(which is available in modern DVCS as well, either as some patch\nmanagement / patch queue extensions, or as git-rebase, Mercurial's\ntransplant extension, or Bazaar graft extension).\n\n\n[...]\n\nUVC> = What, if anything, have we learned from history? =\nUVC> \nUVC> There's a folk saying that \"It's not what you don't know that\nUVC> hurts you, it's what you think you know that ain't so.\" In\nUVC> examining the pattern of development of VCSes, it seems to me\nUVC> that the this sub-field of computer science has been less\nUVC> hampered than most by difficulties in finding appropriate\nUVC> techniques, but more hampered than most by wrong assumptions that\nUVC> hung on far longer than they should have. \nUVC> \nUVC> First wrong assumption: Conflict resolution by merging is\nUVC> intractably difficult, so we'll have to settle for locking. It\nUVC> took at least fifteen and arguably twenty years for VCS designers\nUVC> to get shut of that one. But it's historical now. \n\nI'm not sure if it has place here, i.e. if it was wrong assumption or\njust lack of thought, but I would emphasize that commit-before-merge \nis much, much better than merge-before-commit (or update-before-commit, \nas it is _implicit_ merge that might need to be performed) workflow.\n\nOne of more important tasks of VCS is to not lose your changes; the\nmerge-before-commit does not fill this tack completely, in my opinion,\nespecially nowadays with networked large-group collarboration.\n\n?merge strategies\n\nUVC> \nUVC> Second wrong assumption: Change history representation as a\nUVC> snapshot sequence is perfectly dual to the representation as\nUVC> change/add/delete/rename sequences.. This folk theorem is well\nUVC> expressed in the 2004 essay \"On Arch and Subversion\"[3]. It is\nUVC> appealing, widely held, and dead wrong.\nUVC> \nUVC> File renames break the apparent symmetry. The failure of\nUVC> snapshot-based models to correctly address this has caused\nUVC> endless design failures, subtle bugs, and user misery. \n\nIt is not true.  Example of snapshot-based Git, which with its rename\ndetection deals very well in practice with file renames contradict\nthis theory.  Bazaar which is supposedly snapshot-based, yet support\n\"container identities\" ('file-ids') contradict this further.\n\nThe symmetry might be broken _only_ if there are other operators in\nchangesets than simple delta.  But one has to remember that if\nrepresentation is not equivalent to snapshot-based in the sense that\nyou can do a (3-way) merge based on endpoints (branches to be merged\nand common ancestors aka. merge bases) _only_, and one has to take\n_whole_ history into account when merging, then there is high\nprobability of cases where merge performance suffers badly.  Please do\nnot say that performance doesn't matter, because it does; if operation\ntakes minutes (or more) rather than seconds, then this would affect\nworkflow used, as people try to avoid unpleasant operations...\n\nUVC> \nUVC> Practically speaking, failure to address this broken symmetry\nUVC> goes a long way towards explaining why CVS became such a\nUVC> disaster. But the damage didn't end there, which is why I'm\nUVC> courting controversy by pointing out that it underlies a debate\nUVC> about third-generation designs that is still live today. Should\nUVC> VCSes be purely content-addressable filesystems (the Mercurial\nUVC> and git approach) or should they have container identity (as in\nUVC> Arch, monotone, and bzr)? \nUVC> \nUVC> That debate is not over, but at least VCS designers are grappling\nUVC> with it now. \n\nCVS was disaster no because of (supposedly) broken symmetry between\nsnapshots (snapshot-based operations) and changesets; it was disaster\nbecause its per-file versioning roots were very visible in: non-atomic\ncommits, no support for changesets, heavyweight (bad performance)\nbranching and tagging, complete lack of support for renaming files.\n\nI agree that the issue of content-addressed filesystem versus\ncontainer identities (file-ids) is importnat one, but I would say that\nneither one won over the other...\n\nUVC> \nUVC> I have a guess about the third wrong assumption. I think it goes\nUVC> something like this: The correct choice of abstractions,\nUVC> operations, and containers in a VCS is the one that makes the\nUVC> cleverest sorts of data-shuffling possible. I suspect that, as\nUVC> our algorithms get better, we're going to find that the best\nUVC> choices are not the most theoretically clever ones but rather the\nUVC> ones that are easiest for human beings to intuitively model.\n\nHere I think Git model wins hands down: DAG (Direct Acyclic Graph) of\ncommits (Monotone had it first), clear relation between objects:\ncommit object contain metadata (like authorship and commit message),\nlink to zero (root / initial commit), one (usual case), or two or more\n(merges) parent commits, and link to snapshot of a state of project at\ngiven version...  Having branches and tags (and remote-tracking\nbranches) to exist outside DAG of commits, in a separate namespaces;\nnot that disaster of conventions for imitation of branches and tags in\nSubversion.  Having current branch to be just pointer to one of\nbranches.\n\nHaving delta compression \"under the hood\" in the packfiles allow to\ncombine efficiency of deltas for repository size and network transfer,\nwhile retaining very clear model of snapshot-based VCS.\n\nUVC> \nUVC> This is why, even though I find constructions like today's\nUVC> elaborate merge theory[4] fascinating to think about. I'm not\nUVC> sure it is actually going anywhere useful. Naive merge algorithms\nUVC> with poor behavior at edge cases may actually be preferable to\nUVC> more sophisticated ones that handle edge cases well but that\nUVC> humans have trouble anticipating effectively.\n\nNowadays even Bram Cohen of Codeville merge agrees[citation needed]\nthat 3-way merge, which works in most cases but if fails is easy to\nunderstood, is better than advanced merge strategies which can merge\nautomatically a few more cases, but if fail it is difficult to resolve\nconflicts; note that fail include wrongly resolved merge (merge\nstrategy gives no conflicts, but merge resolution is wrong).\n\nUVC> \nUVC> An even more interesting question is this: what are the fourth,\nUVC> and nth wrong assumptions --- the ones we haven't noticed we're\nUVC> making yet? \nUVC> \nUVC> [3] http://www.reverberate.org/computers/ArchAndSVN.html\nUVC> [4] http://revctrl.org/CategoryMergeAlgorithm\n\nWell, Linus probably would say that concentrating on special case of\nfile and directory renames instead of more generic code movement is a\nwrong idea.\n\nBut what do YOU think?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"102890","messageId":"20090202202424.GG14762@mit.edu","threadId":"17510","inReplyTo":"200902021948.54700.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-02T20:24:24Z","receivedAt":"2009-02-02T20:24:24Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Feb 02, 2009 at 07:48:53PM +0100, Jakub Narebski wrote:\n> In my opinion the most important issue is concentrating on \"container\n> identity\" instead of on the underlying issue of renames in version\n> control, which includes intelligent, rename-aware merge; talk about\n> issues and not about possible solution.  I will concentrate on this\n> issue for now, and leave for example issue of workflows, and of VCS\n> history for possible later posts (it is long enough as is).\n\nThis was discussed in no small amount of detail on the mailing list\nuvc-reviewers, which used to be hosted here:\n\n\thttp://thyrsus.com/mailman/listinfo/uvc-reviewers\n\nUnfortunately, it looks like Eric has taken down his mailman instance\non thyrsus.com.  I have personal archives of the list, and the list\nused to be have public archives, so I don't feel any hesitation\nsharing it with interested parties.\n\n> Below you can find my comments; quoted fragments of \"Understanding\n> Version Control' essay are prefixed with 'UVC> '.  'TODO' refers to\n> http://www.catb.org/esr/writings/version-control/TODO.html\n> \n> Please do participate in this discussion, especially if you have\n> something to say with respect to rename detection versus rename\n> tracking issue.  Thanks in advance.\n\nHeh.  A lot of this has been said already.  I think one of the reasons\nwhy Eric kept things short in his paper, and did *not* say a lot about\nwhether or not container identity tracking was fundamentally needed or\nnot was because we didn't come to any real consensus on the\nuvc-reviewers mailing list.  I believe it is extremely difficult to do\nso given that it's very hard to avoid the slippery slope of advocating\nfor one SCM system versus another.\n\nI'll include some of my writings on the subject from the uvc-reviewers\nmailing list so folks can see where some of this discussion went last\ntime...  (All of this dates from January, 2008, when Eric was last\naggressively updating the paper in question.)  BTW, when I referred to\nSCM's being a horrible hack and \"guessing\" and \"fit only to be used by\namateurs\" if they didn't record function-level identity tracking,\nthere were those who were seriously arguing that any SCM (i.e., like\ngit) that didn't track container identity was fundamentally a \"hack\".\nYes, there are people who seriously take that view, some of which were\nvery bitter that their DSCM didn't win the market/popularity wars, and\nso their pet projects overtaken by SCM's such as git, describing\n$THEIR_PET_SCM_WITH_PROVABLY_CORRECT_SEMANTICS as Betamax, and git as\nVHS.  The argument that without rename-tracking, if git was used to\ndevelopment an software for Air Traffic Control application, airplanes\ncould be dropping out of the sky was also made by these advocates, no\nkidding.  (So was the argument that using a DSCM that didn't do\ncontainer identity tracking might be considered Programming\nMalpractice.)\n\nSo be careful about wanting to reopen this discussion; if the some of\nthe wrong people join in, you may be very sorry!   :-)\n\n   \t     \t       \t       \t    \t       \t    \t - Ted\n\n\n> Here then are some types of identity  \n> tracking one might imagine:\n> \n>    * File identity tracking: tracks the identity of a file through  \n> renames and moves.\n>    * Simple file content tracking: tracks the identity of content  \n> using adds and deletes within a single file.  (Note, there is a  \n> question that could be asked here about the resolution of the  \n> tracking.  Most current systems that track do so on a line-by-line  \n> basis, but one could imagine tracking bytes.  I wont say any more  \n> about this in this email.)\n>    * Movement of content within a file: tracks the identity of  \n> content within a file when lines are moved.\n>    * Movement of content between files: tracks the identity of  \n> content when lines are moved between files.\n\nOne obvious one which isn't in this list is \"Directory Identity\nTracking\", that is when you move a directory, new files which are\ncreated in one branch at the original directory location will be moved\nwhen you merge with another branch where the directory has been moved.\n\nIn private conversations with Tom Lord, he tells me that he had also\nplayed with the concept of \"Function/variable (more generally,\nprogramming structure identifier) identity tracking\".  That is,\nsuppose you had an editor like Eclipse which has as a primitive,\n\"Rename Java identifier (class/method function/variable)\", and this\ninformation was passed into the SCM so it could be tracked.  Then in\none branch, a Java identifier could be renamed, and then in another\nbranch, the use of that same Java identifier could be added in 20\ndifferent places --- and since the SCM knows, at a deep semantic\nlevel, that a rename had taken place in Branch A, when it is merged\nwith Branch B, it could DTRT and change the newly added uses of the\nrenamed identifier to its new name.\n\nAnd like with Directory Identity Tracking, it's not hard to come up\nwith scenarios where without this level of tracking, something\nhorribly wrong could take place as a result of the SCM not tracking\nfunction identities and using them when doing merges.  At the very\nleast, the program would fail to compile, and if the example involved\nan Air Traffic Control system and multiple function renames taking\nplace, you could even come up with a contrived horror scenario where\nplanes would be falling out of the sky --- that is, if you ignore\nregression testing, and simple coding practices that would prevent\nsomething like this from happening.\n\nOf course, the flip argument by people who are trying to promote their\nbrand-spanking new SCM that did function identity tracking (FIT) is\ncritical since SCM's are all about ACCOUNTING, and without FIT,\nsystems that try to merges are just GUESSING, and *obviously* a system\nwhich did FIT is far superior to a SCM that didn't; in fact, a SCM\nthat didn't do FIT is just a Horrible Hack Done By Amateurs.\nFurthermore, using an SCM without this feature would (according to\npromotors of this hypothetical new SCM), be Programming Malpractice.\n\nAnd if this sounds silly, I'm just repeating the exact same arguments\nthat proponents of systems like arch, Bk, et.al, which store the user\nintention information of file and directory renames, have recently\nadvanced against git since it doesn't store this sort of information.\n(It may reconstruct rename information in a lazy fashion when it is\nneeded, but it doesn't store it.)  But if file and directory renames\nis a type of user intention which MUST be stored in order for an SCM\nnot to be a hack, why not function, variable, and class renames?  That\ntoo would be another type of user intention.\n\n\t \t     \t    \t     \t  - Ted\n\n---------------------------\n\n> The second could be called \"location\".  Which file should this patch  \n> be applied to?  Which lines within a file should this hunk be applied  \n> to?  I argue in [2] that Darcs does strictly better at the task of  \n> location than do SVN or GNU diff3.  (I think that SCCS, BK and CDV do  \n> as well, but I don't understand them well enough to be sure.)  I  \n> argue that Darcs does strictly better in the sense that its answers  \n> to the location question are often better and never worse, and that  \n> it does so *not* by having a more sophisticated heuristic or by  \n> getting lucky more often, but by a simple, provably-correct algorithm  \n> which uses valuable information that other algorithms overlook.\n\nHow are you defining \"provably correct\"?  In order to show\ncorrectness, you need to define what correctness means.  \n\nOne approach is that you force the user to tell you --- and if you are\nin the middle of applying a series of 500 patches, you throw up the\nAnnoying GUI Dialog Box which stops the application of patches dead in\nits tracks, and force the user to confirm whether this is a rename, or\na delete followed by an add of remarkably similar content.  Or, if\npatch removes all the files from one directory, and created them all\nin another directory, that what was the user intention was a directory\nrename, and the SCM records it as such.  Here, you are *assuming* that\nwhat the user tells you is correct, and that's part of the lemma you\nuse for proving correctness.  If the user, who is seriously annoyed at\nthe popup boxes, says, \"Yeah, yeah, yeah\", and dismisses the dialog\nbox without changing the defaults (which were selected via a\nhueristics and which were wrong), well, it's not the SCM's fault since\nthe user told it what it wanted, and the user was wrong.  GIGO.\n\nIn your case, you're saying that Darcs is using \"valuable information\"\nthat other algorithms overlook.  OK, so the Darcs people were more\nclever about designing a hueristic which tries to approximate user\nintention, and having designed the hueristic which uses said \"valuable\ninformation\" you can prove whether or not Darcs' algorithm correctly\nimplemented said hueristic.\n\nBut at the end of the day, it's still a hueristic, and the use of\nwords \"provably correct\" is just a semantic trick.  Even svn's lack of\ndirectory rename support could be considered \"provably correct\", if\nthe definition of \"correct\" is an algorithm which determinically\ncreates new files in their original location, even if all other files\nin that directory were deleted and new files with the same name and\nsame content were created somewhere else.  It's still an algorithm,\nand you can prove whether or not it meets its design specs, but to the\nextent that it is less likely to approximate user intenions, people\nwould say that svn might be less useful in such cases than some other\nSCM.\n\n> Let's be careful not to lump these three things together, say \"All  \n> merging involves guessing.\", and thus overlook the interesting fact  \n> that some merge algorithms involve strictly more guessing than do  \n> others.\n\nPart of the problem is that words like \"guessing\" and claims of some\nalgorithm being \"provably correct\" are basically marketing words.\nThey are generally used to denigrate one SCM, and promote someone's\nfavorite SCM as being **better**.\n\nFundamentally, the goal of merging is to Do The Right Thing --- from a\nsemantic point of view, which means that the user's intentions is\nwhat's important at the end of the day.  The question then is whether\nyou record the user's intentions, or try to determine it in from a\nhueristics point of view.\n\nThe people who claim that recording the user's intentions is superior\nwill claim that you can never know for sure what the user meant, so\nyou have to ask him or her to provide that information.  In some cases\nthat's relatively easy; you require the user to use commands like \"bk\nmv\" and \"bk cp\" and \"bk rm\" which not only performs the function, but\nalso records the user's intention.  Unfortunately, if you are applying\na patch, and the patch file hasn't been enhanced to carry this kind of\ninformation, you have to use hueristics and then somehow get the user\nto confirm them --- hence the use of the Annoying Popup Dialog Box.\n\nIn other cases, you can't determine it easily, such as the \"rename a\nJava method function\" case, unless you have a specialized editor which\nhas this as a primitive, or, alternatively, even more Annoying Dialog\nBoxes that pop up as you try to commit a change.  Once you have\nrecorded this information, using it in the merge is relatively easy\n--- or if not easy, at least relatively easy to specify and then show\nwhether or not the information was used correctly.\n\nWhat then tends to happen is that people whose SCM does one kind of\nuser intention recording (such as file renaming) will use this as a\nhuge club to say their system is better than another system, and that\na system which doesn't record this information is \"Guessing\".  They\nwill also say that their system is \"Provably Correct\".  So both of\nthese words are really Red Meat Marketing words, which get used when\npeople try to say that Their SCM Is Superior.\n\nOn the flip side, the people who don't do any recording of this\ninformation will point out that trying to record the information is\nhard, especially when changesets go through a lossy medium, such as\npatch files which are e-mailed around, which doesn't record this kind\nof user intention.  This may or may not applicable for a particular\nproject, but for some projects, it is extremely important, especially\nfor those where e-mail is the primary communication channel and how\npatches get reviewed and passed around.\n\nThe other point which people on \"we don't record user intentions; we\njust record content\" tend to use is that you can always add more\nhueristics later, but in practice, if you didn't record the user\nintention when you made the commit, it's almost impossible to add it\nlater.  So for example, suppose git, which doesn't have function\nrename support today, has a new merge engine added later which works\nspecifically for Java files that correctly intuits and deals with\nmethod function renames.  (Maybe some crazy Java programming\nmethodology does this all the time, so people get motivated to write\nsuch a thing.)  A SCM which works on recorded user intentions will\nneed to add that support, in a way which doesn't break backwards\ncompatibility of their distributed repository, *OR* accept the fact\nthat for function renaming, it will have to use hueristics that are\nrun at merge time.\n\nSo this seems to be fundamentally a tradeoff.  The two *objective*\nthings that can be said is how many user intentions are recorded at\ncommit time (file rename?  directory rename?  function/variable\nrename?), and what sort of information is used at merge time via\nhueristics to determine user intention.  And if you want to call those\nhueristics an \"algorithm\" because it sounds more mathy and provable,\nsure, whatever.  The fact is, the algorithm is still an approximation\non trying to determine user intent, with the goal of making the merge\ndo the right thing from a Semantic Point of View.\n\n\t\t\t\t* * *\n\n>From a project point of view, how often you actually *do* merges is an\ninteresting one.  If merges requiring complicated user intention\ntracking is necessary don't happen very often, maybe focusing on\nthis issue to the extent that SCM geeks tend to do isn't very\nproductive.  In my opinion, the overall usability of the system is\n*far* more important.  In the OSS world, every project is competing\nfor programmers, and if the tool is easier to use, the more likely it\nwill be that you can get people to contribute to your project.  Even\nif they aren't bright enough to work on the core algorithms for your\nproject, they could at least improve the documentation.  That was what\ndrove my decision to switch to Mercurial back in 2005.  Last year, I\nmoved my projects to git because of various features that worked well\nwith my development workflow and which generally improved programmer\nproductivity.  Whether or not file or directory renames were being\ntracked in the SCM had *no* bearing on my decision to switch, because\nmerges happen rarely, and I have an extensive regression test suite\nwhich I run after almost every commit, and *definitely* after every\nmerge.  So if a merge doesn't do the right thing, that's OK; I'll fix\nit up, and use \"git commit --amend\" correct the merge commit.\n\nAnd yet, people seem to focus on recording of user intention because\nit reflects some holy grail of Perfection and Correctness.  And maybe,\nbecause it is easily measurable, whereas usability and improving\nprogrammer productivty are inherently more subjective measures.\nWhat's very sad are the people who are feel profoundly hurt that they\nspent a huge amount of their life working on SCM Correctness, only to\nfind that people chose other SCM's based on other metrics and other\nissues other than the one that they felt was most important.\nUnfortunately there's not a lot that can be done about that.\n\n\t\t      \t       \t      - Ted\n"},{"id":"102892","messageId":"20090202203514.GA26691@thyrsus.com","threadId":"17510","inReplyTo":"20090202202424.GG14762@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2009-02-02T20:35:14Z","receivedAt":"2009-02-02T20:35:14Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Theodore Tso <tytso@MIT.EDU>:\n> Unfortunately, it looks like Eric has taken down his mailman instance\n> on thyrsus.com.  I have personal archives of the list, and the list\n> used to be have public archives, so I don't feel any hesitation\n> sharing it with interested parties.\n\nOops!  I think it got lost during a system upgrade.  I should be able \nto restore it and the archives, now you've reminded me.\n\nThe paper isn't dead, by the way, I've just been buried in other stuff \nthe last year or so.  It's been on my mind lately to resume work on it.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"103024","messageId":"200902032157.51839.jnareb@gmail.com","threadId":"17510","inReplyTo":"20090202203514.GA26691@thyrsus.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-03T20:57:50Z","receivedAt":"2009-02-03T20:57:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 2 Jan 2009, Eric S. Raymond <esr@thyrsus.com> wrote:\n> Theodore Tso <tytso@MIT.EDU>:\n\n> > Unfortunately, it looks like Eric has taken down his mailman instance\n> > on thyrsus.com.  I have personal archives of the list, and the list\n> > used to be have public archives, so I don't feel any hesitation\n> > sharing it with interested parties.\n> \n> Oops!  I think it got lost during a system upgrade.  I should be able \n> to restore it and the archives, now you've reminded me.\n\nBy the way, Mercurial repository of \"Understanding Version Control\"\nsources doesn't work either: http://thyrsus.com/hg/uvc returns\n404 Not Found error.\n\nWhat is strange is that http://thyrsus.com works...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"103059","messageId":"200902040304.05028.jnareb@gmail.com","threadId":"17510","inReplyTo":"20090202202424.GG14762@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-04T02:04:02Z","receivedAt":"2009-02-04T02:04:02Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, Feb 02, 2009, Theodore Tso <tytso@mit.edu> wrote:\n> On Mon, Feb 02, 2009 at 07:48:53PM +0100, Jakub Narebski wrote:\n\n> > In my opinion the most important issue is concentrating on \"container\n> > identity\" instead of on the underlying issue of renames in version\n> > control, which includes intelligent, rename-aware merge; talk about\n> > issues and not about possible solution.  I will concentrate on this\n> > issue for now, and leave for example issue of workflows, and of VCS\n> > history for possible later posts (it is long enough as is).\n> \n> This was discussed in no small amount of detail on the mailing list\n> uvc-reviewers [...]\n\nI guess that this mailing list is subscribe-only, isn't it?  So doing\nCC to uvc-reviewers wouldn't, unfortunately, cut?\n\n> > Below you can find my comments; quoted fragments of \"Understanding\n> > Version Control' essay are prefixed with 'UVC> '.  'TODO' refers to\n> > http://www.catb.org/esr/writings/version-control/TODO.html\n> > \n> > Please do participate in this discussion, especially if you have\n> > something to say with respect to rename detection versus rename\n> > tracking issue.  Thanks in advance.\n> \n> Heh.  A lot of this has been said already.  I think one of the reasons\n> why Eric kept things short in his paper, and did *not* say a lot about\n> whether or not container identity tracking was fundamentally needed or\n> not was because we didn't come to any real consensus on the\n> uvc-reviewers mailing list.  I believe it is extremely difficult to do\n> so given that it's very hard to avoid the slippery slope of advocating\n> for one SCM system versus another.\n\nWell, I tried to be objective, but I know I am biased towards Git.\nI tried to list advantages and disadvantages of all three methods of\ndealing with renames in VCS: 'container identity' aka. 'file-ids'\n(and 'directory-ids') which I think is solution used by Bazaar;\n'rename tracking' or 'recording rename information' which I think\nis the solution used by Mercurial, and 'rename detection' used by\nGit.\n\nBTW. if I remember correctly (either from comments in UVC, or from\nTODO) Eric wanted to have test suite which he could run to examine\nhow well does given VCS support renames in the form of intelligent\nrename-aware merges, and in the form of following file through\nrenames when examining history.  \n\nWhen writing this email I have wanted to add as appendix a proposal\nfor such test, which would also clarify what are the expectations\nwrt. rename support... but I plainly forgot to add it.  You can\nfind some very bare bones version in my post \"Git at Better SCM\nInitiative comparison of VCS (long)\"[1][2] on git mailing list,\naround where I was talking about intelligent_renames section.\n(The post was meant to correct invalid information about Git in\n'Better SCM Initiative' comparison[3]).\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/95809\n[2] My earlier posts with similar title were about _adding_ Git\n    to the comparison.\n[3] http://better-scm.berlios.de/comparison/\n\n> \n> I'll include some of my writings on the subject from the uvc-reviewers\n> mailing list so folks can see where some of this discussion went last\n> time...  (All of this dates from January, 2008, when Eric was last\n> aggressively updating the paper in question.)\n\nThank you very much for those excerpts / fragments, even though\nI'd rather have your fresh comments either on current state of \n\"Understanding Version-Control Systems\", or on my post.\n\n> BTW, when I referred to \n> SCM's being a horrible hack and \"guessing\" and \"fit only to be used by\n> amateurs\" if they didn't record function-level identity tracking,\n> there were those who were seriously arguing that any SCM (i.e., like\n> git) that didn't track container identity was fundamentally a \"hack\".\n> Yes, there are people who seriously take that view, some of which were\n> very bitter that their DSCM didn't win the market/popularity wars, and\n> so their pet projects overtaken by SCM's such as git, describing\n> $THEIR_PET_SCM_WITH_PROVABLY_CORRECT_SEMANTICS as Betamax, and git as\n> VHS.  The argument that without rename-tracking, if git was used to\n> development an software for Air Traffic Control application, airplanes\n> could be dropping out of the sky was also made by these advocates, no\n> kidding.  (So was the argument that using a DSCM that didn't do\n> container identity tracking might be considered Programming\n> Malpractice.)\n\nThat is, I think, the difference between being 'perfect in theory' and\n'good enough in practice', and reminds me of times of discussion about\n\"perfect\" contents merge algorithm (Codeville merge and precise\nCodeville merge, folks at revctrl.org and mark-merge, etc.).  Nowadays\neven Bram Cohen (of the Codeville) that 3-way merge algorithm is good\nenough (\"Version Control Recommended Practices\", '3. Use 3-way merge',\nat http://bramcohen.livejournal.com/52148.html).\n\nFailing gracefully matters (i.e. if there is a conflict it can be\neasily understood), low probability of falsely clean merges matters,\nand performance (to not be exponential in some real cases) matters\ntoo.  More than theoretical perfection.\n\nAnd there are many, many examples of using heuristics over perfect\nalgorithmic solution (if algorithmic solution exists at all).\n\n> \n> So be careful about wanting to reopen this discussion; if the some of\n> the wrong people join in, you may be very sorry!   :-)\n\nThanks for the warning! :-) Well, I hope they are not present on git\nmailing list...\n\n> \n>    \t     \t       \t       \t    \t       \t    \t - Ted\n> \n> \n> > Here then are some types of identity  \n> > tracking one might imagine:\n> > \n> >    * File identity tracking: tracks the identity of a file through  \n> > renames and moves.\n\nGit tracks _contents_ of a file.  Not file identity.  And this works\nquite well in practice (although it is not free from disadvantages).\n\n> >    * Simple file content tracking: tracks the identity of content  \n> > using adds and deletes within a single file.  (Note, there is a  \n> > question that could be asked here about the resolution of the  \n> > tracking.  Most current systems that track do so on a line-by-line  \n> > basis, but one could imagine tracking bytes.  I wont say any more  \n> > about this in this email.)\n> >    * Movement of content within a file: tracks the identity of  \n> > content within a file when lines are moved.\n> >    * Movement of content between files: tracks the identity of  \n> > content when lines are moved between files.\n\n\"git blame -C -C <file>\" supports all three. I think it is the only\nVCS which supports detection of code movement and copying in single\nfile and across files.  That's by the way, of course...\n\n> \n> One obvious one which isn't in this list is \"Directory Identity\n> Tracking\", that is when you move a directory, new files which are\n> created in one branch at the original directory location will be moved\n> when you merge with another branch where the directory has been moved.\n\nOr \"Detecting [Wholesame] Directory Renames\"... which can be done\nusing 'rename detection' paradigm, and we have patches to prove it![4]\nbut unfortunately code didn't made it (yet!) into git.  And it can,\nI think, deal with splitting files into two directories, something\nwhich I guess in 'container identity' (directory-id) based solution\nis simply impossible\n\nThis issue would be in my planned test suite of VCS rename support.\n\n[4] e.g. http://thread.gmane.org/gmane.comp.version-control.git/99529\n\n> In private conversations with Tom Lord, he tells me that he had also\n> played with the concept of \"Function/variable (more generally,\n> programming structure identifier) identity tracking\". [...]\n\nTrue, it is _theoretical_ advantage of changeset based model that\none can use richer set of operators than simple delta (this is IIRC\ntouched in Darcs manual).  But first, nobody implemented such thing,\neven as proof-of-concept prototype.  Second, I GUESS that _in practice_\nthis would require something like Darcs' theory of patches and Darcs'\npatch commutation, and to merge strategy which would have to take\ninto account _whole_ history (and not only endpoints: branches to\nbe merged and common ancestor(s) like in 3-way merge)... which could\nand would lead to exponential time; extremely bad performance.  But\nthat is just my guess.\n\n> ---------------------------\n> \n> > The second could be called \"location\".  Which file should this patch  \n> > be applied to?  Which lines within a file should this hunk be applied  \n> > to?  I argue in [2] that Darcs does strictly better at the task of  \n> > location than do SVN or GNU diff3. [...]\n> \nDoes Darcs still have bug with exponential time of some merges?\n\n> How are you defining \"provably correct\"?  In order to show\n> correctness, you need to define what correctness means.\n\nAdditionally if it would be possible to decide and mathematically\ndefine what is correct way of merge resolution, it still wouldn't\nhelp in the case of semantic conflict.  And not tool would be\never able to solve it.\n\n>\n> One approach is that you force the user to tell you [...]\n\nAnd this is of course _wrong_ solution from the usability point of\nview.  Well, at least I think it is.\n\n> > Let's be careful not to lump these three things together, say \"All  \n> > merging involves guessing.\", and thus overlook the interesting fact  \n> > that some merge algorithms involve strictly more guessing than do  \n> > others.\n> \n> Part of the problem is that words like \"guessing\" and claims of some\n> algorithm being \"provably correct\" are basically marketing words.\n> They are generally used to denigrate one SCM, and promote someone's\n> favorite SCM as being **better**.\n> \n> Fundamentally, the goal of merging is to Do The Right Thing --- from a\n> semantic point of view, which means that the user's intentions is\n> what's important at the end of the day.  The question then is whether\n> you record the user's intentions, or try to determine it in from a\n> heuristics point of view.\n\nAs I wrote: sometimes heuristic algorithms are better; take for\nexample the NP-C 'travelling salesman' problem...\n\nAnd what matters is if given merge strategy (algorithm) is good\nin practice, and not if it is good in theory...  (I'm repeating\nmyself.)\n\n> \n> The people who claim that recording the user's intentions is superior\n> will claim that you can never know for sure what the user meant, so\n> you have to ask him or her to provide that information.  In some cases\n> that's relatively easy; you require the user to use commands like \"bk\n> mv\" and \"bk cp\" and \"bk rm\" which not only performs the function, but\n> also records the user's intention.  Unfortunately, if you are applying\n> a patch, and the patch file hasn't been enhanced to carry this kind of\n> information, you have to use heuristics and then somehow get the user\n> to confirm them --- hence the use of the Annoying Popup Dialog Box.\n\nOrdinary patches, filemanages, your favorite editor (if it doesn't\nhave support for 'rename recording' VCS you use)...\n\nBTW. using heuristic to find renames and record information about\nrenames during commit wouldn't help in the case if the file was\nindependently added but for example under different filename: for\nthis is I think necessary to have rename detection at merge time.\n\n\n[...]\n> And yet, people seem to focus on recording of user intention because\n> it reflects some holy grail of Perfection and Correctness.  And maybe,\n> because it is easily measurable, whereas usability and improving\n> programmer productivty are inherently more subjective measures.\n> What's very sad are the people who are feel profoundly hurt that they\n> spent a huge amount of their life working on SCM Correctness, only to\n> find that people chose other SCM's based on other metrics and other\n> issues other than the one that they felt was most important.\n> Unfortunately there's not a lot that can be done about that.\n> \n> \t\t      \t       \t      - Ted\n\nErm... this certainly shows that discussion on uvc-reviewers mailing\nlist drifted _badly_ away from what \"Understanding Version Control\"\nis about :-( \n\n\nP.S. I hoped that Linus who is strong proponent of 'contents is the\n     king' and superiority of rename detection would write something\n     but perhaps it was too long...\n-- \nJakub Narebski\nPoland\n"},{"id":"103215","messageId":"200902042314.50690.jnareb@gmail.com","threadId":"17510","inReplyTo":"200902021948.54700.jnareb@gmail.com","subject":"Tests for \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-04T22:14:49Z","receivedAt":"2009-02-04T22:14:49Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Sometimes code speaks louder than words. I think it is the case in\nunderstanding how intelligent merge should deal with the presence\nof renames; well commented examples (use cases) are (might be?) better\nthan talking about it.\n\nInstead of talking about 'container identities' it would be, in my\nopinion, to examine what challenges version control system has to\novercome if it wants to tell that it supports renames.  Example tests,\nexample use cases might be a good choice.  With those examples we\ncan check what are the limitations, strong and weak sides of different\nways of dealing with renames: container identities (or 'file-ids'),\ntracking rename information, and heuristic rename detection.\n\n\nBelow there are proposed tests which are meant to check how good merge\nalgorithm (strategy) used by given SCM supports renames.  In all cases\nwe have two branch: branch 'a' (ours) and branch 'b' (theirs), and we\nusually would be merging branch 'b' into (ours) branch 'a'.\n\nNote that for Git we want example files be large enough that\nsimilarity based heuristic rename detection works.  At the bottom\nthere are proposed files, taken from t/t6022-merge-rename.sh test\nfrom git.git repository (which, along with t/t6032-merge-large-rename.sh,\nmight be a good start for more comprehensive test suite for SCMs).\n\n * merging renames: if one side renamed file you should get rename\n   on merge; renaming a file and then merging that rename.\n\n     <switch to branch b>\n     [on branch b]$ scm mv foo bar\n     [on branch b]$ scm commit ... # to commit file rename\n\n     <switch to branch a>\n     [on branch a]$ scm commit ... # to not have fast-forward case\n     [on branch a]$ scm merge b    # merge branch b (with rename)\n\n   expected result:\n     you have file 'bar', and do not have file 'foo'\n\n * applying change to correct file: if our side renamed a file (or\n   rename directory it is in, which does rename full pathname of a\n   file indirectly), and possibly change it, and the other side\n   changed file, we would want merge to bring changes to file after\n   rename.\n\n     <switch to branch a>\n     [on branch a]$ scm mv foo bar\n     [on branch a]$ edit bar && scm commit # optionally\n\n     <switch to branch b>\n     [on branch b]$ edit foo\n     [on branch b]$ scm commit -m 'FOO'    # commit changes\n\n     <switch to branch a>\n     [on branch a]$ scm merge b\n\n   expected result: \n     you have changes made on branch 'b' to file 'foo'\n     (commit 'FOO') in file 'bar'\n\n   Note that like in example in previous item all operations take place\n   _after_ branching point (after creation of branch b off branch a).\n\n   This is I guess what most people think when talking about\n   rename-aware (intelligent) merging.\n\n * renamed directories bring another complication (described for example\n   on Mark Shuttleworth blog in articles about DVCS, promoting Bazaar-NG),\n   namely how to deal with merging changes where other side creates\n   _new files_ in renamed directory.\n\n     <switch to branch a>\n     [on branch a]$ scm mv subdir-foo/ subdir-bar/\n\n     <switch to branch b>\n     [on branch b]$ scm add subdir-foo/baz   # add new file in old dir\n\n     <switch to branch b>\n     [on branch a]$ scm merge b\n\n   expected result:\n     New file subdir-bar/baz\n     Either automatic merge, or a conflict (no commit)\n\n   There is a bit of controversy about this feature, as for example in\n   some programming languages (e.g. Java) or in some project build tool\n   info it is not posible to simply move a file (or create new file in\n   different directory) without changing file contents.  Some say that\n   is better to fail than to do wrongly clean merge.\n\n * independent adding of a file: this is the case where both sides add\n   the same (or nearly the same) file independently, so the file in\n   question doesn't have common ancestor in per-file history.  It\n   might happen because of applying patch independently, for example.\n   I _suspect_ that 'file-id' based solutions would have problems...\n\n   Below there is table of cases that might happen:\n\n              |  1  |  2  |  3  |  4  |  5  |\n     ---------|-----|-----|-----|-----|-----|\n     filename |  =  |  =  | !=  |  =  | !=  |\n     contents |  =  | !=  |  =  | ~=  | ~=  |\n\n   where \n     '=' means that both sides use the same filename, \n         or exactly the same contents; \n    '!=' means files have different contents,\n         or files _started_ with different names\n    '~=' means that sides have slightly different contents,\n         but similarity score is high enough for rename detection.\n\n   Let us examine most complicated case 5 in above table; one can\n   simply omit some commends to get cases 1, 3 and 4.  Note that\n   COPYING file is GNU GPL text.\n\n     <switch to branch a>\n     [on branch a]$ cp ../COPYING COPYING\n     [on branch a]$ scm add COPYING\n     [on branch a]$ scm commit -m 'Added COPYING'\n\n     <switch to branch b>\n     [on branch b]$ cp ../COPYING LICENSE   # optional rename\n     [on branch b]$ sed -e 's/HOWEVER/However/' <LICENSE >LICENSE.1 &&\n                    mv -f LICENSE.1 LICENSE # optional change\n     [on branch b]$ scm add LICENSE\n     [on branch b]$ scm commit -m 'Added LICENSE'\n     [on branch b]$ scm mv LICENSE COPYING\n     [on branch b]$ scm commit -m 'Renamed LICENSE to COPYING'\n\n     <switch to branch a>\n     [on branch a]$ scm merge b\n\n   expected result: \n     Either clean merging changes from branch 'b', or cleanly\n     marked conflict, e.g. CONFLICT (add/add).  What we do not\n     want is one side vanishing.\n\n   But that series of commands was only a preparation.  We now\n   want to repeat merging from branch 'b' into branch 'a', or\n   do reverse merging 'a' into 'b'.\n\n     <switch to branch b>\n     [on branch b]$ sed -e 's/GPL/G.P.L/g' <COPYING >COPYING1 &&\n                    mv -f COPYING COPYING\n     [on branch b]$ scm commit COPYING -m 'Changed COPYING'\n\n     <switch to branch a>\n     [on branch a]$ scm commit ... # to not have fast-forward case\n     [on branch a]$ scm merge b    # merge branch b\n\n   expected result:\n     Clean merge of changes done on branch 'b' into COPYING\n\n\n==================================================\n##################################################\n==================================================\n\nFiles for testing rename detection in Git\n\n........................................\n\nt/t6022-merge-rename.sh\nt/t6022-merge-rename-nocruft.sh\n\ncat >A <<\\EOF &&\na aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\nb bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\nc cccccccccccccccccccccccccccccccccccccccccccccccc\nd dddddddddddddddddddddddddddddddddddddddddddddddd\ne eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee\nf ffffffffffffffffffffffffffffffffffffffffffffffff\ng gggggggggggggggggggggggggggggggggggggggggggggggg\nh hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh\ni iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii\nj jjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj\nk kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk\nl llllllllllllllllllllllllllllllllllllllllllllllll\nm mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm\nn nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn\no oooooooooooooooooooooooooooooooooooooooooooooooo\nEOF\n\ncat >M <<\\EOF &&\nA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nB BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB\nC CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC\nD DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD\nE EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE\nF FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF\nG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG\nH HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH\nI IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII\nJ JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ\nK KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK\nL LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL\nM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM\nN NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN\nO OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO\nEOF\n\nsed -e \"/^g /s/.*/g : master changes a line/\" <A >A+\nsed -e \"/^g /s/.*/g : white changes a line/\" <A >B\nsed -e \"/^G /s/.*/G : colored branch changes a line/\" <M >N\nsed -e \"/^g /s/.*/g : red changes a line/\" <A >B\nsed -e \"/^G /s/.*/G : colored branch changes a line/\" <M >N\n[...]\n\n\n-- \nJakub Narebski\nPoland\n"},{"id":"103244","messageId":"20090204235436.GA8945@mit.edu","threadId":"17510","inReplyTo":"200902040304.05028.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-04T23:54:36Z","receivedAt":"2009-02-04T23:54:36Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, Feb 04, 2009 at 03:04:02AM +0100, Jakub Narebski wrote:\n> \n> I guess that this mailing list is subscribe-only, isn't it?  So doing\n> CC to uvc-reviewers wouldn't, unfortunately, cut?\n\nAccording to the Wayback Archive's record of the uvc-reviewers mailman\nlistinfo was open for anyone to join, and the archives were public,\nwhich is why I don't mind sharing the archives with anyone who asks.\n\n> > I'll include some of my writings on the subject from the uvc-reviewers\n> > mailing list so folks can see where some of this discussion went last\n> > time...  (All of this dates from January, 2008, when Eric was last\n> > aggressively updating the paper in question.)\n> \n> Thank you very much for those excerpts / fragments, even though\n> I'd rather have your fresh comments either on current state of \n> \"Understanding Version-Control Systems\", or on my post.\n\nMy comments haven't changed; as you probably noted, I agree with you,\nand my arguments largely parallel yours.  I was using a Reductio ad\nabsurdum argument to show that the same argument that claims that Git\nis a primitive, hackish, SCM because it doesn't record user intention\nvis-a-vis file renames could also be extended to say that use of all\ncurrent DSCM's amount to \"Programming Malpractice\" because they don't\nallow the recording of higher level \"user intentions\" such as the\nrenaming of variables, functions, types, and class names.\n\nMy comments date from the very end of January 2008, when Eric stopped\nupdating his paper, and before he could start doing an extensive\ndescription and evaluation of bzr, Mercurial and Git, so it's not\nsurprising that they are still relevant today.  I suspect that when he\npicks up this draft again, and starts writing these sections covering\nmodern distributed SCM's, the sections for Mercurial, Git, Bzr,\net. al, will cause a huge amount of controversy, because even though\nhe is claiming to be unbaised, there is very clear in the draft to\ndate that he would very much like to draw a grand sweeping picture of\nprogress and evolution starting from \"first generation systems\" (RCS,\nSCCS, et. al), to \"second generation systems\" (CVS, SVN, et. al), to\n\"third generation systems\" (Arch, Monotone, git, Mercurial, etc.)\n\nThere are hints in the draft that he views \"container identity\" has\nthe next \"evolutionary idea\" which \"more primitive\" systems do not\nhave, and \"more evolved\" systems do have.  This can be seen from this\nexcerpt from his draft:\n\n\tFirst wrong assumption: Conflict resolution by merging is\n\tintractably difficult, so we'll have to settle for locking. It\n\ttook at least fifteen and arguably twenty years for VCS\n\tdesigners to get shut of that one. But it's historical now.\n\n\tSecond wrong assumption: Change history representation as a\n\tsnapshot sequence is perfectly dual to the representation as\n\tchange/add/delete/rename sequences.. This folk theorem is well\n\texpressed in the 2004 essay On Arch and Subversion. It is\n\tappealing, widely held, and dead wrong.\n\n\tFile renames break the apparent symmetry. The failure of\n\tsnapshot-based models to correctly address this has caused\n\tendless design failures, subtle bugs, and user misery.\n\nSo you can see that Eric seems to believe quite strongly that the\nfailure to track file renames is as fundamental an error as what he\nterms the \"First Wrong Assumption\".  He later admits that the idea is\ncontroversial, and that people are still \"grapling\" with it, but I\nthink he's tipped his hand about what he believes the ultimate correct\nanswer is with respect to this issue.\n\nI believe, as I think you do, that the hysteria that states that you\n*must* record user intention leads inexorably to the requirement to\nforce users to indicate \"intention\" by popping up Annoying Dialog\nBoxes whenever they suck in a patch that was sent via e-mail so that\nthe SCM can record information about whether a file rename had\nhappened in a particular commit.  I believe this requirement to do\nrecord user intentions and to pop up these Annoying Dialog Boxes is a\nblind alley ala the vast amount of time wasted arguing over algorithms\nsuch as Codeville precise merges.  I also believe that forcing users\nto record \"user intention\" makes about as much sense as forcing users\nto declare they are about to edit a file by explicitly taking locks on\nfiles ala RCS.\n\nI suspect Eric will disagree with me, but regardless of how he\ncompletes his paper, it will almost certainly end up taking sides one\nway or another on this controversy, at which point one side or the\nother of this particular disagreement will argue that Eric is really\nwriting an advocacy paper pushing Bzr, Mercurial, or Git (depending on\nhow he comes out on this issue).\n\n\nYour suggestion that the proof is going to be in the code makes a lot\nof sense.  The examples I would suggest that we create, and then\ndemonstrate (or make enhancements to git) so that it can handle these\nreal world examples are:\n\n1) In branch A, the directory src/plugin/innodb-experimental is\n   renamed to src/plugin/innodb, and in branch B, a commit (i)\n   modifies a file src/plugin/innodb-experimental/table.c, and (ii)\n   creates a file src/plugin/innodb-experimental/mod-schema.c.  This\n   commit in branch B is then pulled into branch A, where the\n   directory rename has taken place.  The user may not know that a\n   directory rename had taken place under the covers, so they don't\n   give any magic options when they run the \"git cherry-pick\" or \"git\n   merge\" command.  Does the right thing happen such that the right\n   file in src/plugin/innodb is modified, and the new file is created\n   in src/plugin/innodb, even though in the original commit, the\n   changes were made to files in src/plugin/innodb-experimental?\n\n2) And does the right thing happen if the situation is as described\n   above, but in, branch C, which is descended from branch B, a new\n   directory, src/plugin/innodb-experimental is created, such that\n   src/plugin/innodb and src/plugin/innodb-experimental both exist.\n   Now the same commit from branch A is pulled into branch C.  Will\n   the correct thing happen in that the correct files in\n   src/plugin/innodb are modified and created, even though there is a\n   new directory containing a completely unrelated plugin that happens\n   to have the name, \"innodb-experimental\"?\n\n   BTW, it has been asserted that there exists at least one major open\n   source project where this sort of thing happens quite often, and\n   the fact that git did not do the right thing in these conditions\n   was a factor their choosing another DSCM.\n\n> Or \"Detecting [Wholesame] Directory Renames\"... which can be done\n> using 'rename detection' paradigm, and we have patches to prove it![4]\n> but unfortunately code didn't made it (yet!) into git.  And it can,\n> I think, deal with splitting files into two directories, something\n> which I guess in 'container identity' (directory-id) based solution\n> is simply impossible\n\nIt may be that Yann Dirson's patches will handle case (1) above.\nHandling case (2) is much harder, especially without slowing\neverything down massively, since it would effectively mean needing to\nlooking for directory renames along every single commit on the branch.\n(This would obviously have to be cached in some cache file.)\n\nIt can be done, I'm sure, but it would require a lot of code to get\nright.  Whether or not it's worth it is a question which is open to\ndebate, but I believe the bzr folks have asserted that bzr can handle\nboth cases (1) and (2) above, and there are some folks who apparently\ncare.  \n\nWhether or not a particular open source project will really and truly\nrun into this problem is a different question, and one can argue that\nrenaming plugins, and then creating new plugins with the same name as\nolder plugins that have since been renamed will lead to programmer\nconfusion, and so that's a good enough reason to avoid doing such\ncrazy things.  Unfortunately, you know how some programmers\nare.... telling someone they shouldn't do something is often an\ninvitation to do exactly what you tell them is a bad idea, and then\nthey complain when your filesystem or your DSCM doesn't handle that\ncase particularly gracefully.\n\n\t\t\t\t\t\t- Ted\n"},{"id":"103245","messageId":"7v4oz9lpes.fsf@gitster.siamese.dyndns.org","threadId":"17510","inReplyTo":"20090204235436.GA8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-05T00:04:43Z","receivedAt":"2009-02-05T00:04:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@mit.edu> writes:\n\n> 1) In branch A, the directory src/plugin/innodb-experimental is\n>    renamed to src/plugin/innodb, and in branch B, a commit (i)\n>    modifies a file src/plugin/innodb-experimental/table.c, and (ii)\n>    creates a file src/plugin/innodb-experimental/mod-schema.c.  This\n>    commit in branch B is then pulled into branch A, where the\n>    directory rename has taken place.  The user may not know that a\n>    directory rename had taken place under the covers, so they don't\n>    give any magic options when they run the \"git cherry-pick\" or \"git\n>    merge\" command.  Does the right thing happen such that the right\n>    file in src/plugin/innodb is modified, and the new file is created\n>    in src/plugin/innodb, even though in the original commit, the\n>    changes were made to files in src/plugin/innodb-experimental?\n\nCareful.\n\nAlthough it is reasonable to expect that existing file's modification will\nmove to innodb/ directory, it is not as clear-cut as some people seem to\nassume that the new file should always be created in the new directory\ninnodb/.  You seem to imply you understand the issues by having the second\nexample, though.\n"},{"id":"103246","messageId":"200902050108.14033.jnareb@gmail.com","threadId":"17510","inReplyTo":"20090204235436.GA8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-05T00:08:13Z","receivedAt":"2009-02-05T00:08:13Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, 5 Feb 2009, Theodore Tso wrote:\n> On Wed, Feb 04, 2009 at 03:04:02AM +0100, Jakub Narebski wrote:\n> > \n> > I guess that this mailing list is subscribe-only, isn't it?  So doing\n> > CC to uvc-reviewers wouldn't, unfortunately, cut?\n> \n> According to the Wayback Archive's record of the uvc-reviewers mailman\n> listinfo was open for anyone to join, and the archives were public,\n> which is why I don't mind sharing the archives with anyone who asks.\n\nWhat I meant here was whether to send email to uvc-reviewers mailing\nlist I have to be subscribed or not?  Because if you have to be\nsubscribed, even if it is open subscription and not by invitation only,\nit would make it impossible to have discussion on git mailing list as\na main venue, and just CC: (send copies) to uvc-reviewers list.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"103247","messageId":"20090205004905.GE8945@mit.edu","threadId":"17510","inReplyTo":"200902050108.14033.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-05T00:49:06Z","receivedAt":"2009-02-05T00:49:06Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Thu, Feb 05, 2009 at 01:08:13AM +0100, Jakub Narebski wrote:\n> On Thu, 5 Feb 2009, Theodore Tso wrote:\n> > On Wed, Feb 04, 2009 at 03:04:02AM +0100, Jakub Narebski wrote:\n> > > \n> > > I guess that this mailing list is subscribe-only, isn't it?  So doing\n> > > CC to uvc-reviewers wouldn't, unfortunately, cut?\n> > \n> > According to the Wayback Archive's record of the uvc-reviewers mailman\n> > listinfo was open for anyone to join, and the archives were public,\n> > which is why I don't mind sharing the archives with anyone who asks.\n> \n> What I meant here was whether to send email to uvc-reviewers mailing\n> list I have to be subscribed or not?  Because if you have to be\n> subscribed, even if it is open subscription and not by invitation only,\n> it would make it impossible to have discussion on git mailing list as\n> a main venue, and just CC: (send copies) to uvc-reviewers list.\n\nWell, given that the mailman web interface is dead and gone, I'm\nguessing the mailing list itself is also dead, so the point is rather\nmoot.  I assume that if Eric thought the list is useful, he'll set it\nup again when he starts active development on his paper again --- or\nnot.\n\n\t\t\t\t\t\t- Ted\n"},{"id":"103265","messageId":"20090205024333.GH8945@mit.edu","threadId":"17510","inReplyTo":"7v4oz9lpes.fsf@gitster.siamese.dyndns.org","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-05T02:43:33Z","receivedAt":"2009-02-05T02:43:33Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, Feb 04, 2009 at 04:04:43PM -0800, Junio C Hamano wrote:\n> Theodore Tso <tytso@mit.edu> writes:\n> \n> > 1) In branch A, the directory src/plugin/innodb-experimental is\n> >    renamed to src/plugin/innodb, and in branch B, a commit (i)\n> >    modifies a file src/plugin/innodb-experimental/table.c, and (ii)\n> >    creates a file src/plugin/innodb-experimental/mod-schema.c.  This\n> >    commit in branch B is then pulled into branch A, where the\n> >    directory rename has taken place.  The user may not know that a\n> >    directory rename had taken place under the covers, so they don't\n> >    give any magic options when they run the \"git cherry-pick\" or \"git\n> >    merge\" command.  Does the right thing happen such that the right\n> >    file in src/plugin/innodb is modified, and the new file is created\n> >    in src/plugin/innodb, even though in the original commit, the\n> >    changes were made to files in src/plugin/innodb-experimental?\n> \n> Careful.\n> \n> Although it is reasonable to expect that existing file's modification will\n> move to innodb/ directory, it is not as clear-cut as some people seem to\n> assume that the new file should always be created in the new directory\n> innodb/.\n\nCareful; that's actually an argument for recording the directory\nrename.  If the intention is to rename the directory containing some\nplugin, where all of the associated files are for the plugin foobar,\nand we are renaming the directory because plugin has had its name\nchanged to fooblatz, then a commit which introduces a new file, say\ntable.c probably does want to get created in the new directory ---\nespecially if one of the changes was to foobar/Makefile:\n\n--- Makefile.in\t  2009-02-04 21:28:43.977052347 -0500\n+++ Makefile.in.orig\t     2009-02-04 21:28:38.830212569 -0500\n@@ -60,7 +60,7 @@\n #\n #MCHECK= -DMCHECK\n \n-OBJS= table.o crc32.o dict.o unix.o pass1.o pass1b.o pass2.o \\\n+OBJS= crc32.o dict.o unix.o pass1.o pass1b.o pass2.o \\\n       pass3.o pass4.o pass5.o journal.o badblocks.o util.o dirinfo.o \\\n       dx_dirinfo.o ehandler.o problem.o message.o recovery.o region.o\n \nIn other cases, maybe the right thing *is* to drop the new file in the\noriginal directory.  So as the Hg and Bzr apologists might say, if the\nSCM actually records whether the user intention was a *directory*\nrename, versus a series of *file* rename/moves, then it becomes\nobvious what the right thing to do.\n\nBut if the SCM is tracking *content* as git does, then we don't have\nthe benefit of having recorded the user's intention, so we have to use\nhueristics.  We can say that if *all* of the files in the directory\nfoobar had been moved to fooblatz as part of a commit, it's likely\nthat new commits that create new files in foobar should create them in\nfooblatz --- especially if the new file name is mentioned in a file\nnamed \"Makefile\" or \"Makefile.in\".  Or we can give up and ask the user\nwhat was intended at merge time.  That's admittedly as annoying as\nthrowing up the Annoying Dialog Box at commit time when folding in a\npatch (which is how systems that record user intention have to do\nthings).  Ultimately, there are only three choices; either the user\ntells the SCM, the SCM asks the user, or the SCM applies some\nhueristic (i.e., the SCM \"guesses\").  This can happen at commit time\nor it can happen at merge time, but and in some cases the SCM can use\na combination of techniques depending on whether a patch is being\nimported or whether the user is explicitly telling the scm via \"scm\nmv\", \"scm mvdir\", \"scm cp\", et. al.\n\nBut if we throw up our hands and say, it's impossible to guess\ncorrectly, then that leaves us open to the argument from people from\nthe Hg or Bzr camp to say, \"You see, tracking *content* is really for\nthe birds; you really have to let/force the user to *tell you* and\nrecord the whether the directory is being moved, or a series of files\nin the directory are being moved.\"\n\nInstead, I would argue that just as we've shown that while collisions\ncan happen, the costs of locking files ala RCS outweigh the costs, it\nalso is better to use hueristics to determine whether a file should be\ncreated in the original directory or the apparently renamed directory\nlocation, and then let the user fix things up afterwards if the\nalgorithm gets it wrong.  If the corner case only happens 1% of the\ntime, and our algorithm gets it right 99% of the time, the resulting\n0.01% error rate is probably quite acceptable.\n\n      \t  \t    \t  \t     \t\t - Ted\n"},{"id":"103271","messageId":"buo63jpo21o.fsf@dhlpc061.dev.necel.com","threadId":"17510","inReplyTo":"20090204235436.GA8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2009-02-05T06:01:07Z","receivedAt":"2009-02-05T06:01:07Z","isPatch":false,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"Theodore Tso <tytso@mit.edu> writes:\n> I suspect Eric will disagree with me, but regardless of how he\n> completes his paper, it will almost certainly end up taking sides one\n> way or another on this controversy, at which point one side or the\n> other of this particular disagreement will argue that Eric is really\n> writing an advocacy paper pushing Bzr, Mercurial, or Git (depending on\n> how he comes out on this issue).\n\nThat was pretty clear from his comments on the emacs-devel mailing list\n(2008-05 roughly).\n\nHe spent a lot of time trying to sound impartial (and that he was \"still\ndoing research\"), but strongly gave the impression that he had already\nmade up his mind.\n\n-Miles\n\n-- \nInfancy, n. The period of our lives when, according to Wordsworth, 'Heaven\nlies about us.' The world begins lying about us pretty soon afterward.\n"},{"id":"103273","messageId":"7vmyd1ieo6.fsf@gitster.siamese.dyndns.org","threadId":"17510","inReplyTo":"20090205024333.GH8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-05T06:24:57Z","receivedAt":"2009-02-05T06:24:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@mit.edu> writes:\n\n> Careful; that's actually an argument for recording the directory\n> rename.\n\nI do not think so.  More precisely, I can see people could make that\nargument, but I think that argument is weak.\n\nSuppose the original project's implementor only knew about innodb\ninterface, so he had the \"database interface\" directory and innodb access\nmethod file in the source tree, perhaps at <db/inno.c>.\n\nI forked the project, and added gdbm support at <db/gdbm.c>.\n\nYou also forked the project without knowing what I was working on, and you\nstarted working on refining the innodb support.\n\nAll the while, the development community started discussing how the source\ntree should be organized to support multiple backends, and you learned\nthat the plan is to have one directory per larger backend, while keeping\nsingle file ones in <db/*.c>.  Specifically, you learned that innodb\nrelated code will be stored in <innodb/*.c>, and there may be other\n<somedb/*.c> and <someotherdb/*.c> groups added, but you are not\ninterested in anything but enhancing innodb support.\n\nYou rename \"scm mv db innodb\" and then add <innodb/enhanced.c>, or perhaps\nyou may have done it the other way, i.e. added <db/enhanced.c> and then\nrenamed \"scm mv db innodb\".\n\nSuppose you would want to merge my changes, but the upstream's plan hasn't\nhappened yet.  Neither of us merged from the upstream in the meantime.\n\nRecording your \"scm mv db innodb\" as \"the user's intention to rename\ndirectory\" does not help when you want to merge with me to handle the new\nfile <db/gdbm.c> I added.  You not only need to record the \"intent to\nrename db to innodb\", but need to know that the validity of that \"intent\nto rename\" is contingent on the absense of anything unrelated to innodb in\ndb/ directory, in order to merge the two branches correctly.  Otherwise\nyou will end up moving my <db/gdbm.c> to <innodb/gdbm.c>.  The correct\noutcome in this case would probably be to leave it as it is.\n\n> In other cases, maybe the right thing *is* to drop the new file in the\n> original directory.  So as the Hg and Bzr apologists might say, if the\n> SCM actually records whether the user intention was a *directory*\n> rename, versus a series of *file* rename/moves, then it becomes\n> obvious what the right thing to do.\n\nSee how that argument is flawed?  The point of my example is that the line\nbetween your example (1) and (2) in the previous message is blurry.\n"},{"id":"103317","messageId":"20090205093415.GC20844@thyrsus.com","threadId":"17510","inReplyTo":"buo63jpo21o.fsf@dhlpc061.dev.necel.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Eric S. Raymond","fromEmail":"esr@thyrsus.com","sentAt":"2009-02-05T09:34:15Z","receivedAt":"2009-02-05T09:34:15Z","isPatch":false,"sender":{"key":"esr@thyrsus.com","avatar":"https://avatars.githubusercontent.com/u/727961?v=4"},"body":"Miles Bader <miles@gnu.org>:\n> Theodore Tso <tytso@mit.edu> writes:\n> > I suspect Eric will disagree with me, but regardless of how he\n> > completes his paper, it will almost certainly end up taking sides one\n> > way or another on this controversy, at which point one side or the\n> > other of this particular disagreement will argue that Eric is really\n> > writing an advocacy paper pushing Bzr, Mercurial, or Git (depending on\n> > how he comes out on this issue).\n> \n> That was pretty clear from his comments on the emacs-devel mailing list\n> (2008-05 roughly).\n> \n> He spent a lot of time trying to sound impartial (and that he was \"still\n> doing research\"), but strongly gave the impression that he had already\n> made up his mind.\n\nAt the time, I leaned slightly towards Mercurial, but my reasons had\nnothing to do with the cluster of issues Ted is pointing at; rather, I\nliked hg for its interface simplicity.\n\nI remain agnostic about the deep issues around renaming and user\nintentions - in part because I'm by no means sure I completely\nunderstand them yet.\n-- \n\t\t<a href=\"http://www.catb.org/~esr/\">Eric S. Raymond</a>\n"},{"id":"103323","messageId":"200902051223.38992.jnareb@gmail.com","threadId":"17510","inReplyTo":"20090204235436.GA8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-05T11:23:37Z","receivedAt":"2009-02-05T11:23:37Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, Feb 05, 2009, Theodore Tso wrote:\n> On Wed, Feb 04, 2009 at 03:04:02AM +0100, Jakub Narebski wrote:\n\n> My comments date from the very end of January 2008, when Eric stopped\n> updating his paper, and before he could start doing an extensive\n> description and evaluation of bzr, Mercurial and Git,\n\nWhich evaluation is very important, as Git, Mercurial (hg) and Bazaar\n(bzr) in addition to Subversion (svn) dominate the field of open-source\nversion control systems, with Darcs and Monotone having its own niches.\n\n> so it's not \n> surprising that they are still relevant today.  I suspect that when he\n> picks up this draft again, and starts writing these sections covering\n> modern distributed SCM's, the sections for Mercurial, Git, Bzr,\n> et. al, will cause a huge amount of controversy, because even though\n> he is claiming to be unbiased, there is very clear in the draft to\n> date that he would very much like to draw a grand sweeping picture of\n> progress and evolution starting from \"first generation systems\" (RCS,\n> SCCS, et. al), to \"second generation systems\" (CVS, SVN, et. al), to\n> \"third generation systems\" (Arch, Monotone, git, Mercurial, etc.)\n\nThere is progress and evolution... \n * locking -> update-then-commit (or merge-then-commit) -> \n   -> commit-then-merge (and alternate/additional workflow of rebase\n   aka. commit-merge-recommit-push)\n * local -> client / server -> distributed\n\nPerhaps also\n ? per-file history -> whole tree commits\n\nBut there are still controversial issues, like discussed here issue\non _how_ to deal with renames.\n\n> \n> There are hints in the draft that he views \"container identity\" has\n> the next \"evolutionary idea\" which \"more primitive\" systems do not\n> have, and \"more evolved\" systems do have.  This can be seen from this\n> excerpt from his draft:\n> \n> \tFirst wrong assumption: Conflict resolution by merging is\n> \tintractably difficult, so we'll have to settle for locking. It\n> \ttook at least fifteen and arguably twenty years for VCS\n> \tdesigners to get shut of that one. But it's historical now.\n\nThere I think everybody would agree.  Modern VCS rare, if even, have\nsupport for locking model.\n\n> \n> \tSecond wrong assumption: Change history representation as a\n> \tsnapshot sequence is perfectly dual to the representation as\n> \tchange/add/delete/rename sequences.. This folk theorem is well\n> \texpressed in the 2004 essay On Arch and Subversion. It is\n> \tappealing, widely held, and dead wrong.\n> \n> \tFile renames break the apparent symmetry. The failure of\n> \tsnapshot-based models to correctly address this has caused\n> \tendless design failures, subtle bugs, and user misery.\n\nFirst, I have stressed already, the issue of 'container identities'\nfor dealing with renames is totally ORTHOGONAL to the issue whether\nSCM is snapshot based or changeset based.  Case in point: Bazaar (bzr).\nBazaar uses file-ids and directory-ids to deal with renames (here it\nis spiritual child of Arch), but on Bazaar wiki (http://bazaar-vcs.org)\nit is mentioned in the passing that it is _snapshot based_. I think\nthat it had those file-ids even when it used 'weave' in repository\nformat (not deltas / changesets).\n\nSecond, what I also wrote about already, the article cited as argument\nfor changeset based SCM (which you don't have in above excerpt) is not\nto the point, and moreover is totally, utterly _wrong_. The troubles\nwith merging in CVS and Subversion are not caused by the fact that they\nare snapshot based (CVS isn't, by the way), but by the fact that they\ndon't (or in the case of Subversion didn't) track merges.\n\n> \n> So you can see that Eric seems to believe quite strongly that the\n> failure to track file renames is as fundamental an error as what he\n> terms the \"First Wrong Assumption\".  He later admits that the idea is\n> controversial, and that people are still \"grapling\" with it, but I\n> think he's tipped his hand about what he believes the ultimate correct\n> answer is with respect to this issue.\n\nWhat I'd like to see in the next version of \"Understanding Version\nControl Systems\" is to concentrate more on the _issue_ of managing\nrenames, than on specific solution of this problem.  And I very much\nwould like to see 'rename detection' mentioned...\n\nBut I think that the issue of renames is not the main point. The main\npoint is that in modern VCS _merging_ has to be easy[1], from which\nnaturally follows that VCS needs intelligent merge which can deal well\nwith file renames.  Managing renames is needed for easy merging; all\nelse is glitter.\n\nOr, from the other point of view the important thing that _branching_\nis important. Both creating branches, and merging branches (and having\nlarge amount of branches, and being able to delete branches, and having\nlocal (unpublished) and global (published) branches, etc.).\n\n\nBTW. there is excerpt from Junio C. Hamano blog post \"FLOSS weekly #19\nfollow-up (3)\" http://gitster.livejournal.com/9970.html\n\n  By the time the basic structure as we currently know has stabilized,\n  we had help from literally dozens of contributors to add many things\n  on top of the very original version:\n\n  [...]\n  * We did not envision that multiple branches in a single repository\n    would turn out to be such a useful way to work, and did not have\n    support for switching branches.\n\n> \n[...]\n> I suspect Eric will disagree with me, but regardless of how he\n> completes his paper, it will almost certainly end up taking sides one\n> way or another on this controversy, at which point one side or the\n> other of this particular disagreement will argue that Eric is really\n> writing an advocacy paper pushing Bzr, Mercurial, or Git (depending on\n> how he comes out on this issue).\n\nI think, and I hope, that Eric would manage to keep proper scientific\ndecorum[2], balancing or at least mentioning all problems and all\npossible solutions, even if he is biased, and even if this bias shows\n(hopefully a little).\n\n[2] The thing that distinguish true science from cargo-cult science\n    (pseudo-science), which shows only arguments \"for\".\n\n> \n> \n> Your suggestion that the proof is going to be in the code makes a lot\n> of sense.  \n\nI though more about the fact that having 'use cases' examples would\nbe more clean. And also would make possible to test against...\n\n> The examples I would suggest that we create, and then \n> demonstrate (or make enhancements to git) so that it can handle these\n> real world examples are:\n> \n> 1) In branch A, the directory src/plugin/innodb-experimental is\n>    renamed to src/plugin/innodb, and in branch B, a commit (i)\n>    modifies a file src/plugin/innodb-experimental/table.c, and (ii)\n>    creates a file src/plugin/innodb-experimental/mod-schema.c.  This\n>    commit in branch B is then pulled into branch A, where the\n>    directory rename has taken place.  The user may not know that a\n>    directory rename had taken place under the covers, so they don't\n>    give any magic options when they run the \"git cherry-pick\" or \"git\n>    merge\" command.  Does the right thing happen such that the right\n>    file in src/plugin/innodb is modified, and the new file is created\n>    in src/plugin/innodb, even though in the original commit, the\n>    changes were made to files in src/plugin/innodb-experimental?\n\nThis (or similar, at least) example you can find in 'Tests for\n\"Understanding Version Control\" by Eric S. Raymond' subthread...\n\n> \n> 2) And does the right thing happen if the situation is as described\n>    above, but in, branch C, which is descended from branch B, a new\n>    directory, src/plugin/innodb-experimental is created, such that\n>    src/plugin/innodb and src/plugin/innodb-experimental both exist.\n>    Now the same commit from branch A is pulled into branch C.  Will\n>    the correct thing happen in that the correct files in\n>    src/plugin/innodb are modified and created, even though there is a\n>    new directory containing a completely unrelated plugin that happens\n>    to have the name, \"innodb-experimental\"?\n\nErrr... I think that you confused branch 'B' (with innodb-experimental)\nwith branch 'A' (with innodb only) here.\n\n> \n>    BTW, it has been asserted that there exists at least one major open\n>    source project where this sort of thing happens quite often, and\n>    the fact that git did not do the right thing in these conditions\n>    was a factor their choosing another DSCM.\n\nI think that they should change their filesystem hierarchy naming\nconventions and/or use branches more. But that is not terribly \nrelevant...\n\n> \n> > Or \"Detecting [Wholesame] Directory Renames\"... which can be done\n> > using 'rename detection' paradigm, and we have patches to prove it![4]\n> > but unfortunately code didn't made it (yet!) into git.  And it can,\n> > I think, deal with splitting files into two directories, something\n> > which I guess in 'container identity' (directory-id) based solution\n> > is simply impossible\n> \n> It may be that Yann Dirson's patches will handle case (1) above.\n> Handling case (2) is much harder, especially without slowing\n> everything down massively, since it would effectively mean needing to\n> looking for directory renames along every single commit on the branch.\n> (This would obviously have to be cached in some cache file.)\n\nWell, I think it would be a bit simpler: for each _new_ file in merge\nyou have to see where other files in the directory it was created are.\nBut I agree that it would be costly; perhaps it should be triggered by\nseparate option / config, like diff.renames = copies?\n\n> \n> It can be done, I'm sure, but it would require a lot of code to get\n> right.  Whether or not it's worth it is a question which is open to\n> debate, but I believe the bzr folks have asserted that bzr can handle\n> both cases (1) and (2) above, and there are some folks who apparently\n> care.  \n\nOn the other hand I think that fundamentally 'container identity'\nsolutions cannot deal with the case of splitting contents (or reverse,\njoining contents), e.g. splitting file into smaller files, or splitting\ndirectory into few directories (grouping files).  And from what\nI understand at least current implementations of 'file-id' solution\nhave problems with repeated merging in the case of independently added\nfile.\n\n> \n> Whether or not a particular open source project will really and truly\n> run into this problem is a different question, and one can argue that\n> renaming plugins, and then creating new plugins with the same name as\n> older plugins that have since been renamed will lead to programmer\n> confusion, and so that's a good enough reason to avoid doing such\n> crazy things.  Unfortunately, you know how some programmers\n> are.... telling someone they shouldn't do something is often an\n> invitation to do exactly what you tell them is a bad idea, and then\n> they complain when your filesystem or your DSCM doesn't handle that\n> case particularly gracefully.\n> \n> \t\t\t\t\t\t- Ted\n\nFinal words: there is no race. We aren't here to achieve world\ndomination. Sometimes one SCM, with its different choices, might\nbe better solution than the other.  For example if you have large\nmedia files then centralized SCM with partial checkout support might\nbe a best choice.  Another example is how we pointed on #git (sic!)\npeople from IPsec, who wanted each commit to be signed or equivalent,\ntowards Monotone.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"103328","messageId":"20090205131611.GJ8945@mit.edu","threadId":"17510","inReplyTo":"200902051223.38992.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-05T13:16:11Z","receivedAt":"2009-02-05T13:16:11Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Thu, Feb 05, 2009 at 12:23:37PM +0100, Jakub Narebski wrote:\n> > \n> > 2) And does the right thing happen if the situation is as described\n> >    above, but in, branch C, which is descended from branch B, a new\n> >    directory, src/plugin/innodb-experimental is created, such that\n> >    src/plugin/innodb and src/plugin/innodb-experimental both exist.\n> >    Now the same commit from branch A is pulled into branch C.  Will\n> >    the correct thing happen in that the correct files in\n> >    src/plugin/innodb are modified and created, even though there is a\n> >    new directory containing a completely unrelated plugin that happens\n> >    to have the name, \"innodb-experimental\"?\n> \n> Errr... I think that you confused branch 'B' (with innodb-experimental)\n> with branch 'A' (with innodb only) here.\n> \n\nNo, I didn't.   Let me try again.\n\nAt time T:\tProject grows a plugin in directory src/plugins/foo-new\n\nAt time T+1:\tProject releases a stable release, and branches off \"maint\"\n\nAt time T+2:\tProject renames the plugin to be src/plugins/foo, using\n\t\t\"scm mvdir src/plugins/foo-new src/plugins/foo\" on the \n\t\tdevel branch:\n\nAt time T+3:\tA developer wants to implement a new experimental\n\t\t'foo-new' plug in so she creates a completely new\n\t\tsrc/plugins/foo-new directory.  At this point the\n\t\tdevel branch has 'src/plugins/foo' and\n\t\t'src/plugins/foo-new', where src/plugins/foo contains\n\t\tthe plugin which is in the directory\n\t\tsrc/plugins/foo-new on the maint branch (since the\n\t\tmaint branch branched off\n   \t\tbefore the directory renames started happening.\n\nAt time T+4:\tA fix goes into the maint branch that modifies\n   \t\tsrc/plugins/foo-new/interface.c.    The fix needs to be\n   \t\tpulled into the devel branch.    Does the right thing\n   \t\thappen?   (Suppose \"interface.c\" is a commonly used \n\t\tfilename in all plugins and exists in both the 'foo'\n   \t\tand 'foo-new' directories on the devel branch.   Does\n   \t\tthe SCM figure out what is the correct file to\n   \t\tmodify?)\n\nAt Time T+5:\tA commit goes into the maint branch which creates a\n   \t\tnew file, src/plugins/foo-new/table.c, and modifies\n\t\tsrc/plugins/foo-new/Makefile to compile table.c.\n   \t\tWhich directory does the SCM drop table.c into?\n\nThie point is if the project is organized around plugins, which are\nconsidered bundles of code written in a modular way, and there is a\ndesire to rename directories which are the top-level modules, an SCM\nthat can easily deal with directory renames is important.  In practice\nthis doesn't come up in the Linux kernel, and many other OSS projects,\nand if the project's developent style doesn't do directory\nreorganizations often, then this isn't an issue.  If an OSS project\ndoes do this type of reorganization more frequently, then the argument\n\"don't do that\", would seem to be an unnecessary restriction.\n\n\t\t      \t\t     \t     \t  - Ted\n"},{"id":"103330","messageId":"20090205132814.GK8945@mit.edu","threadId":"17510","inReplyTo":"7vmyd1ieo6.fsf@gitster.siamese.dyndns.org","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-05T13:28:14Z","receivedAt":"2009-02-05T13:28:14Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, Feb 04, 2009 at 10:24:57PM -0800, Junio C Hamano wrote:\n> All the while, the development community started discussing how the source\n> tree should be organized to support multiple backends, and you learned\n> that the plan is to have one directory per larger backend, while keeping\n> single file ones in <db/*.c>.  Specifically, you learned that innodb\n> related code will be stored in <innodb/*.c>, and there may be other\n> <somedb/*.c> and <someotherdb/*.c> groups added, but you are not\n> interested in anything but enhancing innodb support.\n> \n> You rename \"scm mv db innodb\" and then add <innodb/enhanced.c>, or perhaps\n> you may have done it the other way, i.e. added <db/enhanced.c> and then\n> renamed \"scm mv db innodb\".\n\nThe argument would be that for SCM that properly tracked user\nintentions, you did the wrong thing.  If the SCM properly understood\ndirectory renames, there is a big differene between this:\n\n\tscm mvdir db innodb\n\nand this\n\n\tscm mv db/* innodb\n\nYou see?  The first moves the *directory* db to innodb.  The second\nmoves all of the *files* that are in db to a new directory, innodb.\nIf, in your example, you had learned that the goal was to keep single\nfile ones in <db/*.c>, and larger backends in <innodb/*.c>, the\ncorrect thing to tell the SCM is *not* to rename the directory db to\ninnodb, but rather, to move all of the files currently in <db/*.c>,\nwhich implement innodb, into the innodb directory.  If an SCM properly\nhandles directory renames, it would distinguish between these two\ncases and record them different, since it implies a different\nintention about what should happen to new files created in <db/*.c> in\nother branches when it comes time to merge them.\n\nOf course, this distinction does not exist in git, because we track\ncontent only.  And a number of other SCM's like Hg, which only track\nfile renames, wouldn't get this right either.  In order to get this\nright, you need to treat directory renames as separate and distinct\noperations from file renames, because they have different merge\nimplications.\n\n> See how that argument is flawed?  The point of my example is that the line\n> between your example (1) and (2) in the previous message is blurry.\n\nIt's blurry if you don't properly make the distinction between file\nand directory renames, yes.  A SCM that only handles file renames\ncan't record the difference between \"move all the files in directory\nfoo to bar\" from \"rename directory foo to bar\".  Just as an SCM (like\ngit) that only handles content that tell the difference between \"move\nall of the lines of content from foo.c to bar.c\" and \"rename foo.c to\nbar.c\".\n\nOur argument for git is that with sufficiently smart merge algorithms\nit doesn't matter, since we can intuit the right thing at merge time.\nHowever, your argument that it's not possible to determine whether the\nnew file should appear as db/gdbm.c or innodb/gdm.c is an argument\ncontent-tracking alone isn't enough. \n\nPersonally, I think the scenario I used of renaming plugins is more\nlikely that the sort of source reorganization which you've posited,\nbut I agree they are both possible scenarios.  The question for git\ndevelopment is whether these sorts of issues ar ones that we should\ntry to handle or not?  After all, one possibility is just to tell\npeople that if they are folks who like to go wild with source tree\nreorganizations all the time, they should go to some other SCM like\nbzr or Hg; that in git's view, the costs of being able to handle\nrandom file and directory renames isn't worth the benefits for what is\nnormally a rare occurrence (and if it's happening all the time, the\nproject is probably doing something else wrong....)\n\n\t\t\t\t\t\t- Ted\n"},{"id":"103357","messageId":"200902051836.44973.jnareb@gmail.com","threadId":"17510","inReplyTo":"20090205131611.GJ8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-05T17:36:42Z","receivedAt":"2009-02-05T17:36:42Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Thu, Feb 05, 2009, Theodore Tso wrote:\n> On Thu, Feb 05, 2009 at 12:23:37PM +0100, Jakub Narebski wrote:\n> > > \n> > > 2) And does the right thing happen if the situation is as described\n> > >    above, but in, branch C, which is descended from branch B, a new\n> > >    directory, src/plugin/innodb-experimental is created, such that\n> > >    src/plugin/innodb and src/plugin/innodb-experimental both exist.\n> > >    Now the same commit from branch A is pulled into branch C.  Will\n> > >    the correct thing happen in that the correct files in\n> > >    src/plugin/innodb are modified and created, even though there is a\n> > >    new directory containing a completely unrelated plugin that happens\n> > >    to have the name, \"innodb-experimental\"?\n> > \n> > Errr... I think that you confused branch 'B' (with innodb-experimental)\n> > with branch 'A' (with innodb only) here.\n> > \n> \n> No, I didn't.   Let me try again.\n> \n> At time T:\tProject grows a plugin in directory src/plugins/foo-new\n> \n> At time T+1:\tProject releases a stable release, and branches off \"maint\"\n> \n> At time T+2:\tProject renames the plugin to be src/plugins/foo, using\n> \t\t\"scm mvdir src/plugins/foo-new src/plugins/foo\" on the \n> \t\tdevel branch:\n\nAnd it is on branch 'A' that it happens.  But it doesn't matter...\nThe example is of 'independent add' in the same filename, different\ncontents case that I put in \"Tests for...\", but for directory not\nfor a filename.  Well, slightly more complicated than that...\n\nWhat I wonder is how directory-id solution deals with situation\nwhere (for example die to some reorganization) where once was single\ndirectory (e.g. lib/) now there are two (include/ and src/); how it\nwould deal with the new file at old directory, hmmm...?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"103397","messageId":"20090205214536.GB9814@mit.edu","threadId":"17510","inReplyTo":"200902051836.44973.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-02-05T21:45:36Z","receivedAt":"2009-02-05T21:45:36Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Thu, Feb 05, 2009 at 06:36:42PM +0100, Jakub Narebski wrote:\n> What I wonder is how directory-id solution deals with situation\n> where (for example die to some reorganization) where once was single\n> directory (e.g. lib/) now there are two (include/ and src/); how it\n> would deal with the new file at old directory, hmmm...?\n\nIn that case, it wouldn't be a directory rename, it would be a series\nof file moves.  So in a hypothetical scm that recorded all of these\nsorts of things, you'd have something like this:\n\nscm mv lib/*.c src\nscm mv lib/*.h include\nscm rmdir lib\n\nNow if you try merging in a commit that creates a files in lib (e.g.,\ncreates lib/foo.c and lib/foo.h and modifies lib/Makefile), presumably\neither a super smart hueristic algorith might be able to figure out\nthe pattern and drop the new files in src and include --- or, more\nlikely, it would flag a merge conflict and ask the user to figure it\nout by hand.\n\nSo yes, there will always be cases where directory-id won't be able to\nhandle a hypothetical source tree reorganization.  It really only\nhelps in the case where you are doing a true, full move of the\ndirectory, i.e.:\n\nscm mvdir src/plugin/innodb src/plugin/innodb-legacy\nscm mvdir src/plugin/innodb-experimental src/plugin/innodb\n\n\t\t\t\t\t\t\t- Ted\n"},{"id":"103403","messageId":"7v63jobi1j.fsf@gitster.siamese.dyndns.org","threadId":"17510","inReplyTo":"20090205132814.GK8945@mit.edu","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-02-05T23:06:16Z","receivedAt":"2009-02-05T23:06:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso <tytso@mit.edu> writes:\n\n> The argument would be that for SCM that properly tracked user\n> intentions, you did the wrong thing.  If the SCM properly understood\n> directory renames, there is a big differene between this:\n>\n> \tscm mvdir db innodb\n>\n> and this\n>\n> \tscm mv db/* innodb\n>\n> You see?  The first moves the *directory* db to innodb.  The second\n\nThen please s/scm mv/scm mvdir/ before reading my example.  The\nhypothetical \"scm mv\" command in my example just knew it was fed an\ndirectory and interpreted it as an intention to move the directory, not\nall its contents.\n\n> Of course, this distinction does not exist in git, because we track\n> content only.  And a number of other SCM's like Hg, which only track\n> file renames, wouldn't get this right either.  In order to get this\n> right, you need to treat directory renames as separate and distinct\n> operations from file renames, because they have different merge\n> implications.\n> ...\n>> See how that argument is flawed?  The point of my example is that the line\n>> between your example (1) and (2) in the previous message is blurry.\n>\n> It's blurry if you don't properly make the distinction between file\n> and directory renames, yes.\n\nMy point was even if you (in the example, who said \"I want to move db\ndirectory to innodb directory\") had two different operations, it is not\nenough, because you cannot capture that \"I want to move db directory to\ninnodb directory\" was contingent on \"because I know everything in my db\ndirectory should belong to innodb -- in fact in my history of db/, there\nis nothing but innodb support\".  The other person you will eventually be\nmerging with may not share that precondition, as the project started out\nto hold anything databasey in db/ and between the two branches being\nmerged, only you changed the semantics of what each directory means.\n\n> However, your argument that it's not possible to determine whether the\n> new file should appear as db/gdbm.c or innodb/gdm.c is an argument\n> content-tracking alone isn't enough. \n\nYes, but it is stronger than that. It is not just \"content-tracking alone\"\nis not enough.  Even systems that have distinction between \"scm mv\" and\n\"scm mvdir\" are not enough.  That is what I was trying to illustrate.\n\nYour plug-in example differentiates two cases, one of which is that the\nrenaming branch would move the directory and the other is the branch moved\nfiles under one directory to a new directory while keeping the original\ndirectory, and two cases should produce different results.  If I\nunderstand your argument correctly, it is that in the latter case the\noutcome may be ambiguous, but in the former case, it is clear that the\nintention of the remaning branch is to rename the directory itself and the\naddition to the directory done in the other branch being merged should\nautomatically be done to the renamed directory while merging.  Most\nimportantly, the argument makes the assumption that the intention of the\nnon-renaming branch (iow, why he added the new files in the directory)\ndoes not matter and does not affect the outcome.\n\nThe source tree restructuring example I brought in questions that\nassumption.  It illustrates that the intention of the side that added the\nnew file matters.  Is it an innodb support enhancement?  Then it should\nfollow the renamer's intention to move rename db/ to innodb/.  Is it\nadding something unrelated to the new meaning of \"innodb\" directory given\nby the renaming side?  Then it is very likely that it should not go to the\nrenamed innodb/ directory, even though we may not be able to decide where\nit *should* go automatically.  The point to consider is that recording the\nrenamer's intention to rename the directory and not just its contents is\nnot enough and does not help the merge.\n\n> Personally, I think the scenario I used of renaming plugins is more\n> likely that the sort of source reorganization which you've posited,\n> but I agree they are both possible scenarios.  The question for git\n> development is whether these sorts of issues ar ones that we should\n> try to handle or not?\n\nThat entirely depends on your definition of \"handle\", I think.\n\nI personally think that it is better for the tool to make its best effort\nbut stop and let the human inspect the result if the validitly of the\nresult is not so cut-and-dried, than blindly saying \"the user said move\nthe directory, so I'll move the directory and move any and all new files\nto it\" and produce a potentially wrong result.  My comment in the previous\nmessage about <db/gdbm.c> was not that it is 100% correct to leave it\nthere, nor it is 100% correct to move it elsewhere.  The point was it is a\ncase you cannot say what is correct even with help from \"scm mvdir\", and\nthe tool should stop and ask for confirmation.\n\nBoasting that \"unlike git that does not record renames, we correctly\nresolve this case automatically, because our superiour design records the\nuser's intention to rename directory\" is simply embarrassing yourself.\nYou may be silently producing a wrong result, which is nothing to boast\nabout.\n\nI think it is Ok to assume that most of the time it is correct to move the\nnew file if the other branch \"renamed\" the directory in the situation the\nexample depicts, and I do not mind if the \"best effort\" the tool makes is\nto move it to make it easy for the user to say \"Yup, that is the right\noutcome\" and conclude the merge, but I think a tool is broken if it does\nnot give the user an opportunity to examine the situation and say \"Oh, no,\nthat is not correct in *this* case\" and fix it up.\n"},{"id":"103939","messageId":"200902100220.07304.jnareb@gmail.com","threadId":"17510","inReplyTo":"200902021948.54700.jnareb@gmail.com","subject":"Re: Comments on \"Understanding Version Control\" by Eric S. Raymond","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-02-10T01:20:06Z","receivedAt":"2009-02-10T01:20:06Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 2 Feb 2009, Jakub Narebski wrote:\n\nUVC = \"Understanding Version-Control Systems\" (draft),\nhttp://www.catb.org/esr/writings/version-control/version-control.html\n\n> UVC> = What, if anything, have we learned from history? =\n> UVC> \n> UVC> There's a folk saying that \"It's not what you don't know that\n> UVC> hurts you, it's what you think you know that ain't so.\" In\n> UVC> examining the pattern of development of VCSes, it seems to me\n> UVC> that the this sub-field of computer science has been less\n> UVC> hampered than most by difficulties in finding appropriate\n> UVC> techniques, but more hampered than most by wrong assumptions that\n> UVC> hung on far longer than they should have. \n> UVC> \n> UVC> First wrong assumption: Conflict resolution by merging is\n> UVC> intractably difficult, so we'll have to settle for locking. It\n> UVC> took at least fifteen and arguably twenty years for VCS designers\n> UVC> to get shut of that one. But it's historical now. \n> UVC> \n> UVC> Second wrong assumption: Change history representation as a\n> UVC> snapshot sequence is perfectly dual to the representation as\n> UVC> change/add/delete/rename sequences.. This folk theorem is well\n> UVC> expressed in the 2004 essay \"On Arch and Subversion\"[3]. It is\n> UVC> appealing, widely held, and dead wrong.\n> UVC> \n> UVC> File renames break the apparent symmetry. The failure of\n> UVC> snapshot-based models to correctly address this has caused\n> UVC> endless design failures, subtle bugs, and user misery. \n> \n> It is not true.  Example of snapshot-based Git, which with its rename\n> detection deals very well in practice with file renames contradict\n> this theory.  Bazaar which is supposedly snapshot-based, yet support\n> \"container identities\" ('file-ids') contradict this further.\n\nNow after thinking about this a bit, I reckon that the second wrong\nassumption is not the fact that snapshots sequences representation\nare perfectly dual to changesets representation, because in practice\n(as in: merge doesn't have exponential time in history size) they are.\nIt is not even assumption that renames are not important, or in other\nwords not dealing correctly with renames and copies.\n\nNo, second wrong assumption (if we want to phrase knowledge from\nhistory of version control in this terms) is not realizing that it\nis _merging_ that has to be easy.  Both to be able to do branching\n(stable, development; feature branches), and for collaboration: the\ndistributed part of distributed version control systems (Linus' \n\"network of trust\").  And intelligent, rename-aware merge strategy\nis _necessary_ component for doing automated merge.  Necessary,\nand very important, but only a _component_.\n\nThat is what Subversion, at least up to Subversion 1.5, got wrong.\nIt made branching (or facsimile / cheap imitation of branching)\neasy, but it *didn't* made merging easy.  Even in SVN 1.5 it is not,\nfrom what I understand, very easy.\n\nEasy merging is extremely important for DVCS in OSS development, as\nusually centralized VCS with need for commit rights simply do not\nscale up to the sizes required by larger OSS projects, especially\nthose with diverse developers.\n\n\nP.S. By the way, the hgbook contains quite good description of DVCS;\ndescription of beginnings of Git can be found at GitHistory page on\nGit Wiki; you can find history of adding features and changing design\nand UI of Git in Junio C Hamano \"Git Chronicles\", presented at\nGitTogether'08.\n-- \nJakub Narebski\nPoland\n"}]}