{"thread":{"id":"4497","subject":"git-cvsimport doesn't quite work, wrt branches","startedAt":"2006-06-13T16:41:49Z","lastAt":"2006-06-15T07:18:24Z","messageCount":10,"participants":["Jim Meyering","Jakub Narebski","Linus Torvalds","Keith Packard","Yann Dirson","Martin Langhoff","sf"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"21722","messageId":"87irn5ovn6.fsf@rho.meyering.net","threadId":"4497","inReplyTo":null,"subject":"git-cvsimport doesn't quite work, wrt branches","fromName":"Jim Meyering","fromEmail":"jim@meyering.net","sentAt":"2006-06-13T16:41:49Z","receivedAt":"2006-06-13T16:41:49Z","isPatch":false,"sender":{"key":"jim@meyering.net","avatar":"https://avatars.githubusercontent.com/u/710630?v=4"},"body":"Here's a test case that shows how git-cvsimport is misbehaving.\nThe script below demonstrates the problem with git-1.3.3 as\nwell as with 1.4.0.rc2.g5e3a6. As for cvsps, I'm using version 2.1.\n\nThe script creates a simple cvs module, with one file on the trunk,\nand one file on a branch, then runs git-cvsimport on that. The error\nis that the resulting git repository has both files on the branch.\n\nFYI, this started when I tried to convert the GNU coreutils repository\n(which takes barely an hour with git-cvsimport -- very quick, for 45K\nrevisions and 90MB of ,v files), but found that with a git-based working\ndirectory, not all files on the b5_9x branch showed up after `git checkout\nb5_9x' -- plus, there were some files there that didn't belong.\n\n-----------------------------\n#!/bin/sh\n# Show that git-cvsimport doesn't quite work when\n# there is one file on a branch, and another on the trunk.\n# The resulting git repository has both files on the branch.\n\nexport PATH=/p/p/git/bin:$PATH\n\ncvs='cvs -f -Q'\n\nt=/tmp/.k\nrm -rf $t\nmkdir -p $t/git $t/cvs\nR=$t/repo\n$cvs -d $R init\nmkdir -p $R/m\n\ncd $t/cvs\n$cvs -d $R co m\ncd m\n# Add a file on the trunk.\ntouch on-trunk\n$cvs add on-trunk\n$cvs ci -m. on-trunk\n\n# Add another file, but destined for a branch.\ntouch on-br\n$cvs add on-br\n$cvs ci -m. on-br\n$cvs tag -b B on-br\n$cvs up -r B\necho x > on-br\n$cvs ci -m. on-br\n# Back to trunk.\n$cvs up -A\n# Remove our only-on-branch file from the trunk.\n$cvs rm -f on-br\n$cvs ci -m. on-br\n\n$cvs up -r B\n\ncd $t/git && git-cvsimport -p -x -v -d $R m >& $t/import-log\ncd $t/git && git checkout B\n\ncd $t\n\n(cd cvs/m; ls -1 on-*) > cvs-files\n(cd git; git-ls-files|sort) > git-files\n\ndiff -u1 cvs-files git-files\n\n# The problem: diff reports the following differences.\n# It should find none.\n# --- cvs-files 2006-06-13 17:48:47.000000000 +0200\n# +++ git-files 2006-06-13 17:48:47.000000000 +0200\n# @@ -1 +1,2 @@\n# ./on-br\n# +./on-trunk\n"},{"id":"21725","messageId":"e6mr9t$gjh$1@sea.gmane.org","threadId":"4497","inReplyTo":"87irn5ovn6.fsf@rho.meyering.net","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-06-13T17:06:13Z","receivedAt":"2006-06-13T17:06:13Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jim Meyering wrote:\n\n> Here's a test case that shows how git-cvsimport is misbehaving.\n> The script below demonstrates the problem with git-1.3.3 as\n> well as with 1.4.0.rc2.g5e3a6. As for cvsps, I'm using version 2.1.\n\nDo parsecvs has the same error?\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"21726","messageId":"Pine.LNX.4.64.0606131008470.5498@g5.osdl.org","threadId":"4497","inReplyTo":"87irn5ovn6.fsf@rho.meyering.net","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-06-13T17:20:10Z","receivedAt":"2006-06-13T17:20:10Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nOn Tue, 13 Jun 2006, Jim Meyering wrote:\n>\n> Here's a test case that shows how git-cvsimport is misbehaving.\n> The script below demonstrates the problem with git-1.3.3 as\n> well as with 1.4.0.rc2.g5e3a6. As for cvsps, I'm using version 2.1.\n\nWell, it's a cvsps problem. \n\nBig surprise.\n\nSadly, it also seems to be one that isn't fixed by the patches _I_ have, \nand looking at Yann's set of patches, I don't think they fix it either.\n\nThis is what (my version of) CVSps reports for your repository:\n\n\t---------------------\n\tPatchSet 1 \n\tDate: 2006/06/13 10:06:42\n\tAuthor: torvalds\n\tBranch: HEAD\n\tTag: (none) \n\tLog:\n\t.\n\t\n\tMembers: \n\t on-br:INITIAL->1.1 \n\t on-trunk:INITIAL->1.1 \n\t\n\t---------------------\n\tPatchSet 2 \n\tDate: 2006/06/13 10:06:44\n\tAuthor: torvalds\n\tBranch: B\n\tAncestor branch: HEAD\n\tTag: (none) \n\tLog:\n\t.\n\t\n\tMembers: \n\t on-br:1.1->1.1.2.1 \n\t\n\t---------------------\n\tPatchSet 3 \n\tDate: 2006/06/13 10:06:46\n\tAuthor: torvalds\n\tBranch: HEAD\n\tTag: (none) \n\tLog:\n\t.\n\t\n\tMembers: \n\t on-br:1.1->1.2(DEAD) \n\n\nand note how the \"on-br\" file is part of the initial PatchSet 1.\n\nSo CVSps basically tells git-cvsimport that commit 2 (on branch B) is \nbased on commit 1, and doesn't say that \"on-trunk\" has gone away, so the \nresulting git repository has branch B containing \"on-trunk\" version 1.1, \nand \"on-br\" version 1.1.2.1.\n\nCVS branches obviously sometimes confuse CVSps. Sadly, they also confuse \n_me_, so I don't see how to fix this particular CVSps bug, because I'm as \nconfused as CVSps is ;)\n\nWe'd need to have CVSps tell git that the \"on-trunk\" file was never added \nto branch B: the simplest way to do that would be to say that it has \nbecome (DEAD) in PatchSet 2 (which is not technically true in CVS terms, \nbut _is_ technically true on git terms - on branch B, that file is \nobviously dead).\n\nYann? Pavel? Anybody? Ideas?\n\n\t\tLinus\n"},{"id":"21742","messageId":"1150224411.20536.79.camel@neko.keithp.com","threadId":"4497","inReplyTo":"Pine.LNX.4.64.0606131008470.5498@g5.osdl.org","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2006-06-13T18:46:51Z","receivedAt":"2006-06-13T18:46:51Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Tue, 2006-06-13 at 10:20 -0700, Linus Torvalds wrote:\n\n> Well, it's a cvsps problem. \n> \n> Big surprise.\n\nYeah, we've got\n\n\tgit-cvsimport\n\tcvsps\n\tcvs rlog\n\t,v files\n\ncvs rlog is designed to 'represent' the history of the repository to\nusers. Cvsps was built as a software analysis tool, and is used by\nputative software engineering researchers. Basing a supposedly lossless\nrepository conversion system on this pair seems foolish to me,\nnotwithstanding the heroic efforts to make it work.\n\n-- \nkeith.packard@intel.com\n"},{"id":"21752","messageId":"20060613211344.GC7766@nowhere.earth","threadId":"4497","inReplyTo":"Pine.LNX.4.64.0606131008470.5498@g5.osdl.org","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Yann Dirson","fromEmail":"ydirson@altern.org","sentAt":"2006-06-13T21:13:44Z","receivedAt":"2006-06-13T21:13:44Z","isPatch":false,"sender":{"key":"ydirson@altern.org","avatar":"https://avatars.githubusercontent.com/u/1190950?v=4"},"body":"On Tue, Jun 13, 2006 at 10:20:10AM -0700, Linus Torvalds wrote:\n> Sadly, it also seems to be one that isn't fixed by the patches _I_ have, \n> and looking at Yann's set of patches, I don't think they fix it either.\n\nI don't think so either.\n\n\n> So CVSps basically tells git-cvsimport that commit 2 (on branch B) is \n> based on commit 1, and doesn't say that \"on-trunk\" has gone away, so the \n> resulting git repository has branch B containing \"on-trunk\" version 1.1, \n> and \"on-br\" version 1.1.2.1.\n> \n> CVS branches obviously sometimes confuse CVSps. Sadly, they also confuse \n> _me_, so I don't see how to fix this particular CVSps bug, because I'm as \n> confused as CVSps is ;)\n>\n> We'd need to have CVSps tell git that the \"on-trunk\" file was never added \n> to branch B: the simplest way to do that would be to say that it has \n> become (DEAD) in PatchSet 2 (which is not technically true in CVS terms, \n> but _is_ technically true on git terms - on branch B, that file is \n> obviously dead).\n> \n> Yann? Pavel? Anybody? Ideas?\n\nThis is exactly the problem I encountered one week ago with one my old\ncvs repos, where I had created a branch only for a part of a source\nhierarchy :)\n\nOne thing that amused me, is that in that case cvsps was DWIM enough\nthat the result was indeed what I expected from the conversion (I had\nforgotten about the particular way that branch was created 3 years\nago). I only discovered the problem when tailor's cvs backend\ngenerated deletions when starting my branch.\n\nSo basically, because of how awkward cvs branches are, cvsps may\nindeed do what many users expect here, because branches in cvs repos\nare sometimes created in strange ways, (in my case, to avoid having to\nmerge changes in unrelevant areas of the tree - nowadays, I'd just use\nstgit to isolate changes).\n\nI don't know what was the particular thing in coreutils developement\nthat led to branching only some files. In my case, it can be seen as\nthe cvs idiom for \"branching a part of the tree\" - something I don't\nthink there is a need to have a special idiom in GIT for.\n\n\nIf we want cvsps to output the exact history derived from cvs\n(ie. what Jim expected, and I think it is reasonable), I fear it would\nrequire substential modification to cvsps. I should check, but I\ndon't think it currently keeps track of which files are part of the\ntree resulting from a changeset, but only of the files actually touhed\nby the changeset. So the change would probably have a big ram\nusage impact, if we store the file refs in each changeset.\n\n\nThat reminds me of another funny cs behaviour I noticed a couple of\nmonths ago (not sure if it was in 1.11.x or 1.12.x): \"cvs import\" was\nnot marking files as dead on the vendor branch when it disappeared\nfrom one upstream version to another, it was just not tagged in the\nnew version. I guess cvsps would have a hard time figuring out what\nhappenned, and would just mark the taks as invalid.\n\n\nFor this type of cvsps issues and cvs tags in general, my latest idea\nwould be to add \"fake\" patchsets on which to apply tags and\nbranchpoints. The ideal way would seem to make those similar to git's\nmerge commits, having as parents all patchsets the tag takes revision\nfrom (obviously it's so biased towards the git model it would be a\npleasure to add support for this in git-cvsimport :) - but that would\nproduce patchsets not fitting well into the current cvsps model, so\nthat may require more thinking.\n\nAnyway, it should provide a way to make sense out of what cvsps\ncurrently considers to be \"invalid\" tags.\n\nBest regards,\n-- \nYann Dirson |\nDebian-related: | Support Debian GNU/Linux:\n | Freedom, Power, Stability, Gratis\n http://ydirson.free.fr/ | Check \n"},{"id":"21756","messageId":"46a038f90606131555m7b1fa744g9770140c87598b7b@mail.gmail.com","threadId":"4497","inReplyTo":"1150224411.20536.79.camel@neko.keithp.com","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-06-13T22:55:06Z","receivedAt":"2006-06-13T22:55:06Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 6/14/06, Keith Packard wrote:\n> cvs rlog is designed to 'represent' the history of the repository to\n> users. Cvsps was built as a software analysis tool, and is used by\n> putative software engineering researchers. Basing a supposedly lossless\n> repository conversion system on this pair seems foolish to me,\n> notwithstanding the heroic efforts to make it work.\n\nYes, cvsps is relying on the wrong things. I am looking at parsecvs\nand the cvs2svn tool and wondering where to from here.\n\nIn terms of history parsing, parsecvs and cvs2svn are similar. I like\ncvs2svn \"many passes\" approach better, though the Python source is\nreally messy. A good thing about cvs2svn is that it is a lot more\nconservative WRT memory use.\n\nSo far, I have been relying on parsecvs for initial imports, and for\ncvsps+git-cvsimport for incrementals on top of that initial import.\nBut parsecvs falls over with large repos.\n\nI am starting to look at what I can do with cvs2svn to get the import\ninto git. It seems to get very good patchsets, and it yields an easily\nreadable DB. I'll either learn Python, or read the DB from Perl\n(probably from git-cvsimport).\n\nThe main problem, however, is that it doesn't do incremental imports,\nso this would be a roundabout way of fixing parsecvs's\nmemory-bound-ness. We still need cvsps :(\n\n\nmartin\n"},{"id":"21757","messageId":"1150241459.20536.98.camel@neko.keithp.com","threadId":"4497","inReplyTo":"46a038f90606131555m7b1fa744g9770140c87598b7b@mail.gmail.com","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2006-06-13T23:30:59Z","receivedAt":"2006-06-13T23:30:59Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Wed, 2006-06-14 at 10:55 +1200, Martin Langhoff wrote:\n\n> In terms of history parsing, parsecvs and cvs2svn are similar. I like\n> cvs2svn \"many passes\" approach better, though the Python source is\n> really messy. A good thing about cvs2svn is that it is a lot more\n> conservative WRT memory use.\n\nI will try to fix parsecvs so it doesn't take so much memory. Of course,\nmy goal was to import various X.org repositories which have horrible\nissues, but aren't all that huge. And, for them, it works just fine.\n \n> So far, I have been relying on parsecvs for initial imports, and for\n> cvsps+git-cvsimport for incrementals on top of that initial import.\n> But parsecvs falls over with large repos.\n\nI'd like some help figuring out how to do incremental imports with\nparsecvs. As parsecvs already constructs the project history from the\npresent into the past, it should be possible to \"notice\" when it hits\nexisting bits in the repository and stop automatically. I think this\nwill just take saving a bit of state in the git repository to mark where\nin CVS the tips of each branch come from.\n\n> The main problem, however, is that it doesn't do incremental imports,\n> so this would be a roundabout way of fixing parsecvs's\n> memory-bound-ness. We still need cvsps :(\n\nParsecvs is currently O(nrev * nfile), and I'd like to make it O(nrev)\ninstead.\n\n-- \nkeith.packard@intel.com\n"},{"id":"21759","messageId":"46a038f90606131856o77d58467le4d3dab8021b32@mail.gmail.com","threadId":"4497","inReplyTo":"1150241459.20536.98.camel@neko.keithp.com","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-06-14T01:56:13Z","receivedAt":"2006-06-14T01:56:13Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 6/14/06, Keith Packard wrote:\n> On Wed, 2006-06-14 at 10:55 +1200, Martin Langhoff wrote:\n>\n> > In terms of history parsing, parsecvs and cvs2svn are similar. I like\n> > cvs2svn \"many passes\" approach better, though the Python source is\n> > really messy. A good thing about cvs2svn is that it is a lot more\n> > conservative WRT memory use.\n>\n> I will try to fix parsecvs so it doesn't take so much memory. Of course,\n> my goal was to import various X.org repositories which have horrible\n> issues, but aren't all that huge. And, for them, it works just fine.\n\nWould it be possible to have it parse the RCS histories from a remote repo?\n\nI had forgotten, but that's something else that the cvsps +\ngit-cvsimport combo can do. In short, to replace cvsps+git-cvsimport\n...\n\n + not memory bound -- or at least must be able to import large\n(mozilla, gentoo) with a decent amount of memory\n\n + must work local and remote (of course local can be faster)\n\n + must do incrementals reasonably well\n\n> I'd like some help figuring out how to do incremental imports with\n> parsecvs. As parsecvs already constructs the project history from the\n> present into the past, it should be possible to \"notice\" when it hits\n> existing bits in the repository and stop automatically. I think this\n> will just take saving a bit of state in the git repository to mark where\n> in CVS the tips of each branch come from.\n\nOk. Before starting to read the RCS files, I would look at all the\nbranch tips in the git repo, and read some metadata of the last commit\nof each head into memory (author, commitmsg, timestamp, diffstat).\n\nWhen parsing RCS files and building changesets to import, compare them\nwith the 'head' data. The timestamp granularity is seconds which is\npretty coarse -- you can ask for history post those timestamps, but\nthere's the risk of missing commits (this affects git-cvsimport today,\nand I'm thinking how to fix it there). So borderline changesets should\nbe compared against the metadata you have.\n\nThere is the chance that your earlier import caught a commit partway\nthrough, so you may end up putting in the 'rest' of the commit. That's\nwhy diffstat can be useful.\n\nIs that useful?\n\n\ncheers,\n\n\n\nmartin\n"},{"id":"21776","messageId":"448FD8E4.9040208@b-i-t.de","threadId":"4497","inReplyTo":"46a038f90606131555m7b1fa744g9770140c87598b7b@mail.gmail.com","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"sf","fromEmail":"sf@b-i-t.de","sentAt":"2006-06-14T09:37:40Z","receivedAt":"2006-06-14T09:37:40Z","isPatch":false,"sender":{"key":"sf@b-i-t.de","avatar":null},"body":"Martin Langhoff wrote:\n...\n> Yes, cvsps is relying on the wrong things. I am looking at parsecvs\n> and the cvs2svn tool and wondering where to from here.\n...\n> I am starting to look at what I can do with cvs2svn to get the import\n> into git. It seems to get very good patchsets, and it yields an easily\n> readable DB. I'll either learn Python, or read the DB from Perl\n> (probably from git-cvsimport).\n\nSVN has a portable format called \"dumpfile\" (see\nhttp://svn.collab.net/repos/svn/trunk/notes/fs_dumprestore.txt) which is\nproduced by \"svnadmin dump ...\" and \"cvs2svn --dump-only ...\".\n\nWhy not use it as input for importing into git?\n\nPros:\n- \"svnadmin dump\" should be fast\n- svn repositories can be tracked with \"svnadmin dump\" (just remember\nthe last imported revision and restart from there)\n- cvs2svn seems to be very good at its job\n- only one tool needed\n\nCons:\n- Both svnadmin and cvs2svn only work on local repositories\n- cvs2svn cannot be used for tracking\n\nRegards\n\tStephan\n"},{"id":"21820","messageId":"20060615071823.GE7766@nowhere.earth","threadId":"4497","inReplyTo":"1150224411.20536.79.camel@neko.keithp.com","subject":"Re: git-cvsimport doesn't quite work, wrt branches","fromName":"Yann Dirson","fromEmail":"ydirson@altern.org","sentAt":"2006-06-15T07:18:24Z","receivedAt":"2006-06-15T07:18:24Z","isPatch":false,"sender":{"key":"ydirson@altern.org","avatar":"https://avatars.githubusercontent.com/u/1190950?v=4"},"body":"On Tue, Jun 13, 2006 at 11:46:51AM -0700, Keith Packard wrote:\n> Yeah, we've got\n> \n> \tgit-cvsimport\n> \tcvsps\n> \tcvs rlog\n> \t,v files\n> \n> cvs rlog is designed to 'represent' the history of the repository to\n> users.\n\nI wouldn't exactly call that \"history of the repository\" :)\n\nAre you thinking about any particular information from the ,v files,\nthat rlog fails to expose ? That is, wouldn't be possible to do a job\nsimilar to what parsecvs does, with remote support ?\n\nBest regards,\n-- \nYann Dirson |\nDebian-related: | Support Debian GNU/Linux:\n | Freedom, Power, Stability, Gratis\n http://ydirson.free.fr/ | Check \n"}]}