cvsps: bad usage: invalid argument --norc

Hi esr.
In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'.
Please give the option back and write something in the man page like:
    This option has no effect; it is present for compatibility

IB> Hi esr. IB> In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'. IB> Please give the option back and write something in the man page like: IB> This option has no effect; it is present for compatibility

Looks like the tool is completely different. I think I'll have to downgrade.

On Sun, Apr 14, 2013 at 12:48:00PM +0400, Ilya Basin wrote:

If you want incremental import support, then that's the best thing to do. There was some discussion about this in January [1] and it doesn't look like cvsps-3 has changed since then, so if you want to incrementally update a Git clone of a CVS repository then cvsps-2 is really the only way to go.

If you're doing a one-off import then I recommend ignoring git-cvsimport and investigating cvs2svn[2] and cvs-fast-export [3].

[1] http://thread.gmane.org/gmane.comp.version-control.git/214258/focus=214305 [2] http://cvs2svn.tigris.org/cvs2git.html [3] https://gitorious.org/cvs-fast-export

Ilya Basin <basinilya@gmail.com>:

Or you could just use 3.x directly rather than through the git-cvsimport wrapper. It works better that way - it actually ships a fast-import stream that git fast-import can consume directly.

Old cvsps (2.x) was very, very broken; there was a bug in it that pretty much guaranteed incorrect conversion of branchy repos. I've fixed that particular bug, and several other serious ones, along with adding the import-stream output stage, but I don't really trust the cvsps code; I think its algorithmic core is weak and klugey and only works semi-by-accident.

I do *not* recommend downgrading, it will pitch you from a bad situation into a worse one. Yes, Junio is still shipping a wrapper for 2.x, but that was very much against my advice.

I'm also now the maintainer of cvs-fast-export, which used to be Keith Packard's utility for lifting the X.org CVS repo to git. That was before I resurrected it and added a fast-import stream output stage.

I *think* cvs-fast-export's algorithms are more correct than cvsps's, but I have not yet been able to pry loose the time to write the really rigorous test suite to verify this. That goal has been second on my prority list for a couple of months now; I keep wanting to get to it but having to fight fires on other projects instead.

I wish I could point you at a convertor I really trust. I can't. There is a third tool, cvs2git (based on the analyzer from cvs2svn) that I don't maintain, which has problems of its own. And those three are about it.

Yes, it's a swamp. The relatively poor capability of the tools isn't anybody's fault; the problem domain is *nasty*. I've been working a closely related but easier one (Subversion stream dump analysis) for a couple years now and understanding it doesn't make it less ugly.

I think with 4-6 weeks of concentrated attention I could clean up the mess and deliver a really high-quality converter, and I'm motivated to do this because I want it as a CVS front end for reposurgeon. But it hasn't happened yet and the incompleteness of my test suite is a blocker in the way.

The topo analysis code in all these tools is really fragile and tends to break on old edge cases when you try to teach it to handle new ones, so a good set of regression tests is especially important. And doesn't yet exist, though I have built a decent start for cvsps based on the tests in the git tree.

Do you have enough interest and spare cycles to help finish the test suite? Another pair of hands on it might speed things up a lot.

On 04/14/2013 01:33 PM, Eric S. Raymond wrote:

                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Again I request that instead of spreading non-specific FUD about
cvs2git, you describe concrete problems so that I can improve the tool.
 So far about the only problems that you have mentioned that have not
been addressed have been stylistic trivia like that you object that
cvs2git writes its outputs by pass, to two separate files, rather than
writing all of its output to stdout, and that you are bothered that it
names its temporary directory "cvs2svn-tmp" rather than "cvs2git-tmp".

I would like to fix any conversion deficiencies you can find. I stand behind this tool and think it is the best one out there for extracting history from CVS.

Michael

Hi Eric.

I tried --fast-export. It's 2 times faster. The first thing that differs: in cvsps2 commits with adjacent timestamps were joined into one (see the attached files). Do you know the reason?

Does this --fast-export thing support what John mentioned, the "incremental import support"? Does 'git fast-import' has it? I need it, because full import takes too long. The central repo of my employer is CVS, other people commit to it and I use git internally to be able to tidy my commit history before exporting to CVS.

Ilya Basin wrote:

Errr... the need to interact with *live* CVS repository is reason behind wanting incremental import, isn't it?

I wonder how hard would be to create a remote helper for CVS, so that one could treat CVS repository as foreign remote, fetching from it and pushing to it...

BTW. can you convince your employer to move to more modern version control system (be it Subversion, Kiln Harmony, Veracity, Bazaar, Mercurial or Git) rather than rely on system which had last release in 2008 i.e. 4 years ago, doesn't support atomic commits, doesn't support versioning of whole repository aka. changesets (beside tags), has slow and complicated branching and even more complicated merging, etc.?

http://stackoverflow.com/questions/802573/difference-between-git-and-cvs/824241#824241

Ilya Basin <basinilya@gmail.com>:

The cvsps guy included code to do that. Keith Packard didn't. Sorry I can't be more helpful, but that's about all I know.

I didn't write either analysis stage; I understand cvsps's, somewhat, because I had to fix several nasty bugs in it. I *don't* understand cvs-fast-export's analysis stage very well yet, because it has no obvious bugs that have required me to dive in. (Keith's notes document one major bug, which may be inherent to the mismatch between file- and changest-orientation and not fixable in the general case, though I will try.)

cvs-fast-export does not have incremental-import support. Whether git-cvs-import has it depend on which version you have and what backend it it is using. I don't maintain that wrapper.

You are out of luck. That feature was dependent on a very fragile coupling between the old output format and a bunch of unmaintainably horrible Perl in the git-cvs-import wrapper script. It didn't work very well; frankly, I'm amazed it worked at all.

The things I had to do to fix the serious bugs in cvsps2 and make it output a fast-import stream had the side effect of breaking that coupling. cvsps3 won't give you that feature. Dropping back to cvsps2 to keep that feature will expose you to the cvsps2 bugs.

I'm sorry these tools are such a mess. I'm trying to fix that, but it's hard, slow work. The problems are deeply ugly and the edge cases have poisoned spikes.

Hi list.
Here's what I mean:
1) a created a git repo from a foreign source using git fast-import
2) new commits were added to the foreign source

Can I create a fast-import input stream not containing the commits already existing in my git repo and import it?

I tried to create such streams with:
    cvsps --fast-export -d ...
and from a shallow git repo, but the new commits are not imported
(unless the import stream contains a new branch)

On Fri, Apr 19, 2013 at 12:29 AM, Ilya Basin <basinilya@gmail.com> wrote:

Yes, see the --import-marks and --export-marks options. The stream has to refer to the marks that were used int the previous run.

Cheers.

Hi Ilya,

Ilya Basin wrote:

Yes, if the foreign source is structured in a way to make it easy. Take a look at the --cat-blob-fd option, the "ls" and "cat-blob" commands, and see svn-fe from contrib/svn-fe/ as an example.

Hope that helps, Jonathan

Hi Eric.

ESR> cvs-fast-export does not have incremental-import support. ESR> Whether git-cvs-import has it depend on which version you have ESR> and what backend it it is using. I don't maintain that wrapper. Did you mean "git-fast-import"? Or do you know any wrapper that already uses cvsps3 --fast-export?

ESR> You are out of luck. That feature was dependent on a very fragile ESR> coupling... OK, OK, I get it.

First of all, I think cvsps3 has almost everithing required for
incremental import: one could just take the date of the last commit
and invoke cvs ps with the '-d' flag. However, to import new commits
into existing branches the stream should contain the "from" command in
oldest commits in each branch (now missing).
If the branch already exists in the target git repo, it's easy to
refer it in the stream:
    from refs/heads/branchname^0

But if the branch is new, but it's parent commit is already imported, I guess, the only way to refer it is by its SHA-1 Eric, what parent information can cvsps provide for the first commit in a branch, when invoked with the '-d' flag?

Apologies for the somewhat belated reply. I've been even busier than usual lately and am about to be traveling for a week.

Ilya Basin <basinilya@gmail.com>:

No, I meant git-cvs-import. I wrote a version of it that supports cvsps3, but Junio chose to keep the old wrapper. Apparently he would rather inflict cvsps2's rather serious known bugs on users than break backward compatibility even a little.

Look at the -i option.  That may do what you need.
 

At the moment it doesn't provide any at all. That case wasn't on my radar when I was fixing the code. If you can specify a behavior you think would be useful, I'm listening.

ESR> At the moment it doesn't provide any at all. That case wasn't on my ESR> radar when I was fixing the code. If you can specify a behavior you ESR> think would be useful, I'm listening.

For new branches the 'from' command can refer the common ancestor in an existing branch. For example:

         /----------E thebranch
        /
    A---B---C---D master

Commit E is newer than D; we already imported D; thebranch is new.
Instead of:
    from refs/heads/thebranch^0
refer the parent as:
    from refs/heads/master^2

ESR> Look at the -i option. That may do what you need. ESR>

OK, something's wrong with the man page: starting with '-A' the description is unstructured:

       -A authormap
           Apply an author-map file to the attribution lines. Each line must be of the form

               ferd = Ferd J. Foonly <foonly@foo.com> America/Chicago

           and will be applied to map the Unix username ferd to the DVCS-style user identity specified after the equals
           sign. The timezone field (after > and whitespace) is optional and (if present) is used to set the timezone
           offset to be attached to the date; acceptable formats for the timezone field are anything that can be in the TZ
           environment variable, including a [+-]hhmm offset. Whitespace around the equals sign is stripped. Lines
           beginning with a # or not containing an equals sign are silently ignored. -R revmap:: Write a revision map to
           the specified argument filename. Each line of the revision map consists of three whitespace-separated fields: a
           filename, an RCS revision number, and the mark of the commit to which that filename-revision pair was assigned.
           -v:: show very verbose parsing messages. -t:: show some brief memory usage statistics. --summary-first:: when
           multiple patchset diffs are being generated, put the patchset summary for all patchsets at the beginning of the
           output. --diffs-opts option string:: send a custom set of options to diff, for example to increase the number
           of context lines, or change the diff format. --debuglvl bitmask:: enable various debug output channels. -Z
           compression:: A value 1-9 which specifies amount of compression. A value of 0 disables compression. --root
           cvsroot:: Override the setting of CVSROOT (overrides working directory and environment). -i:: Incremental
           export. Each commit with no ancestor gets a from pointer name. When importing to an existing repository, this
           will attach each such commit as a child of the last commit on $BRANCH in the existing repository. -k:: Kill
           keywords: will extract files with -kk from the CVS archive to avoid noisy changesets. -T:: Force deterministic
           dates for regression testing. Each patchset will have a monotonic-increasing attributed date computed from its
           patchset ID. --fast-export:: Emit the report as a git import stream. --convert-ignores:: Convert ..cvsignore
           files to .gitignore files. --reposurgeon:: Emit for each commit a list of the CVS file:revision pairs composing
           it as a bzr-style commit property named "cvs-revisions". From version 2.12 onward, reposurgeon can interpret
           these and use them as hints for reference-lifting. -V:: Emit the program version and exit.  module-path::
           Operate on the specified module. If this option is not given, either the CVSROOT environment variable must be
           set to point directly at the module or cvsps must be run in a checkout directory or repository module
           subdirectory.

--
 

Ilya Basin <basinilya@gmail.com>:

Understood. Do you actually need this much generality in practice, or is it a theoretical case?

Interesting. The aciidoc parser got a little confused, but inserting some blank lines fixed it.

ESR> Ilya Basin <basinilya@gmail.com>:

ESR> Understood. Do you actually need this much generality in practice, ESR> or is it a theoretical case?

Such cases occur quite often.