parsecvs tool now creates git repositories

I've hacked in cheesy system(3) calls to invoke various git tools to create a git repository from a parsed cvs repository. It's about the same speed as git-cvsimport now.

The UI is a total disaster, sufficient for testing. You must create an Authors file in the current directory which looks like the git-cvsimport authors file. You must also have a edit-change-log program in your path which edits the commit message in place. /bin/true will work if you don't need to edit the messages.

I should clearly steal the existing git-cvsimport command line arguments and use those.

This tool successfully, and usefully, imports the X.org xserver CVS repository, along with correctly importing several other repositories I've tried. It doesn't quite manage to compute correct branch points for the postgresql CVS repository, so there is clearly work remaining to be done.

CVS - your code's worst nightmare.

On Sat, 2006-04-01 21:36:28 -0800, Keith Packard <keithp@keithp.com> wrote:

Well, at least this sounds quite promising. I'll give it a run once I've arrived back home on the Binutils repository.

MfG, JBG

On Sun, 2006-04-02 11:39:06 +0200, Jan-Benedict Glaw <jbglaw@lug-owl.de> wrote:

Doesn't build for me:

jbglaw@bixie:~/vax/gittish/parsecvs$ make clean rm -f gram.o lex.o parsecvs.o cvsutil.o revlist.o atom.o revcvs.o git.o y.tab.h gram.c parsecvs jbglaw@bixie:~/vax/gittish/parsecvs$ make yacc -d gram.y mv -f y.tab.c gram.c cc -O0 -g -Wall -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fno-strict-aliasing -c -o gram.o gram.c cc -O0 -g -Wall -Wpointer-arith -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fno-strict-aliasing -c -o lex.o lex.c lex.l: In function ‘yylex’: lex.l:69: warning: implicit declaration of function ‘yyget_lineno’ lex.l:69: warning: nested extern declaration of ‘yyget_lineno’ <stdout>: At top level: <stdout>:1747: warning: no previous prototype for ‘yyget_lineno’ <stdout>:1756: warning: no previous prototype for ‘yyget_in’ <stdout>:1764: warning: no previous prototype for ‘yyget_out’ <stdout>:1772: warning: no previous prototype for ‘yyget_leng’ <stdout>:1781: warning: no previous prototype for ‘yyget_text’ <stdout>:1790: warning: no previous prototype for ‘yyset_lineno’ <stdout>:1802: warning: no previous prototype for ‘yyset_in’ <stdout>:1807: warning: no previous prototype for ‘yyset_out’ <stdout>:1812: warning: no previous prototype for ‘yyget_debug’ <stdout>:1817: warning: no previous prototype for ‘yyset_debug’ <stdout>:1823: warning: no previous prototype for ‘yylex_destroy’ lex.l: In function ‘parse_data’: lex.l:90: error: ‘yytext_ptr’ undeclared (first use in this function) lex.l:90: error: (Each undeclared identifier is reported only once lex.l:90: error: for each function it appears in.) make: *** [lex.o] Error 1

MfG, JBG

On Sun, 2006-04-02 at 21:31 +0200, Jan-Benedict Glaw wrote:

I think this is a bug in your version of flex; I'm using standard lex conventions here. I don't know how to make it work for you.

On Sun, 2 Apr 2006, Keith Packard wrote:

I need something like this to make it work with flex/lex..

The "-l" tells flex to be more traditional.

The "clean" rule is obvious.

And the "yylineno" is a lot more traditional than yyget_lineno(), which doesn't work for me at all. I think that's some issue with flex' support for re-entrant parsers.

Whether it works after this, I dunno. But at least it compiles.

		Linus

---

On Sun, 2006-04-02 21:10:56 -0700, Keith Packard <keithp@keithp.com> wrote:

It compiles for me with this patch (thanks to Linus for the hint):

On Mon, Apr 03, 2006 at 09:25:54AM +0200, Jan-Benedict Glaw wrote:

[...]

Almost there. I applied your patch and ran "make clean", but the Makefile forgets to remove lex.c. Here's an updated patch:

On Mon, 2006-04-03 at 09:25 +0200, Jan-Benedict Glaw wrote:

Works for me too; thanks for the fix.

On Mon, 2006-04-03 at 09:54 -0700, Keith Packard wrote:

Well, -l *kinda* works; it places a limit on the maximum token size. And, unlike 'lex', 'flex' places all input into the token buffer, even if handled outside the usual lexer loop. So, my external function that sucked up file contents was losing.

I switched it over to doing one-at-a-time reads from the input file, now the external data function can directly use stdio. This eliminates all calls to 'input' and 'unput' which should make it work for everyone now.

flex -- it's like lex, except less flexible.

On Sat, Apr 01, 2006 at 09:36:28PM -0800, Keith Packard wrote:

What is the current way to use it? I get the impression it reads raw ,v files, but how do I get along with a remote CVS repository?

Erik

Erik Mouw wrote:

tree, then uses 'cvs co ...' for getting the contents.

If you have access to remote CVS repository, it was suggested to use either cvsclone or cvsup.

On Mon, 2006-04-03 at 16:21 +0200, Jakub Narebski wrote:

It's not using cvs co, it's using the rcs 'co' command. I will probably fix it to just generate the files directly as that will be a lot faster. If there was a git command to create blobs directly from file contents, it would be faster still as I could create all of the blobs for a particular file in one pass and then just build trees in the index out of those.

On Mon, 2006-04-03 at 16:03 +0200, Erik Mouw wrote:

You can't. You need to create a local copy of the repository. There is a tool which can do that using the cvs protocol, but I don't recall the name.

It turns out that parsing the ,v files directly is both faster and more accurate than attempting to interpret the output of cvs log.

On Mon, Apr 03, 2006 at 07:37:27AM -0700, Keith Packard wrote:

I believe you're thinking of CVSSuck:
  http://cvs.m17n.org/~akr/cvssuck/

-Peff

On Mon, 03 Apr 2006 16:03:48 +0200, Erik Mouw wrote:

cvsclone, recently released, might be what you are after.

I've only used it on my own CVS repositories so I've no idea just how hard it hits the remote side.

<http://freshmeat.net/projects/cvsclone/>

Anand

Keith,

Looks nifty. Though I thought you'd go for writing a smarter cvsps, so that git-cvsimport could take advantage of it.

Looks like I'll have to brush up on my C to get to play... :-(

m

On Tue, 2006-04-04 at 10:38 +1200, Martin Langhoff wrote:

Once I had the change set information sitting in memory, it was far easier to just generate the appropriate git commands than attempt to recreate the cvsps output format...

Trust me, it wasn't because I wanted to replace git-cvsimport; it's solely that cvsps was generating complete garbage for most of my repositories.

My new tool isn't perfect yet; it isn't getting exactly the expected answers for the postgresql repository, but it's working perfectly for my X server one.

On 4/4/06, Keith Packard <keithp@keithp.com> wrote:

Oh, I don't mind -- we may as well bury cvsimport but I can't do C like I can do Perl, and I surely want to help on this one.

Meh, had you done it in Perl, I'd be helping you with the Pg repo, attic files and ensuring that files created on a branch and then put into HEAD are handled gracefully. (But you'll get Linus' and Junio's attention. Smarty cookie.)

Does it run incrementally? Can it discover non-binary files and pass -kk?

cheers,

martin

On Tue, 2006-04-04 at 14:16 +1200, Martin Langhoff wrote:

I think those parts are working correctly, I've had plenty of examples of that kind of adventure.

It doesn't run incrementally, and it unconditionally passes -kk. It's currently using rcs to check out versions of the files, so it should deal with binary content as well as rcs does. Is there something magic I need to do here? Like for DOS?

On 4/4/06, Keith Packard <keithp@keithp.com> wrote:

Cool. What's the matter with the Pg repo? (Where can I get hold of that repo?)

I thought that the .git-cvs directory it created was to be able to run incrementally (btw, I think it's fair game to create subdirs inside .git for this kind of status-tracking). And passing -kk uncoditionally is destructive in some cases (I know... git-cvsimport does it, and I want to fix that). If you can ask rcs about the mode if the file and not pass -kk for binary files...

We'll let DOS take care of itself ;)

m

On Tue, 2006-04-04 at 14:42 +1200, Martin Langhoff wrote:

As usual, the detection of branch locations is messed up.

The postgresql CVS tree is available at:

        rsync anoncvs.postgresql.org::pgsql-cvs/* postgresql.cvs

It's a fairly hefty 300M.

nah, the .git-cvs directory is purely for debugging; I leave the various command outputs there so I can see what went wrong.

I don't really have a good idea of how we'd do this process incrementally; that's not something I am personally interested in either, I want to run screaming from CVS as fast as I can at this point.

I did discover that rcs has less sophisticated keyword substitution than cvs; not having any ability to customize stuff.

I guess we need to figure out when to pass -ko and when to pass -kk. The other alternative I'd like to get around to trying is to directly generate all of the revision contents from the ,v file.

I've just changed parsecvs to generate blobs for every revision in each ,v file right after they're read in; putting the necessary code right into parsecvs should be reasonably straightforward; we don't need the multi-patch logic as we do want to compute each intermediate version of the file.

With the blobs all generated, the rest of the operation is a simple matter of building suitable indices and creating commits out of them. That's a reasonably fast operation now as it doesn't manipulate any file contents. Plus, I can do all of the index operations using a single git-update-index command, so I eliminate a pile of forking.

Doing the file revision generation in-line would allow us to eliminate most of the remaining forks; we'd run one git-hash-object per file (or so), then a git-update-index, git-write-tree and git-commit-tree per resulting commit.

Keith Packard <keithp@keithp.com> writes:

If you want to go really fast without extra fork, are writing it in C, and have the data for blob in core, you could link with libgit.a and call write_sha1_file() yourself:

	unsigned char sha1[20];
        void *buf;
        unsigned long len;

	write_sha1_file(buf, len, "blob", sha1);

instead of forking "hash-object -w". You feed your blob data in buf, with its length in len, and you will get the blob object name back in sha1[]. buf is owned by you and after write_sha1_file() returns it is safe for you to scribble over it or free() it. sha1[] stores binary object name (20 bytes, not 40-byte hexadecimal), and you can use the helper function sha1_to_hex() if you need a hex representation:

	char *sha1_to_hex(sha1)

which returns a pointer to a static buffer that is valid until next call to sha1_to_hex(), so you need to strdup it if you want to retain it.