git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: I have end-of-lifed cvsps

From
Michael Haggerty <mhagger@alum.mit.edu>
Date
Dec 18, 2013, 23:44 UTC
Message-ID
<52B2335D.2030607@alum.mit.edu>
In-Reply-To
<20131217184724.GA17709@thyrsus.com>
On 12/17/2013 07:47 PM, Eric S. Raymond wrote:
Show 16 quoted lines
> Johan Herland <johan@herland.net>:
>> However, I fear that you underestimate the number of users that want
>> to use Git against CVS repos that are orders of magnitude larger (in
>> both dimensions: #commits and #files) than your example repo.
> 
> You may be right. See below...
> 
> I'm working with Alan Barret now on trying to convert the NetBSD
> repositories. They break cvs-fast-export through sheer bulk of
> metadata, by running the machine out of core.  This is exactly
> the kind of huge case that you're talking about.
> 
> Alan and I are going to take a good hard whack at modifying cvs-fast-export 
> to make this work. Because there really aren't any feasible alternatives.
> The analysis code in cvsps was never good enough. cvs2git, being written
> in Python, would hit the core limit faster than anything written in C.

cvs2git goes to great lengths to store intermediate data to disk and keep the working set small and therefore (despite the Python overhead) I am confident that it scales better than cvs-fast-export. My usual test repo was gcc:

Total CVS Files: 25013 Total CVS Revisions: 578010 Total CVS Branches: 1487929 Total CVS Tags: 11435500 Total Unique Tags: 814 Total Unique Branches: 116 CVS Repos Size in KB: 2074248 Total SVN Commits: 64501

I also regularly converted mozilla (4.2 GB) and emacs (560 MB) for testing purposes. These could all be converted on a 32-bit computer.

Other projects that cvs2svn/cvs2git could handle: FreeBSD, Gentoo, KDE, GNOME, PostgreSQL. (Though for KDE, which I think was in the 16 GB range, I know that they used a giant machine for the conversion.)

If you haven't tried cvs2git yet, please start it up somewhere in the background. It might take a while but it should have no trouble with your repos, and then you can compare the tools based on experience rather than speculation.

Show 5 quoted lines
> Which matters, because right now the set of people working on CVS lifters
> begins with me and ends with Michael Rafferty (cvs2git), who seems even
> less interested in incremental conversion than I am.  Unless somebody
> comes out of nowhere and wants to own that problem, it's not going
> to get solved.

A correct incremental converter could be done (as long as the CVS users don't literally change history retroactively) but it would be a lot of work. Parsing the CVS files isn't the problem; after all, CVS has to do that every time you check out a branch. The problem is the extra bookkeeping that would be needed to keep the overlapping history consistent between runs N and N+1 of the tool. I sketched out what would be necessary once and it came out to several solid weeks of work.

But the traffic on the cvs2svn/cvs2git mailing list has trailed off essentially to zero, so either the software is perfect already (haha) or most everybody has already converted. Therefore I don't invest any significant time in that project these days.

Michael
-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
Previous: Eric S. RaymondNext: Johan Herland
Message 26 of 48 in “I have end-of-lifed cvsps”
  1. Eric S. RaymondDec 12, 2013
  2. Martin LanghoffDec 12, 2013
  3. Eric S. RaymondDec 12, 2013
  4. Martin LanghoffDec 12, 2013
  5. Andreas KreyDec 12, 2013
  6. Martin LanghoffDec 12, 2013
  7. Eric S. RaymondDec 12, 2013
  8. Eric S. RaymondDec 12, 2013
  9. Martin LanghoffDec 12, 2013
  10. Eric S. RaymondDec 12, 2013
  11. Martin LanghoffDec 12, 2013
  12. Eric S. RaymondDec 12, 2013
  13. Martin LanghoffDec 12, 2013
  14. Eric S. RaymondDec 12, 2013
  15. Martin LanghoffDec 13, 2013
  16. Eric S. RaymondDec 13, 2013
  17. Eric S. RaymondDec 12, 2013
  18. Martin LanghoffDec 12, 2013
  19. Jakub NarębskiDec 17, 2013
  20. Johan HerlandDec 17, 2013
  21. Eric S. RaymondDec 17, 2013
  22. Johan HerlandDec 17, 2013
  23. Eric S. RaymondDec 17, 2013
  24. Johan HerlandDec 17, 2013
  25. Eric S. RaymondDec 17, 2013
  26. Michael HaggertyDec 18, 2013
  27. Johan HerlandDec 19, 2013
  28. Michael HaggertyDec 19, 2013
  29. Johan HerlandDec 19, 2013
  30. Michael HaggertyDec 19, 2013
  31. Eric S. RaymondDec 19, 2013
  32. Michael HaggertyDec 19, 2013
  33. Eric S. RaymondDec 17, 2013
  34. Jakub NarębskiDec 17, 2013
  35. Eric S. RaymondDec 17, 2013
  36. Jakub NarębskiDec 18, 2013
  37. Eric S. RaymondDec 18, 2013
  38. Jakub NarębskiDec 18, 2013
  39. incremental fast-import and marks (Re: I have end-of-lifed cvsps)Jonathan Nieder, Dec 18, 2013
  40. Eric S. RaymondDec 18, 2013
  41. Martin LanghoffDec 18, 2013
  42. John KeepingDec 18, 2013
  43. Eric S. RaymondDec 18, 2013
  44. Kent R. SpillnerDec 18, 2013
  45. Jeff KingDec 18, 2013
  46. Eric S. RaymondDec 18, 2013
  47. Andreas SchwabDec 18, 2013
  48. Eric S. RaymondDec 18, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.