{"thread":{"id":"43247","subject":"Re: Mozilla, git and Windows","startedAt":"2006-11-27T15:28:59Z","lastAt":"2006-11-28T12:17:17Z","messageCount":8,"participants":["Jon Smirl","Robin Rosenberg","Sam Vilain","Andy Whitcroft","Petr Baudis","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"297808","messageId":"9e4733910611270728p36e58e08w6cc7a2989b7843ce@mail.gmail.com","threadId":"43247","inReplyTo":null,"subject":"Mozilla, git and Windows","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-11-27T15:28:59Z","receivedAt":"2006-11-27T15:28:59Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"In the other thread we are discussing the conversion of Mozilla CVS to\ngit format. This is something that has to be done but it is not the\nonly issue. Without a native Windows port they won't even consider\nusing git. There is also the risk that the features needed by Mozilla\nwill be completed after they choose to use a different SCM.\n\nEven if we implement all of the needed features git still needs to win\nthe competition against the other possible choices. The last I heard\nthe leading candiate is SVN/SVK.\n\n-- \nJon Smirl\n"},{"id":"297727","messageId":"456B057E.4000904@shadowen.org","threadId":"43247","inReplyTo":"9e4733910611270728p36e58e08w6cc7a2989b7843ce@mail.gmail.com","subject":"Re: Mozilla, git and Windows","fromName":"Andy Whitcroft","fromEmail":"apw@shadowen.org","sentAt":"2006-11-27T15:34:22Z","receivedAt":"2006-11-27T15:34:22Z","isPatch":false,"sender":{"key":"apw@shadowen.org","avatar":"https://gravatar.com/avatar/d3088262854661a913ef35cc40fedcc270142d4461791142bc1ea0b2a4e2e147?d=mp&s=160"},"body":"Jon Smirl wrote:\n> In the other thread we are discussing the conversion of Mozilla CVS to\n> git format. This is something that has to be done but it is not the\n> only issue. Without a native Windows port they won't even consider\n> using git. There is also the risk that the features needed by Mozilla\n> will be completed after they choose to use a different SCM.\n> \n> Even if we implement all of the needed features git still needs to win\n> the competition against the other possible choices. The last I heard\n> the leading candiate is SVN/SVK.\n\nDo we need to worry too much about taking over the world in one day?\nYes of course git is _the_ superior solution etc, but too many new users\nat once is always painful.\n\nI think you are more likely to win letting them convert over to SVN.\nFrom there people naturally start using git mirrors from the SVN trunk.\n Cirtainly I have two projects which do not use git, one in CVS and one\nin SVN.  I just svnimport that and work in git.  I am confident with\ntime the project will migrate, but I am happy other git users are happy\nall without it being the tool of choice.\n\n-apw\n"},{"id":"294736","messageId":"9e4733910611270813g4ff982b0tbe6b57d1d49ca9f7@mail.gmail.com","threadId":"43247","inReplyTo":"456B057E.4000904@shadowen.org","subject":"Re: Mozilla, git and Windows","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-11-27T16:13:10Z","receivedAt":"2006-11-27T16:13:10Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 11/27/06, Andy Whitcroft <apw@shadowen.org> wrote:\n> Jon Smirl wrote:\n> > In the other thread we are discussing the conversion of Mozilla CVS to\n> > git format. This is something that has to be done but it is not the\n> > only issue. Without a native Windows port they won't even consider\n> > using git. There is also the risk that the features needed by Mozilla\n> > will be completed after they choose to use a different SCM.\n> >\n> > Even if we implement all of the needed features git still needs to win\n> > the competition against the other possible choices. The last I heard\n> > the leading candiate is SVN/SVK.\n>\n> Do we need to worry too much about taking over the world in one day?\n> Yes of course git is _the_ superior solution etc, but too many new users\n> at once is always painful.\n>\n> I think you are more likely to win letting them convert over to SVN.\n> From there people naturally start using git mirrors from the SVN trunk.\n>  Cirtainly I have two projects which do not use git, one in CVS and one\n> in SVN.  I just svnimport that and work in git.  I am confident with\n> time the project will migrate, but I am happy other git users are happy\n> all without it being the tool of choice.\n\nThe SVN version of the Mozilla repository is about 3GB. It takes\naround a week of CPU time for svnimport to process it.\n\n-- \nJon Smirl\n"},{"id":"294846","messageId":"200611271737.25733.robin.rosenberg.lists@dewire.com","threadId":"43247","inReplyTo":"9e4733910611270813g4ff982b0tbe6b57d1d49ca9f7@mail.gmail.com","subject":"Re: Mozilla, git and Windows","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2006-11-27T16:37:25Z","receivedAt":"2006-11-27T16:37:25Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"måndag 27 november 2006 17:13 skrev Jon Smirl:\n> On 11/27/06, Andy Whitcroft <apw@shadowen.org> wrote:\n> > I think you are more likely to win letting them convert over to SVN.\n> > From there people naturally start using git mirrors from the SVN trunk.\n> >  Cirtainly I have two projects which do not use git, one in CVS and one\n> > in SVN.  I just svnimport that and work in git.  I am confident with\n> > time the project will migrate, but I am happy other git users are happy\n> > all without it being the tool of choice.\n>\n> The SVN version of the Mozilla repository is about 3GB. It takes\n> around a week of CPU time for svnimport to process it.\n\nYou can track parts of an SVN repo using git-svn. You rarely need the\nwhole history to start working on a project.\n\nIn addition some nice soul with too much hardware will probably make an import \nand publish it and track it so everybody won't have to. We see a lot of \ngit-tracked repos SVN/CVS already.\n\n"},{"id":"298569","messageId":"20061127221338.GP7201@pasky.or.cz","threadId":"43247","inReplyTo":"9e4733910611270813g4ff982b0tbe6b57d1d49ca9f7@mail.gmail.com","subject":"Re: Mozilla, git and Windows","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-11-27T22:13:38Z","receivedAt":"2006-11-27T22:13:38Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Nov 27, 2006 at 05:13:10PM CET, Jon Smirl wrote:\n> The SVN version of the Mozilla repository is about 3GB. It takes\n> around a week of CPU time for svnimport to process it.\n\nIs there a reason why a SVN importer would _have_ to take _longer_ than\na CVS importer? I'd expect the opposite from an optimized importer since\nyou don't have to guess the changesets...\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nThe meaning of Stonehenge in Traflamadorian, when viewed from above, is:\n\"Replacement part being rushed with all possible speed.\"\n"},{"id":"296318","messageId":"456B8310.5020102@vilain.net","threadId":"43247","inReplyTo":"9e4733910611270728p36e58e08w6cc7a2989b7843ce@mail.gmail.com","subject":"Re: Mozilla, git and Windows","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2006-11-28T00:30:08Z","receivedAt":"2006-11-28T00:30:08Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Jon Smirl wrote:\n> In the other thread we are discussing the conversion of Mozilla CVS to\n> git format. This is something that has to be done but it is not the\n> only issue. Without a native Windows port they won't even consider\n> using git. There is also the risk that the features needed by Mozilla\n> will be completed after they choose to use a different SCM.\n>\n> Even if we implement all of the needed features git still needs to win\n> the competition against the other possible choices. The last I heard\n> the leading candiate is SVN/SVK.\n\nJon,\n\nWhen I met clkao in August to discuss the possibility of using git as a\ndepot for SVK, he seemed very open to the idea, and we worked on an\ninitial plan for this.  This should eventually allow svk to be used as a\nporcelain, which might make it more palatable to the Windows crowd.\n\nHowever that doesn't solve the Windows porting issue - it would still\nneed to access the repository.\n\nI've been working on and off on an abstraction to git, in Moose (perl 6\nobjects on perl 5) - what's working so far is making core objects, and\nproducing correct checksums.\n\nYou can see this early implementation at\nhttp://utsl.gen.nz/gitweb/?p=VCS-Git\n\nWhat I've found is that I need a good abstraction of UNIX pipelines that\nis as portable as Perl, so that I can prototype the code basing it on\nsetting up command pipelines on UNIX, and have it gracefully degrade to\neither using temporary files on Windows, or threading if portions are\nre-implemented using Perl (or a ported libgit).  I am currently working\non this and hope to have a release by the end of the week, though I will\nnot have tested the Windows portability by then.\n\nI would love it if anyone interested in the project would like to help\nme complete the OO-based API, provide tests, documentation, or any kind\nof feedback really.\n\n"},{"id":"294131","messageId":"9e4733910611271735y14bed29bk70ae67b5d28eb055@mail.gmail.com","threadId":"43247","inReplyTo":"20061127221338.GP7201@pasky.or.cz","subject":"Re: Mozilla, git and Windows","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-11-28T01:35:05Z","receivedAt":"2006-11-28T01:35:05Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 11/27/06, Petr Baudis <pasky@suse.cz> wrote:\n> On Mon, Nov 27, 2006 at 05:13:10PM CET, Jon Smirl wrote:\n> > The SVN version of the Mozilla repository is about 3GB. It takes\n> > around a week of CPU time for svnimport to process it.\n>\n> Is there a reason why a SVN importer would _have_ to take _longer_ than\n> a CVS importer? I'd expect the opposite from an optimized importer since\n> you don't have to guess the changesets...\n\nThese import programs take forever because they fork off git, SVN or\nCVS millions of times. It really does take a week to fork a CVS\nprocess that many times. It's not the application code that is taking\na week to run, it is the millions of forks.\n\nAs was mentioned in the thread about doing CVS to git import, the\ntrick is to write your own CVS file parser, parse the file once (not\nonce for each revision) and output all of the revisions to the git\ndatabase in a single pass. When code is structured that way I can\nimport the whole Mozilla repository into git in two hours. The\nfast-import back end also works with out forking, it just listens to\ncommand and stdin and acts on them, all of the commands are implement\nin a single binary.\n\nThe speed of fork in Linux is fine for most purposes, but it is not\nfine if you are going to fork off good sized apps several million\ntimes. When I measured those forks in oprofile, 60% of the CPU was\nbeing consumed by the kernel.\n\n-- \nJon Smirl\n"},{"id":"298924","messageId":"456C28CD.6020800@alum.mit.edu","threadId":"43247","inReplyTo":"9e4733910611271735y14bed29bk70ae67b5d28eb055@mail.gmail.com","subject":"Re: Mozilla, git and Windows","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2006-11-28T12:17:17Z","receivedAt":"2006-11-28T12:17:17Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Jon Smirl wrote:\n> As was mentioned in the thread about doing CVS to git import, the\n> trick is to write your own CVS file parser, parse the file once (not\n> once for each revision) and output all of the revisions to the git\n> database in a single pass. When code is structured that way I can\n> import the whole Mozilla repository into git in two hours. The\n> fast-import back end also works with out forking, it just listens to\n> command and stdin and acts on them, all of the commands are implement\n> in a single binary.\n\nUsing cvs2svn, it is now possible to avoid having to invoke CVS/RCS\nzillions of times.  Here is a brief description of how the new hooks work.\n\nThere is an interface called RevisionReader that is used to retrieve the\ncontents of a file.  The RevisionReader that should be used for a run of\ncvs2svn can be set using the --options file method with a line like:\n\nctx.revision_reader = MyRevisionReader()\n\nThe RevisionReader interface includes a method get_revision_recorder(),\nwhich should return an instance of RevisionRecorder.  The\nRevisionRecorder has callback methods that are invoked as the CVS files\nare parsed.  For example, RevisionRecorder.record_text() is passed the\nlog message and text (full text or delta) for each file revision.  The\nrecord_text() method is allowed to return an arbitrary token (for\nexample, a content hash), and that token is stored into\nCVSRevision.revision_recorder_token and carried along by cvs2svn.\n\nThe concrete RevisionReaders included with cvs2svn are RCSRevisionReader\nand CVSRevisionReader, which have do-nothing RevisionRecorders and which\ncall rcs or cvs in OutputPass to get the file contents.  (This repeated\ninvocation of rcs/cvs is the most expensive part of the conversion.)\n\nSo what you would do to speed things up is write your own\nRevisionRecorder, which constructs the file fulltext from the CVS deltas\nand stores the contents in a git store, returning the file revision's\ncontent hash as token.\n\nThen write a RevisionReader that returns an instance of your\nRevisionRecorder to be used in the CollectRevsPass of the conversion.\nFor OutputPass, the RevisionReader has to implement the method\nget_content_stream(), which is passed a CVSRevision instance and has to\nreturn a stream object that produces the file revision's contents.  In\nyour case, you wouldn't need the contents at all, but could just work\nwith CVSRevision.revision_recorder_token, which contains the hash that\nwas generated by your RevisionRecorder.\n\nHow you actually cook these tokens together into a git repository is up\nto you :-)\n\nMichael\n\n"}]}