{"thread":{"id":"12255","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","startedAt":"2008-02-22T07:32:28Z","lastAt":"2008-02-22T19:25:02Z","messageCount":10,"participants":["Shawn O. Pearce","Ian Clatworthy","Johannes Schindelin","Pierre Habouzit","Michael Haggerty","Aidan Van Dyk"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"69556","messageId":"20080222073228.GF8410@spearce.org","threadId":"12255","inReplyTo":"47BE167A.4060005@internode.on.net","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-02-22T07:32:28Z","receivedAt":"2008-02-22T07:32:28Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Ian Clatworthy wrote:\n> FYI. I thought you'd be interested in this as it's inspired by and based\n> on git-fast-import. You can download the Python source from\n> https://code.launchpad.net/bzr-fastimport/. In particular, there's a\n> Python parser of the stream format included that may be useful to the\n> Git community or other VCS communities. The fast-import-info and\n> fast-import-filter commands might also be useful to others.\n\nThis is interesting. I'm not a Python guy, but the info and filter\ncommands do look like they could be useful beyond the Bazaar community.\n\nMichael Haggerty of cvs2svn has spent a good amount of time creating\na git-fast-import backend to cvs2svn. Given that cvs2svn is one of\nthe few tools that can read some of the really strange real world\nCVS trees its good to be able to leverage that work for other systems\n(SVN, Git, and now Bazaar).\n \n> BTW, you might want to either extend the specification (a little) or fix\n> git-fast-export so they match. :-) See doc/notes.txt under\n> http://bazaar.launchpad.net/~bzr/bzr-fastimport/fastimport.dev/files for\n> details. For example, running git-fast-export on 64-bit Hardy Heron\n> produces file modes longer than permitted if the spec was strictly\n> interpreted.\n\nWe may need to take a small hammer to git-fast-export and fix\nits output. Generating long mode strings like your notes suggest\nis incorrect.\n\nThe fast-import format is very strict, to avoid any sort of ambiguous\nbehavior and implicit data corruption during import. This is one\nreason we don't use \"auto format detection\" for dates and instead\nrequire that the frontend tell us what date format it is using,\nand stick to only that format thoughout the stream. Its also a\nreason why we only support a limited number of date formats.\n\nFile modes in Git are very limited. We really only want symlinks\nor regular files with permissions of 644 and 755. Everything else\nis bogus. We also now have the S_IFGITLINK mode to deal with but\nfast-import does not currently support it.\n\n> Looking forward, I'd probably like to extend the spec to support some\n> Bazaar-specific features, e.g. versioning of directories without files\n> inside them. If you have a preferred way of me doing this or would like\n> to work on it together when that time comes, please let me know. To keep\n> backwards compatibility, the first option that springs to mind is using\n> specially marked comments for stuff like this, e.g.\n> \n> ##bzr:: blah blah blah\n\nTechnically its valid for a Git tree to contain an empty subdirectory,\nbut that directory would disappear if the user tries to make a commit\non top of it due to the current limitations of the index file.\n\nSo git-fast-import could actually allow the frontend to create an\nempty directory in the stream format, and record that correctly\nin Git. Its just that building on top of that may cause the\ndirectory to disappear. :)\n\nIf we are heading in the direction of making this a common stream\nformat I'd like to try and work it out such that any additional\nextensions aren't VCS specific, at least as much as we can avoid it.\nThat way exports from a source into this format can be loaded into\nany VCS that recognizes it, and have little or no loss. So yes,\nI am interested in trying to work with you and anyone else who wants\nto extend the format further.\n\n> Finally, thanks for writing git-fast-import and the associated\n> documentation. It's well done. If you have any thoughts on the various\n> front-ends available, I'm interested in hearing them. As well as saving\n> me time vs testing lots of them, your thoughts will give us things to\n> keep in mind when developing bzr-fastexport soon.\n\nI think the fast-import documentation is the longest chunk of docs\nwe have in git, at least as a single manpage. :-)\n\nThe git-p4 importer in the git.git contrib/ directory is probably\nthe most well known and most widely used frontend. It is also an\nincremental tool, and I know a number of Qt developers use it to\nmirror the Qt Perforce tree into Git.\n\nThe t/t9300-fast-import.sh test script contains a number of tests\nfor git-fast-import. The test cases themselves (in terms of the\nstream it feeds in) may be of some use to you as it covers most of\nthe currently recognized stream format.\n \n> -------- Original Message --------\n> Subject: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option\n> Date: Fri, 22 Feb 2008 01:23:34 +1000\n> From: Ian Clatworthy \n> To: Bazaar , bazaar-announce@lists.canonical.com\n> \n> I'm pleased to announce bzr-fastimport, a plugin useful for loading data\n> exported by a large number of foreign VCS tools. Places to start are:\n> \n> * the Launchpad page - https://launchpad.net/bzr-fastimport\n> * the Wiki page - http://bazaar-vcs.org/BzrFastImport.\n> \n> Please note that this is not yet production quality but seems to be\n> working well enough to be useful for a large number of projects.\n> \n> I would *greatly* appreciate testing, feedback and improvements. In\n> particular, I'm using this for migrating the OpenOffice.org repository\n> (76K files and 500K revisions) into Bazaar from Subversion, so I'd\n> really like some help with testing out and enhancing the existing\n> Subversion front-ends.\n\nHeh. OOo is _huge_. I think the best import into Git thus far is\ntaking up about 1.5G of disk space once fully repacked. I don't\nrecall how they did the import, but coming from SVN I think they\nused git-svn, which is not based on git-fast-import.\n\nWhat frontend are you using to go from SVN -> fast-import?\n"},{"id":"69558","messageId":"47BE7E1C.4040303@internode.on.net","threadId":"12255","inReplyTo":"20080222073228.GF8410@spearce.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Ian Clatworthy","fromEmail":"ian.clatworthy@internode.on.net","sentAt":"2008-02-22T07:47:40Z","receivedAt":"2008-02-22T07:47:40Z","isPatch":false,"sender":{"key":"ian.clatworthy@internode.on.net","avatar":null},"body":"Shawn O. Pearce wrote:\n\n> If we are heading in the direction of making this a common stream\n> format I'd like to try and work it out such that any additional\n> extensions aren't VCS specific, at least as much as we can avoid it.\n> That way exports from a source into this format can be loaded into\n> any VCS that recognizes it, and have little or no loss. So yes,\n> I am interested in trying to work with you and anyone else who wants\n> to extend the format further.\n\nExcellent. That sounds the right way to go. I'll contact you if and when\nI want to add stuff.\n\n> Heh. OOo is _huge_. I think the best import into Git thus far is\n> taking up about 1.5G of disk space once fully repacked. I don't\n> recall how they did the import, but coming from SVN I think they\n> used git-svn, which is not based on git-fast-import.\n> \n> What frontend are you using to go from SVN -> fast-import?\n\nThe pack file in the Git clone I have is 2.4G. I thought that was large\nbut it's quite small compared to the 82G svn dump that creates a 55G svn\nrepo!\n\nI'm using svn-fast-export.c currently. I'd rather enhance the Python one\nbut my Subversion binding knowledge is slim and there's a bug wrt \"too\nmany open files\" that causes it to crash almost immediately on the OOo\nrepo. It's not obvious to me how to fix that unfortunately. It worked\nfine for Wordpress OTOH.\n\nThe svn-all-fast-export tool sounds interesting but is completely\nundocumented to my knowledge.\n\nIan C.\n"},{"id":"69570","messageId":"alpine.LSU.1.00.0802221031550.17164@racer.site","threadId":"12255","inReplyTo":"20080222073228.GF8410@spearce.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-22T10:33:57Z","receivedAt":"2008-02-22T10:33:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Feb 2008, Shawn O. Pearce wrote:\n\n> Ian Clatworthy wrote:\n> \n> > BTW, you might want to either extend the specification (a little) or \n> > fix git-fast-export so they match. :-) See doc/notes.txt under \n> > http://bazaar.launchpad.net/~bzr/bzr-fastimport/fastimport.dev/files \n> > for details. For example, running git-fast-export on 64-bit Hardy \n> > Heron produces file modes longer than permitted if the spec was \n> > strictly interpreted.\n> \n> We may need to take a small hammer to git-fast-export and fix its \n> output. Generating long mode strings like your notes suggest is \n> incorrect.\n\nIndeed. Raising the issue with the original author of git-fast-export \nwould not have hurt either.\n\nCiao,\nDscho\n"},{"id":"69571","messageId":"alpine.LSU.1.00.0802221034140.17164@racer.site","threadId":"12255","inReplyTo":"47BE7E1C.4040303@internode.on.net","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-22T10:36:41Z","receivedAt":"2008-02-22T10:36:41Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Feb 2008, Ian Clatworthy wrote:\n\n> Shawn O. Pearce wrote:\n> \n> > Heh. OOo is _huge_. I think the best import into Git thus far is \n> > taking up about 1.5G of disk space once fully repacked. I don't \n> > recall how they did the import, but coming from SVN I think they used \n> > git-svn, which is not based on git-fast-import.\n> > \n> > What frontend are you using to go from SVN -> fast-import?\n> \n> The pack file in the Git clone I have is 2.4G. I thought that was large \n> but it's quite small compared to the 82G svn dump that creates a 55G svn \n> repo!\n\nThe 2.4G have been compressed (loss-lessly ;-) to less than 1.5G. Unlike \nother SCMs, git has transparent access to the object database, which means \nthat we can actually repack _expensively_ for a better compression.\n\nSo yes, the Git clone you have _is_ 2.4G, but that size is not the best \nsize you _can_ have.\n\nCiao,\nDscho\n"},{"id":"69575","messageId":"20080222113737.GA18581@artemis.madism.org","threadId":"12255","inReplyTo":"20080222073228.GF8410@spearce.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-02-22T11:37:37Z","receivedAt":"2008-02-22T11:37:37Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Fri, Feb 22, 2008 at 07:32:28AM +0000, Shawn O. Pearce wrote:\n> Ian Clatworthy wrote:\n> > FYI. I thought you'd be interested in this as it's inspired by and based\n> > on git-fast-import. You can download the Python source from\n> > https://code.launchpad.net/bzr-fastimport/. In particular, there's a\n> > Python parser of the stream format included that may be useful to the\n> > Git community or other VCS communities. The fast-import-info and\n> > fast-import-filter commands might also be useful to others.\n> \n> This is interesting. I'm not a Python guy, but the info and filter\n> commands do look like they could be useful beyond the Bazaar community.\n> \n> Michael Haggerty of cvs2svn has spent a good amount of time creating\n> a git-fast-import backend to cvs2svn. Given that cvs2svn is one of\n> the few tools that can read some of the really strange real world\n> CVS trees its good to be able to leverage that work for other systems\n> (SVN, Git, and now Bazaar).\n\n /me opens bigs ears and eyes: does this mean that we have an\nincremental importer of CVS based on git-fast-import ? I mean I'm really\ninterested into that, as git-cvsimport is really broken with the glibc\nCVS tree, and as the glibc CVSROOT is rsync-able, an incremental\nimporter that has access to the CVSROOT RCS files is probably the most\nefficient way.\n\n Is there any link you can provide to me about these new features of\ncvs2svn ?\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"69577","messageId":"alpine.LSU.1.00.0802221147310.17164@racer.site","threadId":"12255","inReplyTo":"20080222113737.GA18581@artemis.madism.org","subject":"cvs2svn, was Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-02-22T11:47:56Z","receivedAt":"2008-02-22T11:47:56Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 22 Feb 2008, Pierre Habouzit wrote:\n\n> Is there any link you can provide to me about these new features of\n> cvs2svn ?\n\nhttp://article.gmane.org/gmane.comp.version-control.git/74461/match=cvs2svn\n\nHth,\nDscho\n"},{"id":"69579","messageId":"47BECAD2.7000401@alum.mit.edu","threadId":"12255","inReplyTo":"20080222113737.GA18581@artemis.madism.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2008-02-22T13:14:58Z","receivedAt":"2008-02-22T13:14:58Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"Pierre Habouzit wrote:\n> On Fri, Feb 22, 2008 at 07:32:28AM +0000, Shawn O. Pearce wrote:\n>> Michael Haggerty of cvs2svn has spent a good amount of time creating\n>> a git-fast-import backend to cvs2svn. Given that cvs2svn is one of\n>> the few tools that can read some of the really strange real world\n>> CVS trees its good to be able to leverage that work for other systems\n>> (SVN, Git, and now Bazaar).\n> \n> /me opens bigs ears and eyes: does this mean that we have an\n> incremental importer of CVS based on git-fast-import ?\n\ncvs2svn is robust and uses git-fast-import, but it is *not* incremental.\n Incremental conversion would be fun but it would be a lot of work to\nimplement in such a way that it works reliably.\n\nMichael\n"},{"id":"69582","messageId":"20080222144415.GA19974@yugib.highrise.ca","threadId":"12255","inReplyTo":"20080222113737.GA18581@artemis.madism.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Aidan Van Dyk","fromEmail":"aidan@highrise.ca","sentAt":"2008-02-22T14:44:15Z","receivedAt":"2008-02-22T14:44:15Z","isPatch":false,"sender":{"key":"aidan@highrise.ca","avatar":"https://gravatar.com/avatar/853c50d90cce753dc1c390fdc6cbed558f5f969bd43fa4f5cb0118d8f71316f6?d=mp&s=160"},"body":"* Pierre Habouzit [080201 08:20]:\n \n> /me opens bigs ears and eyes: does this mean that we have an\n> incremental importer of CVS based on git-fast-import ? I mean I'm really\n> interested into that, as git-cvsimport is really broken with the glibc\n> CVS tree, and as the glibc CVSROOT is rsync-able, an incremental\n> importer that has access to the CVSROOT RCS files is probably the most\n> efficient way.\n\nIn the repository I convert (PostgreSQL), I'm using the ruby\nfromcvs/togit converter, which has worked well, because git-cvsimport\ndoesn't work.\n\nI actually found the problem with the PostgreSQL CVS repository - it is\na TAG, which seems to have some cyclic dependencies which throws cvsps\ninto a loop. Unfortuntely, I have neither time nor energy to be able to\nlook into fixing cvsps, especially since fromcvs \"just works\" on it.\n\n\thttp://mid.gmane.org/20080220220014.GB16099@yugib.highrise.ca\n\nI don't know what the problem with the glibc CVSROOT, but if it's the\nsame, that might be something to look at.\n\nNote that fromcvs doesn't import tags (that's probably why it didn't\nhave any trouble with the PostgreSQL CVS) but that doesn't bother me,\nsince CVS tags carry none of the authority git tags do, and the git\ncommit ids provide a stable way to refer to particular commits anyways.\n\na.\n\n-- \nAidan Van Dyk Create like a god,\naidan@highrise.ca command like a king,\nhttp://www.highrise.ca/ work like a slave.\n"},{"id":"69602","messageId":"20080222182041.GB4555@artemis.madism.org","threadId":"12255","inReplyTo":"20080222144415.GA19974@yugib.highrise.ca","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Pierre Habouzit","fromEmail":"madcoder@debian.org","sentAt":"2008-02-22T18:20:42Z","receivedAt":"2008-02-22T18:20:42Z","isPatch":false,"sender":{"key":"madcoder@debian.org","avatar":"https://avatars.githubusercontent.com/u/44708?v=4"},"body":"On Fri, Feb 22, 2008 at 02:44:15PM +0000, Aidan Van Dyk wrote:\n> * Pierre Habouzit [080201 08:20]:\n> \n> > /me opens bigs ears and eyes: does this mean that we have an\n> > incremental importer of CVS based on git-fast-import ? I mean I'm really\n> > interested into that, as git-cvsimport is really broken with the glibc\n> > CVS tree, and as the glibc CVSROOT is rsync-able, an incremental\n> > importer that has access to the CVSROOT RCS files is probably the most\n> > efficient way.\n> \n> In the repository I convert (PostgreSQL), I'm using the ruby\n> fromcvs/togit converter, which has worked well, because git-cvsimport\n> doesn't work.\n\n Well, last time I tried, it exploded miserably (big fat OOM) because\nglibc CVS repository comes back to 1984 or so, and has a very nasty big\nfat Changelog with literally thousands of modifications.\n\n\n-- \n·O· Pierre Habouzit\n··O madcoder@debian.org\nOOO http://www.madism.org\n"},{"id":"69609","messageId":"20080222192502.GB10411@yugib.highrise.ca","threadId":"12255","inReplyTo":"20080222182041.GB4555@artemis.madism.org","subject":"Re: [Fwd: [ANNOUNCE] bzr-fastimport plugin, yet another Bazaar import option]","fromName":"Aidan Van Dyk","fromEmail":"aidan@highrise.ca","sentAt":"2008-02-22T19:25:02Z","receivedAt":"2008-02-22T19:25:02Z","isPatch":false,"sender":{"key":"aidan@highrise.ca","avatar":"https://gravatar.com/avatar/853c50d90cce753dc1c390fdc6cbed558f5f969bd43fa4f5cb0118d8f71316f6?d=mp&s=160"},"body":"* Pierre Habouzit [080222 13:40]:\n \n> Well, last time I tried, it exploded miserably (big fat OOM) because\n> glibc CVS repository comes back to 1984 or so, and has a very nasty big\n> fat Changelog with literally thousands of modifications.\n\n\n\tWhen importing a repository like that wouldn't it make sense to nuke\n\tthe redundant ChangeLog,v?\n\n\n\n-- \nAidan Van Dyk Create like a god,\naidan@highrise.ca command like a king,\nhttp://www.highrise.ca/ work like a slave.\n"}]}