{"thread":{"id":"18478","subject":"Importing Bzr revisions","startedAt":"2009-03-23T03:33:34Z","lastAt":"2009-03-26T03:05:20Z","messageCount":11,"participants":["David Reitter","Junio C Hamano","Shawn O. Pearce","Teemu Likonen","Pieter de Bie"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"108995","messageId":"90DBD254-1810-4B11-AA9F-C5661A028FA5@gmail.com","threadId":"18478","inReplyTo":null,"subject":"Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-23T03:33:34Z","receivedAt":"2009-03-23T03:33:34Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"Suppose I have a bzr branch that has been converted (somehow) to a git \nbranch, is it then possible to merge new revisions from the bzr branch \ninto the git one?\nHow would I go about doing that?\n\nThanks\n- David\n\nPS.: please cc me in replies."},{"id":"109016","messageId":"7veiwo8xz7.fsf@gitster.siamese.dyndns.org","threadId":"18478","inReplyTo":"90DBD254-1810-4B11-AA9F-C5661A028FA5@gmail.com","subject":"Re: Importing Bzr revisions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-23T08:06:20Z","receivedAt":"2009-03-23T08:06:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Reitter writes:\n\n> Suppose I have a bzr branch that has been converted (somehow) to a git\n> branch, is it then possible to merge new revisions from the bzr branch\n> into the git one?\n\nIt entirely depends on how that \"somehow\" goes.\n\nIf that \"somehow\" procedure performs a reliably reproducible conversion\n(i.e. not only it will produce the identical git history when you feed the\nsame bzr history to the procedure twice, but it will produce the identical\ngit history followed by new history if you feed the bzr history after new\ncommits are added to the bzr history), you should be able to re-convert\nthe updated bzr history to git and merge the result with the result of the\nearlier conversion. The re-conversion process may not even have to be a\nwhole re-conversion; it could be incremental. But that is entirely up to\nthe quality of the conversion \"somehow\" procedure implements.\n"},{"id":"109049","messageId":"30CDC575-120E-4FAE-8B4D-1971954E913A@gmail.com","threadId":"18478","inReplyTo":"7veiwo8xz7.fsf@gitster.siamese.dyndns.org","subject":"Re: Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-23T12:13:31Z","receivedAt":"2009-03-23T12:13:31Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:\n\n> David Reitter writes:\n>\n>> Suppose I have a bzr branch that has been converted (somehow) to a \n>> git\n>> branch, is it then possible to merge new revisions from the bzr \n>> branch\n>> into the git one?\n>\n> It entirely depends on how that \"somehow\" goes.\n>\n> If that \"somehow\" procedure performs a reliably reproducible \n> conversion\n> (i.e. not only it will produce the identical git history when you \n> feed the\n\nThe question is then: How would one reliably convert a Bzr branch to \ngit? One branch is enough for me, but we're talking >100k revisions, \nso if the procedure involves Bzr tools, it will take longer then is \npracticable.\n\nAlso, my impression is that not many people are doing this, because \nthe fast-export/import combination hasn't seen a lot of development \nactivity. I'm a bit wary to set up my project in git when I don't \njust need to convert once, but in future convert and merge pretty much \nevery day... (The upstream project is going to use Bzr, while I am \ntrying to see if Git is an option for me downstream.)"},{"id":"109057","messageId":"03AC7EDA-2A9F-4626-A67B-CE9F2A88FC7D@gmail.com","threadId":"18478","inReplyTo":"7veiwo8xz7.fsf@gitster.siamese.dyndns.org","subject":"Re: Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-23T13:27:24Z","receivedAt":"2009-03-23T13:27:24Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:\n>\n>> Suppose I have a bzr branch that has been converted (somehow) to a \n>> git\n>> branch, is it then possible to merge new revisions from the bzr \n>> branch\n>> into the git one?\n>\n> It entirely depends on how that \"somehow\" goes.\n>\n> If that \"somehow\" procedure performs a reliably reproducible \n> conversion\n> (i.e. not only it will produce the identical git history when you \n> feed the\n> same bzr history to the procedure twice, but it will produce the \n> identical\n> git history followed by new history if you feed the bzr history \n> after new\n> commits are added to the bzr history), you should be able to re- \n> convert\n\nI'm just experimenting with \"bzr fast-export\", which converts to git, \nand it seems to take about 4 minutes for 1000 revisions on our \n(modern) server. That would be around 7 hours for my emacs \nrepository; I can't do that daily.\n\nI wonder if there's a way for (bzr) fast-export / (git) fast-import to \nwork incrementally, i.e. for selected or most recent revisions.\n\nOr, one could do something like bzr diff -r $REV.. $BBRANCH | (cd \n$GBRANCH; patch -p0; git commit), plus preserving authors and log \nmessages. Is this roughly what the fast-export format does anyways?\n\n"},{"id":"109058","messageId":"20090323135504.GA9732@spearce.org","threadId":"18478","inReplyTo":"03AC7EDA-2A9F-4626-A67B-CE9F2A88FC7D@gmail.com","subject":"Re: Importing Bzr revisions","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-23T13:55:04Z","receivedAt":"2009-03-23T13:55:04Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"David Reitter wrote:\n>\n> I'm just experimenting with \"bzr fast-export\", which converts to git, \n> and it seems to take about 4 minutes for 1000 revisions on our (modern) \n> server. That would be around 7 hours for my emacs repository; I can't do \n> that daily.\n\nNo, you'd want to incrementally do that...\n\n> I wonder if there's a way for (bzr) fast-export / (git) fast-import to \n> work incrementally, i.e. for selected or most recent revisions.\n\nfast-import supports incremental use; git-p4 does it from Perforce.\nThe trick is the application writing the stream (bzr fast-export\nin this case) needs to do something to pick up the prior revisions.\nIt might do that by using the same mark numbers, and requiring you\nto use --import-marks and --export-marks on the git side to save the\nmark database between runs. I don't know, I haven't looked at it.\n\n> Or, one could do something like bzr diff -r $REV.. $BBRANCH | (cd \n> $GBRANCH; patch -p0; git commit), plus preserving authors and log \n> messages. Is this roughly what the fast-export format does anyways?\n\nEh, sort of.\n\nThe fast-import format works on whole files, not patches. So we\nhave to get the entire file from bzr each time it is modified.\nIf the file is a small source file, you almost can't tell the\ndifference in performance. If its a huge binary that changes often,\nit hurts to keep dumping the entire thing over the stream.\n\nBut at the commit level, yes, it preserves authorship and log\nmessages, assuming the bzr fast-export program incldued that data.\nAnd I'm rather certain it does.\n\n-- \nShawn.\n"},{"id":"109064","messageId":"87zlfcz62g.fsf@iki.fi","threadId":"18478","inReplyTo":"03AC7EDA-2A9F-4626-A67B-CE9F2A88FC7D@gmail.com","subject":"Re: Importing Bzr revisions","fromName":"Teemu Likonen","fromEmail":"tlikonen@iki.fi","sentAt":"2009-03-23T14:07:03Z","receivedAt":"2009-03-23T14:07:03Z","isPatch":false,"sender":{"key":"tlikonen@iki.fi","avatar":null},"body":"[I'll Cc to Bazaar list too because this subject is half-Git and\nhalf-Bazaar and we may need help from both sides.]\n\n\nOn 2009-03-23 09:27 (-0400), David Reitter wrote:\n\n> I'm just experimenting with \"bzr fast-export\", which converts to git,\n> and it seems to take about 4 minutes for 1000 revisions on our\n> (modern) server. That would be around 7 hours for my emacs repository;\n> I can't do that daily.\n>\n> I wonder if there's a way for (bzr) fast-export / (git) fast-import to\n> work incrementally, i.e. for selected or most recent revisions.\n\nThey can or should work incrementally, and actually I have succesfully\ndone that. The idea is to use --import-marks and --export-marks options\nwith \"git fast-import\" and --marks option with \"bzr fast-export.\n\nI noticed some problems with newer versions of \"bzr fast-export\", though\n(since it was converted to a proper Bzr command). It seems to corrupt\nthe marks file when doing the first incremental export after the initial\nexport. At least the revisions are not in right order in the marks file\nanymore. \"git fast-import\" can't continue to import from the same\nrevision where it left last time and it seems to create alternative\nhistory -- or something.\n\nReally I don't know if this is a bug in Bzr or in Git and haven't\nfigured out how to file a useful bug report.\n"},{"id":"109068","messageId":"4123DB23-39C4-4651-A3B5-C7876A781365@frim.nl","threadId":"18478","inReplyTo":"03AC7EDA-2A9F-4626-A67B-CE9F2A88FC7D@gmail.com","subject":"Re: Importing Bzr revisions","fromName":"Pieter de Bie","fromEmail":"pieter@frim.nl","sentAt":"2009-03-23T14:18:08Z","receivedAt":"2009-03-23T14:18:08Z","isPatch":false,"sender":{"key":"pieter@frim.nl","avatar":null},"body":"\nOn Mar 23, 2009, at 1:27 PM, David Reitter wrote:\n\n> On Mar 23, 2009, at 4:06 AM, Junio C Hamano wrote:\n>>\n>>> Suppose I have a bzr branch that has been converted (somehow) to a \n>>> git\n>>> branch, is it then possible to merge new revisions from the bzr \n>>> branch\n>>> into the git one?\n>>\n>> It entirely depends on how that \"somehow\" goes.\n>>\n>> If that \"somehow\" procedure performs a reliably reproducible \n>> conversion\n>> (i.e. not only it will produce the identical git history when you \n>> feed the\n>> same bzr history to the procedure twice, but it will produce the \n>> identical\n>> git history followed by new history if you feed the bzr history \n>> after new\n>> commits are added to the bzr history), you should be able to re- \n>> convert\n>\n> I'm just experimenting with \"bzr fast-export\", which converts to \n> git, and it seems to take about 4 minutes for 1000 revisions on our \n> (modern) server. That would be around 7 hours for my emacs \n> repository; I can't do that daily.\n>\n> I wonder if there's a way for (bzr) fast-export / (git) fast-import \n> to work incrementally, i.e. for selected or most recent revisions.\n>\n> Or, one could do something like bzr diff -r $REV.. $BBRANCH | (cd \n> $GBRANCH; patch -p0; git commit), plus preserving authors and log \n> messages. Is this roughly what the fast-export format does anyways?\n\n\nYou might want to take a look at git-bzr (http://github.com/pieter/git-bzr/tree/master \n) it allows incremental bidirectional interaction between git and bzr \nusing the fast-export/import, so it might just work in your case. \nThere are some issues with it, so you might want to check the \n'network' part on github and use one of the other variants.\n\nThat said, it's a 100 line script that hasn't been used much, so good \nluck :)\n\n- Pieter\n"},{"id":"109078","messageId":"D05262B5-9B18-49C9-8312-E69926F967AF@gmail.com","threadId":"18478","inReplyTo":"4123DB23-39C4-4651-A3B5-C7876A781365@frim.nl","subject":"Re: Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-23T15:18:06Z","receivedAt":"2009-03-23T15:18:06Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"On Mar 23, 2009, at 10:18 AM, Pieter de Bie wrote:\n>\n> You might want to take a look at git-bzr (http://github.com/pieter/git-bzr/tree/master \n> ) it allows incremental bidirectional interaction between git and \n> bzr using the fast-export/import, so it might just work in your \n> case. There are some issues with it, so you might want to check the \n> 'network' part on github and use one of the other variants.\n>\n> That said, it's a 100 line script that hasn't been used much, so \n> good luck :)\n\nThanks, this appears to be useful.\n\nAFter reading your script, I tried incremental exports on the bzr \nside: It still supports --import-marks and this seems to work quickly \nenough for my small test project.\n\nCould you update the documentation as to how to install the git-bzr \nscript?\n(I searched for \"plugin\" in the git user manual, and google, and \ndidn't find anything. I haven't used git much, so I'm a bit naive as \nregards to such a question.)\n\n\n"},{"id":"109101","messageId":"CF094EC6-72B6-4523-92ED-6582ADC4C1EE@frim.nl","threadId":"18478","inReplyTo":"D05262B5-9B18-49C9-8312-E69926F967AF@gmail.com","subject":"Re: Importing Bzr revisions","fromName":"Pieter de Bie","fromEmail":"pieter@frim.nl","sentAt":"2009-03-23T20:15:47Z","receivedAt":"2009-03-23T20:15:47Z","isPatch":false,"sender":{"key":"pieter@frim.nl","avatar":null},"body":"\nOn 23 mrt 2009, at 15:18, David Reitter wrote:\n\n> Could you update the documentation as to how to install the git-bzr \n> script?\n> (I searched for \"plugin\" in the git user manual, and google, and \n> didn't find anything. I haven't used git much, so I'm a bit naive \n> as regards to such a question.)\n\nThere's nothing to install, just call the script.. if you run it \nwithout commands it should give some usage information. The README in \nthe repository should tell the rest. If you'd like to call it using \n'git bzr' rather than 'git-bzr', you have to put it somewhere in your \nPATH\n"},{"id":"109103","messageId":"056D9E7A-92CE-4924-BEA8-A28A25C3B914@gmail.com","threadId":"18478","inReplyTo":"CF094EC6-72B6-4523-92ED-6582ADC4C1EE@frim.nl","subject":"Re: Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-23T21:00:36Z","receivedAt":"2009-03-23T21:00:36Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"On Mar 23, 2009, at 4:15 PM, Pieter de Bie wrote:\n>> Could you update the documentation as to how to install the git-bzr \n>> script?\n>> (I searched for \"plugin\" in the git user manual, and google, and \n>> didn't find anything. I haven't used git much, so I'm a bit naive \n>> as regards to such a question.)\n>\n> There's nothing to install, just call the script.. if you run it \n> without commands it should give some usage information. The README \n> in the repository should tell the rest. If you'd like to call it \n> using 'git bzr' rather than 'git-bzr', you have to put it somewhere \n> in your PATH\n\nSure, okay, thanks. (In my bzr mindset I didn't realize that \"git \nbzr\" just elegantly translated to \"git-bzr\".)\n\nI'm currently attempting to convert those 100k revisions from the \nemacs repository, averaging 5.5min/1000 revisions. Judging from CPU \nload (if that's a valid thing to do), I see a 9:1 distribution of the \nworkload (bzr, reading, vs. git, writing).\n\nThanks for helping me out with this so far."},{"id":"109461","messageId":"055EE409-55A1-458F-883D-63CA76B991E3@gmail.com","threadId":"18478","inReplyTo":"87zlfcz62g.fsf@iki.fi","subject":"Re: Importing Bzr revisions","fromName":"David Reitter","fromEmail":"david.reitter@gmail.com","sentAt":"2009-03-26T03:05:20Z","receivedAt":"2009-03-26T03:05:20Z","isPatch":false,"sender":{"key":"david.reitter@gmail.com","avatar":"https://gravatar.com/avatar/0b9f43c735737567aec3272fea5afa461ffa049f180103c1aab629aa6c1207f1?d=mp&s=160"},"body":"On Mar 23, 2009, at 10:07 AM, Teemu Likonen wrote:\n\n>> I'm just experimenting with \"bzr fast-export\", which converts to git,\n>> and it seems to take about 4 minutes for 1000 revisions on our\n>> (modern) server. That would be around 7 hours for my emacs \n>> repository;\n>> I can't do that daily.\n>>\n>> I wonder if there's a way for (bzr) fast-export / (git) fast-import \n>> to\n>> work incrementally, i.e. for selected or most recent revisions.\n>\n> They can or should work incrementally, and actually I have succesfully\n> done that. The idea is to use --import-marks and --export-marks \n> options\n> with \"git fast-import\" and --marks option with \"bzr fast-export.\n>\n> I noticed some problems with newer versions of \"bzr fast-export\", \n> though\n> (since it was converted to a proper Bzr command). It seems to corrupt\n> the marks file when doing the first incremental export after the \n> initial\n> export. At least the revisions are not in right order in the marks \n> file\n> anymore. \"git fast-import\" can't continue to import from the same\n> revision where it left last time and it seems to create alternative\n> history -- or something.\n>\n> Really I don't know if this is a bug in Bzr or in Git and haven't\n> figured out how to file a useful bug report.\n\n\n\nI'm experiencing pretty much the same problem.\n\nLooking at the code (marks_file.py) I don't see why the order would \nmatter (even though it would be nicer if the order was consistent). I \nactually changed this so that it's always sorted, just to help me debug.\n\nNow, I'm getting these errors back from git:\n\nfatal: mark :96985 not declared\nfast-import: dumping crash report to .git/fast_import_crash_74262\nbzr: broken pipe\n\nI couldn't reproduce this with a simple repository.\n\nHowever, if one inspects the output of bzr fast-export, one finds \nstuff like this:\n\ncommit refs/heads/master\nmark :96984\ncommitter 1237847130 +0000\ndata 205\n....\nfrom :96985\nM 644 inline lisp/ChangeLog\ndata 741178\n--- ... --- ... --- ... ---\ncommit refs/heads/master\nmark :96985\ncommitter 1237849747 +0000\n...\nfrom :96984\nM 644 inline src/gtkutil.c\ndata 135796\n\nI'm not sure about the structure of these files, but my educated guess \nwould be that this is a circular reference.\nStrange. One would think that this should never happen.\n\nThat said, I get the same errors in other cases as well without \ncircular reference.\n\nAs an experiment, I deleted the last 1000 or so revisions from the bzr \nmarks file, so that they would be output again.\nA couple of minutes and 430MB in output later, I imported this on the \ngit side, which, after a few seconds, came back with this:\n---------------------------------------------------------------------\nAlloc'd objects: 105000\nTotal objects: 5 ( 5063 duplicates )\n blobs : 0 ( 1549 duplicates 0 deltas)\n trees : 2 ( 2505 duplicates 0 deltas)\n commits: 3 ( 1009 duplicates 0 deltas)\n tags : 0 ( 0 duplicates 0 deltas)\nTotal branches: 1 ( 1 loads )\n marks: 1048576 ( 97012 unique )\n atoms: 1937\nMemory total: 5380 KiB\n pools: 2098 KiB\n objects: 3281 KiB\n---------------------------------------------------------------------\npack_report: getpagesize() = 4096\npack_report: core.packedGitWindowSize = 33554432\npack_report: core.packedGitLimit = 268435456\npack_report: pack_used_ctr = 104764\npack_report: pack_mmap_calls = 6\npack_report: pack_open_windows = 4 / 4\npack_report: pack_mapped = 100666262 / 100666262\n---------------------------------------------------------------------\n\nSo, it seems like there were 3 commits that were missing from previous \ntransfers. The final lines of the marks files for bzr and git seem \ncoherent. Hard to identify the culprit.\n\nFurther changes resulted in good conversion so far. Deleting a good \nnumber of the most recent marks entries seems to be the right thing to \nrecover.\n\nA \"bzr uncommit\" seemed not to make its way to the git side. No \ngood. I wonder if that is going to create a lasting inconsistency. I \ndid get this subsequently:\nwarning: Not updating refs/heads/master (new tip \n6c81ccc916026d020eadeb0ad6e5b12c18aeccd3 does not contain \n3222cfee5bc00412b6f5e52a420f93564f586ee9)\nThis \"not contained\" revision resolved to the uncommitted one - that \nmakes sense.\nBut what consequences does the warning have...?\n\n\n\n"}]}