{"thread":{"id":"33153","subject":"Rewriting references to existing commits in commit messages with filter-branch","startedAt":"2013-03-11T11:45:38Z","lastAt":"2013-03-11T15:58:27Z","messageCount":7,"participants":["Vadim Zeitlin","Lawrence Mitchell","Thomas Rast","Michael Haggerty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"211037","messageId":"E1UF1Ag-0006m3-5k@smtp.tt-solutions.com","threadId":"33153","inReplyTo":null,"subject":"Rewriting references to existing commits in commit messages with filter-branch","fromName":"Vadim Zeitlin","fromEmail":"vz-git@zeitlins.org","sentAt":"2013-03-11T11:45:38Z","receivedAt":"2013-03-11T11:45:38Z","isPatch":false,"sender":{"key":"vz-git@zeitlins.org","avatar":null},"body":" Hello,\n\n I'm in the process of converting an existing svn repository to git. The\nfirst step was a standard \"git svn clone\" that went successfully (after\ntaking 15 hours to complete). However I don't want to stop there and would\nlike massage the repository a little before making it publicly available.\n\n The first thing I'd like to do is to replace all references to subversion\nrevision numbers in the commit messages with the corresponding git commit\nSHA1s. I've written a small message filter script called \"svnmsg2git\" that\nsearches for all occurrences of r12345, runs \"git svn find-rev r12345\" and\nthen -- and this is the important part -- looks up the new commit id\ncorresponding to this under .git-rewrite/map. This seemed to work well in\nlimited testing I did initially but after running\n\ngit filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --all\n\non all ~50k revisions, I have a couple of dozens of errors which happen\nbecause the file .git-rewrite/map/$commit doesn't exist yet when I'm trying\nto look it up.\n\n Does anybody know of a way to fix this? This happens apparently because\nfilter-branch doesn't process the commits in their svn order, and when one\nof them is on a branch while the other one is on the trunk, it can happen\nthat the commit references a previous svn revision is processed before the\ncommit corresponding to this revision itself. At least this is the only\nexplanation I see. But even if my hypothesis is correct, I still have no\nidea about how to force filter-branch to do things in the \"right\" order.\n\n Thanks in advance for any ideas!\nVZ\n"},{"id":"211038","messageId":"87haki2la2.fsf@gmx.li","threadId":"33153","inReplyTo":"E1UF1Ag-0006m3-5k@smtp.tt-solutions.com","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Lawrence Mitchell","fromEmail":"wence@gmx.li","sentAt":"2013-03-11T12:06:29Z","receivedAt":"2013-03-11T12:06:29Z","isPatch":false,"sender":{"key":"wence@gmx.li","avatar":"https://avatars.githubusercontent.com/u/1126981?v=4"},"body":"Vadim Zeitlin wrote:\n\n[...]\n\n\n> git filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --all\n\ngit rev-list lists by default in chronological order.  Do you\nwant to pass --topo-order as one of the rev-list options?\n[...]\n\nLawrence\n-- \nLawrence Mitchell <wence@gmx.li>\n"},{"id":"211040","messageId":"loom.20130311T131746-169@post.gmane.org","threadId":"33153","inReplyTo":"87haki2la2.fsf@gmx.li","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Vadim Zeitlin","fromEmail":"vz-git@zeitlins.org","sentAt":"2013-03-11T12:23:45Z","receivedAt":"2013-03-11T12:23:45Z","isPatch":false,"sender":{"key":"vz-git@zeitlins.org","avatar":null},"body":"Lawrence Mitchell <wence <at> gmx.li> writes:\n\n> Vadim Zeitlin wrote:\n> \n> [...]\n> \n> > git filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --all\n> \n> git rev-list lists by default in chronological order.  Do you\n> want to pass --topo-order as one of the rev-list options?\n\n Thanks, this looked like a good idea but reading git-filter-branch code it\nseems to already do it, at\nhttps://github.com/git/git/blob/master/git-filter-branch.sh#L269 you can see\nthat it does \"git rev-list --reverse --topo-order ...\".\n\n So this probably won't help (I could try it just in case I'm missing something\nbut the first errors appear after almost 2 hours of running...). Notice that I\ncould well be wrong in my explanation of what happens, perhaps it's not related\nto the order of processing of the branches/trunk at all. All I know is that when\na chronologically later commit referring to preceding one on a different branch\nis processed by git-filter-branch, sometimes (or perhaps even always) the file\ncorresponding to the previous commit is not yet present in .git-rewrite/map\ndirectory.\n\n Thanks again for any help with this,\nVZ\n"},{"id":"211042","messageId":"87y5dut72l.fsf@pctrast.inf.ethz.ch","threadId":"33153","inReplyTo":"loom.20130311T131746-169@post.gmane.org","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2013-03-11T13:11:14Z","receivedAt":"2013-03-11T13:11:14Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Vadim Zeitlin <vz-git@zeitlins.org> writes:\n\n> Lawrence Mitchell <wence <at> gmx.li> writes:\n>\n>> Vadim Zeitlin wrote:\n>> \n>> [...]\n>> \n>> > git filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --all\n>> \n>> git rev-list lists by default in chronological order.  Do you\n>> want to pass --topo-order as one of the rev-list options?\n>\n>  Thanks, this looked like a good idea but reading git-filter-branch code it\n> seems to already do it, at\n> https://github.com/git/git/blob/master/git-filter-branch.sh#L269 you can see\n> that it does \"git rev-list --reverse --topo-order ...\".\n\nTry overring that with --date-order (you may have to patch the source).\n--topo-order doesn't order by dates.  --date-order does somewhat\n(respecting topology), which in the absence of clock skew should do what\nyou are looking for.\n\nNote that you cannot *remove* --topo-order and use the default, which is\nto only respect dates and not topology; that would break filter-branch.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"211043","messageId":"513DE1E8.7010302@alum.mit.edu","threadId":"33153","inReplyTo":"E1UF1Ag-0006m3-5k@smtp.tt-solutions.com","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Michael Haggerty","fromEmail":"mhagger@alum.mit.edu","sentAt":"2013-03-11T13:53:44Z","receivedAt":"2013-03-11T13:53:44Z","isPatch":false,"sender":{"key":"mhagger@alum.mit.edu","avatar":"https://avatars.githubusercontent.com/u/119718?v=4"},"body":"On 03/11/2013 12:45 PM, Vadim Zeitlin wrote:\n> [...]\n>  The first thing I'd like to do is to replace all references to subversion\n> revision numbers in the commit messages with the corresponding git commit\n> SHA1s. [...] I have a couple of dozens of errors which happen\n> because the file .git-rewrite/map/$commit doesn't exist yet when I'm trying\n> to look it up.\n\nThe quick and dirty solution would be to rewrite your script such that\nif the commit is still unknown to Git, it emits a warning and leaves the\ncommit message unchanged (i.e., leaves the Subversion revision number\nuntouched).  Then simply run the filter-branch a few times until it\nemits no warnings.\n\nMichael\n\n-- \nMichael Haggerty\nmhagger@alum.mit.edu\nhttp://softwareswirl.blogspot.com/\n"},{"id":"211044","messageId":"loom.20130311T145745-986@post.gmane.org","threadId":"33153","inReplyTo":"513DE1E8.7010302@alum.mit.edu","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Vadim Zeitlin","fromEmail":"vz-git@zeitlins.org","sentAt":"2013-03-11T14:05:46Z","receivedAt":"2013-03-11T14:05:46Z","isPatch":false,"sender":{"key":"vz-git@zeitlins.org","avatar":null},"body":"Michael Haggerty <mhagger <at> alum.mit.edu> writes:\n\n> \n> On 03/11/2013 12:45 PM, Vadim Zeitlin wrote:\n> > [...]\n> >  The first thing I'd like to do is to replace all references to subversion\n> > revision numbers in the commit messages with the corresponding git commit\n> > SHA1s. [...] I have a couple of dozens of errors which happen\n> > because the file .git-rewrite/map/$commit doesn't exist yet when I'm trying\n> > to look it up.\n> \n> The quick and dirty solution would be to rewrite your script such that\n> if the commit is still unknown to Git, it emits a warning and leaves the\n> commit message unchanged (i.e., leaves the Subversion revision number\n> untouched).  Then simply run the filter-branch a few times until it\n> emits no warnings.\n\n Thanks, I did think about this but the trouble is that after the first rewrite\n\"git svn find-rev\" wouldn't work any more, so I'd have to do the substitutions\nmanually. Which might be doable as there are not that many of them but, if\npossible, I'd rather do it automatically.\n\n FWIW I'm playing with --date-order now (thanks Thomas) but somehow it seems to\ncreate other problems while fixing (some of) the existing ones. I need to look\nat this more closely to understand what's going on here...\n\nVZ\n"},{"id":"211049","messageId":"loom.20130311T164232-124@post.gmane.org","threadId":"33153","inReplyTo":"87y5dut72l.fsf@pctrast.inf.ethz.ch","subject":"Re: Rewriting references to existing commits in commit messages with filter-branch","fromName":"Vadim Zeitlin","fromEmail":"vz-git@zeitlins.org","sentAt":"2013-03-11T15:58:27Z","receivedAt":"2013-03-11T15:58:27Z","isPatch":false,"sender":{"key":"vz-git@zeitlins.org","avatar":null},"body":"Thomas Rast <trast <at> student.ethz.ch> writes:\n\n> \n> Vadim Zeitlin <vz-git <at> zeitlins.org> writes:\n> \n> > Lawrence Mitchell <wence <at> gmx.li> writes:\n> >\n> >> Vadim Zeitlin wrote:\n> >> \n> >> [...]\n> >> \n> >> > git filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --all\n> >> \n> >> git rev-list lists by default in chronological order.  Do you\n> >> want to pass --topo-order as one of the rev-list options?\n> >\n> >  Thanks, this looked like a good idea but reading git-filter-branch code it\n> > seems to already do it, at\n> > https://github.com/git/git/blob/master/git-filter-branch.sh#L269 you can see\n> > that it does \"git rev-list --reverse --topo-order ...\".\n> \n> Try overring that with --date-order (you may have to patch the source).\n\n Thanks for the hint, this was indeed the solution. And there is actually no\nneed to patch the source because, considering the way git-filter-branch.sh is\nwritten, the user-specified parameters come after the hard-coded --topo-order\nand it seems that --date-order overrides it if it comes after it. So I just had\nto use\n\ngit filter-branch --msg-filter svnmsg2git --tag-name-filter cat -- --date-order\n--all\n\ninstead of my original command. The only remaining question I have is why isn't\n--date-order the default? At least when using message filter, it seems to me\nthat we always want to rewrite commits in chronological order to deal with\npossible back references (even when not migrating from svn, commit messages can\nstill refer to previous commits, like e.g. the ones created by \"git revert\" do\nand they need to be updated when rewriting history). So why not use it in\ngit-filter-branch.sh?\n\n\n BTW, the explanation for the new errors I was getting with --date-order was\nthat I had some artificial commits generated by cvs2svn in the history of this\nrepository which had _exactly_ the same date as the previous commit and\n--date-order sorted them in the wrong order for some reason. I got round this by\nsimply checking for the specific form of the message (which is \"This commit was\ngenerated by cvs2svn to compensate for changes in rNNNNN, which included commits\nto RCS files with non-trunk default branches.\") and replacing \"rNNNNN\" with \"the\nprevious commit\" in this particular case in order to avoid the problem.\n\n Thanks again for your help!\nVZ\n"}]}