{"thread":{"id":"46216","subject":"Transform log message during migration svn -> git (using git-svn)","startedAt":"2017-06-20T07:33:08Z","lastAt":"2017-06-20T18:50:16Z","messageCount":6,"participants":["paul.mattke@s4m.com","Lars Schneider","Jeff King","Andreas Heiduk"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"322677","messageId":"8ce2bf074fcf4d6181a29eb3e7fff132@GTLBML3EX0005.bagmail.net","threadId":"46216","inReplyTo":null,"subject":"Transform log message during migration svn -> git (using git-svn)","fromName":"","fromEmail":"paul.mattke@s4m.com","sentAt":"2017-06-20T07:32:59Z","receivedAt":"2017-06-20T07:33:08Z","isPatch":false,"sender":{"key":"paul.mattke@s4m.com","avatar":null},"body":"Hi there,\n\nthis is actually not really a bug report, but much more a feature request\n(if I did not oversee an already existing feature like this):\n\nWe want to migrate our SVN repository to GIT and will be using git-svn for\nthat of course. Currently in SVN, all our commit log messages start either\nwith:\n\n123456 (a number, representing the Bug Id in our old legacy bug tracker)\n\nor\n\nT123456 (a number, but prefixed with T, referring a TFS item in this case)\n\nDuring conversion to GIT, we want to replace the T in such log messages with\na #, so commits, referring a TFS item will start with #123456 in the future.\nWe don’t care about log messages which do not start with a T, only the\nTXXXXXX messages need to be transformed here.\n\nI guess an operation like this is currently not possible with git-svn, isn’t\nit? So it would be nice, if a feature could be implemented that gives the\nuser the possibility to specify some kind of script file for example, which\ntransforms the log message in any way we want it.\n\nPaul Mattke\nSoftware Developer\n-------------------------------------------------\nArvato Systems S4M GmbH\nAm Coloneum 3\n50829 Köln\n \nPhone: +49 221 28555-443\nFax: +49 221 28555-210\nE-Mail: paul.mattke@s4m.com\nwww.s4m.arvato-systems.com\n\n"},{"id":"322728","messageId":"EAA4B747-FA00-4A3B-88CD-AF8EBE1F8586@gmail.com","threadId":"46216","inReplyTo":"8ce2bf074fcf4d6181a29eb3e7fff132@GTLBML3EX0005.bagmail.net","subject":"Re: Transform log message during migration svn -> git (using git-svn)","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-20T09:32:08Z","receivedAt":"2017-06-20T09:32:42Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 20 Jun 2017, at 09:32, paul.mattke@s4m.com wrote:\n> \n> Hi there,\n> \n> this is actually not really a bug report, but much more a feature request\n> (if I did not oversee an already existing feature like this):\n> \n> We want to migrate our SVN repository to GIT and will be using git-svn for\n> that of course. Currently in SVN, all our commit log messages start either\n> with:\n> \n> 123456 (a number, representing the Bug Id in our old legacy bug tracker)\n> \n> or\n> \n> T123456 (a number, but prefixed with T, referring a TFS item in this case)\n> \n> During conversion to GIT, we want to replace the T in such log messages with\n> a #, so commits, referring a TFS item will start with #123456 in the future.\n> We don’t care about log messages which do not start with a T, only the\n> TXXXXXX messages need to be transformed here.\n> \n> I guess an operation like this is currently not possible with git-svn, isn’t\n> it? So it would be nice, if a feature could be implemented that gives the\n> user the possibility to specify some kind of script file for example, which\n> transforms the log message in any way we want it.\n\nYou can migrate your repo from SVN to Git as is. Afterwards you can\nfix up the commit messages with the following command:\n\ngit filter-branch -f --msg-filter 'perl -lape \"s/^T(\\d+)/#\\$1/\"'\n\n(this might take a while on a large repo)\n\n- Lars"},{"id":"322731","messageId":"90ee80a910884e1e98fa8a5001ae850c@GTLBML3EX0005.bagmail.net","threadId":"46216","inReplyTo":"EAA4B747-FA00-4A3B-88CD-AF8EBE1F8586@gmail.com","subject":"AW: Transform log message during migration svn -> git (using git-svn)","fromName":"","fromEmail":"paul.mattke@s4m.com","sentAt":"2017-06-20T12:32:35Z","receivedAt":"2017-06-20T12:32:45Z","isPatch":false,"sender":{"key":"paul.mattke@s4m.com","avatar":null},"body":"Well this is a possibility, of course. Our problem is that our SVN\nrepository contains about 220.000 revisions currently. As a colleague of\nmine said that the command you suggest might take about 4 seconds per\nrevision, it would take about 10 days to do this for our whole repository.\nSo of course it could save a lot of time generally if such operation could\nbe done immediately during git-svn.\n\nPaul Mattke\nSoftware Developer\n-------------------------------------------------\nArvato Systems S4M GmbH\nAm Coloneum 3\n50829 Köln\n \nPhone: +49 221 28555-443\nFax: +49 221 28555-210\nE-Mail: paul.mattke@s4m.com\nwww.s4m.arvato-systems.com\n\n\n-----Ursprüngliche Nachricht-----\nVon: Lars Schneider [mailto:larsxschneider@gmail.com] \nGesendet: Dienstag, 20. Juni 2017 11:32\nAn: Mattke, Paul, NMM-BPDD <paul.mattke@s4m.com>\nCc: git@vger.kernel.org\nBetreff: Re: Transform log message during migration svn -> git (using\ngit-svn)\n\n\n> On 20 Jun 2017, at 09:32, paul.mattke@s4m.com wrote:\n> \n> Hi there,\n> \n> this is actually not really a bug report, but much more a feature \n> request (if I did not oversee an already existing feature like this):\n> \n> We want to migrate our SVN repository to GIT and will be using git-svn \n> for that of course. Currently in SVN, all our commit log messages \n> start either\n> with:\n> \n> 123456 (a number, representing the Bug Id in our old legacy bug \n> tracker)\n> \n> or\n> \n> T123456 (a number, but prefixed with T, referring a TFS item in this \n> case)\n> \n> During conversion to GIT, we want to replace the T in such log \n> messages with a #, so commits, referring a TFS item will start with\n#123456 in the future.\n> We don’t care about log messages which do not start with a T, only the \n> TXXXXXX messages need to be transformed here.\n> \n> I guess an operation like this is currently not possible with git-svn, \n> isn’t it? So it would be nice, if a feature could be implemented that \n> gives the user the possibility to specify some kind of script file for \n> example, which transforms the log message in any way we want it.\n\nYou can migrate your repo from SVN to Git as is. Afterwards you can fix up\nthe commit messages with the following command:\n\ngit filter-branch -f --msg-filter 'perl -lape \"s/^T(\\d+)/#\\$1/\"'\n\n(this might take a while on a large repo)\n\n- Lars\n"},{"id":"322732","messageId":"BDA6D349-0DF9-49F3-B2B6-0B1AE3BBD052@gmail.com","threadId":"46216","inReplyTo":"90ee80a910884e1e98fa8a5001ae850c@GTLBML3EX0005.bagmail.net","subject":"Re: Transform log message during migration svn -> git (using git-svn)","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-06-20T12:46:22Z","receivedAt":"2017-06-20T12:46:29Z","isPatch":false,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 20 Jun 2017, at 14:32, <paul.mattke@s4m.com> <paul.mattke@s4m.com> wrote:\n> \n> Well this is a possibility, of course. Our problem is that our SVN\n> repository contains about 220.000 revisions currently. As a colleague of\n> mine said that the command you suggest might take about 4 seconds per\n> revision, it would take about 10 days to do this for our whole repository.\n> So of course it could save a lot of time generally if such operation could\n> be done immediately during git-svn.\n\nYou colleague is most likely correct. I suggested it as this is a one time\noperation and therefore still somewhat practical from my point of view.\n\nIf you don't like the solution then you need to change the git-svn code.\nProbably here somewhere (I am not familiar with this code):\nhttps://github.com/git/git/blob/master/git-svn.perl#L1836\n\n- Lars\n\nPS: Please don't top post on this mailing list :-)\nhttps://en.wikipedia.org/wiki/Posting_style#Top-posting\n\n\n\n> \n> Paul Mattke\n> Software Developer\n> -------------------------------------------------\n> Arvato Systems S4M GmbH\n> Am Coloneum 3\n> 50829 Köln\n>  \n> Phone: +49 221 28555-443\n> Fax: +49 221 28555-210\n> E-Mail: paul.mattke@s4m.com\n> www.s4m.arvato-systems.com\n> \n> \n> -----Ursprüngliche Nachricht-----\n> Von: Lars Schneider [mailto:larsxschneider@gmail.com] \n> Gesendet: Dienstag, 20. Juni 2017 11:32\n> An: Mattke, Paul, NMM-BPDD <paul.mattke@s4m.com>\n> Cc: git@vger.kernel.org\n> Betreff: Re: Transform log message during migration svn -> git (using\n> git-svn)\n> \n> \n>> On 20 Jun 2017, at 09:32, paul.mattke@s4m.com wrote:\n>> \n>> Hi there,\n>> \n>> this is actually not really a bug report, but much more a feature \n>> request (if I did not oversee an already existing feature like this):\n>> \n>> We want to migrate our SVN repository to GIT and will be using git-svn \n>> for that of course. Currently in SVN, all our commit log messages \n>> start either\n>> with:\n>> \n>> 123456 (a number, representing the Bug Id in our old legacy bug \n>> tracker)\n>> \n>> or\n>> \n>> T123456 (a number, but prefixed with T, referring a TFS item in this \n>> case)\n>> \n>> During conversion to GIT, we want to replace the T in such log \n>> messages with a #, so commits, referring a TFS item will start with\n> #123456 in the future.\n>> We don’t care about log messages which do not start with a T, only the \n>> TXXXXXX messages need to be transformed here.\n>> \n>> I guess an operation like this is currently not possible with git-svn, \n>> isn’t it? So it would be nice, if a feature could be implemented that \n>> gives the user the possibility to specify some kind of script file for \n>> example, which transforms the log message in any way we want it.\n> \n> You can migrate your repo from SVN to Git as is. Afterwards you can fix up\n> the commit messages with the following command:\n> \n> git filter-branch -f --msg-filter 'perl -lape \"s/^T(\\d+)/#\\$1/\"'\n> \n> (this might take a while on a large repo)\n> \n> - Lars\n\n"},{"id":"322738","messageId":"20170620145101.6mjikvmxegw5pbwj@sigill.intra.peff.net","threadId":"46216","inReplyTo":"BDA6D349-0DF9-49F3-B2B6-0B1AE3BBD052@gmail.com","subject":"Re: Transform log message during migration svn -> git (using git-svn)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-20T14:51:01Z","receivedAt":"2017-06-20T14:51:11Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2017 at 02:46:22PM +0200, Lars Schneider wrote:\n\n> \n> > On 20 Jun 2017, at 14:32, <paul.mattke@s4m.com> <paul.mattke@s4m.com> wrote:\n> > \n> > Well this is a possibility, of course. Our problem is that our SVN\n> > repository contains about 220.000 revisions currently. As a colleague of\n> > mine said that the command you suggest might take about 4 seconds per\n> > revision, it would take about 10 days to do this for our whole repository.\n> > So of course it could save a lot of time generally if such operation could\n> > be done immediately during git-svn.\n> \n> You colleague is most likely correct. I suggested it as this is a one time\n> operation and therefore still somewhat practical from my point of view.\n\nI didn't follow this whole thread, but I happened to see this bit. I\nthink the command in question is:\n\n  git filter-branch -f --msg-filter 'perl -lape \"s/^T(\\d+)/#\\$1/\"'\n\nI know filter-branch is slow, but a msg-filter should be relatively\nfast.  I'd be surprised at 4 seconds per revision (the main cost is\nkicking off a new perl process per revision). It's more like 120/sec on\nmy machine.\n\nHowever, I think the fastest way would be to do it with fast-export,\nwhere you can just tweak the stream as it flows through:\n\n  # set up a new repo to hold the results; we won't bother\n  # copying the blobs, so just point at the current repo as an\n  # alternate.\n  git init fixed-repo\n  echo \"../../../.git/objects\" >fixed-repo/.git/objects/info/alternates\n\n  git fast-export --no-data --all |\n  perl -ne '\n\t# look for \"data\" chunks which contain the commit message\n\tif (/^data (\\d+)/) {\n\t\tread STDIN, my $buf, $1;\n\t\t$buf =~ s/^T(\\d+)/#$1/;\n\t\tprint \"data \", length($buf), \"\\n\";\n\t\tprint $buf;\n\t} else {\n\t\tprint;\n\t}\n  ' |\n  git -C fixed-repo fast-import\n\nThat runs at about 3600 commits/sec on my machine.\n\nMost of that time goes to doing a tree diff on each commit. Technically\nthat is not required for your use case, but I don't think there's a way\nto get fast-export to skip that (and it's an inherent part of the\nfast-import stream). It's probably fast enough, but it's possible that\na specialized tool like BFG repo cleaner[1] could do better (I don't\nknow offhand if it handles commit message rewrites or not).\n\n-Peff\n\n[1] https://rtyley.github.io/bfg-repo-cleaner/\n"},{"id":"322754","messageId":"e7cb2b23-21ec-f124-5198-7bb72e601184@gmail.com","threadId":"46216","inReplyTo":"90ee80a910884e1e98fa8a5001ae850c@GTLBML3EX0005.bagmail.net","subject":"Re: Transform log message during migration svn -> git (using git-svn)","fromName":"Andreas Heiduk","fromEmail":"asheiduk@gmail.com","sentAt":"2017-06-20T18:50:08Z","receivedAt":"2017-06-20T18:50:16Z","isPatch":false,"sender":{"key":"asheiduk@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9371344?v=4"},"body":"\nAm 20.06.2017 um 14:32 schrieb paul.mattke@s4m.com:\n> Well this is a possibility, of course. Our problem is that our SVN\n> repository contains about 220.000 revisions currently. As a colleague of\n> mine said that the command you suggest might take about 4 seconds per\n> revision, it would take about 10 days to do this for our whole repository.\n> So of course it could save a lot of time generally if such operation could\n> be done immediately during git-svn.\n\nMy data point is this: A \"git filter branch\" run with ~2000 revisions\ntook several hours on a Windows 7 box. That number seems to be roughly\nthe same as your number. A comparable run on a Linux box took only about\n10 minutes.\n\nSo: If your benchmark was done on Windows you might do that also on Linux.\n"}]}