{"thread":{"id":"42588","subject":"Un-paged commit messages in git filter-branch's commit-filter?","startedAt":"2016-06-16T02:19:53Z","lastAt":"2016-08-06T23:18:22Z","messageCount":8,"participants":["Stefan Tauner","Jeff King","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"289081","messageId":"0McmFl-1aux6M2oV7-00Hv9A@mail.gmx.com","threadId":"42588","inReplyTo":null,"subject":"Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Stefan Tauner","fromEmail":"stefan.tauner@gmx.at","sentAt":"2016-06-13T06:28:18Z","receivedAt":"2016-06-16T02:19:53Z","isPatch":false,"sender":{"key":"stefan.tauner@gmx.at","avatar":null},"body":"Hello,\n\nI am trying to do a major cleanup of the repository in one of my\nprojects (and switch from git-svn to native git). I have developed a\ncommit-filter script over the last months that massages partially\ndreadful commit messages into something acceptable. While I am not 100%\nsure I think that upgrading git has broken it partially. AFAICT since\nthe update the commit-filter does not get the original message anymore\nbut at least the subject/first paragraph is run through a pager or\nsomething similar:\nThe first line is broken into multiple lines (i.e. some line breaks are\ninserted about every 72 characters where none have been before).\n\nI have tried to run \"git --no-pager filter-branch ...\" to no avail. I\nhave briefly looked at the source but could not find any proofs...\nAny hints would be appreciated. This is how I run my script:\n\ntmpvar=\"$( I am trying to do a major cleanup of the repository in one of my\n> projects (and switch from git-svn to native git). I have developed a\n> commit-filter script over the last months that massages partially\n> dreadful commit messages into something acceptable. While I am not 100%\n> sure I think that upgrading git has broken it partially. AFAICT since\n> the update the commit-filter does not get the original message anymore\n> but at least the subject/first paragraph is run through a pager or\n> something similar:\n> The first line is broken into multiple lines (i.e. some line breaks are\n> inserted about every 72 characters where none have been before).\n\nThere are some output formats that will wrap lines, but by default,\nfilter-branch should not be using them (and I could not reproduce the\nissue in a simple test). Can you show us what your commit-filter looks\nlike?\n\n-Peff\n"},{"id":"292688","messageId":"0MTjMy-1buKad2Fg8-00QUQV@mail.gmx.com","threadId":"42588","inReplyTo":"20160616095947.GA15988@sigill.intra.peff.net","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Stefan Tauner","fromEmail":"stefan.tauner@gmx.at","sentAt":"2016-07-31T16:39:35Z","receivedAt":"2016-07-31T16:40:49Z","isPatch":false,"sender":{"key":"stefan.tauner@gmx.at","avatar":null},"body":"On Thu, 16 Jun 2016 05:59:47 -0400\nJeff King wrote:\n\n> On Mon, Jun 13, 2016 at 08:28:18AM +0200, Stefan Tauner wrote:\n> \n> > I am trying to do a major cleanup of the repository in one of my\n> > projects (and switch from git-svn to native git). I have developed a\n> > commit-filter script over the last months that massages partially\n> > dreadful commit messages into something acceptable. While I am not 100%\n> > sure I think that upgrading git has broken it partially. AFAICT since\n> > the update the commit-filter does not get the original message anymore\n> > but at least the subject/first paragraph is run through a pager or\n> > something similar:\n> > The first line is broken into multiple lines (i.e. some line breaks are\n> > inserted about every 72 characters where none have been before). \n> \n> There are some output formats that will wrap lines, but by default,\n> filter-branch should not be using them (and I could not reproduce the\n> issue in a simple test). Can you show us what your commit-filter looks\n> like?\n\nThanks for your answer. I have tried to reproduce it in other (newly\ncreated) repositories but failed. However, it seems to relate to some\nkind of persistent paging setting, is that possible?\ngit config -l does not show anything suspicious.\n\nThe following commands produce paged output:\ngit show hash\ngit show --pretty=%B\ngit log hash^..hash\nCommit message in gitk\n\n\nThese do NOT produce paged output:\ngit patch hash^..hash\nCommit message in gitg 0.2.7\n\n\nThis is the script I tried to use to reproduce the problem:\n\n#!/bin/bash\nexport LC_ALL=C\ninput=$(cat)\necho \"===========================\n$input\n===========================\" >> /tmp/paging_bug.txt\ngit commit-tree \"$@\" -m \"$input\"\n\n-- \nKind regards/Mit freundlichen Grüßen, Stefan Tauner\n"},{"id":"292798","messageId":"20160801213631.2ttdlermxw2wbnbi@sigill.intra.peff.net","threadId":"42588","inReplyTo":"0MTjMy-1buKad2Fg8-00QUQV@mail.gmx.com","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-01T21:36:31Z","receivedAt":"2016-08-01T21:37:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jul 31, 2016 at 06:39:35PM +0200, Stefan Tauner wrote:\n\n> > There are some output formats that will wrap lines, but by default,\n> > filter-branch should not be using them (and I could not reproduce the\n> > issue in a simple test). Can you show us what your commit-filter looks\n> > like?\n> \n> Thanks for your answer. I have tried to reproduce it in other (newly\n> created) repositories but failed. However, it seems to relate to some\n> kind of persistent paging setting, is that possible?\n> git config -l does not show anything suspicious.\n> \n> The following commands produce paged output:\n> git show hash\n> git show --pretty=%B\n> git log hash^..hash\n> Commit message in gitk\n> \n> \n> These do NOT produce paged output:\n> git patch hash^..hash\n> Commit message in gitg 0.2.7\n\nWhat is \"git patch\"? An alias for \"format-patch?\".\n\n> This is the script I tried to use to reproduce the problem:\n> \n> #!/bin/bash\n> export LC_ALL=C\n> input=$(cat)\n> echo \"===========================\n> $input\n> ===========================\" >> /tmp/paging_bug.txt\n> git commit-tree \"$@\" -m \"$input\"\n\nCan you be more specific about the input you're feeding to git and the\noutput you're seeing?\n\nFor instance, if I do:\n\n git init\n echo content >file\n git add file\n git commit -m \"$(perl -e 'print join(\" \", 1..100)')\"\n\nI get a commit message with one long unwrapped line, which I can view\nvia git-log, etc. Now if I try to run filter-branch on that:\n\n git filter-branch --commit-filter '\n\tinput=$(cat)\n\t{\n\t\techo \"====================\"\n\t\techo $input\n\t\techo \"====================\"\n\t} >>/tmp/paging_bug.txt\n\tgit commit-tree \"$@\" -m \"$input\"\n '\n\nthen the commit remains unchanged, and paging_bug shows one long line.\nWhat am I missing?\n\n(I wondered at first if the extra \"cat\" and \"-m\" could be messing up\nwhitespace for you, but it should not, as the quoting around \"$input\"\nshould preserve things like newlines. And anyway, the bug in that case\nwould be the _opposite_; I'd expect it to stuff everything onto a single\nline rather than breaking lines).\n\n-Peff\n"},{"id":"292813","messageId":"20160801232429.mb43wxev5lyhxyiw@sigill.intra.peff.net","threadId":"42588","inReplyTo":"0LzskF-1bGAH81g5T-014za7@mail.gmx.com","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-08-01T23:24:29Z","receivedAt":"2016-08-01T23:25:45Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 01, 2016 at 11:49:09PM +0200, Stefan Tauner wrote:\n\n> > For instance, if I do:\n> > \n> > git init\n> > echo content >file\n> > git add file\n> > git commit -m \"$(perl -e 'print join(\" \", 1..100)')\"\n> > \n> > I get a commit message with one long unwrapped line, which I can view\n> > via git-log, etc.\n> \n> That's approximately what I did in my tests as well. And like you, when\n> I do this in a fresh repository, it works like that..\n\nOne thing to look at, I guess, is whether they are corrupted coming in\nto the repository, or when they are being formatted.\n\nIf you do:\n\n git cat-file commit HEAD\n\nyou will get the raw bytes of the commit object stored by git. In the\nexample above, it obviously shows one long line. Have you checked that\nit does so in your cases that misbehave?\n\n> > (I wondered at first if the extra \"cat\" and \"-m\" could be messing up\n> > whitespace for you, but it should not, as the quoting around \"$input\"\n> > should preserve things like newlines. And anyway, the bug in that case\n> > would be the _opposite_; I'd expect it to stuff everything onto a single\n> > line rather than breaking lines).\n> \n> The commit messages I try to process are nothing special really... just\n> very long and not subject-like (because SVN and not giving too much\n> thought to them sometimes). The only special thing I can think of is\n> that they have been processed by git-svn earlier.\n\nHmm. The usual problem with svn-imported commits is not long lines,\nexactly, but rather that the commit message has one big paragraph at the\ntop, rather than a subject/body split.\n\nSo when you ask git for the \"subject\" in such a case, it may paste many\nlines together as a single one. For example:\n\n $ commit=$(seq 1 5 | git commit-tree HEAD^{tree})\n $ git cat-file commit $commit\n tree 07753f428765ac1afe2020b24e40785869bd4a85\n author Jeff King 1470093739 -0400\n committer Jeff King 1470093739 -0400\n\n 1\n 2\n 3\n 4\n 5\n\n $ git log --format=%s $commit\n 1 2 3 4 5\n\nSo could it be that your lines actually _are_ broken in the git objects,\nbut \"%s\" and other tools try to salvage them as a single subject?\n\nI don't recall offhand whether git-svn does line-wrapping or any other\ncommit-message munging.\n\n-Peff\n"},{"id":"292817","messageId":"20160802000736.GA8716@starla","threadId":"42588","inReplyTo":"20160801232429.mb43wxev5lyhxyiw@sigill.intra.peff.net","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2016-08-02T00:07:36Z","receivedAt":"2016-08-02T00:08:36Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jeff King wrote:\n> I don't recall offhand whether git-svn does line-wrapping or any other\n> commit-message munging.\n\nDefinitely no line-wrapping. Munging is minimal:\nit respects i18n.commitencoding, adds a trailing newline,\nand \"git-svn-id:\" line.\n"},{"id":"292838","messageId":"0LzskF-1bGAH81g5T-014za7@mail.gmx.com","threadId":"42588","inReplyTo":"20160801213631.2ttdlermxw2wbnbi@sigill.intra.peff.net","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Stefan Tauner","fromEmail":"stefan.tauner@gmx.at","sentAt":"2016-08-01T21:49:09Z","receivedAt":"2016-08-02T14:35:37Z","isPatch":false,"sender":{"key":"stefan.tauner@gmx.at","avatar":null},"body":"On Mon, 1 Aug 2016 17:36:31 -0400\nJeff King wrote:\n\n> On Sun, Jul 31, 2016 at 06:39:35PM +0200, Stefan Tauner wrote:\n> \n> > > There are some output formats that will wrap lines, but by default,\n> > > filter-branch should not be using them (and I could not reproduce the\n> > > issue in a simple test). Can you show us what your commit-filter looks\n> > > like? \n> > \n> > Thanks for your answer. I have tried to reproduce it in other (newly\n> > created) repositories but failed. However, it seems to relate to some\n> > kind of persistent paging setting, is that possible?\n> > git config -l does not show anything suspicious.\n> > \n> > The following commands produce paged output:\n> > git show hash\n> > git show --pretty=%B\n> > git log hash^..hash\n> > Commit message in gitk\n> > \n> > \n> > These do NOT produce paged output:\n> > git patch hash^..hash\n> > Commit message in gitg 0.2.7 \n> \n> What is \"git patch\"? An alias for \"format-patch?\".\n\nYes, sorry.\nAnd this is the most amazing thing about this behavior... what's so\ndifferent between format-patch and log or show --pretty=%B. Shouldn't\nthese match 100%?\n\n> \n> > This is the script I tried to use to reproduce the problem:\n> > \n> > #!/bin/bash\n> > export LC_ALL=C\n> > input=$(cat)\n> > echo \"===========================\n> > $input\n> > ===========================\" >> /tmp/paging_bug.txt\n> > git commit-tree \"$@\" -m \"$input\" \n> \n> Can you be more specific about the input you're feeding to git and the\n> output you're seeing?\n> \n> For instance, if I do:\n> \n> git init\n> echo content >file\n> git add file\n> git commit -m \"$(perl -e 'print join(\" \", 1..100)')\"\n> \n> I get a commit message with one long unwrapped line, which I can view\n> via git-log, etc.\n\nThat's approximately what I did in my tests as well. And like you, when\nI do this in a fresh repository, it works like that..\n\n> Now if I try to run filter-branch on that:\n> \n> git filter-branch --commit-filter '\n> \tinput=$(cat)\n> \t{\n> \t\techo \"====================\"\n> \t\techo $input\n> \t\techo \"====================\"\n> \t} >>/tmp/paging_bug.txt\n> \tgit commit-tree \"$@\" -m \"$input\"\n> '\n> \n> then the commit remains unchanged, and paging_bug shows one long line.\n\nas well as filter-branch. That's what I meant when I wrote I cannot\nreproduce it with a new repository (to create a MWE). I wrote the first\nmail under the presumption that filter-branch is somehow involved but\napparently it is not the only git command and receives the mangled\ninput already as the commands stated in the last email show.\n\n> What am I missing?\n> \n> (I wondered at first if the extra \"cat\" and \"-m\" could be messing up\n> whitespace for you, but it should not, as the quoting around \"$input\"\n> should preserve things like newlines. And anyway, the bug in that case\n> would be the _opposite_; I'd expect it to stuff everything onto a single\n> line rather than breaking lines).\n\nThe commit messages I try to process are nothing special really... just\nvery long and not subject-like (because SVN and not giving too much\nthought to them sometimes). The only special thing I can think of is\nthat they have been processed by git-svn earlier.\n\n-- \nKind regards/Mit freundlichen Grüßen, Stefan Tauner\n"},{"id":"293310","messageId":"0M0gcI-1bFxtH2ZLy-00usFU@mail.gmx.com","threadId":"42588","inReplyTo":"20160801232429.mb43wxev5lyhxyiw@sigill.intra.peff.net","subject":"Re: Un-paged commit messages in git filter-branch's commit-filter?","fromName":"Stefan Tauner","fromEmail":"stefan.tauner@gmx.at","sentAt":"2016-08-06T09:40:27Z","receivedAt":"2016-08-06T23:18:22Z","isPatch":false,"sender":{"key":"stefan.tauner@gmx.at","avatar":null},"body":"On Mon, 1 Aug 2016 19:24:29 -0400\nJeff King wrote:\n\n> So could it be that your lines actually _are_ broken in the git objects,\n> but \"%s\" and other tools try to salvage them as a single subject?\n\nYES! :)\nThanks so much! I was apparently ignoring this trivial explanation\nbecause I was too much persuaded that the actual commits had missing\nline breaks and only something in git was adding them. But it is\nactually the other way around as you said: the few commands that print\nthe overlong lines are those that rely on %s. gitg for example parses\nthe output of the following:\ngit show --num-stat --pretty=\"format:%s%n%n%b%n\\x01\"\nI was not aware that %s lumps lines together if it does not find a\nproper subject on the first line.\n\nI can work with that now. Thanks again!\n-- \nKind regards/Mit freundlichen Grüßen, Stefan Tauner\n"}]}