{"thread":{"id":"54870","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","startedAt":"2020-12-22T22:41:26Z","lastAt":"2020-12-24T06:14:34Z","messageCount":6,"participants":["Matt Turner","Beat Bolli","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"412867","messageId":"CAEdQ38E9Fepp9gmidcf_HvFMacwPZBr0XgPT5HFs8bHw-SJDZQ@mail.gmail.com","threadId":"54870","inReplyTo":"20201221034452.307153-1-mattst88@gentoo.org","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Matt Turner","fromEmail":"mattst88@gentoo.org","sentAt":"2020-12-22T22:40:29Z","receivedAt":"2020-12-22T22:41:26Z","isPatch":true,"sender":{"key":"mattst88@gentoo.org","avatar":null},"body":"tl;dr:\n\nI want to handle conflicts automatically on lines like\n\n> KEYWORDS=\"~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86\"\n\nwhere conflicts frequently happen by adding/removing ~ before the\narchitecture names or adding/removing whole architectures. I don't\nknow if I should use a custom git merge driver or a custom git merge\nstrategy.\n\n\nSo the program in the patch below works, but it's not ideal, because\nit rejects any hunks that don't touch the KEYWORDS=... assignment.\n\nAs I understand it, a custom git merge driver is intended to be used\nto merge whole file formats, like JSON. As a result, you configure it\nvia gitattributes on a per-extension basis.\n\nI really just want to make the default recursive git merge handle\nKEYWORDS=... conflicts automatically, and I don't expect to be able to\nmake a git merge driver that can handle arbitrary conflicts in\n*.ebuild files. If the merge driver returns non-zero if it was unable\nto resolve the conflicts, but when it does so git evidently doesn't\nfallback and insert the typical <<< HEAD ... === ... >>> markers.\nMaybe I could make my merge driver insert those like git normally\ndoes? Seems like git's logic is probably a bit better about handling\nsome conflicts than my tool would be.\n\nSo... is a git merge strategy the thing I want? I don't know. There\ndoesn't seem to really be any documentation on writing git merge\nstrategies. I've only found [1] and [2].\n\nCc'ing git@vger.kernel.org, since I expect that's where the experts\nare. Hopefully they have suggestions.\n\n\n[1] https://stackoverflow.com/questions/23140240/git-how-do-i-add-a-custom-merge-strategy\n[2] https://stackoverflow.com/questions/54528824/any-documentation-for-writing-a-custom-git-merge-strategy\n\n\nOn Sun, Dec 20, 2020 at 10:44 PM Matt Turner wrote:\n>\n> Since the KEYWORDS=... assignment is a single line, git struggles to\n> handle conflicts. When rebasing a series of commits that modify the\n> KEYWORDS=... it's usually easier to throw them away and reapply on the\n> new tree than it is to manually handle conflicts during the rebase.\n>\n> git allows a 'merge driver' program to handle conflicts; this program\n> handles conflicts in the KEYWORDS=... assignment. E.g., given an ebuild\n> with these keywords:\n>\n> KEYWORDS=\"~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86\"\n>\n> One developer drops the ~alpha keyword and pushes to gentoo.git, and\n> another developer stabilizes hppa. Without this merge driver, git\n> requires the second developer to manually resolve the conflict. With\n> the custom merge driver, it automatically resolves the conflict.\n>\n> gentoo.git/.git/config:\n>\n> [core]\n> ...\n> attributesfile = ~/.gitattributes\n> [merge \"keywords\"]\n> name = KEYWORDS merge driver\n> driver = merge-driver-ekeyword %O %A %B\n>\n> ~/.gitattributes:\n>\n> *.ebuild merge=keywords\n>\n> Signed-off-by: Matt Turner \n> ---\n> One annoying wart in the program is due to the fact that ekeyword\n> won't work on any file not named *.ebuild. I make a symlink (and set up\n> an atexit handler to remove it) to work around this. I'm not sure we\n> could make ekeyword handle arbitrary filenames given its complex multi-\n> argument parameter support. git merge files are named .merge_file_XXXXX\n> according to git-unpack-file(1), so we could allow those. Thoughts?\n>\n> bin/merge-driver-ekeyword | 125 ++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 125 insertions(+)\n> create mode 100755 bin/merge-driver-ekeyword\n>\n> diff --git a/bin/merge-driver-ekeyword b/bin/merge-driver-ekeyword\n> new file mode 100755\n> index 0000000..6e645a9\n> --- /dev/null\n> +++ b/bin/merge-driver-ekeyword\n> @@ -0,0 +1,125 @@\n> +#!/usr/bin/python\n> +#\n> +# Copyright 2020 Gentoo Authors\n> +# Distributed under the terms of the GNU General Public License v2 or later\n> +\n> +\"\"\"\n> +Custom git merge driver for handling conflicts in KEYWORDS assignments\n> +\n> +See https://git-scm.com/docs/gitattributes#_defining_a_custom_merge_driver\n> +\"\"\"\n> +\n> +import atexit\n> +import difflib\n> +import os\n> +import shutil\n> +import sys\n> +\n> +from typing import List, Optional, Tuple\n> +\n> +from gentoolkit.ekeyword import ekeyword\n> +\n> +\n> +def keyword_array(keyword_line: str) -> List[str]:\n> + # Find indices of string inside the double-quotes\n> + i1: int = keyword_line.find('\"') + 1\n> + i2: int = keyword_line.rfind('\"')\n> +\n> + # Split into array of KEYWORDS\n> + return keyword_line[i1:i2].split(' ')\n> +\n> +\n> +def keyword_line_changes(old: str, new: str) -> List[Tuple[Optional[str],\n> + Optional[str]]]:\n> + a: List[str] = keyword_array(old)\n> + b: List[str] = keyword_array(new)\n> +\n> + s = difflib.SequenceMatcher(a=a, b=b)\n> +\n> + changes = []\n> + for tag, i1, i2, j1, j2 in s.opcodes():\n> + if tag == 'replace':\n> + changes.append((a[i1:i2], b[j1:j2]),)\n> + elif tag == 'delete':\n> + changes.append((a[i1:i2], None),)\n> + elif tag == 'insert':\n> + changes.append((None, b[j1:j2]),)\n> + else:\n> + assert tag == 'equal'\n> + return changes\n> +\n> +\n> +def keyword_changes(ebuild1: str, ebuild2: str) -> List[Tuple[Optional[str],\n> + Optional[str]]]:\n> + with open(ebuild1) as e1, open(ebuild2) as e2:\n> + lines1 = e1.readlines()\n> + lines2 = e2.readlines()\n> +\n> + diff = difflib.unified_diff(lines1, lines2, n=0)\n> + assert next(diff) == '--- \\n'\n> + assert next(diff) == '+++ \\n'\n> +\n> + hunk: int = 0\n> + old: str = ''\n> + new: str = ''\n> +\n> + for line in diff:\n> + if line.startswith('@@ '):\n> + if hunk > 0: break\n> + hunk += 1\n> + elif line.startswith('-'):\n> + if old or new: break\n> + old = line\n> + elif line.startswith('+'):\n> + if not old or new: break\n> + new = line\n> + else:\n> + if 'KEYWORDS=' in old and 'KEYWORDS=' in new:\n> + return keyword_line_changes(old, new)\n> + return None\n> +\n> +\n> +def apply_keyword_changes(ebuild: str,\n> + changes: List[Tuple[Optional[str],\n> + Optional[str]]]) -> int:\n> + # ekeyword will only modify files named *.ebuild, so make a symlink\n> + ebuild_symlink = ebuild + '.ebuild'\n> + os.symlink(ebuild, ebuild_symlink)\n> + atexit.register(lambda: os.remove(ebuild_symlink))\n> +\n> + for removals, additions in changes:\n> + args = []\n> + for rem in removals:\n> + # Drop leading '~' and '-' characters and prepend '^'\n> + i = 1 if rem[0] in ('~', '-') else 0\n> + args.append('^' + rem[i:])\n> + if additions:\n> + args.extend(additions)\n> + args.append(ebuild_symlink)\n> +\n> + result = ekeyword.main(args)\n> + if result != 0:\n> + return result\n> + return 0\n> +\n> +\n> +def main(argv):\n> + if len(argv) != 4:\n> + sys.exit(-1)\n> +\n> + O = argv[1] # %O - filename of original\n> + A = argv[2] # %A - filename of our current version\n> + B = argv[3] # %B - filename of the other branch's version\n> +\n> + # Get changes from %O to %B\n> + changes = keyword_changes(O, B)\n> + if not changes:\n> + sys.exit(-1)\n> +\n> + # Apply O -> B changes to A\n> + result: int = apply_keyword_changes(A, changes)\n> + sys.exit(result)\n> +\n> +\n> +if __name__ == \"__main__\":\n> + main(sys.argv)\n> --\n> 2.26.2\n>\n"},{"id":"412939","messageId":"b2adb41b-b2d2-fecf-3a93-7eb3f1f2247d@drbeat.li","threadId":"54870","inReplyTo":"CAEdQ38E9Fepp9gmidcf_HvFMacwPZBr0XgPT5HFs8bHw-SJDZQ@mail.gmail.com","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2020-12-23T18:11:38Z","receivedAt":"2020-12-23T18:19:34Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 22.12.20 23:40, Matt Turner wrote:\n> tl;dr:\n> \n> I want to handle conflicts automatically on lines like\n> \n>> KEYWORDS=\"~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86\"\n> \n> where conflicts frequently happen by adding/removing ~ before the\n> architecture names or adding/removing whole architectures. I don't\n> know if I should use a custom git merge driver or a custom git merge\n> strategy.\n\nYou can probably put each of the keywords on a separate line:\n\nKEYWORDS=\"\n~alpha\n~amd64\n~arm\n~arm64\n~hppa\n~ia64\n~mips\n~ppc\n~ppc64\n~riscv\n~s390\n~sparc~x86\n\"\n\nThe shell should handle both forms about the same.\n\n(I'm not a Gentoo user, just talking about my general shell experience)\n\nRegards\nBeat\n"},{"id":"412940","messageId":"xmqqsg7w8fwx.fsf@gitster.c.googlers.com","threadId":"54870","inReplyTo":"CAEdQ38E9Fepp9gmidcf_HvFMacwPZBr0XgPT5HFs8bHw-SJDZQ@mail.gmail.com","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-23T19:45:50Z","receivedAt":"2020-12-23T19:46:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Turner writes:\n\n> I want to handle conflicts automatically on lines like\n>\n>> KEYWORDS=\"~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86\"\n>\n> where conflicts frequently happen by adding/removing ~ before the\n> architecture names or adding/removing whole architectures. I don't\n> know if I should use a custom git merge driver or a custom git merge\n> strategy.\n\nA merge strategy is about how the changes at the tree level are\nhandled. A merge driver is given three blobs (original, your\nversion, and their version) and comes up with a merged blob.\n\nIn your case, you'd want a custom merge driver if you want to handle\nword changes on a single line, because the default text merge driver\nis pretty much line oriented.\n\n"},{"id":"412941","messageId":"xmqqo8ik8fvf.fsf@gitster.c.googlers.com","threadId":"54870","inReplyTo":"b2adb41b-b2d2-fecf-3a93-7eb3f1f2247d@drbeat.li","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-23T19:46:44Z","receivedAt":"2020-12-23T19:47:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Beat Bolli writes:\n\n> You can probably put each of the keywords on a separate line:\n>\n> KEYWORDS=\"\n> ~alpha\n> ~amd64\n> ~arm\n> ~arm64\n> ~hppa\n> ~ia64\n> ~mips\n> ~ppc\n> ~ppc64\n> ~riscv\n> ~s390\n> ~sparc~x86\n> \"\n>\n> The shell should handle both forms about the same.\n\nI agree that it is a more practical approach than writing an one-off\nmerge driver.\n"},{"id":"412982","messageId":"CAEdQ38En=HrG9A0STgaxFAwqxBjwNGQoV=U26Pia3x=gd-Attg@mail.gmail.com","threadId":"54870","inReplyTo":"xmqqsg7w8fwx.fsf@gitster.c.googlers.com","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Matt Turner","fromEmail":"mattst88@gentoo.org","sentAt":"2020-12-24T04:47:22Z","receivedAt":"2020-12-24T04:49:15Z","isPatch":true,"sender":{"key":"mattst88@gentoo.org","avatar":null},"body":"On Wed, Dec 23, 2020 at 2:46 PM Junio C Hamano wrote:\n>\n> Matt Turner writes:\n>\n> > I want to handle conflicts automatically on lines like\n> >\n> >> KEYWORDS=\"~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ~ppc ~ppc64 ~riscv ~s390 ~sparc ~x86\"\n> >\n> > where conflicts frequently happen by adding/removing ~ before the\n> > architecture names or adding/removing whole architectures. I don't\n> > know if I should use a custom git merge driver or a custom git merge\n> > strategy.\n>\n> A merge strategy is about how the changes at the tree level are\n> handled. A merge driver is given three blobs (original, your\n> version, and their version) and comes up with a merged blob.\n>\n> In your case, you'd want a custom merge driver if you want to handle\n> word changes on a single line, because the default text merge driver\n> is pretty much line oriented.\n\nThanks, that makes sense. The merge driver I've written seems to work\ngreat for handling the KEYWORDS=... line.\n\nIf users could more simply opt into using it (e.g., on the command\nline rather than enabling it via ~/.gitattributes) I think it would be\nfine to use. Better yet, is there a way git can be configured to\nfallback to another merge driver if the first returns a non-zero\nstatus due to unresolved conflicts? For example, if there are changes\nto other lines, how can I fall back to another merge driver?\n\nThank you for your advice!\n"},{"id":"412984","messageId":"xmqqmty34tph.fsf@gitster.c.googlers.com","threadId":"54870","inReplyTo":"CAEdQ38En=HrG9A0STgaxFAwqxBjwNGQoV=U26Pia3x=gd-Attg@mail.gmail.com","subject":"Re: [RFC PATCH gentoolkit] bin: Add merge-driver-ekeyword","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-24T06:13:46Z","receivedAt":"2020-12-24T06:14:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Matt Turner writes:\n\n> ... is there a way git can be configured to\n> fallback to another merge driver if the first returns a non-zero\n> status due to unresolved conflicts? For example, if there are changes\n> to other lines, how can I fall back to another merge driver?\n\nThere is no \"fallback\", but a merge driver should be able to first\nrun another merge driver (e.g. \"git merge-file\" or the \"merge\"\ncommand from the RCS suite of programs would be line-oriented 3-way\ndrivers suitable for text files) and then fix up the leftover bits.\n\nIf your users don't want to contaminate the .gitattributes file that\nis recorded in-tree, they can also use .git/info/attributes to locally\nconfigure Git to use such a driver.\n\n"}]}