{"thread":{"id":"59802","subject":"Automatically re-running commands during an interactive rebase or post commit","startedAt":"2023-05-29T13:38:55Z","lastAt":"2023-06-08T05:11:43Z","messageCount":13,"participants":["Paul Jolly","rsbecker@nexbridge.com","Phillip Wood","Son Luong Ngoc","Oswald Buddenhagen","Elijah Newren"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"477779","messageId":"CACoUkn7TmZ=trtDKcQm0SG5qCqK=-+YxrDV-7xYnLH_XK7K7og@mail.gmail.com","threadId":"59802","inReplyTo":null,"subject":"Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-05-29T13:38:40Z","receivedAt":"2023-05-29T13:38:55Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"Hi all,\n\nI would appreciate some advice on the best way to solve the following problem.\n\nAs part of my project, I have a code generation script that sha256\nhashes a number of files to another file. This produces a\ndeterministic \"has this part of the project changed\" indicator via the\ncode generated file's content, that I then use in various cache\ninvalidation steps.\n\nThis means, however, that I need to re-run that code generation script\nas part of each commit in order to ensure that the code generated hash\nfile is current (I have a step in CI that detects if it is not, which\nre-runs the code generation script to then see if the commit is\n\"clean\").\n\nAs part of my development setup I do a lot of interactive rebasing to\nedit earlier commits in a branch (these \"stacks\" of changes are\nreviewed via Gerrit, which understands a relation chain of changes).\nVia this workflow, I often do a git rebase and edit an earlier commit\nin such a way that I need to re-run the code generation script.\n\nThe challenge is that any commit in such a \"stack\" of commits might\nneed me to re-run the code generation script. But I clearly don't want\nto do this manually!\n\nWhat I'm looking for is a way to automatically re-run this code\ngeneration script when I commit changes, or perform a rebase-edit step\netc.\n\nI've tried to experiment with how I might do this using git commit\nhooks. But so far, my git foo is failing me. It mainly fails because\nwhen doing an edit of an earlier commit via an interactive rebase,\nlater changes might well conflict (in the generated file) with the\nresults of the code generator having been re-run on the edited commit.\nAt this point, my git rebase --continue stops until I have fixed the\nconflict. But in almost all situations, the conflict comes in the\ngenerated hash file. Which I fix by simply re-running the code\ngeneration script (I could optionally fix it by doing a git checkout\n--theirs, and then re-running the code generation script).\n\nThis all feels tantalisingly close to being a perfect workflow! But I\ncan't quite figure out how to make the git hooks \"work\" in such a way\nthat doesn't require any intervention from me (except in those\nsituations where there is a conflict during the rebase that is _not_\nin the code generated file and so does require my intervention).\n\nThe code generation step is incredibly fast if there is nothing to do,\nand is quite fast even when there is something to do (in any case it\ncan't avoid doing this work).\n\nPlease can someone help nudge me in the right direction?\n\nMany thanks,\n\n\nPaul\n"},{"id":"477780","messageId":"002901d99233$3d0c8b80$b725a280$@nexbridge.com","threadId":"59802","inReplyTo":"CACoUkn7TmZ=trtDKcQm0SG5qCqK=-+YxrDV-7xYnLH_XK7K7og@mail.gmail.com","subject":"RE: Automatically re-running commands during an interactive rebase or post commit","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-05-29T13:41:08Z","receivedAt":"2023-05-29T13:41:19Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"\n\n>-----Original Message-----\nOn Monday, May 29, 2023 9:39 AM, Paul Jolly wrote:\n>I would appreciate some advice on the best way to solve the following problem.\n>\n>As part of my project, I have a code generation script that sha256 hashes a number of\n>files to another file. This produces a deterministic \"has this part of the project\n>changed\" indicator via the code generated file's content, that I then use in various\n>cache invalidation steps.\n>\n>This means, however, that I need to re-run that code generation script as part of each\n>commit in order to ensure that the code generated hash file is current (I have a step\n>in CI that detects if it is not, which re-runs the code generation script to then see if\n>the commit is \"clean\").\n>\n>As part of my development setup I do a lot of interactive rebasing to edit earlier\n>commits in a branch (these \"stacks\" of changes are reviewed via Gerrit, which\n>understands a relation chain of changes).\n>Via this workflow, I often do a git rebase and edit an earlier commit in such a way\n>that I need to re-run the code generation script.\n>\n>The challenge is that any commit in such a \"stack\" of commits might need me to re-\n>run the code generation script. But I clearly don't want to do this manually!\n>\n>What I'm looking for is a way to automatically re-run this code generation script\n>when I commit changes, or perform a rebase-edit step etc.\n>\n>I've tried to experiment with how I might do this using git commit hooks. But so far,\n>my git foo is failing me. It mainly fails because when doing an edit of an earlier\n>commit via an interactive rebase, later changes might well conflict (in the generated\n>file) with the results of the code generator having been re-run on the edited commit.\n>At this point, my git rebase --continue stops until I have fixed the conflict. But in\n>almost all situations, the conflict comes in the generated hash file. Which I fix by\n>simply re-running the code generation script (I could optionally fix it by doing a git\n>checkout --theirs, and then re-running the code generation script).\n>\n>This all feels tantalisingly close to being a perfect workflow! But I can't quite figure\n>out how to make the git hooks \"work\" in such a way that doesn't require any\n>intervention from me (except in those situations where there is a conflict during the\n>rebase that is _not_ in the code generated file and so does require my intervention).\n>\n>The code generation step is incredibly fast if there is nothing to do, and is quite fast\n>even when there is something to do (in any case it can't avoid doing this work).\n>\n>Please can someone help nudge me in the right direction?\n\nI wonder whether setting up a clean/smudge filter might help. You might want to look into a clean filter that runs your code generator.\n\n--Randall\n\n"},{"id":"477781","messageId":"CACoUkn6wzoHph+BU3eYWqmf4ZWGji08Qhhjn+z4zrWJp1csPbQ@mail.gmail.com","threadId":"59802","inReplyTo":"002901d99233$3d0c8b80$b725a280$@nexbridge.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-05-29T14:01:50Z","receivedAt":"2023-05-29T14:02:40Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"> >Please can someone help nudge me in the right direction?\n>\n> I wonder whether setting up a clean/smudge filter might help. You might want to look into a clean filter that runs your code generator.\n\nThanks. I'm not clear that this will help, because the main problem is\nthat the rebase stops when trying to apply a later commit. In this\ncase, the previous commit will be \"clean\", but the commit the rebase\nis trying to apply will conflict and the rebase process will stop\n\nReading elsewhere, I'm now looking into whether wrapping the default\nmergetool will help provide the \"hooks\" I need.\n"},{"id":"477782","messageId":"CACoUkn6F_=JR5SYBWL2sVDxDVAZbxcV5Ruif6wLba-_q_QNiZg@mail.gmail.com","threadId":"59802","inReplyTo":"CACoUkn7TmZ=trtDKcQm0SG5qCqK=-+YxrDV-7xYnLH_XK7K7og@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-05-29T16:02:18Z","receivedAt":"2023-05-29T16:02:35Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"> I would appreciate some advice on the best way to solve the following problem.\n\nReplying to self with some updates based on further investigations.\n\nThe lack of pre-commit like hook during a rebase is documented in a\nfew places (https://adamj.eu/tech/2022/11/07/pre-commit-run-hooks-rebase/\nto list one). The -x flag provides a means by which I can\nretrospectively amend the \"previous\" commit if that is not clean. Not\nideal, but a solution of sorts.\n\nHowever, the main problem appears to be the lack of \"hook\" to do\nsomething when git rebase detects conflicts.\n\nIdeally, I would like to have a hook be called when git rebase detects\na conflict, a hook in which I could attempt to \"recover\" the\nsituation. If that hook failed to fully recover the situation, git\nrebase would fail as it currently does. Otherwise, it would be able to\ncontinue as normal.\n\nMy use of git is pretty basic, so part of me feels like I can't be the\nfirst person to run into this problem!\n\nAs ever, any pointers are greatly appreciated.\n"},{"id":"477783","messageId":"198eea19-fc40-de0c-8854-3cdc94925f32@gmail.com","threadId":"59802","inReplyTo":"CACoUkn6F_=JR5SYBWL2sVDxDVAZbxcV5Ruif6wLba-_q_QNiZg@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-05-29T17:53:25Z","receivedAt":"2023-05-29T17:53:32Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Paul\n\nOn 29/05/2023 17:02, Paul Jolly wrote:\n>> I would appreciate some advice on the best way to solve the following problem.\n> \n> Replying to self with some updates based on further investigations.\n> \n> The lack of pre-commit like hook during a rebase is documented in a\n> few places (https://adamj.eu/tech/2022/11/07/pre-commit-run-hooks-rebase/\n> to list one). The -x flag provides a means by which I can\n> retrospectively amend the \"previous\" commit if that is not clean. Not\n> ideal, but a solution of sorts.\n> \n> However, the main problem appears to be the lack of \"hook\" to do\n> something when git rebase detects conflicts.\n\nI think you could get around this by using a script that wraps \"git \nrebase\". Something like\n\nwhile true\ndo\n     if test -d \"$(git rev-parse --git-path rebase-merge)\"\n     then\n\tgit rebase --continue\n     else\n\tgit rebase \"$@\"\n     fi\n     if test $? -eq 0\n     then\n\texit 0\n     fi\n     # Handle conflicts here\n\n     if automatic-conflict-resolution-failed\n     then\n         exit 1\n     fi\ndone\n\n\nYou can get a list of the files that have conflicts with\n\n\tgit diff-index --name-only --diff-filter=U HEAD --\n\nor by parsing the output of\n\n\tgit status --porcelain\n\nI did wonder if you could use a custom merge driver (see man \ngitattributes) for the file with the checksums but that would be run \nbefore all the other files have been merged. Another alternative might \nbe to use a custom merge strategy that runs \"git merge-ort\" and then \ncleans up the conflicts in the checksum file. You can specify a custom \nstrategy by naming your script \"git-merge-foo\" and passing \"-s foo\" to \ngit rebase.\n\nBest Wishes\n\nPhillip\n\n> Ideally, I would like to have a hook be called when git rebase detects\n> a conflict, a hook in which I could attempt to \"recover\" the\n> situation. If that hook failed to fully recover the situation, git\n> rebase would fail as it currently does. Otherwise, it would be able to\n> continue as normal.\n> \n> My use of git is pretty basic, so part of me feels like I can't be the\n> first person to run into this problem!\n> \n> As ever, any pointers are greatly appreciated.\n\n"},{"id":"477785","messageId":"CACoUkn4TOGi7xojSatEowF0CcFJHJof+F5XR-QtPTL5iznZDKw@mail.gmail.com","threadId":"59802","inReplyTo":"198eea19-fc40-de0c-8854-3cdc94925f32@gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-05-29T19:08:07Z","receivedAt":"2023-05-29T19:08:23Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"Hi Phillip,\n\nMany thanks for your response.\n\n> I think you could get around this by using a script that wraps \"git\n> rebase\". Something like\n\nThat's an interesting thought. I will give this some thought, thank you.\n\n> I did wonder if you could use a custom merge driver (see man\n> gitattributes) for the file with the checksums but that would be run\n> before all the other files have been merged. Another alternative might\n> be to use a custom merge strategy that runs \"git merge-ort\" and then\n> cleans up the conflicts in the checksum file. You can specify a custom\n> strategy by naming your script \"git-merge-foo\" and passing \"-s foo\" to\n> git rebase.\n\nThis approach with a custom merge strategy is very much closer to what\nfeels like it will work best.\n\nI just gave this a try, and I seem to be falling at somewhat the first hurdle.\n\nI have a custom merge strategy working, insofar as I've written a\nscript that is being called. The script simply wraps \"git merge-ort\"\nas you suggest.\n\nBut I can't seem to work out how to actually run 'git merge-ort'! I've\ntried 'git merge -s \"$@\"' in my wrapper script, but doing so my custom\nstrategy behaves differently to if I use no strategy, which seems to\nsuggest I'm doing something wrong.\n\nDo you have any suggestions/pointers?\n\nBest,\n\n\nPaul\n"},{"id":"477789","messageId":"cc93b0ed-1167-a017-2138-181a490d26e2@gmail.com","threadId":"59802","inReplyTo":"CACoUkn4TOGi7xojSatEowF0CcFJHJof+F5XR-QtPTL5iznZDKw@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-05-29T19:48:53Z","receivedAt":"2023-05-29T19:49:03Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Paul\n\nOn 29/05/2023 20:08, Paul Jolly wrote:\n> Hi Phillip,\n> \n> Many thanks for your response.\n> \n>> I think you could get around this by using a script that wraps \"git\n>> rebase\". Something like\n> \n> That's an interesting thought. I will give this some thought, thank you.\n> \n>> I did wonder if you could use a custom merge driver (see man\n>> gitattributes) for the file with the checksums but that would be run\n>> before all the other files have been merged. Another alternative might\n>> be to use a custom merge strategy that runs \"git merge-ort\" and then\n>> cleans up the conflicts in the checksum file. You can specify a custom\n>> strategy by naming your script \"git-merge-foo\" and passing \"-s foo\" to\n>> git rebase.\n> \n> This approach with a custom merge strategy is very much closer to what\n> feels like it will work best.\n> \n> I just gave this a try, and I seem to be falling at somewhat the first hurdle.\n> \n> I have a custom merge strategy working, insofar as I've written a\n> script that is being called. The script simply wraps \"git merge-ort\"\n> as you suggest.\n> \n> But I can't seem to work out how to actually run 'git merge-ort'! I've\n> tried 'git merge -s \"$@\"' in my wrapper script, but doing so my custom\n> strategy behaves differently to if I use no strategy, which seems to\n> suggest I'm doing something wrong.\n> \n> Do you have any suggestions/pointers?\n\nSorry, I'd assumed there was a merge-ort command in the same way as \nthere is a merge-recursive command but it does not exist. I think the \nbest you can do is run \"git merge-recursive\" rather than \"git merge-ort\" \nin your script. I've cc'd Elijah in case he has a better suggestion.\n\nBest Wishes\n\nPhillip\n\n> Best,\n> \n> \n> Paul\n"},{"id":"477792","messageId":"CAL3xRKfj6R9BfnKEcEqcD5Z+y_A1uocSgEsPyoaV95VXenjHbQ@mail.gmail.com","threadId":"59802","inReplyTo":"CACoUkn7TmZ=trtDKcQm0SG5qCqK=-+YxrDV-7xYnLH_XK7K7og@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Son Luong Ngoc","fromEmail":"sluongng@gmail.com","sentAt":"2023-05-30T07:22:41Z","receivedAt":"2023-05-30T07:23:51Z","isPatch":false,"sender":{"key":"sluongng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/26684313?v=4"},"body":"Hey Paul,\n\nOn Mon, May 29, 2023 at 3:44 PM Paul Jolly <paul@myitcv.io> wrote:\n>\n> Hi all,\n>\n> I would appreciate some advice on the best way to solve the following problem.\n>\n...\n>\n> I've tried to experiment with how I might do this using git commit\n> hooks. But so far, my git foo is failing me. It mainly fails because\n> when doing an edit of an earlier commit via an interactive rebase,\n> later changes might well conflict (in the generated file) with the\n> results of the code generator having been re-run on the edited commit.\n> At this point, my git rebase --continue stops until I have fixed the\n> conflict. But in almost all situations, the conflict comes in the\n> generated hash file. Which I fix by simply re-running the code\n> generation script (I could optionally fix it by doing a git checkout\n> --theirs, and then re-running the code generation script).\n>\n> This all feels tantalisingly close to being a perfect workflow! But I\n> can't quite figure out how to make the git hooks \"work\" in such a way\n> that doesn't require any intervention from me (except in those\n> situations where there is a conflict during the rebase that is _not_\n> in the code generated file and so does require my intervention).\n>\n> The code generation step is incredibly fast if there is nothing to do,\n> and is quite fast even when there is something to do (in any case it\n> can't avoid doing this work).\n>\n> Please can someone help nudge me in the right direction?\n\nIn general, there are 2 cases that you would want to handle:\n1. Inserting format directive in between commit rebase that DOES NOT\n    come with merge conflicts\n2. Same but DOES come with merge conflicts.\n\nFor (1), you might be interested in tools such as\n- Git Absorb(a) that automatically fixup your stack of commits with your\n  current dirty changes.\n- Git Branchless(b) \"git test\" feature\n\nBoth of these tools are heavily influenced by Meta's internal Phabricator\nmercurial workflow. Since the release of these tools, Meta has also\nopen-sourced their internal tool at Sapling SCM(c) which they touted\nto be git-compatible.\n\nFor (2), and if none of the tools above solve your problem,\nthen I recommend using git-rebase interactive with a vim macro to\ngenerate the needed rebase todo. You can find my comment in (d)\nto see what such a rebase todo list would look like.\n\nTools such as Restack (e) take it a step further by providing a custom Git\n`sequence.editor` to programmatically generate the rebase todo for you.\nThis could be a bash script, or a perl script... or a custom Go binary of\nyour choosing. You might want to go down this route if a vim macro is\nnot sufficient and you require some custom logic.\n\nFinally, I would recommend turning on rerere.enabled (f) config to store\nthe conflict resolution for subsequent rebase attempts. This way, you would\nonly need to resolve each rebase conflict once.\n\n(a): https://github.com/tummychow/git-absorb\n(b): https://github.com/arxanas/git-branchless/wiki/Command:-git-test#fixing-formatting-and-linting-issues\n(c): https://sapling-scm.com/docs/commands/absorb\n(d): https://github.com/arxanas/git-branchless/discussions/45#discussioncomment-3364792\n(e): https://github.com/abhinav/restack\n(f): https://git-scm.com/docs/git-config#Documentation/git-config.txt-rerereenabled\n\n>\n> Many thanks,\n>\n>\n> Paul\n\nCheers,\nSon Luong.\n"},{"id":"477794","messageId":"ZHXFdRnrwzNCA227@ugly","threadId":"59802","inReplyTo":"CACoUkn7TmZ=trtDKcQm0SG5qCqK=-+YxrDV-7xYnLH_XK7K7og@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Oswald Buddenhagen","fromEmail":"oswald.buddenhagen@gmx.de","sentAt":"2023-05-30T09:44:21Z","receivedAt":"2023-05-30T09:44:27Z","isPatch":false,"sender":{"key":"oswald.buddenhagen@gmx.de","avatar":"https://avatars.githubusercontent.com/u/812380?v=4"},"body":"On Mon, May 29, 2023 at 02:38:40PM +0100, Paul Jolly wrote:\n>As part of my project, I have a code generation script that sha256\n>hashes a number of files to another file. This produces a\n>deterministic \"has this part of the project changed\" indicator via the\n>code generated file's content, that I then use in various cache\n>invalidation steps.\n>\n>This means, however, that I need to re-run that code generation script\n>as part of each commit in order to ensure that the code generated hash\n>file is current (I have a step in CI that detects if it is not, which\n>re-runs the code generation script to then see if the commit is\n>\"clean\").\n>\ni would recommend taking a step back and considering whether you're \nactually trying to fix the right problem.\n\nwhy are you checking in an auto-generated file, esp. given that it can \nbe generated very quickly as you report?\n\nusually, this should be done by the build system.\n\nif the used build tool really is too dumb to integrate it into the build \nsystem, you might have luck with a post-checkout hook.\n\nyou can also obtain existing hashes directly from git, with ls-tree, \nthough this would again require some kind of integration with the build \nor checkout process.\n\nif you can't get around checking in the hash, i can think of hacking it \nusing rebase --exec. basically, before each pick you'd create a commit \nthat reverts the hash change (by checking out that path from the parent \nof the last commit that touched it, found automatically with git log), \nand after the pick you'd squash away the revert (using `reset HEAD~2 && \ncommit -C @{1}` or something to that effect). very ugly, very fragile.\n\nregards,\nossi\n"},{"id":"477795","messageId":"CACoUkn5i_nR5Tpd_ZBBQyP4e0ObYKVXXa9d+umkdpF5PPXa-pg@mail.gmail.com","threadId":"59802","inReplyTo":"ZHXFdRnrwzNCA227@ugly","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-05-30T10:27:15Z","receivedAt":"2023-05-30T10:29:03Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"Hi ossi,\n\nThanks for the response.\n\n> i would recommend taking a step back and considering whether you're\n> actually trying to fix the right problem.\n>\n> why are you checking in an auto-generated file, esp. given that it can\n> be generated very quickly as you report?\n>\n> usually, this should be done by the build system.\n\nThanks for asking about this and forcing me to think more on the point.\n\nTo add a bit more context, and explanation behind the current design decisions.\n\nThe system I have described is a pipeline that supports a\ndocumentation site for https://cuelang.org/. The architecture I have\ndescribed is that of the preprocessor, a tool which helps to automate\nthe testing of examples in documentation. Content authors write\ndocumentation in format X, the preprocessor validates (and runs, as\nrequired) that content, and produces format Y. Format Y is the input\nto a Hugo (https://gohugo.io/) static site; hugo processes format Y to\nproduce format Z, the HTML that then renders the site.\n\nThe generation of hashes that I referred to before relates to the\ncontents of format X. If when the preprocessor runs it detects\n(according to the cache files commited with the content) a cache hit,\nthen there is no need to re-run an example in some documentation. We\ncommit those hashes for now to sidestep needing to create and maintain\na shared preprocessor cache (a cache that is shared between CI systems\nand users). We might move to a system like that in the future; for now\nthis feels like a sufficient setup.\n\nThe cache right now is very dumb; well-known files are updated with\nhash values. And this is what creates the git conflicts. One thing we\ncould do to eliminate the conflicts altogether is commit a\ncontent-addressed cache. This would have the problem of growing over\ntime... but I think we could solve that problem a different way.\n\n> if the used build tool really is too dumb to integrate it into the build\n> system, you might have luck with a post-checkout hook.\n>\n> you can also obtain existing hashes directly from git, with ls-tree,\n> though this would again require some kind of integration with the build\n> or checkout process.\n>\n> if you can't get around checking in the hash, i can think of hacking it\n> using rebase --exec. basically, before each pick you'd create a commit\n> that reverts the hash change (by checking out that path from the parent\n> of the last commit that touched it, found automatically with git log),\n> and after the pick you'd squash away the revert (using `reset HEAD~2 &&\n> commit -C @{1}` or something to that effect). very ugly, very fragile.\n\nThanks. I have a working setup now using a combination of git rebase\n-x and a script that I run whenever git rebase fails because of a\nconflict. This works but is not ideal for a couple of reasons:\n\n1. Each interactive rebase is \"littered\" with exec lines which should\nbe a detail\n2. I need to re-run the script manually when conflicts are detected\n\nPoint 1 would be nicely addressed by a git hook that fires \"pre\ncommit\" during a rebase.\n\nPoint 2 could be solved by a custom merge driver, but that's seemingly\nnot possible right now:\nhttps://lore.kernel.org/git/ZHXFdRnrwzNCA227@ugly/T/#m14b204843fea1fe9ff1c7500244049a43ed610eb.\nAlternatively it could be solved by another hook that fires when\nrebase detects a conflict, a hook that attempts to \"recover\" the\nsituation before rebase actually fails.\n\nThanks again for asking about whether we are solving the right problem\nhere. Writing my response above prompted me to think again about\ndifferent solutions.\n\nBest,\n\n\nPaul\n"},{"id":"477824","messageId":"CABPp-BEAnB-6ogcohMWMmktr9SrwMtM0-dw1pypcJedcMMcsCg@mail.gmail.com","threadId":"59802","inReplyTo":"cc93b0ed-1167-a017-2138-181a490d26e2@gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-05-31T04:14:09Z","receivedAt":"2023-05-31T04:14:29Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 29, 2023 at 12:48 PM Phillip Wood <phillip.wood123@gmail.com> wrote:\n> On 29/05/2023 20:08, Paul Jolly wrote:\n[...]\n> > I have a custom merge strategy working, insofar as I've written a\n> > script that is being called. The script simply wraps \"git merge-ort\"\n> > as you suggest.\n> >\n> > But I can't seem to work out how to actually run 'git merge-ort'! I've\n> > tried 'git merge -s \"$@\"' in my wrapper script, but doing so my custom\n> > strategy behaves differently to if I use no strategy, which seems to\n> > suggest I'm doing something wrong.\n\nDon't call `git merge` from your custom merge strategy.  That's just\nasking for trouble; we should probably modify the code to throw huge\nerrors if people attempt that.\n\nUse lower-level primitives, such as `git merge-tree`, `git\nmerge-file`, `git read-tree`, `git update-index`, `git merge-index`,\n`git merge-one-file`, etc.  Bias towards the first two of those.\n\n> > Do you have any suggestions/pointers?\n>\n> Sorry, I'd assumed there was a merge-ort command in the same way as\n> there is a merge-recursive command but it does not exist. I think the\n> best you can do is run \"git merge-recursive\" rather than \"git merge-ort\"\n> in your script. I've cc'd Elijah in case he has a better suggestion.\n\nIn short, I'd say use `git merge-tree` instead of `git\nmerge-recursive`, but the long story about the latter command is:\n\nFirst of all, we _currently_ have a bunch of executables & scripts,\none per merge strategy:\n  * git-merge-octopus\n  * git-merge-resolve\n  * git-merge-recursive\n  * git-merge-ours\nBut, in this set, git-merge-recursive is builtin and isn't called as\nan external program, so that's not a reason I think it should exist,\nand we certainly shouldn't add another for `ort` based on this\nrationale.  (The others may also be replaced with builtins; there were\neven patches in that direction, though they weren't followed up on.)\n\nSecond of all, we've had multiple ways to expose users to \"doing a merge\":\n  * git-merge\n  * git-merge-recursive\n  * git-merge-tree\nThe first is the one most users are familiar with, `git merge`.  So,\nabout the others...  Traditionally, git-merge-tree was a virtually\nuseless implementation that fell way short of the good idea to 'make a\nmerge without touching the worktree or index'.  In contrast,\ngit-merge-recursive was in between git-merge-tree and git-merge in\nterms of how low level it was.  It still required updating the working\ntree and index, which limited what could be built on top of it, but\nunlike git-merge-tree it at least provided useful functionality in\nthat it could do-all-parts-of-the-merge-except-commit-and-update-branch.\nWhile not optimal, that was enough that git-merge-recursive got used\nin some places (such as one of the rebase backends once upon a time).\nNowadays, though, git-merge-tree is the correct low-level primitive --\nit can do a merge without updating branches, creating a commit, or\nupdating the worktree or index.  That means that git-merge-recursive\nisn't really useful from this angle either; everything can be built on\ntop of something a little lower level.\n\nThe two angles above were the reasons that git-merge-recursive\nexisted, but neither is useful nowadays, so git-merge-recursive is\nbasically a historical relic (which we are keeping for backward\ncompatibility purposes, not for any intrinsic value I can see).  I\ndon't want to add another useless historical relic, in the form of\n`git-merge-ort`.\n\nBut there's a third reason to not provide `git-merge-ort` as well: The\nplan has always been to use `ort` as a drop-in replacement for\n`recursive` and make _all_ calls within the code unconditionally\ntranslate requests for `recursive` to call the `ort` code instead.  I\njust haven't gotten around to finishing those plans yet.  (In fact,\nthere are only two places in the code that still call the recursive\nbackend by default -- git-merge-recursive and git-am, both of which\nI've been meaning to fix up).  So, making a `git-merge-ort` would make\nit really weird in the future in that we'd have both\n`git-merge-recursive` and `git-merge-ort` calling the same thing.\n\nAnyway, if you want a low-level do-a-merge thing, use `git merge-tree`.\n"},{"id":"477926","messageId":"6716d478-08a5-b9ef-f1a3-792f591506da@gmail.com","threadId":"59802","inReplyTo":"CABPp-BEAnB-6ogcohMWMmktr9SrwMtM0-dw1pypcJedcMMcsCg@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-06-02T09:57:42Z","receivedAt":"2023-06-02T09:57:48Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Elijah\n\nOn 31/05/2023 05:14, Elijah Newren wrote:\n >\n> ... lots of helpful comments ...\n> \n> Anyway, if you want a low-level do-a-merge thing, use `git merge-tree`.\n\nThanks for taking the time to explain that, I hadn't really paid \nattention to the changes to \"git merge-tree\" it looks nice - I should \nswitch a couple of scripts I've got from using \"git merge-recursive\" \nover to that.\n\nBest Wishes\n\nPhillip\n"},{"id":"478163","messageId":"CACoUkn5OsWNMzUe=O_ObjJoDk=ULnRij90idYutgn42Z3-cWgw@mail.gmail.com","threadId":"59802","inReplyTo":"CABPp-BEAnB-6ogcohMWMmktr9SrwMtM0-dw1pypcJedcMMcsCg@mail.gmail.com","subject":"Re: Automatically re-running commands during an interactive rebase or post commit","fromName":"Paul Jolly","fromEmail":"paul@myitcv.io","sentAt":"2023-06-08T05:11:25Z","receivedAt":"2023-06-08T05:11:43Z","isPatch":false,"sender":{"key":"paul@myitcv.io","avatar":"https://gravatar.com/avatar/6a83179b2ef0386af693cc03a9c50ea897f5246c5dcc0fa5d126171ba17c3a63?d=mp&s=160"},"body":"> Anyway, if you want a low-level do-a-merge thing, use `git merge-tree`.\n\nElijah - please accept my apologies for my belated reply.\n\nThank you very much for this detail. I will give the custom merge\nstrategy approach another whirl!\n\nBest,\n\n\nPaul\n"}]}