{"thread":{"id":"45002","subject":"git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","startedAt":"2017-01-29T21:37:11Z","lastAt":"2017-02-04T18:44:03Z","messageCount":6,"participants":["Carlo Wood","Junio C Hamano","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"310517","messageId":"20170129203348.1a8c0722@hikaru","threadId":"45002","inReplyTo":null,"subject":"git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Carlo Wood","fromEmail":"carlo@alinoe.com","sentAt":"2017-01-29T19:33:48Z","receivedAt":"2017-01-29T21:37:11Z","isPatch":false,"sender":{"key":"carlo@alinoe.com","avatar":"https://gravatar.com/avatar/f9ef6306f40295ad6047a49139a3e240d5c297092eafab3da2e34416ba1193d9?d=mp&s=160"},"body":"Hi,\n\nthere seems to be a problem with using 'git commit --amend' in\ngit submodules when using 'git push --recurse-submodules=on-demand'\nin the parent.\n\nThe latter fails, saying \"The following submodule paths contain changes\nthat can not be found on any remote:\" for such submodule, even though\nthe submodule is clean, pushed and reports 'Everything up-to-date'\nwhen trying to push it.\n\nI believe that the reason has to be that the parent repository thinks\nthat the comment that was amended, but not pushed, must be on the remote\ntoo, while the whole point of amend is that this commit is not pushed.\n\nI wrote a little script that demonstrates the problem.\nPlease run in an empty directory.\n\nSTART-OF-SCRIPT\n\n#! /bin/bash\n\n# This script demonstrates a bug in git where it reports\n#\n# The following submodule paths contain changes that can\n# not be found on any remote:\n#\n# for a submodule that is clean and pushed.\n#\n# Create an empty directory, put this script in it\n# and run the script.\n#\n# Carlo Wood, 2017/01/29\n\n# Clean a possible previous run:\nrm -rf parent remote.parent remote.subm\n\nREMOTE_BASE=\"$(pwd)\"\n\n# Create a 'remote' for the submodule 'subm'.\nmkdir remote.subm\npushd remote.subm\ngit init --bare\npopd\n\n# Create a 'remote' for the 'parent' repository.\nmkdir remote.parent\npushd remote.parent\ngit init --bare\npopd\n\n# Create initial parent/subm directory structore.\nmkdir -p parent/subm\n\n# Create an initial subm git repository.\npushd parent/subm\ngit init\ngit remote add local \"$REMOTE_BASE/remote.subm\"\ntouch s ; git add s\ngit commit -m 'Initial commit.'\ngit push --set-upstream local master\npopd\n\n# Create an initial parent git repository with subm as submodule\n# and push.recurseSubmodules = on-demand.\npushd parent\ngit init\ngit config push.recurseSubmodules on-demand\ngit remote add local \"$REMOTE_BASE/remote.parent\"\ntouch p ; git add p\ngit submodule add \"$REMOTE_BASE/remote.subm\" subm\ngit add .gitmodules subm\ngit commit -m 'Initial commit.'\ngit push --set-upstream local master\npopd\n\n# Do some commit in subm, but do not push it to the remote.\npushd parent/subm\necho \"My frist commit.\" > s\ngit commit -a -m 'Change s'\npopd\n\n# Add the subm hash to the parent.\npushd parent\ngit add subm\ngit commit -m 'Updated subm.'\npopd\n\n# Amend the commit in subm (and optionally push it).\npushd parent/subm\necho \"My first commit.\" > s\ngit commit -a --amend -m 'Change s'\npopd\n\n# Correct that in the parent too:\npushd parent\ngit add subm\ngit commit -m 'Updated subm.'\npopd\n\n# At this point nothing was published yes, so the\n# amend shouldn't have caused a problem. But it did.\npushd parent\ngit push\npopd\n\necho \"THE ABOVE ERROR CAN NOW BE REPRODUCED INDEFINITELY,\"\necho \"FOR EXAMPLE, DO:\"\necho\necho \"cd parent/subm\"\necho \"git push\"\necho \"cd ..\"\necho \"git push\"\n\n\nEND-OF-SCRIPT\n\nTested with current master 4e59582ff70d299f5a88449891e78d15b4b3fabe\n\nRegards,\nCarlo\n\n-- \nCarlo Wood \n"},{"id":"310518","messageId":"xmqqvasxwee1.fsf@gitster.mtv.corp.google.com","threadId":"45002","inReplyTo":"20170129203348.1a8c0722@hikaru","subject":"Re: git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-01-30T01:00:22Z","receivedAt":"2017-01-30T01:00:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Carlo Wood writes:\n\n> there seems to be a problem with using 'git commit --amend' in\n> git submodules when using 'git push --recurse-submodules=on-demand'\n> in the parent.\n>\n> The latter fails, saying \"The following submodule paths contain changes\n> that can not be found on any remote:\" for such submodule, even though\n> the submodule is clean, pushed and reports 'Everything up-to-date'\n> when trying to push it.\n>\n> I believe that the reason has to be that the parent repository thinks\n> that the comment that was amended, but not pushed, must be on the remote\n> too, while the whole point of amend is that this commit is not pushed.\n\nI am not super familiar with the actualy implementation of the\ncodepaths involved in this, so CC'ed the folks who can help you\nbetter.\n\nI suspect the submodule folks would say it is working as intended,\nif \\\n\n - you made a commit in the submodule;\n - recorded the resulting commit in the superproject;\n - you amended the commit in the submodule; and then\n - you did \"push, while pushing out in the submodule as needed\" from\n the superproject.\n\nThere are two commits in the submodule that are involved in the\nabove scenario, and the first one before amending is needed by the\nother participants of the project in order for them to check out\nwhat you are trying to push in the superproject, because that is\nwhat the superproject's tree records. You somehow need to make that\ncommit available to them, but after you amended, the original commit\nmay no longer be reachable from any branch in your submodule, so\neven if you (or the \"on-demand\" mechanism) pushed any and all\nbranches out, that would not make the needed commit available to\nothers. If you push your top-level superproject out in such a\nsituation, you would break others.\n\nI think you have two options.\n\n 1. If the amend was done to improve things in submodule but is not\n quite ready, then get rid of that amended commit and restore the\n branch in the submodule to the state before you amended, i.e.\n the tip of the branch will become the same commit as the one\n that is recorded in the superproject. Then push the submodule\n and the superproject out. After that, move the submodule branch\n to point at the amended commit (or record the amended commit as\n a child of the commit you pushed out).\n\n 2. If the amend is good and ready to go, \"git add\" to update the\n superproject to make that amended result the one that is needed\n in the submodule.\n"},{"id":"310623","messageId":"CAGZ79kbCfKVDq+9Pr5LmOtT=+uB+u+EMQg1=FUNS2umCvtvHhg@mail.gmail.com","threadId":"45002","inReplyTo":"xmqqvasxwee1.fsf@gitster.mtv.corp.google.com","subject":"Re: git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-01-31T22:08:41Z","receivedAt":"2017-01-31T22:09:32Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Jan 29, 2017 at 5:00 PM, Junio C Hamano wrote:\n> Carlo Wood writes:\n>\n>> there seems to be a problem with using 'git commit --amend' in\n>> git submodules when using 'git push --recurse-submodules=on-demand'\n>> in the parent.\n>>\n>> The latter fails, saying \"The following submodule paths contain changes\n>> that can not be found on any remote:\" for such submodule, even though\n>> the submodule is clean, pushed and reports 'Everything up-to-date'\n>> when trying to push it.\n>>\n>> I believe that the reason has to be that the parent repository thinks\n>> that the comment that was amended, but not pushed, must be on the remote\n>> too, while the whole point of amend is that this commit is not pushed.\n>\n> I am not super familiar with the actualy implementation of the\n> codepaths involved in this, so CC'ed the folks who can help you\n> better.\n>\n> I suspect the submodule folks would say it is working as intended,\n> if \\\n>\n> - you made a commit in the submodule;\n> - recorded the resulting commit in the superproject;\n> - you amended the commit in the submodule; and then\n> - you did \"push, while pushing out in the submodule as needed\" from\n> the superproject.\n\nYes, for the current state of affairs, this is it.\n\n>\n> There are two commits in the submodule that are involved in the\n> above scenario, and the first one before amending is needed by the\n> other participants of the project in order for them to check out\n> what you are trying to push in the superproject, because that is\n> what the superproject's tree records. You somehow need to make that\n> commit available to them, but after you amended, the original commit\n> may no longer be reachable from any branch in your submodule, so\n> even if you (or the \"on-demand\" mechanism) pushed any and all\n> branches out, that would not make the needed commit available to\n> others. If you push your top-level superproject out in such a\n> situation, you would break others.\n\nIn the long term future, we may want to reject non-fastforward\nsubmodule updates. (Not sure if that is feasible)\n\n>\n> I think you have two options.\n>\n> 1. If the amend was done to improve things in submodule but is not\n> quite ready, then get rid of that amended commit and restore the\n> branch in the submodule to the state before you amended, i.e.\n> the tip of the branch will become the same commit as the one\n> that is recorded in the superproject. Then push the submodule\n> and the superproject out. After that, move the submodule branch\n> to point at the amended commit (or record the amended commit as\n> a child of the commit you pushed out).\n>\n> 2. If the amend is good and ready to go, \"git add\" to update the\n> superproject to make that amended result the one that is needed\n> in the submodule.\n\nyup.\n"},{"id":"310635","messageId":"20170201021040.4f6cc827@hikaru","threadId":"45002","inReplyTo":"CAGZ79kbCfKVDq+9Pr5LmOtT=+uB+u+EMQg1=FUNS2umCvtvHhg@mail.gmail.com","subject":"Re: git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Carlo Wood","fromEmail":"carlo@alinoe.com","sentAt":"2017-02-01T01:10:40Z","receivedAt":"2017-02-01T01:48:32Z","isPatch":false,"sender":{"key":"carlo@alinoe.com","avatar":"https://gravatar.com/avatar/f9ef6306f40295ad6047a49139a3e240d5c297092eafab3da2e34416ba1193d9?d=mp&s=160"},"body":"On Tue, 31 Jan 2017 14:08:41 -0800\nStefan Beller wrote:\n\n> On Sun, Jan 29, 2017 at 5:00 PM, Junio C Hamano \n> wrote:\n> > 2. If the amend is good and ready to go, \"git add\" to update the\n> > superproject to make that amended result the one that is needed\n> > in the submodule. \n> \n> yup.\n\nBut that is what I am doing. The amended commit IS already\nadded to the superproject (and pushed to the remote).\n\nPlease have a look at my script, this happens here:\n\n# Correct that in the parent too:\npushd parent\ngit add subm\ngit commit -m 'Updated subm.'\npopd\n\nThe commit from before the amend was added to the super\nproject (but never pushed) but has now been completely\nreplaced. I still think this is a flaw in git. It shouldn't\nnot complain and simply push.\n\n-- \nCarlo Wood \n"},{"id":"310636","messageId":"20170201022203.4b6489de@hikaru","threadId":"45002","inReplyTo":"xmqqvasxwee1.fsf@gitster.mtv.corp.google.com","subject":"Re: git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Carlo Wood","fromEmail":"carlo@alinoe.com","sentAt":"2017-02-01T01:22:03Z","receivedAt":"2017-02-01T02:16:40Z","isPatch":false,"sender":{"key":"carlo@alinoe.com","avatar":"https://gravatar.com/avatar/f9ef6306f40295ad6047a49139a3e240d5c297092eafab3da2e34416ba1193d9?d=mp&s=160"},"body":"On Sun, 29 Jan 2017 17:00:22 -0800\nJunio C Hamano wrote:\n\n> I suspect the submodule folks would say it is working as intended,\n> if \\\n> \n> - you made a commit in the submodule;\n> - recorded the resulting commit in the superproject;\n> - you amended the commit in the submodule; and then\n> - you did \"push, while pushing out in the submodule as needed\" from\n> the superproject.\n\nThis is not what I'm doing.\nThis is what I'm doing (see my script):\n\n - you made a commit in the submodule;\n - recorded the resulting commit in the superproject;\n - you amended the commit in the submodule;\n - you record the amended commit in the superproject; <=== !\n - you push the submodule out (or not, the on-demand does that\n anyway)\n - you try to push the superproject, but that fails,\n as long as you use recurseSubmodules=on-demand.\n\n> \n> There are two commits in the submodule that are involved in the\n> above scenario, and the first one before amending is needed by the\n> other participants of the project in order for them to check out\n> what you are trying to push in the superproject, because that is\n> what the superproject's tree records.\n\nI never pushed anything of that, so the other participants don't\nknow, nor have, the pre-amended commit.\n\nIt is true that the superproject THINKS that the pre-amended commit\nis a normal commit though: the last recorded (amended) commit is\ninternally listed as being on top of the amended commit (which is\nincorrect). This is why the superproject assumes that the current\nadd commit of the submodule needs the pre-amended commit to be\navailable too. This is not correct however, it is not needed to\nbe available to others and does not need to be pushed to a remote.\n\n> I think you have two options.\n> \n> 1. If the amend was done to improve things in submodule but is not\n> quite ready, then get rid of that amended commit and restore the\n> branch in the submodule to the state before you amended, i.e.\n> the tip of the branch will become the same commit as the one\n> that is recorded in the superproject. Then push the submodule\n> and the superproject out. After that, move the submodule branch\n> to point at the amended commit (or record the amended commit as\n> a child of the commit you pushed out).\n\nThat would work, but would be a horrible workaround for an existing\nbug :p\n\n> 2. If the amend is good and ready to go, \"git add\" to update the\n> superproject to make that amended result the one that is needed\n> in the submodule.\n\nThis was already done, also in the script that I provided.\nYet, the push in the superproject is still rejected.\n\n-- \nCarlo Wood \n"},{"id":"310859","messageId":"CAGZ79kYXc-cT-059ZpoVrUjLodY-x61bW2ANQqtcAQLzGiRZ3A@mail.gmail.com","threadId":"45002","inReplyTo":"20170201021040.4f6cc827@hikaru","subject":"Re: git push failing when push.recurseSubmodules on-demand and git commit --amend was used in submodule.","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-02-04T18:43:56Z","receivedAt":"2017-02-04T18:44:03Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 31, 2017 at 5:10 PM, Carlo Wood wrote:\n> On Tue, 31 Jan 2017 14:08:41 -0800\n> Stefan Beller wrote:\n>\n>> On Sun, Jan 29, 2017 at 5:00 PM, Junio C Hamano \n>> wrote:\n>> > 2. If the amend is good and ready to go, \"git add\" to update the\n>> > superproject to make that amended result the one that is needed\n>> > in the submodule.\n>>\n>> yup.\n>\n> But that is what I am doing. The amended commit IS already\n> added to the superproject (and pushed to the remote).\n>\n> Please have a look at my script, this happens here:\n>\n> # Correct that in the parent too:\n> pushd parent\n> git add subm\n> git commit -m 'Updated subm.'\n> popd\n\nAnd if you were to use ammend here, too; there would be no problem;\nIn the parent there are now two commits, the first one pointing at the\nfirst (unammended) commit in the submodule, the second pointing to\nthe corrected commit.\n\n>\n> The commit from before the amend was added to the super\n> project (but never pushed) but has now been completely\n> replaced. I still think this is a flaw in git. It shouldn't\n> not complain and simply push.\n\nThe problem here is in the design.\nWhen \"on-demand\" is set, the parent repo determines which\nsumbodules needs pushing, then runs a plain \"git push\" in them\nand then checks again.\n\nThe push operation in the submodule did not push out the un-amended\ncommit (which is correct IMO).\n\nThe parent in the second check determines that there is a commit, that\nis not pushed, though. Maybe we need an option there \"on-demand-but-no-recheck\"\nas a weaker promise what Git can deliver there.\n\n>\n> --\n> Carlo Wood \n"}]}