{"thread":{"id":"57931","subject":"request for development/build workflow suggestion/fix","startedAt":"2022-05-30T01:50:16Z","lastAt":"2022-05-30T22:04:33Z","messageCount":4,"participants":["graham.menhennitt@c4i.com","rsbecker@nexbridge.com","Kerry, Richard"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"456362","messageId":"BN2P110MB17460C18D525E831A63C4376C5DD9@BN2P110MB1746.NAMP110.PROD.OUTLOOK.COM","threadId":"57931","inReplyTo":null,"subject":"request for development/build workflow suggestion/fix","fromName":"","fromEmail":"graham.menhennitt@c4i.com","sentAt":"2022-05-30T01:45:04Z","receivedAt":"2022-05-30T01:50:16Z","isPatch":false,"sender":{"key":"graham.menhennitt@c4i.com","avatar":null},"body":"We have a (small) problem in our development/build workflow. I'd like to request suggestions on either:\n- how we can improve the workflow to avoid the problem, or\n- how we can fix our build system so that the problem doesn't occur. \n\nOur (simplified) development workflow for fixing a bug or adding functionality:\n- clone the repo and create a private branch off the Integration branch\n- fix the bug or develop the functionality (and test)\n- merge the private branch to Integration\n- create a tag to identify the fix\n\nOur build system:\n- clone the repo's Integration branch\n- edit a 'version' file in the repo that contains the current build number and increment that number\n- build the software\n- commit the edited version file\n- create a tag identifying the build number\n- produce a list of changes since the previous build by diffing between the newly created tag and the previous build tag\n\nThe problem:\n- if a developer merges to Integration and creates a tag while the build is in progress, his tag gets included in the diffs even though his changes aren't actually included in the build.\n\nBelow is a shell script that demonstrates the problem. A quick description:\n- create an empty git repo 'origin'\n- clone it to repo 'buildClone', create an Integration branch, add a source file and 'version' file with version = 1, tag that as build 1, push\n- increment the version to 2 as if we were doing build 2, commit and tag as build 2 but don't push yet\n- create a second clone 'develClone', create a development branch, modify the source file, commit, merge the branch to Integration, tag with 'develTag', push to origin\n- in the build clone, pull and then push to origin\n- increment the version file as if doing build 3, commit, tag as build 3, pull and push\n- get the diffs between builds 1 & 2, and 2 & 3 - develTag should not be in the first set of diffs but it should be in the second set\n\nSo, my question: can anybody please suggest an improvement to our development or build workflows to fix/avoid the problem? The obvious change of omitting the pulls before pushing from buildClone doesn't work - git won't allow the push.\n\nMany thanks in advance for any suggestions.\n\tGraham\n\n\n========================================== shell script ==========================\nexport GIT_COMMITTER_NAME=committer\nexport GIT_COMMITTER_EMAIL=committer@example.com\nexport GIT_AUTHOR_NAME=author\nexport GIT_AUTHOR_EMAIL=author@example.com\n\nset -e # exit on error\n\nset -x # for debugging\n\n# clean up any previous leftovers and start afresh\nrm -rf origin buildClone develClone\nmkdir -p origin buildClone develClone\n\n# create a bare repo in 'origin'\ngit -C origin init --bare\n\n# and a clone for building\ngit -C buildClone clone ../origin .\n\n# create an Integration branch and check it out\ngit -C buildClone checkout -b Integration\n\n# create a source file, commit it, and push\necho \"this is some source code\" > buildClone/sourceFile\ngit -C buildClone add sourceFile\ngit -C buildClone commit --all --message \"add source file\"\ngit -C buildClone push --set-upstream origin refs/heads/Integration:refs/heads/Integration\n\n# create a version file, commit it, and push (as if build 1 had just been completed)\necho \"version=1\" > buildClone/version\ngit -C buildClone add version\ngit -C buildClone commit --all --message \"add version\"\ngit -C buildClone push\n# and add a tag to it\ngit -C buildClone tag build1\ngit -C buildClone push --tags\n\n# edit the file to increment the version, and commit it (but don't push it yet)\nsed --in-place --expression \"s/1/2/\" buildClone/version\ngit -C buildClone commit --all --message \"increment version\"\n# and create a tag on the build\ngit -C buildClone tag build2\n\n\n                        # develClone repo\n\n                        # clone again for development work and checkout Integration branch\n                        git -C develClone clone -b Integration ../origin .\n\n                        # create a branch and check it out, edit the source file, commit it, and push it (as if a developer was making changes)\n                        git -C develClone checkout -b develBranch\n                        sed --in-place --expression \"s/$/ with a change/\" develClone/sourceFile\n                        git -C develClone commit --all --message \"edit source file\"\n                        git -C develClone push --set-upstream origin refs/heads/develBranch:refs/heads/develBranch\n\n                        # merge the branch into Integration\n                        git -C develClone checkout Integration\n                        git -C develClone merge --no-ff --commit -m \"merge develBranch to Integration\" develBranch\n\n                        # create a development tag\n                        git -C develClone tag develTag\n\n                        # and push everything\n                        git -C develClone push\n                        git -C develClone push --tags\n\n# back in buildClone\n\n# get any changes from origin that have occurred since we originally cloned\ngit -C buildClone pull --ff --no-edit\n\n# and push everything\ngit -C buildClone push --force\ngit -C buildClone push --tags --force\n\n# now do build 3\n\n# edit the file to increment the version, and commit it (but don't push it yet)\nsed --in-place --expression \"s/2/3/\" buildClone/version\ngit -C buildClone commit --all --message \"increment version\"\n# and create a tag on the build\ngit -C buildClone tag build3\n\n# get any changes from origin that have occurred since we originally cloned\ngit -C buildClone pull --ff --no-edit\n\n# and push everything\ngit -C buildClone push --force\ngit -C buildClone push --tags --force\n\n# finally, test the results\nfailed=false\n\n# we do not want the following to contain develTag (but, of course, it does)\nif git -C buildClone tag --contains build1 --no-contains build2 | grep develTag; then\n    echo \"build2 contains develTag when it shouldn't\"\n    failed=true\nfi\n\n# and we do want the following to contain develTag (but, of course, it doesn't)\nif ! git -C buildClone tag --contains build2 --no-contains build3 | grep develTag; then\n    echo \"build3 doesn't contain develTag when it should\"\n    failed=true\nfi\n\nif $failed; then\n    echo \"!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed !!!!!!!!!!!!!!!!!!!!!!\"\n    exit 1\nfi\n"},{"id":"456363","messageId":"033c01d873cb$3001f6a0$9005e3e0$@nexbridge.com","threadId":"57931","inReplyTo":"BN2P110MB17460C18D525E831A63C4376C5DD9@BN2P110MB1746.NAMP110.PROD.OUTLOOK.COM","subject":"RE: request for development/build workflow suggestion/fix","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2022-05-30T02:15:47Z","receivedAt":"2022-05-30T02:15:58Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On May 29, 2022 9:45 PM, Graham Menhennitt wrote:\n>We have a (small) problem in our development/build workflow. I'd like to\nrequest\n>suggestions on either:\n>- how we can improve the workflow to avoid the problem, or\n>- how we can fix our build system so that the problem doesn't occur.\n>\n>Our (simplified) development workflow for fixing a bug or adding\nfunctionality:\n>- clone the repo and create a private branch off the Integration branch\n>- fix the bug or develop the functionality (and test)\n>- merge the private branch to Integration\n>- create a tag to identify the fix\n>\n>Our build system:\n>- clone the repo's Integration branch\n>- edit a 'version' file in the repo that contains the current build number\nand\n>increment that number\n>- build the software\n>- commit the edited version file\n>- create a tag identifying the build number\n>- produce a list of changes since the previous build by diffing between the\nnewly\n>created tag and the previous build tag\n>\n>The problem:\n>- if a developer merges to Integration and creates a tag while the build is\nin\n>progress, his tag gets included in the diffs even though his changes aren't\nactually\n>included in the build.\n>\n>Below is a shell script that demonstrates the problem. A quick description:\n>- create an empty git repo 'origin'\n>- clone it to repo 'buildClone', create an Integration branch, add a source\nfile and\n>'version' file with version = 1, tag that as build 1, push\n>- increment the version to 2 as if we were doing build 2, commit and tag as\nbuild 2\n>but don't push yet\n>- create a second clone 'develClone', create a development branch, modify\nthe\n>source file, commit, merge the branch to Integration, tag with 'develTag',\npush to\n>origin\n>- in the build clone, pull and then push to origin\n>- increment the version file as if doing build 3, commit, tag as build 3,\npull and push\n>- get the diffs between builds 1 & 2, and 2 & 3 - develTag should not be in\nthe first\n>set of diffs but it should be in the second set\n>\n>So, my question: can anybody please suggest an improvement to our\n>development or build workflows to fix/avoid the problem? The obvious change\nof\n>omitting the pulls before pushing from buildClone doesn't work - git won't\nallow\n>the push.\n>\n>Many thanks in advance for any suggestions.\n>\tGraham\n>\n>\n>========================================== shell script\n>========================== export GIT_COMMITTER_NAME=committer\n>export GIT_COMMITTER_EMAIL=committer@example.com\n>export GIT_AUTHOR_NAME=author\n>export GIT_AUTHOR_EMAIL=author@example.com\n>\n>set -e # exit on error\n>\n>set -x # for debugging\n>\n># clean up any previous leftovers and start afresh rm -rf origin buildClone\n>develClone mkdir -p origin buildClone develClone\n>\n># create a bare repo in 'origin'\n>git -C origin init --bare\n>\n># and a clone for building\n>git -C buildClone clone ../origin .\n>\n># create an Integration branch and check it out git -C buildClone checkout\n-b\n>Integration\n>\n># create a source file, commit it, and push echo \"this is some source code\"\n>\n>buildClone/sourceFile git -C buildClone add sourceFile git -C buildClone\ncommit --\n>all --message \"add source file\"\n>git -C buildClone push --set-upstream origin\n>refs/heads/Integration:refs/heads/Integration\n>\n># create a version file, commit it, and push (as if build 1 had just been\ncompleted)\n>echo \"version=1\" > buildClone/version git -C buildClone add version git -C\n>buildClone commit --all --message \"add version\"\n>git -C buildClone push\n># and add a tag to it\n>git -C buildClone tag build1\n>git -C buildClone push --tags\n>\n># edit the file to increment the version, and commit it (but don't push it\nyet) sed --\n>in-place --expression \"s/1/2/\" buildClone/version git -C buildClone commit\n--all --\n>message \"increment version\"\n># and create a tag on the build\n>git -C buildClone tag build2\n>\n>\n>                        # develClone repo\n>\n>                        # clone again for development work and checkout\nIntegration branch\n>                        git -C develClone clone -b Integration ../origin .\n>\n>                        # create a branch and check it out, edit the source\nfile, commit it, and\n>push it (as if a developer was making changes)\n>                        git -C develClone checkout -b develBranch\n>                        sed --in-place --expression \"s/$/ with a change/\"\n>develClone/sourceFile\n>                        git -C develClone commit --all --message \"edit\nsource file\"\n>                        git -C develClone push --set-upstream origin\n>refs/heads/develBranch:refs/heads/develBranch\n>\n>                        # merge the branch into Integration\n>                        git -C develClone checkout Integration\n>                        git -C develClone merge --no-ff --commit -m \"merge\ndevelBranch to\n>Integration\" develBranch\n>\n>                        # create a development tag\n>                        git -C develClone tag develTag\n>\n>                        # and push everything\n>                        git -C develClone push\n>                        git -C develClone push --tags\n>\n># back in buildClone\n>\n># get any changes from origin that have occurred since we originally cloned\ngit -C\n>buildClone pull --ff --no-edit\n>\n># and push everything\n>git -C buildClone push --force\n>git -C buildClone push --tags --force\n>\n># now do build 3\n>\n># edit the file to increment the version, and commit it (but don't push it\nyet) sed --\n>in-place --expression \"s/2/3/\" buildClone/version git -C buildClone commit\n--all --\n>message \"increment version\"\n># and create a tag on the build\n>git -C buildClone tag build3\n>\n># get any changes from origin that have occurred since we originally cloned\ngit -C\n>buildClone pull --ff --no-edit\n>\n># and push everything\n>git -C buildClone push --force\n>git -C buildClone push --tags --force\n>\n># finally, test the results\n>failed=false\n>\n># we do not want the following to contain develTag (but, of course, it\ndoes) if git -\n>C buildClone tag --contains build1 --no-contains build2 | grep develTag;\nthen\n>    echo \"build2 contains develTag when it shouldn't\"\n>    failed=true\n>fi\n>\n># and we do want the following to contain develTag (but, of course, it\ndoesn't) if !\n>git -C buildClone tag --contains build2 --no-contains build3 | grep\ndevelTag; then\n>    echo \"build3 doesn't contain develTag when it should\"\n>    failed=true\n>fi\n>\n>if $failed; then\n>    echo \"!!!!!!!!!!!!!!!!!!!!!!!!!!!! failed !!!!!!!!!!!!!!!!!!!!!!\"\n>    exit 1\n>fi\n\nI don't know whether your process might be able to distinguish between\nannotated and unannotated tags. git describe does. Maybe using one or the\nother might enhance what you are trying to do.\n--Randall\n\n"},{"id":"456364","messageId":"AS8PR02MB7302BAA142824332D1BE24939CDD9@AS8PR02MB7302.eurprd02.prod.outlook.com","threadId":"57931","inReplyTo":"BN2P110MB17460C18D525E831A63C4376C5DD9@BN2P110MB1746.NAMP110.PROD.OUTLOOK.COM","subject":"RE: request for development/build workflow suggestion/fix","fromName":"Kerry, Richard","fromEmail":"richard.kerry@atos.net","sentAt":"2022-05-30T09:31:37Z","receivedAt":"2022-05-30T09:32:50Z","isPatch":false,"sender":{"key":"richard.kerry@atos.net","avatar":null},"body":"> -----Original Message-----\n> From: Graham.Menhennitt@c4i.com <Graham.Menhennitt@c4i.com>\n> Sent: 30 May 2022 02:45\n> To: git@vger.kernel.org\n> Subject: request for development/build workflow suggestion/fix\n> \n\n> Our (simplified) development workflow for fixing a bug or adding\n> functionality:\n> - clone the repo and create a private branch off the Integration branch\n> - fix the bug or develop the functionality (and test)\n> - merge the private branch to Integration\n> - create a tag to identify the fix\n> \n> Our build system:\n> - clone the repo's Integration branch\n> - edit a 'version' file in the repo that contains the current build number and\n> increment that number\n> - build the software\n> - commit the edited version file\n> - create a tag identifying the build number\n> - produce a list of changes since the previous build by diffing between the\n> newly created tag and the previous build tag\n> \n> The problem:\n> - if a developer merges to Integration and creates a tag while the build is in\n> progress, his tag gets included in the diffs even though his changes aren't\n> actually included in the build.\n \n> So, my question: can anybody please suggest an improvement to our\n> development or build workflows to fix/avoid the problem? The obvious\n> change of omitting the pulls before pushing from buildClone doesn't work -\n> git won't allow the push.\n> \n> Many thanks in advance for any suggestions.\n> \tGraham\n> \n\nDon't merge to the integration branch, commit to the development branch (ie the trunk)\nAt the moment when a build is to be started, branch it generating a new integration branch (maybe also tag its start point).  Build from that.\nThus at that point in time it is frozen.  No-one commits to it.\n\nIntegration manager creates the integration branch and is then in control of what may be merged to it.  The rest of the team can commit to the development branch and their work will not hit integration without something being done explicitly to put it there.\n\nMaybe,\nRichard.\n\n\n"},{"id":"456379","messageId":"BN2P110MB174675FF841676169C7F578DC5DD9@BN2P110MB1746.NAMP110.PROD.OUTLOOK.COM","threadId":"57931","inReplyTo":"AS8PR02MB7302BAA142824332D1BE24939CDD9@AS8PR02MB7302.eurprd02.prod.outlook.com","subject":"RE: request for development/build workflow suggestion/fix","fromName":"","fromEmail":"graham.menhennitt@c4i.com","sentAt":"2022-05-30T21:59:17Z","receivedAt":"2022-05-30T22:04:33Z","isPatch":false,"sender":{"key":"graham.menhennitt@c4i.com","avatar":null},"body":"Thanks for replying, Richard.\n\nI'll have a think about your suggestion. It's a fairly radical change from what we currently do, but I'll consider it.\n\nThanks,\n\tGraham\n\n-----Original Message-----\nFrom: Kerry, Richard <richard.kerry@atos.net> \nSent: Monday, 30 May 2022 7:32 PM\nTo: Graham.Menhennitt@c4i.com; git@vger.kernel.org\nSubject: [EXTERNAL] RE: request for development/build workflow suggestion/fix\n\nWARNING: Email originated from an EXTERNAL source. Do NOT click links, open attachments or reply unless you recognize sender and know the content is safe.\n\n> -----Original Message-----\n> From: Graham.Menhennitt@c4i.com <Graham.Menhennitt@c4i.com>\n> Sent: 30 May 2022 02:45\n> To: git@vger.kernel.org\n> Subject: request for development/build workflow suggestion/fix\n>\n\n> Our (simplified) development workflow for fixing a bug or adding\n> functionality:\n> - clone the repo and create a private branch off the Integration \n> branch\n> - fix the bug or develop the functionality (and test)\n> - merge the private branch to Integration\n> - create a tag to identify the fix\n>\n> Our build system:\n> - clone the repo's Integration branch\n> - edit a 'version' file in the repo that contains the current build \n> number and increment that number\n> - build the software\n> - commit the edited version file\n> - create a tag identifying the build number\n> - produce a list of changes since the previous build by diffing \n> between the newly created tag and the previous build tag\n>\n> The problem:\n> - if a developer merges to Integration and creates a tag while the \n> build is in progress, his tag gets included in the diffs even though \n> his changes aren't actually included in the build.\n\n> So, my question: can anybody please suggest an improvement to our \n> development or build workflows to fix/avoid the problem? The obvious \n> change of omitting the pulls before pushing from buildClone doesn't \n> work - git won't allow the push.\n>\n> Many thanks in advance for any suggestions.\n>       Graham\n>\n\nDon't merge to the integration branch, commit to the development branch (ie the trunk) At the moment when a build is to be started, branch it generating a new integration branch (maybe also tag its start point).  Build from that.\nThus at that point in time it is frozen.  No-one commits to it.\n\nIntegration manager creates the integration branch and is then in control of what may be merged to it.  The rest of the team can commit to the development branch and their work will not hit integration without something being done explicitly to put it there.\n\nMaybe,\nRichard.\n\n\n"}]}