{"thread":{"id":"45280","subject":"Finding a tag that introduced a submodule change","startedAt":"2017-03-03T15:41:32Z","lastAt":"2017-03-16T10:17:36Z","messageCount":6,"participants":["Robert Dailey","Jacob Keller","Junio C Hamano","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"313185","messageId":"CAHd499CfJnPtLmi8qzr=_jrfCgMw85MOUv-wPKmAHFUyDFXhRA@mail.gmail.com","threadId":"45280","inReplyTo":null,"subject":"Finding a tag that introduced a submodule change","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-03-03T15:40:04Z","receivedAt":"2017-03-03T15:41:32Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"I have a repository with a single submodule in it. Since the parent\nrepository represents the code base for an actual product, I tag\nrelease versions in the parent repository. I do not put tags in the\nsubmodule since multiple other products may be using it there and I\nwanted to avoid ambiguous tags.\n\nSometimes I run into a situation where I need to find out which\nrelease of the product a submodule change was introduced in. This is\nnontrivial, since there are no tags in the submodule itself. This is\none thing I tried:\n\n1. Do a `git log` in the submodule to find the SHA1 representing the\nchange I want to check for\n2. In the parent repository, do a git log with pickaxe to determine\nwhen the submodule itself changed to the value of that SHA1.\n3. Based on the result of #2, do a `git tag --contains` to see the\nlowest-version tag that contains the SHA1, which will identify the\nfirst release that introduced that change\n\nHowever, I was not able to get past #2 because apparently there are\ncases where when we move the submodule \"forward\", we skip over\ncommits, so the value of the submodule itself never was set to that\nSHA1.\n\nI'm at a loss here on how to easily do this. Can someone recommend a\nway to do this? Obviously the easier the better, as I have to somehow\ntrain my team how to do this on their own.\n\nThanks in advance.\n"},{"id":"313210","messageId":"CA+P7+xr=RmPy45DL_Qc2LcWKS-3AzVcPXqQaosZdf4jhnRp1TQ@mail.gmail.com","threadId":"45280","inReplyTo":"CAHd499CfJnPtLmi8qzr=_jrfCgMw85MOUv-wPKmAHFUyDFXhRA@mail.gmail.com","subject":"Re: Finding a tag that introduced a submodule change","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-03-03T16:39:21Z","receivedAt":"2017-03-03T17:46:13Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Fri, Mar 3, 2017 at 7:40 AM, Robert Dailey wrote:\n> I have a repository with a single submodule in it. Since the parent\n> repository represents the code base for an actual product, I tag\n> release versions in the parent repository. I do not put tags in the\n> submodule since multiple other products may be using it there and I\n> wanted to avoid ambiguous tags.\n>\n\nHi,\n\nI agree you shouldn't use tags in the submodules.\n\n> Sometimes I run into a situation where I need to find out which\n> release of the product a submodule change was introduced in. This is\n> nontrivial, since there are no tags in the submodule itself. This is\n> one thing I tried:\n>\n\nI've run into this exact problem at $DAYJOB.\n\n> 1. Do a `git log` in the submodule to find the SHA1 representing the\n> change I want to check for\n> 2. In the parent repository, do a git log with pickaxe to determine\n> when the submodule itself changed to the value of that SHA1.\n> 3. Based on the result of #2, do a `git tag --contains` to see the\n> lowest-version tag that contains the SHA1, which will identify the\n> first release that introduced that change\n>\n> However, I was not able to get past #2 because apparently there are\n> cases where when we move the submodule \"forward\", we skip over\n> commits, so the value of the submodule itself never was set to that\n> SHA1.\n>\n> I'm at a loss here on how to easily do this. Can someone recommend a\n> way to do this? Obviously the easier the better, as I have to somehow\n> train my team how to do this on their own.\n>\n> Thanks in advance.\n\nSo there's better ways to do this, but I do think there would be value\nin adding some plumbing to make it easier.\n\nHere is how I would do this, best if written into a shell script or\nsimilar to automate the tricky part of #2\n\n1. Do a git-log of the *parent* project, filtering out to show only\nthe path to the submodule\n\n2. For each commit here, you find the new and old values of the\nsubmodule pointer.\n\n3. Use git merge-base --is-ancestor to ensure that \"old\" is an\nancestor of \"submodule sha1id\" and then\n\n4. Use git-merge-base to ensure that \"submodule sha1id\" is an\nanscestor of \"new\".\n\nIf both these are tree, then you know that the commit was included\ninto the parent project at this point.\n\nI've had to do this once or twice, but I don't actually remember\nexactly how I did 3. One sneaky way would be to add new tags for each\nsubmodule change something like the following might work and be more\nefficient. I'm not really sure but here's how I would go that route:\n\n1. git log --pretty=%h | parallel git ls-tree {}\npath-to-submodule\n\nThe above more or less prints every submodule value as it changed over\ntime in the parent project.\n\nNext, for each submodule change:\n\n2. git -C tag parent/ \n\nCreate a new tag prefixed by \"parent\" that includes the sha1id of the\nparent commit, and create it inside the submodule\n\n3. git -C submodule describe --contains --match=\"parent/*\" \n\nOnce you're done you can also delete all the tags that are in the\n\"parent\" prefix if you dont' really wanna see them again.\n\nBasically, re-use the machinery to tag and then use describe\n--contains to find the commit.\n\nI *really* think a similar algorithm could be embedded as a plumbing\nsubcommand, since I think this is tedious to do by hand.\n\nI'm not really sure if this is the \"best\" algorithm either, but it's\npretty much what I've used in the past. Either the tag way or the log\nyourself one at a time way.\n"},{"id":"313213","messageId":"xmqqy3wm1bkm.fsf@junio-linux.mtv.corp.google.com","threadId":"45280","inReplyTo":"CAHd499CfJnPtLmi8qzr=_jrfCgMw85MOUv-wPKmAHFUyDFXhRA@mail.gmail.com","subject":"Re: Finding a tag that introduced a submodule change","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-03T18:04:25Z","receivedAt":"2017-03-03T18:08:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robert Dailey writes:\n\n> Sometimes I run into a situation where I need to find out which\n> release of the product a submodule change was introduced in. This is\n> nontrivial, since there are no tags in the submodule itself.\n\nDoes your superproject rewind the commit in the submodule project as\nit goes forward? That is, is this property guaranteed to hold by\nyour project's discipline:\n\n\tGiven any two commits C1 and C2 in the superproject, and the\n\tcommit in the submodule bound to C1's and C2's tree (call\n\tthem S1 and S2, respectively), if C1 is an ancestor of C2,\n\tthen S1 is the same as S2 or an ancestor of S2.\n\nIf so, I think you can do a bisection of the history in the\nsuperproject. Pick an old commit in the superproject that binds an\nold commit from the submodule that does not have the change and call\nit \"good\". Similarly pick a new one in the superproject that binds\na newer commit from the submodule that does have the change, and\ncall it \"bad\". Then do\n\n\t$ git bisect start $bad $good -- $path_to_submodule\n\nwhich would suggest you to test commits that change what commit is\nbound at the submodule's path.\n\nWhen testing each of these commits, you would see if the commit\nbound at the submodule's path has the change or not.\n\n\t$ current=$(git rev-parse HEAD:$path_to_submodule)\n\nwould give you the object name of that commit, and then\n\n\t$ git -C $path_to_submodule merge-base --is-ancestor $change $current\n\nwould tell you if the $change you are interested in is already\ncontained in that $current commit. Then you say \"git bisect good\"\nif $current is too old to contain the $change, and \"git bisect bad\"\nif $current is sufficiently new and contains the $change, to\ncontinue.\n\nIf your superproject rewinds the commit in the submodule as it goes\nforward, e.g. an older commit in the superproject used submodule\ncommit from day X, but somebody who made yesterday's commit in the\nsuperproject realized that that submodule commit was broken and used\nan older commit in the submodule from day (X-1), then you cannot\nbisect. In such a case, I think you would essentially need to check\nall superproject commits that changed the commit bound at the\nsubmodule's path.\n\n\t$ git rev-list $bad..$good -- $path_to_submodule\n\nwould give a list of such commits, and you would do the \"merge-base\"\ncheck for all them to see which ones have and do not have the\n$change (replace \"HEAD\" with the commit you are testing in the\ncomputation that gives you $current).\n\n\n"},{"id":"314136","messageId":"CAHd499Bf-Y5uuZKPO5dOaJqYJ7ZGhG0nX7nQmw7+KO8tZr8eMQ@mail.gmail.com","threadId":"45280","inReplyTo":"xmqqy3wm1bkm.fsf@junio-linux.mtv.corp.google.com","subject":"Re: Finding a tag that introduced a submodule change","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2017-03-15T14:12:55Z","receivedAt":"2017-03-15T14:13:53Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Fri, Mar 3, 2017 at 12:04 PM, Junio C Hamano wrote:\n> Robert Dailey writes:\n>\n>> Sometimes I run into a situation where I need to find out which\n>> release of the product a submodule change was introduced in. This is\n>> nontrivial, since there are no tags in the submodule itself.\n>\n> Does your superproject rewind the commit in the submodule project as\n> it goes forward? That is, is this property guaranteed to hold by\n> your project's discipline:\n>\n> Given any two commits C1 and C2 in the superproject, and the\n> commit in the submodule bound to C1's and C2's tree (call\n> them S1 and S2, respectively), if C1 is an ancestor of C2,\n> then S1 is the same as S2 or an ancestor of S2.\n>\n> If so, I think you can do a bisection of the history in the\n> superproject. Pick an old commit in the superproject that binds an\n> old commit from the submodule that does not have the change and call\n> it \"good\". Similarly pick a new one in the superproject that binds\n> a newer commit from the submodule that does have the change, and\n> call it \"bad\". Then do\n>\n> $ git bisect start $bad $good -- $path_to_submodule\n>\n> which would suggest you to test commits that change what commit is\n> bound at the submodule's path.\n>\n> When testing each of these commits, you would see if the commit\n> bound at the submodule's path has the change or not.\n>\n> $ current=$(git rev-parse HEAD:$path_to_submodule)\n>\n> would give you the object name of that commit, and then\n>\n> $ git -C $path_to_submodule merge-base --is-ancestor $change $current\n>\n> would tell you if the $change you are interested in is already\n> contained in that $current commit. Then you say \"git bisect good\"\n> if $current is too old to contain the $change, and \"git bisect bad\"\n> if $current is sufficiently new and contains the $change, to\n> continue.\n>\n> If your superproject rewinds the commit in the submodule as it goes\n> forward, e.g. an older commit in the superproject used submodule\n> commit from day X, but somebody who made yesterday's commit in the\n> superproject realized that that submodule commit was broken and used\n> an older commit in the submodule from day (X-1), then you cannot\n> bisect. In such a case, I think you would essentially need to check\n> all superproject commits that changed the commit bound at the\n> submodule's path.\n>\n> $ git rev-list $bad..$good -- $path_to_submodule\n>\n> would give a list of such commits, and you would do the \"merge-base\"\n> check for all them to see which ones have and do not have the\n> $change (replace \"HEAD\" with the commit you are testing in the\n> computation that gives you $current).\n\n\nHi Junio, my apologies for the very late response.\n\nI really like your idea, however unfortunately often times people on\nmy team accidentally rewind the submodule. However your latter\nsuggestion about just doing merge-base on each change would be a\nworthy solution to try. Thank you very much, I will certainly give\nthis a try!!\n"},{"id":"314148","messageId":"CAGZ79ka56DtHeAMVK7T-voXm3aRrGvUPNYW9MC1YR+kcGriWJw@mail.gmail.com","threadId":"45280","inReplyTo":"CAHd499CfJnPtLmi8qzr=_jrfCgMw85MOUv-wPKmAHFUyDFXhRA@mail.gmail.com","subject":"Re: Finding a tag that introduced a submodule change","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-15T17:10:21Z","receivedAt":"2017-03-15T17:10:53Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Fri, Mar 3, 2017 at 7:40 AM, Robert Dailey wrote:\n> I have a repository with a single submodule in it. Since the parent\n> repository represents the code base for an actual product, I tag\n> release versions in the parent repository. I do not put tags in the\n> submodule since multiple other products may be using it there and I\n> wanted to avoid ambiguous tags.\n>\n> Sometimes I run into a situation where I need to find out which\n> release of the product a submodule change was introduced in. This is\n> nontrivial, since there are no tags in the submodule itself. This is\n> one thing I tried:\n>\n> 1. Do a `git log` in the submodule to find the SHA1 representing the\n> change I want to check for\n> 2. In the parent repository, do a git log with pickaxe to determine\n> when the submodule itself changed to the value of that SHA1.\n> 3. Based on the result of #2, do a `git tag --contains` to see the\n> lowest-version tag that contains the SHA1, which will identify the\n> first release that introduced that change\n>\n> However, I was not able to get past #2 because apparently there are\n> cases where when we move the submodule \"forward\", we skip over\n> commits, so the value of the submodule itself never was set to that\n> SHA1.\n>\n> I'm at a loss here on how to easily do this. Can someone recommend a\n> way to do this? Obviously the easier the better, as I have to somehow\n> train my team how to do this on their own.\n>\n> Thanks in advance.\n\nI cannot offer an easy way. However I can come up with a proposal\nhow to make this easy in the future. ;)\n\n\"git-{branch,tag} --contains\" currently only takes a commit id as that is\neasy to check for. (Just a revwalk from all commits, as we walk over the\ncommits in the graph)\n\nWe should extend the possible arguments to --contains, such that you can\ndo\n\n # check that a given path had this exact tree/blob id\n git tag --contains :\n # check if the given tree/blob was at any path\n git tag --contains \n # generalizing from above:\n git tag --contains [:]\n\nWith this designed API you could ask for\n\n git tag --contains submodule:\n\nFor the implementation of this feature the revwalk would also need\nto walk the object graph (as restricted by the pathspec) and\nsee if there is the given object for each tag.\n\nThanks,\nStefan\n"},{"id":"314214","messageId":"CA+P7+xoU3AA1FGRZno2mi+K4vhGy9EN3d4YDv_vqKOPqp7vboA@mail.gmail.com","threadId":"45280","inReplyTo":"CAGZ79ka56DtHeAMVK7T-voXm3aRrGvUPNYW9MC1YR+kcGriWJw@mail.gmail.com","subject":"Re: Finding a tag that introduced a submodule change","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-03-16T10:17:03Z","receivedAt":"2017-03-16T10:17:36Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Wed, Mar 15, 2017 at 10:10 AM, Stefan Beller wrote:\n> On Fri, Mar 3, 2017 at 7:40 AM, Robert Dailey wrote:\n>> I have a repository with a single submodule in it. Since the parent\n>> repository represents the code base for an actual product, I tag\n>> release versions in the parent repository. I do not put tags in the\n>> submodule since multiple other products may be using it there and I\n>> wanted to avoid ambiguous tags.\n>>\n>> Sometimes I run into a situation where I need to find out which\n>> release of the product a submodule change was introduced in. This is\n>> nontrivial, since there are no tags in the submodule itself. This is\n>> one thing I tried:\n>>\n>> 1. Do a `git log` in the submodule to find the SHA1 representing the\n>> change I want to check for\n>> 2. In the parent repository, do a git log with pickaxe to determine\n>> when the submodule itself changed to the value of that SHA1.\n>> 3. Based on the result of #2, do a `git tag --contains` to see the\n>> lowest-version tag that contains the SHA1, which will identify the\n>> first release that introduced that change\n>>\n>> However, I was not able to get past #2 because apparently there are\n>> cases where when we move the submodule \"forward\", we skip over\n>> commits, so the value of the submodule itself never was set to that\n>> SHA1.\n>>\n>> I'm at a loss here on how to easily do this. Can someone recommend a\n>> way to do this? Obviously the easier the better, as I have to somehow\n>> train my team how to do this on their own.\n>>\n>> Thanks in advance.\n>\n> I cannot offer an easy way. However I can come up with a proposal\n> how to make this easy in the future. ;)\n>\n> \"git-{branch,tag} --contains\" currently only takes a commit id as that is\n> easy to check for. (Just a revwalk from all commits, as we walk over the\n> commits in the graph)\n>\n> We should extend the possible arguments to --contains, such that you can\n> do\n>\n> # check that a given path had this exact tree/blob id\n> git tag --contains :\n> # check if the given tree/blob was at any path\n> git tag --contains \n> # generalizing from above:\n> git tag --contains [:]\n>\n> With this designed API you could ask for\n>\n> git tag --contains submodule:\n>\n> For the implementation of this feature the revwalk would also need\n> to walk the object graph (as restricted by the pathspec) and\n> see if there is the given object for each tag.\n>\n> Thanks,\n> Stefan\n\nThis sounds useful, but has a limitation in regards to submodules.\nLets say that parent project points submodule commit 1.\n\nIn the submodule, you create commit 2, commit 3, and commit 4.\n\nThen, in the parent project, you new move the submodule forward to commit 4\n\nI think the general goal for submodules is to say \"which parent commit\nincluded this submodule commit\" but the parent never ACTUALLY included\ncommit 3, it only included commit 4 which happens to contain commit 3.\n\nI'm wondering if it might be worth adding an (optional) mode for\nsubmodules which would disallow adding a submodule pointer if the\ncurrent submodule pointer is not an ancestor of the new value. This\nseems like a valuable protection for many uses cases (and preserves\nthe behavior of a bisect to find which commit added something). It\nobviously shouldn't be mandatory since people often re-wind the\nsubmodule pointer. If you have this enabled the only way to rewind the\nsubmodule pointer would be to rewmind the parent history itself.\n\nYou could make the --contains logic above smart enough to try and\ndetect \"ancestor of\" like now, but I think that wouldn't necessarily\nbuy us too much and seems pretty submodule specific.\n\nThanks,\nJake\n"}]}