{"thread":{"id":"52406","subject":"RE: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","startedAt":"2019-12-08T10:31:04Z","lastAt":"2020-03-12T10:40:29Z","messageCount":20,"participants":["Nadav SInai","Strain, Roger L.","Marc Balmer","Johannes Schindelin","Ed Maste","Tom Clarkson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"387712","messageId":"CANxxO2MGJ2Wo6Y-33KzzPXz6vktRACk0Oi2Y6o_s-cDFRhG7+Q@mail.gmail.com","threadId":"52406","inReplyTo":null,"subject":"RE: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Nadav SInai","fromEmail":"ns@nadavsinai.com","sentAt":"2019-12-08T10:30:48Z","receivedAt":"2019-12-08T10:31:04Z","isPatch":false,"sender":{"key":"ns@nadavsinai.com","avatar":null},"body":"Hi, I'm curious if any of you had any luck in preventing that\nseg-fault in git-subtree script\nI'm encountering it myself using git 2.24.0.windows.2., seg-fault is\nin the same while loop (currently on line 757)\nWhen I tried your suggestion of adding the ($parents) ($rev) to the\nprogress print I see that the last commit have only one revision\nprinted\nlike this:\n\n259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n(a69ee056f66acf66c63f89f55d26c0cc17036623)\n259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n(843dd34090d36dfabd6a2e3e8459a4887427313b)\n259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n(f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n259/290 (528) [276]\n(7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n259/290 (529) [277]\n(a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n259/290 (530) [278]\n(90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n259/290 (531) [279]\n(9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n259/290 (532) [280]\n(f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n259/290 (533) [281]\n(c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n259/290 (534) [282]\n(3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n259/290 (535) [283]\n(134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n259/290 (536) [284]\n(edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n259/290 (537) [285]\n(dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n259/290 (538) [286]\n(a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\nC:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n8853 Done eval \"$grl\"\n 8854 Segmentation fault (core dumped) | while read rev parents; do\n process_split_commit \"$rev\" \"$parents\" 0;\ndone\n\nI downgraded git to 2.19.0-windows.1 and it works now.\n\n\nI'm thankful for your insights\nNadav Sinai\nWeb Tech lead\nPhilips-Algotec\n"},{"id":"387778","messageId":"3b9408a9bd87ea488c4a6b9bc2583aba56ce3949.camel@swri.org","threadId":"52406","inReplyTo":"CANxxO2MGJ2Wo6Y-33KzzPXz6vktRACk0Oi2Y6o_s-cDFRhG7+Q@mail.gmail.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Strain, Roger L.","fromEmail":"roger.strain@swri.org","sentAt":"2019-12-09T14:11:03Z","receivedAt":"2019-12-09T14:29:22Z","isPatch":false,"sender":{"key":"roger.strain@swri.org","avatar":"https://avatars.githubusercontent.com/u/52041877?v=4"},"body":"I haven't been able to find anything relating to the issue, but I also\nhaven't had a repo that exposes the problem to test more thoroughly\nagainst. If this happens to be a public repo somewhere, I'd be more\nthan happy to take a second look.\n\nThat being said, if the community feels it would be better to revert\nthe changes that were introduced, I won't object. I've had to further\ncustomize the script for our internal use, and those changes aren't\nsomething that would be useful for the public at large. (A few changes relate to the presence/absence of a specific file, which I certainly wouldn't expect anyone else to have.) Short story is we're going to have to use a custom script going forward, so keeping or reverting the changes here make no difference to us. I still feel that the changes which were made make the script more correct, but clearly there's some undiagnosed logic error somewhere.\n\nHonestly, I'm surprised we didn't see this particular issue show up on\nour own repo; it's ridiculously large and complex. At least if it had,\nI'd be able to troubleshoot it more reliably.\n\n-- \nRoger Strain\n\n-----Original Message-----\nFrom: Nadav SInai \nTo: roger.strain@swri.org\nCc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\nmarc@msys.ch, pclouds@gmail.com\nSubject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n315a84f9aa0e2e629b0680068646b0032518ebed\nDate: Sun, 08 Dec 2019 12:30:48 +0200\n\n[EXTERNAL EMAIL]\n\nHi, I'm curious if any of you had any luck in preventing that\nseg-fault in git-subtree script\nI'm encountering it myself using git 2.24.0.windows.2., seg-fault is\nin the same while loop (currently on line 757)\nWhen I tried your suggestion of adding the ($parents) ($rev) to the\nprogress print I see that the last commit have only one revision\nprinted\nlike this:\n\n259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n(a69ee056f66acf66c63f89f55d26c0cc17036623)\n259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n(843dd34090d36dfabd6a2e3e8459a4887427313b)\n259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n(f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n259/290 (528) [276]\n(7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n259/290 (529) [277]\n(a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n259/290 (530) [278]\n(90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n259/290 (531) [279]\n(9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n259/290 (532) [280]\n(f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n259/290 (533) [281]\n(c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n259/290 (534) [282]\n(3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n259/290 (535) [283]\n(134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n259/290 (536) [284]\n(edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n259/290 (537) [285]\n(dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n259/290 (538) [286]\n(a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\nC:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n8853 Done eval \"$grl\"\n 8854 Segmentation fault (core dumped) | while read rev\nparents; do\n process_split_commit \"$rev\" \"$parents\" 0;\ndone\n\nI downgraded git to 2.19.0-windows.1 and it works now.\n\n\nI'm thankful for your insights\nNadav Sinai\nWeb Tech lead\nPhilips-Algotec\n\n\n"},{"id":"387779","messageId":"1676A670-C8B7-482B-B330-B03B35CB6B5E@msys.ch","threadId":"52406","inReplyTo":"e19d7992acf0efc990aea356716a3afa34e13cb7.camel@swri.org","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Marc Balmer","fromEmail":"marc@msys.ch","sentAt":"2019-12-09T14:30:07Z","receivedAt":"2019-12-09T14:30:19Z","isPatch":false,"sender":{"key":"marc@msys.ch","avatar":"https://gravatar.com/avatar/beb9137fb845e4986641aeb23660b07d3a6dda8a3048d2c9fe6f6e16b5167517?d=mp&s=160"},"body":"I am not familiar with the source code, so I can not send in that revert. I can, however, say that I am grateful to whomever does it ;)\n\n- Marc\n\n\n> Am 09.12.2019 um 15:18 schrieb Strain, Roger L. :\n> \n> As I said, I'm using a custom script here. I don't know if anybody else\n> benefited from the change and hasn't said anything, but I won't object\n> to someone submitting that revert.\n> \n> -- \n> Roger\n> \n> -----Original Message-----\n> From: Marc Balmer \n> To: \"Strain, Roger L.\" \n> Cc: ns@nadavsinai.com , git@vger.kernel.org <\n> git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n> Johannes.Schindelin@gmx.de>, gitster@pobox.com , \n> pclouds@gmail.com \n> Subject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n> 315a84f9aa0e2e629b0680068646b0032518ebed\n> Date: Mon, 09 Dec 2019 15:13:47 +0100\n> \n> Roger,\n> \n> I am all for reverting it. if that does not cause any other regressions\n> or headaches (or both...)\n> \n> - Marc\n> \n> \n> \n> Am 09.12.2019 um 15:11 schrieb Strain, Roger L. \n> :\n> \n> I haven't been able to find anything relating to the issue, but I also\n> haven't had a repo that exposes the problem to test more thoroughly\n> against. If this happens to be a public repo somewhere, I'd be more\n> than happy to take a second look.\n> \n> That being said, if the community feels it would be better to revert\n> the changes that were introduced, I won't object. I've had to further\n> customize the script for our internal use, and those changes aren't\n> something that would be useful for the public at large. (A few changes\n> relate to the presence/absence of a specific file, which I certainly\n> wouldn't expect anyone else to have.) Short story is we're going to\n> have to use a custom script going forward, so keeping or reverting the\n> changes here make no difference to us. I still feel that the changes\n> which were made make the script more correct, but clearly there's some\n> undiagnosed logic error somewhere.\n> \n> Honestly, I'm surprised we didn't see this particular issue show up on\n> our own repo; it's ridiculously large and complex. At least if it had,\n> I'd be able to troubleshoot it more reliably.\n> \n> -- \n> Roger Strain\n> \n> -----Original Message-----\n> From: Nadav SInai \n> To: roger.strain@swri.org\n> Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n> marc@msys.ch, pclouds@gmail.com\n> Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n> 315a84f9aa0e2e629b0680068646b0032518ebed\n> Date: Sun, 08 Dec 2019 12:30:48 +0200\n> \n> [EXTERNAL EMAIL]\n> \n> Hi, I'm curious if any of you had any luck in preventing that\n> seg-fault in git-subtree script\n> I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n> in the same while loop (currently on line 757)\n> When I tried your suggestion of adding the ($parents) ($rev) to the\n> progress print I see that the last commit have only one revision\n> printed\n> like this:\n> \n> 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> (a69ee056f66acf66c63f89f55d26c0cc17036623)\n> 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n> (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> 259/290 (528) [276]\n> (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n> 259/290 (529) [277]\n> (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n> 259/290 (530) [278]\n> (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n> 259/290 (531) [279]\n> (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n> 259/290 (532) [280]\n> (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n> 259/290 (533) [281]\n> (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n> 259/290 (534) [282]\n> (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n> 259/290 (535) [283]\n> (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n> 259/290 (536) [284]\n> (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n> 259/290 (537) [285]\n> (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n> 259/290 (538) [286]\n> (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n> C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n> 8853 Done eval \"$grl\"\n> 8854 Segmentation fault (core dumped) | while read rev\n> parents; do\n> process_split_commit \"$rev\" \"$parents\" 0;\n> done\n> \n> I downgraded git to 2.19.0-windows.1 and it works now.\n> \n> \n> I'm thankful for your insights\n> Nadav Sinai\n> Web Tech lead\n> Philips-Algotec\n> \n> \n> \n> \n> \n\n"},{"id":"387780","messageId":"E8FC0E5F-DD83-4470-B068-21865ECA84D2@msys.ch","threadId":"52406","inReplyTo":"3b9408a9bd87ea488c4a6b9bc2583aba56ce3949.camel@swri.org","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Marc Balmer","fromEmail":"marc@msys.ch","sentAt":"2019-12-09T14:13:47Z","receivedAt":"2019-12-09T14:36:52Z","isPatch":false,"sender":{"key":"marc@msys.ch","avatar":"https://gravatar.com/avatar/beb9137fb845e4986641aeb23660b07d3a6dda8a3048d2c9fe6f6e16b5167517?d=mp&s=160"},"body":"Roger,\n\nI am all for reverting it. if that does not cause any other regressions or headaches (or both...)\n\n- Marc\n\n\n> Am 09.12.2019 um 15:11 schrieb Strain, Roger L. :\n> \n> I haven't been able to find anything relating to the issue, but I also\n> haven't had a repo that exposes the problem to test more thoroughly\n> against. If this happens to be a public repo somewhere, I'd be more\n> than happy to take a second look.\n> \n> That being said, if the community feels it would be better to revert\n> the changes that were introduced, I won't object. I've had to further\n> customize the script for our internal use, and those changes aren't\n> something that would be useful for the public at large. (A few changes relate to the presence/absence of a specific file, which I certainly wouldn't expect anyone else to have.) Short story is we're going to have to use a custom script going forward, so keeping or reverting the changes here make no difference to us. I still feel that the changes which were made make the script more correct, but clearly there's some undiagnosed logic error somewhere.\n> \n> Honestly, I'm surprised we didn't see this particular issue show up on\n> our own repo; it's ridiculously large and complex. At least if it had,\n> I'd be able to troubleshoot it more reliably.\n> \n> -- \n> Roger Strain\n> \n> -----Original Message-----\n> From: Nadav SInai \n> To: roger.strain@swri.org\n> Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n> marc@msys.ch, pclouds@gmail.com\n> Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n> 315a84f9aa0e2e629b0680068646b0032518ebed\n> Date: Sun, 08 Dec 2019 12:30:48 +0200\n> \n> [EXTERNAL EMAIL]\n> \n> Hi, I'm curious if any of you had any luck in preventing that\n> seg-fault in git-subtree script\n> I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n> in the same while loop (currently on line 757)\n> When I tried your suggestion of adding the ($parents) ($rev) to the\n> progress print I see that the last commit have only one revision\n> printed\n> like this:\n> \n> 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> (a69ee056f66acf66c63f89f55d26c0cc17036623)\n> 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n> (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> 259/290 (528) [276]\n> (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n> 259/290 (529) [277]\n> (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n> 259/290 (530) [278]\n> (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n> 259/290 (531) [279]\n> (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n> 259/290 (532) [280]\n> (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n> 259/290 (533) [281]\n> (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n> 259/290 (534) [282]\n> (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n> 259/290 (535) [283]\n> (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n> 259/290 (536) [284]\n> (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n> 259/290 (537) [285]\n> (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n> 259/290 (538) [286]\n> (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n> C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n> 8853 Done eval \"$grl\"\n> 8854 Segmentation fault (core dumped) | while read rev\n> parents; do\n> process_split_commit \"$rev\" \"$parents\" 0;\n> done\n> \n> I downgraded git to 2.19.0-windows.1 and it works now.\n> \n> \n> I'm thankful for your insights\n> Nadav Sinai\n> Web Tech lead\n> Philips-Algotec\n> \n> \n\n"},{"id":"387785","messageId":"e19d7992acf0efc990aea356716a3afa34e13cb7.camel@swri.org","threadId":"52406","inReplyTo":"E8FC0E5F-DD83-4470-B068-21865ECA84D2@msys.ch","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Strain, Roger L.","fromEmail":"roger.strain@swri.org","sentAt":"2019-12-09T14:18:43Z","receivedAt":"2019-12-09T15:20:23Z","isPatch":false,"sender":{"key":"roger.strain@swri.org","avatar":"https://avatars.githubusercontent.com/u/52041877?v=4"},"body":"As I said, I'm using a custom script here. I don't know if anybody else\nbenefited from the change and hasn't said anything, but I won't object\nto someone submitting that revert.\n\n-- \nRoger\n\n-----Original Message-----\nFrom: Marc Balmer \nTo: \"Strain, Roger L.\" \nCc: ns@nadavsinai.com , git@vger.kernel.org <\ngit@vger.kernel.org>, Johannes.Schindelin@gmx.de <\nJohannes.Schindelin@gmx.de>, gitster@pobox.com , \npclouds@gmail.com \nSubject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n315a84f9aa0e2e629b0680068646b0032518ebed\nDate: Mon, 09 Dec 2019 15:13:47 +0100\n\nRoger,\n\nI am all for reverting it. if that does not cause any other regressions\nor headaches (or both...)\n\n- Marc\n\n\n\nAm 09.12.2019 um 15:11 schrieb Strain, Roger L. \n:\n\nI haven't been able to find anything relating to the issue, but I also\nhaven't had a repo that exposes the problem to test more thoroughly\nagainst. If this happens to be a public repo somewhere, I'd be more\nthan happy to take a second look.\n\nThat being said, if the community feels it would be better to revert\nthe changes that were introduced, I won't object. I've had to further\ncustomize the script for our internal use, and those changes aren't\nsomething that would be useful for the public at large. (A few changes\nrelate to the presence/absence of a specific file, which I certainly\nwouldn't expect anyone else to have.) Short story is we're going to\nhave to use a custom script going forward, so keeping or reverting the\nchanges here make no difference to us. I still feel that the changes\nwhich were made make the script more correct, but clearly there's some\nundiagnosed logic error somewhere.\n\nHonestly, I'm surprised we didn't see this particular issue show up on\nour own repo; it's ridiculously large and complex. At least if it had,\nI'd be able to troubleshoot it more reliably.\n\n-- \nRoger Strain\n\n-----Original Message-----\nFrom: Nadav SInai \nTo: roger.strain@swri.org\nCc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\nmarc@msys.ch, pclouds@gmail.com\nSubject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n315a84f9aa0e2e629b0680068646b0032518ebed\nDate: Sun, 08 Dec 2019 12:30:48 +0200\n\n[EXTERNAL EMAIL]\n\nHi, I'm curious if any of you had any luck in preventing that\nseg-fault in git-subtree script\nI'm encountering it myself using git 2.24.0.windows.2., seg-fault is\nin the same while loop (currently on line 757)\nWhen I tried your suggestion of adding the ($parents) ($rev) to the\nprogress print I see that the last commit have only one revision\nprinted\nlike this:\n\n259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n(a69ee056f66acf66c63f89f55d26c0cc17036623)\n259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n(843dd34090d36dfabd6a2e3e8459a4887427313b)\n259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n(f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n259/290 (528) [276]\n(7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n259/290 (529) [277]\n(a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n259/290 (530) [278]\n(90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n259/290 (531) [279]\n(9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n259/290 (532) [280]\n(f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n259/290 (533) [281]\n(c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n259/290 (534) [282]\n(3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n259/290 (535) [283]\n(134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n259/290 (536) [284]\n(edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n259/290 (537) [285]\n(dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n259/290 (538) [286]\n(a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\nC:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n8853 Done eval \"$grl\"\n 8854 Segmentation fault (core dumped) | while read rev\nparents; do\n process_split_commit \"$rev\" \"$parents\" 0;\ndone\n\nI downgraded git to 2.19.0-windows.1 and it works now.\n\n\nI'm thankful for your insights\nNadav Sinai\nWeb Tech lead\nPhilips-Algotec\n\n\n\n\n\n"},{"id":"387787","messageId":"nycvar.QRO.7.76.6.1912091615200.31080@tvgsbejvaqbjf.bet","threadId":"52406","inReplyTo":"1676A670-C8B7-482B-B330-B03B35CB6B5E@msys.ch","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-09T15:26:31Z","receivedAt":"2019-12-09T15:27:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 9 Dec 2019, Marc Balmer wrote:\n\n> I am not familiar with the source code, so I can not send in that\n> revert. I can, however, say that I am grateful to whomever does it ;)\n\nI am against reverting the change without knowing the root cause.\n\nThe recent reporter only compared Git for Windows v2.19.0 vs v2.20.1,\nwhich is _quite_ a big difference.\n\nFor what I know, the problem might be a change in the MSYS2 runtime that\nis mistaken by some malware for malicious code (we did introduce some code\nto emulate Ctrl+C in MinTTY which injects a remote thread and executes\nExitProcess() there, which might very well be construed as an attack, even\nif it is actually very much desired behavior).\n\nThese segmentation faults in `git subtree` on Windows have traditionally\nbeen _all_ because of overzealous anti-malware.\n\nSo first, a much more fine-grained analysis would be required, e.g.\ncomparing v2.20.1 against v2.20.0, then copying _just_ the `git-subtree`\nfile from a working into a non-working version (or vice versa; I would\nhighly recommend using the portable versions for such side-by side\ncomparison).\n\nCiao,\nJohannes\n\n>\n> - Marc\n>\n>\n> > Am 09.12.2019 um 15:18 schrieb Strain, Roger L. :\n> >\n> > As I said, I'm using a custom script here. I don't know if anybody else\n> > benefited from the change and hasn't said anything, but I won't object\n> > to someone submitting that revert.\n> >\n> > --\n> > Roger\n> >\n> > -----Original Message-----\n> > From: Marc Balmer \n> > To: \"Strain, Roger L.\" \n> > Cc: ns@nadavsinai.com , git@vger.kernel.org <\n> > git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n> > Johannes.Schindelin@gmx.de>, gitster@pobox.com ,\n> > pclouds@gmail.com \n> > Subject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n> > 315a84f9aa0e2e629b0680068646b0032518ebed\n> > Date: Mon, 09 Dec 2019 15:13:47 +0100\n> >\n> > Roger,\n> >\n> > I am all for reverting it. if that does not cause any other regressions\n> > or headaches (or both...)\n> >\n> > - Marc\n> >\n> >\n> >\n> > Am 09.12.2019 um 15:11 schrieb Strain, Roger L. \n> > :\n> >\n> > I haven't been able to find anything relating to the issue, but I also\n> > haven't had a repo that exposes the problem to test more thoroughly\n> > against. If this happens to be a public repo somewhere, I'd be more\n> > than happy to take a second look.\n> >\n> > That being said, if the community feels it would be better to revert\n> > the changes that were introduced, I won't object. I've had to further\n> > customize the script for our internal use, and those changes aren't\n> > something that would be useful for the public at large. (A few changes\n> > relate to the presence/absence of a specific file, which I certainly\n> > wouldn't expect anyone else to have.) Short story is we're going to\n> > have to use a custom script going forward, so keeping or reverting the\n> > changes here make no difference to us. I still feel that the changes\n> > which were made make the script more correct, but clearly there's some\n> > undiagnosed logic error somewhere.\n> >\n> > Honestly, I'm surprised we didn't see this particular issue show up on\n> > our own repo; it's ridiculously large and complex. At least if it had,\n> > I'd be able to troubleshoot it more reliably.\n> >\n> > --\n> > Roger Strain\n> >\n> > -----Original Message-----\n> > From: Nadav SInai \n> > To: roger.strain@swri.org\n> > Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n> > marc@msys.ch, pclouds@gmail.com\n> > Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n> > 315a84f9aa0e2e629b0680068646b0032518ebed\n> > Date: Sun, 08 Dec 2019 12:30:48 +0200\n> >\n> > [EXTERNAL EMAIL]\n> >\n> > Hi, I'm curious if any of you had any luck in preventing that\n> > seg-fault in git-subtree script\n> > I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n> > in the same while loop (currently on line 757)\n> > When I tried your suggestion of adding the ($parents) ($rev) to the\n> > progress print I see that the last commit have only one revision\n> > printed\n> > like this:\n> >\n> > 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> > (a69ee056f66acf66c63f89f55d26c0cc17036623)\n> > 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> > (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> > 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n> > (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> > 259/290 (528) [276]\n> > (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n> > 259/290 (529) [277]\n> > (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n> > 259/290 (530) [278]\n> > (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n> > 259/290 (531) [279]\n> > (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n> > 259/290 (532) [280]\n> > (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n> > 259/290 (533) [281]\n> > (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n> > 259/290 (534) [282]\n> > (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n> > 259/290 (535) [283]\n> > (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n> > 259/290 (536) [284]\n> > (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n> > 259/290 (537) [285]\n> > (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n> > 259/290 (538) [286]\n> > (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n> > C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n> > 8853 Done eval \"$grl\"\n> > 8854 Segmentation fault (core dumped) | while read rev\n> > parents; do\n> > process_split_commit \"$rev\" \"$parents\" 0;\n> > done\n> >\n> > I downgraded git to 2.19.0-windows.1 and it works now.\n> >\n> >\n> > I'm thankful for your insights\n> > Nadav Sinai\n> > Web Tech lead\n> > Philips-Algotec\n> >\n> >\n> >\n> >\n> >\n>\n>\n"},{"id":"387788","messageId":"7E95BE86-BD96-482F-9ECA-DBDD9C10D114@msys.ch","threadId":"52406","inReplyTo":"nycvar.QRO.7.76.6.1912091615200.31080@tvgsbejvaqbjf.bet","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Marc Balmer","fromEmail":"marc@msys.ch","sentAt":"2019-12-09T15:31:44Z","receivedAt":"2019-12-09T15:31:59Z","isPatch":false,"sender":{"key":"marc@msys.ch","avatar":"https://gravatar.com/avatar/beb9137fb845e4986641aeb23660b07d3a6dda8a3048d2c9fe6f6e16b5167517?d=mp&s=160"},"body":"Fwiw, I see the problem on Linux.\n\nIt hay nothing to do with overzealos antimalware, it is a regression and it has been well documented.\n\n\n> Am 09.12.2019 um 17:20 schrieb Johannes Schindelin :\n> \n> Hi,\n> \n>> On Mon, 9 Dec 2019, Marc Balmer wrote:\n>> \n>> I am not familiar with the source code, so I can not send in that\n>> revert. I can, however, say that I am grateful to whomever does it ;)\n> \n> I am against reverting the change without knowing the root cause.\n> \n> The recent reporter only compared Git for Windows v2.19.0 vs v2.20.1,\n> which is _quite_ a big difference.\n> \n> For what I know, the problem might be a change in the MSYS2 runtime that\n> is mistaken by some malware for malicious code (we did introduce some code\n> to emulate Ctrl+C in MinTTY which injects a remote thread and executes\n> ExitProcess() there, which might very well be construed as an attack, even\n> if it is actually very much desired behavior).\n> \n> These segmentation faults in `git subtree` on Windows have traditionally\n> been _all_ because of overzealous anti-malware.\n> \n> So first, a much more fine-grained analysis would be required, e.g.\n> comparing v2.20.1 against v2.20.0, then copying _just_ the `git-subtree`\n> file from a working into a non-working version (or vice versa; I would\n> highly recommend using the portable versions for such side-by side\n> comparison).\n> \n> Ciao,\n> Johannes\n> \n>> \n>> - Marc\n>> \n>> \n>>>> Am 09.12.2019 um 15:18 schrieb Strain, Roger L. :\n>>> \n>>> As I said, I'm using a custom script here. I don't know if anybody else\n>>> benefited from the change and hasn't said anything, but I won't object\n>>> to someone submitting that revert.\n>>> \n>>> --\n>>> Roger\n>>> \n>>> -----Original Message-----\n>>> From: Marc Balmer \n>>> To: \"Strain, Roger L.\" \n>>> Cc: ns@nadavsinai.com , git@vger.kernel.org <\n>>> git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n>>> Johannes.Schindelin@gmx.de>, gitster@pobox.com ,\n>>> pclouds@gmail.com \n>>> Subject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n>>> 315a84f9aa0e2e629b0680068646b0032518ebed\n>>> Date: Mon, 09 Dec 2019 15:13:47 +0100\n>>> \n>>> Roger,\n>>> \n>>> I am all for reverting it. if that does not cause any other regressions\n>>> or headaches (or both...)\n>>> \n>>> - Marc\n>>> \n>>> \n>>> \n>>> Am 09.12.2019 um 15:11 schrieb Strain, Roger L. \n>>> :\n>>> \n>>> I haven't been able to find anything relating to the issue, but I also\n>>> haven't had a repo that exposes the problem to test more thoroughly\n>>> against. If this happens to be a public repo somewhere, I'd be more\n>>> than happy to take a second look.\n>>> \n>>> That being said, if the community feels it would be better to revert\n>>> the changes that were introduced, I won't object. I've had to further\n>>> customize the script for our internal use, and those changes aren't\n>>> something that would be useful for the public at large. (A few changes\n>>> relate to the presence/absence of a specific file, which I certainly\n>>> wouldn't expect anyone else to have.) Short story is we're going to\n>>> have to use a custom script going forward, so keeping or reverting the\n>>> changes here make no difference to us. I still feel that the changes\n>>> which were made make the script more correct, but clearly there's some\n>>> undiagnosed logic error somewhere.\n>>> \n>>> Honestly, I'm surprised we didn't see this particular issue show up on\n>>> our own repo; it's ridiculously large and complex. At least if it had,\n>>> I'd be able to troubleshoot it more reliably.\n>>> \n>>> --\n>>> Roger Strain\n>>> \n>>> -----Original Message-----\n>>> From: Nadav SInai \n>>> To: roger.strain@swri.org\n>>> Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n>>> marc@msys.ch, pclouds@gmail.com\n>>> Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n>>> 315a84f9aa0e2e629b0680068646b0032518ebed\n>>> Date: Sun, 08 Dec 2019 12:30:48 +0200\n>>> \n>>> [EXTERNAL EMAIL]\n>>> \n>>> Hi, I'm curious if any of you had any luck in preventing that\n>>> seg-fault in git-subtree script\n>>> I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n>>> in the same while loop (currently on line 757)\n>>> When I tried your suggestion of adding the ($parents) ($rev) to the\n>>> progress print I see that the last commit have only one revision\n>>> printed\n>>> like this:\n>>> \n>>> 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n>>> (a69ee056f66acf66c63f89f55d26c0cc17036623)\n>>> 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n>>> (843dd34090d36dfabd6a2e3e8459a4887427313b)\n>>> 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n>>> (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n>>> 259/290 (528) [276]\n>>> (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n>>> 259/290 (529) [277]\n>>> (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n>>> 259/290 (530) [278]\n>>> (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n>>> 259/290 (531) [279]\n>>> (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n>>> 259/290 (532) [280]\n>>> (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n>>> 259/290 (533) [281]\n>>> (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n>>> 259/290 (534) [282]\n>>> (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n>>> 259/290 (535) [283]\n>>> (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n>>> 259/290 (536) [284]\n>>> (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n>>> 259/290 (537) [285]\n>>> (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n>>> 259/290 (538) [286]\n>>> (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n>>> C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n>>> 8853 Done eval \"$grl\"\n>>> 8854 Segmentation fault (core dumped) | while read rev\n>>> parents; do\n>>> process_split_commit \"$rev\" \"$parents\" 0;\n>>> done\n>>> \n>>> I downgraded git to 2.19.0-windows.1 and it works now.\n>>> \n>>> \n>>> I'm thankful for your insights\n>>> Nadav Sinai\n>>> Web Tech lead\n>>> Philips-Algotec\n>>> \n>>> \n>>> \n>>> \n>>> \n>> \n>> \n\n"},{"id":"387789","messageId":"CAPyFy2BjWx2Hp+H__kDFRFZZjcK4hc99oKqkZQjXPLfQE=2SPg@mail.gmail.com","threadId":"52406","inReplyTo":"3b9408a9bd87ea488c4a6b9bc2583aba56ce3949.camel@swri.org","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2019-12-09T11:45:57Z","receivedAt":"2019-12-09T15:32:28Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Mon, 9 Dec 2019 at 09:29, Strain, Roger L. wrote:\n>\n> I've had to further\n> customize the script for our internal use, and those changes aren't\n> something that would be useful for the public at large.\n\nWould you describe the sort of problem you have to work around with\ncustom changes?\n\nI'm starting on a path of trying to fix git-subtree for failures[1]\nencountered in a prototype conversion of the FreeBSD repository from\nsvn to git. The misbehaviour I encounter occurs when split encounters\na commit for which the path being split is empty in 'git ls-tree', and\nthe commit is actually not a subtree commit. I'm currently\nexperimenting with hacks to skip specific hashes during the initial\nsubtree split. On reading your mail I realize I could address my issue\nby testing for the existence of a specific file though, which makes me\nwonder if the issue you have is similar.\n\n[1] https://lore.kernel.org/git/CAPyFy2AsmaxU-BDf_teZJE5hiaVpTSZc8fftnuXPb_4-j7j5Fw@mail.gmail.com/\n"},{"id":"387804","messageId":"1c62a65d7effb49c6bd258f462f34221ac1c7ff1.camel@swri.org","threadId":"52406","inReplyTo":"CAPyFy2BjWx2Hp+H__kDFRFZZjcK4hc99oKqkZQjXPLfQE=2SPg@mail.gmail.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Strain, Roger L.","fromEmail":"roger.strain@swri.org","sentAt":"2019-12-09T16:19:09Z","receivedAt":"2019-12-09T16:19:18Z","isPatch":false,"sender":{"key":"roger.strain@swri.org","avatar":"https://avatars.githubusercontent.com/u/52041877?v=4"},"body":"So it's been quite a while since I made this specific change, but I'll\nattach the relevant portion of the diff below. I may be completely\nmisremembering portions, and apologize in advance. This was based on an\nearlier version of the script, and I can see some other changes have\nbeen made since I forked, but perhaps this will still explain what I\ntried to do to work around our problem.\n\nWithin process_split_commit, there's logic that tries to distinguish\nbetween commits which are mainline and commits which are subtree.\nThere's even a comment in the relevant section asking \"Is there no\nbetter way? Does it matter?\" Well, the answer was yes, it mattered,\nbecause we were picking up mainline commits that there before the\ninitial add of a subtree, and those were getting sucked in as if they\nwere subtree commits, and then all the remaining hashes were off.\n\nWhat this change was meant to do was to check for the existence of a\nsingle, known file. We keep a file called \"subtrees.csv\" in the root of\nour mainline repo, and it defines the various subtrees that comprise\nthe mainline. Therefore, if that file exists, I can say with certainty\nthat it is a mainline commit. So when that dodgy check comes up, it\nchecks for the file first, then falls back to the old behavior.\n\nPartial diff follows, feel free to try it out if it sounds like a\nsimilar problem that you're facing. Change the specific filename for\nyour needs, obviously.\n\nTo be clear, this is NOT something I'm submitting for inclusion in the\ngeneral release; it's very repo-specific, and I just hope it might help\na fellow soul.\n\n@@ -506,6 +499,20 @@ subtree_for_commit () {\n done\n }\n \n+subtree_for_csv () {\n+ commit=\"$1\"\n+ dir=\"$2\"\n+ git ls-tree \"$commit\" -- \"$dir\" |\n+ while read mode type tree name\n+ do\n+ assert test \"$name\" = \"$dir\"\n+ assert test \"$type\" = \"blob\" -o \"$type\" = \"commit\"\n+ test \"$type\" = \"commit\" && continue # ignore\nsubmodules\n+ echo $tree\n+ break\n+ done\n+}\n+\n tree_changed () {\n tree=$1\n shift\n@@ -667,9 +674,17 @@ process_split_commit () {\n if test -z \"$tree\"\n then\n set_notree \"$rev\"\n- if test -n \"$newparents\"\n+ subtreescsv=$(subtree_for_csv \"$rev\" \"subtrees.csv\")\n+ debug \"${indentprefix} subtrees.csv tree is:\n$subtreescsv\"\n+\n+ # ugly. is there no better way to tell if this is a\nsubtree\n+ # vs. a mainline commit? Does it matter?\n+ if test -z \"$subtreescsv\"\n then\n- cache_set \"$rev\" \"$rev\"\n+ if test -n \"$newparents\"\n+ then\n+ cache_set \"$rev\" \"$rev\"\n+ fi\n fi\n return\n fi\n\n\n-- \nRoger Strain\n\n-----Original Message-----\nFrom: Ed Maste \nTo: \"Strain, Roger L.\" \nCc: git@vger.kernel.org , marc@msys.ch <\nmarc@msys.ch>\nSubject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n315a84f9aa0e2e629b0680068646b0032518ebed\nDate: Mon, 09 Dec 2019 06:45:57 -0500\n\n[EXTERNAL EMAIL]\n\nOn Mon, 9 Dec 2019 at 09:29, Strain, Roger L. \nwrote:\n\nI've had to further\ncustomize the script for our internal use, and those changes aren't\nsomething that would be useful for the public at large.\n\nWould you describe the sort of problem you have to work around with\ncustom changes?\n\nI'm starting on a path of trying to fix git-subtree for failures[1]\nencountered in a prototype conversion of the FreeBSD repository from\nsvn to git. The misbehaviour I encounter occurs when split encounters\na commit for which the path being split is empty in 'git ls-tree', and\nthe commit is actually not a subtree commit. I'm currently\nexperimenting with hacks to skip specific hashes during the initial\nsubtree split. On reading your mail I realize I could address my issue\nby testing for the existence of a specific file though, which makes me\nwonder if the issue you have is similar.\n\n[1] \nhttps://lore.kernel.org/git/CAPyFy2AsmaxU-BDf_teZJE5hiaVpTSZc8fftnuXPb_4-j7j5Fw@mail.gmail.com/\n\n\n"},{"id":"387824","messageId":"nycvar.QRO.7.76.6.1912092037540.31080@tvgsbejvaqbjf.bet","threadId":"52406","inReplyTo":"7E95BE86-BD96-482F-9ECA-DBDD9C10D114@msys.ch","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-09T19:38:58Z","receivedAt":"2019-12-09T19:39:30Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Marc,\n\nOn Mon, 9 Dec 2019, Marc Balmer wrote:\n\n> Fwiw, I see the problem on Linux.\n\nOkay, I must have overlooked that piece of information.\n\n> It hay nothing to do with overzealos antimalware, it is a regression and\n> it has been well documented.\n\nIs there a minimal, complete and verifiable example that other developers\ncould use to analyze the bug?\n\nCiao,\nJohannes\n\n>\n>\n> > Am 09.12.2019 um 17:20 schrieb Johannes Schindelin :\n> >\n> > Hi,\n> >\n> >> On Mon, 9 Dec 2019, Marc Balmer wrote:\n> >>\n> >> I am not familiar with the source code, so I can not send in that\n> >> revert. I can, however, say that I am grateful to whomever does it ;)\n> >\n> > I am against reverting the change without knowing the root cause.\n> >\n> > The recent reporter only compared Git for Windows v2.19.0 vs v2.20.1,\n> > which is _quite_ a big difference.\n> >\n> > For what I know, the problem might be a change in the MSYS2 runtime that\n> > is mistaken by some malware for malicious code (we did introduce some code\n> > to emulate Ctrl+C in MinTTY which injects a remote thread and executes\n> > ExitProcess() there, which might very well be construed as an attack, even\n> > if it is actually very much desired behavior).\n> >\n> > These segmentation faults in `git subtree` on Windows have traditionally\n> > been _all_ because of overzealous anti-malware.\n> >\n> > So first, a much more fine-grained analysis would be required, e.g.\n> > comparing v2.20.1 against v2.20.0, then copying _just_ the `git-subtree`\n> > file from a working into a non-working version (or vice versa; I would\n> > highly recommend using the portable versions for such side-by side\n> > comparison).\n> >\n> > Ciao,\n> > Johannes\n> >\n> >>\n> >> - Marc\n> >>\n> >>\n> >>>> Am 09.12.2019 um 15:18 schrieb Strain, Roger L. :\n> >>>\n> >>> As I said, I'm using a custom script here. I don't know if anybody else\n> >>> benefited from the change and hasn't said anything, but I won't object\n> >>> to someone submitting that revert.\n> >>>\n> >>> --\n> >>> Roger\n> >>>\n> >>> -----Original Message-----\n> >>> From: Marc Balmer \n> >>> To: \"Strain, Roger L.\" \n> >>> Cc: ns@nadavsinai.com , git@vger.kernel.org <\n> >>> git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n> >>> Johannes.Schindelin@gmx.de>, gitster@pobox.com ,\n> >>> pclouds@gmail.com \n> >>> Subject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n> >>> 315a84f9aa0e2e629b0680068646b0032518ebed\n> >>> Date: Mon, 09 Dec 2019 15:13:47 +0100\n> >>>\n> >>> Roger,\n> >>>\n> >>> I am all for reverting it. if that does not cause any other regressions\n> >>> or headaches (or both...)\n> >>>\n> >>> - Marc\n> >>>\n> >>>\n> >>>\n> >>> Am 09.12.2019 um 15:11 schrieb Strain, Roger L. \n> >>> :\n> >>>\n> >>> I haven't been able to find anything relating to the issue, but I also\n> >>> haven't had a repo that exposes the problem to test more thoroughly\n> >>> against. If this happens to be a public repo somewhere, I'd be more\n> >>> than happy to take a second look.\n> >>>\n> >>> That being said, if the community feels it would be better to revert\n> >>> the changes that were introduced, I won't object. I've had to further\n> >>> customize the script for our internal use, and those changes aren't\n> >>> something that would be useful for the public at large. (A few changes\n> >>> relate to the presence/absence of a specific file, which I certainly\n> >>> wouldn't expect anyone else to have.) Short story is we're going to\n> >>> have to use a custom script going forward, so keeping or reverting the\n> >>> changes here make no difference to us. I still feel that the changes\n> >>> which were made make the script more correct, but clearly there's some\n> >>> undiagnosed logic error somewhere.\n> >>>\n> >>> Honestly, I'm surprised we didn't see this particular issue show up on\n> >>> our own repo; it's ridiculously large and complex. At least if it had,\n> >>> I'd be able to troubleshoot it more reliably.\n> >>>\n> >>> --\n> >>> Roger Strain\n> >>>\n> >>> -----Original Message-----\n> >>> From: Nadav SInai \n> >>> To: roger.strain@swri.org\n> >>> Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n> >>> marc@msys.ch, pclouds@gmail.com\n> >>> Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n> >>> 315a84f9aa0e2e629b0680068646b0032518ebed\n> >>> Date: Sun, 08 Dec 2019 12:30:48 +0200\n> >>>\n> >>> [EXTERNAL EMAIL]\n> >>>\n> >>> Hi, I'm curious if any of you had any luck in preventing that\n> >>> seg-fault in git-subtree script\n> >>> I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n> >>> in the same while loop (currently on line 757)\n> >>> When I tried your suggestion of adding the ($parents) ($rev) to the\n> >>> progress print I see that the last commit have only one revision\n> >>> printed\n> >>> like this:\n> >>>\n> >>> 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> >>> (a69ee056f66acf66c63f89f55d26c0cc17036623)\n> >>> 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> >>> (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> >>> 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n> >>> (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> >>> 259/290 (528) [276]\n> >>> (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n> >>> 259/290 (529) [277]\n> >>> (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n> >>> 259/290 (530) [278]\n> >>> (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n> >>> 259/290 (531) [279]\n> >>> (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n> >>> 259/290 (532) [280]\n> >>> (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n> >>> 259/290 (533) [281]\n> >>> (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n> >>> 259/290 (534) [282]\n> >>> (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n> >>> 259/290 (535) [283]\n> >>> (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n> >>> 259/290 (536) [284]\n> >>> (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n> >>> 259/290 (537) [285]\n> >>> (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n> >>> 259/290 (538) [286]\n> >>> (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n> >>> C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n> >>> 8853 Done eval \"$grl\"\n> >>> 8854 Segmentation fault (core dumped) | while read rev\n> >>> parents; do\n> >>> process_split_commit \"$rev\" \"$parents\" 0;\n> >>> done\n> >>>\n> >>> I downgraded git to 2.19.0-windows.1 and it works now.\n> >>>\n> >>>\n> >>> I'm thankful for your insights\n> >>> Nadav Sinai\n> >>> Web Tech lead\n> >>> Philips-Algotec\n> >>>\n> >>>\n> >>>\n> >>>\n> >>>\n> >>\n> >>\n>\n>\n"},{"id":"387931","messageId":"D99ED706-EC49-4A52-8186-5C9B0B5BC744@icloud.com","threadId":"52406","inReplyTo":"nycvar.QRO.7.76.6.1912092037540.31080@tvgsbejvaqbjf.bet","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-11T05:43:22Z","receivedAt":"2019-12-11T05:51:56Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n> Is there a minimal, complete and verifiable example that other developers\n> could use to analyze the bug?\n\nI ran into this bug today, and while not much closer to a solution, I believe I understand why it is happening. \n\nThe recursive search is required because the original rev-list based approach could leave out some relevant commits - ie reversion would replace an obvious bug with a hidden one.\n\nThe search will stop when it reaches either a root commit or one already mapped to a subtree commit.\n\nIf you have a small repo or run git subtree add directly on your master branch, the search will terminate fairly quickly.\n\nHowever, if you do everything via pull requests, the search will hit a merge commit where one side is just ahead of the subtree mapping, but the other side is several thousand commits with no sign of either a root or any subtrees.\n\nI’m not sure yet if it is the number of commits or merges specifically, but the script seems to be able to handle around 400-500 commits before it falls over.\n\n>> \n>> \n>>> Am 09.12.2019 um 17:20 schrieb Johannes Schindelin :\n>>> \n>>> Hi,\n>>> \n>>>> On Mon, 9 Dec 2019, Marc Balmer wrote:\n>>>> \n>>>> I am not familiar with the source code, so I can not send in that\n>>>> revert. I can, however, say that I am grateful to whomever does it ;)\n>>> \n>>> I am against reverting the change without knowing the root cause.\n>>> \n>>> The recent reporter only compared Git for Windows v2.19.0 vs v2.20.1,\n>>> which is _quite_ a big difference.\n>>> \n>>> For what I know, the problem might be a change in the MSYS2 runtime that\n>>> is mistaken by some malware for malicious code (we did introduce some code\n>>> to emulate Ctrl+C in MinTTY which injects a remote thread and executes\n>>> ExitProcess() there, which might very well be construed as an attack, even\n>>> if it is actually very much desired behavior).\n>>> \n>>> These segmentation faults in `git subtree` on Windows have traditionally\n>>> been _all_ because of overzealous anti-malware.\n>>> \n>>> So first, a much more fine-grained analysis would be required, e.g.\n>>> comparing v2.20.1 against v2.20.0, then copying _just_ the `git-subtree`\n>>> file from a working into a non-working version (or vice versa; I would\n>>> highly recommend using the portable versions for such side-by side\n>>> comparison).\n>>> \n>>> Ciao,\n>>> Johannes\n>>> \n>>>> \n>>>> - Marc\n>>>> \n>>>> \n>>>>>> Am 09.12.2019 um 15:18 schrieb Strain, Roger L. :\n>>>>> \n>>>>> As I said, I'm using a custom script here. I don't know if anybody else\n>>>>> benefited from the change and hasn't said anything, but I won't object\n>>>>> to someone submitting that revert.\n>>>>> \n>>>>> --\n>>>>> Roger\n>>>>> \n>>>>> -----Original Message-----\n>>>>> From: Marc Balmer \n>>>>> To: \"Strain, Roger L.\" \n>>>>> Cc: ns@nadavsinai.com , git@vger.kernel.org <\n>>>>> git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n>>>>> Johannes.Schindelin@gmx.de>, gitster@pobox.com ,\n>>>>> pclouds@gmail.com \n>>>>> Subject: Re: Regression in git-subtree.sh, introduced in 2.20.1, after\n>>>>> 315a84f9aa0e2e629b0680068646b0032518ebed\n>>>>> Date: Mon, 09 Dec 2019 15:13:47 +0100\n>>>>> \n>>>>> Roger,\n>>>>> \n>>>>> I am all for reverting it. if that does not cause any other regressions\n>>>>> or headaches (or both...)\n>>>>> \n>>>>> - Marc\n>>>>> \n>>>>> \n>>>>> \n>>>>> Am 09.12.2019 um 15:11 schrieb Strain, Roger L. \n>>>>> :\n>>>>> \n>>>>> I haven't been able to find anything relating to the issue, but I also\n>>>>> haven't had a repo that exposes the problem to test more thoroughly\n>>>>> against. If this happens to be a public repo somewhere, I'd be more\n>>>>> than happy to take a second look.\n>>>>> \n>>>>> That being said, if the community feels it would be better to revert\n>>>>> the changes that were introduced, I won't object. I've had to further\n>>>>> customize the script for our internal use, and those changes aren't\n>>>>> something that would be useful for the public at large. (A few changes\n>>>>> relate to the presence/absence of a specific file, which I certainly\n>>>>> wouldn't expect anyone else to have.) Short story is we're going to\n>>>>> have to use a custom script going forward, so keeping or reverting the\n>>>>> changes here make no difference to us. I still feel that the changes\n>>>>> which were made make the script more correct, but clearly there's some\n>>>>> undiagnosed logic error somewhere.\n>>>>> \n>>>>> Honestly, I'm surprised we didn't see this particular issue show up on\n>>>>> our own repo; it's ridiculously large and complex. At least if it had,\n>>>>> I'd be able to troubleshoot it more reliably.\n>>>>> \n>>>>> --\n>>>>> Roger Strain\n>>>>> \n>>>>> -----Original Message-----\n>>>>> From: Nadav SInai \n>>>>> To: roger.strain@swri.org\n>>>>> Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, gitster@pobox.com,\n>>>>> marc@msys.ch, pclouds@gmail.com\n>>>>> Subject: RE: Regression in git-subtree.sh, introduced in 2.20.1, after\n>>>>> 315a84f9aa0e2e629b0680068646b0032518ebed\n>>>>> Date: Sun, 08 Dec 2019 12:30:48 +0200\n>>>>> \n>>>>> [EXTERNAL EMAIL]\n>>>>> \n>>>>> Hi, I'm curious if any of you had any luck in preventing that\n>>>>> seg-fault in git-subtree script\n>>>>> I'm encountering it myself using git 2.24.0.windows.2., seg-fault is\n>>>>> in the same while loop (currently on line 757)\n>>>>> When I tried your suggestion of adding the ($parents) ($rev) to the\n>>>>> progress print I see that the last commit have only one revision\n>>>>> printed\n>>>>> like this:\n>>>>> \n>>>>> 259/290 (523) [271] (843dd34090d36dfabd6a2e3e8459a4887427313b)\n>>>>> (a69ee056f66acf66c63f89f55d26c0cc17036623)\n>>>>> 259/290 (525) [273] (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n>>>>> (843dd34090d36dfabd6a2e3e8459a4887427313b)\n>>>>> 259/290 (527) [275] (82303752a428cf1d789ac9f156008adb2798b7b5)\n>>>>> (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n>>>>> 259/290 (528) [276]\n>>>>> (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a35135)\n>>>>> 259/290 (529) [277]\n>>>>> (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798b7b5)\n>>>>> 259/290 (530) [278]\n>>>>> (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f8603728ff)\n>>>>> 259/290 (531) [279]\n>>>>> (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a22cad)\n>>>>> 259/290 (532) [280]\n>>>>> (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a50bfc)\n>>>>> 259/290 (533) [281]\n>>>>> (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc4291a7f)\n>>>>> 259/290 (534) [282]\n>>>>> (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28caf3b8)\n>>>>> 259/290 (535) [283]\n>>>>> (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b8521eb)\n>>>>> 259/290 (536) [284]\n>>>>> (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99e7b1)\n>>>>> 259/290 (537) [285]\n>>>>> (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43ce0e5)\n>>>>> 259/290 (538) [286]\n>>>>> (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df4b37)\n>>>>> C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree: line 757:\n>>>>> 8853 Done eval \"$grl\"\n>>>>> 8854 Segmentation fault (core dumped) | while read rev\n>>>>> parents; do\n>>>>> process_split_commit \"$rev\" \"$parents\" 0;\n>>>>> done\n>>>>> \n>>>>> I downgraded git to 2.19.0-windows.1 and it works now.\n>>>>> \n>>>>> \n>>>>> I'm thankful for your insights\n>>>>> Nadav Sinai\n>>>>> Web Tech lead\n>>>>> Philips-Algotec\n>>>>> \n>>>>> \n>>>>> \n>>>>> \n>>>>> \n>>>> \n>>>> \n>> \n>> \n\n"},{"id":"387944","messageId":"038c72f0349174bb92e1dd9c3b38f02543ba1d95.camel@swri.org","threadId":"52406","inReplyTo":"D99ED706-EC49-4A52-8186-5C9B0B5BC744@icloud.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Strain, Roger L.","fromEmail":"roger.strain@swri.org","sentAt":"2019-12-11T14:39:04Z","receivedAt":"2019-12-11T14:39:23Z","isPatch":false,"sender":{"key":"roger.strain@swri.org","avatar":"https://avatars.githubusercontent.com/u/52041877?v=4"},"body":"The comment about \"400-500 commits\" is interesting to me; our repos\nregularly have to process thousands (!) of commits like this, which is\npart of the reason I felt a need to try to fix the behavior. When I\nmanage to get things into a reasonably sane state, the commit count may\nstay in the hundreds, but if things go too long, I've regularly seen\ncounts well over 5000.\n\nThis makes me wonder if the problem is perhaps related to the hardware\ninvolved; maybe the algorithm is doing exactly what it should, but the\navailable RAM isn't sufficient. If that's the problem, perhaps we could\nfind a way to perform the recursive work without using actual\nrecursion, reducing the number of instances on the stack.\n\n-- \nRoger Strain\n\nOn Wed, 2019-12-11 at 16:43 +1100, Tom Clarkson wrote:\n> > Is there a minimal, complete and verifiable example that other\n> > developers\n> > could use to analyze the bug?\n> \n> I ran into this bug today, and while not much closer to a solution, I\n> believe I understand why it is happening. \n> \n> The recursive search is required because the original rev-list based\n> approach could leave out some relevant commits - ie reversion would\n> replace an obvious bug with a hidden one.\n> \n> The search will stop when it reaches either a root commit or one\n> already mapped to a subtree commit.\n> \n> If you have a small repo or run git subtree add directly on your\n> master branch, the search will terminate fairly quickly.\n> \n> However, if you do everything via pull requests, the search will hit\n> a merge commit where one side is just ahead of the subtree mapping,\n> but the other side is several thousand commits with no sign of either\n> a root or any subtrees.\n> \n> I’m not sure yet if it is the number of commits or merges\n> specifically, but the script seems to be able to handle around 400-\n> 500 commits before it falls over.\n> \n> > > \n> > > \n> > > > Am 09.12.2019 um 17:20 schrieb Johannes Schindelin <\n> > > > Johannes.Schindelin@gmx.de>:\n> > > > \n> > > > Hi,\n> > > > \n> > > > > On Mon, 9 Dec 2019, Marc Balmer wrote:\n> > > > > \n> > > > > I am not familiar with the source code, so I can not send in\n> > > > > that\n> > > > > revert. I can, however, say that I am grateful to whomever\n> > > > > does it ;)\n> > > > \n> > > > I am against reverting the change without knowing the root\n> > > > cause.\n> > > > \n> > > > The recent reporter only compared Git for Windows v2.19.0 vs\n> > > > v2.20.1,\n> > > > which is _quite_ a big difference.\n> > > > \n> > > > For what I know, the problem might be a change in the MSYS2\n> > > > runtime that\n> > > > is mistaken by some malware for malicious code (we did\n> > > > introduce some code\n> > > > to emulate Ctrl+C in MinTTY which injects a remote thread and\n> > > > executes\n> > > > ExitProcess() there, which might very well be construed as an\n> > > > attack, even\n> > > > if it is actually very much desired behavior).\n> > > > \n> > > > These segmentation faults in `git subtree` on Windows have\n> > > > traditionally\n> > > > been _all_ because of overzealous anti-malware.\n> > > > \n> > > > So first, a much more fine-grained analysis would be required,\n> > > > e.g.\n> > > > comparing v2.20.1 against v2.20.0, then copying _just_ the\n> > > > `git-subtree`\n> > > > file from a working into a non-working version (or vice versa;\n> > > > I would\n> > > > highly recommend using the portable versions for such side-by\n> > > > side\n> > > > comparison).\n> > > > \n> > > > Ciao,\n> > > > Johannes\n> > > > \n> > > > > \n> > > > > - Marc\n> > > > > \n> > > > > \n> > > > > > > Am 09.12.2019 um 15:18 schrieb Strain, Roger L. <\n> > > > > > > roger.strain@swri.org>:\n> > > > > > \n> > > > > > As I said, I'm using a custom script here. I don't know if\n> > > > > > anybody else\n> > > > > > benefited from the change and hasn't said anything, but I\n> > > > > > won't object\n> > > > > > to someone submitting that revert.\n> > > > > > \n> > > > > > --\n> > > > > > Roger\n> > > > > > \n> > > > > > -----Original Message-----\n> > > > > > From: Marc Balmer \n> > > > > > To: \"Strain, Roger L.\" \n> > > > > > Cc: ns@nadavsinai.com , \n> > > > > > git@vger.kernel.org <\n> > > > > > git@vger.kernel.org>, Johannes.Schindelin@gmx.de <\n> > > > > > Johannes.Schindelin@gmx.de>, gitster@pobox.com <\n> > > > > > gitster@pobox.com>,\n> > > > > > pclouds@gmail.com \n> > > > > > Subject: Re: Regression in git-subtree.sh, introduced in\n> > > > > > 2.20.1, after\n> > > > > > 315a84f9aa0e2e629b0680068646b0032518ebed\n> > > > > > Date: Mon, 09 Dec 2019 15:13:47 +0100\n> > > > > > \n> > > > > > Roger,\n> > > > > > \n> > > > > > I am all for reverting it. if that does not cause any other\n> > > > > > regressions\n> > > > > > or headaches (or both...)\n> > > > > > \n> > > > > > - Marc\n> > > > > > \n> > > > > > \n> > > > > > \n> > > > > > Am 09.12.2019 um 15:11 schrieb Strain, Roger L. <\n> > > > > > roger.strain@swri.org>\n> > > > > > :\n> > > > > > \n> > > > > > I haven't been able to find anything relating to the issue,\n> > > > > > but I also\n> > > > > > haven't had a repo that exposes the problem to test more\n> > > > > > thoroughly\n> > > > > > against. If this happens to be a public repo somewhere, I'd\n> > > > > > be more\n> > > > > > than happy to take a second look.\n> > > > > > \n> > > > > > That being said, if the community feels it would be better\n> > > > > > to revert\n> > > > > > the changes that were introduced, I won't object. I've had\n> > > > > > to further\n> > > > > > customize the script for our internal use, and those\n> > > > > > changes aren't\n> > > > > > something that would be useful for the public at large. (A\n> > > > > > few changes\n> > > > > > relate to the presence/absence of a specific file, which I\n> > > > > > certainly\n> > > > > > wouldn't expect anyone else to have.) Short story is we're\n> > > > > > going to\n> > > > > > have to use a custom script going forward, so keeping or\n> > > > > > reverting the\n> > > > > > changes here make no difference to us. I still feel that\n> > > > > > the changes\n> > > > > > which were made make the script more correct, but clearly\n> > > > > > there's some\n> > > > > > undiagnosed logic error somewhere.\n> > > > > > \n> > > > > > Honestly, I'm surprised we didn't see this particular issue\n> > > > > > show up on\n> > > > > > our own repo; it's ridiculously large and complex. At least\n> > > > > > if it had,\n> > > > > > I'd be able to troubleshoot it more reliably.\n> > > > > > \n> > > > > > --\n> > > > > > Roger Strain\n> > > > > > \n> > > > > > -----Original Message-----\n> > > > > > From: Nadav SInai \n> > > > > > To: roger.strain@swri.org\n> > > > > > Cc: Johannes.Schindelin@gmx.de, git@vger.kernel.org, \n> > > > > > gitster@pobox.com,\n> > > > > > marc@msys.ch, pclouds@gmail.com\n> > > > > > Subject: RE: Regression in git-subtree.sh, introduced in\n> > > > > > 2.20.1, after\n> > > > > > 315a84f9aa0e2e629b0680068646b0032518ebed\n> > > > > > Date: Sun, 08 Dec 2019 12:30:48 +0200\n> > > > > > \n> > > > > > [EXTERNAL EMAIL]\n> > > > > > \n> > > > > > Hi, I'm curious if any of you had any luck in preventing\n> > > > > > that\n> > > > > > seg-fault in git-subtree script\n> > > > > > I'm encountering it myself using git 2.24.0.windows.2.,\n> > > > > > seg-fault is\n> > > > > > in the same while loop (currently on line 757)\n> > > > > > When I tried your suggestion of adding the ($parents)\n> > > > > > ($rev) to the\n> > > > > > progress print I see that the last commit have only one\n> > > > > > revision\n> > > > > > printed\n> > > > > > like this:\n> > > > > > \n> > > > > > 259/290 (523) [271]\n> > > > > > (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> > > > > > (a69ee056f66acf66c63f89f55d26c0cc17036623)\n> > > > > > 259/290 (525) [273]\n> > > > > > (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> > > > > > (843dd34090d36dfabd6a2e3e8459a4887427313b)\n> > > > > > 259/290 (527) [275]\n> > > > > > (82303752a428cf1d789ac9f156008adb2798b7b5)\n> > > > > > (f5eea1a3cbe1e16acba53e8a9fe07b6525a8b97c)\n> > > > > > 259/290 (528) [276]\n> > > > > > (7187897883c9fb4d33d4c87a02b876f8603728ff05f0945ae2ce9f98a3\n> > > > > > 5135)\n> > > > > > 259/290 (529) [277]\n> > > > > > (a00a3665343439a426671958dd90ed0407a22cad9ac9f156008adb2798\n> > > > > > b7b5)\n> > > > > > 259/290 (530) [278]\n> > > > > > (90beb94ebd331c457d79d05341453f5829a50bfcd4c87a02b876f86037\n> > > > > > 28ff)\n> > > > > > 259/290 (531) [279]\n> > > > > > (9582e0acbed1910173564e250f350b5cc4291a7f671958dd90ed0407a2\n> > > > > > 2cad)\n> > > > > > 259/290 (532) [280]\n> > > > > > (f183930d6fabd3dccdddc5ec35d754ad28caf3b879d05341453f5829a5\n> > > > > > 0bfc)\n> > > > > > 259/290 (533) [281]\n> > > > > > (c9309f3a38c41f7991d9e78ddb47f7e85b8521eb564e250f350b5cc429\n> > > > > > 1a7f)\n> > > > > > 259/290 (534) [282]\n> > > > > > (3bcf08f63a0e2b93ecc376bd679a16c80e99e7b1ddc5ec35d754ad28ca\n> > > > > > f3b8)\n> > > > > > 259/290 (535) [283]\n> > > > > > (134621bb55a0470cdf6519ce08d6909af43ce0e5d9e78ddb47f7e85b85\n> > > > > > 21eb)\n> > > > > > 259/290 (536) [284]\n> > > > > > (edb3471fbba29748f9784d29b3cee1dee2df4b37c376bd679a16c80e99\n> > > > > > e7b1)\n> > > > > > 259/290 (537) [285]\n> > > > > > (dd947a095df07a32dfd56666a395a7c42b25ca116519ce08d6909af43c\n> > > > > > e0e5)\n> > > > > > 259/290 (538) [286]\n> > > > > > (a639e09d2cbe1ea1149c080c1c95b8b018340ae2784d29b3cee1dee2df\n> > > > > > 4b37)\n> > > > > > C:/Program Files/Git/mingw64/libexec/git-core\\git-subtree:\n> > > > > > line 757:\n> > > > > > 8853 Done eval \"$grl\"\n> > > > > > 8854 Segmentation fault (core dumped) | while read\n> > > > > > rev\n> > > > > > parents; do\n> > > > > > process_split_commit \"$rev\" \"$parents\" 0;\n> > > > > > done\n> > > > > > \n> > > > > > I downgraded git to 2.19.0-windows.1 and it works now.\n> > > > > > \n> > > > > > \n> > > > > > I'm thankful for your insights\n> > > > > > Nadav Sinai\n> > > > > > Web Tech lead\n> > > > > > Philips-Algotec\n> > > > > > \n> > > > > > \n> > > > > > \n> > > > > > \n> > > > > > \n> > > > > \n> > > > > \n> > > \n> > > \n> \n> \n"},{"id":"388005","messageId":"5C8CA727-370E-4CEE-BBF9-F336C5921D98@icloud.com","threadId":"52406","inReplyTo":"038c72f0349174bb92e1dd9c3b38f02543ba1d95.camel@swri.org","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-12T05:02:50Z","receivedAt":"2019-12-12T05:02:58Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n> This makes me wonder if the problem is perhaps related to the hardware\n> involved; maybe the algorithm is doing exactly what it should, but the\n> available RAM isn't sufficient. If that's the problem, perhaps we could\n> find a way to perform the recursive work without using actual\n> recursion, reducing the number of instances on the stack.\n\nIt’s not so much hardware as OS I think - After adding stack depth (the indent parameter on check_parents) to the log, I have been able to get different results with ulimit settings.\n\nWith the default stack size on macOS of 8MB, It falls over at depth 445. Being less than the shortest path to the root commit, that matches my initial count, which was just the number of lines in the log.\n\nReducing the stack size with ulimit -s 4096 makes it fall over at 225\n\nIncreasing to the hard limit of 64MB should allow a depth of around 4000, and as it turns out that did allow the script to complete, reaching a maximum depth of 1148.\n\nI’m not seeing any issues with the hashes being wrong (all show no parents or subtree) but processing all those commits that resolve to nothing does take forever.\n\nThe mainline commit test seems to work ok on my repo, but it’s fairly easy to see scenarios where it would break, such as having a subfolder with the same name within the subtree.\n\nSo while part of the fix will be a more reliable test, it also needs to work before parent commits are processed to mitigate the recursion issues.\n\nThe rules I have come up with so far are below. There are still scenarios where the recursion is unavoidable such as running an initial split on a large repo, but that should be much less common than using a small subtree with a more complex existing repo.\n\nIn the initial setup of cmd_split, collect some extra information:\n\n\t- Add rev-list of all git-subtree-split values to the cache. I’d expect subtrees to usually be smaller than mainline, but since we can do that non-recursively we may as well.\n\n\t- Find the git-subtree-mainline value from subtree add/rejoin. Anything in its rev list should only be reachable by mainline commits. If not (which probably requires doing something convoluted like having subtree include mainline as its own subtree), this is a good place to check that and fall back to the existing behavior. \n\n\nWhen processing each commit:\n\nIf no prior splits were found, we only have mainline commits.\n\n \t- If $dir exists, it is a mainline commit needing copy - use existing process.\n\t- If $dir does not exist, it is a mainline commit that will map to nothing - no need to process further.\n\nIf we do have some known subtree commits:\n\n\t- If it is in the cache, it is a subtree commit we don’t need to process further.\n\t- If subtree root is not reachable (rev-list or merge-base), must be mainline pre subtree add. Map to nothing and skip further processing.\n\t- If any subtree root is reachable, could be either mainline commit with subtree merged in, or subtree commit newer than the last add/squash (subtree pull/merge without squash does not use a custom commit message)\n\t\t- If $dir does not exist, must be subtree - add to the cache as mapped to self, no need to process parents.\n\t\t- If the folder does exist, it is either a mainline commit to be processed normally, or a subtree that happens to contain a folder with the same name. Check if mainline root is reachable.\n\n\n\n"},{"id":"388109","messageId":"nycvar.QRO.7.76.6.1912131440400.46@tvgsbejvaqbjf.bet","threadId":"52406","inReplyTo":"5C8CA727-370E-4CEE-BBF9-F336C5921D98@icloud.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-13T13:41:27Z","receivedAt":"2019-12-13T20:37:25Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Tom,\n\nOn Thu, 12 Dec 2019, Tom Clarkson wrote:\n\n>\n> > This makes me wonder if the problem is perhaps related to the hardware\n> > involved; maybe the algorithm is doing exactly what it should, but the\n> > available RAM isn't sufficient. If that's the problem, perhaps we could\n> > find a way to perform the recursive work without using actual\n> > recursion, reducing the number of instances on the stack.\n>\n> It’s not so much hardware as OS I think - After adding stack depth (the indent parameter on check_parents) to the log, I have been able to get different results with ulimit settings.\n\nDo you mean to say that the stack overflow is reported as a segmentation\nfault? If so, that message was sure a red herring...\n\nThanks,\nDscho\n\n>\n> With the default stack size on macOS of 8MB, It falls over at depth 445. Being less than the shortest path to the root commit, that matches my initial count, which was just the number of lines in the log.\n>\n> Reducing the stack size with ulimit -s 4096 makes it fall over at 225\n>\n> Increasing to the hard limit of 64MB should allow a depth of around 4000, and as it turns out that did allow the script to complete, reaching a maximum depth of 1148.\n>\n> I’m not seeing any issues with the hashes being wrong (all show no parents or subtree) but processing all those commits that resolve to nothing does take forever.\n>\n> The mainline commit test seems to work ok on my repo, but it’s fairly easy to see scenarios where it would break, such as having a subfolder with the same name within the subtree.\n>\n> So while part of the fix will be a more reliable test, it also needs to work before parent commits are processed to mitigate the recursion issues.\n>\n> The rules I have come up with so far are below. There are still scenarios where the recursion is unavoidable such as running an initial split on a large repo, but that should be much less common than using a small subtree with a more complex existing repo.\n>\n> In the initial setup of cmd_split, collect some extra information:\n>\n> \t- Add rev-list of all git-subtree-split values to the cache. I’d expect subtrees to usually be smaller than mainline, but since we can do that non-recursively we may as well.\n>\n> \t- Find the git-subtree-mainline value from subtree add/rejoin. Anything in its rev list should only be reachable by mainline commits. If not (which probably requires doing something convoluted like having subtree include mainline as its own subtree), this is a good place to check that and fall back to the existing behavior.\n>\n>\n> When processing each commit:\n>\n> If no prior splits were found, we only have mainline commits.\n>\n> \t- If $dir exists, it is a mainline commit needing copy - use existing process.\n> \t- If $dir does not exist, it is a mainline commit that will map to nothing - no need to process further.\n>\n> If we do have some known subtree commits:\n>\n> \t- If it is in the cache, it is a subtree commit we don’t need to process further.\n> \t- If subtree root is not reachable (rev-list or merge-base), must be mainline pre subtree add. Map to nothing and skip further processing.\n> \t- If any subtree root is reachable, could be either mainline commit with subtree merged in, or subtree commit newer than the last add/squash (subtree pull/merge without squash does not use a custom commit message)\n> \t\t- If $dir does not exist, must be subtree - add to the cache as mapped to self, no need to process parents.\n> \t\t- If the folder does exist, it is either a mainline commit to be processed normally, or a subtree that happens to contain a folder with the same name. Check if mainline root is reachable.\n>\n>\n>\n>\n"},{"id":"388204","messageId":"2BB5D7E6-D565-4DCF-8E4D-D410AC1F91F3@msys.ch","threadId":"52406","inReplyTo":"nycvar.QRO.7.76.6.1912131440400.46@tvgsbejvaqbjf.bet","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Marc Balmer","fromEmail":"marc@msys.ch","sentAt":"2019-12-14T08:29:10Z","receivedAt":"2019-12-14T08:29:32Z","isPatch":false,"sender":{"key":"marc@msys.ch","avatar":"https://gravatar.com/avatar/beb9137fb845e4986641aeb23660b07d3a6dda8a3048d2c9fe6f6e16b5167517?d=mp&s=160"},"body":"\n\n> Am 13.12.2019 um 14:41 schrieb Johannes Schindelin :\n> \n> Hi Tom,\n> \n> On Thu, 12 Dec 2019, Tom Clarkson wrote:\n> \n>> \n>>> This makes me wonder if the problem is perhaps related to the hardware\n>>> involved; maybe the algorithm is doing exactly what it should, but the\n>>> available RAM isn't sufficient. If that's the problem, perhaps we could\n>>> find a way to perform the recursive work without using actual\n>>> recursion, reducing the number of instances on the stack.\n>> \n>> It’s not so much hardware as OS I think - After adding stack depth (the indent parameter on check_parents) to the log, I have been able to get different results with ulimit settings.\n> \n> Do you mean to say that the stack overflow is reported as a segmentation\n> fault? If so, that message was sure a red herring...\n\nFWIW, changing the stack limit using ulimit does not change anything on my (Fedora) system. At some point, with exactly the same two leading numbers (those separated with a /)it seems to enter an endless (recursive) loop, eventually eating up all memory. And then, after some time, it segfaults.\n\n\n> \n> Thanks,\n> Dscho\n> \n>> \n>> With the default stack size on macOS of 8MB, It falls over at depth 445. Being less than the shortest path to the root commit, that matches my initial count, which was just the number of lines in the log.\n>> \n>> Reducing the stack size with ulimit -s 4096 makes it fall over at 225\n>> \n>> Increasing to the hard limit of 64MB should allow a depth of around 4000, and as it turns out that did allow the script to complete, reaching a maximum depth of 1148.\n>> \n>> I’m not seeing any issues with the hashes being wrong (all show no parents or subtree) but processing all those commits that resolve to nothing does take forever.\n>> \n>> The mainline commit test seems to work ok on my repo, but it’s fairly easy to see scenarios where it would break, such as having a subfolder with the same name within the subtree.\n>> \n>> So while part of the fix will be a more reliable test, it also needs to work before parent commits are processed to mitigate the recursion issues.\n>> \n>> The rules I have come up with so far are below. There are still scenarios where the recursion is unavoidable such as running an initial split on a large repo, but that should be much less common than using a small subtree with a more complex existing repo.\n>> \n>> In the initial setup of cmd_split, collect some extra information:\n>> \n>> \t- Add rev-list of all git-subtree-split values to the cache. I’d expect subtrees to usually be smaller than mainline, but since we can do that non-recursively we may as well.\n>> \n>> \t- Find the git-subtree-mainline value from subtree add/rejoin. Anything in its rev list should only be reachable by mainline commits. If not (which probably requires doing something convoluted like having subtree include mainline as its own subtree), this is a good place to check that and fall back to the existing behavior.\n>> \n>> \n>> When processing each commit:\n>> \n>> If no prior splits were found, we only have mainline commits.\n>> \n>> \t- If $dir exists, it is a mainline commit needing copy - use existing process.\n>> \t- If $dir does not exist, it is a mainline commit that will map to nothing - no need to process further.\n>> \n>> If we do have some known subtree commits:\n>> \n>> \t- If it is in the cache, it is a subtree commit we don’t need to process further.\n>> \t- If subtree root is not reachable (rev-list or merge-base), must be mainline pre subtree add. Map to nothing and skip further processing.\n>> \t- If any subtree root is reachable, could be either mainline commit with subtree merged in, or subtree commit newer than the last add/squash (subtree pull/merge without squash does not use a custom commit message)\n>> \t\t- If $dir does not exist, must be subtree - add to the cache as mapped to self, no need to process parents.\n>> \t\t- If the folder does exist, it is either a mainline commit to be processed normally, or a subtree that happens to contain a folder with the same name. Check if mainline root is reachable.\n>> \n>> \n>> \n>> \n\n"},{"id":"388209","messageId":"10B64ECE-6635-4735-A8AA-44E66BF0E5DA@icloud.com","threadId":"52406","inReplyTo":"BAB4CF6D-6904-4698-ACE1-EBEEC745E569@msys.ch","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-14T14:27:32Z","receivedAt":"2019-12-14T14:27:39Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":">> FWIW, changing the stack limit using ulimit does not change anything on my (Fedora) system. At some point, with exactly the same two leading numbers (those separated with a /)it seems to enter an endless (recursive) loop, eventually eating up all memory. And then, after some time, it segfaults.\n>> \n> \n> So today I tried a subtree push again. It took hours.... Then it pushed every single commit that was ever done to repository.\n\nSo not actually an infinite loop, but close enough to make no difference? I think we have about three bugs interacting in interesting ways here - Is your repo one where the subtree was originally extracted from mainline?\n\nWhen you say it pushed every commit, does that mean that a bunch of mainline commits erroneously ended up in the subtree repo?\n\n"},{"id":"388241","messageId":"4F323B43-44AF-44C0-87DC-5A5C7C17FEB1@icloud.com","threadId":"52406","inReplyTo":"5C8CA727-370E-4CEE-BBF9-F336C5921D98@icloud.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-16T03:50:04Z","receivedAt":"2019-12-16T03:50:13Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"I have put together a patch (currently testing on GitGitGadget) that at least fixes things for my repo - Commits from before subtree add are recognized as a dead end, so it no longer runs out of stack space while finding its way to a root commit.\n\nI think the issue Marc ran into is a bit more complex in that the recursion eventually producing the wrong result suggests that correct identification of mainline commits remains an issue. While I have some ideas on how to improve that, it’s probably best handled separately.\n\nHowever, there is a decent chance that excluding a large number of known irrelevant commits will catch the problematic ones in that scenario - and should match the previous behavior of treating the problem commits as initial.\n\n"},{"id":"388259","messageId":"CAPyFy2ANiDQ+Ed+3vG-MAxeAV=CRhJow56F7tBooBpJ-Q9B-bA@mail.gmail.com","threadId":"52406","inReplyTo":"10B64ECE-6635-4735-A8AA-44E66BF0E5DA@icloud.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Ed Maste","fromEmail":"emaste@freebsd.org","sentAt":"2019-12-16T11:30:50Z","receivedAt":"2019-12-16T15:17:15Z","isPatch":false,"sender":{"key":"emaste@freebsd.org","avatar":"https://avatars.githubusercontent.com/u/1034582?v=4"},"body":"On Sat, 14 Dec 2019 at 09:27, Tom Clarkson wrote:\n>\n> When you say it pushed every commit, does that mean that a bunch of mainline commits erroneously ended up in the subtree repo?\n\nWe encounter this case when trying to use subtree on the FreeBSD\nrepository. In our case it's caused by commit that should not be\nclassified as a commit to the subtree, but has no files in the\nsubtree. In our case it looks like an artifact of svn-git conversion\nof an odd working branch, but the same issue is reproducible in other\nways. For example, it will appear if the subtree is deleted at some\npoint and later re-added.\n"},{"id":"388411","messageId":"C4578D90-519D-4C24-9E62-C7E949D2FE0E@icloud.com","threadId":"52406","inReplyTo":"CAPyFy2ANiDQ+Ed+3vG-MAxeAV=CRhJow56F7tBooBpJ-Q9B-bA@mail.gmail.com","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Tom Clarkson","fromEmail":"tqclarkson@icloud.com","sentAt":"2019-12-18T00:15:10Z","receivedAt":"2019-12-18T00:15:19Z","isPatch":false,"sender":{"key":"tqclarkson@icloud.com","avatar":"https://gravatar.com/avatar/02ea2942fcaf5dc1140ccda995e0dee545ff210ca8adee02d50f35de7205e301?d=mp&s=160"},"body":"\n\n> On 16 Dec 2019, at 10:30 pm, Ed Maste wrote:\n> \n> On Sat, 14 Dec 2019 at 09:27, Tom Clarkson wrote:\n>> \n>> When you say it pushed every commit, does that mean that a bunch of mainline commits erroneously ended up in the subtree repo?\n> \n> We encounter this case when trying to use subtree on the FreeBSD\n> repository. In our case it's caused by commit that should not be\n> classified as a commit to the subtree, but has no files in the\n> subtree. In our case it looks like an artifact of svn-git conversion\n> of an odd working branch, but the same issue is reproducible in other\n> ways. For example, it will appear if the subtree is deleted at some\n> point and later re-added.\n\nDeleting and re-adding a subtree is an interesting case. My patch won’t avoid the recursion there, because it can only be certain about the irrelevance of commits from before the first add.\n\nHowever, I think that may be ok to leave in as something of an edge case - you may get more recursion than your system can handle, but assuming process_split_commit is correct, you can work around it by increasing ulimit, and can avoid any subsequent performance issues with a rejoin commit. Maybe we could display some sort of “your repo is doing something weird” warning to make it clearer where there are problems to be worked around.\n\nAlthough the recursion would no doubt fall over on pretty much any machine when depth gets to 200k, it looks like the FreeBSD repo isn’t getting to that point, so let’s cover the details of more reliable mainline detection on its own thread.\n\n"},{"id":"393120","messageId":"6FCBB30F-4557-46E4-8255-B9746887F151@msys.ch","threadId":"52406","inReplyTo":"BAB4CF6D-6904-4698-ACE1-EBEEC745E569@msys.ch","subject":"Re: Regression in git-subtree.sh, introduced in 2.20.1, after 315a84f9aa0e2e629b0680068646b0032518ebed","fromName":"Marc Balmer","fromEmail":"marc@msys.ch","sentAt":"2020-03-12T10:40:09Z","receivedAt":"2020-03-12T10:40:29Z","isPatch":false,"sender":{"key":"marc@msys.ch","avatar":"https://gravatar.com/avatar/beb9137fb845e4986641aeb23660b07d3a6dda8a3048d2c9fe6f6e16b5167517?d=mp&s=160"},"body":"G'day\n\nDue to some issue in git subtree, a subtree push pushed all commits (over 8000) ever done to the main repository. So the history of the subtree'ed repository not only showed commits done to the particular subtree, but all commits in the whole project (see E-mail exchange below).\n\nToday we decided to no longer use subtrees, but to use two independend repository and managing merges manually.\n\nHow can we get rid of a subtree cache data? Is it enough to remove the .git/subtree-cache directory? Or is that dangerous? Does git-subtree store any data anywhere else?\n\nThanks and regards,\nMarc\n\n\n> Am 14.12.2019 um 14:59 schrieb Marc Balmer :\n> \n> \n> \n>> Am 14.12.2019 um 09:29 schrieb Marc Balmer :\n>> \n>> \n>> \n>>> Am 13.12.2019 um 14:41 schrieb Johannes Schindelin :\n>>> \n>>> Hi Tom,\n>>> \n>>> On Thu, 12 Dec 2019, Tom Clarkson wrote:\n>>> \n>>>> \n>>>>> This makes me wonder if the problem is perhaps related to the hardware\n>>>>> involved; maybe the algorithm is doing exactly what it should, but the\n>>>>> available RAM isn't sufficient. If that's the problem, perhaps we could\n>>>>> find a way to perform the recursive work without using actual\n>>>>> recursion, reducing the number of instances on the stack.\n>>>> \n>>>> It’s not so much hardware as OS I think - After adding stack depth (the indent parameter on check_parents) to the log, I have been able to get different results with ulimit settings.\n>>> \n>>> Do you mean to say that the stack overflow is reported as a segmentation\n>>> fault? If so, that message was sure a red herring...\n>> \n>> FWIW, changing the stack limit using ulimit does not change anything on my (Fedora) system. At some point, with exactly the same two leading numbers (those separated with a /)it seems to enter an endless (recursive) loop, eventually eating up all memory. And then, after some time, it segfaults.\n>> \n>> \n> \n> So today I tried a subtree push again. It took hours.... Then it pushed every single commit that was ever done to repository.\n> \n> I can definitely say that git subtree is totally broken and unusable at this moment.\n> \n> We will now split out what once was a subtree into a proper repository of it's own. git subtree was a nice idea, but it does not work.\n> \n>>> \n>>> Thanks,\n>>> Dscho\n>>> \n>>>> \n>>>> With the default stack size on macOS of 8MB, It falls over at depth 445. Being less than the shortest path to the root commit, that matches my initial count, which was just the number of lines in the log.\n>>>> \n>>>> Reducing the stack size with ulimit -s 4096 makes it fall over at 225\n>>>> \n>>>> Increasing to the hard limit of 64MB should allow a depth of around 4000, and as it turns out that did allow the script to complete, reaching a maximum depth of 1148.\n>>>> \n>>>> I’m not seeing any issues with the hashes being wrong (all show no parents or subtree) but processing all those commits that resolve to nothing does take forever.\n>>>> \n>>>> The mainline commit test seems to work ok on my repo, but it’s fairly easy to see scenarios where it would break, such as having a subfolder with the same name within the subtree.\n>>>> \n>>>> So while part of the fix will be a more reliable test, it also needs to work before parent commits are processed to mitigate the recursion issues.\n>>>> \n>>>> The rules I have come up with so far are below. There are still scenarios where the recursion is unavoidable such as running an initial split on a large repo, but that should be much less common than using a small subtree with a more complex existing repo.\n>>>> \n>>>> In the initial setup of cmd_split, collect some extra information:\n>>>> \n>>>> \t- Add rev-list of all git-subtree-split values to the cache. I’d expect subtrees to usually be smaller than mainline, but since we can do that non-recursively we may as well.\n>>>> \n>>>> \t- Find the git-subtree-mainline value from subtree add/rejoin. Anything in its rev list should only be reachable by mainline commits. If not (which probably requires doing something convoluted like having subtree include mainline as its own subtree), this is a good place to check that and fall back to the existing behavior.\n>>>> \n>>>> \n>>>> When processing each commit:\n>>>> \n>>>> If no prior splits were found, we only have mainline commits.\n>>>> \n>>>> \t- If $dir exists, it is a mainline commit needing copy - use existing process.\n>>>> \t- If $dir does not exist, it is a mainline commit that will map to nothing - no need to process further.\n>>>> \n>>>> If we do have some known subtree commits:\n>>>> \n>>>> \t- If it is in the cache, it is a subtree commit we don’t need to process further.\n>>>> \t- If subtree root is not reachable (rev-list or merge-base), must be mainline pre subtree add. Map to nothing and skip further processing.\n>>>> \t- If any subtree root is reachable, could be either mainline commit with subtree merged in, or subtree commit newer than the last add/squash (subtree pull/merge without squash does not use a custom commit message)\n>>>> \t\t- If $dir does not exist, must be subtree - add to the cache as mapped to self, no need to process parents.\n>>>> \t\t- If the folder does exist, it is either a mainline commit to be processed normally, or a subtree that happens to contain a folder with the same name. Check if mainline root is reachable.\n\n"}]}