{"thread":{"id":"60476","subject":"Git Rename Detection Bug","startedAt":"2023-11-06T12:01:01Z","lastAt":"2023-12-28T15:59:58Z","messageCount":13,"participants":["Jeremy Pridmore","Elijah Newren","Philip Oakley","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"484460","messageId":"LO6P265MB6736043BE8FB607DB671D21EFAAAA@LO6P265MB6736.GBRP265.PROD.OUTLOOK.COM","threadId":"60476","inReplyTo":null,"subject":"Git Rename Detection Bug","fromName":"Jeremy Pridmore","fromEmail":"jpridmore@rdt.co.uk","sentAt":"2023-11-06T12:00:53Z","receivedAt":"2023-11-06T12:01:01Z","isPatch":false,"sender":{"key":"jpridmore@rdt.co.uk","avatar":null},"body":"Thank you for filling out a Git bug report!\nPlease answer the following questions to help us understand your issue.\n\nWhat did you do before the bug happened? (Steps to reproduce your issue)\nI have two GIT repositories (A and B). Both migrated from the same TFS server using git-tfs tool. I migrated code into A and made lots of changes, including moving 50,000+ files from folder \"/Landscape\" to \"/Landscape/src\". B contains the same code but with various other changes made since my original migration from TFS to A. All the files in B are still in the \"/Landscape\" folder. I recently needed to merge my changes from A to B, so I added A as a remote to B and then performed a number of cherry-picks from A to B, but got stuck when trying to cherry-pick the commit containing the results of moving all files into \"/Landscape/src\".\n\nWhat did you expect to happen? (Expected behavior)\nI expected the git rename detection to match all files in A \"/Landscape\" to files in B \"/Landscape/src\".\n\nWhat happened instead? (Actual behavior)\nAlthough many files were matched successfully, git mismatched over two dozen similarly named files, e.g.\n\nIncorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IAccountsIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IAccountsIntegration.vb\nIncorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb\nIncorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\nIncorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\nIncorrect name match: Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\nIncorrect path match: Landscape/Documentation/uiDocumentation/licenses.licx -> Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\nIncorrect path match: Landscape/Import/uiImport/My Project/licenses.licx -> Landscape/src/Documentation/uiDocumentation/licenses.licx\nIncorrect path match: Landscape/Main/uiMain.Workflow/My Project/licenses.licx -> Landscape/src/Import/uiImport/My Project/licenses.licx\nIncorrect path match: Landscape/Main/uiMain/My Project/licenses.licx -> Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\nIncorrect path match: Landscape/LandscapeApiService.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/Main/uiMain.Workflow/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx -> Landscape/src/Main/uiMain/My Project/licenses.licx\nIncorrect path match: Landscape/Main/uiMain.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Main/uiMain/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/Policy/Rdt.Policy.UI/Properties/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx\nIncorrect path match: Landscape/Rates/uiRates/My Project/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI/Properties/licenses.licx\nIncorrect path match: Landscape/Rdt.Claim.UI/Properties/licenses.licx -> Landscape/src/Rates/uiRates/My Project/licenses.licx\nIncorrect path match: Landscape/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Claim.UI/Properties/licenses.licx\nIncorrect path match: Landscape/Rdt.Landscape.UI.Templates/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx\nIncorrect path match: Landscape/Rdt.Landscape.UI.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates/Properties/licenses.licx\nIncorrect path match: Landscape/Rdt.Landscape.UI/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Workflow/Properties/licenses.licx\nIncorrect path match: Landscape/StandardLetters/uiStandardLetters/My Project/licenses.licx -> Landscape/src/Rdt.Landscape.UI/Properties/licenses.licx\nIncorrect path match: Landscape/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb -> Landscape/src/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb\nIncorrect path match: Landscape/SystemEvents/uiSystemEvents/My Project/licenses.licx -> Landscape/src/StandardLetters/uiStandardLetters/My Project/licenses.licx\nIncorrect path match: Landscape/Services/busServices/RDT_Logo.ico -> Landscape/src/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/Startup/uiStartup/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup32/RDT_Logo.ico\nIncorrect path match: Landscape/Startup/uiStartup/Resources/newrdlogogradiant48shad.ico -> Landscape/src/Startup/uiStartup32/newrdlogogradiant48shad.ico\nIncorrect path match: Landscape/Templates/uiTemplates.Workflow/My Project/licenses.licx -> Landscape/src/SystemEvents/uiSystemEvents/My Project/licenses.licx\nIncorrect path match: Landscape/Utils/Rdt.Utils.UI/Properties/licenses.licx -> Landscape/src/Templates/uiTemplates.Workflow/My Project/licenses.licx\nIncorrect path match: Landscape/Utils/uiUtils/My Project/licenses.licx -> Landscape/src/Utils/Rdt.Utils.UI/Properties/licenses.licx\nIncorrect name match: Landscape/WebServices/ServiceFabric/Policy/Rdt.Policy.Repository.Service.Fabric.Host/PackageRoot/Data/Swagger/Examples/POST_UKSTasks_Response.json -> Landscape/src/Utils/uiUtils/My Project/licenses.licx\n\n\nWhat's different between what you expected and what actually happened?\n\nAs you can see, although the filenames (and content) are the same, there are a few files that have been incorrectly matched to files with teh same name, but in a different sub folder. In some cases, it seems that the catalyst has been git thinking that a file from B has been deleted from A, when in fact it has not actually been deleted at all.\nFor example, the file Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1 has not been deleted in A or B, therefore git should not have attempted to rename Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 to Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1, especially as it then attempts to rename Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 to Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 and so on.\n\nGit status contains, for example:\n deleted by them: Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n renamed: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n\nThe correct renames should have been:\ngit mv \"Landscape/Complaints/Rdt.Complaints.UI/Interfaces/IAccountsIntegration.vb\" \"Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IAccountsIntegration.vb\"\ngit mv \"Landscape/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb\" \"Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb\"\ngit mv \"Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\" \"Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\"\ngit mv \"Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\" \"Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\"\ngit mv \"Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\" \"Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\"\ngit mv \"Landscape/Documentation/uiDocumentation/licenses.licx\" \"Landscape/src/Documentation/uiDocumentation/licenses.licx\"\ngit mv \"Landscape/Import/uiImport/My Project/licenses.licx\" \"Landscape/src/Import/uiImport/My Project/licenses.licx\"\ngit mv \"Landscape/Main/uiMain.Workflow/My Project/licenses.licx\" \"Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\"\ngit mv \"Landscape/Main/uiMain.Workflow/Resources/RDT_Logo.ico\" \"Landscape/src/Main/uiMain.Workflow/Resources/RDT_Logo.ico\"\ngit mv \"Landscape/Main/uiMain/My Project/licenses.licx\" \"Landscape/src/Main/uiMain/My Project/licenses.licx\"\ngit mv \"Landscape/Main/uiMain/Resources/RDT_Logo.ico\" \"Landscape/src/Main/uiMain/Resources/RDT_Logo.ico\"\ngit mv \"Landscape/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx\" \"Landscape/src/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx\"\ngit mv \"Landscape/Policy/Rdt.Policy.UI/Properties/licenses.licx\" \"Landscape/src/Policy/Rdt.Policy.UI/Properties/licenses.licx\"\ngit mv \"Landscape/Rdt.Claim.UI/Properties/licenses.licx\" \"Landscape/src/Rdt.Claim.UI/Properties/licenses.licx\"\ngit mv \"Landscape/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx\" \"Landscape/src/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx\"\ngit mv \"Landscape/Rdt.Landscape.UI.Templates/Properties/licenses.licx\" \"Landscape/src/Rdt.Landscape.UI.Templates/Properties/licenses.licx\"\ngit mv \"Landscape/Rdt.Landscape.UI.Workflow/Properties/licenses.licx\" \"Landscape/src/Rdt.Landscape.UI.Workflow/Properties/licenses.licx\"\ngit mv \"Landscape/Rdt.Landscape.UI/Properties/licenses.licx\" \"Landscape/src/Rdt.Landscape.UI/Properties/licenses.licx\"\ngit mv \"Landscape/Services/busServices/RDT_Logo.ico\" \"Landscape/src/Services/busServices/RDT_Logo.ico\"\ngit mv \"Landscape/Services/uiServices/Complaints/Interfaces/IAccountsIntegration.vb\" \"Landscape/src/Services/uiServices/Complaints/Interfaces/IAccountsIntegration.vb\"\ngit mv \"Landscape/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb\" \"Landscape/src/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb\"\ngit mv \"Landscape/StandardLetters/uiStandardLetters/My Project/licenses.licx\" \"Landscape/src/StandardLetters/uiStandardLetters/My Project/licenses.licx\"\ngit mv \"Landscape/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\" \"Landscape/src/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\"\ngit mv \"Landscape/Startup/uiStartup/Resources/RDT_Logo.ico\" \"Landscape/src/Startup/uiStartup/Resources/RDT_Logo.ico\"\ngit mv \"Landscape/SystemEvents/uiSystemEvents/My Project/licenses.licx\" \"Landscape/src/SystemEvents/uiSystemEvents/My Project/licenses.licx\"\ngit mv \"Landscape/Templates/uiTemplates.Workflow/My Project/licenses.licx\" \"Landscape/src/Templates/uiTemplates.Workflow/My Project/licenses.licx\"\ngit mv \"Landscape/Utils/Rdt.Utils.UI/Properties/licenses.licx\" \"Landscape/src/Utils/Rdt.Utils.UI/Properties/licenses.licx\"\ngit mv \"Landscape/Utils/uiUtils/My Project/licenses.licx\" \"Landscape/src/Utils/uiUtils/My Project/licenses.licx\"\n\n\nAnything else you want to add:\nI can't help but think that this is related to changes made by Palantir:\nhttps://blog.palantir.com/optimizing-gits-merge-machinery-1-127ceb0ef2a1\n\nI have tried to unstage these renames using \"git restore --staged \" so I can then apply the correct \"git mv\" commands, but bizzarely, this then results in \"git status\" reporting a different, smaller set of mismatched names:\n\nIncorrect name match: Landscape/Services/busServices/Service References/DataCollectorService/busServices.DataCollectorService.GetDataResponse.datasource -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\nIncorrect name match: Landscape/Services/busServices/Service References/GlobalGatewayService/busServices.GlobalGatewayService.OrderResponse.datasource -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\nIncorrect name match: Landscape/WebServices/ServiceFabric/Policy/Rdt.Policy.Repository.Service.Fabric.Host/PackageRoot/Data/Swagger/Examples/POST_UKSTasks_Response.json -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\nIncorrect path match: Landscape/LandscapeApiService.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/Main/uiMain.Workflow/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/WebServices/WCFServices/Landscape WCF Service/wsWcfLandscapeServices/RDT_Logo.ico -> Landscape/src/Main/uiMain/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/WindowsServices/Rdt.BatchProcessingService/Rdt.BatchProcessingService.Integration/RDT_Logo.ico -> Landscape/src/Services/busServices/RDT_Logo.ico\nIncorrect path match: Landscape/WindowsServices/Rdt.BatchProcessingService/Rdt.BatchProcessingService.Setup/UIContent/RDT_Logo.ico -> Landscape/src/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/WindowsServices/Rdt.BatchProcessingService/Rdt.BatchProcessingService/RDT_Logo.ico -> Landscape/src/Startup/uiStartup/Resources/RDT_Logo.ico\nIncorrect path match: Landscape/_Tests/Rdt.BatchProcessingService.Tests/RDT_Logo.ico -> Landscape/src/Startup/uiStartup32/RDT_Logo.ico\nIncorrect path match: Landscape/Startup/uiStartup/Resources/newrdlogogradiant48shad.ico -> Landscape/src/Startup/uiStartup32/newrdlogogradiant48shad.ico\nIncorrect path match: Landscape/uiStartup.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/WebServices/WCFServices/Landscape WCF Service/wsWcfLandscapeServices/RDT_Logo.ico\nIncorrect path match: Landscape/wisWorkflow.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/WindowsServices/Rdt.BatchProcessingService/Rdt.BatchProcessingService.Integration/RDT_Logo.ico\nIncorrect path match: Landscape/wsXmlLandscapeServices.Setup/Setup/Binary/RDT_Logo.ico -> Landscape/src/WindowsServices/Rdt.BatchProcessingService/Rdt.BatchProcessingService.Setup/UIContent/RDT_Logo.ico\n\nNotice how \"busServices.DataCollectorService.GetDataResponse.datasource\" is now renamed to \"Landscape.Net/pre-req.ps1\", whereas the datasource file wasn't even listed previously?\n\nPlease help. I've been on this for a couple of weeks now and I'm running out of ideas.\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.42.0.windows.2\ncpu: x86_64\nbuilt from commit: 2f819d1670fff9a1818f63b6722e9959405378e3\nsizeof-long: 4\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Windows 10.0 19044\ncompiler info: gnuc: 13.2\nlibc info: no libc information available\n$SHELL (typically, interactive shell): \n\n\n[Enabled Hooks]\n\n\n\nRegards,\n\nJeremy Pridmore\nLead Solution Architect\n\n________________________________\n\nDISCLAIMER This email is confidential. It should only be read by those persons to whom it is addressed. RDT Ltd accept no liability for the consequences of any person acting, or refraining from acting, on any information contained within this e-mail or any attached documents prior to the receipt by those persons of subsequent written confirmation of that information. If you think this e-mail may not be intended for you, do not use, pass on or copy the transmission in any way. While all reasonable precautions are taken to minimise the risk of transmitting software viruses we advise you to carry out your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses.\n"},{"id":"484512","messageId":"CABPp-BHYaxa7QoXabM=7hW-93hQLK-=KayGtDHtWxxdAnJCcJw@mail.gmail.com","threadId":"60476","inReplyTo":"LO6P265MB6736043BE8FB607DB671D21EFAAAA@LO6P265MB6736.GBRP265.PROD.OUTLOOK.COM","subject":"Re: Git Rename Detection Bug","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-07T08:05:54Z","receivedAt":"2023-11-07T08:06:11Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Mon, Nov 6, 2023 at 4:01 AM Jeremy Pridmore wrote:\n>\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n\nI think it might be worthwhile to point out a few facts about rename\nhandling in Git, as background information that might clarify a few\nthings about how Git's mental model seems to differ from yours:\n\n * In git, renames are not tracked; they are detected (based on file\nsimilarity of the commits being compared).\n * So, when you run \"git mv A B\", there is no rename recorded. It's\nbasically the same as \"git rm A\", followed by creating B with the same\ncontents, followed by \"git add B\".\n * The detection happens whenever an operation (diff, log -p, merge,\nstatus, etc.) needs or wants to know about renames.\n * In git, directory renames are detected _after_ normal renames, and\nvia amalgamation of the individual renames.\n * As a corollary of the last item, the only way individual renames\ncan be affected by directory renames, is if the individual rename on\none side of history was into a directory that the other side of\nhistory renamed away; in such a case, we apply an _extra_ rename to\nmove it into the new directory. But we don't \"undo\" individual\nrenames to make them fit the majority-determined directory rename or\nanything like that.\n\n(Also, if it matters, all of this is true of both `recursive` and\n`ort` merge strategies, i.e. the old default merge backend and the new\none.)\n\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> I have two GIT repositories (A and B). Both migrated from the same TFS server using git-tfs tool. I migrated code into A and made lots of changes, including moving 50,000+ files from folder \"/Landscape\" to \"/Landscape/src\". B contains the same code but with various other changes made since my original migration from TFS to A. All the files in B are still in the \"/Landscape\" folder. I recently needed to merge my changes from A to B, so I added A as a remote to B and then performed a number of cherry-picks from A to B, but got stuck when trying to cherry-pick the commit containing the results of moving all files into \"/Landscape/src\".\n\nIn case anyone else wants to dig into this, note that this problem\nsetup precludes directory rename detection being involved. Directory\nrename detection has a rule where if the source directory wasn't\nentirely removed on one side, then that directory was not renamed on\nthat side. Seems obvious, but the upshot of that rule is that a\ndirectory cannot be renamed into a subdirectory of itself, because by\nvirtue of being a subdirectory that means its parent directory still\nexists.\n\nSo, this is a problem where only regular rename detection (i.e. rename\ndetection of individual files) is going to be at play.\n\n> What did you expect to happen? (Expected behavior)\n> I expected the git rename detection to match all files in A \"/Landscape\" to files in B \"/Landscape/src\".\n\nAre all files under \"/Landscape\" from the merge base commit\nindividually more similar to the counterpart under \"/Landscape/src\"\nthan to files under any other directory? If not, the expectation goes\nagainst how rename detection has worked in git from the beginning.\n\n> What happened instead? (Actual behavior)\n> Although many files were matched successfully, git mismatched over two dozen similarly named files, e.g.\n>\n> Incorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IAccountsIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IAccountsIntegration.vb\n> Incorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n> Incorrect name match: Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n> Incorrect path match: Landscape/Documentation/uiDocumentation/licenses.licx -> Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Import/uiImport/My Project/licenses.licx -> Landscape/src/Documentation/uiDocumentation/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain.Workflow/My Project/licenses.licx -> Landscape/src/Import/uiImport/My Project/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain/My Project/licenses.licx -> Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\n> Incorrect path match: Landscape/LandscapeApiService.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/Main/uiMain.Workflow/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx -> Landscape/src/Main/uiMain/My Project/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Main/uiMain/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Policy/Rdt.Policy.UI/Properties/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx\n> Incorrect path match: Landscape/Rates/uiRates/My Project/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Claim.UI/Properties/licenses.licx -> Landscape/src/Rates/uiRates/My Project/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Claim.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Templates/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Workflow/Properties/licenses.licx\n> Incorrect path match: Landscape/StandardLetters/uiStandardLetters/My Project/licenses.licx -> Landscape/src/Rdt.Landscape.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb -> Landscape/src/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb\n> Incorrect path match: Landscape/SystemEvents/uiSystemEvents/My Project/licenses.licx -> Landscape/src/StandardLetters/uiStandardLetters/My Project/licenses.licx\n> Incorrect path match: Landscape/Services/busServices/RDT_Logo.ico -> Landscape/src/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup32/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup/Resources/newrdlogogradiant48shad.ico -> Landscape/src/Startup/uiStartup32/newrdlogogradiant48shad.ico\n> Incorrect path match: Landscape/Templates/uiTemplates.Workflow/My Project/licenses.licx -> Landscape/src/SystemEvents/uiSystemEvents/My Project/licenses.licx\n> Incorrect path match: Landscape/Utils/Rdt.Utils.UI/Properties/licenses.licx -> Landscape/src/Templates/uiTemplates.Workflow/My Project/licenses.licx\n> Incorrect path match: Landscape/Utils/uiUtils/My Project/licenses.licx -> Landscape/src/Utils/Rdt.Utils.UI/Properties/licenses.licx\n> Incorrect name match: Landscape/WebServices/ServiceFabric/Policy/Rdt.Policy.Repository.Service.Fabric.Host/PackageRoot/Data/Swagger/Examples/POST_UKSTasks_Response.json -> Landscape/src/Utils/uiUtils/My Project/licenses.licx\n>\n>\n> What's different between what you expected and what actually happened?\n>\n> As you can see, although the filenames (and content) are the same,\n\nThe content is the same as well? So, these renames that you label as\nincorrect are actually _exact_ renames -- and further, in most cases\nthey also have an identical basename for the file as well.\n\nExact renames are detected first, before any other method of rename\ndetection is employed, and other than giving a preference to files\nwith the same basename, if there are multiple choices it just picks\none (what I'd call at random, though technically based on what the\ninternal processing order happens to be; see the \"Too many identical\nalternatives? Pick one\" code comment).\n\nAnd this, too, is true of both the `recursve` and `ort` backends; no\nchange has been made to how exact renames are handled.\n\n> In some cases, it seems that the catalyst has been git thinking that a file from B has been deleted from A, when in fact it has not actually been deleted at all.\n>\n> For example, the file Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1 has not been deleted in A or B, therefore git should not have attempted to rename Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 to Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1, especially as it then attempts to rename Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 to Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 and so on.\n\nRenamed files, from Git's perspective, always involve files that have\nbeen deleted.\n\n> Git status contains, for example:\n> deleted by them: Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n\nThis means that it wasn't sufficiently similar to any of the new\nfiles...or that _other_ deleted files were more similar to the new\nfiles and thus that they were paired up instead of this file, leaving\nthis file to simply be marked as deleted. (Or that other deleted\nfiles were just as similar; tie-breakers are kinda random in such a\ncase.)\n\n[...]\n\n> Anything else you want to add:\n> I can't help but think that this is related to changes made by Palantir:\n> https://blog.palantir.com/optimizing-gits-merge-machinery-1-127ceb0ef2a1\n\nCurious. What makes you think it's related?\n\nIf there is some reason you think it's related, there's an easy way to\ncheck -- just repeat the cherry-pick with the \"-s recursive\" flag to\nuse the old merge backend and compare the results.\n\nI'll be somewhat surprised if it's related, though.\n\n> I have tried to unstage these renames using \"git restore --staged \" so I can then apply the correct \"git mv\" commands\n\nWhy? Just modify all the files to have the correct end results and then commit.\n\n>, but bizzarely, this then results in \"git status\" reporting a different, smaller set of mismatched names:\n\nAs mentioned earlier, git does _not_ record renames. So, running the\ncorrect \"git mv\" command doesn't really mean much. If you use\ncompletely \"incorrect\" git-mv commands, but then manually tweak files\nuntil they have the correct results, then what's recorded is exactly\nthe same as if you had used the \"correct\" git-mv commands.\n\nFurther, when you run \"git status\", it can't access any renames you\ndid because that information isn't recorded anywhere. It instead\nrecomputes renames on the fly. And it does so each and every time you\nrun \"git status\", even if you make no changes between two invocations.\nIn fact, from this you can probably also deduce that there are other\nways to affect what will be shown as renames, when you have multiple\nfiles similar to any given source file. In particular, you can cause\na different pairing modifying one of the similar files enough that it\nbecomes the most similar to the source file, or so that it becomes no\nlonger the most similar to the source file. However, what \"git\nstatus\" reports for renames is irrelevant, since that info won't be\nrecorded in the commit. Renames are never recorded. Anywhere.\n\nIn fact, you can even run \"git status --no-renames\" to just see the\nold filenames that were removed and the new ones that were added\nwithout having all the files be paired up as renames.\n\n\n\nHope that helps,\nElijah\n"},{"id":"484708","messageId":"LO6P265MB6736F5F9E8368A9DE95D294FFAA9A@LO6P265MB6736.GBRP265.PROD.OUTLOOK.COM","threadId":"60476","inReplyTo":"CABPp-BHYaxa7QoXabM=7hW-93hQLK-=KayGtDHtWxxdAnJCcJw@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Jeremy Pridmore","fromEmail":"jpridmore@rdt.co.uk","sentAt":"2023-11-10T11:28:00Z","receivedAt":"2023-11-10T11:28:07Z","isPatch":false,"sender":{"key":"jpridmore@rdt.co.uk","avatar":null},"body":"Hi Elijah,\n\nMany thanks for your reply, the detail is much appreciated. I was aware, from recently read articles, that git doesn't record renames as such, hence my investigations into the rename detection, but I also found some interesting points in your email, such as the \"git status --no-renames\" flag.\n\nI think the issue I'm encountering is described by what you say here:\n\"Exact renames are detected first, before any other method of rename\ndetection is employed, and other than giving a preference to files\nwith the same basename, if there are multiple choices it just picks\none (what I'd call at random, though technically based on what the\ninternal processing order happens to be)\"\n\nThat is close to the behaviour I'm seeing. As I mentioned, git seems to think a file has been deleted and then as it continues to detect renames, it's as if it is going through lists of \"Local-Base\" and \"Base-Remote\" changes trying to match them up, but the directories of the files being matched are \"offset\", as highlighted by this list of mismatches:\n\n(I'd put the paths in a table for easier analysis, but for some reason the emails need to be plain text)\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n> Incorrect name match: Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n> Incorrect path match: Landscape/Documentation/uiDocumentation/licenses.licx -> Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Import/uiImport/My Project/licenses.licx -> Landscape/src/Documentation/uiDocumentation/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain.Workflow/My Project/licenses.licx -> Landscape/src/Import/uiImport/My Project/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain/My Project/licenses.licx -> Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\n\nGiven git compares both the content and the directory\\filenames, and as the repositories have unrelated histories, the \"Base\" file is going to be empty, therefore, even if Local and Remote are identical, they are both 100% different to Base. That given, I'm not sure why git would state that Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx and Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 are a \"both added\" conflict given their file names and paths are completely different. Any ideas?\n\nI wrote a script to resolve the conflicts best I can which categorises the files into sets according to the file status (i.e. \"added by them\", \"added by us\" etc), and then either does a \"git checkout head -- \" or a \"git rm \" based upon which set the file is in and whether it is in another set or not. This has worked really well and helped me through the large changeset with 3k conflicts.\n\nAs git only needs to try and match files in the \"deleted by us\" and \"deleted by them\" sets (although including the \"deleted in both\" set would allow matching renames/moves on both sides), an idea for a potential improvement to the matching algorithm (where you say there's a comment \"too many alternatives, pick one\") could be to compute a \"difference value\" for the path\\filename of those files in one of the other sets (i.e. \"added by us\", \"added by them\" or \"added in both\"), and chose a potential rename based upon the smallest calculated difference. The difference value would be the number of differences in folder names, e.g.\n\ndeleted in both: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n\nadded in both: Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n(path\\name difference = 2)\nadded in both: Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n(path\\name difference = 1)\nadded in both: Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n(path\\name difference = 2)\n\nSo, given the above, git would chose the second \"added in both\" entry.\n\nFood for thought? Happy to discuss the idea further.\n\nRegards,\n\nJeremy Pridmore\nLead Solution Architect\n\n\nFrom: Elijah Newren \nSent: 07 November 2023 08:05\nTo: Jeremy Pridmore \nCc: git@vger.kernel.org ; Paul Baumgartner \nSubject: Re: Git Rename Detection Bug\n\n[You don't often get email from newren@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]\n\nCaution: This email originated from outside of the organisation. Please treat any attachments or links with caution. If in doubt please contact IT\n\n\nHi,\n\nOn Mon, Nov 6, 2023 at 4:01 AM Jeremy Pridmore wrote:\n>\n> Thank you for filling out a Git bug report!\n> Please answer the following questions to help us understand your issue.\n\nI think it might be worthwhile to point out a few facts about rename\nhandling in Git, as background information that might clarify a few\nthings about how Git's mental model seems to differ from yours:\n\n * In git, renames are not tracked; they are detected (based on file\nsimilarity of the commits being compared).\n * So, when you run \"git mv A B\", there is no rename recorded. It's\nbasically the same as \"git rm A\", followed by creating B with the same\ncontents, followed by \"git add B\".\n * The detection happens whenever an operation (diff, log -p, merge,\nstatus, etc.) needs or wants to know about renames.\n * In git, directory renames are detected _after_ normal renames, and\nvia amalgamation of the individual renames.\n * As a corollary of the last item, the only way individual renames\ncan be affected by directory renames, is if the individual rename on\none side of history was into a directory that the other side of\nhistory renamed away; in such a case, we apply an _extra_ rename to\nmove it into the new directory. But we don't \"undo\" individual\nrenames to make them fit the majority-determined directory rename or\nanything like that.\n\n(Also, if it matters, all of this is true of both `recursive` and\n`ort` merge strategies, i.e. the old default merge backend and the new\none.)\n\n> What did you do before the bug happened? (Steps to reproduce your issue)\n> I have two GIT repositories (A and B). Both migrated from the same TFS server using git-tfs tool. I migrated code into A and made lots of changes, including moving 50,000+ files from folder \"/Landscape\" to \"/Landscape/src\". B contains the same code but with various other changes made since my original migration from TFS to A. All the files in B are still in the \"/Landscape\" folder. I recently needed to merge my changes from A to B, so I added A as a remote to B and then performed a number of cherry-picks from A to B, but got stuck when trying to cherry-pick the commit containing the results of moving all files into \"/Landscape/src\".\n\nIn case anyone else wants to dig into this, note that this problem\nsetup precludes directory rename detection being involved. Directory\nrename detection has a rule where if the source directory wasn't\nentirely removed on one side, then that directory was not renamed on\nthat side. Seems obvious, but the upshot of that rule is that a\ndirectory cannot be renamed into a subdirectory of itself, because by\nvirtue of being a subdirectory that means its parent directory still\nexists.\n\nSo, this is a problem where only regular rename detection (i.e. rename\ndetection of individual files) is going to be at play.\n\n> What did you expect to happen? (Expected behavior)\n> I expected the git rename detection to match all files in A \"/Landscape\" to files in B \"/Landscape/src\".\n\nAre all files under \"/Landscape\" from the merge base commit\nindividually more similar to the counterpart under \"/Landscape/src\"\nthan to files under any other directory? If not, the expectation goes\nagainst how rename detection has worked in git from the beginning.\n\n> What happened instead? (Actual behavior)\n> Although many files were matched successfully, git mismatched over two dozen similarly named files, e.g.\n>\n> Incorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IAccountsIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IAccountsIntegration.vb\n> Incorrect path match: Landscape/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb -> Landscape/src/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n> Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n> Incorrect name match: Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n> Incorrect path match: Landscape/Documentation/uiDocumentation/licenses.licx -> Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Import/uiImport/My Project/licenses.licx -> Landscape/src/Documentation/uiDocumentation/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain.Workflow/My Project/licenses.licx -> Landscape/src/Import/uiImport/My Project/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain/My Project/licenses.licx -> Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\n> Incorrect path match: Landscape/LandscapeApiService.Setup/Setup/UIContent/RDT_Logo.ico -> Landscape/src/Main/uiMain.Workflow/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx -> Landscape/src/Main/uiMain/My Project/licenses.licx\n> Incorrect path match: Landscape/Main/uiMain.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Main/uiMain/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Policy/Rdt.Policy.UI/Properties/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI.Templates/Properties/licenses.licx\n> Incorrect path match: Landscape/Rates/uiRates/My Project/licenses.licx -> Landscape/src/Policy/Rdt.Policy.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Claim.UI/Properties/licenses.licx -> Landscape/src/Rates/uiRates/My Project/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Claim.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Templates/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates.Workflow/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI.Workflow/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Templates/Properties/licenses.licx\n> Incorrect path match: Landscape/Rdt.Landscape.UI/Properties/licenses.licx -> Landscape/src/Rdt.Landscape.UI.Workflow/Properties/licenses.licx\n> Incorrect path match: Landscape/StandardLetters/uiStandardLetters/My Project/licenses.licx -> Landscape/src/Rdt.Landscape.UI/Properties/licenses.licx\n> Incorrect path match: Landscape/Complaints/Rdt.Complaints.UI/Interfaces/IDocumentIntegration.vb -> Landscape/src/Services/uiServices/Complaints/Interfaces/IDocumentIntegration.vb\n> Incorrect path match: Landscape/SystemEvents/uiSystemEvents/My Project/licenses.licx -> Landscape/src/StandardLetters/uiStandardLetters/My Project/licenses.licx\n> Incorrect path match: Landscape/Services/busServices/RDT_Logo.ico -> Landscape/src/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup.Workflow/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup/Resources/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup/Resources/RDT_Logo.ico -> Landscape/src/Startup/uiStartup32/RDT_Logo.ico\n> Incorrect path match: Landscape/Startup/uiStartup/Resources/newrdlogogradiant48shad.ico -> Landscape/src/Startup/uiStartup32/newrdlogogradiant48shad.ico\n> Incorrect path match: Landscape/Templates/uiTemplates.Workflow/My Project/licenses.licx -> Landscape/src/SystemEvents/uiSystemEvents/My Project/licenses.licx\n> Incorrect path match: Landscape/Utils/Rdt.Utils.UI/Properties/licenses.licx -> Landscape/src/Templates/uiTemplates.Workflow/My Project/licenses.licx\n> Incorrect path match: Landscape/Utils/uiUtils/My Project/licenses.licx -> Landscape/src/Utils/Rdt.Utils.UI/Properties/licenses.licx\n> Incorrect name match: Landscape/WebServices/ServiceFabric/Policy/Rdt.Policy.Repository.Service.Fabric.Host/PackageRoot/Data/Swagger/Examples/POST_UKSTasks_Response.json -> Landscape/src/Utils/uiUtils/My Project/licenses.licx\n>\n>\n> What's different between what you expected and what actually happened?\n>\n> As you can see, although the filenames (and content) are the same,\n\nThe content is the same as well? So, these renames that you label as\nincorrect are actually _exact_ renames -- and further, in most cases\nthey also have an identical basename for the file as well.\n\nExact renames are detected first, before any other method of rename\ndetection is employed, and other than giving a preference to files\nwith the same basename, if there are multiple choices it just picks\none (what I'd call at random, though technically based on what the\ninternal processing order happens to be; see the \"Too many identical\nalternatives? Pick one\" code comment).\n\nAnd this, too, is true of both the `recursve` and `ort` backends; no\nchange has been made to how exact renames are handled.\n\n> In some cases, it seems that the catalyst has been git thinking that a file from B has been deleted from A, when in fact it has not actually been deleted at all.\n>\n> For example, the file Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1 has not been deleted in A or B, therefore git should not have attempted to rename Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 to Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1, especially as it then attempts to rename Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 to Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 and so on.\n\nRenamed files, from Git's perspective, always involve files that have\nbeen deleted.\n\n> Git status contains, for example:\n> deleted by them: Landscape/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n\nThis means that it wasn't sufficiently similar to any of the new\nfiles...or that _other_ deleted files were more similar to the new\nfiles and thus that they were paired up instead of this file, leaving\nthis file to simply be marked as deleted. (Or that other deleted\nfiles were just as similar; tie-breakers are kinda random in such a\ncase.)\n\n[...]\n\n> Anything else you want to add:\n> I can't help but think that this is related to changes made by Palantir:\n> https://blog.palantir.com/optimizing-gits-merge-machinery-1-127ceb0ef2a1\n\nCurious. What makes you think it's related?\n\nIf there is some reason you think it's related, there's an easy way to\ncheck -- just repeat the cherry-pick with the \"-s recursive\" flag to\nuse the old merge backend and compare the results.\n\nI'll be somewhat surprised if it's related, though.\n\n> I have tried to unstage these renames using \"git restore --staged \" so I can then apply the correct \"git mv\" commands\n\nWhy? Just modify all the files to have the correct end results and then commit.\n\n>, but bizzarely, this then results in \"git status\" reporting a different, smaller set of mismatched names:\n\nAs mentioned earlier, git does _not_ record renames. So, running the\ncorrect \"git mv\" command doesn't really mean much. If you use\ncompletely \"incorrect\" git-mv commands, but then manually tweak files\nuntil they have the correct results, then what's recorded is exactly\nthe same as if you had used the \"correct\" git-mv commands.\n\nFurther, when you run \"git status\", it can't access any renames you\ndid because that information isn't recorded anywhere. It instead\nrecomputes renames on the fly. And it does so each and every time you\nrun \"git status\", even if you make no changes between two invocations.\nIn fact, from this you can probably also deduce that there are other\nways to affect what will be shown as renames, when you have multiple\nfiles similar to any given source file. In particular, you can cause\na different pairing modifying one of the similar files enough that it\nbecomes the most similar to the source file, or so that it becomes no\nlonger the most similar to the source file. However, what \"git\nstatus\" reports for renames is irrelevant, since that info won't be\nrecorded in the commit. Renames are never recorded. Anywhere.\n\nIn fact, you can even run \"git status --no-renames\" to just see the\nold filenames that were removed and the new ones that were added\nwithout having all the files be paired up as renames.\n\n\n\nHope that helps,\nElijah\n\n________________________________\n\nDISCLAIMER This email is confidential. It should only be read by those persons to whom it is addressed. RDT Ltd accept no liability for the consequences of any person acting, or refraining from acting, on any information contained within this e-mail or any attached documents prior to the receipt by those persons of subsequent written confirmation of that information. If you think this e-mail may not be intended for you, do not use, pass on or copy the transmission in any way. While all reasonable precautions are taken to minimise the risk of transmitting software viruses we advise you to carry out your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses.\n"},{"id":"484749","messageId":"CABPp-BHEX+SyophEfgRqDbNdrAS3=bptt_cKzHLBSutnBAxexw@mail.gmail.com","threadId":"60476","inReplyTo":"LO6P265MB6736F5F9E8368A9DE95D294FFAA9A@LO6P265MB6736.GBRP265.PROD.OUTLOOK.COM","subject":"Re: Git Rename Detection Bug","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-11T05:46:00Z","receivedAt":"2023-11-11T05:47:46Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Jeremy,\n\nOn Fri, Nov 10, 2023 at 3:28 AM Jeremy Pridmore wrote:\n>\n> Hi Elijah,\n>\n> Many thanks for your reply, the detail is much appreciated. I was aware, from recently read articles, that git doesn't record renames as such, hence my investigations into the rename detection, but I also found some interesting points in your email, such as the \"git status --no-renames\" flag.\n\nThe fact that you were trying to \"undo\" renames and \"redo the correct\nones\" suggested there's something you still didn't understand about\nrename detection, though. The fact that you were worried that \"git\nstatus\" showing the \"wrong renames\" and the implication that it needed\nto show the right ones before you committed also suggested there's\nsomething that was still not being understood. Likewise, the fact\nthat the renames reported by \"git status\" change even when you haven't\nrenamed files further but have simply made additional changes to the\ncontents of some files suggests there is something that was still not\nbeing understood about the phrase that \"renames are detected rather\nthan recorded\".\n\nWhile renames are used in the merge algorithm in order to know what\nfiles to match up for three-way content merges (so that changes from\nboth sides to the \"same\" file can be incorporated into the end\nresult), once the merge stops to ask you to resolve conflicts, the\ndetection of renames doesn't matter beyond that point. The fact that\nrenames aren't recorded means whatever renames are shown is only there\nas a guide to help you understand. All that matters to Git is that\nall files have the intended content. If some of the files have the\nwrong content, then by all means go and correct it. If \"git mv\"\ncommands help you do that, great. If simply editing all files of\ninterest (including adding and deleting files) until they match the\nexpected contents works, that's fine too. Once all files have the\ncorrect content, commit it. Git will have no way whatsoever of\nknowing which of those two routes you picked, and won't behave any\ndifferently in the future based on which way you ended up with the\nright contents in each file. You could have even done the extreme of\n\"git merge -s ours --no-commit ${OTHER_BRANCH}\" (merge the other\nbranch but completely ignore *every* change the other side made and\nthen stop for user to make further changes), followed by opening and\nediting every relevant file to include changes made by the other side,\ndeleting and adding files as relevant, to end up with the right\ncontents in all files and then committed, and Git wouldn't know the\ndifference and no one who pulled your merge commit would be able to\ntell the difference either.\n\n> I think the issue I'm encountering is described by what you say here:\n> \"Exact renames are detected first, before any other method of rename\n> detection is employed, and other than giving a preference to files\n> with the same basename, if there are multiple choices it just picks\n> one (what I'd call at random, though technically based on what the\n> internal processing order happens to be)\"\n>\n> That is close to the behaviour I'm seeing. As I mentioned, git seems to think a file has been deleted and then as it continues to detect renames, it's as if it is going through lists of \"Local-Base\" and \"Base-Remote\" changes trying to match them up, but the directories of the files being matched are \"offset\", as highlighted by this list of mismatches:\n>\n> (I'd put the paths in a table for easier analysis, but for some reason the emails need to be plain text)\n> > Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n> > Incorrect path match: Landscape/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 -> Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n> > Incorrect name match: Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx -> Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n> > Incorrect path match: Landscape/Documentation/uiDocumentation/licenses.licx -> Landscape/src/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\n> > Incorrect path match: Landscape/Import/uiImport/My Project/licenses.licx -> Landscape/src/Documentation/uiDocumentation/licenses.licx\n> > Incorrect path match: Landscape/Main/uiMain.Workflow/My Project/licenses.licx -> Landscape/src/Import/uiImport/My Project/licenses.licx\n> > Incorrect path match: Landscape/Main/uiMain/My Project/licenses.licx -> Landscape/src/Main/uiMain.Workflow/My Project/licenses.licx\n>\n> Given git compares both the content and the directory\\filenames,\n\nFor exact renames, it only will look at filenames if there are\nmultiple 100% matches. If one match is 100%, and the other 99.9999%\nmatch, the 100% match is taken.\n\n(Since we think that exact renames seem to be describing your problem,\nthere's no point discussing the inexact rename handling.)\n\n> and as the repositories have unrelated histories,\n\nWhat do you mean by this? If there's no common point in history (i.e.\nno merge base), rename detection doesn't even get invoked, so you must\nmean something different by this phrase than what I would normally\ntake it to mean.\n\nAny chance, if you're still in the middle of the merge, that you could run\n git merge-base --all HEAD MERGE_HEAD\nand report the output? It'll be commit hashes that don't mean a lot\nto anyone who doesn't have your repository, but I'm interested in how\nmany commit hashes it responds with (0, 1, or more than 1).\n\n> the \"Base\" file is going to be empty, therefore, even if Local and Remote are identical, they are both 100% different to Base.\n\nI assume by \"Base\" you are referring to the commit in history common\nto both branches being merged (i.e. what we call the \"merge base\"), or\nto the relevant file from that commit. Is that right?\n\nEven if I'm right about that, this comment has me lost. Could you\nclarify it, with one particular example? For example (I'm making\nstuff up since I'm not familiar with your repo, but showing how to\nclarify for a given set of path(s)):\n\n * In this repository, with the renames detected above,\nLandscape/Main/somefile.licx is an empty file in the merge base.\n * On my local side, I renamed Landscape/Main/somefile.licx to\nLandscape/src/Main/somefile.licx and populated it with some content\n(with hash A). There is also another new file on the local side (at\nleast new relative to the merge base) named\nLandscape/src/Other/somefile.licx that happens to have hash A.\n * On the remote side, Landscape/Main/somefile.licx was left in\nplace but populated with some content (with hash A).\n * Git is detecting the rename as Landscape/Main/somefile.licx ->\nLandscape/src/Other/somefile.licx, when I wanted it to detect a rename\nto Landscape/src/Main/somefile.licx.\n\nI'm pretty sure this example is not what you're seeing, even if\ncomponents of it are, because the empty file thing is impossible with\nthe rest of the story.\n\n> That given, I'm not sure why git would state that Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx and Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1 are a \"both added\" conflict given their file names and paths are completely different. Any ideas?\n\nThe fact that Landscape/Documentation/Rdt.Documentation.UI/Properties/licenses.licx\nis marked as \"both added\" means that this file\n * did not exist in the merge base\n * did exist on your local side\n * did exist on the remote side\n * the version of this file on the local and remote sides do not match\nBasically, both sides added a new file and they are not the same, so\nyou have a conflict.\n\nThe answer for Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1,\nis an identical set of bullet points; it was a file added by both\nsides of history that did not exist in the merge base, and the sides\nare different so you have a conflict in this file too.\n\n> I wrote a script to resolve the conflicts best I can which categorises the files into sets according to the file status (i.e. \"added by them\", \"added by us\" etc), and then either does a \"git checkout head --\n\" or a \"git rm \" based upon which set the file is in and\nwhether it is in another set or not. This has worked really well and\nhelped me through the large changeset with 3k conflicts.\n\nSo, you're simply throwing away the changes made by the remote side?\nI mean, that's one way to merge, and it might be right in your case,\nbut to someone unfamiliar with your repo it smells like a hack to just\nignore conflicts and throw away other people's changes in order to\ncomplete the merge.\n\n> As git only needs to try and match files in the \"deleted by us\" and \"deleted by them\" sets (although including the \"deleted in both\" set would allow matching renames/moves on both sides),\n\n\"deleted by us\" and \"deleted by them\" means no rename was detected for\nthe file (at least on the side that the delete is reported for). So,\na \"deleted in both\" only happens when neither side detects a rename,\nand if the file isn't renamed on either side and both removed it, then\nthere's no conflict -- just delete the file.\n\n> an idea for a potential improvement to the matching algorithm (where you say there's a comment \"too many alternatives, pick one\") could be to compute a \"difference value\" for the path\\filename of those files in one of the other sets (i.e. \"added by us\", \"added by them\" or \"added in both\"), and chose a potential rename based upon the smallest calculated difference. The difference value would be the number of differences in folder names, e.g.\n>\n> deleted in both: Landscape/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n>\n> added in both: Landscape/src/Deployment/PowershellScripts/pre-req/Landscape.Net/pre-req.ps1\n> (path\\name difference = 2)\n> added in both: Landscape/src/Deployment/PowershellScripts/pre-req/Rdt.BatchProcessingService.Setup/pre-req.ps1\n> (path\\name difference = 1)\n> added in both: Landscape/src/Deployment/PowershellScripts/pre-req/Workflow/pre-req.ps1\n> (path\\name difference = 2)\n>\n> So, given the above, git would chose the second \"added in both\" entry.\n\nThree problems here:\n\n* If git were to report \"deleted in both\" for one path and \"added in\nboth\" in another as you suggest, that would only be because the files\nare dissimilar. Not only would they not be an exact match, their\ncontents would have less than 50% similarity. Thus augmenting exact\nmatching like this just wouldn't work, because the files aren't\nmatches at all. The only correct thing to do would be to not report\nthe \"deleted in both\" because that file is not a rename of anything\nelse and has simply been deleted by both sides.\n\n* filename similarity is extraordinarily expensive compared to exact\nrenames, and if not carefully handled, can sometimes rival the cost of\nfile content similarity computations given our spanhash\nrepresentations. Exact renames are tasked with finding renames even\nif they are known to not be relevant, simply because exact renames can\ndo so very quickly. If we change that, we throw a monkey wrench in\nour performance handling elsewhere and have to rethink a number of\nother things.\n\n* While I was optimizing rename detection while investigating the new\nmerge backend, I actually attempted a few versions of filename\nsimilarity looking for something that was predictive and useful.\nWhile I think the idea was potentially helpful for some repositories,\nit has a significant risk of hurting merges in other repositories.\nWhile what I tried was far from an exhaustive checking of all filename\nsimilarity ideas, I came away doubting there was a useful heuristic\nother than exact matches of basenames (i.e. exact matches of\neverything in the filename after the final slash). If someone else\nwants to try more ideas, and do a study on various existing\nrepositories, they can go ahead, but I suspect most work here is going\nto end up at a dead end and I'm unwilling to put further time of my\nown into it.\n\n> Food for thought? Happy to discuss the idea further.\n\nSo, I've occasionally seen repositories that have something like the following:\n\n * base version: directory named library-x-1.7/\n * stable branch: many changes to files under library-x-1.7/\n * development branch: library-x-1.7/ no longer exists. However,\nlibrary-x-1.8/ and library-x-1.9/ both do. Both are obviously\n\"similar\" to library-x-1.7/ but both have many changes.\n\nWhat happens when someone tries to merge the stable branch into the\ndevelopment branch? There are two obvious guesses:\n\n * Changes from library-x-1.7/ on the stable branch are applied to\nlibrary-x-1.8/\n * Changes from library-x-1.7/ on the stable branch are applied to\nlibrary-x-1.9/\n\nEither answer can be suboptimal depending on your viewpoint.\n(Applying the changes to both directories would also have other\nsuboptimal effects even if it might sound right based for this exact\nproblem as I've worded it. But git doesn't do copy detection as part\nof merges so Git won't choose this third choice ever.) So, which of\nthose two happens? Well, since renames are detected based upon file\nsimilarity, the changes will go to whatever file is most similar.\nWhat does that mean? It means that both answers above are wrong.\nInstead:\n\n * Some of the changes from library-x-1.7/ on the stable branch are\napplied to files from library-x-1.8/, while others are applied to\nfiles from library-x-1.9/, and to determine which files from which\ndirectory are matched up is an individual file choice based on which\nfile in library-x-1.8/ or library-x-1.9/ is most similar to the file\nfrom the base version in library-x-1.7/.\n\nThis answer is clearly worse than either of the two above, and is\nvirtually never what people would want. But it's also fundamental to\nthe idea of matching up files and detecting renames individually based\nupon file similarity. It's part of both the old and new merge\nbackends in Git, because both were based upon this fundamental idea.\n\nSo, if you have this kind of situation, or even something like it\nwhere files from one old directory could match files from multiple\nother directories, it's just something you have to be aware of.\n\nAll that said, here's something that might help:\n\n*** Hack to workaround rename detection in special cases where there\nare directories of multiple possible matches ***\n\n1. Get back to a clean slate from before the merge.\n\n$ git merge --abort\nOR\n$ cd ${OTHER_DIRECTORY} && git clone ${url} && cd ${REPONAME} && git\ncheckout ${relevant_branch}\n\n2. Temporarily undo your local renames and make a temporary commit\n\n$ git mv Landscape/src/* Landscape/\n$ git commit -m \"TEMPORARY COMMIT\"\n\n3. Perform the merge (files will be in Landscape/ instead of\nLandscape/src/ for now). Don't worry, we'll fix the merge commit\nlater.\n\n$ git merge ${REMOTE_BRANCH}\n[...fix up any conflicts and commit, if needed...]\n\n4. Rename Landscape/ back to Landscape/src/ and make another (temporary) commit.\n\n5. Create a corrected merge commit with the current tree, the commit\nmessage from your merge commit, and the correct parents:\n$ git commit-tree -p HEAD~3 -p HEAD~1^2 -F $(git log -1 --format=%B\nHEAD~1) HEAD^{tree}\n[...the above command will print out a new commit id for a corrected\nmerge commit. You can inspect it first, but we just need to pass this\nto reset --hard...]\n\n6. Reset your branch to this corrected merge commit (which will orphan\nthe temporary commits from steps 2, 3, and 4 so they can later be\ngarbage collected)\n$ git reset --hard [...output of commit-tree command...]\n\n\nHope that helps,\nElijah\n"},{"id":"484761","messageId":"CABPp-BEtva2WTGQG3Qs4EbZLK_RJC9vuA-2OYxkTPExgowwvqQ@mail.gmail.com","threadId":"60476","inReplyTo":"9baca4af-a570-4b7a-a1ee-de91b809e79c@iee.email","subject":"Re: Git Rename Detection Bug","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-11T15:13:28Z","receivedAt":"2023-11-11T15:13:44Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Sat, Nov 11, 2023 at 3:08 AM Philip Oakley wrote:\n>\n> Hi all,\n>\n> On 11/11/2023 05:46, Elijah Newren wrote:\n> > The fact that you were trying to \"undo\" renames and \"redo the correct\n> > ones\" suggested there's something you still didn't understand about\n> > rename detection, though.\n>\n>\n> Could I suggest that we are missing a piece of terminology, to wit,\n> BLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\n> history simplification (based on a tree's pathspec, most commonly a\n> commit's top level path).\n\nWe could add it, but I'm not sure how it helps. We already had 'exact\nrename' which seems to fit the bill as well, and 'blob' is something\nsomeone new to Git is unlikely to know.\n\nPerhaps it's useful in some other context, though?\n\n> File rename, at it's most basic, is when the blob associated with that\n> changed path is identical, i.e. BLOBSAME. There is no need to 'record'\n> the action of renaming, moving or whatever, the content sameness is\n> right there, in plain sight, as an identical blob name. After that\n> (files with slight variations) it is a load of heuristics, but starting\n> with BLOBSAME we see how easy the basic rename detection is, and why\n> renames (and de-dup) don't need recording.\n\nThis is incorrect. Let's say you have a file foo:\n * base version: foo has hash A\n * our version: foo has been renamed to bar, but bar still has hash A\n * their version: foo has been modified; it now has hash B\n\nThe foo->bar is an exact rename (or they are BLOBSAME if you prefer),\nbut the renaming/moving/whatever is a critical piece of information\nbecause the changes to foo in 'their' version need to be applied to\nbar to get the correct end results.\n\nI do not know if in Jeremy's case foo has been modified on the\nunrenamed side. But the following hypothetical is exactly the type of\nproblem Jeremy is hitting: what should happen when 'our' version has\nboth a new 'bar' and a new 'baz' file that each have hash A? In that\ncase, to which one was foo renamed? It's inherently ambiguous.\n\n> The heuristics of 'rename with small change' is trickier, but for a\n> basic understanding, starting at BLOBSAME (and TREESAME for directory\n> renames) should make it easier to grasp the concepts.\n\nInteresting; TREESAME isn't used within directory rename detection\ncurrently; it is only used currently when two (or three) trees with\nthe same name are TREESAME, in order to potentially avoid recursing\ninto the tree. But even then, having two trees with the same name be\nTREESAME isn't enough on its own to avoid recursing into that tree,\nbecause the other side could have added files within the same-named\ntree and we need to know about those added files because they could be\npart of renames involving other files outside that tree. There would\nprobably be similar challenges to attempting to apply the concept of\nTREESAME to directory rename detection to two trees of different\nnames, but it's at least an interesting idea. Hmm....\n"},{"id":"484763","messageId":"9baca4af-a570-4b7a-a1ee-de91b809e79c@iee.email","threadId":"60476","inReplyTo":"CABPp-BHEX+SyophEfgRqDbNdrAS3=bptt_cKzHLBSutnBAxexw@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-11-11T11:08:10Z","receivedAt":"2023-11-11T16:54:11Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi all,\n\nOn 11/11/2023 05:46, Elijah Newren wrote:\n> The fact that you were trying to \"undo\" renames and \"redo the correct\n> ones\" suggested there's something you still didn't understand about\n> rename detection, though.\n\n\nCould I suggest that we are missing a piece of terminology, to wit,\nBLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\nhistory simplification (based on a tree's pathspec, most commonly a\ncommit's top level path).\n\nFile rename, at it's most basic, is when the blob associated with that\nchanged path is identical, i.e. BLOBSAME. There is no need to 'record'\nthe action of renaming, moving or whatever, the content sameness is\nright there, in plain sight, as an identical blob name. After that\n(files with slight variations) it is a load of heuristics, but starting\nwith BLOBSAME we see how easy the basic rename detection is, and why\nrenames (and de-dup) don't need recording.\n\nThe heuristics of 'rename with small change' is trickier, but for a\nbasic understanding, starting at BLOBSAME (and TREESAME for directory\nrenames) should make it easier to grasp the concepts.\n\n--\n\nPhilip\n"},{"id":"484777","messageId":"xmqqzfzimuv2.fsf@gitster.g","threadId":"60476","inReplyTo":"CABPp-BEtva2WTGQG3Qs4EbZLK_RJC9vuA-2OYxkTPExgowwvqQ@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-11-12T23:09:53Z","receivedAt":"2023-11-12T23:10:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> Could I suggest that we are missing a piece of terminology, to wit,\n>> BLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\n>> history simplification (based on a tree's pathspec, most commonly a\n>> commit's top level path).\n>\n> We could add it, but I'm not sure how it helps. We already had 'exact\n> rename' which seems to fit the bill as well, and 'blob' is something\n> someone new to Git is unlikely to know.\n\nAlso, as Philip said, TREESAME is a concept foreign to rename\ndetection codepath. It is a property of a commit (not a tree) and\ntells us if it has the same tree object as its relevant parents (in\nwhich case it can be simplified away if it is a merge). I do not\nmind rename codepath using a jargon (or two) to express \"in trees A\nand B, this subtree of A records the same tree object as a subtree\nof B at a different path (i.e., the contents of these two subtrees\nat different paths are the same)\" but the word used to express that\nshould not be TREESAME to avoid confusion. And the other word to\nexpress \"this path in tree A records a blob object that is identical\nto this other path in tree B\" should not be BLOBSAME, as the word\nstrongly hints it is somehow related to TREESAME.\n\nThanks.\n\n"},{"id":"484934","messageId":"781fc667-6597-4327-80d5-721fb273d2e2@iee.email","threadId":"60476","inReplyTo":"CABPp-BEtva2WTGQG3Qs4EbZLK_RJC9vuA-2OYxkTPExgowwvqQ@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-11-15T14:36:27Z","receivedAt":"2023-11-15T14:36:31Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Elijah,\nsorry for the delay in replying.\n\nOn 11/11/2023 15:13, Elijah Newren wrote:\n> Hi,\n> \n> On Sat, Nov 11, 2023 at 3:08 AM Philip Oakley wrote:\n>>\n>> Hi all,\n>>\n>> On 11/11/2023 05:46, Elijah Newren wrote:\n>>> The fact that you were trying to \"undo\" renames and \"redo the correct\n>>> ones\" suggested there's something you still didn't understand about\n>>> rename detection, though.\n>>\n>>\n>> Could I suggest that we are missing a piece of terminology, to wit,\n>> BLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\n>> history simplification (based on a tree's pathspec, most commonly a\n>> commit's top level path).\n> \n> We could add it, but I'm not sure how it helps. We already had 'exact\n> rename' which seems to fit the bill as well,\n\nMy point was that we already had the confusion of mental models, with\nboth sides essentially thinking they had an \"exact rename\", hence my\nthought was to add a rather distinct technical name which reflected the\nGit mind-shift. Without something to bring folks up short they'll\ncontinue, erroneously, with their prior mental models.\n\n\n and 'blob' is something\n> someone new to Git is unlikely to know.\n\nI'd agree that BLOBSAME is new, but we should be proactive in ensuring\nfolk do have the mind shift from the old centralised VCS misunderstandings.\n\n> \n> Perhaps it's useful in some other context, though?\n> \n>> File rename, at it's most basic, is when the blob associated with that\n>> changed path is identical, i.e. BLOBSAME. There is no need to 'record'\n>> the action of renaming, moving or whatever, the content sameness is\n>> right there, in plain sight, as an identical blob name. After that\n>> (files with slight variations) it is a load of heuristics, but starting\n>> with BLOBSAME we see how easy the basic rename detection is, and why\n>> renames (and de-dup) don't need recording.\n> \n> This is incorrect. Let's say you have a file foo:\n> * base version: foo has hash A\n> * our version: foo has been renamed to bar, but bar still has hash A\n> * their version: foo has been modified; it now has hash B\n> \n> The foo->bar is an exact rename (or they are BLOBSAME if you prefer),\n> but the renaming/moving/whatever is a critical piece of information\n> because the changes to foo in 'their' version need to be applied to\n> bar to get the correct end results.\n\nIsn't that what I thought I'd said?\nHash A = Hash A => identical content;\nHash A != B => different content.\n\n> \n> I do not know if in Jeremy's case foo has been modified on the\n> unrenamed side. But the following hypothetical is exactly the type of\n> problem Jeremy is hitting: what should happen when 'our' version has\n> both a new 'bar' and a new 'baz' file that each have hash A? In that\n> case, to which one was foo renamed? It's inherently ambiguous.\n\ntrue, the terminology hasn't kept up with the methodology for blob\ncontent, and the independent meta-data. In previous 'ort' discussions I\ndidn't really understand what the '1/2' renames (and other\nnomenclatures) really meant with respect to paths, filenames, content\nand the ours / theirs / base distinctions.\n> \n>> The heuristics of 'rename with small change' is trickier, but for a\n>> basic understanding, starting at BLOBSAME (and TREESAME for directory\n>> renames) should make it easier to grasp the concepts.\n> \n> Interesting; TREESAME isn't used within directory rename detection\n> currently; it is only used currently when two (or three) trees with\n> the same name are TREESAME, in order to potentially avoid recursing\n> into the tree. But even then, having two trees with the same name be\n> TREESAME isn't enough on its own to avoid recursing into that tree,\n> because the other side could have added files within the same-named\n> tree and we need to know about those added files because they could be\n> part of renames involving other files outside that tree. \n\ndefinitely easy to get confused on these cases...\n\n> There would\n> probably be similar challenges to attempting to apply the concept of\n> TREESAME to directory rename detection to two trees of different\n> names, but it's at least an interesting idea. Hmm....\n> \n\n\nThanks for the insights.\n\nPhilip\n"},{"id":"484938","messageId":"dc2b2553-419b-4d93-9b72-204838a6011f@iee.email","threadId":"60476","inReplyTo":"xmqqzfzimuv2.fsf@gitster.g","subject":"Re: Git Rename Detection Bug","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-11-15T15:35:33Z","receivedAt":"2023-11-15T15:35:36Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 12/11/2023 23:09, Junio C Hamano wrote:\n> Elijah Newren writes:\n> \n>>> Could I suggest that we are missing a piece of terminology, to wit,\n>>> BLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\n>>> history simplification (based on a tree's pathspec, most commonly a\n>>> commit's top level path).\n>>\n>> We could add it, but I'm not sure how it helps. We already had 'exact\n>> rename' which seems to fit the bill as well, and 'blob' is something\n>> someone new to Git is unlikely to know.\n> \n> Also, as Philip said, TREESAME is a concept foreign to rename\n> detection codepath. It is a property of a commit (not a tree)\n\nThat (property of a commit?) wasn't really obvious to me.\n\nI'd always thought of it as a comparison between two trees, commonly\nthose associated with two commits. Though it could also be thought of as\nthe operation \"TREESAME to\" that binds in the second tree.\n\n and\n> tells us if it has the same tree object as its relevant parents (in\n> which case it can be simplified away if it is a merge). I do not\n> mind rename codepath using a jargon (or two) to express \"in trees A\n> and B, this subtree of A records the same tree object as a subtree\n> of B at a different path (i.e., the contents of these two subtrees\n> at different paths are the same)\" but the word used to express that\n> should not be TREESAME to avoid confusion.\n\nMaybe it's that the explanation of TREESAME in\nrev-list-options.txt#L419-L436 [1] has a similar set of confusions about\nhow subtrees are considered and the path v filename confusions.\n\n And the other word to\n> express \"this path in tree A records a blob object that is identical\n> to this other path in tree B\" should not be BLOBSAME, as the word\n> strongly hints it is somehow related to TREESAME.\n\nYep. Naming is hard.\n\n> \n> Thanks.\n> \n> \nPhilip\n\n[1]\nhttps://github.com/git/git/blob/master/Documentation/rev-list-options.txt#L419-L436\n"},{"id":"484940","messageId":"990ab7d5-e29a-4766-b112-c8908a7ed196@iee.email","threadId":"60476","inReplyTo":"CABPp-BHEX+SyophEfgRqDbNdrAS3=bptt_cKzHLBSutnBAxexw@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-11-15T16:51:53Z","receivedAt":"2023-11-15T16:51:53Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Elijah,\n\nOn 11/11/2023 05:46, Elijah Newren wrote:\n> * filename similarity is extraordinarily expensive compared to exact\n> renames, and if not carefully handled, can sometimes rival the cost of\n> file content similarity computations given our spanhash\n> representations.\n\nI've not heard of spanhash representation before. Any references or\nfurther reading?\n\n> Exact renames are tasked with finding renames even\n> if they are known to not be relevant, simply because exact renames can\n> do so very quickly. If we change that, we throw a monkey wrench in\n> our performance handling elsewhere and have to rethink a number of\n> other things.\n\n--\nPhilip\n"},{"id":"484968","messageId":"CABPp-BFSW+8qzc-bCB-0Mk=cRF8Mduk90Or3N+ZdkdAuhADY4A@mail.gmail.com","threadId":"60476","inReplyTo":"781fc667-6597-4327-80d5-721fb273d2e2@iee.email","subject":"Re: Git Rename Detection Bug","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-11-16T06:26:41Z","receivedAt":"2023-11-16T06:26:56Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Philip,\n\nOn Wed, Nov 15, 2023 at 6:36 AM Philip Oakley wrote:\n>\n> Hi Elijah,\n> sorry for the delay in replying.\n\nIt was only a few days; no need to apologize.\n\n>\n> On 11/11/2023 15:13, Elijah Newren wrote:\n> > Hi,\n> >\n> > On Sat, Nov 11, 2023 at 3:08 AM Philip Oakley wrote:\n> >>\n> >> Hi all,\n> >>\n> >> On 11/11/2023 05:46, Elijah Newren wrote:\n> >>> The fact that you were trying to \"undo\" renames and \"redo the correct\n> >>> ones\" suggested there's something you still didn't understand about\n> >>> rename detection, though.\n> >>\n> >>\n> >> Could I suggest that we are missing a piece of terminology, to wit,\n> >> BLOBSAME. It's a compatriot to TREESAME, as used in `git log` for\n> >> history simplification (based on a tree's pathspec, most commonly a\n> >> commit's top level path).\n> >\n> > We could add it, but I'm not sure how it helps. We already had 'exact\n> > rename' which seems to fit the bill as well,\n>\n> My point was that we already had the confusion of mental models, with\n> both sides essentially thinking they had an \"exact rename\", hence my\n> thought was to add a rather distinct technical name which reflected the\n> Git mind-shift. Without something to bring folks up short they'll\n> continue, erroneously, with their prior mental models.\n\nMaybe I'm missing the other mental model you're alluding to? The\nmental model problems I suspected Jeremy had did not appear to hinge\non exact renames; the fact that Jeremy was trying to \"undo\" renames\nand \"redo the correct ones\" suggested to me that it's an issue with\nunderstanding detecting vs. tracking renames, something for which it\ndoesn't matter whether the renames are exact or inexact. (Also, the\nfact that he thought it was a bug that the detected renames could\nchange in `git status` output by merely editing files, suggested also\na detected vs. tracked rename misunderstanding -- and in that case,\nthere's virtually no chance that an explanation of exact renames or\nBLOBSAME or whatever could help since the editing of files in question\nmeans that they aren't going to have the same contents anymore.)\n\n> and 'blob' is something\n> > someone new to Git is unlikely to know.\n>\n> I'd agree that BLOBSAME is new, but we should be proactive in ensuring\n> folk do have the mind shift from the old centralised VCS misunderstandings.\n\nI agree it's good to help folks make the mind shift, I'm just not\nfollowing how BLOBSAME or any other alternative for \"exact rename\"\ncould help in this particular case.\n\nMaybe it'd help to understand why I brought up \"exact renames\" in the\nfirst place? When most people see \"rename detection bug\" (as in the\nsubject of this thread), they're likely to assume some heuristic went\nhaywire. When Jeremy pointed out that the content of the \"mismatched\"\nfile was identical to the source file of interest, I thought it worth\npointing out so anyone else watching the thread would notice. It\nmeans all the normal heuristics involved in inexact renames cannot be\nthe source of this particular problem. And, in particular, that the\nchanges that ort brought aren't relevant to this particular problem\neither.\n\n> > Perhaps it's useful in some other context, though?\n> >\n> >> File rename, at it's most basic, is when the blob associated with that\n> >> changed path is identical, i.e. BLOBSAME. There is no need to 'record'\n> >> the action of renaming, moving or whatever, the content sameness is\n> >> right there, in plain sight, as an identical blob name. After that\n> >> (files with slight variations) it is a load of heuristics, but starting\n> >> with BLOBSAME we see how easy the basic rename detection is, and why\n> >> renames (and de-dup) don't need recording.\n> >\n> > This is incorrect. Let's say you have a file foo:\n> > * base version: foo has hash A\n> > * our version: foo has been renamed to bar, but bar still has hash A\n> > * their version: foo has been modified; it now has hash B\n> >\n> > The foo->bar is an exact rename (or they are BLOBSAME if you prefer),\n> > but the renaming/moving/whatever is a critical piece of information\n> > because the changes to foo in 'their' version need to be applied to\n> > bar to get the correct end results.\n>\n> Isn't that what I thought I'd said?\n> Hash A = Hash A => identical content;\n> Hash A != B => different content.\n\nMy apologies, I misread what you had written. I had somehow read you\nas saying that the rename wasn't needed, but you didn't say that at\nall.\n\nNow that I've re-read what you wrote, and hopefully understand better,\nI did notice something else that might be worth responding to, though.\nLet me re-quote a piece of what you said and respond to it:\n\n> >> ...the content sameness is\n> >> right there, in plain sight, as an identical blob name. After that\n> >> (files with slight variations) it is a load of heuristics, ...\n\nThis suggests there aren't heuristics involved when we have identical\nblob names. That's not quite accurate, though. When there are\nmultiple identical blob names that a given source file could be paired\nwith (e.g. old/foo.txt deleted, and all three of A/foo.txt and\nB/bar.txt and C/foo.txt all added on the same side of history and all\nwith identical contents to the old/foo.txt), we give a higher score to\nfiles that share the same basename (so A/foo.txt and C/foo.txt would\nbe picked as rename targets over B/bar.txt). If we have multiple\nfiles with an identical blob name and the same file basename (i.e.\nA/foo.txt and C/foo.txt in this case), then we pick one, basically at\nrandom as far as the user is concerned (so old/foo.txt could be a\nrename to either A/foo.txt or C/foo.txt, just depending on processing\norder).\n\n> > I do not know if in Jeremy's case foo has been modified on the\n> > unrenamed side. But the following hypothetical is exactly the type of\n> > problem Jeremy is hitting: what should happen when 'our' version has\n> > both a new 'bar' and a new 'baz' file that each have hash A? In that\n> > case, to which one was foo renamed? It's inherently ambiguous.\n>\n> true, the terminology hasn't kept up with the methodology for blob\n> content, and the independent meta-data. In previous 'ort' discussions I\n> didn't really understand what the '1/2' renames (and other\n> nomenclatures) really meant with respect to paths, filenames, content\n> and the ours / theirs / base distinctions.\n\nNone of that other nomenclature in 'ort' really matters; the problem\npresented is almost certainly unrelated to any changes that came with\n'ort'. If he had switched to git anytime in the last 15 years he'd\nprobably see exactly the same problem.\n\nI'm guessing by \"'1/2' renames\" that you're referring to what I termed\n\"rename/rename(1to2)\" cases, which exist in both merge-recursive and\nmerge-ort. Those are cases where the base version had a file named A,\nand one side of history renamed A->B, while the other side of history\nrenamed A->C. That kind of situation is not relevant to the current\nproblem; for the current problem we have a file named D that was\nrenamed on only one side, but there are both files E and F each with\ncontents identical to D and rename detection has to decide whether D\nwas renamed to E or renamed to F on that side.\n\n(This contrasts with \"rename/rename(2to1)\" cases, where the base\nversion has both files G & H, and then one side of history renames\nG->I and the other side renames H->I. Or in short, 2to1 means two\nfiles are renamed to one file, and 1to2 means one file is renamed to\ntwo.)\n\n> >> The heuristics of 'rename with small change' is trickier, but for a\n> >> basic understanding, starting at BLOBSAME (and TREESAME for directory\n> >> renames) should make it easier to grasp the concepts.\n> >\n> > Interesting; TREESAME isn't used within directory rename detection\n> > currently; it is only used currently when two (or three) trees with\n> > the same name are TREESAME, in order to potentially avoid recursing\n> > into the tree. But even then, having two trees with the same name be\n> > TREESAME isn't enough on its own to avoid recursing into that tree,\n> > because the other side could have added files within the same-named\n> > tree and we need to know about those added files because they could be\n> > part of renames involving other files outside that tree.\n>\n> definitely easy to get confused on these cases...\n\nyeah, and apparently I shouldn't have used TREESAME this way as per\nJunio's comment elsewhere in the thread. Oops.\n\n> Thanks for the insights.\n\nThanks for your comments, and again, my apologies for misreading what\nyou wrote the first time. I hope I didn't do it again.\n"},{"id":"486009","messageId":"CABPp-BEdSGBt7DCrJCmOtG+RgZ2F3fNZQJ91PjZQxNa-ShKf8g@mail.gmail.com","threadId":"60476","inReplyTo":"990ab7d5-e29a-4766-b112-c8908a7ed196@iee.email","subject":"Re: Git Rename Detection Bug","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2023-12-24T07:46:01Z","receivedAt":"2023-12-24T07:46:15Z","isPatch":false,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Philip,\n\nSorry for the late reply; I somehow missed this earlier.\n\nOn Wed, Nov 15, 2023 at 8:51 AM Philip Oakley wrote:\n>\n> Hi Elijah,\n>\n> On 11/11/2023 05:46, Elijah Newren wrote:\n> > * filename similarity is extraordinarily expensive compared to exact\n> > renames, and if not carefully handled, can sometimes rival the cost of\n> > file content similarity computations given our spanhash\n> > representations.\n>\n> I've not heard of spanhash representation before. Any references or\n> further reading?\n\nYou can find more in diffcore-delta.c, especially the big comment near\nthe top of the file. But here's a short explanation of spanhashes:\n * Split files into chunks delimited either by LF or 64 bytes,\nwhichever comes first.\n * Hash every chunk into an integer between 0 and 107926\n * Keep a character count for each of those integers as well (thus if\na line has N characters, but appears twice in the file, the associated\ncount for that integer will be 2N).\n * A \"spanhash\" is the combination of the integer that a chunk (or\nspan) hashes to, plus the count associated with it.\n * The list/array of spanhashes for a file (i.e. the list/array of\nintegers and character counts) is used to compare one file to another.\n\nNow, why do I claim that comparison of filenames can rival cost of\nfile content similarity? Well, in a monorepo of interest, the median\nsized file is named something close to\n\"modules/client-resources/src/main/resources/buttons/smallTriangleBlackRight.png\"\nand is 2875 bytes. As a png, all its chunks are probably the full 64\ncharacters, which works out to about 45 chunks (assuming the 64-byte\nchunks are different from each other). The filename is 79 characters.\nSo, for this case, 45 pairs of integers vs 79 characters. So, the\ncomparison cost is roughly the same order of magnitude.\n(Yes, creating the spanhashes is a heavy overhead; however, we only\ninitialize it once and then do N comparisons of each spanhash to the\nother spanhashes. And we'd be doing N comparisons of each filename to\nother filenames, so the overhead of creating the spanhashes can be\noverlooked if your merge has enough files modified on both sides of\nhistory.)\n\nYes, this particular repository is a case I randomly picked that you\ncan argue is special. But rather than look at the particular example,\nI think it's interesting to check how the spanhash size vs. filename\nsize scale with repository size. From my experience: (1) I don't\nthink the median-sized file varies all that much between small and big\nrepositories; every time I check a repo the median size seems to be\norder of a few thousand bytes, regardless of whether the repository\nI'm looking at is tiny or huge, (2) while small repositories often\nhave much shorter filenames, big repositories often will have\nfilenames even longer than my example; length of filename tends to\ngrow with repository size from deep directory nestings. So, between\nthese two facts, I'd expect the filename comparison costs to grow\nrelative to file content comparison costs, when considering only\nmedian-sized files being modified. And since it's common to have\nmerges or rebases or diffs where only approximately-median-sized files\nare involved, I think this is relevant to look at. Finally, since I\nalready had an example that showed the cost likely roughly comparable\nfor a random repository of interest, and it's not even all that big a\nrepository compared to many out there, I think the combination\nmotivates pretty well my claim that filename similarity costs _could_\nrival file content similarity costs if one wasn't careful.\n\nI don't have a rigorous proof here. And, in fact, I ended up doing\nthis rough back-of-the-envelope analysis _after_ implementing some\nfilename similarity comparison ideas and seeing performance degrade\nbadly, and wondering why it made such a difference. I don't know if I\never got exact numbers, but I certainly didn't record them. This\nrough analysis, though, was what made me realize that I needed to be\ncareful with any such added filename comparisons, though, and is why\nI'm leery of adding more.\n"},{"id":"486122","messageId":"0438051f-7943-46df-bea7-7b790cddd72b@iee.email","threadId":"60476","inReplyTo":"CABPp-BEdSGBt7DCrJCmOtG+RgZ2F3fNZQJ91PjZQxNa-ShKf8g@mail.gmail.com","subject":"Re: Git Rename Detection Bug","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-12-28T15:33:43Z","receivedAt":"2023-12-28T15:59:58Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Elijah,\nMany thanks.. personal notes in-line.\n\nOn 24/12/2023 07:46, Elijah Newren wrote:\n> Hi Philip,\n> \n> Sorry for the late reply; I somehow missed this earlier.\n> \n> On Wed, Nov 15, 2023 at 8:51 AM Philip Oakley wrote:\n>>\n>> Hi Elijah,\n>>\n>> On 11/11/2023 05:46, Elijah Newren wrote:\n>>> * filename similarity is extraordinarily expensive compared to exact\n>>> renames, and if not carefully handled, can sometimes rival the cost of\n>>> file content similarity computations given our spanhash\n>>> representations.\n>>\n>> I've not heard of spanhash representation before. Any references or\n>> further reading?\n> \n> You can find more in diffcore-delta.c, especially the big comment near\n> the top of the file.\n\n+1\n\n> But here's a short explanation of spanhashes:\n> * Split files into chunks delimited either by LF or 64 bytes,\n> whichever comes first.\n\nneat\n\n\n> * Hash every chunk into an integer between 0 and 107926\n\nas per the comment, this is 1 less than a nice prime 107927 that fits\n17bits.\nSome discussions at\nhttps://lore.kernel.org/git/7vwtezt202.fsf@assigned-by-dhcp.cox.net/ and\nsurrounding messages.\n\nThe hash is very similar to a CRC, a rotating 64bit value, using 7 bit\nshifts and a 8bit char addition, then reduced to a hash computed at ~#L157\n\n> * Keep a character count for each of those integers as well (thus if\n> a line has N characters, but appears twice in the file, the associated\n> count for that integer will be 2N).\n> * A \"spanhash\" is the combination of the integer that a chunk (or\n> span) hashes to, plus the count associated with it.\n> * The list/array of spanhashes for a file (i.e. the list/array of\n> integers and character counts) is used to compare one file to another.\n\nI was surprised to see that I'd been in the area at #L162 ;-)\n\nThank you for the useful summary.\n\n\n> \n> Now, why do I claim that comparison of filenames can rival cost of\n> file content similarity? Well, in a monorepo of interest, the median\n> sized file is named something close to\n> \"modules/client-resources/src/main/resources/buttons/smallTriangleBlackRight.png\"\n> and is 2875 bytes. As a png, all its chunks are probably the full 64\n> characters, which works out to about 45 chunks (assuming the 64-byte\n> chunks are different from each other). The filename is 79 characters.\n> So, for this case, 45 pairs of integers vs 79 characters. So, the\n> comparison cost is roughly the same order of magnitude.\n> (Yes, creating the spanhashes is a heavy overhead; however, we only\n> initialize it once and then do N comparisons of each spanhash to the\n> other spanhashes. And we'd be doing N comparisons of each filename to\n> other filenames, so the overhead of creating the spanhashes can be\n> overlooked if your merge has enough files modified on both sides of\n> history.)\n\nNice point about the hashes only being computed once.\n\n> \n> Yes, this particular repository is a case I randomly picked that you\n> can argue is special. But rather than look at the particular example,\n> I think it's interesting to check how the spanhash size vs. filename\n> size scale with repository size. From my experience: (1) I don't\n> think the median-sized file varies all that much between small and big\n> repositories; every time I check a repo the median size seems to be\n> order of a few thousand bytes, regardless of whether the repository\n> I'm looking at is tiny or huge, (2) while small repositories often\n> have much shorter filenames, big repositories often will have\n> filenames even longer than my example; length of filename tends to\n> grow with repository size from deep directory nestings. So, between\n> these two facts, I'd expect the filename comparison costs to grow\n> relative to file content comparison costs, when considering only\n> median-sized files being modified. And since it's common to have\n> merges or rebases or diffs where only approximately-median-sized files\n> are involved, I think this is relevant to look at. Finally, since I\n> already had an example that showed the cost likely roughly comparable\n> for a random repository of interest, and it's not even all that big a\n> repository compared to many out there, I think the combination\n> motivates pretty well my claim that filename similarity costs _could_\n> rival file content similarity costs if one wasn't careful.\n> \n> I don't have a rigorous proof here. And, in fact, I ended up doing\n> this rough back-of-the-envelope analysis _after_ implementing some\n> filename similarity comparison ideas and seeing performance degrade\n> badly, and wondering why it made such a difference. I don't know if I\n> ever got exact numbers, but I certainly didn't record them. This\n> rough analysis, though, was what made me realize that I needed to be\n> careful with any such added filename comparisons, though, and is why\n> I'm leery of adding more.\n\nThanks again.\n"}]}