{"thread":{"id":"39994","subject":"[PATCH] git-p4: fix faulty paths for case insensitive systems","startedAt":"2015-08-02T15:15:16Z","lastAt":"2015-08-05T05:36:37Z","messageCount":4,"participants":["larsxschneider@gmail.com","Luke Diamand","Lars Schneider"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"267285","messageId":"1438528517-5028-1-git-send-email-larsxschneider@gmail.com","threadId":"39994","inReplyTo":null,"subject":"[PATCH] git-p4: fix faulty paths for case insensitive systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-08-02T15:15:16Z","receivedAt":"2015-08-02T15:15:16Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nHi,\n\nI want to propose this patch as it helped us to migrate a big source code base\nsuccessfully from P4 to Git. I am sorry that I don't provide a test case, yet.\nI would like to get advise on the patch and on the best strategy to provide a\ntest. Do you only run git-p4 integration tests in \"t/t98??-git-p4-*.sh\"? If yes,\nwhich version of \"start_p4d\" should I use?\n\nThanks,\nLars\n\nLars Schneider (1):\n git-p4: fix faulty paths for case insensitive systems\n\n git-p4.py | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 77 insertions(+), 4 deletions(-)\n\n--\n1.9.5 (Apple Git-50.3)\n"},{"id":"267286","messageId":"1438528517-5028-2-git-send-email-larsxschneider@gmail.com","threadId":"39994","inReplyTo":"1438528517-5028-1-git-send-email-larsxschneider@gmail.com","subject":"[PATCH] git-p4: fix faulty paths for case insensitive systems","fromName":"","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-08-02T15:15:17Z","receivedAt":"2015-08-02T15:15:17Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"From: Lars Schneider \n\nPROBLEM:\nWe run P4 servers on Linux and P4 clients on Windows. For an unknown\nreason the file path for a number of files in P4 does not match the\ndirectory path with respect to case sensitivity.\n\nE.g. `p4 files` might return\n//depot/path/to/file1\n//depot/PATH/to/file2\n\nIf you use P4/P4V then these files end up in the same directory, e.g.\n//depot/path/to/file1\n//depot/path/to/file2\n\nIf you use git-p4 then all files not matching the correct file path\n(e.g. `file2`) will be ignored.\n\nSOLUTION:\nIdentify files that are different with respect to case sensitivity.\nIf there are any then run `p4 dirs` to build up a dictionary\ncontaining the correct cases for each path. Upon `clone` this\ndictionary is used to fix the paths. All this is only applied if the\ngit config \"core.ignorecase\" is set.\n\nSigned-off-by: Lars Schneider \n---\n git-p4.py | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 77 insertions(+), 4 deletions(-)\n\ndiff --git a/git-p4.py b/git-p4.py\nindex 549022e..692f1f4 100755\n--- a/git-p4.py\n+++ b/git-p4.py\n@@ -1859,7 +1859,7 @@ class View(object):\n (self.client_prefix, clientFile))\n return clientFile[len(self.client_prefix):]\n \n- def update_client_spec_path_cache(self, files):\n+ def update_client_spec_path_cache(self, files, fixPathCase = None):\n \"\"\" Caching file paths by \"p4 where\" batch query \"\"\"\n \n # List depot file paths exclude that already cached\n@@ -1878,6 +1878,8 @@ class View(object):\n if \"unmap\" in res:\n # it will list all of them, but only one not unmap-ped\n continue\n+ if fixPathCase:\n+ res['depotFile'] = fixPathCase(res['depotFile'])\n self.client_spec_path_cache[res['depotFile']] = self.convert_client_path(res[\"clientFile\"])\n \n # not found files or unmap files set to \"\"\n@@ -1973,7 +1975,8 @@ class P4Sync(Command, P4UserMap):\n files = []\n fnum = 0\n while commit.has_key(\"depotFile%s\" % fnum):\n- path = commit[\"depotFile%s\" % fnum]\n+ path = commit[\"depotFile%s\" % fnum]\n+ path = self.fixPathCase(path)\n \n if [p for p in self.cloneExclude\n if p4PathStartsWith(path, p)]:\n@@ -2037,7 +2040,9 @@ class P4Sync(Command, P4UserMap):\n branches = {}\n fnum = 0\n while commit.has_key(\"depotFile%s\" % fnum):\n- path = commit[\"depotFile%s\" % fnum]\n+ path = commit[\"depotFile%s\" % fnum]\n+ path = self.fixPathCase(path)\n+\n found = [p for p in self.depotPaths\n if p4PathStartsWith(path, p)]\n if not found:\n@@ -2164,6 +2169,10 @@ class P4Sync(Command, P4UserMap):\n if marshalled[\"code\"] == \"error\":\n if \"data\" in marshalled:\n err = marshalled[\"data\"].rstrip()\n+\n+ if \"depotFile\" in marshalled:\n+ marshalled['depotFile'] = self.fixPathCase(marshalled['depotFile'])\n+\n if err:\n f = None\n if self.stream_have_file_info:\n@@ -2238,6 +2247,7 @@ class P4Sync(Command, P4UserMap):\n \n # do the last chunk\n if self.stream_file.has_key('depotFile'):\n+ self.stream_file['depotFile'] = self.fixPathCase(self.stream_file['depotFile'])\n self.streamOneP4File(self.stream_file, self.stream_contents)\n \n def make_email(self, userid):\n@@ -2295,7 +2305,8 @@ class P4Sync(Command, P4UserMap):\n sys.stderr.write(\"Ignoring file outside of prefix: %s\\n\" % f['path'])\n \n if self.clientSpecDirs:\n- self.clientSpecDirs.update_client_spec_path_cache(files)\n+ self.clientSpecDirs.update_client_spec_path_cache(\n+ files, lambda x: self.fixPathCase(x))\n \n self.gitStream.write(\"commit %s\\n\" % branch)\n # gitStream.write(\"mark :%s\\n\" % details[\"change\"])\n@@ -2759,6 +2770,63 @@ class P4Sync(Command, P4UserMap):\n print \"IO error with git fast-import. Is your git version recent enough?\"\n print self.gitError.read()\n \n+ def fixPathCase(self, path):\n+ if self.caseCorrectedPaths:\n+ components = path.split('/')\n+ filename = components.pop()\n+ dirname = '/'.join(components).lower() + '/'\n+ if dirname in self.caseCorrectedPaths:\n+ path = self.caseCorrectedPaths[dirname] + filename\n+ return path\n+\n+ def generatePathCaseDict(self, depotPaths):\n+ # Query all files and generate a list of all used paths\n+ # e.g. this files list:\n+ # //depot/path/to/file1\n+ # //depot/PATH/to/file2\n+ #\n+ # result in this path list:\n+ # //depot/\n+ # //depot/PATH/\n+ # //depot/path/\n+ # //depot/PATH/to/\n+ # //depot/path/to/\n+ p4_paths = set()\n+ for p in depotPaths:\n+ for f in p4CmdList([\"files\", p+\"...\"]):\n+ components = f[\"depotFile\"].split('/')[0:-1]\n+ for i in range(3, len(components)+1):\n+ p4_paths.add('/'.join(components[0:i]) + '/')\n+ p4_paths = sorted(list(p4_paths), key=len)\n+\n+ if len(p4_paths) > len(set([p.lower() for p in p4_paths])):\n+ print \"ATTENTION: File paths with different case variations detected. Fixing may take a while...\"\n+ found_variations = True\n+ while found_variations:\n+ for path in p4_paths:\n+ found_variations = False\n+ path_variations = [p for p in p4_paths if p.lower() == path.lower()]\n+\n+ if len(path_variations) > 1:\n+ print \"%i different case variations for path '%s' detected.\" % (len(path_variations), path)\n+ # If we detect path variations (e.g. //depot/path and //depot/PATH) then we query P4 to list\n+ # the subdirectories of the parent (e.g //depot/*). P4 will return these subdirectories with\n+ # the correct case.\n+ parent_path = '/'.join(path.split('/')[0:-2])\n+ case_ok_paths = [p[\"dir\"] + '/' for p in p4CmdList([\"dirs\", \"-D\", parent_path + '/*'])]\n+\n+ # Replace all known paths with the case corrected path from P4 dirs command\n+ for case_ok_path in case_ok_paths:\n+ pattern = re.compile(\"^\" + case_ok_path, re.IGNORECASE)\n+ p4_paths = sorted(list(set([pattern.sub(case_ok_path, p) for p in p4_paths])), key=len)\n+\n+ found_variations = True\n+ break\n+ return dict((p.lower(), p) for p in p4_paths)\n+ else:\n+ if self.verbose:\n+ print \"All file paths have consistent case\"\n+ return None\n \n def run(self, args):\n self.depotPaths = []\n@@ -2930,6 +2998,11 @@ class P4Sync(Command, P4UserMap):\n \n self.depotPaths = newPaths\n \n+ if gitConfigBool(\"core.ignorecase\"):\n+ self.caseCorrectedPaths = self.generatePathCaseDict(self.depotPaths)\n+ else:\n+ self.caseCorrectedPaths = None\n+\n # --detect-branches may change this for each branch\n self.branchPrefixes = self.depotPaths\n \n-- \n1.9.5 (Apple Git-50.3)\n"},{"id":"267487","messageId":"55C1374E.5060808@diamand.org","threadId":"39994","inReplyTo":"1438528517-5028-1-git-send-email-larsxschneider@gmail.com","subject":"Re: [PATCH] git-p4: fix faulty paths for case insensitive systems","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2015-08-04T22:06:06Z","receivedAt":"2015-08-04T22:06:06Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 02/08/15 16:15, larsxschneider@gmail.com wrote:\n> From: Lars Schneider \n>\n> Hi,\n>\n> I want to propose this patch as it helped us to migrate a big source code base\n> successfully from P4 to Git. I am sorry that I don't provide a test case, yet.\n\nCase sensitivity is a pretty tricky area with p4 - it's very brave of \nyou to have a go at fixing it!\n\n> I would like to get advise on the patch and on the best strategy to provide a\n> test. Do you only run git-p4 integration tests in \"t/t98??-git-p4-*.sh\"? If yes,\n> which version of \"start_p4d\" should I use?\n\nOnly the t98* tests relate to git-p4 so if you just copy one of those it \nshould do the right thing.\n\nt9819-git-p4-case-folding.sh already has a few failing tests for this \nproblem. I wrote it a while back just to illustrate the problem, so it \nmight be of use to you, or you might need to start again.\n\nWon't your change make importing much slower for people with this problem?\n\nAlso, I'm not sure you can use \"core.ignorecase\" to trigger this: the \nproblem will arise if the *server* is ignoring case as well (which I \nthink you can detect by querying the server).\n\nI'm not trying to be negative - but this problem does have some annoying \npitfalls! Let me know if you think I can help though.\n\nRegards!\nLuke\n"},{"id":"267494","messageId":"42A92EAD-493D-458C-9D8B-BDC78E1A1614@gmail.com","threadId":"39994","inReplyTo":"55C1374E.5060808@diamand.org","subject":"Re: [PATCH] git-p4: fix faulty paths for case insensitive systems","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2015-08-05T05:36:37Z","receivedAt":"2015-08-05T05:36:37Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"Thank you for your reply. Your t8919 test case looks exactly like the right thing. Unfortuantly I don’t have Internet access for the next two weeks. Afterwards I will provide a proper test cases for the patch.\n\nYou are correct about the speed. All these initial “p4 dirs” calls make the clone pretty slow. However, for us it is a one time history migration and therefore speed is not an issue. I also understand your “core.ignorecase” comment. Let’s assume the path correction works as expected, how and when would you trigger it? Would you only rely on the “server ignoring case” flag?\n\nCheers,\nLars\n\nOn 05 Aug 2015, at 00:06, Luke Diamand wrote:\n\n> On 02/08/15 16:15, larsxschneider@gmail.com wrote:\n>> From: Lars Schneider \n>> \n>> Hi,\n>> \n>> I want to propose this patch as it helped us to migrate a big source code base\n>> successfully from P4 to Git. I am sorry that I don't provide a test case, yet.\n> \n> Case sensitivity is a pretty tricky area with p4 - it's very brave of you to have a go at fixing it!\n> \n>> I would like to get advise on the patch and on the best strategy to provide a\n>> test. Do you only run git-p4 integration tests in \"t/t98??-git-p4-*.sh\"? If yes,\n>> which version of \"start_p4d\" should I use?\n> \n> Only the t98* tests relate to git-p4 so if you just copy one of those it should do the right thing.\n> \n> t9819-git-p4-case-folding.sh already has a few failing tests for this problem. I wrote it a while back just to illustrate the problem, so it might be of use to you, or you might need to start again.\n> \n> Won't your change make importing much slower for people with this problem?\n> \n> Also, I'm not sure you can use \"core.ignorecase\" to trigger this: the problem will arise if the *server* is ignoring case as well (which I think you can detect by querying the server).\n> \n> I'm not trying to be negative - but this problem does have some annoying pitfalls! Let me know if you think I can help though.\n> \n> Regards!\n> Luke\n> \n"}]}