{"thread":{"id":"55786","subject":"Bug: git p4 clone fails if non unicode files with +k or+ko attributes are present","startedAt":"2021-05-28T04:18:31Z","lastAt":"2021-05-28T04:18:31Z","messageCount":1,"participants":["Arthur Moore"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"425732","messageId":"01000179b131beca-113a6722-77cb-4320-8930-6904b197a003-000000@email.amazonses.com","threadId":"55786","inReplyTo":null,"subject":"Bug: git p4 clone fails if non unicode files with +k or+ko attributes are present","fromName":"Arthur Moore","fromEmail":"arthur.moore.git@cd-net.net","sentAt":"2021-05-28T04:18:18Z","receivedAt":"2021-05-28T04:18:31Z","isPatch":false,"sender":{"key":"arthur.moore.git@cd-net.net","avatar":null},"body":"Hello,\n\nI would like to report a problem which is preventing me from using\ngit's Perforce integration.\n\nProblem Command:\ngit p4 clone //path/to/files/\n\nSteps to reproduce:\n* Create/have a configured and running Perforce server and client.\n* Create a file with any encoding aside from UTF-8 or UTF-16\ncontaining at least one character outside the ASCII range.\n* Adjust the file's type to text and include the \"+k\" or \"+ko\" attributes.\n  * This can be done in the graphical p4v client by right clicking the\nfile and selecting \"Change Filetype\"\n* Submit that file to Perforce.\n* Attempt to run `git p4 clone //path/to/files/`\n\nExpected behavior:\nThe Perforce repository is successfully cloned.\n\nActual behavior:\n\nThe clone failed with the following error:\n```\nTraceback (most recent call last):\n  File \"/usr/libexec/git-core/git-p4\", line 4441, in <module>\n    main()\n  File \"/usr/libexec/git-core/git-p4\", line 4435, in main\n    if not cmd.run(args):\n  File \"/usr/libexec/git-core/git-p4\", line 4187, in run\n    if not P4Sync.run(self, depotPaths):\n  File \"/usr/libexec/git-core/git-p4\", line 4053, in run\n    self.importHeadRevision(revision)\n  File \"/usr/libexec/git-core/git-p4\", line 3761, in importHeadRevision\n    self.commit(details, self.extractFilesFromCommit(details), self.branch)\n  File \"/usr/libexec/git-core/git-p4\", line 3311, in commit\n    self.streamP4Files(files)\n  File \"/usr/libexec/git-core/git-p4\", line 3155, in streamP4Files\n    p4CmdList([\"-x\", \"-\", \"print\"],\n  File \"/usr/libexec/git-core/git-p4\", line 784, in p4CmdList\n    cb(entry)\n  File \"/usr/libexec/git-core/git-p4\", line 3142, in streamP4FilesCbSelf\n    self.streamP4FilesCb(entry)\n  File \"/usr/libexec/git-core/git-p4\", line 3090, in streamP4FilesCb\n    self.streamOneP4File(self.stream_file, self.stream_contents)\n  File \"/usr/libexec/git-core/git-p4\", line 3035, in streamOneP4File\n    text = ''.join(decode_text_stream(c) for c in contents)\n  File \"/usr/libexec/git-core/git-p4\", line 3035, in <genexpr>\n    text = ''.join(decode_text_stream(c) for c in contents)\n  File \"/usr/libexec/git-core/git-p4\", line 184, in decode_text_stream\n    return s.decode() if isinstance(s, bytes) else s\nUnicodeDecodeError: 'utf-8' codec can't decode byte 0xde in position\n6276: invalid continuation byte\n```\n\nFailure Analysis:\n\nThe problem appears to be partly in `streamOneP4File` and partly in `\np4_keywords_regexp_for_type`.\n\nPerforce allows embedding special keywords in files that are expanded\nat checkout, and this function strips the expansion.\nTo do so,  it converts the binary data to a real string, performs a\nregex replace, then converts it back to binary data.\nUnfortunately, that assumes the file is encoded in UTF-8, and when\nthis is not the case an uncaught exception is thrown.\n\nImportantly, perforce also allows injecting those keywords in raw\nbinary files, so this goes beyond a simple \"support more encodings\"\nissue.\n\nhttps://community.perforce.com/s/article/3482\n\n[System Info]\ngit version 2.31.1\n\n[Enabled Hooks]\nnot run from a git repository - no hooks to show\n\nSincerely,\nArthur Moore\n"}]}