{"thread":{"id":"13752","subject":"[StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","startedAt":"2008-06-01T08:46:50Z","lastAt":"2008-06-03T09:27:27Z","messageCount":18,"participants":["Jakub Narebski","Catalin Marinas","Karl Hasselström"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"78280","messageId":"200806011046.51872.jnareb@gmail.com","threadId":"13752","inReplyTo":null,"subject":"[StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-01T08:46:50Z","receivedAt":"2008-06-01T08:46:50Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"StGIT errors out on rebasing patch which deletes file with Unicode\ncharacters in filename (with characters outside US-ASCII in filename).\nThe patch in question is patch deleting gitweb/test/* in git directory,\nand is present already on the 'origin' branch (the branch we rebase\nonto), so stg-rebase should result in an empty patch (as first patch).\n\n \"gitweb/test/M\\303\\244rchen\" | 2 --\n gitweb/test/file with spaces | 4 ----\n gitweb/test/file+plus+sign | 6 ------\n 3 files changed, 0 insertions(+), 12 deletions(-)\n delete mode 100644 gitweb/test/Märchen\n delete mode 100644 gitweb/test/file with spaces\n delete mode 100644 gitweb/test/file+plus+sign\n\nI guess the error is caused by using unescaped (quoted) filename.\n\nBelow is StGIT's error message:\n\n1248:[gitweb/web@git]# stg rebase origin\nChecking for changes in the working directory ... done\nPopping all applied patches ... done\nRebasing to \"origin\" ... Traceback (most recent call last):\n File \"/usr/bin/stg\", line 43, in ?\n main()\n File \"/usr/lib/python2.4/site-packages/stgit/main.py\", line 281, in main\n command.func(parser, options, args)\n File \"/usr/lib/python2.4/site-packages/stgit/commands/rebase.py\", line 70, in func\n rebase(crt_series, args[0])\n File \"/usr/lib/python2.4/site-packages/stgit/commands/common.py\", line 356, in rebase\n git.rebase(tree_id = tree_id)\n File \"/usr/lib/python2.4/site-packages/stgit/git.py\", line 927, in rebase\n reset(tree_id = tree_id)\n File \"/usr/lib/python2.4/site-packages/stgit/git.py\", line 872, in reset\n map(os.remove, rm_files)\nOSError: [Errno 2] No such file or directory: '\"gitweb/test/M\\\\303\\\\244rchen\"'\n1249:[gitweb/web@git]# stg version\nStacked GIT 0.14.2\ngit version 1.5.5.3\nPython version 2.4.3 (#1, Jun 13 2006, 16:41:18) \n[GCC 4.0.2 20051125 (Red Hat 4.0.2-8)]\n\n-- \nJakub Narebski\nPoland\n"},{"id":"78339","messageId":"200806020939.05902.jnareb@gmail.com","threadId":"13752","inReplyTo":"200806011046.51872.jnareb@gmail.com","subject":"Re: [StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-02T07:39:05Z","receivedAt":"2008-06-02T07:39:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Sun, 1 June 2008, Jakub Narebski wrote:\n\n> StGIT errors out on rebasing patch which deletes file with Unicode\n> characters in filename (with characters outside US-ASCII in filename).\n> The patch in question is patch deleting gitweb/test/* in git directory,\n> and is present already on the 'origin' branch (the branch we rebase\n> onto), so stg-rebase should result in an empty patch (as first patch).\n> \n> \"gitweb/test/M\\303\\244rchen\" | 2 --\n> gitweb/test/file with spaces | 4 ----\n> gitweb/test/file+plus+sign | 6 ------\n> 3 files changed, 0 insertions(+), 12 deletions(-)\n> delete mode 100644 gitweb/test/Märchen\n> delete mode 100644 gitweb/test/file with spaces\n> delete mode 100644 gitweb/test/file+plus+sign\n> \n> I guess the error is caused by using unescaped (quoted) filename.\n\nYou can WORKAROUND this bug by setting core.quotepath to false. This\nallowed me to make stg-rebase.\n \n-- \nJakub Narebski\nPoland\n"},{"id":"78360","messageId":"b0943d9e0806020626s3a1009d6q2a4cdb2526be9c8d@mail.gmail.com","threadId":"13752","inReplyTo":"200806020939.05902.jnareb@gmail.com","subject":"Re: [StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2008-06-02T13:26:08Z","receivedAt":"2008-06-02T13:26:08Z","isPatch":false,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"2008/6/2 Jakub Narebski :\n> On Sun, 1 June 2008, Jakub Narebski wrote:\n>\n>> StGIT errors out on rebasing patch which deletes file with Unicode\n>> characters in filename (with characters outside US-ASCII in filename).\n>> The patch in question is patch deleting gitweb/test/* in git directory,\n>> and is present already on the 'origin' branch (the branch we rebase\n>> onto), so stg-rebase should result in an empty patch (as first patch).\n>>\n>> \"gitweb/test/M\\303\\244rchen\" | 2 --\n>> gitweb/test/file with spaces | 4 ----\n>> gitweb/test/file+plus+sign | 6 ------\n>> 3 files changed, 0 insertions(+), 12 deletions(-)\n>> delete mode 100644 gitweb/test/Märchen\n>> delete mode 100644 gitweb/test/file with spaces\n>> delete mode 100644 gitweb/test/file+plus+sign\n>>\n>> I guess the error is caused by using unescaped (quoted) filename.\n>\n> You can WORKAROUND this bug by setting core.quotepath to false. This\n> allowed me to make stg-rebase.\n\nI can add a workaround in StGIT to actually ignore the exception\nraised by os.remove() but I don't know how to convert the quoted file\nname back to its unicode value in Python.\n\n-- \nCatalin\n"},{"id":"78375","messageId":"200806021747.35243.jnareb@gmail.com","threadId":"13752","inReplyTo":"b0943d9e0806020626s3a1009d6q2a4cdb2526be9c8d@mail.gmail.com","subject":"Re: [StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-02T15:47:34Z","receivedAt":"2008-06-02T15:47:34Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia poniedziałek 2. czerwca 2008 15:26, Catalin Marinas napisał:\n> 2008/6/2 Jakub Narebski :\n>> On Sun, 1 June 2008, Jakub Narebski wrote:\n>>\n>>> StGIT errors out on rebasing patch which deletes file with Unicode\n>>> characters in filename (with characters outside US-ASCII in filename).\n>>> The patch in question is patch deleting gitweb/test/* in git directory,\n>>> and is present already on the 'origin' branch (the branch we rebase\n>>> onto), so stg-rebase should result in an empty patch (as first patch).\n>>>\n>>> \"gitweb/test/M\\303\\244rchen\" | 2 --\n>>> gitweb/test/file with spaces | 4 ----\n>>> gitweb/test/file+plus+sign | 6 ------\n>>> 3 files changed, 0 insertions(+), 12 deletions(-)\n>>> delete mode 100644 gitweb/test/Märchen\n>>> delete mode 100644 gitweb/test/file with spaces\n>>> delete mode 100644 gitweb/test/file+plus+sign\n>>>\n>>> I guess the error is caused by using unescaped (quoted) filename.\n>>\n>> You can WORKAROUND this bug by setting core.quotepath to false. This\n>> allowed me to make stg-rebase.\n> \n> I can add a workaround in StGIT to actually ignore the exception\n> raised by os.remove() but I don't know how to convert the quoted file\n> name back to its unicode value in Python.\n\nIn Perl it is as simple as (see unquote() in gitweb/gitweb.perl)\n\n\tif ($str =~ m/^\"(.*)\"$/) {\n\t\t# needs unquoting\n\t\t$str = $1;\n\t\t$str =~ s/\\\\([^0-7]|[0-7]{1,3})/unq($1)/eg;\n\t}\n\nwhere\n\n\tsub unq {\n\n\t\t#...\n\n\t\tif ($seq =~ m/^[0-7]{1,3}$/) {\n\t\t\t# octal char sequence\n\t\t\treturn chr(oct($seq));\n\t\t} elsif (exists $es{$seq}) {\n\t\t\t# C escape sequence, aka character escape code\n\t\t\treturn $es{$seq};\n\t\t}\n\t\t# quoted ordinary character\n\t\treturn $seq;\n\nAlthough I haven't tested this extensively.\n\nOr you can just use '-z' switch to git-diff-tree and/or git-ls-files,\nto get filename without quoting (but then you use '\\0' as record\ndelimiter, not \"\\n\").\n\n-- \nJakub Narebski\nPoland\n"},{"id":"78380","messageId":"b0943d9e0806020911l63f5c5e5t39bb3b513d7aaccd@mail.gmail.com","threadId":"13752","inReplyTo":"200806021747.35243.jnareb@gmail.com","subject":"Re: [StGIT BUG] StGIT errors out on rebasing patch deleting file with Unicode filename","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2008-06-02T16:11:59Z","receivedAt":"2008-06-02T16:11:59Z","isPatch":false,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"2008/6/2 Jakub Narebski :\n> Dnia poniedziałek 2. czerwca 2008 15:26, Catalin Marinas napisał:\n>> 2008/6/2 Jakub Narebski :\n>>> On Sun, 1 June 2008, Jakub Narebski wrote:\n>>>\n>>>> StGIT errors out on rebasing patch which deletes file with Unicode\n>>>> characters in filename (with characters outside US-ASCII in filename).\n>>>> The patch in question is patch deleting gitweb/test/* in git directory,\n>>>> and is present already on the 'origin' branch (the branch we rebase\n>>>> onto), so stg-rebase should result in an empty patch (as first patch).\n>>>>\n>>>> \"gitweb/test/M\\303\\244rchen\" | 2 --\n>>>> gitweb/test/file with spaces | 4 ----\n>>>> gitweb/test/file+plus+sign | 6 ------\n>>>> 3 files changed, 0 insertions(+), 12 deletions(-)\n>>>> delete mode 100644 gitweb/test/Märchen\n>>>> delete mode 100644 gitweb/test/file with spaces\n>>>> delete mode 100644 gitweb/test/file+plus+sign\n>>>>\n>>>> I guess the error is caused by using unescaped (quoted) filename.\n>>>\n>>> You can WORKAROUND this bug by setting core.quotepath to false. This\n>>> allowed me to make stg-rebase.\n>>\n>> I can add a workaround in StGIT to actually ignore the exception\n>> raised by os.remove() but I don't know how to convert the quoted file\n>> name back to its unicode value in Python.\n>\n> In Perl it is as simple as (see unquote() in gitweb/gitweb.perl)\n\nActually, Python seems OK at handling quoted characters, the only\nconfusion is that it doubles the backslash in the string since it\ndoesn't know that the text received from Git is quoted or not. I would\nprobably have to check for the \"core.quotepath\" value and act\naccordingly.\n\n-- \nCatalin\n"},{"id":"78404","messageId":"20080602202148.24899.55294.stgit@yoghurt","threadId":"13752","inReplyTo":"200806011046.51872.jnareb@gmail.com","subject":"[PATCH] Add rebase test for when upstream has deleted a non-ASCII file","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T20:23:21Z","receivedAt":"2008-06-02T20:23:21Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Test that stg rebase can handle upstream deleting a file with a\nnon-ASCII name. It currently can't.\n\nBug spotted by Jakub Narebski .\n\nSigned-off-by: Karl Hasselström \n\n---\n\nHere's a little something for the test suite. Note that it's enough\nthat upstream deletes the file -- no patch in the StGit stack needs to\ntouch it.\n\n t/t2201-rebase-nonascii.sh | 27 +++++++++++++++++++++++++++\n 1 files changed, 27 insertions(+), 0 deletions(-)\n create mode 100755 t/t2201-rebase-nonascii.sh\n\n\ndiff --git a/t/t2201-rebase-nonascii.sh b/t/t2201-rebase-nonascii.sh\nnew file mode 100755\nindex 0000000..42c062f\n--- /dev/null\n+++ b/t/t2201-rebase-nonascii.sh\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+test_description='Rebase onto changed non-ASCII file'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'Setup' '\n+ echo \"Fjäderholmarna\" > skärgårdsö.txt &&\n+ git add skärgårdsö.txt &&\n+ git commit -m \"Create island\" &&\n+ stg init &&\n+ echo foo > unrelated.txt &&\n+ git add unrelated.txt &&\n+ stg new -m \"Unrelated file\" &&\n+ stg refresh &&\n+ stg pop &&\n+ rm skärgårdsö.txt &&\n+ git commit -a -m \"Remove island\" &&\n+ git tag upstream &&\n+ git reset --hard HEAD^ &&\n+ stg push\n+'\n+\n+test_expect_failure 'Rebase onto changed non-ASCII file' '\n+ stg rebase upstream\n+'\n+\n+test_done\n"},{"id":"78419","messageId":"20080602214212.18768.63775.stgit@yoghurt","threadId":"13752","inReplyTo":"200806011046.51872.jnareb@gmail.com","subject":"[StGit PATCH 0/4] Handle non-ASCII filenames","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T21:46:10Z","receivedAt":"2008-06-02T21:46:10Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"I fixed the first problem, and while doing so noticed that a nearby\nblock of code had exactly the same bug. So I fixed that as well.\n\nCatalin, this should go on the stable branch, I believe. It probably\nwarrants a new release too, since anyone rebasing patches past the\npoint where the \"Märchen\" file was removed from git.git is going to\nhit the same bug Jakub did.\n\n---\n\nKarl Hasselström (4):\n Handle refresh of changed files with non-ASCII names\n Test for another filename quoting issue in tree_status()\n Handle changed files with non-ASCII names\n Add rebase test for when upstream has deleted a non-ASCII file\n\n\n stgit/git.py | 31 +++++++++++++++++++++----------\n t/t3200-non-ascii-filenames.sh | 38 ++++++++++++++++++++++++++++++++++++++\n 2 files changed, 59 insertions(+), 10 deletions(-)\n create mode 100755 t/t3200-non-ascii-filenames.sh\n\n-- \nKarl Hasselström, kha@treskal.com\n www.treskal.com/kalle\n"},{"id":"78420","messageId":"20080602214618.18768.63334.stgit@yoghurt","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"[PATCH 1/4] Add rebase test for when upstream has deleted a non-ASCII file","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T21:46:19Z","receivedAt":"2008-06-02T21:46:19Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Test that stg rebase can handle upstream deleting a file with a\nnon-ASCII name. It currently can't.\n\nBug spotted by Jakub Narebski .\n\nSigned-off-by: Karl Hasselström \n\n---\n\n t/t3200-non-ascii-filenames.sh | 27 +++++++++++++++++++++++++++\n 1 files changed, 27 insertions(+), 0 deletions(-)\n create mode 100755 t/t3200-non-ascii-filenames.sh\n\n\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nnew file mode 100755\nindex 0000000..1d82a9f\n--- /dev/null\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+test_description='Handle files with non-ASCII characters in their names'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'Setup' '\n+ echo \"Fjäderholmarna\" > skärgårdsö.txt &&\n+ git add skärgårdsö.txt &&\n+ git commit -m \"Create island\" &&\n+ stg init &&\n+ echo foo > unrelated.txt &&\n+ git add unrelated.txt &&\n+ stg new -m \"Unrelated file\" &&\n+ stg refresh &&\n+ stg pop &&\n+ rm skärgårdsö.txt &&\n+ git commit -a -m \"Remove island\" &&\n+ git tag upstream &&\n+ git reset --hard HEAD^ &&\n+ stg push\n+'\n+\n+test_expect_failure 'Rebase onto changed non-ASCII file' '\n+ stg rebase upstream\n+'\n+\n+test_done\n"},{"id":"78422","messageId":"20080602214624.18768.53465.stgit@yoghurt","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"[PATCH 2/4] Handle changed files with non-ASCII names","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T21:46:24Z","receivedAt":"2008-06-02T21:46:24Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Git was quoting them for us, which was not what we wanted. So call\ndiff-index with the -z flag, so that it doesn't.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n stgit/git.py | 18 +++++++++++++-----\n t/t3200-non-ascii-filenames.sh | 2 +-\n 2 files changed, 14 insertions(+), 6 deletions(-)\n\n\ndiff --git a/stgit/git.py b/stgit/git.py\nindex deb5efc..d4cd946 100644\n--- a/stgit/git.py\n+++ b/stgit/git.py\n@@ -236,11 +236,19 @@ def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n args = diff_flags + [tree_id]\n if files:\n args += ['--'] + files\n- for line in GRun('diff-index', *args).output_lines():\n- fs = tuple(line.rstrip().split(' ',4)[-1].split('\\t',1))\n- if fs[1] not in reported_files:\n- cache_files.append(fs)\n- reported_files.add(fs[1])\n+ t = None\n+ for line in GRun('diff-index', '-z', *args).raw_output().split('\\0'):\n+ if not line:\n+ # There's a zero byte at the end of the output, which\n+ # gives us an empty string as the last \"line\".\n+ continue\n+ if t == None:\n+ mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n+ else:\n+ if not line in reported_files:\n+ cache_files.append((t, line))\n+ reported_files.add(line)\n+ t = None\n \n # files in the index but changed on (or removed from) disk\n args = list(diff_flags)\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex 1d82a9f..a04ead8 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -20,7 +20,7 @@ test_expect_success 'Setup' '\n stg push\n '\n \n-test_expect_failure 'Rebase onto changed non-ASCII file' '\n+test_expect_success 'Rebase onto changed non-ASCII file' '\n stg rebase upstream\n '\n \n"},{"id":"78423","messageId":"20080602214630.18768.72825.stgit@yoghurt","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"[PATCH 3/4] Test for another filename quoting issue in tree_status()","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T21:46:30Z","receivedAt":"2008-06-02T21:46:30Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"stgit.git.tree_status() had another filename quoting issue, similar to\nthe one just fixed. Test for that one too.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n t/t3200-non-ascii-filenames.sh | 13 ++++++++++++-\n 1 files changed, 12 insertions(+), 1 deletions(-)\n\n\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex a04ead8..055b152 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -10,7 +10,7 @@ test_expect_success 'Setup' '\n stg init &&\n echo foo > unrelated.txt &&\n git add unrelated.txt &&\n- stg new -m \"Unrelated file\" &&\n+ stg new p0 -m \"Unrelated file\" &&\n stg refresh &&\n stg pop &&\n rm skärgårdsö.txt &&\n@@ -24,4 +24,15 @@ test_expect_success 'Rebase onto changed non-ASCII file' '\n stg rebase upstream\n '\n \n+test_expect_success 'Setup' '\n+ stg delete p0 &&\n+ git reset --hard HEAD^ &&\n+ echo \"-- ett liv mitt ute i vattnet\" >> skärgårdsö.txt &&\n+ stg new p1 -m \"Describe island\"\n+'\n+\n+test_expect_failure 'Refresh changes to non-ASCII file' '\n+ stg refresh\n+'\n+\n test_done\n"},{"id":"78421","messageId":"20080602214636.18768.21389.stgit@yoghurt","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"[PATCH 4/4] Handle refresh of changed files with non-ASCII names","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-02T21:46:36Z","receivedAt":"2008-06-02T21:46:36Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Without -z, git diff-files was quoting them for us.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n stgit/git.py | 39 +++++++++++++++++++++------------------\n t/t3200-non-ascii-filenames.sh | 2 +-\n 2 files changed, 22 insertions(+), 19 deletions(-)\n\n\ndiff --git a/stgit/git.py b/stgit/git.py\nindex d4cd946..9032ddd 100644\n--- a/stgit/git.py\n+++ b/stgit/git.py\n@@ -191,6 +191,19 @@ def ls_files(files, tree = None, full_name = True):\n raise GitException, \\\n 'Some of the given paths are either missing or not known to GIT'\n \n+def parse_git_ls(output):\n+ t = None\n+ for line in output.split('\\0'):\n+ if not line:\n+ # There's a zero byte at the end of the output, which\n+ # gives us an empty string as the last \"line\".\n+ continue\n+ if t == None:\n+ mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n+ else:\n+ yield (t, line)\n+ t = None\n+\n def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n noexclude = True, verbose = False, diff_flags = []):\n \"\"\"Get the status of all changed files, or of a selected set of\n@@ -236,29 +249,19 @@ def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n args = diff_flags + [tree_id]\n if files:\n args += ['--'] + files\n- t = None\n- for line in GRun('diff-index', '-z', *args).raw_output().split('\\0'):\n- if not line:\n- # There's a zero byte at the end of the output, which\n- # gives us an empty string as the last \"line\".\n- continue\n- if t == None:\n- mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n- else:\n- if not line in reported_files:\n- cache_files.append((t, line))\n- reported_files.add(line)\n- t = None\n+ for t, fn in parse_git_ls(GRun('diff-index', '-z', *args).raw_output()):\n+ if not fn in reported_files:\n+ cache_files.append((t, fn))\n+ reported_files.add(fn)\n \n # files in the index but changed on (or removed from) disk\n args = list(diff_flags)\n if files:\n args += ['--'] + files\n- for line in GRun('diff-files', *args).output_lines():\n- fs = tuple(line.rstrip().split(' ',4)[-1].split('\\t',1))\n- if fs[1] not in reported_files:\n- cache_files.append(fs)\n- reported_files.add(fs[1])\n+ for t, fn in parse_git_ls(GRun('diff-files', '-z', *args).raw_output()):\n+ if not fn in reported_files:\n+ cache_files.append((t, fn))\n+ reported_files.add(fn)\n \n if verbose:\n out.done()\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex 055b152..892ebc9 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -31,7 +31,7 @@ test_expect_success 'Setup' '\n stg new p1 -m \"Describe island\"\n '\n \n-test_expect_failure 'Refresh changes to non-ASCII file' '\n+test_expect_success 'Refresh changes to non-ASCII file' '\n stg refresh\n '\n \n"},{"id":"78443","messageId":"20080603003846.25028.49353.stgit@yoghurt","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"[StGit PATCH v2 0/4] Handle non-ASCII filenames","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T00:41:32Z","receivedAt":"2008-06-03T00:41:32Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"I fixed the first problem, and while doing so noticed that a nearby\nblock of code had exactly the same bug. So I fixed that as well.\n\nCatalin, this should go on the stable branch, I believe. It probably\nwarrants a new release too, since anyone rebasing patches past the\npoint where the \"Märchen\" file was removed from git.git is going to\nhit the same bug Jakub did.\n\n( This is a second version of the series, this time based on Catalin's\n stable and not stable~1. As luck would have it, that last commit was\n touching the precise same part of stgit/git.py that my series did.\n Argh. )\n\n---\n\nKarl Hasselström (4):\n Handle refresh of changed files with non-ASCII names\n Test for another filename quoting issue in tree_status()\n Handle changed files with non-ASCII names\n Add rebase test for when upstream has deleted a non-ASCII file\n\n\n stgit/git.py | 31 ++++++++++++++-------\n t/t3200-non-ascii-filenames.sh | 59 ++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 80 insertions(+), 10 deletions(-)\n create mode 100755 t/t3200-non-ascii-filenames.sh\n\n-- \nKarl Hasselström, kha@treskal.com\n www.treskal.com/kalle\n"},{"id":"78444","messageId":"20080603004139.25028.33202.stgit@yoghurt","threadId":"13752","inReplyTo":"20080603003846.25028.49353.stgit@yoghurt","subject":"[StGit PATCH v2 1/4] Add rebase test for when upstream has deleted a non-ASCII file","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T00:41:39Z","receivedAt":"2008-06-03T00:41:39Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Test that stg rebase can handle upstream deleting a file with a\nnon-ASCII name. It currently can't.\n\nBug spotted by Jakub Narebski .\n\nSigned-off-by: Karl Hasselström \n\n---\n\n t/t3200-non-ascii-filenames.sh | 27 +++++++++++++++++++++++++++\n 1 files changed, 27 insertions(+), 0 deletions(-)\n create mode 100755 t/t3200-non-ascii-filenames.sh\n\n\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nnew file mode 100755\nindex 0000000..1d82a9f\n--- /dev/null\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -0,0 +1,27 @@\n+#!/bin/sh\n+test_description='Handle files with non-ASCII characters in their names'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'Setup' '\n+ echo \"Fjäderholmarna\" > skärgårdsö.txt &&\n+ git add skärgårdsö.txt &&\n+ git commit -m \"Create island\" &&\n+ stg init &&\n+ echo foo > unrelated.txt &&\n+ git add unrelated.txt &&\n+ stg new -m \"Unrelated file\" &&\n+ stg refresh &&\n+ stg pop &&\n+ rm skärgårdsö.txt &&\n+ git commit -a -m \"Remove island\" &&\n+ git tag upstream &&\n+ git reset --hard HEAD^ &&\n+ stg push\n+'\n+\n+test_expect_failure 'Rebase onto changed non-ASCII file' '\n+ stg rebase upstream\n+'\n+\n+test_done\n"},{"id":"78446","messageId":"20080603004145.25028.34246.stgit@yoghurt","threadId":"13752","inReplyTo":"20080603003846.25028.49353.stgit@yoghurt","subject":"[StGit PATCH v2 2/4] Handle changed files with non-ASCII names","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T00:41:45Z","receivedAt":"2008-06-03T00:41:45Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Git was quoting them for us, which was not what we wanted. So call\ndiff-index with the -z flag, so that it doesn't.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n stgit/git.py | 22 +++++++++++++++-------\n t/t3200-non-ascii-filenames.sh | 2 +-\n 2 files changed, 16 insertions(+), 8 deletions(-)\n\n\ndiff --git a/stgit/git.py b/stgit/git.py\nindex 6140fd9..8c637d5 100644\n--- a/stgit/git.py\n+++ b/stgit/git.py\n@@ -242,13 +242,21 @@ def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n args = diff_flags + [tree_id]\n if files_left:\n args += ['--'] + files_left\n- for line in GRun('diff-index', *args).output_lines():\n- fs = tuple(line.rstrip().split(' ',4)[-1].split('\\t',1))\n- # the condition is needed in case files is emtpy and\n- # diff-index lists those already reported\n- if fs[1] not in reported_files:\n- cache_files.append(fs)\n- reported_files.add(fs[1])\n+ t = None\n+ for line in GRun('diff-index', '-z', *args).raw_output().split('\\0'):\n+ if not line:\n+ # There's a zero byte at the end of the output, which\n+ # gives us an empty string as the last \"line\".\n+ continue\n+ if t == None:\n+ mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n+ else:\n+ # the condition is needed in case files is emtpy and\n+ # diff-index lists those already reported\n+ if not line in reported_files:\n+ cache_files.append((t, line))\n+ reported_files.add(line)\n+ t = None\n files_left = [f for f in files if f not in reported_files]\n \n # files in the index but changed on (or removed from) disk. Only\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex 1d82a9f..a04ead8 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -20,7 +20,7 @@ test_expect_success 'Setup' '\n stg push\n '\n \n-test_expect_failure 'Rebase onto changed non-ASCII file' '\n+test_expect_success 'Rebase onto changed non-ASCII file' '\n stg rebase upstream\n '\n \n"},{"id":"78447","messageId":"20080603004151.25028.99694.stgit@yoghurt","threadId":"13752","inReplyTo":"20080603003846.25028.49353.stgit@yoghurt","subject":"[StGit PATCH v2 3/4] Test for another filename quoting issue in tree_status()","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T00:41:51Z","receivedAt":"2008-06-03T00:41:51Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"stgit.git.tree_status() had another filename quoting issue, similar to\nthe one just fixed. Test for that one too.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n t/t3200-non-ascii-filenames.sh | 34 +++++++++++++++++++++++++++++++++-\n 1 files changed, 33 insertions(+), 1 deletions(-)\n\n\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex a04ead8..3146b8d 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -3,6 +3,12 @@ test_description='Handle files with non-ASCII characters in their names'\n \n . ./test-lib.sh\n \n+# Ignore our own output files.\n+cat > .git/info/exclude < skärgårdsö.txt &&\n git add skärgårdsö.txt &&\n@@ -10,7 +16,7 @@ test_expect_success 'Setup' '\n stg init &&\n echo foo > unrelated.txt &&\n git add unrelated.txt &&\n- stg new -m \"Unrelated file\" &&\n+ stg new p0 -m \"Unrelated file\" &&\n stg refresh &&\n stg pop &&\n rm skärgårdsö.txt &&\n@@ -24,4 +30,30 @@ test_expect_success 'Rebase onto changed non-ASCII file' '\n stg rebase upstream\n '\n \n+test_expect_success 'Setup' '\n+ stg delete p0 &&\n+ git reset --hard HEAD^ &&\n+ echo \"-- ett liv mitt ute i vattnet\" >> skärgårdsö.txt &&\n+ stg new p1 -m \"Describe island\"\n+'\n+\n+cat > expected.txt < output.txt &&\n+ diff -u expected.txt output.txt\n+'\n+\n+test_expect_success 'Refresh changes to non-ASCII file' '\n+ stg refresh\n+'\n+\n+cat > expected.txt < output.txt &&\n+ diff -u expected.txt output.txt\n+'\n+\n test_done\n"},{"id":"78445","messageId":"20080603004157.25028.65176.stgit@yoghurt","threadId":"13752","inReplyTo":"20080603003846.25028.49353.stgit@yoghurt","subject":"[StGit PATCH v2 4/4] Handle refresh of changed files with non-ASCII names","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T00:41:57Z","receivedAt":"2008-06-03T00:41:57Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"Without -z, git diff-files was quoting them for us.\n\nSigned-off-by: Karl Hasselström \n\n---\n\n stgit/git.py | 43 +++++++++++++++++++++-------------------\n t/t3200-non-ascii-filenames.sh | 2 +-\n 2 files changed, 24 insertions(+), 21 deletions(-)\n\n\ndiff --git a/stgit/git.py b/stgit/git.py\nindex 8c637d5..8e6bdf4 100644\n--- a/stgit/git.py\n+++ b/stgit/git.py\n@@ -191,6 +191,19 @@ def ls_files(files, tree = 'HEAD', full_name = True):\n raise GitException, \\\n 'Some of the given paths are either missing or not known to GIT'\n \n+def parse_git_ls(output):\n+ t = None\n+ for line in output.split('\\0'):\n+ if not line:\n+ # There's a zero byte at the end of the output, which\n+ # gives us an empty string as the last \"line\".\n+ continue\n+ if t == None:\n+ mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n+ else:\n+ yield (t, line)\n+ t = None\n+\n def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n noexclude = True, verbose = False, diff_flags = []):\n \"\"\"Get the status of all changed files, or of a selected set of\n@@ -242,21 +255,12 @@ def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n args = diff_flags + [tree_id]\n if files_left:\n args += ['--'] + files_left\n- t = None\n- for line in GRun('diff-index', '-z', *args).raw_output().split('\\0'):\n- if not line:\n- # There's a zero byte at the end of the output, which\n- # gives us an empty string as the last \"line\".\n- continue\n- if t == None:\n- mode_a, mode_b, sha1_a, sha1_b, t = line.split(' ')\n- else:\n- # the condition is needed in case files is emtpy and\n- # diff-index lists those already reported\n- if not line in reported_files:\n- cache_files.append((t, line))\n- reported_files.add(line)\n- t = None\n+ for t, fn in parse_git_ls(GRun('diff-index', '-z', *args).raw_output()):\n+ # the condition is needed in case files is emtpy and\n+ # diff-index lists those already reported\n+ if not fn in reported_files:\n+ cache_files.append((t, fn))\n+ reported_files.add(fn)\n files_left = [f for f in files if f not in reported_files]\n \n # files in the index but changed on (or removed from) disk. Only\n@@ -267,13 +271,12 @@ def tree_status(files = None, tree_id = 'HEAD', unknown = False,\n args = list(diff_flags)\n if files_left:\n args += ['--'] + files_left\n- for line in GRun('diff-files', *args).output_lines():\n- fs = tuple(line.rstrip().split(' ',4)[-1].split('\\t',1))\n+ for t, fn in parse_git_ls(GRun('diff-files', '-z', *args).raw_output()):\n # the condition is needed in case files is empty and\n # diff-files lists those already reported\n- if fs[1] not in reported_files:\n- cache_files.append(fs)\n- reported_files.add(fs[1])\n+ if not fn in reported_files:\n+ cache_files.append((t, fn))\n+ reported_files.add(fn)\n \n if verbose:\n out.done()\ndiff --git a/t/t3200-non-ascii-filenames.sh b/t/t3200-non-ascii-filenames.sh\nindex 3146b8d..1aa78ed 100755\n--- a/t/t3200-non-ascii-filenames.sh\n+++ b/t/t3200-non-ascii-filenames.sh\n@@ -40,7 +40,7 @@ test_expect_success 'Setup' '\n cat > expected.txt < output.txt &&\n diff -u expected.txt output.txt\n '\n"},{"id":"78467","messageId":"b0943d9e0806030056x588474dcha3e14dd8dca952ba@mail.gmail.com","threadId":"13752","inReplyTo":"20080602214212.18768.63775.stgit@yoghurt","subject":"Re: [StGit PATCH 0/4] Handle non-ASCII filenames","fromName":"Catalin Marinas","fromEmail":"catalin.marinas@gmail.com","sentAt":"2008-06-03T07:56:38Z","receivedAt":"2008-06-03T07:56:38Z","isPatch":true,"sender":{"key":"catalin.marinas@gmail.com","avatar":null},"body":"On 02/06/2008, Karl Hasselström wrote:\n> I fixed the first problem, and while doing so noticed that a nearby\n> block of code had exactly the same bug. So I fixed that as well.\n\nThat was fast :-). thanks.\n\n> Catalin, this should go on the stable branch, I believe. It probably\n> warrants a new release too, since anyone rebasing patches past the\n> point where the \"Märchen\" file was removed from git.git is going to\n> hit the same bug Jakub did.\n\nYes, it will go in both stable and master. There are some more patches\non stable already, I'll try to release 0.14.3 this weekend.\n\nIs the new infrastructure affected? We haven't got to the point of\nconverting \"rebase\" yet (doesn't have all the functionality it needs\nin the new infrastructure). BTW, I created a new lib.git.Branch class\nas the parent of lib.git.Stack and it takes care the extra things used\nwhen creating a new stack like setting parent and remote branches.\nI'll post the patches in the next few days.\n\n-- \nCatalin\n"},{"id":"78470","messageId":"20080603092727.GA13132@diana.vm.bytemark.co.uk","threadId":"13752","inReplyTo":"b0943d9e0806030056x588474dcha3e14dd8dca952ba@mail.gmail.com","subject":"Re: [StGit PATCH 0/4] Handle non-ASCII filenames","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2008-06-03T09:27:27Z","receivedAt":"2008-06-03T09:27:27Z","isPatch":true,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"On 2008-06-03 08:56:38 +0100, Catalin Marinas wrote:\n\n> On 02/06/2008, Karl Hasselström wrote:\n>\n> > I fixed the first problem, and while doing so noticed that a\n> > nearby block of code had exactly the same bug. So I fixed that as\n> > well.\n>\n> That was fast :-). thanks.\n\nJakub reported the bug right before I was about to start hacking\nanyway. :-)\n\n> > Catalin, this should go on the stable branch, I believe. It\n> > probably warrants a new release too, since anyone rebasing\n> > patches past the point where the \"Märchen\" file was removed from\n> > git.git is going to hit the same bug Jakub did.\n>\n> Yes, it will go in both stable and master. There are some more\n> patches on stable already, I'll try to release 0.14.3 this weekend.\n\nSplendid. You can pull from my \"stable\" branch if you like.\n\n> Is the new infrastructure affected? We haven't got to the point of\n> converting \"rebase\" yet (doesn't have all the functionality it needs\n> in the new infrastructure).\n\nThe problem wasn't actually about rebase. It was about mangling of\ndiff-index and diff-files output in tree_status(). I think if one were\nto look closely, a largish number of commands would be found to be\naffected.\n\nThe new infrastructure won't have this problem, by virtue of not\nhaving any such calls yet. Plus, I try to use the -z flag whenever I\ncan from the outset.\n\n> BTW, I created a new lib.git.Branch class as the parent of\n> lib.git.Stack and it takes care the extra things used when creating\n> a new stack like setting parent and remote branches. I'll post the\n> patches in the next few days.\n\nLooking forward to it.\n\n-- \nKarl Hasselström, kha@treskal.com\n www.treskal.com/kalle\n"}]}