Re: [PATCH 01/11] t9902-completion: add tests demonstrating issues with quoted pathnames

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Apr 18, 2018, 12:31 UTC
Message-ID: <nycvar.QRO.7.76.6.1804181421590.4241@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz>
In-Reply-To: <20180416224113.16993-2-szeder.dev@gmail.com>

Hi Gábor,

On Tue, 17 Apr 2018, SZEDER Gábor wrote:

Show 16 quoted lines

> Completion functions see all words on the command line verbatim,
> including any backslash-escapes, single and double quotes that might
> be there.  Furthermore, git commands quote pathnames if they contain
> certain special characters.  All these create various issues when
> doing git-aware path completion.
> 
> Add a couple of failing tests to demonstrate these issues.
> 
> Later patches in this series will discuss these issues in detail as
> they fix them.
> 
> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
> ---
> 
> Notes:
>     Do any more new tests need FUNNYNAMES* prereq?

Yes.

Show 36 quoted lines

>  t/t9902-completion.sh | 91 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 91 insertions(+)
> 
> diff --git a/t/t9902-completion.sh b/t/t9902-completion.sh
> index b7f5b1e632..ff2e4a8f5f 100755
> --- a/t/t9902-completion.sh
> +++ b/t/t9902-completion.sh
> @@ -1427,6 +1427,97 @@ test_expect_success 'complete files' '
>  	test_completion "git add mom" "momified"
>  '
>  
> +# The next tests only care about how the completion script deals with
> +# unusual characters in path names.  By defining a custom completion
> +# function to list untracked files they won't be influenced by future
> +# changes of the completion functions of real git commands, and we
> +# don't have to bother with adding files to the index in these tests.
> +_git_test_path_comp ()
> +{
> +	__git_complete_index_file --others
> +}
> +
> +test_expect_failure 'complete files - escaped characters on cmdline' '
> +	test_when_finished "rm -rf \"New|Dir\"" &&
> +	mkdir "New|Dir" &&
> +	>"New|Dir/New&File.c" &&
> +
> +	test_completion "git test-path-comp N" \
> +			"New|Dir" &&	# Bash will turn this into "New\|Dir/"
> +	test_completion "git test-path-comp New\\|D" \
> +			"New|Dir" &&
> +	test_completion "git test-path-comp New\\|Dir/N" \
> +			"New|Dir/New&File.c" &&	# Bash will turn this into
> +						# "New\|Dir/New\&File.c "
> +	test_completion "git test-path-comp New\\|Dir/New\\&F" \
> +			"New|Dir/New&File.c"
> +'

This fails with:

2018-04-18T11:12:55.0436371Z expecting success: 2018-04-18T11:12:55.0436665Z test_when_finished "rm -rf \"New|Dir\"" && 2018-04-18T11:12:55.0436799Z mkdir "New|Dir" && 2018-04-18T11:12:55.0436904Z >"New|Dir/New&File.c" && 2018-04-18T11:12:55.0436972Z 2018-04-18T11:12:55.0437158Z test_completion "git test-path-comp N" \ 2018-04-18T11:12:55.0437296Z "New|Dir" && # Bash will turn this into "New\|Dir/" 2018-04-18T11:12:55.0437413Z test_completion "git test-path-comp New\\|D" \ 2018-04-18T11:12:55.0437522Z "New|Dir" && 2018-04-18T11:12:55.0437629Z test_completion "git test-path-comp New\\|Dir/N" \ 2018-04-18T11:12:55.0437767Z "New|Dir/New&File.c" && # Bash will turn this into 2018-04-18T11:12:55.0438040Z # "New\|Dir/New\&File.c " 2018-04-18T11:12:55.0438152Z test_completion "git test-path-comp New\\|Dir/New\\&F" \ 2018-04-18T11:12:55.0438504Z "New|Dir/New&File.c" 2018-04-18T11:12:55.0438742Z 2018-04-18T11:12:55.0590984Z ++ test_when_finished 'rm -rf "New|Dir"' 2018-04-18T11:12:55.0591722Z ++ test 0 = 0 2018-04-18T11:12:55.0592001Z ++ test_cleanup='{ rm -rf "New|Dir" 2018-04-18T11:12:55.0592290Z } && (exit "$eval_ret"); eval_ret=$?; :' 2018-04-18T11:12:55.0592472Z ++ mkdir 'New|Dir' 2018-04-18T11:12:55.0717255Z ++ test_completion 'git test-path-comp N' 'New|Dir' 2018-04-18T11:12:55.0717680Z ++ test 2 -gt 1 2018-04-18T11:12:55.0718062Z ++ printf '%s\n' 'New|Dir' 2018-04-18T11:12:55.0718275Z ++ run_completion 'git test-path-comp N' 2018-04-18T11:12:55.0718447Z ++ local -a COMPREPLY _words 2018-04-18T11:12:55.0718631Z ++ local _cword 2018-04-18T11:12:55.0718806Z ++ _words=($1) 2018-04-18T11:12:55.0718965Z ++ test N = ' ' 2018-04-18T11:12:55.0719124Z ++ (( _cword = 3 - 1 )) 2018-04-18T11:12:55.0719286Z ++ __git_wrap__git_main 2018-04-18T11:12:55.0719467Z ++ __git_func_wrap __git_main 2018-04-18T11:12:55.0719633Z ++ local cur words cword prev 2018-04-18T11:12:55.0719801Z ++ _get_comp_words_by_ref -n =: cur words cword prev 2018-04-18T11:12:55.0720074Z ++ '[' 6 -gt 0 ']' 2018-04-18T11:12:55.0720239Z ++ case "$1" in 2018-04-18T11:12:55.0720406Z ++ shift 2018-04-18T11:12:55.0720584Z ++ '[' 5 -gt 0 ']' 2018-04-18T11:12:55.0720742Z ++ case "$1" in 2018-04-18T11:12:55.0720899Z ++ shift 2018-04-18T11:12:55.0721054Z ++ '[' 4 -gt 0 ']' 2018-04-18T11:12:55.0721240Z ++ case "$1" in 2018-04-18T11:12:55.0721392Z ++ cur=N 2018-04-18T11:12:55.0721547Z ++ shift 2018-04-18T11:12:55.0721717Z ++ '[' 3 -gt 0 ']' 2018-04-18T11:12:55.0721879Z ++ case "$1" in 2018-04-18T11:12:55.0722040Z ++ words=("${_words[@]}") 2018-04-18T11:12:55.0722201Z ++ shift 2018-04-18T11:12:55.0722396Z ++ '[' 2 -gt 0 ']' 2018-04-18T11:12:55.0722931Z ++ case "$1" in 2018-04-18T11:12:55.0723070Z ++ cword=2 2018-04-18T11:12:55.0723221Z ++ shift 2018-04-18T11:12:55.0723357Z ++ '[' 1 -gt 0 ']' 2018-04-18T11:12:55.0723575Z ++ case "$1" in 2018-04-18T11:12:55.0723735Z ++ prev=test-path-comp 2018-04-18T11:12:55.0723874Z ++ shift 2018-04-18T11:12:55.0724009Z ++ '[' 0 -gt 0 ']' 2018-04-18T11:12:55.0724397Z ++ __git_main 2018-04-18T11:12:55.0724984Z ++ local i c=1 command __git_dir __git_repo_path 2018-04-18T11:12:55.0725183Z ++ local __git_C_args C_args_count=0 2018-04-18T11:12:55.0725353Z ++ '[' 1 -lt 2 ']' 2018-04-18T11:12:55.0725537Z ++ i=test-path-comp 2018-04-18T11:12:55.0725712Z ++ case "$i" in 2018-04-18T11:12:55.0725882Z ++ command=test-path-comp 2018-04-18T11:12:55.0726057Z ++ break 2018-04-18T11:12:55.0726270Z ++ '[' -z test-path-comp ']' 2018-04-18T11:12:55.0726446Z ++ __git_complete_command test-path-comp 2018-04-18T11:12:55.0726621Z ++ local command=test-path-comp 2018-04-18T11:12:55.0726816Z ++ local completion_func=_git_test_path_comp 2018-04-18T11:12:55.0726992Z ++ declare -f _git_test_path_comp 2018-04-18T11:12:55.0727353Z ++ declare -f _git_test_path_comp 2018-04-18T11:12:55.0727547Z ++ _git_test_path_comp 2018-04-18T11:12:55.0727716Z ++ __git_complete_index_file --others 2018-04-18T11:12:55.0727890Z ++ local dequoted_word pfx= cur_ 2018-04-18T11:12:55.0728234Z ++ __git_dequote N 2018-04-18T11:12:55.0728418Z ++ local rest=N len ch 2018-04-18T11:12:55.0728869Z ++ dequoted_word= 2018-04-18T11:12:55.0729020Z ++ test -n N 2018-04-18T11:12:55.0729152Z ++ len=0 2018-04-18T11:12:55.0729309Z ++ dequoted_word=N 2018-04-18T11:12:55.0729440Z ++ rest= 2018-04-18T11:12:55.0729666Z ++ case "${rest:0:1}" in 2018-04-18T11:12:55.0729822Z ++ test -n '' 2018-04-18T11:12:55.0729993Z ++ case "$dequoted_word" in 2018-04-18T11:12:55.0730133Z ++ cur_=N 2018-04-18T11:12:55.0782504Z +++ __git_index_files --others '' N 2018-04-18T11:12:55.0782805Z +++ local root= match=N 2018-04-18T11:12:55.0845235Z +++ __git_ls_files_helper '' --others N 2018-04-18T11:12:55.0845440Z +++ '[' --others == --committable ']' 2018-04-18T11:12:55.0845567Z +++ __git -C '' -c core.quotePath=false ls-files --exclude-standard --others -- 'N*' 2018-04-18T11:12:55.0845706Z +++ git -C '' -c core.quotePath=false ls-files --exclude-standard --others -- 'N*' 2018-04-18T11:12:55.0907632Z +++ awk -F / -v pfx= '{ 2018-04-18T11:12:55.0907806Z paths[$1] = 1 2018-04-18T11:12:55.0908985Z } 2018-04-18T11:12:55.0942839Z END { 2018-04-18T11:12:55.0943072Z for (p in paths) { 2018-04-18T11:12:55.0949175Z if (substr(p, 1, 1) != "\"") { 2018-04-18T11:12:55.0949458Z # No special characters, easy! 2018-04-18T11:12:55.0949659Z print pfx p 2018-04-18T11:12:55.0949823Z continue 2018-04-18T11:12:55.0949999Z } 2018-04-18T11:12:55.0950121Z 2018-04-18T11:12:55.0950335Z # The path is quoted. 2018-04-18T11:12:55.0950829Z p = dequote(p) 2018-04-18T11:12:55.0951171Z if (p == "") 2018-04-18T11:12:55.0951555Z continue 2018-04-18T11:12:55.0951672Z 2018-04-18T11:12:55.0951856Z # Even when a directory name itself does not contain 2018-04-18T11:12:55.0952038Z # any special characters, it will still be quoted if 2018-04-18T11:12:55.0952213Z # any of its (stripped) trailing path components do. 2018-04-18T11:12:55.0952407Z # Because of this we may have seen the same direcory 2018-04-18T11:12:55.0952583Z # both quoted and unquoted. 2018-04-18T11:12:55.0952762Z if (p in paths) 2018-04-18T11:12:55.0952948Z # We have seen the same directory unquoted, 2018-04-18T11:12:55.0953117Z # skip it. 2018-04-18T11:12:55.0953276Z continue 2018-04-18T11:12:55.0953441Z else 2018-04-18T11:12:55.0953613Z print pfx p 2018-04-18T11:12:55.0953766Z } 2018-04-18T11:12:55.0953914Z } 2018-04-18T11:12:55.0954461Z function dequote(p, bs_idx, out, esc, esc_idx, dec) { 2018-04-18T11:12:55.0954650Z # Skip opening double quote. 2018-04-18T11:12:55.0954813Z p = substr(p, 2) 2018-04-18T11:12:55.0954935Z 2018-04-18T11:12:55.0955237Z # Interpret backslash escape sequences. 2018-04-18T11:12:55.0955415Z while ((bs_idx = index(p, "\\")) != 0) { 2018-04-18T11:12:55.0955533Z out = out substr(p, 1, bs_idx - 1) 2018-04-18T11:12:55.0955638Z esc = substr(p, bs_idx + 1, 1) 2018-04-18T11:12:55.0955743Z p = substr(p, bs_idx + 2) 2018-04-18T11:12:55.0955830Z 2018-04-18T11:12:55.0955939Z if ((esc_idx = index("abtvfr\"\\", esc)) != 0) { 2018-04-18T11:12:55.0956079Z # C-style one-character escape sequence. 2018-04-18T11:12:55.0956513Z out = out substr("\a\b\t\v\f\r\"\\", 2018-04-18T11:12:55.0956631Z esc_idx, 1) 2018-04-18T11:12:55.0956745Z } else if (esc == "n") { 2018-04-18T11:12:55.0956853Z # Uh-oh, a newline character. 2018-04-18T11:12:55.0956973Z # We cant reliably put a pathname 2018-04-18T11:12:55.0957086Z # containing a newline into COMPREPLY, 2018-04-18T11:12:55.0957193Z # and the newline would create a mess. 2018-04-18T11:12:55.0957300Z # Skip this path. 2018-04-18T11:12:55.0957413Z return "" 2018-04-18T11:12:55.0957510Z } else { 2018-04-18T11:12:55.0957808Z # Must be a \nnn octal value, then. 2018-04-18T11:12:55.0958070Z dec = esc * 64 + \ 2018-04-18T11:12:55.0958184Z substr(p, 1, 1) * 8 + \ 2018-04-18T11:12:55.0958274Z substr(p, 2, 1) 2018-04-18T11:12:55.0958369Z out = out sprintf("%c", dec) 2018-04-18T11:12:55.0958587Z p = substr(p, 3) 2018-04-18T11:12:55.0958692Z } 2018-04-18T11:12:55.0958769Z } 2018-04-18T11:12:55.0958862Z # Drop closing double quote, if there is one. 2018-04-18T11:12:55.0958969Z # (There isnt any if this is a directory, as it was 2018-04-18T11:12:55.0959153Z # already stripped with the trailing path components.) 2018-04-18T11:12:55.0959256Z if (substr(p, length(p), 1) == "\"") 2018-04-18T11:12:55.0959356Z out = out substr(p, 1, length(p) - 1) 2018-04-18T11:12:55.0959441Z else 2018-04-18T11:12:55.0959541Z out = out p 2018-04-18T11:12:55.0959598Z 2018-04-18T11:12:55.0959682Z return out 2018-04-18T11:12:55.0959763Z }' 2018-04-18T11:12:55.1182135Z ++ __gitcomp_file_direct $'New∩\201╝Dir' 2018-04-18T11:12:55.1182355Z ++ local 'IFS= 2018-04-18T11:12:55.1182439Z ' 2018-04-18T11:12:55.1182518Z ++ COMPREPLY=($1) 2018-04-18T11:12:55.1182622Z ++ compopt -o filenames +o nospace 2018-04-18T11:12:55.1182877Z ++ compgen -f /non-existing-dir/ 2018-04-18T11:12:55.1182979Z ++ return 0 2018-04-18T11:12:55.1183055Z ++ return 2018-04-18T11:12:55.1183147Z ++ print_comp 2018-04-18T11:12:55.1183224Z ++ local 'IFS= 2018-04-18T11:12:55.1183300Z ' 2018-04-18T11:12:55.1183398Z ++ echo $'New∩\201╝Dir' 2018-04-18T11:12:55.1183508Z ++ sort out 2018-04-18T11:12:55.1183605Z ++ /usr/bin/sort out 2018-04-18T11:12:55.1306331Z ++ test_cmp expected out_sorted 2018-04-18T11:12:55.1306825Z ++ mingw_test_cmp expected out_sorted 2018-04-18T11:12:55.1307024Z ++ local test_cmp_a= test_cmp_b= 2018-04-18T11:12:55.1307233Z ++ local stdin_for_diff= 2018-04-18T11:12:55.1307401Z ++ test -s expected 2018-04-18T11:12:55.1307568Z ++ test -s out_sorted 2018-04-18T11:12:55.1307742Z ++ mingw_read_file_strip_cr_ test_cmp_a 2018-04-18T11:12:55.1308083Z ++ local line 2018-04-18T11:12:55.1308424Z ++ : 2018-04-18T11:12:55.1308566Z ++ IFS=$'\r' 2018-04-18T11:12:55.1308717Z ++ read -r -d ' 2018-04-18T11:12:55.1308852Z ' line 2018-04-18T11:12:55.1317521Z ++ line='New|Dir 2018-04-18T11:12:55.1317784Z ' 2018-04-18T11:12:55.1318257Z ++ eval 'test_cmp_a=$test_cmp_a$line' 2018-04-18T11:12:55.1318424Z +++ test_cmp_a='New|Dir 2018-04-18T11:12:55.1318569Z ' 2018-04-18T11:12:55.1318724Z ++ : 2018-04-18T11:12:55.1318871Z ++ IFS=$'\r' 2018-04-18T11:12:55.1319027Z ++ read -r -d ' 2018-04-18T11:12:55.1319170Z ' line 2018-04-18T11:12:55.1319334Z ++ test -z '' 2018-04-18T11:12:55.1319476Z ++ break 2018-04-18T11:12:55.1319628Z ++ mingw_read_file_strip_cr_ test_cmp_b 2018-04-18T11:12:55.1319797Z ++ local line 2018-04-18T11:12:55.1319939Z ++ : 2018-04-18T11:12:55.1320081Z ++ IFS=$'\r' 2018-04-18T11:12:55.1320240Z ++ read -r -d ' 2018-04-18T11:12:55.1320384Z ' line 2018-04-18T11:12:55.1320555Z ++ line='New∩ü╝Dir 2018-04-18T11:12:55.1320915Z ' 2018-04-18T11:12:55.1321099Z ++ eval 'test_cmp_b=$test_cmp_b$line' 2018-04-18T11:12:55.1321266Z +++ test_cmp_b='New∩ü╝Dir 2018-04-18T11:12:55.1321422Z ' 2018-04-18T11:12:55.1321570Z ++ : 2018-04-18T11:12:55.1321705Z ++ IFS=$'\r' 2018-04-18T11:12:55.1321859Z ++ read -r -d ' 2018-04-18T11:12:55.1321994Z ' line 2018-04-18T11:12:55.1322219Z ++ test -z '' 2018-04-18T11:12:55.1322361Z ++ break 2018-04-18T11:12:55.1322497Z ++ test -n 'New|Dir 2018-04-18T11:12:55.1322649Z ' 2018-04-18T11:12:55.1322828Z ++ test -n 'New∩ü╝Dir 2018-04-18T11:12:55.1322977Z ' 2018-04-18T11:12:55.1323109Z ++ test 'New|Dir 2018-04-18T11:12:55.1323397Z ' = 'New∩ü╝Dir 2018-04-18T11:12:55.1323540Z ' 2018-04-18T11:12:55.1323680Z ++ eval 'diff -u "$@" ' 2018-04-18T11:12:55.1323840Z +++ diff -u expected out_sorted 2018-04-18T11:12:55.1454977Z --- expected 2018-04-18 11:12:55.065444100 +0000 2018-04-18T11:12:55.1455785Z error: last command exited with $?=1 2018-04-18T11:12:55.1456722Z +++ out_sorted 2018-04-18 11:12:55.127568400 +0000 2018-04-18T11:12:55.1457211Z @@ -1 +1 @@ 2018-04-18T11:12:55.1457408Z -New|Dir 2018-04-18T11:12:55.1457752Z +New∩ü╝Dir 2018-04-18T11:12:55.1457975Z not ok 111 - complete files - escaped characters on cmdline 2018-04-18T11:12:55.1645995Z # 2018-04-18T11:12:55.1646221Z # test_when_finished "rm -rf \"New|Dir\"" && 2018-04-18T11:12:55.1646380Z # mkdir "New|Dir" && 2018-04-18T11:12:55.1646487Z # >"New|Dir/New&File.c" && 2018-04-18T11:12:55.1646583Z # 2018-04-18T11:12:55.1646865Z # test_completion "git test-path-comp N" \ 2018-04-18T11:12:55.1646986Z # "New|Dir" && # Bash will turn this into "New\|Dir/" 2018-04-18T11:12:55.1647108Z # test_completion "git test-path-comp New\\|D" \ 2018-04-18T11:12:55.1647212Z # "New|Dir" && 2018-04-18T11:12:55.1647346Z # test_completion "git test-path-comp New\\|Dir/N" \ 2018-04-18T11:12:55.1647510Z # "New|Dir/New&File.c" && # Bash will turn this into 2018-04-18T11:12:55.1647636Z # # "New\|Dir/New\&File.c " 2018-04-18T11:12:55.1647775Z # test_completion "git test-path-comp New\\|Dir/New\\&F" \ 2018-04-18T11:12:55.1647886Z # "New|Dir/New&File.c"

I suspect that the culprit is once again Cygwin's trick where illegal characters are mapped into a private Unicode page. Cygwin (and therefore MSYS2 runtime, and therefore the Bash used to run the test script) can use those filenames all right, but Git cannot.

So even testing whether you could write an illegal file name via shell script is *not* enough to determine whether the file system supports funny characters.

As far as I can tell from a *really* cursory glance, this is the only affected test case. Apparently your prereq catches, somehow, on Windows:

2018-04-18T11:12:43.0459702Z Your filesystem does not allow \ and " in filenames. 2018-04-18T11:12:43.0459823Z skipped: complete files - C-style escapes in ls-files output (missing FUNNYNAMES_BS_DQ)

Ciao, Dscho