{"thread":{"id":"61174","subject":"[PATCH 3/6] fast-import: release unfreed strbufs","startedAt":"2024-03-22T00:03:45Z","lastAt":"2024-04-15T17:10:26Z","messageCount":84,"participants":["Thalia Archibald","Patrick Steinhardt","Eric Sunshine","Junio C Hamano","Chris Torek"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"491189","messageId":"20240322000304.76810-2-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 1/6] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:18Z","receivedAt":"2024-03-22T00:03:45Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Path parsing in fast-import is inconsistent and many unquoting errors\nare suppressed.\n\n`` appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n filecopy ::= 'C' SP SP LF\n filerename ::= 'R' SP SP LF\n ls ::= 'ls' SP SP LF\n ls-commit ::= 'ls' SP LF\n\nand fast-import.c parses them in five different ways:\n\n1. For filemodify and filedelete:\n If `` is a valid quoted string, unquote it;\n otherwise, treat it as literal bytes (including any number of SP).\n2. For filecopy (source) and filerename (source):\n If `` is a valid quoted string, unquote it;\n otherwise, treat it as literal bytes until the next SP.\n3. For filecopy (dest) and filerename (dest):\n Like 1., but an unquoted empty string is an error.\n4. For ls:\n If `` starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes (including any number of SP).\n5. For ls-commit:\n Unquote `` and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\nIn the first three, any errors from trying to unquote a string are\nsuppressed, so a quoted string that contains invalid escapes would be\ninterpreted as literal bytes. For example, `\"\\xff\"` would fail to\nunquote (because hex escapes are not supported), and it would instead be\ninterpreted as the byte sequence `\"` `\\` `x` `f` `f` `\"`, which is\ncertainly not intended. Some front-ends erroneously use their language's\nstandard quoting routine and could silently introduce escapes that would\nbe incorrectly parsed due to this.\n\nThe documentation states that “To use a source path that contains SP the\npath must be quoted.”, so it is expected that some implementations\ndepend on spaces being allowed in paths in the final position. Thus we\nhave two documented ways to parse paths, so simplify the implementation\nto that.\n\nNow we have:\n\n1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n If `` starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes (including any number of SP).\n Garbage after a quoted string or an unquoted empty string are errors.\n (In ls-commit, it must be quoted to disambiguate from ls.)\n\n2. `parse_path_space` for filecopy (source) and filerename (source):\n\n If `` starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes until the next SP.\n It must be followed by a SP. An unquoted empty string is an error.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 3 +-\n builtin/fast-import.c | 115 ++++++++------\n t/t9300-fast-import.sh | 252 +++++++++++++++++++++++++++++-\n 3 files changed, 316 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex b2607366b9..271bd63a10 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -649,7 +649,8 @@ The value of `` must be in canonical form. That is it must not:\n * contain the special component `.` or `..` (e.g. `foo/./bar` and\n `foo/../bar` are invalid).\n \n-The root of the tree can be represented by an empty string as ``.\n+The root of the tree can be represented by a quoted empty string (`\"\"`)\n+as ``.\n \n It is recommended that `` always be encoded using UTF-8.\n \ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 71a195ca22..b2adec8d9a 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2224,7 +2224,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n *\n * idnum ::= ':' bigint;\n *\n- * Return the first character after the value in *endptr.\n+ * Update *endptr to point to the first character after the value.\n *\n * Complain if the following character is not what is expected,\n * either a space or end of the string.\n@@ -2257,8 +2257,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n }\n \n /*\n- * Parse the mark reference, demanding a trailing space. Return a\n- * pointer to the space.\n+ * Parse the mark reference, demanding a trailing space. Update *p to\n+ * point to the first character after the space.\n */\n static uintmax_t parse_mark_ref_space(const char **p)\n {\n@@ -2272,10 +2272,57 @@ static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n \n+/*\n+ * Parse the path string into the strbuf. It may be quoted with escape sequences\n+ * or unquoted without escape sequences. When unquoted, it may only contain a\n+ * space if `allow_spaces` is nonzero.\n+ */\n+static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n+{\n+\tstrbuf_reset(sb);\n+\tif (*p == '\"') {\n+\t\tif (unquote_c_style(sb, p, endp))\n+\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t} else {\n+\t\tif (allow_spaces)\n+\t\t\t*endp = p + strlen(p);\n+\t\telse\n+\t\t\t*endp = strchr(p, ' ');\n+\t\tif (*endp == p)\n+\t\t\tdie(\"Missing %s: %s\", field, command_buf.buf);\n+\t\tstrbuf_add(sb, p, *endp - p);\n+\t}\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and complain if this is not the end of\n+ * the string. It may contain spaces even when unquoted.\n+ */\n+static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n+{\n+\tconst char *end;\n+\n+\tparse_path(sb, p, &end, 1, field);\n+\tif (*end)\n+\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and ensure it is followed by a space.\n+ * It may not contain spaces when unquoted. Update *endp to point to the first\n+ * character after the space.\n+ */\n+static void parse_path_space(struct strbuf *sb, const char *p, const char **endp, const char *field)\n+{\n+\tparse_path(sb, p, endp, 0, field);\n+\tif (**endp != ' ')\n+\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n+\t(*endp)++;\n+}\n+\n static void file_change_m(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2312,12 +2359,8 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n@@ -2381,48 +2424,23 @@ static void file_change_m(const char *p, struct branch *b)\n static void file_change_d(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \n-\tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n }\n \n-static void file_change_cr(const char *s, struct branch *b, int rename)\n+static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *d;\n+\tconst char *s, *d;\n \tstatic struct strbuf s_uq = STRBUF_INIT;\n \tstatic struct strbuf d_uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct tree_entry leaf;\n \n-\tstrbuf_reset(&s_uq);\n-\tif (!unquote_c_style(&s_uq, s, &endp)) {\n-\t\tif (*endp != ' ')\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t} else {\n-\t\tendp = strchr(s, ' ');\n-\t\tif (!endp)\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t\tstrbuf_add(&s_uq, s, endp - s);\n-\t}\n+\tparse_path_space(&s_uq, p, &p, \"source\");\n+\tparse_path_eol(&d_uq, p, \"dest\");\n \ts = s_uq.buf;\n-\n-\tendp++;\n-\tif (!*endp)\n-\t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\n-\td = endp;\n-\tstrbuf_reset(&d_uq);\n-\tif (!unquote_c_style(&d_uq, d, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n-\t\td = d_uq.buf;\n-\t}\n+\td = d_uq.buf;\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n@@ -3168,6 +3186,7 @@ static void parse_ls(const char *p, struct branch *b)\n {\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n+\tstatic struct strbuf uq = STRBUF_INIT;\n \n \t/* ls SP ( SP)? */\n \tif (*p == '\"') {\n@@ -3182,16 +3201,8 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tif (*p == '\"') {\n-\t\tstatic struct strbuf uq = STRBUF_INIT;\n-\t\tconst char *endp;\n-\t\tstrbuf_reset(&uq);\n-\t\tif (unquote_c_style(&uq, p, &endp))\n-\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex dbb5042b0b..ef04b43f46 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -2146,6 +2146,7 @@ test_expect_success 'Q: deny note on empty branch' '\n \tEOF\n \ttest_must_fail git fast-import has a\n+# special case. Test every occurrence of in the grammar against every\n+# error case.\n+#\n+\n+#\n+# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n+# filerename (dest), and ls.\n+#\n+# commit :301 from root -- modify hello.c\n+# commit :302 from :301 -- modify $path\n+# commit :303 from :302 -- delete $path\n+# commit :304 from :301 -- copy hello.c $path\n+# commit :305 from :301 -- rename hello.c $path\n+# ls :305 $path\n+#\n+test_path_eol_success () {\n+\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths at EOL with $test must work\" '\n+\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_m.exp &&\n+\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n+\t\ttest_cmp tree_m.exp tree_m.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n+\t\tgit ls-tree $commit_d >tree_d.out &&\n+\t\ttest_cmp tree_d.exp tree_d.out &&\n+\n+\t\t( printf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" &&\n+\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\ttest_cmp out tree_r.exp &&\n+\n+\t\tgit branch -D path-eol\n+\t'\n+}\n+\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+\n+#\n+# Valid paths before a space: filecopy (source) and filerename (source).\n+#\n+# commit :301 from root -- modify $path\n+# commit :302 from :301 -- copy $path hello2.c\n+# commit :303 from :301 -- rename $path hello2.c\n+#\n+test_path_space_success () {\n+\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths before space with $test must work\" '\n+\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\tgit branch -D path-space\n+\t'\n+}\n+\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+\n+#\n+# Test a single commit change with an invalid path. Run it with all occurrences\n+# of in the grammar against all error kinds.\n+#\n+test_path_fail () {\n+\twhat=\"$1\" path=\"$2\" err_grep=\"$3\"\n+\ttest_expect_success \"S: $change with $what must fail\" '\n+\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :1\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <, the must be quoted.\n+change='ls (without tree-ish in commit)' prefix='ls ' field=path suffix='' \\\n+test_path_eol_quoted_fail &&\n+test_path_fail 'empty unquoted path' '' \"Invalid dataref\"\n+\n ###\n ### series T (ls)\n ###\n-- \n2.44.0\n\n\n"},{"id":"491188","messageId":"20240322000304.76810-4-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 3/6] fast-import: release unfreed strbufs","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:33Z","receivedAt":"2024-03-22T00:03:46Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"These strbufs are owned. Release them at the end of their scopes.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 29 ++++++++++++++++++-----------\n 1 file changed, 18 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 1b3d6784c1..d6f998f363 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2364,6 +2364,7 @@ static void file_change_m(const char *p, struct branch *b)\n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n \t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n+\t\tstrbuf_release(&path);\n \t\treturn;\n \t}\n \n@@ -2409,11 +2410,11 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*path.buf) {\n+\tif (*path.buf)\n+\t\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n+\telse\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n-\t\treturn;\n-\t}\n-\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n+\tstrbuf_release(&path);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n@@ -2422,6 +2423,7 @@ static void file_change_d(const char *p, struct branch *b)\n \n \tparse_path_eol(&path, p, \"path\");\n \ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n+\tstrbuf_release(&path);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n@@ -2440,17 +2442,18 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n \t\tdie(\"Path %s not in branch\", source.buf);\n-\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n+\tif (*dest.buf)\n+\t\ttree_content_set(&b->branch_tree, dest.buf,\n+\t\t\t&leaf.versions[1].oid,\n+\t\t\tleaf.versions[1].mode,\n+\t\t\tleaf.tree);\n+\telse\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n-\t\treturn;\n-\t}\n-\ttree_content_set(&b->branch_tree, dest.buf,\n-\t\t&leaf.versions[1].oid,\n-\t\tleaf.versions[1].mode,\n-\t\tleaf.tree);\n+\tstrbuf_release(&source);\n+\tstrbuf_release(&dest);\n }\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n@@ -2804,6 +2807,7 @@ static void parse_new_commit(const char *arg)\n \tfree(author);\n \tfree(committer);\n \tfree(encoding);\n+\tstrbuf_release(&msg);\n \n \tif (!store_object(OBJ_COMMIT, &new_data, NULL, &b->oid, next_mark))\n \t\tb->pack_id = pack_id;\n@@ -2886,6 +2890,7 @@ static void parse_new_tag(const char *arg)\n \tstrbuf_addch(&new_data, '\\n');\n \tstrbuf_addbuf(&new_data, &msg);\n \tfree(tagger);\n+\tstrbuf_release(&msg);\n \n \tif (store_object(OBJ_TAG, &new_data, NULL, &t->oid, next_mark))\n \t\tt->pack_id = MAX_PACK_ID;\n@@ -3171,6 +3176,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \t\tstrbuf_addch(&line, '\\n');\n \t}\n \tcat_blob_write(line.buf, line.len);\n+\tstrbuf_release(&line);\n }\n \n static void parse_ls(const char *p, struct branch *b)\n@@ -3206,6 +3212,7 @@ static void parse_ls(const char *p, struct branch *b)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n \t\trelease_tree_entry(root);\n+\tstrbuf_release(&path);\n }\n \n static void checkpoint(void)\n-- \n2.44.0\n\n\n"},{"id":"491190","messageId":"20240322000304.76810-3-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 2/6] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:25Z","receivedAt":"2024-03-22T00:03:51Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Previously, one case would not write the path to the strbuf: when the\npath is unquoted and at the end of the string. It was essentially\ncopy-on-write. However, with the logic simplification of the previous\ncommit, this case was eliminated and the strbuf is always populated.\n\nDirectly use the strbufs now instead of an alias.\n\nSince this already changes all the lines that use the strbufs, rename\nthem from `uq` to be more descriptive. That they are unquoted is not\ntheir most important property, so name them after what they carry.\n\nAdditionally, `file_change_m` no longer needs to copy the path before\nreading inline data.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 54 ++++++++++++++++++-------------------------\n 1 file changed, 22 insertions(+), 32 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex b2adec8d9a..1b3d6784c1 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2322,7 +2322,7 @@ static void parse_path_space(struct strbuf *sb, const char *p, const char **endp\n \n static void file_change_m(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2359,12 +2359,11 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n+\tparse_path_eol(&path, p, \"path\");\n \n \t/* Git does not track empty, non-toplevel directories. */\n-\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n-\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n+\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n+\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n \t\treturn;\n \t}\n \n@@ -2385,10 +2384,6 @@ static void file_change_m(const char *p, struct branch *b)\n \t\tif (S_ISDIR(mode))\n \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n \t\t\t\tcommand_buf.buf);\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\twhile (read_next_command() != EOF) {\n \t\t\tconst char *v;\n \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n@@ -2414,49 +2409,45 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*p) {\n+\tif (!*path.buf) {\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n+\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *s, *d;\n-\tstatic struct strbuf s_uq = STRBUF_INIT;\n-\tstatic struct strbuf d_uq = STRBUF_INIT;\n+\tstatic struct strbuf source = STRBUF_INIT;\n+\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n \n-\tparse_path_space(&s_uq, p, &p, \"source\");\n-\tparse_path_eol(&d_uq, p, \"dest\");\n-\ts = s_uq.buf;\n-\td = d_uq.buf;\n+\tparse_path_space(&source, p, &p, \"source\");\n+\tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n-\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n \telse\n-\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n-\t\tdie(\"Path %s not in branch\", s);\n-\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n+\t\tdie(\"Path %s not in branch\", source.buf);\n+\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, d,\n+\ttree_content_set(&b->branch_tree, dest.buf,\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n@@ -3186,7 +3177,7 @@ static void parse_ls(const char *p, struct branch *b)\n {\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n \t/* ls SP ( SP)? */\n \tif (*p == '\"') {\n@@ -3201,9 +3192,8 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_get(root, p, &leaf, 1);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\n \t * until it is saved. Save, for simplicity.\n@@ -3211,7 +3201,7 @@ static void parse_ls(const char *p, struct branch *b)\n \tif (S_ISDIR(leaf.versions[1].mode))\n \t\tstore_tree(&leaf);\n \n-\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n \tif (leaf.tree)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n-- \n2.44.0\n\n\n"},{"id":"491191","messageId":"20240322000304.76810-5-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 4/6] fast-import: remove dead strbuf","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:40Z","receivedAt":"2024-03-22T00:03:54Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The strbuf in `note_change_n` has been unused since the function was\ncreated in a8dd2e7d2b (fast-import: Add support for importing commit\nnotes, 2009-10-09) and looks to be a fossil from adapting\n`note_change_m`. Remove it.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex d6f998f363..ae8494d0ac 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2458,7 +2458,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct branch *s;\n \tstruct object_id oid, commit_oid;\n@@ -2523,10 +2522,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\tread_next_command();\n \t\tparse_and_store_blob(&last_blob, &oid, 0);\n \t} else if (oe) {\n-- \n2.44.0\n\n\n"},{"id":"491192","messageId":"20240322000304.76810-7-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 6/6] fast-import: forbid escaped NUL in paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:55Z","receivedAt":"2024-03-22T00:04:12Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"NUL cannot appear in paths. Even disregarding filesystem path\nlimitations, the tree object format delimits with NUL, so such a path\ncannot be encoded by Git.\n\nWhen a quoted path is unquoted, it could possibly contain NUL from\n\"\\000\". Forbid it so it isn't truncated.\n\nfast-import still has other issues with NUL, but those will be addressed\nlater.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 1 +\n builtin/fast-import.c | 2 ++\n t/t9300-fast-import.sh | 1 +\n 3 files changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 4aa8ccbefd..411413e8c3 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -657,6 +657,7 @@ and must be in canonical form. That is it must not:\n The root of the tree can be represented by a quoted empty string (`\"\"`)\n as ``.\n \n+`` cannot contain NUL, either literally or escaped as `\\000`.\n It is recommended that `` always be encoded using UTF-8.\n \n `filedelete`\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex ae8494d0ac..e36f59084e 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2283,6 +2283,8 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp, int\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t\tif (strlen(sb->buf) != sb->len)\n+\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n \t} else {\n \t\tif (allow_spaces)\n \t\t\t*endp = p + strlen(p);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex ef04b43f46..994a80e442 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3285,6 +3285,7 @@ test_path_fail () {\n test_path_base_fail () {\n \ttest_path_fail 'unclosed \" in '\"$field\" '\"hello.c' \"Invalid $field\"\n \ttest_path_fail \"invalid escape in quoted $field\" '\"hello\\xff\"' \"Invalid $field\"\n+\ttest_path_fail \"escaped NUL in quoted $field\" '\"hello\\000\"' \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n \ttest_path_base_fail\n-- \n2.44.0\n\n\n"},{"id":"491193","messageId":"20240322000304.76810-6-thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH 5/6] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:47Z","receivedAt":"2024-03-22T00:04:16Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Simply saying “C-style” string quoting is imprecise, as only a subset of\nC escapes are supported. Document the exact escapes.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 21 +++++++++++++--------\n 1 file changed, 13 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 271bd63a10..4aa8ccbefd 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n In both formats `` is the complete path of the file to be added\n (if not already existing) or modified (if already existing).\n \n-A `` string must use UNIX-style directory separators (forward\n-slash `/`), may contain any byte other than `LF`, and must not\n-start with double quote (`\"`).\n+A `` string may contain any byte other than `LF`, and must not\n+start with double quote (`\"`). It is interpreted as literal bytes\n+without escaping.\n \n A path can use C-style string quoting; this is accepted in all cases\n and mandatory if the filename starts with double quote or contains\n-`LF`. In C-style quoting, the complete name should be surrounded with\n-double quotes, and any `LF`, backslash, or double quote characters\n-must be escaped by preceding them with a backslash (e.g.,\n-`\"path/with\\n, \\\\ and \\\" in it\"`).\n+`LF`. In C-style quoting, the complete name is surrounded with\n+double quotes (`\"`) and certain characters must be escaped by preceding\n+them with a backslash: `LF` is written as `\\n`, backslash as `\\\\`, and\n+double quote as `\\\"`. Additionally, some characters may may optionally\n+be written with escape sequences: `\\a` for bell, `\\b` for backspace,\n+`\\f` for form feed, `\\n` for line feed, `\\r` for carriage return, `\\t`\n+for horizontal tab, and `\\v` for vertical tab. Any byte can be written\n+with 3-digit octal codes (e.g., `\\033`).\n \n-The value of `` must be in canonical form. That is it must not:\n+A `` must use UNIX-style directory separators (forward slash `/`)\n+and must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n-- \n2.44.0\n\n\n"},{"id":"491194","messageId":"E3673B5C-3D46-41A4-A204-9574F04F71D6@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-2-thalia@archibald.dev","subject":"Re: [PATCH 1/6] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:11:21Z","receivedAt":"2024-03-22T00:11:38Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Looks like my cover letter was dropped and placing each Cc: on a separate line\nonly sends to the last one. Let’s try again. Here's my cover letter and the full\nrelevant list from contrib/contacts is now CC'd:\n\n> fast-import has subtle differences in how it parses file paths between each\n> occurrence of in the grammar. Many errors were suppressed or not checked,\n> which could lead to silent data corruption. A particularly bad case was when a\n> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> supported), it would be treated as literal bytes instead of a quoted string.\n> \n> Bring path parsing into line with the documented behavior and improve\n> documentation to fill in missing details.\n> \n> This patch series is patterned after 06454cb9a3 (fast-import: tighten parsing of\n> datarefs, 2012-04-07), which did similar fixes across the grammar, but for\n> marks.\n> \n> This is my first contribution to Git, so please let me know if there's something\n> I've missed. I'm working on a tool for advanced repo transformations (like a\n> union of filter-repo and Reposurgeon workflows), so I've been living in\n> fast-import code and I have more parsing fixes planned.\n\n"},{"id":"491195","messageId":"20240322000304.76810-1-thalia@archibald.dev","threadId":"61174","inReplyTo":null,"subject":"[PATCH 0/6] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-03-22T00:03:10Z","receivedAt":"2024-03-22T00:21:50Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"fast-import has subtle differences in how it parses file paths between each\noccurrence of in the grammar. Many errors are suppressed or not checked,\nwhich could lead to silent data corruption. A particularly bad case is when a\nfront-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\nsupported), it would be treated as literal bytes instead of a quoted string.\n\nBring path parsing into line with the documented behavior and improve\ndocumentation to fill in missing details.\n\nThis patch series is patterned after 06454cb9a3 (fast-import: tighten parsing of\ndatarefs, 2012-04-07), which did similar fixes across the grammar, but for\nmarks.\n\nThis is my first contribution to Git, so please let me know if there's something\nI've missed. I'm working on a tool for advanced repo transformations (like a\nunion of filter-repo and Reposurgeon workflows), so I've been living in\nfast-import code and I have more parsing fixes planned.\n\nThalia\n\n\n"},{"id":"491728","messageId":"ZgUobnAVRCfyu1k7@tanuki","threadId":"61174","inReplyTo":"20240322000304.76810-2-thalia@archibald.dev","subject":"Re: [PATCH 1/6] fast-import: tighten parsing of paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-28T08:21:02Z","receivedAt":"2024-03-28T08:21:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 22, 2024 at 12:03:18AM +0000, Thalia Archibald wrote:\n> Path parsing in fast-import is inconsistent and many unquoting errors\n> are suppressed.\n> \n> `` appears in the grammar in these places:\n> \n> filemodify ::= 'M' SP ( | 'inline') SP LF\n> filedelete ::= 'D' SP LF\n> filecopy ::= 'C' SP SP LF\n> filerename ::= 'R' SP SP LF\n> ls ::= 'ls' SP SP LF\n> ls-commit ::= 'ls' SP LF\n> \n> and fast-import.c parses them in five different ways:\n> \n> 1. For filemodify and filedelete:\n> If `` is a valid quoted string, unquote it;\n> otherwise, treat it as literal bytes (including any number of SP).\n> 2. For filecopy (source) and filerename (source):\n> If `` is a valid quoted string, unquote it;\n> otherwise, treat it as literal bytes until the next SP.\n> 3. For filecopy (dest) and filerename (dest):\n> Like 1., but an unquoted empty string is an error.\n> 4. For ls:\n> If `` starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes (including any number of SP).\n> 5. For ls-commit:\n> Unquote `` and report parse errors.\n> (It must start with `\"` to disambiguate from ls.)\n> \n> In the first three, any errors from trying to unquote a string are\n> suppressed, so a quoted string that contains invalid escapes would be\n> interpreted as literal bytes. For example, `\"\\xff\"` would fail to\n> unquote (because hex escapes are not supported), and it would instead be\n> interpreted as the byte sequence `\"` `\\` `x` `f` `f` `\"`, which is\n> certainly not intended. Some front-ends erroneously use their language's\n> standard quoting routine and could silently introduce escapes that would\n> be incorrectly parsed due to this.\n> \n> The documentation states that “To use a source path that contains SP the\n> path must be quoted.”, so it is expected that some implementations\n> depend on spaces being allowed in paths in the final position. Thus we\n> have two documented ways to parse paths, so simplify the implementation\n> to that.\n> \n> Now we have:\n> \n> 1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n> filerename (dest), ls, and ls-commit:\n> \n> If `` starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes (including any number of SP).\n> Garbage after a quoted string or an unquoted empty string are errors.\n> (In ls-commit, it must be quoted to disambiguate from ls.)\n> \n> 2. `parse_path_space` for filecopy (source) and filerename (source):\n> \n> If `` starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes until the next SP.\n> It must be followed by a SP. An unquoted empty string is an error.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> Documentation/git-fast-import.txt | 3 +-\n> builtin/fast-import.c | 115 ++++++++------\n> t/t9300-fast-import.sh | 252 +++++++++++++++++++++++++++++-\n> 3 files changed, 316 insertions(+), 54 deletions(-)\n> \n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index b2607366b9..271bd63a10 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -649,7 +649,8 @@ The value of `` must be in canonical form. That is it must not:\n> * contain the special component `.` or `..` (e.g. `foo/./bar` and\n> `foo/../bar` are invalid).\n> \n> -The root of the tree can be represented by an empty string as ``.\n> +The root of the tree can be represented by a quoted empty string (`\"\"`)\n> +as ``.\n\nSo this is part of the \"filemodify\" section with the following syntax:\n\n 'M' SP SP SP LF\n\nThe way I interpret this change is that could previously be empty\n(`SP LF`), but now it needs to be quoted (`SP '\"' '\"' LF). This seems to\nbe related to cases (1) and (3) of your commit messages, where\n\"filemodify\" could contain an unquoted empty string whereas \"filecopy\"\nand \"filerename\" would complain about such an unquoted string.\n\nIn any case I don't see a strong argument why exactly it should be\nforbidden to have an unquoted empty path here, and I do wonder whether\nit would break existing writers of the format when we retroactively\ntighten this case. Isn't it possible to instead loosen it such that all\nthree of the above actions know to handle unquoted empty paths?\n\n> It is recommended that `` always be encoded using UTF-8.\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 71a195ca22..b2adec8d9a 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2224,7 +2224,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n> *\n> * idnum ::= ':' bigint;\n> *\n> - * Return the first character after the value in *endptr.\n> + * Update *endptr to point to the first character after the value.\n\nI think it would make sense to put these improvements for comments into\na separate commit. Otherwise it makes you wonder whether this behaviour\nis new now.\n\n> * Complain if the following character is not what is expected,\n> * either a space or end of the string.\n> @@ -2257,8 +2257,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n> }\n> \n> /*\n> - * Parse the mark reference, demanding a trailing space. Return a\n> - * pointer to the space.\n> + * Parse the mark reference, demanding a trailing space. Update *p to\n> + * point to the first character after the space.\n> */\n\nSame.\n\n> static uintmax_t parse_mark_ref_space(const char **p)\n> {\n> @@ -2272,10 +2272,57 @@ static uintmax_t parse_mark_ref_space(const char **p)\n> \treturn mark;\n> }\n> \n> +/*\n> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n> + * or unquoted without escape sequences. When unquoted, it may only contain a\n> + * space if `allow_spaces` is nonzero.\n> + */\n> +static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n> +{\n> +\tstrbuf_reset(sb);\n\nIt's not all that customary in our codebase to have the function reset\nthe `strbuf` for the caller because it does make the function less\nflexible. I would either keep the `strbuf_reset()` on the caller side or\nat least document this behaviour in the comment.\n\n> +\tif (*p == '\"') {\n> +\t\tif (unquote_c_style(sb, p, endp))\n> +\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n> +\t} else {\n> +\t\tif (allow_spaces)\n> +\t\t\t*endp = p + strlen(p);\n\nI wonder whether `stop_at_unquoted_space` might be more telling. It's\nnot like we disallow spaces here, it's that we treat them as the\nseparator to the next field.\n\n> +\t\telse\n> +\t\t\t*endp = strchr(p, ' ');\n> +\t\tif (*endp == p)\n> +\t\t\tdie(\"Missing %s: %s\", field, command_buf.buf);\n\nError messages should start with a lower-case letter and be\ntranslateable. But these are simply moved over from the previous code,\nso I don't mind much if you want to keep them as-is.\n\n> +\t\tstrbuf_add(sb, p, *endp - p);\n> +\t}\n> +}\n> +\n> +/*\n> + * Parse the path string into the strbuf, and complain if this is not the end of\n> + * the string. It may contain spaces even when unquoted.\n> + */\n> +static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n> +{\n> +\tconst char *end;\n> +\n> +\tparse_path(sb, p, &end, 1, field);\n> +\tif (*end)\n> +\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n> +}\n> +\n> +/*\n> + * Parse the path string into the strbuf, and ensure it is followed by a space.\n> + * It may not contain spaces when unquoted. Update *endp to point to the first\n> + * character after the space.\n> + */\n> +static void parse_path_space(struct strbuf *sb, const char *p, const char **endp, const char *field)\n> +{\n> +\tparse_path(sb, p, endp, 0, field);\n> +\tif (**endp != ' ')\n> +\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n> +\t(*endp)++;\n> +}\n> +\n> static void file_change_m(const char *p, struct branch *b)\n> {\n> \tstatic struct strbuf uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \tstruct object_entry *oe;\n> \tstruct object_id oid;\n> \tuint16_t mode, inline_data = 0;\n> @@ -2312,12 +2359,8 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n> \t}\n> \n> -\tstrbuf_reset(&uq);\n> -\tif (!unquote_c_style(&uq, p, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n\nThis is loosening the condition so that we also accept unquoted paths\nnow. Okay.\n\n> \n> \t/* Git does not track empty, non-toplevel directories. */\n> \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n> @@ -2381,48 +2424,23 @@ static void file_change_m(const char *p, struct branch *b)\n> static void file_change_d(const char *p, struct branch *b)\n> {\n> \tstatic struct strbuf uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \n> -\tstrbuf_reset(&uq);\n> -\tif (!unquote_c_style(&uq, p, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n> \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n> }\n\nSame.\n\n> -static void file_change_cr(const char *s, struct branch *b, int rename)\n> +static void file_change_cr(const char *p, struct branch *b, int rename)\n> {\n> -\tconst char *d;\n> +\tconst char *s, *d;\n> \tstatic struct strbuf s_uq = STRBUF_INIT;\n> \tstatic struct strbuf d_uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \tstruct tree_entry leaf;\n> \n> -\tstrbuf_reset(&s_uq);\n> -\tif (!unquote_c_style(&s_uq, s, &endp)) {\n> -\t\tif (*endp != ' ')\n> -\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n> -\t} else {\n> -\t\tendp = strchr(s, ' ');\n> -\t\tif (!endp)\n> -\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n> -\t\tstrbuf_add(&s_uq, s, endp - s);\n> -\t}\n> +\tparse_path_space(&s_uq, p, &p, \"source\");\n> +\tparse_path_eol(&d_uq, p, \"dest\");\n> \ts = s_uq.buf;\n> -\n> -\tendp++;\n> -\tif (!*endp)\n> -\t\tdie(\"Missing dest: %s\", command_buf.buf);\n> -\n> -\td = endp;\n> -\tstrbuf_reset(&d_uq);\n> -\tif (!unquote_c_style(&d_uq, d, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n> -\t\td = d_uq.buf;\n> -\t}\n> +\td = d_uq.buf;\n\nNice simplification. The source path should behave the same, and parsing\nof the destination path has been loosened to also allow unquoted paths.\n\n> \tmemset(&leaf, 0, sizeof(leaf));\n> \tif (rename)\n> @@ -3168,6 +3186,7 @@ static void parse_ls(const char *p, struct branch *b)\n> {\n> \tstruct tree_entry *root = NULL;\n> \tstruct tree_entry leaf = {NULL};\n> +\tstatic struct strbuf uq = STRBUF_INIT;\n\nI know the code had this as a static variable before, as well. But is\nthis really necessary? Can't we leave it as non-static and then release\nthe buffer at the end of this function?\n\n> \t/* ls SP ( SP)? */\n> \tif (*p == '\"') {\n> @@ -3182,16 +3201,8 @@ static void parse_ls(const char *p, struct branch *b)\n> \t\t\troot->versions[1].mode = S_IFDIR;\n> \t\tload_tree(root);\n> \t}\n> -\tif (*p == '\"') {\n> -\t\tstatic struct strbuf uq = STRBUF_INIT;\n> -\t\tconst char *endp;\n> -\t\tstrbuf_reset(&uq);\n> -\t\tif (unquote_c_style(&uq, p, &endp))\n> -\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n\nAnd this case should behave the same.\n\n> \ttree_content_get(root, p, &leaf, 1);\n> \t/*\n> \t * A directory in preparation would have a sha1 of zero\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index dbb5042b0b..ef04b43f46 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -2146,6 +2146,7 @@ test_expect_success 'Q: deny note on empty branch' '\n> \tEOF\n> \ttest_must_fail git fast-import '\n> +\n> ###\n> ### series R (feature and option)\n> ###\n> @@ -2794,7 +2795,7 @@ test_expect_success 'R: blob appears only once' '\n> '\n> \n> ###\n> -### series S\n> +### series S (mark and path parsing)\n> ###\n> #\n> # Make sure missing spaces and EOLs after mark references\n> @@ -3064,6 +3065,255 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n> \ttest_grep \"space after tree-ish\" err\n> '\n> \n> +#\n> +# Path parsing\n> +#\n> +# There are two sorts of ways a path can be parsed, depending on whether it is\n> +# the last field on the line. Additionally, ls without a has a\n> +# special case. Test every occurrence of in the grammar against every\n> +# error case.\n> +#\n> +\n> +#\n> +# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n> +# filerename (dest), and ls.\n> +#\n> +# commit :301 from root -- modify hello.c\n> +# commit :302 from :301 -- modify $path\n> +# commit :303 from :302 -- delete $path\n> +# commit :304 from :301 -- copy hello.c $path\n> +# commit :305 from :301 -- rename hello.c $path\n> +# ls :305 $path\n> +#\n> +test_path_eol_success () {\n> +\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n\nShould these variables be local?\n\n> +\ttest_expect_success \"S: paths at EOL with $test must work\" '\n> +\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n> +\t\tblob\n> +\t\tmark :401\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tblob\n> +\t\tmark :402\n> +\t\tdata < +\t\thallo welt\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/path-eol\n> +\t\tmark :301\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tinitial commit\n> +\t\tCOMMIT\n> +\t\tM 100644 :401 hello.c\n> +\n> +\t\tcommit refs/heads/path-eol\n> +\t\tmark :302\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filemodify\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tM 100644 :402 '\"$path\"'\n> +\n> +\t\tcommit refs/heads/path-eol\n> +\t\tmark :303\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filedelete\n> +\t\tCOMMIT\n> +\t\tfrom :302\n> +\t\tD '\"$path\"'\n> +\n> +\t\tcommit refs/heads/path-eol\n> +\t\tmark :304\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filecopy dest\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tC hello.c '\"$path\"'\n> +\n> +\t\tcommit refs/heads/path-eol\n> +\t\tmark :305\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filerename dest\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tR hello.c '\"$path\"'\n> +\n> +\t\tls :305 '\"$path\"'\n> +\t\tEOF\n> +\n> +\t\tcommit_m=$(grep :302 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_d=$(grep :303 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_c=$(grep :304 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_r=$(grep :305 marks.out | cut -d\\ -f2) &&\n> +\t\tblob1=$(grep :401 marks.out | cut -d\\ -f2) &&\n> +\t\tblob2=$(grep :402 marks.out | cut -d\\ -f2) &&\n> +\n> +\t\t( printf \"100644 blob $blob2\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_m.exp &&\n> +\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n> +\t\ttest_cmp tree_m.exp tree_m.out &&\n> +\n> +\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n> +\t\tgit ls-tree $commit_d >tree_d.out &&\n> +\t\ttest_cmp tree_d.exp tree_d.out &&\n> +\n> +\t\t( printf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_c.exp &&\n> +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n> +\t\ttest_cmp tree_c.exp tree_c.out &&\n> +\n> +\t\tprintf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" >tree_r.exp &&\n> +\t\tgit ls-tree $commit_r >tree_r.out &&\n> +\t\ttest_cmp tree_r.exp tree_r.out &&\n> +\n> +\t\ttest_cmp out tree_r.exp &&\n> +\n> +\t\tgit branch -D path-eol\n> +\t'\n> +}\n> +\n> +test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n> +\n> +#\n> +# Valid paths before a space: filecopy (source) and filerename (source).\n> +#\n> +# commit :301 from root -- modify $path\n> +# commit :302 from :301 -- copy $path hello2.c\n> +# commit :303 from :301 -- rename $path hello2.c\n> +#\n> +test_path_space_success () {\n> +\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n\nSame question here, should these be local?\n\n> +\ttest_expect_success \"S: paths before space with $test must work\" '\n> +\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n> +\t\tblob\n> +\t\tmark :401\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/path-space\n> +\t\tmark :301\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tinitial commit\n> +\t\tCOMMIT\n> +\t\tM 100644 :401 '\"$path\"'\n> +\n> +\t\tcommit refs/heads/path-space\n> +\t\tmark :302\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filecopy source\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tC '\"$path\"' hello2.c\n> +\n> +\t\tcommit refs/heads/path-space\n> +\t\tmark :303\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filerename source\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tR '\"$path\"' hello2.c\n> +\n> +\t\tEOF\n> +\n> +\t\tcommit_c=$(grep :302 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_r=$(grep :303 marks.out | cut -d\\ -f2) &&\n> +\t\tblob=$(grep :401 marks.out | cut -d\\ -f2) &&\n> +\n> +\t\t( printf \"100644 blob $blob\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob\\thello2.c\\n\" ) | sort >tree_c.exp &&\n> +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n> +\t\ttest_cmp tree_c.exp tree_c.out &&\n> +\n> +\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n> +\t\tgit ls-tree $commit_r >tree_r.out &&\n> +\t\ttest_cmp tree_r.exp tree_r.out &&\n> +\n> +\t\tgit branch -D path-space\n> +\t'\n> +}\n> +\n> +test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n> +\n> +#\n> +# Test a single commit change with an invalid path. Run it with all occurrences\n> +# of in the grammar against all error kinds.\n> +#\n> +test_path_fail () {\n> +\twhat=\"$1\" path=\"$2\" err_grep=\"$3\"\n\nSame.\n\n> +\ttest_expect_success \"S: $change with $what must fail\" '\n> +\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n> +\t\tblob\n> +\t\tmark :1\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/S-path-fail\n> +\t\tmark :2\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit setup\n> +\t\tCOMMIT\n> +\t\tM 100644 :1 hello.c\n> +\n> +\t\tcommit refs/heads/S-path-fail\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit with bad path\n> +\t\tCOMMIT\n> +\t\tfrom :2\n> +\t\t'\"$prefix$path$suffix\"'\n> +\t\tEOF\n> +\n> +\t\ttest_grep '\"'$err_grep'\"' err\n> +\t'\n> +}\n> +\n> +test_path_base_fail () {\n> +\ttest_path_fail 'unclosed \" in '\"$field\" '\"hello.c' \"Invalid $field\"\n> +\ttest_path_fail \"invalid escape in quoted $field\" '\"hello\\xff\"' \"Invalid $field\"\n> +}\n> +test_path_eol_quoted_fail () {\n> +\ttest_path_base_fail\n> +\ttest_path_fail \"garbage after quoted $field\" '\"hello.c\"x' \"Garbage after $field\"\n> +\ttest_path_fail \"space after quoted $field\" '\"hello.c\" ' \"Garbage after $field\"\n> +}\n> +test_path_eol_fail () {\n> +\ttest_path_eol_quoted_fail\n> +\ttest_path_fail 'empty unquoted path' '' \"Missing $field\"\n> +}\n> +test_path_space_fail () {\n> +\ttest_path_base_fail\n> +\ttest_path_fail 'empty unquoted path' '' \"Missing $field\"\n> +\ttest_path_fail \"missing space after quoted $field\" '\"hello.c\"x' \"Missing space after $field\"\n> +}\n> +\n> +change=filemodify prefix='M 100644 :1 ' field=path suffix='' test_path_eol_fail\n> +change=filedelete prefix='D ' field=path suffix='' test_path_eol_fail\n> +change=filecopy prefix='C ' field=source suffix=' world.c' test_path_space_fail\n> +change=filecopy prefix='C hello.c ' field=dest suffix='' test_path_eol_fail\n> +change=filerename prefix='R ' field=source suffix=' world.c' test_path_space_fail\n> +change=filerename prefix='R hello.c ' field=dest suffix='' test_path_eol_fail\n> +change='ls (in commit)' prefix='ls :2 ' field=path suffix='' test_path_eol_fail\n\nThis is quite confusing because you now mix two different styles, where\nsome of the functions take arguments while others pass arguments via\nvariables. I think it would be preferable to pass all arguments as\nproper function arguments.\n\nPatrick\n\n> +# When 'ls' has no , the must be quoted.\n> +change='ls (without tree-ish in commit)' prefix='ls ' field=path suffix='' \\\n> +test_path_eol_quoted_fail &&\n> +test_path_fail 'empty unquoted path' '' \"Invalid dataref\"\n> +\n> ###\n> ### series T (ls)\n> ###\n> -- \n> 2.44.0\n> \n> \n> \n"},{"id":"491729","messageId":"ZgUocuwnNQ0fsRhI@tanuki","threadId":"61174","inReplyTo":"20240322000304.76810-3-thalia@archibald.dev","subject":"Re: [PATCH 2/6] fast-import: directly use strbufs for paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-28T08:21:06Z","receivedAt":"2024-03-28T08:21:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 22, 2024 at 12:03:25AM +0000, Thalia Archibald wrote:\n> Previously, one case would not write the path to the strbuf: when the\n> path is unquoted and at the end of the string. It was essentially\n> copy-on-write. However, with the logic simplification of the previous\n> commit, this case was eliminated and the strbuf is always populated.\n> \n> Directly use the strbufs now instead of an alias.\n> \n> Since this already changes all the lines that use the strbufs, rename\n> them from `uq` to be more descriptive. That they are unquoted is not\n> their most important property, so name them after what they carry.\n> \n> Additionally, `file_change_m` no longer needs to copy the path before\n> reading inline data.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 54 ++++++++++++++++++-------------------------\n> 1 file changed, 22 insertions(+), 32 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index b2adec8d9a..1b3d6784c1 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2322,7 +2322,7 @@ static void parse_path_space(struct strbuf *sb, const char *p, const char **endp\n> \n> static void file_change_m(const char *p, struct branch *b)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n\nI was about to propose that we should likely also change all of these\nstatic variables to be local instead. I don't think that we use the\nvariables after the function calls. But now that I see that we do it\nlike this in all of these helpers I think what's going on is that this\nis a memory optimization to avoid reallocating buffers all the time.\n\nUgly, but so be it. We could refactor the code to pass in scratch\nbuffers from the outside to remove those static variables. But that\ncertainly would be a bigger change and thus likely outside of the scope\nof this patch series.\n\nPatrick\n\n> \tstruct object_entry *oe;\n> \tstruct object_id oid;\n> \tuint16_t mode, inline_data = 0;\n> @@ -2359,12 +2359,11 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n> \t}\n> \n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> +\tparse_path_eol(&path, p, \"path\");\n> \n> \t/* Git does not track empty, non-toplevel directories. */\n> -\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n> -\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n> +\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n> +\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n> \t\treturn;\n> \t}\n> \n> @@ -2385,10 +2384,6 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\tif (S_ISDIR(mode))\n> \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n> \t\t\t\tcommand_buf.buf);\n> -\t\tif (p != uq.buf) {\n> -\t\t\tstrbuf_addstr(&uq, p);\n> -\t\t\tp = uq.buf;\n> -\t\t}\n> \t\twhile (read_next_command() != EOF) {\n> \t\t\tconst char *v;\n> \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n> @@ -2414,49 +2409,45 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\t\tcommand_buf.buf);\n> \t}\n> \n> -\tif (!*p) {\n> +\tif (!*path.buf) {\n> \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n> \t\treturn;\n> \t}\n> -\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n> +\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n> }\n> \n> static void file_change_d(const char *p, struct branch *b)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> \n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> -\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n> +\tparse_path_eol(&path, p, \"path\");\n> +\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n> }\n> \n> static void file_change_cr(const char *p, struct branch *b, int rename)\n> {\n> -\tconst char *s, *d;\n> -\tstatic struct strbuf s_uq = STRBUF_INIT;\n> -\tstatic struct strbuf d_uq = STRBUF_INIT;\n> +\tstatic struct strbuf source = STRBUF_INIT;\n> +\tstatic struct strbuf dest = STRBUF_INIT;\n> \tstruct tree_entry leaf;\n> \n> -\tparse_path_space(&s_uq, p, &p, \"source\");\n> -\tparse_path_eol(&d_uq, p, \"dest\");\n> -\ts = s_uq.buf;\n> -\td = d_uq.buf;\n> +\tparse_path_space(&source, p, &p, \"source\");\n> +\tparse_path_eol(&dest, p, \"dest\");\n> \n> \tmemset(&leaf, 0, sizeof(leaf));\n> \tif (rename)\n> -\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n> +\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n> \telse\n> -\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n> +\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n> \tif (!leaf.versions[1].mode)\n> -\t\tdie(\"Path %s not in branch\", s);\n> -\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n> +\t\tdie(\"Path %s not in branch\", source.buf);\n> +\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n> \t\ttree_content_replace(&b->branch_tree,\n> \t\t\t&leaf.versions[1].oid,\n> \t\t\tleaf.versions[1].mode,\n> \t\t\tleaf.tree);\n> \t\treturn;\n> \t}\n> -\ttree_content_set(&b->branch_tree, d,\n> +\ttree_content_set(&b->branch_tree, dest.buf,\n> \t\t&leaf.versions[1].oid,\n> \t\tleaf.versions[1].mode,\n> \t\tleaf.tree);\n> @@ -3186,7 +3177,7 @@ static void parse_ls(const char *p, struct branch *b)\n> {\n> \tstruct tree_entry *root = NULL;\n> \tstruct tree_entry leaf = {NULL};\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> \n> \t/* ls SP ( SP)? */\n> \tif (*p == '\"') {\n> @@ -3201,9 +3192,8 @@ static void parse_ls(const char *p, struct branch *b)\n> \t\t\troot->versions[1].mode = S_IFDIR;\n> \t\tload_tree(root);\n> \t}\n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> -\ttree_content_get(root, p, &leaf, 1);\n> +\tparse_path_eol(&path, p, \"path\");\n> +\ttree_content_get(root, path.buf, &leaf, 1);\n> \t/*\n> \t * A directory in preparation would have a sha1 of zero\n> \t * until it is saved. Save, for simplicity.\n> @@ -3211,7 +3201,7 @@ static void parse_ls(const char *p, struct branch *b)\n> \tif (S_ISDIR(leaf.versions[1].mode))\n> \t\tstore_tree(&leaf);\n> \n> -\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n> +\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n> \tif (leaf.tree)\n> \t\trelease_tree_content_recursive(leaf.tree);\n> \tif (!b || root != &b->branch_tree)\n> -- \n> 2.44.0\n> \n> \n> \n"},{"id":"491730","messageId":"ZgUod3QKptINaPBF@tanuki","threadId":"61174","inReplyTo":"20240322000304.76810-4-thalia@archibald.dev","subject":"Re: [PATCH 3/6] fast-import: release unfreed strbufs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-28T08:21:11Z","receivedAt":"2024-03-28T08:21:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 22, 2024 at 12:03:33AM +0000, Thalia Archibald wrote:\n> These strbufs are owned. Release them at the end of their scopes.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 29 ++++++++++++++++++-----------\n> 1 file changed, 18 insertions(+), 11 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 1b3d6784c1..d6f998f363 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2364,6 +2364,7 @@ static void file_change_m(const char *p, struct branch *b)\n> \t/* Git does not track empty, non-toplevel directories. */\n> \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n> \t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n> +\t\tstrbuf_release(&path);\n> \t\treturn;\n> \t}\n\nOh, now you get to my comment in the preceding patch. With this patch\nwe're now in a somewhat weird in-between state where the buffers are\nstill static, but we release their memory after each call. So we kind of\nget the worst of both worlds: static variables without being able to\nreuse the buffer's memory.\n\nIf we were to change this then we should definitely mark the buffers as\nnon-static. If so, it would be great to demonstrate that this does not\nsignificantly impact performance.\n\nThe same is true for all the other instances.\n\nPatrick\n\n> @@ -2409,11 +2410,11 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\t\tcommand_buf.buf);\n> \t}\n> \n> -\tif (!*path.buf) {\n> +\tif (*path.buf)\n> +\t\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n> +\telse\n> \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n> -\t\treturn;\n> -\t}\n> -\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n> +\tstrbuf_release(&path);\n> }\n> \n> static void file_change_d(const char *p, struct branch *b)\n> @@ -2422,6 +2423,7 @@ static void file_change_d(const char *p, struct branch *b)\n> \n> \tparse_path_eol(&path, p, \"path\");\n> \ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n> +\tstrbuf_release(&path);\n> }\n> \n> static void file_change_cr(const char *p, struct branch *b, int rename)\n> @@ -2440,17 +2442,18 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n> \t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n> \tif (!leaf.versions[1].mode)\n> \t\tdie(\"Path %s not in branch\", source.buf);\n> -\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n> +\tif (*dest.buf)\n> +\t\ttree_content_set(&b->branch_tree, dest.buf,\n> +\t\t\t&leaf.versions[1].oid,\n> +\t\t\tleaf.versions[1].mode,\n> +\t\t\tleaf.tree);\n> +\telse\t/* C \"path/to/subdir\" \"\" */\n> \t\ttree_content_replace(&b->branch_tree,\n> \t\t\t&leaf.versions[1].oid,\n> \t\t\tleaf.versions[1].mode,\n> \t\t\tleaf.tree);\n> -\t\treturn;\n> -\t}\n> -\ttree_content_set(&b->branch_tree, dest.buf,\n> -\t\t&leaf.versions[1].oid,\n> -\t\tleaf.versions[1].mode,\n> -\t\tleaf.tree);\n> +\tstrbuf_release(&source);\n> +\tstrbuf_release(&dest);\n> }\n> \n> static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n> @@ -2804,6 +2807,7 @@ static void parse_new_commit(const char *arg)\n> \tfree(author);\n> \tfree(committer);\n> \tfree(encoding);\n> +\tstrbuf_release(&msg);\n> \n> \tif (!store_object(OBJ_COMMIT, &new_data, NULL, &b->oid, next_mark))\n> \t\tb->pack_id = pack_id;\n> @@ -2886,6 +2890,7 @@ static void parse_new_tag(const char *arg)\n> \tstrbuf_addch(&new_data, '\\n');\n> \tstrbuf_addbuf(&new_data, &msg);\n> \tfree(tagger);\n> +\tstrbuf_release(&msg);\n> \n> \tif (store_object(OBJ_TAG, &new_data, NULL, &t->oid, next_mark))\n> \t\tt->pack_id = MAX_PACK_ID;\n> @@ -3171,6 +3176,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n> \t\tstrbuf_addch(&line, '\\n');\n> \t}\n> \tcat_blob_write(line.buf, line.len);\n> +\tstrbuf_release(&line);\n> }\n> \n> static void parse_ls(const char *p, struct branch *b)\n> @@ -3206,6 +3212,7 @@ static void parse_ls(const char *p, struct branch *b)\n> \t\trelease_tree_content_recursive(leaf.tree);\n> \tif (!b || root != &b->branch_tree)\n> \t\trelease_tree_entry(root);\n> +\tstrbuf_release(&path);\n> }\n> \n> static void checkpoint(void)\n> -- \n> 2.44.0\n> \n> \n> \n"},{"id":"491731","messageId":"ZgUofCh36-W7f-yo@tanuki","threadId":"61174","inReplyTo":"20240322000304.76810-5-thalia@archibald.dev","subject":"Re: [PATCH 4/6] fast-import: remove dead strbuf","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-28T08:21:16Z","receivedAt":"2024-03-28T08:21:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 22, 2024 at 12:03:40AM +0000, Thalia Archibald wrote:\n> The strbuf in `note_change_n` has been unused since the function was\n> created in a8dd2e7d2b (fast-import: Add support for importing commit\n> notes, 2009-10-09) and looks to be a fossil from adapting\n> `note_change_m`. Remove it.\n\nJust from inspecting the diff it's not clear that it is actually unused\ngiven that we assign `p = uq.buf`. The message here should probably\nmention the important detail that `p` is not actually used after the\nassignment.\n\nPatrick\n\n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 5 -----\n> 1 file changed, 5 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index d6f998f363..ae8494d0ac 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2458,7 +2458,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n> \n> static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> \tstruct object_entry *oe;\n> \tstruct branch *s;\n> \tstruct object_id oid, commit_oid;\n> @@ -2523,10 +2522,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n> \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n> \n> \tif (inline_data) {\n> -\t\tif (p != uq.buf) {\n> -\t\t\tstrbuf_addstr(&uq, p);\n> -\t\t\tp = uq.buf;\n> -\t\t}\n> \t\tread_next_command();\n> \t\tparse_and_store_blob(&last_blob, &oid, 0);\n> \t} else if (oe) {\n> -- \n> 2.44.0\n> \n> \n> \n"},{"id":"491732","messageId":"ZgUog3UFu3WIc0L0@tanuki","threadId":"61174","inReplyTo":"20240322000304.76810-6-thalia@archibald.dev","subject":"Re: [PATCH 5/6] fast-import: document C-style escapes for paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-28T08:21:23Z","receivedAt":"2024-03-28T08:21:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 22, 2024 at 12:03:47AM +0000, Thalia Archibald wrote:\n> Simply saying “C-style” string quoting is imprecise, as only a subset of\n> C escapes are supported. Document the exact escapes.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> Documentation/git-fast-import.txt | 21 +++++++++++++--------\n> 1 file changed, 13 insertions(+), 8 deletions(-)\n> \n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index 271bd63a10..4aa8ccbefd 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n> In both formats `` is the complete path of the file to be added\n> (if not already existing) or modified (if already existing).\n> \n> -A `` string must use UNIX-style directory separators (forward\n> -slash `/`), may contain any byte other than `LF`, and must not\n> -start with double quote (`\"`).\n> +A `` string may contain any byte other than `LF`, and must not\n> +start with double quote (`\"`). It is interpreted as literal bytes\n> +without escaping.\n\nPaths also mustn't start with a space in many cases, right?\n\nPatrick\n\n> A path can use C-style string quoting; this is accepted in all cases\n> and mandatory if the filename starts with double quote or contains\n> -`LF`. In C-style quoting, the complete name should be surrounded with\n> -double quotes, and any `LF`, backslash, or double quote characters\n> -must be escaped by preceding them with a backslash (e.g.,\n> -`\"path/with\\n, \\\\ and \\\" in it\"`).\n> +`LF`. In C-style quoting, the complete name is surrounded with\n> +double quotes (`\"`) and certain characters must be escaped by preceding\n> +them with a backslash: `LF` is written as `\\n`, backslash as `\\\\`, and\n> +double quote as `\\\"`. Additionally, some characters may may optionally\n> +be written with escape sequences: `\\a` for bell, `\\b` for backspace,\n> +`\\f` for form feed, `\\n` for line feed, `\\r` for carriage return, `\\t`\n> +for horizontal tab, and `\\v` for vertical tab. Any byte can be written\n> +with 3-digit octal codes (e.g., `\\033`).\n> \n> -The value of `` must be in canonical form. That is it must not:\n> +A `` must use UNIX-style directory separators (forward slash `/`)\n> +and must be in canonical form. That is it must not:\n> \n> * contain an empty directory component (e.g. `foo//bar` is invalid),\n> * end with a directory separator (e.g. `foo/` is invalid),\n> -- \n> 2.44.0\n> \n> \n> \n"},{"id":"491942","messageId":"cover.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"20240322000304.76810-1-thalia@archibald.dev","subject":"[PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:02:03Z","receivedAt":"2024-04-01T09:02:18Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"> fast-import has subtle differences in how it parses file paths between each\n> occurrence of in the grammar. Many errors are suppressed or not checked,\n> which could lead to silent data corruption. A particularly bad case is when a\n> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> supported), it would be treated as literal bytes instead of a quoted string.\n>\n> Bring path parsing into line with the documented behavior and improve\n> documentation to fill in missing details.\n\nThanks for the review, Patrick. I've made several changes, which I think address\nyour feedback. What's the protocol for adding `Reviewed-by`? Since I don't know\nwhether I, you, or Junio add it, I've refrained from attaching your name to\nthese patches.\n\nChanges since v1:\n* In fast-import:\n * Move `strbuf_release` outside of `parse_path_space` and `parse_path_eol`.\n * Retain allocations for static `strbuf`s.\n * Rename `allow_spaces` parameter of `parse_path` to `include_spaces`.\n * Extract change to neighboring comments as patch 8.\n* In tests:\n * Test `` for the root path additionally in all tests using `\"\"`.\n * Pass all arguments by positional variables.\n * Use `local`.\n * Use `test_when_finished` instead of manual cleanup.\n* In documentation:\n * Better document conditions under which a path is considered quoted or\n unquoted.\n* Reword commit messages.\n\nThalia\n\n\nThalia Archibald (8):\n fast-import: tighten path unquoting\n fast-import: directly use strbufs for paths\n fast-import: allow unquoted empty path for root\n fast-import: remove dead strbuf\n fast-import: improve documentation for unquoted paths\n fast-import: document C-style escapes for paths\n fast-import: forbid escaped NUL in paths\n fast-import: make comments more precise\n\n Documentation/git-fast-import.txt | 30 +-\n builtin/fast-import.c | 156 ++++----\n t/t9300-fast-import.sh | 617 +++++++++++++++++++++---------\n 3 files changed, 541 insertions(+), 262 deletions(-)\n\nRange-diff against v1:\n1: 8d9e0b25cb ! 1: e790bdf714 fast-import: tighten parsing of paths\n @@ Metadata\n Author: Thalia Archibald \n\n ## Commit message ##\n - fast-import: tighten parsing of paths\n + fast-import: tighten path unquoting\n\n Path parsing in fast-import is inconsistent and many unquoting errors\n - are suppressed.\n + are suppressed or not checked.\n\n - `` appears in the grammar in these places:\n + appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n @@ Commit message\n and fast-import.c parses them in five different ways:\n\n 1. For filemodify and filedelete:\n - If `` is a valid quoted string, unquote it;\n - otherwise, treat it as literal bytes (including any number of SP).\n + Try to unquote . If it unquotes without errors, use the\n + unquoted version; otherwise, treat it as literal bytes to the end of\n + the line (including any number of SP).\n 2. For filecopy (source) and filerename (source):\n - If `` is a valid quoted string, unquote it;\n - otherwise, treat it as literal bytes until the next SP.\n + Try to unquote . If it unquotes without errors, use the\n + unquoted version; otherwise, treat it as literal bytes up to, but not\n + including, the next SP.\n 3. For filecopy (dest) and filerename (dest):\n - Like 1., but an unquoted empty string is an error.\n + Like 1., but an unquoted empty string is forbidden.\n 4. For ls:\n - If `` starts with `\"`, unquote it and report parse errors;\n - otherwise, treat it as literal bytes (including any number of SP).\n + If starts with `\"`, unquote it and report parse errors;\n + otherwise, treat it as literal bytes to the end of the line\n + (including any number of SP).\n 5. For ls-commit:\n - Unquote `` and report parse errors.\n + Unquote and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\n In the first three, any errors from trying to unquote a string are\n suppressed, so a quoted string that contains invalid escapes would be\n interpreted as literal bytes. For example, `\"\\xff\"` would fail to\n unquote (because hex escapes are not supported), and it would instead be\n - interpreted as the byte sequence `\"` `\\` `x` `f` `f` `\"`, which is\n + interpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\n certainly not intended. Some front-ends erroneously use their language's\n - standard quoting routine and could silently introduce escapes that would\n - be incorrectly parsed due to this.\n + standard quoting routine instead of matching Git's, which could silently\n + introduce escapes that would be incorrectly parsed due to this and lead\n + to data corruption.\n\n - The documentation states that “To use a source path that contains SP the\n - path must be quoted.”, so it is expected that some implementations\n - depend on spaces being allowed in paths in the final position. Thus we\n - have two documented ways to parse paths, so simplify the implementation\n - to that.\n + The documentation states “To use a source path that contains SP the path\n + must be quoted.”, so it is expected that some implementations depend on\n + spaces being allowed in paths in the final position. Thus we have two\n + documented ways to parse paths, so simplify the implementation to that.\n\n Now we have:\n\n 1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n - If `` starts with `\"`, unquote it and report parse errors;\n - otherwise, treat it as literal bytes (including any number of SP).\n - Garbage after a quoted string or an unquoted empty string are errors.\n - (In ls-commit, it must be quoted to disambiguate from ls.)\n + If starts with `\"`, unquote it and report parse errors;\n + otherwise, treat it as literal bytes to the end of the line\n + (including any number of SP).\n\n 2. `parse_path_space` for filecopy (source) and filerename (source):\n\n - If `` starts with `\"`, unquote it and report parse errors;\n - otherwise, treat it as literal bytes until the next SP.\n - It must be followed by a SP. An unquoted empty string is an error.\n + If starts with `\"`, unquote it and report parse errors;\n + otherwise, treat it as literal bytes up to, but not including, the\n + next SP. It must be followed by SP.\n +\n + There remain two special cases: The dest in filecopy and rename\n + cannot be an unquoted empty string (this will be addressed subsequently)\n + and in ls-commit must be quoted to disambiguate it from ls.\n\n Signed-off-by: Thalia Archibald \n\n - ## Documentation/git-fast-import.txt ##\n -@@ Documentation/git-fast-import.txt: The value of `` must be in canonical form. That is it must not:\n - * contain the special component `.` or `..` (e.g. `foo/./bar` and\n - `foo/../bar` are invalid).\n -\n --The root of the tree can be represented by an empty string as ``.\n -+The root of the tree can be represented by a quoted empty string (`\"\"`)\n -+as ``.\n -\n - It is recommended that `` always be encoded using UTF-8.\n -\n -\n ## builtin/fast-import.c ##\n -@@ builtin/fast-import.c: static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n - *\n - * idnum ::= ':' bigint;\n - *\n -- * Return the first character after the value in *endptr.\n -+ * Update *endptr to point to the first character after the value.\n - *\n - * Complain if the following character is not what is expected,\n - * either a space or end of the string.\n -@@ builtin/fast-import.c: static uintmax_t parse_mark_ref_eol(const char *p)\n - }\n -\n - /*\n -- * Parse the mark reference, demanding a trailing space. Return a\n -- * pointer to the space.\n -+ * Parse the mark reference, demanding a trailing space. Update *p to\n -+ * point to the first character after the space.\n - */\n - static uintmax_t parse_mark_ref_space(const char **p)\n - {\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n +/*\n + * Parse the path string into the strbuf. It may be quoted with escape sequences\n + * or unquoted without escape sequences. When unquoted, it may only contain a\n -+ * space if `allow_spaces` is nonzero.\n ++ * space if `include_spaces` is nonzero.\n + */\n -+static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n ++static void parse_path(struct strbuf *sb, const char *p, const char **endp, int include_spaces, const char *field)\n +{\n -+\tstrbuf_reset(sb);\n +\tif (*p == '\"') {\n +\t\tif (unquote_c_style(sb, p, endp))\n +\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n +\t} else {\n -+\t\tif (allow_spaces)\n ++\t\tif (include_spaces)\n +\t\t\t*endp = p + strlen(p);\n +\t\telse\n +\t\t\t*endp = strchr(p, ' ');\n -+\t\tif (*endp == p)\n -+\t\t\tdie(\"Missing %s: %s\", field, command_buf.buf);\n +\t\tstrbuf_add(sb, p, *endp - p);\n +\t}\n +}\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b)\n - \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n\n --\tstrbuf_reset(&uq);\n + \tstrbuf_reset(&uq);\n -\tif (!unquote_c_style(&uq, p, &endp)) {\n -\t\tif (*endp)\n -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n \tstatic struct strbuf uq = STRBUF_INIT;\n -\tconst char *endp;\n\n --\tstrbuf_reset(&uq);\n + \tstrbuf_reset(&uq);\n -\tif (!unquote_c_style(&uq, p, &endp)) {\n -\t\tif (*endp)\n -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n -\tconst char *endp;\n \tstruct tree_entry leaf;\n\n --\tstrbuf_reset(&s_uq);\n + \tstrbuf_reset(&s_uq);\n -\tif (!unquote_c_style(&s_uq, s, &endp)) {\n -\t\tif (*endp != ' ')\n -\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n -\t\tstrbuf_add(&s_uq, s, endp - s);\n -\t}\n +\tparse_path_space(&s_uq, p, &p, \"source\");\n -+\tparse_path_eol(&d_uq, p, \"dest\");\n \ts = s_uq.buf;\n --\n +\n -\tendp++;\n -\tif (!*endp)\n --\t\tdie(\"Missing dest: %s\", command_buf.buf);\n ++\tif (!p)\n + \t\tdie(\"Missing dest: %s\", command_buf.buf);\n -\n -\td = endp;\n --\tstrbuf_reset(&d_uq);\n + \tstrbuf_reset(&d_uq);\n -\tif (!unquote_c_style(&d_uq, d, &endp)) {\n -\t\tif (*endp)\n -\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n -\t\td = d_uq.buf;\n -\t}\n ++\tparse_path_eol(&d_uq, p, \"dest\");\n +\td = d_uq.buf;\n\n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n -@@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n +@@ builtin/fast-import.c: static void print_ls(int mode, const unsigned char *hash, const char *path)\n +\n + static void parse_ls(const char *p, struct branch *b)\n {\n ++\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n -+\tstatic struct strbuf uq = STRBUF_INIT;\n\n - \t/* ls SP ( SP)? */\n - \tif (*p == '\"') {\n @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n -\t\tp = uq.buf;\n -\t}\n ++\tstrbuf_reset(&uq);\n +\tparse_path_eol(&uq, p, \"path\");\n +\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +# Path parsing\n +#\n +# There are two sorts of ways a path can be parsed, depending on whether it is\n -+# the last field on the line. Additionally, ls without a has a\n -+# special case. Test every occurrence of in the grammar against every\n -+# error case.\n ++# the last field on the line. Additionally, ls without a has a special\n ++# case. Test every occurrence of in the grammar against every error case.\n +#\n +\n +#\n +# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n +# filerename (dest), and ls.\n +#\n -+# commit :301 from root -- modify hello.c\n ++# commit :301 from root -- modify hello.c (for setup)\n +# commit :302 from :301 -- modify $path\n +# commit :303 from :302 -- delete $path\n +# commit :304 from :301 -- copy hello.c $path\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +# ls :305 $path\n +#\n +test_path_eol_success () {\n -+\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n ++\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n +\ttest_expect_success \"S: paths at EOL with $test must work\" '\n ++\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n ++\n +\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n +\t\tblob\n +\t\tmark :401\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\thallo welt\n +\t\tBLOB\n +\n -+\t\tcommit refs/heads/path-eol\n ++\t\tcommit refs/heads/S-path-eol\n +\t\tmark :301\n +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata <tree_r.out &&\n +\t\ttest_cmp tree_r.exp tree_r.out &&\n +\n -+\t\ttest_cmp out tree_r.exp &&\n -+\n -+\t\tgit branch -D path-eol\n ++\t\ttest_cmp out tree_r.exp\n +\t'\n +}\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +#\n +# Valid paths before a space: filecopy (source) and filerename (source).\n +#\n -+# commit :301 from root -- modify $path\n ++# commit :301 from root -- modify $path (for setup)\n +# commit :302 from :301 -- copy $path hello2.c\n +# commit :303 from :301 -- rename $path hello2.c\n +#\n +test_path_space_success () {\n -+\ttest=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n ++\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n +\ttest_expect_success \"S: paths before space with $test must work\" '\n ++\t\ttest_when_finished \"git branch -D S-path-space\" &&\n ++\n +\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n +\t\tblob\n +\t\tmark :401\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\thello world\n +\t\tBLOB\n +\n -+\t\tcommit refs/heads/path-space\n ++\t\tcommit refs/heads/S-path-space\n +\t\tmark :301\n +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata < $GIT_COMMITTER_DATE\n +\t\tdata <tree_r.exp &&\n +\t\tgit ls-tree $commit_r >tree_r.out &&\n -+\t\ttest_cmp tree_r.exp tree_r.out &&\n -+\n -+\t\tgit branch -D path-space\n ++\t\ttest_cmp tree_r.exp tree_r.out\n +\t'\n +}\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +# of in the grammar against all error kinds.\n +#\n +test_path_fail () {\n -+\twhat=\"$1\" path=\"$2\" err_grep=\"$3\"\n ++\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n +\ttest_expect_success \"S: $change with $what must fail\" '\n +\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n +\t\tblob\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +}\n +\n +test_path_base_fail () {\n -+\ttest_path_fail 'unclosed \" in '\"$field\" '\"hello.c' \"Invalid $field\"\n -+\ttest_path_fail \"invalid escape in quoted $field\" '\"hello\\xff\"' \"Invalid $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n ++\ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n ++\ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n +}\n +test_path_eol_quoted_fail () {\n -+\ttest_path_base_fail\n -+\ttest_path_fail \"garbage after quoted $field\" '\"hello.c\"x' \"Garbage after $field\"\n -+\ttest_path_fail \"space after quoted $field\" '\"hello.c\" ' \"Garbage after $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n ++\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n ++\ttest_path_fail \"$change\" \"garbage after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Garbage after $field\"\n ++\ttest_path_fail \"$change\" \"space after quoted $field\" \"$prefix\" '\"hello.c\" ' \"$suffix\" \"Garbage after $field\"\n +}\n +test_path_eol_fail () {\n -+\ttest_path_eol_quoted_fail\n -+\ttest_path_fail 'empty unquoted path' '' \"Missing $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n ++\ttest_path_eol_quoted_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n +}\n +test_path_space_fail () {\n -+\ttest_path_base_fail\n -+\ttest_path_fail 'empty unquoted path' '' \"Missing $field\"\n -+\ttest_path_fail \"missing space after quoted $field\" '\"hello.c\"x' \"Missing space after $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n ++\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n ++\ttest_path_fail \"$change\" \"missing space after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Missing space after $field\"\n +}\n +\n -+change=filemodify prefix='M 100644 :1 ' field=path suffix='' test_path_eol_fail\n -+change=filedelete prefix='D ' field=path suffix='' test_path_eol_fail\n -+change=filecopy prefix='C ' field=source suffix=' world.c' test_path_space_fail\n -+change=filecopy prefix='C hello.c ' field=dest suffix='' test_path_eol_fail\n -+change=filerename prefix='R ' field=source suffix=' world.c' test_path_space_fail\n -+change=filerename prefix='R hello.c ' field=dest suffix='' test_path_eol_fail\n -+change='ls (in commit)' prefix='ls :2 ' field=path suffix='' test_path_eol_fail\n ++test_path_eol_fail filemodify 'M 100644 :1 ' path ''\n ++test_path_eol_fail filedelete 'D ' path ''\n ++test_path_space_fail filecopy 'C ' source ' world.c'\n ++test_path_eol_fail filecopy 'C hello.c ' dest ''\n ++test_path_space_fail filerename 'R ' source ' world.c'\n ++test_path_eol_fail filerename 'R hello.c ' dest ''\n ++test_path_eol_fail 'ls (in commit)' 'ls :2 ' path ''\n +\n -+# When 'ls' has no , the must be quoted.\n -+change='ls (without tree-ish in commit)' prefix='ls ' field=path suffix='' \\\n -+test_path_eol_quoted_fail &&\n -+test_path_fail 'empty unquoted path' '' \"Invalid dataref\"\n ++# When 'ls' has no , the must be quoted.\n ++test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n +\n ###\n ### series T (ls)\n2: a2aca9f9e6 ! 2: 82a6f53c13 fast-import: directly use strbufs for paths\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n\n +-\tstrbuf_reset(&uq);\n -\tparse_path_eol(&uq, p, \"path\");\n -\tp = uq.buf;\n ++\tstrbuf_reset(&path);\n +\tparse_path_eol(&path, p, \"path\");\n\n \t/* Git does not track empty, non-toplevel directories. */\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n -\tstatic struct strbuf uq = STRBUF_INIT;\n +\tstatic struct strbuf path = STRBUF_INIT;\n\n +-\tstrbuf_reset(&uq);\n -\tparse_path_eol(&uq, p, \"path\");\n -\tp = uq.buf;\n -\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n ++\tstrbuf_reset(&path);\n +\tparse_path_eol(&path, p, \"path\");\n +\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n +\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n\n +-\tstrbuf_reset(&s_uq);\n -\tparse_path_space(&s_uq, p, &p, \"source\");\n --\tparse_path_eol(&d_uq, p, \"dest\");\n -\ts = s_uq.buf;\n --\td = d_uq.buf;\n ++\tstrbuf_reset(&source);\n +\tparse_path_space(&source, p, &p, \"source\");\n +\n + \tif (!p)\n + \t\tdie(\"Missing dest: %s\", command_buf.buf);\n +-\tstrbuf_reset(&d_uq);\n +-\tparse_path_eol(&d_uq, p, \"dest\");\n +-\td = d_uq.buf;\n ++\tstrbuf_reset(&dest);\n +\tparse_path_eol(&dest, p, \"dest\");\n\n \tmemset(&leaf, 0, sizeof(leaf));\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n -@@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n +@@ builtin/fast-import.c: static void print_ls(int mode, const unsigned char *hash, const char *path)\n +\n + static void parse_ls(const char *p, struct branch *b)\n {\n - \tstruct tree_entry *root = NULL;\n - \tstruct tree_entry leaf = {NULL};\n -\tstatic struct strbuf uq = STRBUF_INIT;\n +\tstatic struct strbuf path = STRBUF_INIT;\n + \tstruct tree_entry *root = NULL;\n + \tstruct tree_entry leaf = {NULL};\n\n - \t/* ls SP ( SP)? */\n - \tif (*p == '\"') {\n @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n +-\tstrbuf_reset(&uq);\n -\tparse_path_eol(&uq, p, \"path\");\n -\tp = uq.buf;\n -\ttree_content_get(root, p, &leaf, 1);\n ++\tstrbuf_reset(&path);\n +\tparse_path_eol(&path, p, \"path\");\n +\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n3: ecaf4883d1 < -: ---------- fast-import: release unfreed strbufs\n-: ---------- > 3: 893bbf5e73 fast-import: allow unquoted empty path for root\n4: 058a38416a ! 4: cb05a184e6 fast-import: remove dead strbuf\n @@ Metadata\n ## Commit message ##\n fast-import: remove dead strbuf\n\n - The strbuf in `note_change_n` has been unused since the function was\n + The strbuf in `note_change_n` is to copy the remainder of `p` before\n + potentially invalidating it when reading the next line. However, `p` is\n + not used after that point. It has been unused since the function was\n created in a8dd2e7d2b (fast-import: Add support for importing commit\n notes, 2009-10-09) and looks to be a fossil from adapting\n - `note_change_m`. Remove it.\n + `file_change_m`. Remove it.\n\n Signed-off-by: Thalia Archibald \n\n5: a5e8df0759 < -: ---------- fast-import: document C-style escapes for paths\n6: 9792940ba9 < -: ---------- fast-import: forbid escaped NUL in paths\n-: ---------- > 5: 1f34b632d7 fast-import: improve documentation for unquoted paths\n-: ---------- > 6: 82a4da68af fast-import: document C-style escapes for paths\n-: ---------- > 7: c087c6a860 fast-import: forbid escaped NUL in paths\n-: ---------- > 8: a503c55b83 fast-import: make comments more precise\n--\n2.44.0\n\n"},{"id":"491943","messageId":"e790bdf714bb4c2a67708c016a97cf4f9e79ac48.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:02:47Z","receivedAt":"2024-04-01T09:02:56Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Path parsing in fast-import is inconsistent and many unquoting errors\nare suppressed or not checked.\n\n appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n filecopy ::= 'C' SP SP LF\n filerename ::= 'R' SP SP LF\n ls ::= 'ls' SP SP LF\n ls-commit ::= 'ls' SP LF\n\nand fast-import.c parses them in five different ways:\n\n1. For filemodify and filedelete:\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes to the end of\n the line (including any number of SP).\n2. For filecopy (source) and filerename (source):\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes up to, but not\n including, the next SP.\n3. For filecopy (dest) and filerename (dest):\n Like 1., but an unquoted empty string is forbidden.\n4. For ls:\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n5. For ls-commit:\n Unquote and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\nIn the first three, any errors from trying to unquote a string are\nsuppressed, so a quoted string that contains invalid escapes would be\ninterpreted as literal bytes. For example, `\"\\xff\"` would fail to\nunquote (because hex escapes are not supported), and it would instead be\ninterpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\ncertainly not intended. Some front-ends erroneously use their language's\nstandard quoting routine instead of matching Git's, which could silently\nintroduce escapes that would be incorrectly parsed due to this and lead\nto data corruption.\n\nThe documentation states “To use a source path that contains SP the path\nmust be quoted.”, so it is expected that some implementations depend on\nspaces being allowed in paths in the final position. Thus we have two\ndocumented ways to parse paths, so simplify the implementation to that.\n\nNow we have:\n\n1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n\n2. `parse_path_space` for filecopy (source) and filerename (source):\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes up to, but not including, the\n next SP. It must be followed by SP.\n\nThere remain two special cases: The dest in filecopy and rename\ncannot be an unquoted empty string (this will be addressed subsequently)\nand in ls-commit must be quoted to disambiguate it from ls.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 102 ++++++++++-------\n t/t9300-fast-import.sh | 251 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 309 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 782bda007c..6f9048a037 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2258,10 +2258,54 @@ static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n \n+/*\n+ * Parse the path string into the strbuf. It may be quoted with escape sequences\n+ * or unquoted without escape sequences. When unquoted, it may only contain a\n+ * space if `include_spaces` is nonzero.\n+ */\n+static void parse_path(struct strbuf *sb, const char *p, const char **endp, int include_spaces, const char *field)\n+{\n+\tif (*p == '\"') {\n+\t\tif (unquote_c_style(sb, p, endp))\n+\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t} else {\n+\t\tif (include_spaces)\n+\t\t\t*endp = p + strlen(p);\n+\t\telse\n+\t\t\t*endp = strchr(p, ' ');\n+\t\tstrbuf_add(sb, p, *endp - p);\n+\t}\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and complain if this is not the end of\n+ * the string. It may contain spaces even when unquoted.\n+ */\n+static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n+{\n+\tconst char *end;\n+\n+\tparse_path(sb, p, &end, 1, field);\n+\tif (*end)\n+\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and ensure it is followed by a space.\n+ * It may not contain spaces when unquoted. Update *endp to point to the first\n+ * character after the space.\n+ */\n+static void parse_path_space(struct strbuf *sb, const char *p, const char **endp, const char *field)\n+{\n+\tparse_path(sb, p, endp, 0, field);\n+\tif (**endp != ' ')\n+\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n+\t(*endp)++;\n+}\n+\n static void file_change_m(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2299,11 +2343,8 @@ static void file_change_m(const char *p, struct branch *b)\n \t}\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n@@ -2367,48 +2408,29 @@ static void file_change_m(const char *p, struct branch *b)\n static void file_change_d(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n }\n \n-static void file_change_cr(const char *s, struct branch *b, int rename)\n+static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *d;\n+\tconst char *s, *d;\n \tstatic struct strbuf s_uq = STRBUF_INIT;\n \tstatic struct strbuf d_uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&s_uq);\n-\tif (!unquote_c_style(&s_uq, s, &endp)) {\n-\t\tif (*endp != ' ')\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t} else {\n-\t\tendp = strchr(s, ' ');\n-\t\tif (!endp)\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t\tstrbuf_add(&s_uq, s, endp - s);\n-\t}\n+\tparse_path_space(&s_uq, p, &p, \"source\");\n \ts = s_uq.buf;\n \n-\tendp++;\n-\tif (!*endp)\n+\tif (!p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\n-\td = endp;\n \tstrbuf_reset(&d_uq);\n-\tif (!unquote_c_style(&d_uq, d, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n-\t\td = d_uq.buf;\n-\t}\n+\tparse_path_eol(&d_uq, p, \"dest\");\n+\td = d_uq.buf;\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n@@ -3152,6 +3174,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n+\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3168,16 +3191,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tif (*p == '\"') {\n-\t\tstatic struct strbuf uq = STRBUF_INIT;\n-\t\tconst char *endp;\n-\t\tstrbuf_reset(&uq);\n-\t\tif (unquote_c_style(&uq, p, &endp))\n-\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tstrbuf_reset(&uq);\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 60e30fed3c..0fb5612b07 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -2142,6 +2142,7 @@ test_expect_success 'Q: deny note on empty branch' '\n \tEOF\n \ttest_must_fail git fast-import has a special\n+# case. Test every occurrence of in the grammar against every error case.\n+#\n+\n+#\n+# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n+# filerename (dest), and ls.\n+#\n+# commit :301 from root -- modify hello.c (for setup)\n+# commit :302 from :301 -- modify $path\n+# commit :303 from :302 -- delete $path\n+# commit :304 from :301 -- copy hello.c $path\n+# commit :305 from :301 -- rename hello.c $path\n+# ls :305 $path\n+#\n+test_path_eol_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths at EOL with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_m.exp &&\n+\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n+\t\ttest_cmp tree_m.exp tree_m.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n+\t\tgit ls-tree $commit_d >tree_d.out &&\n+\t\ttest_cmp tree_d.exp tree_d.out &&\n+\n+\t\t( printf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" &&\n+\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\ttest_cmp out tree_r.exp\n+\t'\n+}\n+\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+\n+#\n+# Valid paths before a space: filecopy (source) and filerename (source).\n+#\n+# commit :301 from root -- modify $path (for setup)\n+# commit :302 from :301 -- copy $path hello2.c\n+# commit :303 from :301 -- rename $path hello2.c\n+#\n+test_path_space_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths before space with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-space\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out\n+\t'\n+}\n+\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+\n+#\n+# Test a single commit change with an invalid path. Run it with all occurrences\n+# of in the grammar against all error kinds.\n+#\n+test_path_fail () {\n+\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n+\ttest_expect_success \"S: $change with $what must fail\" '\n+\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :1\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <, the must be quoted.\n+test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n+\n ###\n ### series T (ls)\n ###\n-- \n2.44.0\n\n\n"},{"id":"491944","messageId":"82a6f53c1326a420348eb70461f5929340a930d3.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 2/8] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:06Z","receivedAt":"2024-04-01T09:03:15Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Previously, one case would not write the path to the strbuf: when the\npath is unquoted and at the end of the string. It was essentially\ncopy-on-write. However, with the logic simplification of the previous\ncommit, this case was eliminated and the strbuf is always populated.\n\nDirectly use the strbufs now instead of an alias.\n\nSince this already changes all the lines that use the strbufs, rename\nthem from `uq` to be more descriptive. That they are unquoted is not\ntheir most important property, so name them after what they carry.\n\nAdditionally, `file_change_m` no longer needs to copy the path before\nreading inline data.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 64 ++++++++++++++++++-------------------------\n 1 file changed, 27 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 6f9048a037..fad9324e59 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2305,7 +2305,7 @@ static void parse_path_space(struct strbuf *sb, const char *p, const char **endp\n \n static void file_change_m(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2342,13 +2342,12 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n \n \t/* Git does not track empty, non-toplevel directories. */\n-\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n-\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n+\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n+\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n \t\treturn;\n \t}\n \n@@ -2369,10 +2368,6 @@ static void file_change_m(const char *p, struct branch *b)\n \t\tif (S_ISDIR(mode))\n \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n \t\t\t\tcommand_buf.buf);\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\twhile (read_next_command() != EOF) {\n \t\t\tconst char *v;\n \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n@@ -2398,55 +2393,51 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*p) {\n+\tif (!*path.buf) {\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n+\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *s, *d;\n-\tstatic struct strbuf s_uq = STRBUF_INIT;\n-\tstatic struct strbuf d_uq = STRBUF_INIT;\n+\tstatic struct strbuf source = STRBUF_INIT;\n+\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n \n-\tstrbuf_reset(&s_uq);\n-\tparse_path_space(&s_uq, p, &p, \"source\");\n-\ts = s_uq.buf;\n+\tstrbuf_reset(&source);\n+\tparse_path_space(&source, p, &p, \"source\");\n \n \tif (!p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\tstrbuf_reset(&d_uq);\n-\tparse_path_eol(&d_uq, p, \"dest\");\n-\td = d_uq.buf;\n+\tstrbuf_reset(&dest);\n+\tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n-\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n \telse\n-\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n-\t\tdie(\"Path %s not in branch\", s);\n-\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n+\t\tdie(\"Path %s not in branch\", source.buf);\n+\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, d,\n+\ttree_content_set(&b->branch_tree, dest.buf,\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n@@ -3174,7 +3165,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3191,10 +3182,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_get(root, p, &leaf, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\n \t * until it is saved. Save, for simplicity.\n@@ -3202,7 +3192,7 @@ static void parse_ls(const char *p, struct branch *b)\n \tif (S_ISDIR(leaf.versions[1].mode))\n \t\tstore_tree(&leaf);\n \n-\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n \tif (leaf.tree)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n-- \n2.44.0\n\n\n"},{"id":"491945","messageId":"893bbf5e734903382c22893121252ed49b3c7a69.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 3/8] fast-import: allow unquoted empty path for root","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:17Z","receivedAt":"2024-04-01T09:03:24Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Ever since filerename was added in f39a946a1f (Support wholesale\ndirectory renames in fast-import, 2007-07-09) and filecopy in b6f3481bb4\n(Teach fast-import to recursively copy files/directories, 2007-07-15),\nboth have produced an error when the destination path is empty. Later,\nwhen support for targeting the root directory with an empty string was\nadded in 2794ad5244 (fast-import: Allow filemodify to set the root,\n2010-10-10), this had the effect of allowing the quoted empty string\n(`\"\"`), but forbidding its unquoted variant (``). This seems to have\nbeen intended as simple data validation for parsing two paths, rather\nthan a syntax restriction, because it was not extended to the other\noperations.\n\nAll other occurrences of paths (in filemodify, filedelete, the source of\nfilecopy and filerename, and ls) allow both.\n\nFor most of this feature's lifetime, the documentation has not\nprescribed the use of quoted empty strings. In e5959106d6\n(Documentation/fast-import: put explanation of M 040000 \"\" in\ncontext, 2011-01-15), its documentation was changed from “`` may\nalso be an empty string (`\"\"`) to specify the root of the tree” to “The\nroot of the tree can be represented by an empty string as ``”.\n\nThus, we can assume that some front-ends have depended on this behavior.\n\nRemove this restriction for the destination paths of filecopy and\nfilerename and change tests targeting the root to test `\"\"` and ``.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 +-\n t/t9300-fast-import.sh | 363 +++++++++++++++++++++--------------------\n 2 files changed, 191 insertions(+), 177 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex fad9324e59..58cc8d4ede 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2416,11 +2416,8 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&source);\n-\tparse_path_space(&source, p, &p, \"source\");\n-\n-\tif (!p)\n-\t\tdie(\"Missing dest: %s\", command_buf.buf);\n \tstrbuf_reset(&dest);\n+\tparse_path_space(&source, p, &p, \"source\");\n \tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 0fb5612b07..635b1b9af7 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1059,30 +1059,33 @@ test_expect_success 'M: rename subdirectory to new subdirectory' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'M: rename root to subdirectory' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/M4\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/M4\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n-\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n-\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n-\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n-\tEOF\n-\tgit fast-import actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n+\t\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n+\t\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n+\t\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n+\t\tEOF\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series N\n@@ -1259,49 +1262,52 @@ test_expect_success PIPE 'N: empty directory reads as missing' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'N: copy root directory by tree hash' '\n-\tcat >expect <<-EOF &&\n-\t:100755 000000 $newf $zero D\tfile3/newf\n-\t:100644 000000 $oldf $zero D\tfile3/oldf\n-\tEOF\n-\troot=$(git rev-parse refs/heads/branch^0^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N6\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 000000 $newf $zero D\tfile3/newf\n+\t\t:100644 000000 $oldf $zero D\tfile3/oldf\n+\t\tEOF\n+\t\troot_tree=$(git rev-parse refs/heads/branch^0^{tree}) &&\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N6\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tM 040000 $root_tree '\"$root\"'\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n \n-test_expect_success 'N: copy root by path' '\n-\tcat >expect <<-EOF &&\n-\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n-\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n-\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n-\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n-\tEOF\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N-copy-root-path\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n+\t\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n+\t\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n+\t\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n+\t\tEOF\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N-copy-root-path\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tC '\"$root\"' oldroot\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n test_expect_success 'N: delete directory by copying' '\n \tcat >expect <<-\\EOF &&\n@@ -1431,98 +1437,102 @@ test_expect_success 'N: reject foo/ syntax in ls argument' '\n \tINPUT_END\n '\n \n-test_expect_success 'N: copy to root by id and modify' '\n-\techo \"hello, world\" >expect.foo &&\n-\techo hello >expect.bar &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N7\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.foo &&\n+\t\techo hello >expect.bar &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N7\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.foo &&\n-\tgit show N8:foo/bar >actual.bar &&\n-\ttest_cmp expect.foo actual.foo &&\n-\ttest_cmp expect.bar actual.bar\n-'\n+\t\tM 040000 $tree '\"$root\"'\n+\t\tM 644 inline foo/foo\n+\t\tdata <actual.foo &&\n+\t\tgit show N8:foo/bar >actual.bar &&\n+\t\ttest_cmp expect.foo actual.foo &&\n+\t\ttest_cmp expect.bar actual.bar\n+\t'\n \n-test_expect_success 'N: extract subtree' '\n-\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N9\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/N9\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect.baz &&\n-\techo hello, world >expect.qux &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N10\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.baz &&\n+\t\techo hello, world >expect.qux &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N10\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.baz &&\n-\tgit show N11:bar/qux >actual.qux &&\n-\tgit show N11:bar/quux >actual.quux &&\n-\ttest_cmp expect.baz actual.baz &&\n-\ttest_cmp expect.qux actual.qux &&\n-\ttest_cmp expect.qux actual.quux'\n+\t\tM 040000 $tree '\"$root\"'\n+\t\tM 100644 inline foo/bar/qux\n+\t\tdata <actual.baz &&\n+\t\tgit show N11:bar/qux >actual.qux &&\n+\t\tgit show N11:bar/quux >actual.quux &&\n+\t\ttest_cmp expect.baz actual.baz &&\n+\t\ttest_cmp expect.qux actual.qux &&\n+\t\ttest_cmp expect.qux actual.quux\n+\t'\n+done\n \n ###\n ### series O\n@@ -3067,6 +3077,7 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n # There are two sorts of ways a path can be parsed, depending on whether it is\n # the last field on the line. Additionally, ls without a has a special\n # case. Test every occurrence of in the grammar against every error case.\n+# Paths for the root (empty strings) are tested elsewhere.\n #\n \n #\n@@ -3314,16 +3325,19 @@ test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n ###\n # Setup is carried over from series S.\n \n-test_expect_success 'T: ls root tree' '\n-\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n-\t040000 tree $(git rev-parse S^{tree})\tZ\n-\tEOF\n-\tsha1=$(git rev-parse --verify S) &&\n-\tgit fast-import --import-marks=marks <<-EOF >actual &&\n-\tls $sha1 \"\"\n-\tEOF\n-\ttest_cmp expect actual\n-'\n+for root in '\"\"' ''\n+do\n+\ttest_expect_success \"T: ls root ($root) tree\" '\n+\t\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n+\t\t040000 tree $(git rev-parse S^{tree})\tZ\n+\t\tEOF\n+\t\tsha1=$(git rev-parse --verify S) &&\n+\t\tgit fast-import --import-marks=marks <<-EOF >actual &&\n+\t\tls $sha1 $root\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n \n test_expect_success 'T: delete branch' '\n \tgit branch to-delete &&\n@@ -3425,30 +3439,33 @@ test_expect_success 'U: validate directory delete result' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'U: filedelete root succeeds' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/U\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/U-delete-root\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 000000 $f7id $ZERO_OID D\thello.c\n-\tEOF\n+\ttest_expect_success \"U: validate root ($root) delete result\" '\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 000000 $f7id $ZERO_OID D\thello.c\n+\t\tEOF\n \n-\tgit diff-tree -M -r U^1 U >actual &&\n+\t\tgit diff-tree -M -r U U-delete-root >actual &&\n \n-\tcompare_diff_raw expect actual\n-'\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series V (checkpoint)\n-- \n2.44.0\n\n\n"},{"id":"491946","messageId":"cb05a184e6431edd29fbdfa42980f3fda0cd85ff.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 4/8] fast-import: remove dead strbuf","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:25Z","receivedAt":"2024-04-01T09:03:34Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The strbuf in `note_change_n` is to copy the remainder of `p` before\npotentially invalidating it when reading the next line. However, `p` is\nnot used after that point. It has been unused since the function was\ncreated in a8dd2e7d2b (fast-import: Add support for importing commit\nnotes, 2009-10-09) and looks to be a fossil from adapting\n`file_change_m`. Remove it.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 58cc8d4ede..fc6eeaf89c 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2442,7 +2442,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct branch *s;\n \tstruct object_id oid, commit_oid;\n@@ -2507,10 +2506,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\tread_next_command();\n \t\tparse_and_store_blob(&last_blob, &oid, 0);\n \t} else if (oe) {\n-- \n2.44.0\n\n\n"},{"id":"491947","messageId":"1f34b632d72b0b04d22da54b8eac1520e262620a.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 5/8] fast-import: improve documentation for unquoted paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:33Z","receivedAt":"2024-04-01T09:03:43Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"It describes what cannot be in an unquoted path, but not what it is.\nReframe it as a definition of unquoted paths. The requirement that it\nnot start with `\"` is the core element that implies the rest.\n\nThe restriction that the source paths of filecopy and filerename cannot\ncontain SP is only stated in their respective sections. Restate it in\nthe section.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 19 ++++++++++++-------\n 1 file changed, 12 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex b2607366b9..f26d7a8571 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n In both formats `` is the complete path of the file to be added\n (if not already existing) or modified (if already existing).\n \n-A `` string must use UNIX-style directory separators (forward\n-slash `/`), may contain any byte other than `LF`, and must not\n-start with double quote (`\"`).\n+A `` can be written as unquoted bytes or a C-style quoted string:\n \n-A path can use C-style string quoting; this is accepted in all cases\n-and mandatory if the filename starts with double quote or contains\n-`LF`. In C-style quoting, the complete name should be surrounded with\n+When a `` does not start with double quote (`\"`), it is an\n+unquoted string and is parsed as literal bytes without any escape\n+sequences. However, if the filename contains `LF` or starts with double\n+quote, it must be written as a quoted string. Additionally, the source\n+`` in `filecopy` or `filerename` must be quoted if it contains SP.\n+\n+A `` can use C-style string quoting; this is accepted in all cases\n+and mandatory in the cases where the filename cannot be represented as\n+an unquoted string. In C-style quoting, the complete name should be surrounded with\n double quotes, and any `LF`, backslash, or double quote characters\n must be escaped by preceding them with a backslash (e.g.,\n `\"path/with\\n, \\\\ and \\\" in it\"`).\n \n-The value of `` must be in canonical form. That is it must not:\n+A `` must use UNIX-style directory separators (forward slash `/`)\n+and must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n-- \n2.44.0\n\n\n"},{"id":"491948","messageId":"82a4da68afbc92925ac716a9b098f1a79a3b2ab5.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 6/8] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:42Z","receivedAt":"2024-04-01T09:03:57Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Simply saying “C-style” string quoting is imprecise, as only a subset of\nC escapes are supported. Document the exact escapes.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 12 ++++++++----\n t/t9300-fast-import.sh | 10 ++++++----\n 2 files changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f26d7a8571..db53b50268 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -640,10 +640,14 @@ quote, it must be written as a quoted string. Additionally, the source\n \n A `` can use C-style string quoting; this is accepted in all cases\n and mandatory in the cases where the filename cannot be represented as\n-an unquoted string. In C-style quoting, the complete name should be surrounded with\n-double quotes, and any `LF`, backslash, or double quote characters\n-must be escaped by preceding them with a backslash (e.g.,\n-`\"path/with\\n, \\\\ and \\\" in it\"`).\n+an unquoted string. In C-style quoting, the complete filename is\n+surrounded with double quote (`\"`) and certain characters must be\n+escaped by preceding them with a backslash: `LF` is written as `\\n`,\n+backslash as `\\\\`, and double quote as `\\\"`. Some characters may may\n+optionally be written with escape sequences: `\\a` for bell, `\\b` for\n+backspace, `\\f` for form feed, `\\n` for line feed, `\\r` for carriage\n+return, `\\t` for horizontal tab, and `\\v` for vertical tab. Any byte can\n+be written with 3-digit octal codes (e.g., `\\033`).\n \n A `` must use UNIX-style directory separators (forward slash `/`)\n and must be in canonical form. That is it must not:\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 635b1b9af7..e10962dffe 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3185,8 +3185,9 @@ test_path_eol_success () {\n \t'\n }\n \n-test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Valid paths before a space: filecopy (source) and filerename (source).\n@@ -3250,8 +3251,9 @@ test_path_space_success () {\n \t'\n }\n \n-test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Test a single commit change with an invalid path. Run it with all occurrences\n-- \n2.44.0\n\n\n"},{"id":"491949","messageId":"c087c6a8604ddcbda5f09bb41fb9d7e746ed9c55.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 7/8] fast-import: forbid escaped NUL in paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:50Z","receivedAt":"2024-04-01T09:03:59Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"NUL cannot appear in paths. Even disregarding filesystem path\nlimitations, the tree object format delimits with NUL, so such a path\ncannot be encoded by Git.\n\nWhen a quoted path is unquoted, it could possibly contain NUL from\n\"\\000\". Forbid it so it isn't truncated.\n\nfast-import still has other issues with NUL, but those will be addressed\nlater.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 1 +\n builtin/fast-import.c | 2 ++\n t/t9300-fast-import.sh | 1 +\n 3 files changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex db53b50268..edda30f90c 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -660,6 +660,7 @@ and must be in canonical form. That is it must not:\n \n The root of the tree can be represented by an empty string as ``.\n \n+`` cannot contain NUL, either literally or escaped as `\\000`.\n It is recommended that `` always be encoded using UTF-8.\n \n `filedelete`\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex fc6eeaf89c..9d0f53fe04 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2268,6 +2268,8 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp, int\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t\tif (strlen(sb->buf) != sb->len)\n+\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n \t} else {\n \t\tif (include_spaces)\n \t\t\t*endp = p + strlen(p);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex e10962dffe..794a96df38 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3294,6 +3294,7 @@ test_path_base_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n \ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n \ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n+\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n-- \n2.44.0\n\n\n"},{"id":"491950","messageId":"a503c55b83bc610aeefd72190b514264aacb8430.1711960552.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v2 8/8] fast-import: make comments more precise","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:03:58Z","receivedAt":"2024-04-01T09:04:05Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The former is somewhat imprecise. The latter became out of sync with the\nbehavior in e814c39c2f (fast-import: refactor parsing of spaces,\n2014-06-18).\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 9d0f53fe04..9b66ffd2d0 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2210,7 +2210,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n *\n * idnum ::= ':' bigint;\n *\n- * Return the first character after the value in *endptr.\n+ * Update *endptr to point to the first character after the value.\n *\n * Complain if the following character is not what is expected,\n * either a space or end of the string.\n@@ -2243,8 +2243,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n }\n \n /*\n- * Parse the mark reference, demanding a trailing space. Return a\n- * pointer to the space.\n+ * Parse the mark reference, demanding a trailing space. Update *p to\n+ * point to the first character after the space.\n */\n static uintmax_t parse_mark_ref_space(const char **p)\n {\n-- \n2.44.0\n\n\n"},{"id":"491951","messageId":"4523FBB5-68AE-4D98-BB5F-6B498ADF1C25@archibald.dev","threadId":"61174","inReplyTo":"E01C617F-3720-42C0-83EE-04BB01643C86@archibald.dev","subject":"Re: [PATCH 1/6] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:05:48Z","receivedAt":"2024-04-01T09:05:56Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"(Sorry for first sending as HTML)\n\nOn Mar 28, 2024, at 01:21, Patrick Steinhardt wrote:\n> \n> So this is part of the \"filemodify\" section with the following syntax:\n> \n> 'M' SP SP SP LF\n> \n> The way I interpret this change is that could previously be empty\n> (`SP LF`), but now it needs to be quoted (`SP '\"' '\"' LF). This seems to\n> be related to cases (1) and (3) of your commit messages, where\n> \"filemodify\" could contain an unquoted empty string whereas \"filecopy\"\n> and \"filerename\" would complain about such an unquoted string.\n> \n> In any case I don't see a strong argument why exactly it should be\n> forbidden to have an unquoted empty path here, and I do wonder whether\n> it would break existing writers of the format when we retroactively\n> tighten this case. Isn't it possible to instead loosen it such that all\n> three of the above actions know to handle unquoted empty paths?\n\nAt first, I strongly thought that we should forbid this case of unquoted empty\npaths. It's a somewhat peculiar case in that it refers to the root of the repo\nand few front-ends use it. I surveyed git fast-export, git-filter-repo,\nReposurgeon, hg-fast-export, cvs-fast-export (by Eric S. Raymond),\ncvs-fast-export (by Roger Vaughn), svn2git, bzr-fastexport, and bk fast-export,\nbut none of them ever target the root of the repo. I assumed that if an unquoted\nempty path was ever emitted, it was likely an bug that should not be accepted\n(e.g., a null byte array somehow).\n\nHowever, most occurrences of in the grammar have allowed unquoted empty\nstrings to mean the root for 14 years and documentation has implied that it's\nallowed for 13 years. It's just the two cases of the destination paths of\nfilecopy and filerename that don't allow it, and those are less-used operations,\nso front-ends may never encounter that error.\n\nSome assumed errors in emitting empty paths are caught by validation with file\nmodes, so even if it's loosened it's still fairly safe. filemodify must be\n040000 when it targets the root, and filecopy and filerename to the root must\nhave a source path that's a directory. The worst case is filedelete\nunintentionally targeting the root, but that's been allowed to be an unquoted\nempty path for almost its entire lifetime, so I don't think should be changed.\n\nI've changed it to allow unquoted empty paths for all operations in patch v2\n3/8 (fast-import: allow unquoted empty path for root).\n\n> This is loosening the condition so that we also accept unquoted paths\n> now. Okay.\n\nOn the contrary, v1 tightens all paths to forbid unquoted empty strings. v2 now\nallows it and should make that change more clear.\n\n> On Fri, Mar 22, 2024 at 12:03:18AM +0000, Thalia Archibald wrote:\n>> +/*\n>> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n>> + * or unquoted without escape sequences. When unquoted, it may only contain a\n>> + * space if `allow_spaces` is nonzero.\n>> + */\n>> +static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n>> +{\n>> + strbuf_reset(sb);\n>> + if (*p == '\"') {\n>> + if (unquote_c_style(sb, p, endp))\n>> + die(\"Invalid %s: %s\", field, command_buf.buf);\n>> + } else {\n>> + if (allow_spaces)\n>> + *endp = p + strlen(p);\n> \n> I wonder whether `stop_at_unquoted_space` might be more telling. It's\n> not like we disallow spaces here, it's that we treat them as the\n> separator to the next field.\n\nI agree, but I’d rather something shorter, so I’ve changed it to `include_spaces`.\n\n>> + else\n>> + *endp = strchr(p, ' ');\n>> + if (*endp == p)\n>> + die(\"Missing %s: %s\", field, command_buf.buf);\n> \n> Error messages should start with a lower-case letter and be\n> translateable. But these are simply moved over from the previous code,\n> so I don't mind much if you want to keep them as-is.\n> \n>> + strbuf_add(sb, p, *endp - p);\n>> + }\n>> +}\n\n\nfast-import isn’t a porcelain command, AFAIK, so I thought the convention is\nthat its output isn't translated?\n\nFrom po/README.md:\n> \n> The output from Git's plumbing utilities will primarily be read by\n> programs and would break scripts under non-C locales if it was\n> translated. Plumbing strings should not be translated, since\n> they're part of Git's API.\n\n\nI would, however, like to improve its error messages. For example, diagnosing\nerrors more precisely or changing the outdated “GIT” to “Git”.\n\nTo what extent should the exact error messages be considered part of Git's API?\n\nThalia\n"},{"id":"491952","messageId":"9DF24328-0128-4386-82E7-662D10267242@archibald.dev","threadId":"61174","inReplyTo":"ZgUod3QKptINaPBF@tanuki","subject":"Re: [PATCH 3/6] fast-import: release unfreed strbufs","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:06:24Z","receivedAt":"2024-04-01T09:06:39Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"(Resending as plain text)\n\nOn Mar 28, 2024, at 01:21, Patrick Steinhardt wrote:\n> I was about to propose that we should likely also change all of these\n> static variables to be local instead. I don't think that we use the\n> variables after the function calls. But now that I see that we do it\n> like this in all of these helpers I think what's going on is that this\n> is a memory optimization to avoid reallocating buffers all the time.\n> \n> Ugly, but so be it. We could refactor the code to pass in scratch\n> buffers from the outside to remove those static variables. But that\n> certainly would be a bigger change and thus likely outside of the scope\n> of this patch series.\n\n\n> Oh, now you get to my comment in the preceding patch. With this patch\n> we're now in a somewhat weird in-between state where the buffers are\n> still static, but we release their memory after each call. So we kind of\n> get the worst of both worlds: static variables without being able to\n> reuse the buffer's memory.\n> \n> If we were to change this then we should definitely mark the buffers as\n> non-static. If so, it would be great to demonstrate that this does not\n> significantly impact performance.\n> \n> The same is true for all the other instances.\n\nI had glossed that they're `static`, since I've grown accustomed to Rust, where\nthis sort of non-reentrant code is discouraged. However, this pattern is great\nfor fast-import, because all of its data is simply freed when it exits at the\nend of the stream. I dropped this patch in v2.\n\nI don't think it's worth hoisting these `strbuf`s out. It would only reduce it\nfrom 5 to 2 total static `strbuf`s for paths, but would make ownership less\nclear.\n\nThalia\n\n\n"},{"id":"491953","messageId":"4C726A5B-902B-4E2F-BFB1-11660E551187@archibald.dev","threadId":"61174","inReplyTo":"ZgUog3UFu3WIc0L0@tanuki","subject":"Re: [PATCH 5/6] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-01T09:06:55Z","receivedAt":"2024-04-01T09:07:03Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"(Sending again as plain text)\n\nOn Mar 28, 2024, at 01:21, Patrick Steinhardt wrote:\n> On Fri, Mar 22, 2024 at 12:03:47AM +0000, Thalia Archibald wrote:\n>> \n>> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n>> index 271bd63a10..4aa8ccbefd 100644\n>> --- a/Documentation/git-fast-import.txt\n>> +++ b/Documentation/git-fast-import.txt\n>> @@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n>> In both formats `` is the complete path of the file to be added\n>> (if not already existing) or modified (if already existing).\n>> \n>> -A `` string must use UNIX-style directory separators (forward\n>> -slash `/`), may contain any byte other than `LF`, and must not\n>> -start with double quote (`\"`).\n>> +A `` string may contain any byte other than `LF`, and must not\n>> +start with double quote (`\"`). It is interpreted as literal bytes\n>> +without escaping.\n> \n> Paths also mustn't start with a space in many cases, right?\n\nIt talks about starting with double quote, because that's what determines\nwhether it's parsed as a quoted or unquoted string.\n\nContaining spaces is different. When unquoted, a path can only contain a space\nif it's the last field on the line; that's all paths except the source paths of\nfilecopy and filerename. That note was already remarked in the filecopy and\nfilerename sections, but it would help to note it in the general section,\nso I've done that and clarified quoting in patch v2 5/8 (fast-import: improve\ndocumentation for unquoted paths).\n\nThalia\n"},{"id":"492476","messageId":"A820E635-14FA-403C-B932-D7F9FE57C092@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-07T21:19:08Z","receivedAt":"2024-04-07T21:19:25Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 1, 2024, at 02:02, Thalia Archibald wrote:\n>> fast-import has subtle differences in how it parses file paths between each\n>> occurrence of in the grammar. Many errors are suppressed or not checked,\n>> which could lead to silent data corruption. A particularly bad case is when a\n>> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n>> supported), it would be treated as literal bytes instead of a quoted string.\n>> \n>> Bring path parsing into line with the documented behavior and improve\n>> documentation to fill in missing details.\n> \n> Thanks for the review, Patrick. I've made several changes, which I think address\n> your feedback. What's the protocol for adding `Reviewed-by`? Since I don't know\n> whether I, you, or Junio add it, I've refrained from attaching your name to\n> these patches.\n\nHello! Friendly ping here. It’s been a week since the last emails for this patch\nset, so I’d like to check in on the status.\n\nAs a new contributor to the project, I don’t yet have a full view of the\ncontribution flow, aside from what I’ve read. I suspect the latency is because I\nmay not have cc’d all the area experts. (When I sent v1, I used separate Cc\nlines in send-email --compose, but it dropped all but the last. After Patrick\nreviewed it, I figured I could leave the cc list as-is for v2, assuming I’d get\nanother review.) I’ve now cc’d everyone listed by contrib/contacts, as well as\nthe maintainer. For anyone not a part of the earlier discussion, the latest\nversion is at https://lore.kernel.org/git/cover.1711960552.git.thalia@archibald.dev/.\nIf that’s not the problem, I’d be keen to know what I could do better.\n\nI have several more patch sets in the works, that I’ve held back on sending\nuntil this one is finished, in case I’ve been doing something wrong. I want to\nmove forward. Thank you for your time.\n\nThalia\n\n> Changes since v1:\n> * In fast-import:\n> * Move `strbuf_release` outside of `parse_path_space` and `parse_path_eol`.\n> * Retain allocations for static `strbuf`s.\n> * Rename `allow_spaces` parameter of `parse_path` to `include_spaces`.\n> * Extract change to neighboring comments as patch 8.\n> * In tests:\n> * Test `` for the root path additionally in all tests using `\"\"`.\n> * Pass all arguments by positional variables.\n> * Use `local`.\n> * Use `test_when_finished` instead of manual cleanup.\n> * In documentation:\n> * Better document conditions under which a path is considered quoted or\n> unquoted.\n> * Reword commit messages.\n> \n> Thalia\n> \n> \n> Thalia Archibald (8):\n> fast-import: tighten path unquoting\n> fast-import: directly use strbufs for paths\n> fast-import: allow unquoted empty path for root\n> fast-import: remove dead strbuf\n> fast-import: improve documentation for unquoted paths\n> fast-import: document C-style escapes for paths\n> fast-import: forbid escaped NUL in paths\n> fast-import: make comments more precise\n> \n> Documentation/git-fast-import.txt | 30 +-\n> builtin/fast-import.c | 156 ++++----\n> t/t9300-fast-import.sh | 617 +++++++++++++++++++++---------\n> 3 files changed, 541 insertions(+), 262 deletions(-)\n> \n> Range-diff against v1:\n> 1: 8d9e0b25cb ! 1: e790bdf714 fast-import: tighten parsing of paths\n> @@ Metadata\n> Author: Thalia Archibald \n> \n> ## Commit message ##\n> - fast-import: tighten parsing of paths\n> + fast-import: tighten path unquoting\n> \n> Path parsing in fast-import is inconsistent and many unquoting errors\n> - are suppressed.\n> + are suppressed or not checked.\n> \n> - `` appears in the grammar in these places:\n> + appears in the grammar in these places:\n> \n> filemodify ::= 'M' SP ( | 'inline') SP LF\n> filedelete ::= 'D' SP LF\n> @@ Commit message\n> and fast-import.c parses them in five different ways:\n> \n> 1. For filemodify and filedelete:\n> - If `` is a valid quoted string, unquote it;\n> - otherwise, treat it as literal bytes (including any number of SP).\n> + Try to unquote . If it unquotes without errors, use the\n> + unquoted version; otherwise, treat it as literal bytes to the end of\n> + the line (including any number of SP).\n> 2. For filecopy (source) and filerename (source):\n> - If `` is a valid quoted string, unquote it;\n> - otherwise, treat it as literal bytes until the next SP.\n> + Try to unquote . If it unquotes without errors, use the\n> + unquoted version; otherwise, treat it as literal bytes up to, but not\n> + including, the next SP.\n> 3. For filecopy (dest) and filerename (dest):\n> - Like 1., but an unquoted empty string is an error.\n> + Like 1., but an unquoted empty string is forbidden.\n> 4. For ls:\n> - If `` starts with `\"`, unquote it and report parse errors;\n> - otherwise, treat it as literal bytes (including any number of SP).\n> + If starts with `\"`, unquote it and report parse errors;\n> + otherwise, treat it as literal bytes to the end of the line\n> + (including any number of SP).\n> 5. For ls-commit:\n> - Unquote `` and report parse errors.\n> + Unquote and report parse errors.\n> (It must start with `\"` to disambiguate from ls.)\n> \n> In the first three, any errors from trying to unquote a string are\n> suppressed, so a quoted string that contains invalid escapes would be\n> interpreted as literal bytes. For example, `\"\\xff\"` would fail to\n> unquote (because hex escapes are not supported), and it would instead be\n> - interpreted as the byte sequence `\"` `\\` `x` `f` `f` `\"`, which is\n> + interpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\n> certainly not intended. Some front-ends erroneously use their language's\n> - standard quoting routine and could silently introduce escapes that would\n> - be incorrectly parsed due to this.\n> + standard quoting routine instead of matching Git's, which could silently\n> + introduce escapes that would be incorrectly parsed due to this and lead\n> + to data corruption.\n> \n> - The documentation states that “To use a source path that contains SP the\n> - path must be quoted.”, so it is expected that some implementations\n> - depend on spaces being allowed in paths in the final position. Thus we\n> - have two documented ways to parse paths, so simplify the implementation\n> - to that.\n> + The documentation states “To use a source path that contains SP the path\n> + must be quoted.”, so it is expected that some implementations depend on\n> + spaces being allowed in paths in the final position. Thus we have two\n> + documented ways to parse paths, so simplify the implementation to that.\n> \n> Now we have:\n> \n> 1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n> filerename (dest), ls, and ls-commit:\n> \n> - If `` starts with `\"`, unquote it and report parse errors;\n> - otherwise, treat it as literal bytes (including any number of SP).\n> - Garbage after a quoted string or an unquoted empty string are errors.\n> - (In ls-commit, it must be quoted to disambiguate from ls.)\n> + If starts with `\"`, unquote it and report parse errors;\n> + otherwise, treat it as literal bytes to the end of the line\n> + (including any number of SP).\n> \n> 2. `parse_path_space` for filecopy (source) and filerename (source):\n> \n> - If `` starts with `\"`, unquote it and report parse errors;\n> - otherwise, treat it as literal bytes until the next SP.\n> - It must be followed by a SP. An unquoted empty string is an error.\n> + If starts with `\"`, unquote it and report parse errors;\n> + otherwise, treat it as literal bytes up to, but not including, the\n> + next SP. It must be followed by SP.\n> +\n> + There remain two special cases: The dest in filecopy and rename\n> + cannot be an unquoted empty string (this will be addressed subsequently)\n> + and in ls-commit must be quoted to disambiguate it from ls.\n> \n> Signed-off-by: Thalia Archibald \n> \n> - ## Documentation/git-fast-import.txt ##\n> -@@ Documentation/git-fast-import.txt: The value of `` must be in canonical form. That is it must not:\n> - * contain the special component `.` or `..` (e.g. `foo/./bar` and\n> - `foo/../bar` are invalid).\n> -\n> --The root of the tree can be represented by an empty string as ``.\n> -+The root of the tree can be represented by a quoted empty string (`\"\"`)\n> -+as ``.\n> -\n> - It is recommended that `` always be encoded using UTF-8.\n> -\n> -\n> ## builtin/fast-import.c ##\n> -@@ builtin/fast-import.c: static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n> - *\n> - * idnum ::= ':' bigint;\n> - *\n> -- * Return the first character after the value in *endptr.\n> -+ * Update *endptr to point to the first character after the value.\n> - *\n> - * Complain if the following character is not what is expected,\n> - * either a space or end of the string.\n> -@@ builtin/fast-import.c: static uintmax_t parse_mark_ref_eol(const char *p)\n> - }\n> -\n> - /*\n> -- * Parse the mark reference, demanding a trailing space. Return a\n> -- * pointer to the space.\n> -+ * Parse the mark reference, demanding a trailing space. Update *p to\n> -+ * point to the first character after the space.\n> - */\n> - static uintmax_t parse_mark_ref_space(const char **p)\n> - {\n> @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n> return mark;\n> }\n> @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n> +/*\n> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n> + * or unquoted without escape sequences. When unquoted, it may only contain a\n> -+ * space if `allow_spaces` is nonzero.\n> ++ * space if `include_spaces` is nonzero.\n> + */\n> -+static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n> ++static void parse_path(struct strbuf *sb, const char *p, const char **endp, int include_spaces, const char *field)\n> +{\n> -+ strbuf_reset(sb);\n> + if (*p == '\"') {\n> + if (unquote_c_style(sb, p, endp))\n> + die(\"Invalid %s: %s\", field, command_buf.buf);\n> + } else {\n> -+ if (allow_spaces)\n> ++ if (include_spaces)\n> + *endp = p + strlen(p);\n> + else\n> + *endp = strchr(p, ' ');\n> -+ if (*endp == p)\n> -+ die(\"Missing %s: %s\", field, command_buf.buf);\n> + strbuf_add(sb, p, *endp - p);\n> + }\n> +}\n> @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n> struct object_id oid;\n> uint16_t mode, inline_data = 0;\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b)\n> - die(\"Missing space after SHA1: %s\", command_buf.buf);\n> }\n> \n> -- strbuf_reset(&uq);\n> + strbuf_reset(&uq);\n> - if (!unquote_c_style(&uq, p, &endp)) {\n> - if (*endp)\n> - die(\"Garbage after path in: %s\", command_buf.buf);\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> static struct strbuf uq = STRBUF_INIT;\n> - const char *endp;\n> \n> -- strbuf_reset(&uq);\n> + strbuf_reset(&uq);\n> - if (!unquote_c_style(&uq, p, &endp)) {\n> - if (*endp)\n> - die(\"Garbage after path in: %s\", command_buf.buf);\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> - const char *endp;\n> struct tree_entry leaf;\n> \n> -- strbuf_reset(&s_uq);\n> + strbuf_reset(&s_uq);\n> - if (!unquote_c_style(&s_uq, s, &endp)) {\n> - if (*endp != ' ')\n> - die(\"Missing space after source: %s\", command_buf.buf);\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> - strbuf_add(&s_uq, s, endp - s);\n> - }\n> + parse_path_space(&s_uq, p, &p, \"source\");\n> -+ parse_path_eol(&d_uq, p, \"dest\");\n> s = s_uq.buf;\n> --\n> +\n> - endp++;\n> - if (!*endp)\n> -- die(\"Missing dest: %s\", command_buf.buf);\n> ++ if (!p)\n> + die(\"Missing dest: %s\", command_buf.buf);\n> -\n> - d = endp;\n> -- strbuf_reset(&d_uq);\n> + strbuf_reset(&d_uq);\n> - if (!unquote_c_style(&d_uq, d, &endp)) {\n> - if (*endp)\n> - die(\"Garbage after dest in: %s\", command_buf.buf);\n> - d = d_uq.buf;\n> - }\n> ++ parse_path_eol(&d_uq, p, \"dest\");\n> + d = d_uq.buf;\n> \n> memset(&leaf, 0, sizeof(leaf));\n> if (rename)\n> -@@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n> +@@ builtin/fast-import.c: static void print_ls(int mode, const unsigned char *hash, const char *path)\n> +\n> + static void parse_ls(const char *p, struct branch *b)\n> {\n> ++ static struct strbuf uq = STRBUF_INIT;\n> struct tree_entry *root = NULL;\n> struct tree_entry leaf = {NULL};\n> -+ static struct strbuf uq = STRBUF_INIT;\n> \n> - /* ls SP ( SP)? */\n> - if (*p == '\"') {\n> @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n> root->versions[1].mode = S_IFDIR;\n> load_tree(root);\n> @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n> - die(\"Garbage after path in: %s\", command_buf.buf);\n> - p = uq.buf;\n> - }\n> ++ strbuf_reset(&uq);\n> + parse_path_eol(&uq, p, \"path\");\n> + p = uq.buf;\n> tree_content_get(root, p, &leaf, 1);\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +# Path parsing\n> +#\n> +# There are two sorts of ways a path can be parsed, depending on whether it is\n> -+# the last field on the line. Additionally, ls without a has a\n> -+# special case. Test every occurrence of in the grammar against every\n> -+# error case.\n> ++# the last field on the line. Additionally, ls without a has a special\n> ++# case. Test every occurrence of in the grammar against every error case.\n> +#\n> +\n> +#\n> +# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n> +# filerename (dest), and ls.\n> +#\n> -+# commit :301 from root -- modify hello.c\n> ++# commit :301 from root -- modify hello.c (for setup)\n> +# commit :302 from :301 -- modify $path\n> +# commit :303 from :302 -- delete $path\n> +# commit :304 from :301 -- copy hello.c $path\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +# ls :305 $path\n> +#\n> +test_path_eol_success () {\n> -+ test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> ++ local test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> + test_expect_success \"S: paths at EOL with $test must work\" '\n> ++ test_when_finished \"git branch -D S-path-eol\" &&\n> ++\n> + git fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n> + blob\n> + mark :401\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + hallo welt\n> + BLOB\n> +\n> -+ commit refs/heads/path-eol\n> ++ commit refs/heads/S-path-eol\n> + mark :301\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + COMMIT\n> + M 100644 :401 hello.c\n> +\n> -+ commit refs/heads/path-eol\n> ++ commit refs/heads/S-path-eol\n> + mark :302\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + from :301\n> + M 100644 :402 '\"$path\"'\n> +\n> -+ commit refs/heads/path-eol\n> ++ commit refs/heads/S-path-eol\n> + mark :303\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + from :302\n> + D '\"$path\"'\n> +\n> -+ commit refs/heads/path-eol\n> ++ commit refs/heads/S-path-eol\n> + mark :304\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + from :301\n> + C hello.c '\"$path\"'\n> +\n> -+ commit refs/heads/path-eol\n> ++ commit refs/heads/S-path-eol\n> + mark :305\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + git ls-tree $commit_r >tree_r.out &&\n> + test_cmp tree_r.exp tree_r.out &&\n> +\n> -+ test_cmp out tree_r.exp &&\n> -+\n> -+ git branch -D path-eol\n> ++ test_cmp out tree_r.exp\n> + '\n> +}\n> +\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +#\n> +# Valid paths before a space: filecopy (source) and filerename (source).\n> +#\n> -+# commit :301 from root -- modify $path\n> ++# commit :301 from root -- modify $path (for setup)\n> +# commit :302 from :301 -- copy $path hello2.c\n> +# commit :303 from :301 -- rename $path hello2.c\n> +#\n> +test_path_space_success () {\n> -+ test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> ++ local test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> + test_expect_success \"S: paths before space with $test must work\" '\n> ++ test_when_finished \"git branch -D S-path-space\" &&\n> ++\n> + git fast-import --export-marks=marks.out <<-EOF 2>err &&\n> + blob\n> + mark :401\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + hello world\n> + BLOB\n> +\n> -+ commit refs/heads/path-space\n> ++ commit refs/heads/S-path-space\n> + mark :301\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + COMMIT\n> + M 100644 :401 '\"$path\"'\n> +\n> -+ commit refs/heads/path-space\n> ++ commit refs/heads/S-path-space\n> + mark :302\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> + from :301\n> + C '\"$path\"' hello2.c\n> +\n> -+ commit refs/heads/path-space\n> ++ commit refs/heads/S-path-space\n> + mark :303\n> + committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> + data < @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +\n> + printf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n> + git ls-tree $commit_r >tree_r.out &&\n> -+ test_cmp tree_r.exp tree_r.out &&\n> -+\n> -+ git branch -D path-space\n> ++ test_cmp tree_r.exp tree_r.out\n> + '\n> +}\n> +\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +# of in the grammar against all error kinds.\n> +#\n> +test_path_fail () {\n> -+ what=\"$1\" path=\"$2\" err_grep=\"$3\"\n> ++ local change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n> + test_expect_success \"S: $change with $what must fail\" '\n> + test_must_fail git fast-import <<-EOF 2>err &&\n> + blob\n> @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n> +}\n> +\n> +test_path_base_fail () {\n> -+ test_path_fail 'unclosed \" in '\"$field\" '\"hello.c' \"Invalid $field\"\n> -+ test_path_fail \"invalid escape in quoted $field\" '\"hello\\xff\"' \"Invalid $field\"\n> ++ local change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> ++ test_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n> ++ test_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n> +}\n> +test_path_eol_quoted_fail () {\n> -+ test_path_base_fail\n> -+ test_path_fail \"garbage after quoted $field\" '\"hello.c\"x' \"Garbage after $field\"\n> -+ test_path_fail \"space after quoted $field\" '\"hello.c\" ' \"Garbage after $field\"\n> ++ local change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> ++ test_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> ++ test_path_fail \"$change\" \"garbage after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Garbage after $field\"\n> ++ test_path_fail \"$change\" \"space after quoted $field\" \"$prefix\" '\"hello.c\" ' \"$suffix\" \"Garbage after $field\"\n> +}\n> +test_path_eol_fail () {\n> -+ test_path_eol_quoted_fail\n> -+ test_path_fail 'empty unquoted path' '' \"Missing $field\"\n> ++ local change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> ++ test_path_eol_quoted_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> +}\n> +test_path_space_fail () {\n> -+ test_path_base_fail\n> -+ test_path_fail 'empty unquoted path' '' \"Missing $field\"\n> -+ test_path_fail \"missing space after quoted $field\" '\"hello.c\"x' \"Missing space after $field\"\n> ++ local change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> ++ test_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> ++ test_path_fail \"$change\" \"missing space after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Missing space after $field\"\n> +}\n> +\n> -+change=filemodify prefix='M 100644 :1 ' field=path suffix='' test_path_eol_fail\n> -+change=filedelete prefix='D ' field=path suffix='' test_path_eol_fail\n> -+change=filecopy prefix='C ' field=source suffix=' world.c' test_path_space_fail\n> -+change=filecopy prefix='C hello.c ' field=dest suffix='' test_path_eol_fail\n> -+change=filerename prefix='R ' field=source suffix=' world.c' test_path_space_fail\n> -+change=filerename prefix='R hello.c ' field=dest suffix='' test_path_eol_fail\n> -+change='ls (in commit)' prefix='ls :2 ' field=path suffix='' test_path_eol_fail\n> ++test_path_eol_fail filemodify 'M 100644 :1 ' path ''\n> ++test_path_eol_fail filedelete 'D ' path ''\n> ++test_path_space_fail filecopy 'C ' source ' world.c'\n> ++test_path_eol_fail filecopy 'C hello.c ' dest ''\n> ++test_path_space_fail filerename 'R ' source ' world.c'\n> ++test_path_eol_fail filerename 'R hello.c ' dest ''\n> ++test_path_eol_fail 'ls (in commit)' 'ls :2 ' path ''\n> +\n> -+# When 'ls' has no , the must be quoted.\n> -+change='ls (without tree-ish in commit)' prefix='ls ' field=path suffix='' \\\n> -+test_path_eol_quoted_fail &&\n> -+test_path_fail 'empty unquoted path' '' \"Invalid dataref\"\n> ++# When 'ls' has no , the must be quoted.\n> ++test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n> +\n> ###\n> ### series T (ls)\n> 2: a2aca9f9e6 ! 2: 82a6f53c13 fast-import: directly use strbufs for paths\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> die(\"Missing space after SHA1: %s\", command_buf.buf);\n> }\n> \n> +- strbuf_reset(&uq);\n> - parse_path_eol(&uq, p, \"path\");\n> - p = uq.buf;\n> ++ strbuf_reset(&path);\n> + parse_path_eol(&path, p, \"path\");\n> \n> /* Git does not track empty, non-toplevel directories. */\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> - static struct strbuf uq = STRBUF_INIT;\n> + static struct strbuf path = STRBUF_INIT;\n> \n> +- strbuf_reset(&uq);\n> - parse_path_eol(&uq, p, \"path\");\n> - p = uq.buf;\n> - tree_content_remove(&b->branch_tree, p, NULL, 1);\n> ++ strbuf_reset(&path);\n> + parse_path_eol(&path, p, \"path\");\n> + tree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n> }\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> + static struct strbuf dest = STRBUF_INIT;\n> struct tree_entry leaf;\n> \n> +- strbuf_reset(&s_uq);\n> - parse_path_space(&s_uq, p, &p, \"source\");\n> -- parse_path_eol(&d_uq, p, \"dest\");\n> - s = s_uq.buf;\n> -- d = d_uq.buf;\n> ++ strbuf_reset(&source);\n> + parse_path_space(&source, p, &p, \"source\");\n> +\n> + if (!p)\n> + die(\"Missing dest: %s\", command_buf.buf);\n> +- strbuf_reset(&d_uq);\n> +- parse_path_eol(&d_uq, p, \"dest\");\n> +- d = d_uq.buf;\n> ++ strbuf_reset(&dest);\n> + parse_path_eol(&dest, p, \"dest\");\n> \n> memset(&leaf, 0, sizeof(leaf));\n> @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n> &leaf.versions[1].oid,\n> leaf.versions[1].mode,\n> leaf.tree);\n> -@@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n> +@@ builtin/fast-import.c: static void print_ls(int mode, const unsigned char *hash, const char *path)\n> +\n> + static void parse_ls(const char *p, struct branch *b)\n> {\n> - struct tree_entry *root = NULL;\n> - struct tree_entry leaf = {NULL};\n> - static struct strbuf uq = STRBUF_INIT;\n> + static struct strbuf path = STRBUF_INIT;\n> + struct tree_entry *root = NULL;\n> + struct tree_entry leaf = {NULL};\n> \n> - /* ls SP ( SP)? */\n> - if (*p == '\"') {\n> @@ builtin/fast-import.c: static void parse_ls(const char *p, struct branch *b)\n> root->versions[1].mode = S_IFDIR;\n> load_tree(root);\n> }\n> +- strbuf_reset(&uq);\n> - parse_path_eol(&uq, p, \"path\");\n> - p = uq.buf;\n> - tree_content_get(root, p, &leaf, 1);\n> ++ strbuf_reset(&path);\n> + parse_path_eol(&path, p, \"path\");\n> + tree_content_get(root, path.buf, &leaf, 1);\n> /*\n> 3: ecaf4883d1 < -: ---------- fast-import: release unfreed strbufs\n> -: ---------- > 3: 893bbf5e73 fast-import: allow unquoted empty path for root\n> 4: 058a38416a ! 4: cb05a184e6 fast-import: remove dead strbuf\n> @@ Metadata\n> ## Commit message ##\n> fast-import: remove dead strbuf\n> \n> - The strbuf in `note_change_n` has been unused since the function was\n> + The strbuf in `note_change_n` is to copy the remainder of `p` before\n> + potentially invalidating it when reading the next line. However, `p` is\n> + not used after that point. It has been unused since the function was\n> created in a8dd2e7d2b (fast-import: Add support for importing commit\n> notes, 2009-10-09) and looks to be a fossil from adapting\n> - `note_change_m`. Remove it.\n> + `file_change_m`. Remove it.\n> \n> Signed-off-by: Thalia Archibald \n> \n> 5: a5e8df0759 < -: ---------- fast-import: document C-style escapes for paths\n> 6: 9792940ba9 < -: ---------- fast-import: forbid escaped NUL in paths\n> -: ---------- > 5: 1f34b632d7 fast-import: improve documentation for unquoted paths\n> -: ---------- > 6: 82a4da68af fast-import: document C-style escapes for paths\n> -: ---------- > 7: c087c6a860 fast-import: forbid escaped NUL in paths\n> -: ---------- > 8: a503c55b83 fast-import: make comments more precise\n> --\n> 2.44.0\n\n\n"},{"id":"492481","messageId":"CAPig+cSu5dLoDew-efAB-H95B53fteDGNO=_1jc9i_MUqdpw-g@mail.gmail.com","threadId":"61174","inReplyTo":"A820E635-14FA-403C-B932-D7F9FE57C092@archibald.dev","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-07T23:46:52Z","receivedAt":"2024-04-07T23:47:04Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Apr 7, 2024 at 5:19 PM Thalia Archibald wrote:\n> On Apr 1, 2024, at 02:02, Thalia Archibald wrote:\n> >> fast-import has subtle differences in how it parses file paths between each\n> >> occurrence of in the grammar. Many errors are suppressed or not checked,\n> >> which could lead to silent data corruption. A particularly bad case is when a\n> >> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> >> supported), it would be treated as literal bytes instead of a quoted string.\n> >>\n> >> Bring path parsing into line with the documented behavior and improve\n> >> documentation to fill in missing details.\n> >\n> > Thanks for the review, Patrick. I've made several changes, which I think address\n> > your feedback. What's the protocol for adding `Reviewed-by`? Since I don't know\n> > whether I, you, or Junio add it, I've refrained from attaching your name to\n> > these patches.\n>\n> Hello! Friendly ping here. It’s been a week since the last emails for this patch\n> set, so I’d like to check in on the status.\n\nPinging is certainly the correct thing to do.\n\nRegarding `Reviewed-by:`: that trailer doesn't mean that someone\nmerely read and commented on a patch. Instead, it's explicitly _given_\nby a reviewer to indicate that the reviewer has thoroughly reviewed\nthe patch set and is confident that it is ready to be merged into the\nproject, at which point Junio typically adds the `Reviewed-by:`.\n\n> As a new contributor to the project, I don’t yet have a full view of the\n> contribution flow, aside from what I’ve read. I suspect the latency is because I\n> may not have cc’d all the area experts. (When I sent v1, I used separate Cc\n> lines in send-email --compose, but it dropped all but the last. After Patrick\n> reviewed it, I figured I could leave the cc list as-is for v2, assuming I’d get\n> another review.) I’ve now cc’d everyone listed by contrib/contacts, as well as\n> the maintainer. For anyone not a part of the earlier discussion, the latest\n> version is at https://lore.kernel.org/git/cover.1711960552.git.thalia@archibald.dev/.\n> If that’s not the problem, I’d be keen to know what I could do better.\n\nLack of response may be due to the series being overlooked, or it\ncould mean that nobody has any particular interest in the changes\n(which is not to say that there is anything wrong with the changes),\nor that people are simply busy elsewhere. It could also mean that this\nreroll is good enough and reviewers have nothing else to add. So,\ncc'ing potentially interested people makes sense.\n\n> I have several more patch sets in the works, that I’ve held back on sending\n> until this one is finished, in case I’ve been doing something wrong. I want to\n> move forward. Thank you for your time.\n\nIf the additional patch sets are unrelated to this patch set, then I\ndon't see a reason to hold them back. Feel free to send them. Even if\nthey are related to this patch set, you may still want to send them.\nAfter all, doing so may get the ball rolling again on this patch set.\n"},{"id":"492491","messageId":"ZhONyBIFlWbNHNwN@tanuki","threadId":"61174","inReplyTo":"CAPig+cSu5dLoDew-efAB-H95B53fteDGNO=_1jc9i_MUqdpw-g@mail.gmail.com","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:25:12Z","receivedAt":"2024-04-08T06:25:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Apr 07, 2024 at 07:46:52PM -0400, Eric Sunshine wrote:\n> On Sun, Apr 7, 2024 at 5:19 PM Thalia Archibald wrote:\n> > On Apr 1, 2024, at 02:02, Thalia Archibald wrote:\n> > >> fast-import has subtle differences in how it parses file paths between each\n> > >> occurrence of in the grammar. Many errors are suppressed or not checked,\n> > >> which could lead to silent data corruption. A particularly bad case is when a\n> > >> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> > >> supported), it would be treated as literal bytes instead of a quoted string.\n> > >>\n> > >> Bring path parsing into line with the documented behavior and improve\n> > >> documentation to fill in missing details.\n> > >\n> > > Thanks for the review, Patrick. I've made several changes, which I think address\n> > > your feedback. What's the protocol for adding `Reviewed-by`? Since I don't know\n> > > whether I, you, or Junio add it, I've refrained from attaching your name to\n> > > these patches.\n> >\n> > Hello! Friendly ping here. It’s been a week since the last emails for this patch\n> > set, so I’d like to check in on the status.\n> \n> Pinging is certainly the correct thing to do.\n> \n> Regarding `Reviewed-by:`: that trailer doesn't mean that someone\n> merely read and commented on a patch. Instead, it's explicitly _given_\n> by a reviewer to indicate that the reviewer has thoroughly reviewed\n> the patch set and is confident that it is ready to be merged into the\n> project, at which point Junio typically adds the `Reviewed-by:`.\n> \n> > As a new contributor to the project, I don’t yet have a full view of the\n> > contribution flow, aside from what I’ve read. I suspect the latency is because I\n> > may not have cc’d all the area experts. (When I sent v1, I used separate Cc\n> > lines in send-email --compose, but it dropped all but the last. After Patrick\n> > reviewed it, I figured I could leave the cc list as-is for v2, assuming I’d get\n> > another review.) I’ve now cc’d everyone listed by contrib/contacts, as well as\n> > the maintainer. For anyone not a part of the earlier discussion, the latest\n> > version is at https://lore.kernel.org/git/cover.1711960552.git.thalia@archibald.dev/.\n> > If that’s not the problem, I’d be keen to know what I could do better.\n> \n> Lack of response may be due to the series being overlooked, or it\n> could mean that nobody has any particular interest in the changes\n> (which is not to say that there is anything wrong with the changes),\n> or that people are simply busy elsewhere. It could also mean that this\n> reroll is good enough and reviewers have nothing else to add. So,\n> cc'ing potentially interested people makes sense.\n\nYeah, for this patch series I think it's mostly a lack of interest.\nWhich is too bad, because it does address some real issues with\ngit-fast-import(1). Part of the problem is also that this area does not\nreally have an area expert at all -- if you git-shortlog(1) for example\n\"builtin/fast-import.c\" for the last year you will see that it didn't\nget much attention at all.\n\nAnyway, I'm currently trying to make it a habit to pick up and review\nrandom patch series that didn't get any attention at all every once in a\nwhile, which is also why I reviewed your first version. I'm taking these\na bit slower though, also in the hope that some initial discussion may\nmotivate others to chime in, as well. Which may explain why I didn't yet\nreview your v2.\n\nIn any case I do have it in my backlog and will get to it somewhen this\nweek.\n\n> > I have several more patch sets in the works, that I’ve held back on sending\n> > until this one is finished, in case I’ve been doing something wrong. I want to\n> > move forward. Thank you for your time.\n> \n> If the additional patch sets are unrelated to this patch set, then I\n> don't see a reason to hold them back. Feel free to send them. Even if\n> they are related to this patch set, you may still want to send them.\n> After all, doing so may get the ball rolling again on this patch set.\n\nAgreed. Especially given that this is your first contribution, the\nquality of your patch series is quite high. So I don't see much of a\nreason to hold back the other patch series in case they are unrelated.\n\nPatrick\n"},{"id":"492507","messageId":"1B15EC5E-E0E7-474E-A0D8-3089918394A5@archibald.dev","threadId":"61174","inReplyTo":"ZhONyBIFlWbNHNwN@tanuki","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-08T07:15:35Z","receivedAt":"2024-04-08T07:15:51Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 7, 2024, at 23:25, Patrick Steinhardt wrote:\n> On Sun, Apr 07, 2024 at 07:46:52PM -0400, Eric Sunshine wrote:\n>> \n>> Lack of response may be due to the series being overlooked, or it\n>> could mean that nobody has any particular interest in the changes\n>> (which is not to say that there is anything wrong with the changes),\n>> or that people are simply busy elsewhere. It could also mean that this\n>> reroll is good enough and reviewers have nothing else to add. So,\n>> cc'ing potentially interested people makes sense.\n> \n> Yeah, for this patch series I think it's mostly a lack of interest.\n> Which is too bad, because it does address some real issues with\n> git-fast-import(1). Part of the problem is also that this area does not\n> really have an area expert at all -- if you git-shortlog(1) for example\n> \"builtin/fast-import.c\" for the last year you will see that it didn't\n> get much attention at all.\n\nUnfortunately, my upcoming patches will probably suffer the same fate, as\nthey're mostly fixing parsing issues in fast-import.\n\n> Anyway, I'm currently trying to make it a habit to pick up and review\n> random patch series that didn't get any attention at all every once in a\n> while, which is also why I reviewed your first version. I'm taking these\n> a bit slower though, also in the hope that some initial discussion may\n> motivate others to chime in, as well. Which may explain why I didn't yet\n> review your v2.\n> \n> In any case I do have it in my backlog and will get to it somewhen this\n> week.\n\nThank you!\n\n>>> I have several more patch sets in the works, that I’ve held back on sending\n>>> until this one is finished, in case I’ve been doing something wrong. I want to\n>>> move forward. Thank you for your time.\n>> \n>> If the additional patch sets are unrelated to this patch set, then I\n>> don't see a reason to hold them back. Feel free to send them. Even if\n>> they are related to this patch set, you may still want to send them.\n>> After all, doing so may get the ball rolling again on this patch set.\n> \n> Agreed. Especially given that this is your first contribution, the\n> quality of your patch series is quite high. So I don't see much of a\n> reason to hold back the other patch series in case they are unrelated.\n\nMy effort comes from reimplementing fast-import parsing as a Rust library,\nfollowing the implementation, not just the documentation, so I’ve noticed many\nmismatches between the concrete and abstract grammars. Perhaps it would save\nreviewer time to submit those around the same time, so knowledge of fast-import\nneeds to be evicted and loaded from cache less.\n\nThalia\n"},{"id":"492509","messageId":"ZhOztsTfuEjCqpD1@tanuki","threadId":"61174","inReplyTo":"1B15EC5E-E0E7-474E-A0D8-3089918394A5@archibald.dev","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T09:07:02Z","receivedAt":"2024-04-08T09:07:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 07:15:35AM +0000, Thalia Archibald wrote:\n> On Apr 7, 2024, at 23:25, Patrick Steinhardt wrote:\n> > On Sun, Apr 07, 2024 at 07:46:52PM -0400, Eric Sunshine wrote:\n[snip]\n> >>> I have several more patch sets in the works, that I’ve held back on sending\n> >>> until this one is finished, in case I’ve been doing something wrong. I want to\n> >>> move forward. Thank you for your time.\n> >> \n> >> If the additional patch sets are unrelated to this patch set, then I\n> >> don't see a reason to hold them back. Feel free to send them. Even if\n> >> they are related to this patch set, you may still want to send them.\n> >> After all, doing so may get the ball rolling again on this patch set.\n> > \n> > Agreed. Especially given that this is your first contribution, the\n> > quality of your patch series is quite high. So I don't see much of a\n> > reason to hold back the other patch series in case they are unrelated.\n> \n> My effort comes from reimplementing fast-import parsing as a Rust library,\n> following the implementation, not just the documentation, so I’ve noticed many\n> mismatches between the concrete and abstract grammars. Perhaps it would save\n> reviewer time to submit those around the same time, so knowledge of fast-import\n> needs to be evicted and loaded from cache less.\n\nIn this case I think it depends on whether or not these patch series\nwould conflict with each other. If they do it's preferable to land them\nsequentially. If they don't conflict then it should be fine to send\nseparate patch series and parallelize the review to a certain degree.\n\nPatrick\n"},{"id":"492539","messageId":"xmqqpluzgb4t.fsf@gitster.g","threadId":"61174","inReplyTo":"ZhONyBIFlWbNHNwN@tanuki","subject":"Re: [PATCH v2 0/8] fast-import: tighten parsing of paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T14:52:18Z","receivedAt":"2024-04-08T14:52:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> Regarding `Reviewed-by:`: that trailer doesn't mean that someone\n>> merely read and commented on a patch. Instead, it's explicitly _given_\n>> by a reviewer to indicate that the reviewer has thoroughly reviewed\n>> the patch set and is confident that it is ready to be merged into the\n>> project, at which point Junio typically adds the `Reviewed-by:`.\n\nJust to clarify, \"adds\" means \"cuts the reviewed-by lines written by\nthe reviewers in their review messages and pasts them into the\ncommit message, either amending the commits after the fact or while\napplying them\". I do _not_ judge from the sideline the quality of\nreviews given by others and say \"yeah, that is an adequate review,\nso I'll forge _their_ reviewed-by trailer\".\n\n> Anyway, I'm currently trying to make it a habit to pick up and review\n> random patch series that didn't get any attention at all every once in a\n> while, ...\n\nI have noticed your effort and appreciated it very much. I wish\nthere were many more others who do the same. I of course have been\nplaying that role for some time as the final catch-all reviewer, but\na single person does not scale.\n\nThanks.\n"},{"id":"492680","messageId":"ZhYxNYR33ftEfcPk@tanuki","threadId":"61174","inReplyTo":"e790bdf714bb4c2a67708c016a97cf4f9e79ac48.1711960552.git.thalia@archibald.dev","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-10T06:27:01Z","receivedAt":"2024-04-10T06:27:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 01, 2024 at 09:02:47AM +0000, Thalia Archibald wrote:\n> Path parsing in fast-import is inconsistent and many unquoting errors\n> are suppressed or not checked.\n> \n> appears in the grammar in these places:\n> \n> filemodify ::= 'M' SP ( | 'inline') SP LF\n> filedelete ::= 'D' SP LF\n> filecopy ::= 'C' SP SP LF\n> filerename ::= 'R' SP SP LF\n> ls ::= 'ls' SP SP LF\n> ls-commit ::= 'ls' SP LF\n> \n> and fast-import.c parses them in five different ways:\n> \n> 1. For filemodify and filedelete:\n> Try to unquote . If it unquotes without errors, use the\n> unquoted version; otherwise, treat it as literal bytes to the end of\n> the line (including any number of SP).\n> 2. For filecopy (source) and filerename (source):\n> Try to unquote . If it unquotes without errors, use the\n> unquoted version; otherwise, treat it as literal bytes up to, but not\n> including, the next SP.\n> 3. For filecopy (dest) and filerename (dest):\n> Like 1., but an unquoted empty string is forbidden.\n> 4. For ls:\n> If starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes to the end of the line\n> (including any number of SP).\n> 5. For ls-commit:\n> Unquote and report parse errors.\n> (It must start with `\"` to disambiguate from ls.)\n> \n> In the first three, any errors from trying to unquote a string are\n> suppressed, so a quoted string that contains invalid escapes would be\n> interpreted as literal bytes. For example, `\"\\xff\"` would fail to\n> unquote (because hex escapes are not supported), and it would instead be\n> interpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\n> certainly not intended. Some front-ends erroneously use their language's\n> standard quoting routine instead of matching Git's, which could silently\n> introduce escapes that would be incorrectly parsed due to this and lead\n> to data corruption.\n> \n> The documentation states “To use a source path that contains SP the path\n> must be quoted.”, so it is expected that some implementations depend on\n> spaces being allowed in paths in the final position. Thus we have two\n> documented ways to parse paths, so simplify the implementation to that.\n> \n> Now we have:\n> \n> 1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n> filerename (dest), ls, and ls-commit:\n> \n> If starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes to the end of the line\n> (including any number of SP).\n> \n> 2. `parse_path_space` for filecopy (source) and filerename (source):\n> \n> If starts with `\"`, unquote it and report parse errors;\n> otherwise, treat it as literal bytes up to, but not including, the\n> next SP. It must be followed by SP.\n> \n> There remain two special cases: The dest in filecopy and rename\n> cannot be an unquoted empty string (this will be addressed subsequently)\n> and in ls-commit must be quoted to disambiguate it from ls.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 102 ++++++++++-------\n> t/t9300-fast-import.sh | 251 ++++++++++++++++++++++++++++++++++++++++-\n> 2 files changed, 309 insertions(+), 44 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 782bda007c..6f9048a037 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2258,10 +2258,54 @@ static uintmax_t parse_mark_ref_space(const char **p)\n> \treturn mark;\n> }\n> \n> +/*\n> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n> + * or unquoted without escape sequences. When unquoted, it may only contain a\n> + * space if `include_spaces` is nonzero.\n> + */\n> +static void parse_path(struct strbuf *sb, const char *p, const char **endp, int include_spaces, const char *field)\n\nLet's break this overly long line, for example after `**endp,`.\n\n> +{\n> +\tif (*p == '\"') {\n> +\t\tif (unquote_c_style(sb, p, endp))\n> +\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n> +\t} else {\n> +\t\tif (include_spaces)\n> +\t\t\t*endp = p + strlen(p);\n> +\t\telse\n> +\t\t\t*endp = strchr(p, ' ');\n> +\t\tstrbuf_add(sb, p, *endp - p);\n\nstrchr(3P) may return a NULL pointer in case there is no space, which\nwould make us segfault here when dereferencing `*endp`. We should\nprobably add a testcase that would hit this edge case.\n\n> +\t}\n> +}\n> +\n> +/*\n> + * Parse the path string into the strbuf, and complain if this is not the end of\n> + * the string. It may contain spaces even when unquoted.\n> + */\n> +static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n> +{\n> +\tconst char *end;\n> +\n> +\tparse_path(sb, p, &end, 1, field);\n> +\tif (*end)\n> +\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n> +}\n> +\n> +/*\n> + * Parse the path string into the strbuf, and ensure it is followed by a space.\n> + * It may not contain spaces when unquoted. Update *endp to point to the first\n> + * character after the space.\n> + */\n> +static void parse_path_space(struct strbuf *sb, const char *p, const char **endp, const char *field)\n> +{\n> +\tparse_path(sb, p, endp, 0, field);\n> +\tif (**endp != ' ')\n> +\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n> +\t(*endp)++;\n> +}\n> +\n> static void file_change_m(const char *p, struct branch *b)\n> {\n> \tstatic struct strbuf uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \tstruct object_entry *oe;\n> \tstruct object_id oid;\n> \tuint16_t mode, inline_data = 0;\n> @@ -2299,11 +2343,8 @@ static void file_change_m(const char *p, struct branch *b)\n> \t}\n> \n> \tstrbuf_reset(&uq);\n> -\tif (!unquote_c_style(&uq, p, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n> \n> \t/* Git does not track empty, non-toplevel directories. */\n> \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n> @@ -2367,48 +2408,29 @@ static void file_change_m(const char *p, struct branch *b)\n> static void file_change_d(const char *p, struct branch *b)\n> {\n> \tstatic struct strbuf uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \n> \tstrbuf_reset(&uq);\n> -\tif (!unquote_c_style(&uq, p, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n> \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n> }\n> \n> -static void file_change_cr(const char *s, struct branch *b, int rename)\n> +static void file_change_cr(const char *p, struct branch *b, int rename)\n> {\n> -\tconst char *d;\n> +\tconst char *s, *d;\n> \tstatic struct strbuf s_uq = STRBUF_INIT;\n> \tstatic struct strbuf d_uq = STRBUF_INIT;\n> -\tconst char *endp;\n> \tstruct tree_entry leaf;\n> \n> \tstrbuf_reset(&s_uq);\n> -\tif (!unquote_c_style(&s_uq, s, &endp)) {\n> -\t\tif (*endp != ' ')\n> -\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n> -\t} else {\n> -\t\tendp = strchr(s, ' ');\n> -\t\tif (!endp)\n> -\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n> -\t\tstrbuf_add(&s_uq, s, endp - s);\n> -\t}\n> +\tparse_path_space(&s_uq, p, &p, \"source\");\n> \ts = s_uq.buf;\n> \n> -\tendp++;\n> -\tif (!*endp)\n> +\tif (!p)\n> \t\tdie(\"Missing dest: %s\", command_buf.buf);\n\nSo this statement right now doesn't make a whole lot of sense because\n`p` cannot ever be `NULL` -- we'd segfault before that. Once we update\n`parse_path()` to handle this correctly it will work as expected though.\n\nI was briefly wondering though whether we really want `parse_path()` to\nset `p` to be a NULL pointer. If we didn't, we could retain the previous\nbehaviour here and instead check for `!*p`.\n\nPatrick\n\n> -\td = endp;\n> \tstrbuf_reset(&d_uq);\n> -\tif (!unquote_c_style(&d_uq, d, &endp)) {\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n> -\t\td = d_uq.buf;\n> -\t}\n> +\tparse_path_eol(&d_uq, p, \"dest\");\n> +\td = d_uq.buf;\n> \n> \tmemset(&leaf, 0, sizeof(leaf));\n> \tif (rename)\n> @@ -3152,6 +3174,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n> \n> static void parse_ls(const char *p, struct branch *b)\n> {\n> +\tstatic struct strbuf uq = STRBUF_INIT;\n> \tstruct tree_entry *root = NULL;\n> \tstruct tree_entry leaf = {NULL};\n> \n> @@ -3168,16 +3191,9 @@ static void parse_ls(const char *p, struct branch *b)\n> \t\t\troot->versions[1].mode = S_IFDIR;\n> \t\tload_tree(root);\n> \t}\n> -\tif (*p == '\"') {\n> -\t\tstatic struct strbuf uq = STRBUF_INIT;\n> -\t\tconst char *endp;\n> -\t\tstrbuf_reset(&uq);\n> -\t\tif (unquote_c_style(&uq, p, &endp))\n> -\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n> -\t\tif (*endp)\n> -\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n> -\t\tp = uq.buf;\n> -\t}\n> +\tstrbuf_reset(&uq);\n> +\tparse_path_eol(&uq, p, \"path\");\n> +\tp = uq.buf;\n> \ttree_content_get(root, p, &leaf, 1);\n> \t/*\n> \t * A directory in preparation would have a sha1 of zero\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 60e30fed3c..0fb5612b07 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -2142,6 +2142,7 @@ test_expect_success 'Q: deny note on empty branch' '\n> \tEOF\n> \ttest_must_fail git fast-import '\n> +\n> ###\n> ### series R (feature and option)\n> ###\n> @@ -2790,7 +2791,7 @@ test_expect_success 'R: blob appears only once' '\n> '\n> \n> ###\n> -### series S\n> +### series S (mark and path parsing)\n> ###\n> #\n> # Make sure missing spaces and EOLs after mark references\n> @@ -3060,6 +3061,254 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n> \ttest_grep \"space after tree-ish\" err\n> '\n> \n> +#\n> +# Path parsing\n> +#\n> +# There are two sorts of ways a path can be parsed, depending on whether it is\n> +# the last field on the line. Additionally, ls without a has a special\n> +# case. Test every occurrence of in the grammar against every error case.\n> +#\n> +\n> +#\n> +# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n> +# filerename (dest), and ls.\n> +#\n> +# commit :301 from root -- modify hello.c (for setup)\n> +# commit :302 from :301 -- modify $path\n> +# commit :303 from :302 -- delete $path\n> +# commit :304 from :301 -- copy hello.c $path\n> +# commit :305 from :301 -- rename hello.c $path\n> +# ls :305 $path\n> +#\n> +test_path_eol_success () {\n> +\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> +\ttest_expect_success \"S: paths at EOL with $test must work\" '\n> +\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n> +\n> +\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n> +\t\tblob\n> +\t\tmark :401\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tblob\n> +\t\tmark :402\n> +\t\tdata < +\t\thallo welt\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/S-path-eol\n> +\t\tmark :301\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tinitial commit\n> +\t\tCOMMIT\n> +\t\tM 100644 :401 hello.c\n> +\n> +\t\tcommit refs/heads/S-path-eol\n> +\t\tmark :302\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filemodify\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tM 100644 :402 '\"$path\"'\n> +\n> +\t\tcommit refs/heads/S-path-eol\n> +\t\tmark :303\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filedelete\n> +\t\tCOMMIT\n> +\t\tfrom :302\n> +\t\tD '\"$path\"'\n> +\n> +\t\tcommit refs/heads/S-path-eol\n> +\t\tmark :304\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filecopy dest\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tC hello.c '\"$path\"'\n> +\n> +\t\tcommit refs/heads/S-path-eol\n> +\t\tmark :305\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filerename dest\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tR hello.c '\"$path\"'\n> +\n> +\t\tls :305 '\"$path\"'\n> +\t\tEOF\n> +\n> +\t\tcommit_m=$(grep :302 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_d=$(grep :303 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_c=$(grep :304 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_r=$(grep :305 marks.out | cut -d\\ -f2) &&\n> +\t\tblob1=$(grep :401 marks.out | cut -d\\ -f2) &&\n> +\t\tblob2=$(grep :402 marks.out | cut -d\\ -f2) &&\n> +\n> +\t\t( printf \"100644 blob $blob2\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_m.exp &&\n\nI think it is more customary to format as follows:\n\n\t(\n\t\tprintf \"100644 blob $blob2\\t'\"$unquoted_path\"'\\n\" &&\n\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n\t) | sort >tree_m.exp &&\n\nSame for other statements further down.\n\nAlso, there is no need to do `'\"$unuoted_path\"'` here. You should be\nable to refer to `$unquoted_path` just fine even without unquoting again\nbecause we use eval to execute the code block. In fact, it can even be\nharmful as it is known to break shells under some circumstances. See\nalso 7c4449eb31 (t/README: document how to loop around test cases,\n2024-03-22), which I think should apply in your case, too.\n\nPatrick\n\n> +\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n> +\t\ttest_cmp tree_m.exp tree_m.out &&\n> +\n> +\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n> +\t\tgit ls-tree $commit_d >tree_d.out &&\n> +\t\ttest_cmp tree_d.exp tree_d.out &&\n> +\n> +\t\t( printf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_c.exp &&\n> +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n> +\t\ttest_cmp tree_c.exp tree_c.out &&\n> +\n> +\t\tprintf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" >tree_r.exp &&\n> +\t\tgit ls-tree $commit_r >tree_r.out &&\n> +\t\ttest_cmp tree_r.exp tree_r.out &&\n> +\n> +\t\ttest_cmp out tree_r.exp\n> +\t'\n> +}\n> +\n> +test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n> +\n> +#\n> +# Valid paths before a space: filecopy (source) and filerename (source).\n> +#\n> +# commit :301 from root -- modify $path (for setup)\n> +# commit :302 from :301 -- copy $path hello2.c\n> +# commit :303 from :301 -- rename $path hello2.c\n> +#\n> +test_path_space_success () {\n> +\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n> +\ttest_expect_success \"S: paths before space with $test must work\" '\n> +\t\ttest_when_finished \"git branch -D S-path-space\" &&\n> +\n> +\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n> +\t\tblob\n> +\t\tmark :401\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/S-path-space\n> +\t\tmark :301\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tinitial commit\n> +\t\tCOMMIT\n> +\t\tM 100644 :401 '\"$path\"'\n> +\n> +\t\tcommit refs/heads/S-path-space\n> +\t\tmark :302\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filecopy source\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tC '\"$path\"' hello2.c\n> +\n> +\t\tcommit refs/heads/S-path-space\n> +\t\tmark :303\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit filerename source\n> +\t\tCOMMIT\n> +\t\tfrom :301\n> +\t\tR '\"$path\"' hello2.c\n> +\n> +\t\tEOF\n> +\n> +\t\tcommit_c=$(grep :302 marks.out | cut -d\\ -f2) &&\n> +\t\tcommit_r=$(grep :303 marks.out | cut -d\\ -f2) &&\n> +\t\tblob=$(grep :401 marks.out | cut -d\\ -f2) &&\n> +\n> +\t\t( printf \"100644 blob $blob\\t'\"$unquoted_path\"'\\n\" &&\n> +\t\t printf \"100644 blob $blob\\thello2.c\\n\" ) | sort >tree_c.exp &&\n> +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n> +\t\ttest_cmp tree_c.exp tree_c.out &&\n> +\n> +\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n> +\t\tgit ls-tree $commit_r >tree_r.out &&\n> +\t\ttest_cmp tree_r.exp tree_r.out\n> +\t'\n> +}\n> +\n> +test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n> +\n> +#\n> +# Test a single commit change with an invalid path. Run it with all occurrences\n> +# of in the grammar against all error kinds.\n> +#\n> +test_path_fail () {\n> +\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n> +\ttest_expect_success \"S: $change with $what must fail\" '\n> +\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n> +\t\tblob\n> +\t\tmark :1\n> +\t\tdata < +\t\thello world\n> +\t\tBLOB\n> +\n> +\t\tcommit refs/heads/S-path-fail\n> +\t\tmark :2\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit setup\n> +\t\tCOMMIT\n> +\t\tM 100644 :1 hello.c\n> +\n> +\t\tcommit refs/heads/S-path-fail\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcommit with bad path\n> +\t\tCOMMIT\n> +\t\tfrom :2\n> +\t\t'\"$prefix$path$suffix\"'\n> +\t\tEOF\n> +\n> +\t\ttest_grep '\"'$err_grep'\"' err\n> +\t'\n> +}\n> +\n> +test_path_base_fail () {\n> +\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> +\ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n> +\ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n> +}\n> +test_path_eol_quoted_fail () {\n> +\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> +\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> +\ttest_path_fail \"$change\" \"garbage after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Garbage after $field\"\n> +\ttest_path_fail \"$change\" \"space after quoted $field\" \"$prefix\" '\"hello.c\" ' \"$suffix\" \"Garbage after $field\"\n> +}\n> +test_path_eol_fail () {\n> +\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> +\ttest_path_eol_quoted_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> +}\n> +test_path_space_fail () {\n> +\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> +\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n> +\ttest_path_fail \"$change\" \"missing space after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Missing space after $field\"\n> +}\n> +\n> +test_path_eol_fail filemodify 'M 100644 :1 ' path ''\n> +test_path_eol_fail filedelete 'D ' path ''\n> +test_path_space_fail filecopy 'C ' source ' world.c'\n> +test_path_eol_fail filecopy 'C hello.c ' dest ''\n> +test_path_space_fail filerename 'R ' source ' world.c'\n> +test_path_eol_fail filerename 'R hello.c ' dest ''\n> +test_path_eol_fail 'ls (in commit)' 'ls :2 ' path ''\n> +\n> +# When 'ls' has no , the must be quoted.\n> +test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n> +\n> ###\n> ### series T (ls)\n> ###\n> -- \n> 2.44.0\n> \n"},{"id":"492681","messageId":"ZhYxP42Br2h1mq5r@tanuki","threadId":"61174","inReplyTo":"82a6f53c1326a420348eb70461f5929340a930d3.1711960552.git.thalia@archibald.dev","subject":"Re: [PATCH v2 2/8] fast-import: directly use strbufs for paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-10T06:27:11Z","receivedAt":"2024-04-10T06:27:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 01, 2024 at 09:03:06AM +0000, Thalia Archibald wrote:\n> Previously, one case would not write the path to the strbuf: when the\n> path is unquoted and at the end of the string. It was essentially\n> copy-on-write. However, with the logic simplification of the previous\n> commit, this case was eliminated and the strbuf is always populated.\n> \n> Directly use the strbufs now instead of an alias.\n> \n> Since this already changes all the lines that use the strbufs, rename\n> them from `uq` to be more descriptive. That they are unquoted is not\n> their most important property, so name them after what they carry.\n> \n> Additionally, `file_change_m` no longer needs to copy the path before\n> reading inline data.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 64 ++++++++++++++++++-------------------------\n> 1 file changed, 27 insertions(+), 37 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 6f9048a037..fad9324e59 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2305,7 +2305,\n> 7 @@ static void parse_path_space(struct strbuf *sb, const char *p, const char **endp\n> \n> static void file_change_m(const char *p, struct branch *b)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> \tstruct object_entry *oe;\n> \tstruct object_id oid;\n> \tuint16_t mode, inline_data = 0;\n> @@ -2342,13 +2342,12 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n> \t}\n> \n> -\tstrbuf_reset(&uq);\n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> +\tstrbuf_reset(&path);\n> +\tparse_path_eol(&path, p, \"path\");\n> \n> \t/* Git does not track empty, non-toplevel directories. */\n> -\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n> -\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n> +\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n> +\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n> \t\treturn;\n> \t}\n> \n> @@ -2369,10 +2368,6 @@ static void file_change_m(const char *p, str\n> uct branch *b)\n> \t\tif (S_ISDIR(mode))\n> \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n> \t\t\t\tcommand_buf.buf);\n> -\t\tif (p != uq.buf) {\n> -\t\t\tstrbuf_addstr(&uq, p);\n> -\t\t\tp = uq.buf;\n> -\t\t}\n> \t\twhile (read_next_command() != EOF) {\n> \t\t\tconst char *v;\n> \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n> @@ -2398,55 +2393,51 @@ static void file_change_m(const char *p, struct branch *b)\n> \t\t\t\tcommand_buf.buf);\n> \t}\n> \n> -\tif (!*p) {\n> +\tif (!*path.buf) {\n> \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n> \t\treturn;\n> \t}\n> -\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n> +\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n> }\n> \n> static void file_change_d(const char *p, struct branch *b)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> \n> -\tstrbuf_reset(&uq);\n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> -\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n> +\tstrbuf_reset(&path);\n> +\tparse_path_eol(&path, p\n> , \"path\");\n\nThis looks weird. Did you manually edit the patch or is there some weird\ncharacter in here that breaks diff generation?\n\n> +\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n> }\n> \n> static void file_change_cr(const char *p, struct branch *b, int rename)\n> {\n> -\tconst char *s, *d;\n> -\tstatic struct strbuf s_uq = STRBUF_INIT;\n> -\tstatic struct strbuf d_uq = STRBUF_INIT;\n> +\tstatic struct strbuf source = STRBUF_INIT;\n> +\tstatic struct strbuf dest = STRBUF_INIT;\n> \tstruct tree_entry leaf;\n> \n> -\tstrbuf_reset(&s_uq);\n> -\tparse_path_space(&s_uq, p, &p, \"source\");\n> -\ts = s_uq.buf;\n> +\tstrbuf_reset(&source);\n> +\tparse_path_space(&source, p, &p, \"source\");\n> \n> \tif (!p)\n> \t\tdie(\"Missing dest: %s\", command_buf.buf);\n> -\tstrbuf_reset(&d_uq);\n> -\tparse_path_eol(&d_uq, p, \"dest\");\n> -\td = d_uq.buf;\n> +\tstrbuf_reset(&dest);\n> +\tparse_path_eol(&dest, p, \"dest\");\n> \n> \tmemset(&leaf, 0, sizeof(leaf));\n> \tif (rename)\n> -\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n> +\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n> \telse\n> -\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n> +\t\ttree_content_get(&b-\n> >branch_tree, source.buf, &leaf, 1);\n\nSame here. Is your mail agent maybe wrapping lines?\n\n> \tif (!leaf.versions[1].mode)\n> -\t\tdie(\"Path %s not in branch\", s);\n> -\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n> +\t\tdie(\"Path %s not in branch\", source.buf);\n> +\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n> \t\ttree_content_replace(&b->branch_tree,\n> \t\t\t&leaf.versions[1].oid,\n> \t\t\tleaf.versions[1].mode,\n> \t\t\tleaf.tree);\n> \t\treturn;\n> \t}\n> -\ttree_content_set(&b->branch_tree, d,\n> +\ttree_content_set(&b->branch_tree, dest.buf,\n> \t\t&leaf.versions[1].oid,\n> \t\tleaf.versions[1].mode,\n> \t\tleaf.tree);\n> @@ -3174,7 +3165,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n> \n> static void parse_ls(const char *p, struct branch *b)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> +\tstatic struct strbuf path = STRBUF_INIT;\n> \tstruct tree_entry *root = NULL;\n> \tstruct tree_entry leaf = {NULL};\n> \n> @@ -3191,10 +3182,9 @@ static void parse_ls(const char *p, struct branch *b)\n> \t\t\troot->versions[1].mode = S_IFDIR;\n> \t\tload_tree(root);\n> \t}\n> -\ts\n> trbuf_reset(&uq);\n\nAnd here.\n\nOther than those formatting issues this patch looks fine to me.\n\nPatrick\n\n> -\tparse_path_eol(&uq, p, \"path\");\n> -\tp = uq.buf;\n> -\ttree_content_get(root, p, &leaf, 1);\n> +\tstrbuf_reset(&path);\n> +\tparse_path_eol(&path, p, \"path\");\n> +\ttree_content_get(root, path.buf, &leaf, 1);\n> \t/*\n> \t * A directory in preparation would have a sha1 of zero\n> \t * until it is saved. Save, for simplicity.\n> @@ -3202,7 +3192,7 @@ static void parse_ls(const char *p, struct branch *b)\n> \tif (S_ISDIR(leaf.versions[1].mode))\n> \t\tstore_tree(&leaf);\n> \n> -\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n> +\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n> \tif (leaf.tree)\n> \t\trelease_tree_content_recursive(leaf.tree);\n> \tif (!b || root != &b->branch_tree)\n> -- \n> 2.44.0\n> \n"},{"id":"492682","messageId":"ZhYxSSdqUFx821Hh@tanuki","threadId":"61174","inReplyTo":"893bbf5e734903382c22893121252ed49b3c7a69.1711960552.git.thalia@archibald.dev","subject":"Re: [PATCH v2 3/8] fast-import: allow unquoted empty path for root","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-10T06:27:21Z","receivedAt":"2024-04-10T06:27:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 01, 2024 at 09:03:17AM +0000, Thalia Archibald wrote:\n> Ever since filerename was added in f39a946a1f (Support wholesale\n> directory renames in fast-import, 2007-07-09) and filecopy in b6f3481bb4\n> (Teach fast-import to recursively copy files/directories, 2007-07-15),\n> both have produced an error when the destination path is empty. Later,\n> when support for targeting the root directory with an empty string was\n> added in 2794ad5244 (fast-import: Allow filemodify to set the root,\n> 2010-10-10), this had the effect of allowing the quoted empty string\n> (`\"\"`), but forbidding its unquoted variant (``). This seems to have\n> been intended as simple data validation for parsing two paths, rather\n> than a syntax restriction, because it was not extended to the other\n> operations.\n> \n> All other occurrences of paths (in filemodify, filedelete, the source of\n> filecopy and filerename, and ls) allow both.\n> \n> For most of this feature's lifetime, the documentation has not\n> prescribed the use of quoted empty strings. In e5959106d6\n> (Documentation/fast-import: put explanation of M 040000 \"\" in\n> context, 2011-01-15), its documentation was changed from “`` may\n> also be an empty string (`\"\"`) to specify the root of the tree” to “The\n> root of the tree can be represented by an empty string as ``”.\n> \n> Thus, we can assume that some front-ends have depended on this behavior.\n> \n> Remove this restriction for the destination paths of filecopy and\n> filerename and change tests targeting the root to test `\"\"` and ``.\n> \n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 5 +-\n> t/t9300-fast-import.sh | 363 +++++++++++++++++++++--------------------\n> 2 files changed, 191 insertions(+), 177 deletions(-)\n> \n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index fad9324e59..58cc8d4ede 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2416,11 +2416,8 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n> \tstruct tree_entry leaf;\n> \n> \tstrbuf_reset(&source);\n> -\tparse_path_space(&source, p, &p, \"source\");\n\nNit: the diff would be a bit easier to read if you retained the sequence\nof `strbuf_reset()` and `parse_path_space()`.\n\n> -\n> -\tif (!p)\n> -\t\tdie(\"Missing dest: %s\", command_buf.buf);\n\n> \tstrbuf_reset(&dest);\n\nI also wonder why this actually makes a difference. As mentioned in a\npreceding mail, `if (!p)` cannot really do anything because the only\ncase where `p` could be a `NULL` pointer is when strchr(3P) did not\nfound a subsequent space in `parse_path()`. And in that case we would\nhave segfaulted because we do dereference `p` afterwards.\n\n> +\tparse_path_space(&source, p, &p, \"source\");\n> \tparse_path_eol(&dest, p, \"dest\");\n> \n> \tmemset(&leaf, 0, sizeof(leaf));\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 0fb5612b07..635b1b9af7 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -1059,30 +1059,33 @@ test_expect_success 'M: rename subdirectory to new subdirectory' '\n> \tcompare_diff_raw expect actual\n> '\n> \n> -test_expect_success 'M: rename root to subdirectory' '\n> -\tcat >input <<-INPUT_END &&\n> -\tcommit refs/heads/M4\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\trename root\n> -\tCOMMIT\n> +for root in '\"\"' ''\n> +do\n> +\ttest_expect_success \"M: rename root ($root) to subdirectory\" '\n> +\t\tcat >input <<-INPUT_END &&\n> +\t\tcommit refs/heads/M4\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\trename root\n> +\t\tCOMMIT\n> \n> -\tfrom refs/heads/M2^0\n> -\tR \"\" sub\n> +\t\tfrom refs/heads/M2^0\n> +\t\tR '\"$root\"' sub\n\nSame comment here, we should not do the `'\"$root\"'` dance but can\ninstead just refer to the variable directly in the quoted block.\n\nPatrick\n\n> -\tINPUT_END\n> +\t\tINPUT_END\n> \n> -\tcat >expect <<-EOF &&\n> -\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n> -\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n> -\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n> -\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n> -\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n> -\tEOF\n> -\tgit fast-import -\tgit diff-tree -M -r M4^ M4 >actual &&\n> -\tcompare_diff_raw expect actual\n> -'\n> +\t\tcat >expect <<-EOF &&\n> +\t\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n> +\t\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n> +\t\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n> +\t\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n> +\t\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n> +\t\tEOF\n> +\t\tgit fast-import +\t\tgit diff-tree -M -r M4^ M4 >actual &&\n> +\t\tcompare_diff_raw expect actual\n> +\t'\n> +done\n> \n> ###\n> ### series N\n> @@ -1259,49 +1262,52 @@ test_expect_success PIPE 'N: empty directory reads as missing' '\n> \ttest_cmp expect actual\n> '\n> \n> -test_expect_success 'N: copy root directory by tree hash' '\n> -\tcat >expect <<-EOF &&\n> -\t:100755 000000 $newf $zero D\tfile3/newf\n> -\t:100644 000000 $oldf $zero D\tfile3/oldf\n> -\tEOF\n> -\troot=$(git rev-parse refs/heads/branch^0^{tree}) &&\n> -\tcat >input <<-INPUT_END &&\n> -\tcommit refs/heads/N6\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\tcopy root directory by tree hash\n> -\tCOMMIT\n> +for root in '\"\"' ''\n> +do\n> +\ttest_expect_success \"N: copy root ($root) by tree hash\" '\n> +\t\tcat >expect <<-EOF &&\n> +\t\t:100755 000000 $newf $zero D\tfile3/newf\n> +\t\t:100644 000000 $oldf $zero D\tfile3/oldf\n> +\t\tEOF\n> +\t\troot_tree=$(git rev-parse refs/heads/branch^0^{tree}) &&\n> +\t\tcat >input <<-INPUT_END &&\n> +\t\tcommit refs/heads/N6\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcopy root directory by tree hash\n> +\t\tCOMMIT\n> \n> -\tfrom refs/heads/branch^0\n> -\tM 040000 $root \"\"\n> -\tINPUT_END\n> -\tgit fast-import -\tgit diff-tree -C --find-copies-harder -r N4 N6 >actual &&\n> -\tcompare_diff_raw expect actual\n> -'\n> +\t\tfrom refs/heads/branch^0\n> +\t\tM 040000 $root_tree '\"$root\"'\n> +\t\tINPUT_END\n> +\t\tgit fast-import +\t\tgit diff-tree -C --find-copies-harder -r N4 N6 >actual &&\n> +\t\tcompare_diff_raw expect actual\n> +\t'\n> \n> -test_expect_success 'N: copy root by path' '\n> -\tcat >expect <<-EOF &&\n> -\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n> -\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n> -\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n> -\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n> -\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n> -\tEOF\n> -\tcat >input <<-INPUT_END &&\n> -\tcommit refs/heads/N-copy-root-path\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\tcopy root directory by (empty) path\n> -\tCOMMIT\n> +\ttest_expect_success \"N: copy root ($root) by path\" '\n> +\t\tcat >expect <<-EOF &&\n> +\t\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n> +\t\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n> +\t\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n> +\t\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n> +\t\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n> +\t\tEOF\n> +\t\tcat >input <<-INPUT_END &&\n> +\t\tcommit refs/heads/N-copy-root-path\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcopy root directory by (empty) path\n> +\t\tCOMMIT\n> \n> -\tfrom refs/heads/branch^0\n> -\tC \"\" oldroot\n> -\tINPUT_END\n> -\tgit fast-import -\tgit diff-tree -C --find-copies-harder -r branch N-copy-root-path >actual &&\n> -\tcompare_diff_raw expect actual\n> -'\n> +\t\tfrom refs/heads/branch^0\n> +\t\tC '\"$root\"' oldroot\n> +\t\tINPUT_END\n> +\t\tgit fast-import +\t\tgit diff-tree -C --find-copies-harder -r branch N-copy-root-path >actual &&\n> +\t\tcompare_diff_raw expect actual\n> +\t'\n> +done\n> \n> test_expect_success 'N: delete directory by copying' '\n> \tcat >expect <<-\\EOF &&\n> @@ -1431,98 +1437,102 @@ test_expect_success 'N: reject foo/ syntax in ls argument' '\n> \tINPUT_END\n> '\n> \n> -test_expect_success 'N: copy to root by id and modify' '\n> -\techo \"hello, world\" >expect.foo &&\n> -\techo hello >expect.bar &&\n> -\tgit fast-import <<-SETUP_END &&\n> -\tcommit refs/heads/N7\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\thello, tree\n> -\tCOMMIT\n> +for root in '\"\"' ''\n> +do\n> +\ttest_expect_success \"N: copy to root ($root) by id and modify\" '\n> +\t\techo \"hello, world\" >expect.foo &&\n> +\t\techo hello >expect.bar &&\n> +\t\tgit fast-import <<-SETUP_END &&\n> +\t\tcommit refs/heads/N7\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\thello, tree\n> +\t\tCOMMIT\n> \n> -\tdeleteall\n> -\tM 644 inline foo/bar\n> -\tdata < -\thello\n> -\tEOF\n> -\tSETUP_END\n> +\t\tdeleteall\n> +\t\tM 644 inline foo/bar\n> +\t\tdata < +\t\thello\n> +\t\tEOF\n> +\t\tSETUP_END\n> \n> -\ttree=$(git rev-parse --verify N7:) &&\n> -\tgit fast-import <<-INPUT_END &&\n> -\tcommit refs/heads/N8\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\tcopy to root by id and modify\n> -\tCOMMIT\n> +\t\ttree=$(git rev-parse --verify N7:) &&\n> +\t\tgit fast-import <<-INPUT_END &&\n> +\t\tcommit refs/heads/N8\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcopy to root by id and modify\n> +\t\tCOMMIT\n> \n> -\tM 040000 $tree \"\"\n> -\tM 644 inline foo/foo\n> -\tdata < -\thello, world\n> -\tEOF\n> -\tINPUT_END\n> -\tgit show N8:foo/foo >actual.foo &&\n> -\tgit show N8:foo/bar >actual.bar &&\n> -\ttest_cmp expect.foo actual.foo &&\n> -\ttest_cmp expect.bar actual.bar\n> -'\n> +\t\tM 040000 $tree '\"$root\"'\n> +\t\tM 644 inline foo/foo\n> +\t\tdata < +\t\thello, world\n> +\t\tEOF\n> +\t\tINPUT_END\n> +\t\tgit show N8:foo/foo >actual.foo &&\n> +\t\tgit show N8:foo/bar >actual.bar &&\n> +\t\ttest_cmp expect.foo actual.foo &&\n> +\t\ttest_cmp expect.bar actual.bar\n> +\t'\n> \n> -test_expect_success 'N: extract subtree' '\n> -\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n> -\tcat >input <<-INPUT_END &&\n> -\tcommit refs/heads/N9\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\textract subtree branch:newdir\n> -\tCOMMIT\n> +\ttest_expect_success \"N: extract subtree to the root ($root)\" '\n> +\t\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n> +\t\tcat >input <<-INPUT_END &&\n> +\t\tcommit refs/heads/N9\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\textract subtree branch:newdir\n> +\t\tCOMMIT\n> \n> -\tM 040000 $branch \"\"\n> -\tC \"newdir\" \"\"\n> -\tINPUT_END\n> -\tgit fast-import -\tgit diff --exit-code branch:newdir N9\n> -'\n> +\t\tM 040000 $branch '\"$root\"'\n> +\t\tC \"newdir\" '\"$root\"'\n> +\t\tINPUT_END\n> +\t\tgit fast-import +\t\tgit diff --exit-code branch:newdir N9\n> +\t'\n> \n> -test_expect_success 'N: modify subtree, extract it, and modify again' '\n> -\techo hello >expect.baz &&\n> -\techo hello, world >expect.qux &&\n> -\tgit fast-import <<-SETUP_END &&\n> -\tcommit refs/heads/N10\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\thello, tree\n> -\tCOMMIT\n> +\ttest_expect_success \"N: modify subtree, extract it to the root ($root), and modify again\" '\n> +\t\techo hello >expect.baz &&\n> +\t\techo hello, world >expect.qux &&\n> +\t\tgit fast-import <<-SETUP_END &&\n> +\t\tcommit refs/heads/N10\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\thello, tree\n> +\t\tCOMMIT\n> \n> -\tdeleteall\n> -\tM 644 inline foo/bar/baz\n> -\tdata < -\thello\n> -\tEOF\n> -\tSETUP_END\n> +\t\tdeleteall\n> +\t\tM 644 inline foo/bar/baz\n> +\t\tdata < +\t\thello\n> +\t\tEOF\n> +\t\tSETUP_END\n> \n> -\ttree=$(git rev-parse --verify N10:) &&\n> -\tgit fast-import <<-INPUT_END &&\n> -\tcommit refs/heads/N11\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\tcopy to root by id and modify\n> -\tCOMMIT\n> +\t\ttree=$(git rev-parse --verify N10:) &&\n> +\t\tgit fast-import <<-INPUT_END &&\n> +\t\tcommit refs/heads/N11\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tcopy to root by id and modify\n> +\t\tCOMMIT\n> \n> -\tM 040000 $tree \"\"\n> -\tM 100644 inline foo/bar/qux\n> -\tdata < -\thello, world\n> -\tEOF\n> -\tR \"foo\" \"\"\n> -\tC \"bar/qux\" \"bar/quux\"\n> -\tINPUT_END\n> -\tgit show N11:bar/baz >actual.baz &&\n> -\tgit show N11:bar/qux >actual.qux &&\n> -\tgit show N11:bar/quux >actual.quux &&\n> -\ttest_cmp expect.baz actual.baz &&\n> -\ttest_cmp expect.qux actual.qux &&\n> -\ttest_cmp expect.qux actual.quux'\n> +\t\tM 040000 $tree '\"$root\"'\n> +\t\tM 100644 inline foo/bar/qux\n> +\t\tdata < +\t\thello, world\n> +\t\tEOF\n> +\t\tR \"foo\" '\"$root\"'\n> +\t\tC \"bar/qux\" \"bar/quux\"\n> +\t\tINPUT_END\n> +\t\tgit show N11:bar/baz >actual.baz &&\n> +\t\tgit show N11:bar/qux >actual.qux &&\n> +\t\tgit show N11:bar/quux >actual.quux &&\n> +\t\ttest_cmp expect.baz actual.baz &&\n> +\t\ttest_cmp expect.qux actual.qux &&\n> +\t\ttest_cmp expect.qux actual.quux\n> +\t'\n> +done\n> \n> ###\n> ### series O\n> @@ -3067,6 +3077,7 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n> # There are two sorts of ways a path can be parsed, depending on whether it is\n> # the last field on the line. Additionally, ls without a has a special\n> # case. Test every occurrence of in the grammar against every error case.\n> +# Paths for the root (empty strings) are tested elsewhere.\n> #\n> \n> #\n> @@ -3314,16 +3325,19 @@ test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n> ###\n> # Setup is carried over from series S.\n> \n> -test_expect_success 'T: ls root tree' '\n> -\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n> -\t040000 tree $(git rev-parse S^{tree})\tZ\n> -\tEOF\n> -\tsha1=$(git rev-parse --verify S) &&\n> -\tgit fast-import --import-marks=marks <<-EOF >actual &&\n> -\tls $sha1 \"\"\n> -\tEOF\n> -\ttest_cmp expect actual\n> -'\n> +for root in '\"\"' ''\n> +do\n> +\ttest_expect_success \"T: ls root ($root) tree\" '\n> +\t\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n> +\t\t040000 tree $(git rev-parse S^{tree})\tZ\n> +\t\tEOF\n> +\t\tsha1=$(git rev-parse --verify S) &&\n> +\t\tgit fast-import --import-marks=marks <<-EOF >actual &&\n> +\t\tls $sha1 $root\n> +\t\tEOF\n> +\t\ttest_cmp expect actual\n> +\t'\n> +done\n> \n> test_expect_success 'T: delete branch' '\n> \tgit branch to-delete &&\n> @@ -3425,30 +3439,33 @@ test_expect_success 'U: validate directory delete result' '\n> \tcompare_diff_raw expect actual\n> '\n> \n> -test_expect_success 'U: filedelete root succeeds' '\n> -\tcat >input <<-INPUT_END &&\n> -\tcommit refs/heads/U\n> -\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> -\tdata < -\tmust succeed\n> -\tCOMMIT\n> -\tfrom refs/heads/U^0\n> -\tD \"\"\n> +for root in '\"\"' ''\n> +do\n> +\ttest_expect_success \"U: filedelete root ($root) succeeds\" '\n> +\t\tcat >input <<-INPUT_END &&\n> +\t\tcommit refs/heads/U-delete-root\n> +\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n> +\t\tdata < +\t\tmust succeed\n> +\t\tCOMMIT\n> +\t\tfrom refs/heads/U^0\n> +\t\tD '\"$root\"'\n> \n> -\tINPUT_END\n> +\t\tINPUT_END\n> \n> -\tgit fast-import -'\n> +\t\tgit fast-import +\t'\n> \n> -test_expect_success 'U: validate root delete result' '\n> -\tcat >expect <<-EOF &&\n> -\t:100644 000000 $f7id $ZERO_OID D\thello.c\n> -\tEOF\n> +\ttest_expect_success \"U: validate root ($root) delete result\" '\n> +\t\tcat >expect <<-EOF &&\n> +\t\t:100644 000000 $f7id $ZERO_OID D\thello.c\n> +\t\tEOF\n> \n> -\tgit diff-tree -M -r U^1 U >actual &&\n> +\t\tgit diff-tree -M -r U U-delete-root >actual &&\n> \n> -\tcompare_diff_raw expect actual\n> -'\n> +\t\tcompare_diff_raw expect actual\n> +\t'\n> +done\n> \n> ###\n> ### series V (checkpoint)\n> -- \n> 2.44.0\n> \n"},{"id":"492692","messageId":"CAPx1GvfgC46n_5fk3dHxg7dn393UVMi0CtHKqQ_GAaqnV_ECCQ@mail.gmail.com","threadId":"61174","inReplyTo":"ZhYxNYR33ftEfcPk@tanuki","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2024-04-10T08:18:12Z","receivedAt":"2024-04-10T08:18:25Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Tue, Apr 9, 2024 at 11:30 PM Patrick Steinhardt wrote:\n> > + if (include_spaces)\n> > + *endp = p + strlen(p);\n> > + else\n> > + *endp = strchr(p, ' ');\n> > + strbuf_add(sb, p, *endp - p);\n>\n> strchr(3P) may return a NULL pointer in case there is no space, which\n> would make us segfault here when dereferencing `*endp`. We should\n> probably add a testcase that would hit this edge case.\n\nNote that you can do:\n\n *endp = p + strcspn(p, \" \");\n\n(though `strcspn` is a fundamentally harder operation since it\ntakes a string argument). Everything depends on whether you\nwant to test for an explicit \"there was no space at all\" case of\ncourse; performance considerations are secondary.\n\nChris\n"},{"id":"492693","messageId":"784B5318-F155-4D93-8085-62ACA63432E5@archibald.dev","threadId":"61174","inReplyTo":"CAPx1GvfgC46n_5fk3dHxg7dn393UVMi0CtHKqQ_GAaqnV_ECCQ@mail.gmail.com","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T08:44:49Z","receivedAt":"2024-04-10T08:47:44Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 10, 2024, at 01:18, Chris Torek wrote:\n> On Tue, Apr 9, 2024 at 11:30 PM Patrick Steinhardt wrote:\n>>> + if (include_spaces)\n>>> + *endp = p + strlen(p);\n>>> + else\n>>> + *endp = strchr(p, ' ');\n>>> + strbuf_add(sb, p, *endp - p);\n>> \n>> strchr(3P) may return a NULL pointer in case there is no space, which\n>> would make us segfault here when dereferencing `*endp`. We should\n>> probably add a testcase that would hit this edge case.\n> \n> Note that you can do:\n> \n> *endp = p + strcspn(p, \" \");\n> \n> (though `strcspn` is a fundamentally harder operation since it\n> takes a string argument). Everything depends on whether you\n> want to test for an explicit \"there was no space at all\" case of\n> course; performance considerations are secondary.\n\nI thought strchr returned a pointer to the terminating NUL byte if the needle\nwas not found. Turns out it does return NULL in that case, as you say. strchrnul\ndoes what I want here and I’ve replaced it with that.\n\nI’ve added a test covering this case.\n\nThalia\n\n\n"},{"id":"492694","messageId":"CAPx1GvfUs-dR_RbqnBT3YhDJPMB7b=+fzyow8qjJgJQnJwg0Pw@mail.gmail.com","threadId":"61174","inReplyTo":"784B5318-F155-4D93-8085-62ACA63432E5@archibald.dev","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2024-04-10T08:51:51Z","receivedAt":"2024-04-10T08:52:04Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Wed, Apr 10, 2024 at 1:47 AM Thalia Archibald wrote:\n> strchrnul does what I want here and I’ve replaced it with that.\n\n`strchrnul` is a GNU extension (found on a lot of systems, but not\npart of C90 or C99).\n\nChris\n"},{"id":"492695","messageId":"11351F50-D21D-4C9B-8441-43513A11336A@archibald.dev","threadId":"61174","inReplyTo":"ZhYxNYR33ftEfcPk@tanuki","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:12:44Z","receivedAt":"2024-04-10T09:12:59Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"(Sorry for re-sending)\n\nOn Apr 9, 2024, at 23:27, Patrick Steinhardt wrote:\n> On Mon, Apr 01, 2024 at 09:02:47AM +0000, Thalia Archibald wrote:\n>> \n>> - if (!*endp)\n>> + if (!p)\n>> die(\"Missing dest: %s\", command_buf.buf);\n> \n> So this statement right now doesn't make a whole lot of sense because\n> `p` cannot ever be `NULL` -- we'd segfault before that. Once we update\n> `parse_path()` to handle this correctly it will work as expected though.\n> \n> I was briefly wondering though whether we really want `parse_path()` to\n> set `p` to be a NULL pointer. If we didn't, we could retain the previous\n> behaviour here and instead check for `!*p`.\n\nGood catch. There should be a deref there.\n\nThis mistake was because I originally planned to not allow unquoted empty\nstrings and had factored that condition into parse_path. After your round 1\nfeedback, I changed my mind after reanalysis. The condition you see here is\nsupposed to match the behavior for before and is removed in patch 3/8. There was\nno test before my series exercising this branch and my test for it is added in\n3/8, so it wasn't caught in this intermediate version.\n\n>> + ( printf \"100644 blob $blob2\\t'\"$unquoted_path\"'\\n\" &&\n>> + printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_m.exp &&\n> \n> Also, there is no need to do `'\"$unuoted_path\"'` here. You should be\n> able to refer to `$unquoted_path` just fine even without unquoting again\n> because we use eval to execute the code block. In fact, it can even be\n> harmful as it is known to break shells under some circumstances. See\n> also 7c4449eb31 (t/README: document how to loop around test cases,\n> 2024-03-22), which I think should apply in your case, too.\n\nI agree it makes it less finicky. The one upside to string splicing is that when\na test fails, the substitutions are visible in the dump of the shell script. I\nfound that useful while debugging. The titles can uniquely identify the\n$prefix/$path/$suffix values when looking in the source, since they're all\n1-to-1. Considering the downsides, I've switched to plain substitutions. \n\nThalia\n"},{"id":"492696","messageId":"6C467327-0DB1-4E1F-8CB5-23DE45271348@archibald.dev","threadId":"61174","inReplyTo":"CAPx1GvfUs-dR_RbqnBT3YhDJPMB7b=+fzyow8qjJgJQnJwg0Pw@mail.gmail.com","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:14:16Z","receivedAt":"2024-04-10T09:14:30Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 10, 2024, at 01:51, Chris Torek wrote:\n> On Wed, Apr 10, 2024 at 1:47 AM Thalia Archibald wrote:\n>> strchrnul does what I want here and I’ve replaced it with that.\n> \n> `strchrnul` is a GNU extension (found on a lot of systems, but not\n> part of C90 or C99).\n\nI can’t speak to Git standards, but it seems broadly used in Git, including\nthree times already in fast-import:\n\n$ rg --count-matches --sort=path strchrnul\nadd-patch.c:1\nadvice.c:1\napply.c:2\narchive.c:1\nattr.c:1\nbuiltin/am.c:1\nbuiltin/fast-export.c:5\nbuiltin/fast-import.c:4\nbuiltin/stash.c:1\ncache-tree.c:2\ncommit.c:5\ncompat/mingw.c:1\ncompat/terminal.c:1\nconfig.c:1\ndiff.c:1\nfmt-merge-msg.c:2\nfsck.c:2\ngit-compat-util.h:3\ngit.c:1\ngpg-interface.c:8\ngraph.c:1\nhelp.c:1\nhttp.c:2\nident.c:2\nlog-tree.c:1\nmailmap.c:1\nmatch-trees.c:1\nnotes.c:1\nobject-file.c:2\nparse-options.c:2\npath.c:1\npretty.c:2\nref-filter.c:5\nrefs/debug.c:1\nremote-curl.c:1\nremote.c:1\nrun-command.c:1\nscalar.c:1\nsequencer.c:5\nstrbuf.c:1\ntrailer.c:1\ntransport-helper.c:1\nutf8.c:1\nws.c:1\nwt-status.c:1\n\nThalia\n"},{"id":"492697","messageId":"DF42695D-2FA3-425C-B553-FF33D653CCCC@archibald.dev","threadId":"61174","inReplyTo":"CAPx1GvfUs-dR_RbqnBT3YhDJPMB7b=+fzyow8qjJgJQnJwg0Pw@mail.gmail.com","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:16:24Z","receivedAt":"2024-04-10T09:16:42Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 10, 2024, at 02:14, Thalia Archibald wrote:\n> On Apr 10, 2024, at 01:51, Chris Torek wrote:\n>> On Wed, Apr 10, 2024 at 1:47 AM Thalia Archibald wrote:\n>>> strchrnul does what I want here and I’ve replaced it with that.\n>> \n>> `strchrnul` is a GNU extension (found on a lot of systems, but not\n>> part of C90 or C99).\n> \n> I can’t speak to Git standards, but it seems broadly used in Git, including\n> three times already in fast-import.\n\n… and that would be because it is supplied when unavailable:\n\ngit-compat-util.h\n\n#ifndef HAVE_STRCHRNUL\n#define strchrnul gitstrchrnul\nstatic inline char *gitstrchrnul(const char *s, int c)\n{\n\twhile (*s && *s != c)\n\t\ts++;\n\treturn (char *)s;\n}\n#endif\n\nThalia\n"},{"id":"492699","messageId":"ZhZfF5YH-uwx-wYb@tanuki","threadId":"61174","inReplyTo":"6C467327-0DB1-4E1F-8CB5-23DE45271348@archibald.dev","subject":"Re: [PATCH v2 1/8] fast-import: tighten path unquoting","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-10T09:42:47Z","receivedAt":"2024-04-10T09:42:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 09:14:16AM +0000, Thalia Archibald wrote:\n> On Apr 10, 2024, at 01:51, Chris Torek wrote:\n> > On Wed, Apr 10, 2024 at 1:47 AM Thalia Archibald wrote:\n> >> strchrnul does what I want here and I’ve replaced it with that.\n> > \n> > `strchrnul` is a GNU extension (found on a lot of systems, but not\n> > part of C90 or C99).\n> \n> I can’t speak to Git standards, but it seems broadly used in Git, including\n> three times already in fast-import:\n\nIt's fine to use `strchrnul()` in Git. In case libc doesn't provide it\nwe have a fallback implementation in \"git-compat-util.h\".\n\nPatrick\n"},{"id":"492700","messageId":"cover.1712741870.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1711960552.git.thalia@archibald.dev","subject":"[PATCH v3 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:54:57Z","receivedAt":"2024-04-10T09:55:11Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"> fast-import has subtle differences in how it parses file paths between each\n> occurrence of in the grammar. Many errors are suppressed or not checked,\n> which could lead to silent data corruption. A particularly bad case is when a\n> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> supported), it would be treated as literal bytes instead of a quoted string.\n>\n> Bring path parsing into line with the documented behavior and improve\n> documentation to fill in missing details.\n\nUpdated to address review comments. Thanks, Patrick!\n\nChanges since v2:\n* Fix NUL overrun by replacing `strchr(p, ' ')` with `strchrnul(p, ' ')` in\n patch 1/8\n* Fix \"Missing dest\" error condition in patch 1/8\n* Test missing space after unquoted path\n* Substitute shell parameters in test_expect_success call, instead of with\n string splicing\n* Reformat (-subshells\n* Rewrap long lines in `parse_path` and `parse_path_space`\n\nHopefully, this series sends without any rewrapped lines. I use Proton Mail via\nProton Mail Bridge and Apple Mail. I have no idea how to control this, or if I\neven can, and see no relevant-looking settings in any of the three. In v2 and\nnow v3, I only manually modified the cover letter after using format-patch, not\nany of the others.\n\nThalia\n\n\nThalia Archibald (8):\n fast-import: tighten path unquoting\n fast-import: directly use strbufs for paths\n fast-import: allow unquoted empty path for root\n fast-import: remove dead strbuf\n fast-import: improve documentation for unquoted paths\n fast-import: document C-style escapes for paths\n fast-import: forbid escaped NUL in paths\n fast-import: make comments more precise\n\n Documentation/git-fast-import.txt | 30 +-\n builtin/fast-import.c | 158 ++++----\n t/t9300-fast-import.sh | 624 +++++++++++++++++++++---------\n 3 files changed, 550 insertions(+), 262 deletions(-)\n\nRange-diff against v2:\n1: e790bdf714 ! 1: d9ab0c6a75 fast-import: tighten path unquoting\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n + * or unquoted without escape sequences. When unquoted, it may only contain a\n + * space if `include_spaces` is nonzero.\n + */\n -+static void parse_path(struct strbuf *sb, const char *p, const char **endp, int include_spaces, const char *field)\n ++static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n ++\t\tint include_spaces, const char *field)\n +{\n +\tif (*p == '\"') {\n +\t\tif (unquote_c_style(sb, p, endp))\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n +\t\tif (include_spaces)\n +\t\t\t*endp = p + strlen(p);\n +\t\telse\n -+\t\t\t*endp = strchr(p, ' ');\n ++\t\t\t*endp = strchrnul(p, ' ');\n +\t\tstrbuf_add(sb, p, *endp - p);\n +\t}\n +}\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n + * It may not contain spaces when unquoted. Update *endp to point to the first\n + * character after the space.\n + */\n -+static void parse_path_space(struct strbuf *sb, const char *p, const char **endp, const char *field)\n ++static void parse_path_space(struct strbuf *sb, const char *p,\n ++\t\tconst char **endp, const char *field)\n +{\n +\tparse_path(sb, p, endp, 0, field);\n +\tif (**endp != ' ')\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n \n -\tendp++;\n -\tif (!*endp)\n -+\tif (!p)\n ++\tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n -\n -\td = endp;\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tcommit filemodify\n +\t\tCOMMIT\n +\t\tfrom :301\n -+\t\tM 100644 :402 '\"$path\"'\n ++\t\tM 100644 :402 $path\n +\n +\t\tcommit refs/heads/S-path-eol\n +\t\tmark :303\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tcommit filedelete\n +\t\tCOMMIT\n +\t\tfrom :302\n -+\t\tD '\"$path\"'\n ++\t\tD $path\n +\n +\t\tcommit refs/heads/S-path-eol\n +\t\tmark :304\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tcommit filecopy dest\n +\t\tCOMMIT\n +\t\tfrom :301\n -+\t\tC hello.c '\"$path\"'\n ++\t\tC hello.c $path\n +\n +\t\tcommit refs/heads/S-path-eol\n +\t\tmark :305\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tcommit filerename dest\n +\t\tCOMMIT\n +\t\tfrom :301\n -+\t\tR hello.c '\"$path\"'\n ++\t\tR hello.c $path\n +\n -+\t\tls :305 '\"$path\"'\n ++\t\tls :305 $path\n +\t\tEOF\n +\n +\t\tcommit_m=$(grep :302 marks.out | cut -d\\ -f2) &&\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tblob1=$(grep :401 marks.out | cut -d\\ -f2) &&\n +\t\tblob2=$(grep :402 marks.out | cut -d\\ -f2) &&\n +\n -+\t\t( printf \"100644 blob $blob2\\t'\"$unquoted_path\"'\\n\" &&\n -+\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_m.exp &&\n ++\t\t(\n ++\t\t\tprintf \"100644 blob $blob2\\t$unquoted_path\\n\" &&\n ++\t\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n ++\t\t) | sort >tree_m.exp &&\n +\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n +\t\ttest_cmp tree_m.exp tree_m.out &&\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tgit ls-tree $commit_d >tree_d.out &&\n +\t\ttest_cmp tree_d.exp tree_d.out &&\n +\n -+\t\t( printf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" &&\n -+\t\t printf \"100644 blob $blob1\\thello.c\\n\" ) | sort >tree_c.exp &&\n ++\t\t(\n ++\t\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" &&\n ++\t\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n ++\t\t) | sort >tree_c.exp &&\n +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n +\t\ttest_cmp tree_c.exp tree_c.out &&\n +\n -+\t\tprintf \"100644 blob $blob1\\t'\"$unquoted_path\"'\\n\" >tree_r.exp &&\n ++\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" >tree_r.exp &&\n +\t\tgit ls-tree $commit_r >tree_r.out &&\n +\t\ttest_cmp tree_r.exp tree_r.out &&\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tdata <tree_c.exp &&\n ++\t\t(\n ++\t\t\tprintf \"100644 blob $blob\\t$unquoted_path\\n\" &&\n ++\t\t\tprintf \"100644 blob $blob\\thello2.c\\n\"\n ++\t\t) | sort >tree_c.exp &&\n +\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n +\t\ttest_cmp tree_c.exp tree_c.out &&\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\t\tcommit with bad path\n +\t\tCOMMIT\n +\t\tfrom :2\n -+\t\t'\"$prefix$path$suffix\"'\n ++\t\t$prefix$path$suffix\n +\t\tEOF\n +\n -+\t\ttest_grep '\"'$err_grep'\"' err\n ++\t\ttest_grep \"$err_grep\" err\n +\t'\n +}\n +\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n +\ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n +}\n +test_path_eol_quoted_fail () {\n -+\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n -+\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n -+\ttest_path_fail \"$change\" \"garbage after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Garbage after $field\"\n -+\ttest_path_fail \"$change\" \"space after quoted $field\" \"$prefix\" '\"hello.c\" ' \"$suffix\" \"Garbage after $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n ++\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" ''\n ++\ttest_path_fail \"$change\" \"garbage after quoted $field\" \"$prefix\" '\"hello.c\"' 'x' \"Garbage after $field\"\n ++\ttest_path_fail \"$change\" \"space after quoted $field\" \"$prefix\" '\"hello.c\"' ' ' \"Garbage after $field\"\n +}\n +test_path_eol_fail () {\n -+\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n -+\ttest_path_eol_quoted_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n ++\ttest_path_eol_quoted_fail \"$change\" \"$prefix\" \"$field\"\n +}\n +test_path_space_fail () {\n -+\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n -+\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" \"$suffix\"\n -+\ttest_path_fail \"$change\" \"missing space after quoted $field\" \"$prefix\" '\"hello.c\"x' \"$suffix\" \"Missing space after $field\"\n ++\tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n ++\ttest_path_base_fail \"$change\" \"$prefix\" \"$field\" ' world.c'\n ++\ttest_path_fail \"$change\" \"missing space after quoted $field\" \"$prefix\" '\"hello.c\"' 'x world.c' \"Missing space after $field\"\n ++\ttest_path_fail \"$change\" \"missing space after unquoted $field\" \"$prefix\" 'hello.c' '' \"Missing space after $field\"\n +}\n +\n -+test_path_eol_fail filemodify 'M 100644 :1 ' path ''\n -+test_path_eol_fail filedelete 'D ' path ''\n -+test_path_space_fail filecopy 'C ' source ' world.c'\n -+test_path_eol_fail filecopy 'C hello.c ' dest ''\n -+test_path_space_fail filerename 'R ' source ' world.c'\n -+test_path_eol_fail filerename 'R hello.c ' dest ''\n -+test_path_eol_fail 'ls (in commit)' 'ls :2 ' path ''\n ++test_path_eol_fail filemodify 'M 100644 :1 ' path\n ++test_path_eol_fail filedelete 'D ' path\n ++test_path_space_fail filecopy 'C ' source\n ++test_path_eol_fail filecopy 'C hello.c ' dest\n ++test_path_space_fail filerename 'R ' source\n ++test_path_eol_fail filerename 'R hello.c ' dest\n ++test_path_eol_fail 'ls (in commit)' 'ls :2 ' path\n +\n +# When 'ls' has no , the must be quoted.\n -+test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n ++test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n +\n ###\n ### series T (ls)\n2: 82a6f53c13 ! 2: 696ca27bb7 fast-import: directly use strbufs for paths\n @@ Commit message\n Signed-off-by: Thalia Archibald \n \n ## builtin/fast-import.c ##\n -@@ builtin/fast-import.c: static void parse_path_space(struct strbuf *sb, const char *p, const char **endp\n +@@ builtin/fast-import.c: static void parse_path_space(struct strbuf *sb, const char *p,\n \n static void file_change_m(const char *p, struct branch *b)\n {\n @@ builtin/fast-import.c: static void file_change_m(const char *p, struct branch *b\n +\tstrbuf_reset(&source);\n +\tparse_path_space(&source, p, &p, \"source\");\n \n - \tif (!p)\n + \tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n -\tstrbuf_reset(&d_uq);\n -\tparse_path_eol(&d_uq, p, \"dest\");\n3: 893bbf5e73 ! 3: 39879d0a66 fast-import: allow unquoted empty path for root\n @@ Commit message\n \n ## builtin/fast-import.c ##\n @@ builtin/fast-import.c: static void file_change_cr(const char *p, struct branch *b, int rename)\n - \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&source);\n --\tparse_path_space(&source, p, &p, \"source\");\n + \tparse_path_space(&source, p, &p, \"source\");\n -\n --\tif (!p)\n +-\tif (!*p)\n -\t\tdie(\"Missing dest: %s\", command_buf.buf);\n \tstrbuf_reset(&dest);\n -+\tparse_path_space(&source, p, &p, \"source\");\n \tparse_path_eol(&dest, p, \"dest\");\n \n - \tmemset(&leaf, 0, sizeof(leaf));\n \n ## t/t9300-fast-import.sh ##\n @@ t/t9300-fast-import.sh: test_expect_success 'M: rename subdirectory to new subdirectory' '\n @@ t/t9300-fast-import.sh: test_expect_success 'M: rename subdirectory to new subdi\n -\tfrom refs/heads/M2^0\n -\tR \"\" sub\n +\t\tfrom refs/heads/M2^0\n -+\t\tR '\"$root\"' sub\n ++\t\tR $root sub\n \n -\tINPUT_END\n +\t\tINPUT_END\n @@ t/t9300-fast-import.sh: test_expect_success PIPE 'N: empty directory reads as mi\n -\tcompare_diff_raw expect actual\n -'\n +\t\tfrom refs/heads/branch^0\n -+\t\tM 040000 $root_tree '\"$root\"'\n ++\t\tM 040000 $root_tree $root\n +\t\tINPUT_END\n +\t\tgit fast-import actual &&\n @@ t/t9300-fast-import.sh: test_expect_success PIPE 'N: empty directory reads as mi\n -\tcompare_diff_raw expect actual\n -'\n +\t\tfrom refs/heads/branch^0\n -+\t\tC '\"$root\"' oldroot\n ++\t\tC $root oldroot\n +\t\tINPUT_END\n +\t\tgit fast-import actual &&\n @@ t/t9300-fast-import.sh: test_expect_success 'N: reject foo/ syntax in ls argumen\n -\ttest_cmp expect.foo actual.foo &&\n -\ttest_cmp expect.bar actual.bar\n -'\n -+\t\tM 040000 $tree '\"$root\"'\n ++\t\tM 040000 $tree $root\n +\t\tM 644 inline foo/foo\n +\t\tdata <actual.baz &&\n @@ t/t9300-fast-import.sh: test_expect_success 'S: ls with garbage after sha1 must\n #\n \n #\n -@@ t/t9300-fast-import.sh: test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path ''\n +@@ t/t9300-fast-import.sh: test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n ###\n # Setup is carried over from series S.\n \n @@ t/t9300-fast-import.sh: test_expect_success 'U: validate directory delete result\n +\t\tmust succeed\n +\t\tCOMMIT\n +\t\tfrom refs/heads/U^0\n -+\t\tD '\"$root\"'\n ++\t\tD $root\n \n -\tINPUT_END\n +\t\tINPUT_END\n4: cb05a184e6 = 4: 1cef05e59a fast-import: remove dead strbuf\n5: 1f34b632d7 = 5: 2e78690023 fast-import: improve documentation for unquoted paths\n6: 82a4da68af = 6: 1b07ddffe0 fast-import: document C-style escapes for paths\n7: c087c6a860 ! 7: dc67464b6a fast-import: forbid escaped NUL in paths\n @@ Documentation/git-fast-import.txt: and must be in canonical form. That is it mus\n `filedelete`\n \n ## builtin/fast-import.c ##\n -@@ builtin/fast-import.c: static void parse_path(struct strbuf *sb, const char *p, const char **endp, int\n +@@ builtin/fast-import.c: static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n @@ t/t9300-fast-import.sh: test_path_base_fail () {\n +\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n - \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n + \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n8: a503c55b83 = 8: 5e02d887bc fast-import: make comments more precise\n-- \n2.44.0\n\n\n"},{"id":"492701","messageId":"d9ab0c6a750f6df222d0cb3f46e282000f98680d.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:55:25Z","receivedAt":"2024-04-10T09:55:38Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Path parsing in fast-import is inconsistent and many unquoting errors\nare suppressed or not checked.\n\n appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n filecopy ::= 'C' SP SP LF\n filerename ::= 'R' SP SP LF\n ls ::= 'ls' SP SP LF\n ls-commit ::= 'ls' SP LF\n\nand fast-import.c parses them in five different ways:\n\n1. For filemodify and filedelete:\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes to the end of\n the line (including any number of SP).\n2. For filecopy (source) and filerename (source):\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes up to, but not\n including, the next SP.\n3. For filecopy (dest) and filerename (dest):\n Like 1., but an unquoted empty string is forbidden.\n4. For ls:\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n5. For ls-commit:\n Unquote and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\nIn the first three, any errors from trying to unquote a string are\nsuppressed, so a quoted string that contains invalid escapes would be\ninterpreted as literal bytes. For example, `\"\\xff\"` would fail to\nunquote (because hex escapes are not supported), and it would instead be\ninterpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\ncertainly not intended. Some front-ends erroneously use their language's\nstandard quoting routine instead of matching Git's, which could silently\nintroduce escapes that would be incorrectly parsed due to this and lead\nto data corruption.\n\nThe documentation states “To use a source path that contains SP the path\nmust be quoted.”, so it is expected that some implementations depend on\nspaces being allowed in paths in the final position. Thus we have two\ndocumented ways to parse paths, so simplify the implementation to that.\n\nNow we have:\n\n1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n\n2. `parse_path_space` for filecopy (source) and filerename (source):\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes up to, but not including, the\n next SP. It must be followed by SP.\n\nThere remain two special cases: The dest in filecopy and rename\ncannot be an unquoted empty string (this will be addressed subsequently)\nand in ls-commit must be quoted to disambiguate it from ls.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 104 ++++++++++-------\n t/t9300-fast-import.sh | 258 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 318 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 782bda007c..ce9231afe6 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2258,10 +2258,56 @@ static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n \n+/*\n+ * Parse the path string into the strbuf. It may be quoted with escape sequences\n+ * or unquoted without escape sequences. When unquoted, it may only contain a\n+ * space if `include_spaces` is nonzero.\n+ */\n+static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n+\t\tint include_spaces, const char *field)\n+{\n+\tif (*p == '\"') {\n+\t\tif (unquote_c_style(sb, p, endp))\n+\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t} else {\n+\t\tif (include_spaces)\n+\t\t\t*endp = p + strlen(p);\n+\t\telse\n+\t\t\t*endp = strchrnul(p, ' ');\n+\t\tstrbuf_add(sb, p, *endp - p);\n+\t}\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and complain if this is not the end of\n+ * the string. It may contain spaces even when unquoted.\n+ */\n+static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n+{\n+\tconst char *end;\n+\n+\tparse_path(sb, p, &end, 1, field);\n+\tif (*end)\n+\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and ensure it is followed by a space.\n+ * It may not contain spaces when unquoted. Update *endp to point to the first\n+ * character after the space.\n+ */\n+static void parse_path_space(struct strbuf *sb, const char *p,\n+\t\tconst char **endp, const char *field)\n+{\n+\tparse_path(sb, p, endp, 0, field);\n+\tif (**endp != ' ')\n+\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n+\t(*endp)++;\n+}\n+\n static void file_change_m(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2299,11 +2345,8 @@ static void file_change_m(const char *p, struct branch *b)\n \t}\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n@@ -2367,48 +2410,29 @@ static void file_change_m(const char *p, struct branch *b)\n static void file_change_d(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n }\n \n-static void file_change_cr(const char *s, struct branch *b, int rename)\n+static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *d;\n+\tconst char *s, *d;\n \tstatic struct strbuf s_uq = STRBUF_INIT;\n \tstatic struct strbuf d_uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&s_uq);\n-\tif (!unquote_c_style(&s_uq, s, &endp)) {\n-\t\tif (*endp != ' ')\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t} else {\n-\t\tendp = strchr(s, ' ');\n-\t\tif (!endp)\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t\tstrbuf_add(&s_uq, s, endp - s);\n-\t}\n+\tparse_path_space(&s_uq, p, &p, \"source\");\n \ts = s_uq.buf;\n \n-\tendp++;\n-\tif (!*endp)\n+\tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\n-\td = endp;\n \tstrbuf_reset(&d_uq);\n-\tif (!unquote_c_style(&d_uq, d, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n-\t\td = d_uq.buf;\n-\t}\n+\tparse_path_eol(&d_uq, p, \"dest\");\n+\td = d_uq.buf;\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n@@ -3152,6 +3176,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n+\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3168,16 +3193,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tif (*p == '\"') {\n-\t\tstatic struct strbuf uq = STRBUF_INIT;\n-\t\tconst char *endp;\n-\t\tstrbuf_reset(&uq);\n-\t\tif (unquote_c_style(&uq, p, &endp))\n-\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tstrbuf_reset(&uq);\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 60e30fed3c..de2f1304e8 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -2142,6 +2142,7 @@ test_expect_success 'Q: deny note on empty branch' '\n \tEOF\n \ttest_must_fail git fast-import has a special\n+# case. Test every occurrence of in the grammar against every error case.\n+#\n+\n+#\n+# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n+# filerename (dest), and ls.\n+#\n+# commit :301 from root -- modify hello.c (for setup)\n+# commit :302 from :301 -- modify $path\n+# commit :303 from :302 -- delete $path\n+# commit :304 from :301 -- copy hello.c $path\n+# commit :305 from :301 -- rename hello.c $path\n+# ls :305 $path\n+#\n+test_path_eol_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths at EOL with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_m.exp &&\n+\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n+\t\ttest_cmp tree_m.exp tree_m.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n+\t\tgit ls-tree $commit_d >tree_d.out &&\n+\t\ttest_cmp tree_d.exp tree_d.out &&\n+\n+\t\t(\n+\t\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" &&\n+\t\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n+\t\t) | sort >tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\ttest_cmp out tree_r.exp\n+\t'\n+}\n+\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+\n+#\n+# Valid paths before a space: filecopy (source) and filerename (source).\n+#\n+# commit :301 from root -- modify $path (for setup)\n+# commit :302 from :301 -- copy $path hello2.c\n+# commit :303 from :301 -- rename $path hello2.c\n+#\n+test_path_space_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths before space with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-space\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out\n+\t'\n+}\n+\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+\n+#\n+# Test a single commit change with an invalid path. Run it with all occurrences\n+# of in the grammar against all error kinds.\n+#\n+test_path_fail () {\n+\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n+\ttest_expect_success \"S: $change with $what must fail\" '\n+\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :1\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <, the must be quoted.\n+test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n+\n ###\n ### series T (ls)\n ###\n-- \n2.44.0\n\n\n"},{"id":"492702","messageId":"696ca27bb7267d7c881cb18eb74473ebf271bb5c.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 2/8] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:55:33Z","receivedAt":"2024-04-10T09:55:40Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Previously, one case would not write the path to the strbuf: when the\npath is unquoted and at the end of the string. It was essentially\ncopy-on-write. However, with the logic simplification of the previous\ncommit, this case was eliminated and the strbuf is always populated.\n\nDirectly use the strbufs now instead of an alias.\n\nSince this already changes all the lines that use the strbufs, rename\nthem from `uq` to be more descriptive. That they are unquoted is not\ntheir most important property, so name them after what they carry.\n\nAdditionally, `file_change_m` no longer needs to copy the path before\nreading inline data.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 64 ++++++++++++++++++-------------------------\n 1 file changed, 27 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex ce9231afe6..8f6312fbaf 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2307,7 +2307,7 @@ static void parse_path_space(struct strbuf *sb, const char *p,\n \n static void file_change_m(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2344,13 +2344,12 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n \n \t/* Git does not track empty, non-toplevel directories. */\n-\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n-\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n+\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n+\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n \t\treturn;\n \t}\n \n@@ -2371,10 +2370,6 @@ static void file_change_m(const char *p, struct branch *b)\n \t\tif (S_ISDIR(mode))\n \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n \t\t\t\tcommand_buf.buf);\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\twhile (read_next_command() != EOF) {\n \t\t\tconst char *v;\n \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n@@ -2400,55 +2395,51 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*p) {\n+\tif (!*path.buf) {\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n+\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *s, *d;\n-\tstatic struct strbuf s_uq = STRBUF_INIT;\n-\tstatic struct strbuf d_uq = STRBUF_INIT;\n+\tstatic struct strbuf source = STRBUF_INIT;\n+\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n \n-\tstrbuf_reset(&s_uq);\n-\tparse_path_space(&s_uq, p, &p, \"source\");\n-\ts = s_uq.buf;\n+\tstrbuf_reset(&source);\n+\tparse_path_space(&source, p, &p, \"source\");\n \n \tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\tstrbuf_reset(&d_uq);\n-\tparse_path_eol(&d_uq, p, \"dest\");\n-\td = d_uq.buf;\n+\tstrbuf_reset(&dest);\n+\tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n-\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n \telse\n-\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n-\t\tdie(\"Path %s not in branch\", s);\n-\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n+\t\tdie(\"Path %s not in branch\", source.buf);\n+\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, d,\n+\ttree_content_set(&b->branch_tree, dest.buf,\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n@@ -3176,7 +3167,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3193,10 +3184,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_get(root, p, &leaf, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\n \t * until it is saved. Save, for simplicity.\n@@ -3204,7 +3194,7 @@ static void parse_ls(const char *p, struct branch *b)\n \tif (S_ISDIR(leaf.versions[1].mode))\n \t\tstore_tree(&leaf);\n \n-\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n \tif (leaf.tree)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n-- \n2.44.0\n\n\n"},{"id":"492703","messageId":"39879d0a6627b62f685b38b9afcfe19e4c14c38e.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 3/8] fast-import: allow unquoted empty path for root","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:55:41Z","receivedAt":"2024-04-10T09:56:01Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Ever since filerename was added in f39a946a1f (Support wholesale\ndirectory renames in fast-import, 2007-07-09) and filecopy in b6f3481bb4\n(Teach fast-import to recursively copy files/directories, 2007-07-15),\nboth have produced an error when the destination path is empty. Later,\nwhen support for targeting the root directory with an empty string was\nadded in 2794ad5244 (fast-import: Allow filemodify to set the root,\n2010-10-10), this had the effect of allowing the quoted empty string\n(`\"\"`), but forbidding its unquoted variant (``). This seems to have\nbeen intended as simple data validation for parsing two paths, rather\nthan a syntax restriction, because it was not extended to the other\noperations.\n\nAll other occurrences of paths (in filemodify, filedelete, the source of\nfilecopy and filerename, and ls) allow both.\n\nFor most of this feature's lifetime, the documentation has not\nprescribed the use of quoted empty strings. In e5959106d6\n(Documentation/fast-import: put explanation of M 040000 \"\" in\ncontext, 2011-01-15), its documentation was changed from “`` may\nalso be an empty string (`\"\"`) to specify the root of the tree” to “The\nroot of the tree can be represented by an empty string as ``”.\n\nThus, we can assume that some front-ends have depended on this behavior.\n\nRemove this restriction for the destination paths of filecopy and\nfilerename and change tests targeting the root to test `\"\"` and ``.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 3 -\n t/t9300-fast-import.sh | 363 +++++++++++++++++++++--------------------\n 2 files changed, 190 insertions(+), 176 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 8f6312fbaf..0da7e8a5a5 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2419,9 +2419,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n \tstrbuf_reset(&source);\n \tparse_path_space(&source, p, &p, \"source\");\n-\n-\tif (!*p)\n-\t\tdie(\"Missing dest: %s\", command_buf.buf);\n \tstrbuf_reset(&dest);\n \tparse_path_eol(&dest, p, \"dest\");\n \ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex de2f1304e8..13f98e6688 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1059,30 +1059,33 @@ test_expect_success 'M: rename subdirectory to new subdirectory' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'M: rename root to subdirectory' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/M4\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/M4\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n-\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n-\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n-\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n-\tEOF\n-\tgit fast-import actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n+\t\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n+\t\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n+\t\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n+\t\tEOF\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series N\n@@ -1259,49 +1262,52 @@ test_expect_success PIPE 'N: empty directory reads as missing' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'N: copy root directory by tree hash' '\n-\tcat >expect <<-EOF &&\n-\t:100755 000000 $newf $zero D\tfile3/newf\n-\t:100644 000000 $oldf $zero D\tfile3/oldf\n-\tEOF\n-\troot=$(git rev-parse refs/heads/branch^0^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N6\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 000000 $newf $zero D\tfile3/newf\n+\t\t:100644 000000 $oldf $zero D\tfile3/oldf\n+\t\tEOF\n+\t\troot_tree=$(git rev-parse refs/heads/branch^0^{tree}) &&\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N6\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tM 040000 $root_tree $root\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n \n-test_expect_success 'N: copy root by path' '\n-\tcat >expect <<-EOF &&\n-\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n-\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n-\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n-\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n-\tEOF\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N-copy-root-path\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n+\t\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n+\t\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n+\t\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n+\t\tEOF\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N-copy-root-path\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tC $root oldroot\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n test_expect_success 'N: delete directory by copying' '\n \tcat >expect <<-\\EOF &&\n@@ -1431,98 +1437,102 @@ test_expect_success 'N: reject foo/ syntax in ls argument' '\n \tINPUT_END\n '\n \n-test_expect_success 'N: copy to root by id and modify' '\n-\techo \"hello, world\" >expect.foo &&\n-\techo hello >expect.bar &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N7\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.foo &&\n+\t\techo hello >expect.bar &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N7\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.foo &&\n-\tgit show N8:foo/bar >actual.bar &&\n-\ttest_cmp expect.foo actual.foo &&\n-\ttest_cmp expect.bar actual.bar\n-'\n+\t\tM 040000 $tree $root\n+\t\tM 644 inline foo/foo\n+\t\tdata <actual.foo &&\n+\t\tgit show N8:foo/bar >actual.bar &&\n+\t\ttest_cmp expect.foo actual.foo &&\n+\t\ttest_cmp expect.bar actual.bar\n+\t'\n \n-test_expect_success 'N: extract subtree' '\n-\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N9\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/N9\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect.baz &&\n-\techo hello, world >expect.qux &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N10\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.baz &&\n+\t\techo hello, world >expect.qux &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N10\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.baz &&\n-\tgit show N11:bar/qux >actual.qux &&\n-\tgit show N11:bar/quux >actual.quux &&\n-\ttest_cmp expect.baz actual.baz &&\n-\ttest_cmp expect.qux actual.qux &&\n-\ttest_cmp expect.qux actual.quux'\n+\t\tM 040000 $tree $root\n+\t\tM 100644 inline foo/bar/qux\n+\t\tdata <actual.baz &&\n+\t\tgit show N11:bar/qux >actual.qux &&\n+\t\tgit show N11:bar/quux >actual.quux &&\n+\t\ttest_cmp expect.baz actual.baz &&\n+\t\ttest_cmp expect.qux actual.qux &&\n+\t\ttest_cmp expect.qux actual.quux\n+\t'\n+done\n \n ###\n ### series O\n@@ -3067,6 +3077,7 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n # There are two sorts of ways a path can be parsed, depending on whether it is\n # the last field on the line. Additionally, ls without a has a special\n # case. Test every occurrence of in the grammar against every error case.\n+# Paths for the root (empty strings) are tested elsewhere.\n #\n \n #\n@@ -3321,16 +3332,19 @@ test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n ###\n # Setup is carried over from series S.\n \n-test_expect_success 'T: ls root tree' '\n-\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n-\t040000 tree $(git rev-parse S^{tree})\tZ\n-\tEOF\n-\tsha1=$(git rev-parse --verify S) &&\n-\tgit fast-import --import-marks=marks <<-EOF >actual &&\n-\tls $sha1 \"\"\n-\tEOF\n-\ttest_cmp expect actual\n-'\n+for root in '\"\"' ''\n+do\n+\ttest_expect_success \"T: ls root ($root) tree\" '\n+\t\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n+\t\t040000 tree $(git rev-parse S^{tree})\tZ\n+\t\tEOF\n+\t\tsha1=$(git rev-parse --verify S) &&\n+\t\tgit fast-import --import-marks=marks <<-EOF >actual &&\n+\t\tls $sha1 $root\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n \n test_expect_success 'T: delete branch' '\n \tgit branch to-delete &&\n@@ -3432,30 +3446,33 @@ test_expect_success 'U: validate directory delete result' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'U: filedelete root succeeds' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/U\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/U-delete-root\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 000000 $f7id $ZERO_OID D\thello.c\n-\tEOF\n+\ttest_expect_success \"U: validate root ($root) delete result\" '\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 000000 $f7id $ZERO_OID D\thello.c\n+\t\tEOF\n \n-\tgit diff-tree -M -r U^1 U >actual &&\n+\t\tgit diff-tree -M -r U U-delete-root >actual &&\n \n-\tcompare_diff_raw expect actual\n-'\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series V (checkpoint)\n-- \n2.44.0\n\n\n"},{"id":"492704","messageId":"2e78690023f653436ec2eba0c5fb9dfa6bb62624.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 5/8] fast-import: improve documentation for unquoted paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:55:56Z","receivedAt":"2024-04-10T09:56:08Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"It describes what cannot be in an unquoted path, but not what it is.\nReframe it as a definition of unquoted paths. The requirement that it\nnot start with `\"` is the core element that implies the rest.\n\nThe restriction that the source paths of filecopy and filerename cannot\ncontain SP is only stated in their respective sections. Restate it in\nthe section.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 19 ++++++++++++-------\n 1 file changed, 12 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex b2607366b9..f26d7a8571 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n In both formats `` is the complete path of the file to be added\n (if not already existing) or modified (if already existing).\n \n-A `` string must use UNIX-style directory separators (forward\n-slash `/`), may contain any byte other than `LF`, and must not\n-start with double quote (`\"`).\n+A `` can be written as unquoted bytes or a C-style quoted string:\n \n-A path can use C-style string quoting; this is accepted in all cases\n-and mandatory if the filename starts with double quote or contains\n-`LF`. In C-style quoting, the complete name should be surrounded with\n+When a `` does not start with double quote (`\"`), it is an\n+unquoted string and is parsed as literal bytes without any escape\n+sequences. However, if the filename contains `LF` or starts with double\n+quote, it must be written as a quoted string. Additionally, the source\n+`` in `filecopy` or `filerename` must be quoted if it contains SP.\n+\n+A `` can use C-style string quoting; this is accepted in all cases\n+and mandatory in the cases where the filename cannot be represented as\n+an unquoted string. In C-style quoting, the complete name should be surrounded with\n double quotes, and any `LF`, backslash, or double quote characters\n must be escaped by preceding them with a backslash (e.g.,\n `\"path/with\\n, \\\\ and \\\" in it\"`).\n \n-The value of `` must be in canonical form. That is it must not:\n+A `` must use UNIX-style directory separators (forward slash `/`)\n+and must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n-- \n2.44.0\n\n\n"},{"id":"492705","messageId":"1b07ddffe000ed2ab34bd41f4f0558ae8b2dd663.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 6/8] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:56:03Z","receivedAt":"2024-04-10T09:56:15Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Simply saying “C-style” string quoting is imprecise, as only a subset of\nC escapes are supported. Document the exact escapes.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 12 ++++++++----\n t/t9300-fast-import.sh | 10 ++++++----\n 2 files changed, 14 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex f26d7a8571..db53b50268 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -640,10 +640,14 @@ quote, it must be written as a quoted string. Additionally, the source\n \n A `` can use C-style string quoting; this is accepted in all cases\n and mandatory in the cases where the filename cannot be represented as\n-an unquoted string. In C-style quoting, the complete name should be surrounded with\n-double quotes, and any `LF`, backslash, or double quote characters\n-must be escaped by preceding them with a backslash (e.g.,\n-`\"path/with\\n, \\\\ and \\\" in it\"`).\n+an unquoted string. In C-style quoting, the complete filename is\n+surrounded with double quote (`\"`) and certain characters must be\n+escaped by preceding them with a backslash: `LF` is written as `\\n`,\n+backslash as `\\\\`, and double quote as `\\\"`. Some characters may may\n+optionally be written with escape sequences: `\\a` for bell, `\\b` for\n+backspace, `\\f` for form feed, `\\n` for line feed, `\\r` for carriage\n+return, `\\t` for horizontal tab, and `\\v` for vertical tab. Any byte can\n+be written with 3-digit octal codes (e.g., `\\033`).\n \n A `` must use UNIX-style directory separators (forward slash `/`)\n and must be in canonical form. That is it must not:\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 13f98e6688..5cde8f8d01 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3189,8 +3189,9 @@ test_path_eol_success () {\n \t'\n }\n \n-test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Valid paths before a space: filecopy (source) and filerename (source).\n@@ -3256,8 +3257,9 @@ test_path_space_success () {\n \t'\n }\n \n-test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Test a single commit change with an invalid path. Run it with all occurrences\n-- \n2.44.0\n\n\n"},{"id":"492706","messageId":"dc67464b6aec46c5b6ac0d9cb8549d48a35c5353.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 7/8] fast-import: forbid escaped NUL in paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:56:11Z","receivedAt":"2024-04-10T09:56:20Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"NUL cannot appear in paths. Even disregarding filesystem path\nlimitations, the tree object format delimits with NUL, so such a path\ncannot be encoded by Git.\n\nWhen a quoted path is unquoted, it could possibly contain NUL from\n\"\\000\". Forbid it so it isn't truncated.\n\nfast-import still has other issues with NUL, but those will be addressed\nlater.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 1 +\n builtin/fast-import.c | 2 ++\n t/t9300-fast-import.sh | 1 +\n 3 files changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex db53b50268..edda30f90c 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -660,6 +660,7 @@ and must be in canonical form. That is it must not:\n \n The root of the tree can be represented by an empty string as ``.\n \n+`` cannot contain NUL, either literally or escaped as `\\000`.\n It is recommended that `` always be encoded using UTF-8.\n \n `filedelete`\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 7a398dc975..98096b6fa7 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2269,6 +2269,8 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t\tif (strlen(sb->buf) != sb->len)\n+\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n \t} else {\n \t\tif (include_spaces)\n \t\t\t*endp = p + strlen(p);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 5cde8f8d01..1e68426852 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3300,6 +3300,7 @@ test_path_base_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n \ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n \ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n+\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n-- \n2.44.0\n\n\n"},{"id":"492707","messageId":"5e02d887bc93a4212499c3bd13b36c401cc19081.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 8/8] fast-import: make comments more precise","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:56:18Z","receivedAt":"2024-04-10T09:56:25Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The former is somewhat imprecise. The latter became out of sync with the\nbehavior in e814c39c2f (fast-import: refactor parsing of spaces,\n2014-06-18).\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 98096b6fa7..fd23a00150 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2210,7 +2210,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n *\n * idnum ::= ':' bigint;\n *\n- * Return the first character after the value in *endptr.\n+ * Update *endptr to point to the first character after the value.\n *\n * Complain if the following character is not what is expected,\n * either a space or end of the string.\n@@ -2243,8 +2243,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n }\n \n /*\n- * Parse the mark reference, demanding a trailing space. Return a\n- * pointer to the space.\n+ * Parse the mark reference, demanding a trailing space. Update *p to\n+ * point to the first character after the space.\n */\n static uintmax_t parse_mark_ref_space(const char **p)\n {\n-- \n2.44.0\n\n\n"},{"id":"492708","messageId":"1cef05e59a806069cc921511329dea67eee80dba.1712741871.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v3 4/8] fast-import: remove dead strbuf","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T09:55:49Z","receivedAt":"2024-04-10T09:56:35Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The strbuf in `note_change_n` is to copy the remainder of `p` before\npotentially invalidating it when reading the next line. However, `p` is\nnot used after that point. It has been unused since the function was\ncreated in a8dd2e7d2b (fast-import: Add support for importing commit\nnotes, 2009-10-09) and looks to be a fossil from adapting\n`file_change_m`. Remove it.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 0da7e8a5a5..7a398dc975 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2444,7 +2444,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct branch *s;\n \tstruct object_id oid, commit_oid;\n@@ -2509,10 +2508,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\tread_next_command();\n \t\tparse_and_store_blob(&last_blob, &oid, 0);\n \t} else if (oe) {\n-- \n2.44.0\n\n\n"},{"id":"492709","messageId":"17089372-8AF1-49A5-A582-8153E87380AE@archibald.dev","threadId":"61174","inReplyTo":"ZhYxP42Br2h1mq5r@tanuki","subject":"Re: [PATCH v2 2/8] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T10:07:13Z","receivedAt":"2024-04-10T10:08:16Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 9, 2024, at 23:27, Patrick Steinhardt wrote:\n> On Mon, Apr 01, 2024 at 09:03:06AM +0000, Thalia Archibald wrote:\n>> \n>> + parse_path_eol(&path, p\n>> , \"path\");\n> \n> This looks weird. Did you manually edit the patch or is there some weird\n> character in here that breaks diff generation?\n> \n>> + tree_content_get(&b-\n>>> branch_tree, source.buf, &leaf, 1);\n> \n> Same here. Is your mail agent maybe wrapping lines?\n> \n>> - s\n>> trbuf_reset(&uq);\n> \n> And here.\n> \n> Other than those formatting issues this patch looks fine to me.\n\nI’m not able to reproduce these rewrapping issues anywhere I view this email: in\nmy outbox, inbox, or the archive. I think it’s on your end.\n\nhttps://lore.kernel.org/git/82a6f53c1326a420348eb70461f5929340a930d3.1711960552.git.thalia@archibald.dev/\n\nThalia\n"},{"id":"492711","messageId":"ZhZnfHqMD-KtIsF2@tanuki","threadId":"61174","inReplyTo":"17089372-8AF1-49A5-A582-8153E87380AE@archibald.dev","subject":"Re: [PATCH v2 2/8] fast-import: directly use strbufs for paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-10T10:18:36Z","receivedAt":"2024-04-10T10:18:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 10:07:13AM +0000, Thalia Archibald wrote:\n> On Apr 9, 2024, at 23:27, Patrick Steinhardt wrote:\n> > On Mon, Apr 01, 2024 at 09:03:06AM +0000, Thalia Archibald wrote:\n> >> \n> >> + parse_path_eol(&path, p\n> >> , \"path\");\n> > \n> > This looks weird. Did you manually edit the patch or is there some weird\n> > character in here that breaks diff generation?\n> > \n> >> + tree_content_get(&b-\n> >>> branch_tree, source.buf, &leaf, 1);\n> > \n> > Same here. Is your mail agent maybe wrapping lines?\n> > \n> >> - s\n> >> trbuf_reset(&uq);\n> > \n> > And here.\n> > \n> > Other than those formatting issues this patch looks fine to me.\n> \n> I’m not able to reproduce these rewrapping issues anywhere I view this email: in\n> my outbox, inbox, or the archive. I think it’s on your end.\n> \n> https://lore.kernel.org/git/82a6f53c1326a420348eb70461f5929340a930d3.1711960552.git.thalia@archibald.dev/\n\nCould be that this is happening because the mails you sent to me are\nactually encrypted.\n\nPatrick\n"},{"id":"492733","messageId":"xmqqjzl5t6mn.fsf@gitster.g","threadId":"61174","inReplyTo":"1b07ddffe000ed2ab34bd41f4f0558ae8b2dd663.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 6/8] fast-import: document C-style escapes for paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-10T18:28:00Z","receivedAt":"2024-04-10T18:28:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> +an unquoted string. In C-style quoting, the complete filename is\n> +surrounded with double quote (`\"`) and certain characters must be\n> +escaped by preceding them with a backslash: `LF` is written as `\\n`,\n> +backslash as `\\\\`, and double quote as `\\\"`. Some characters may may\n\n\"may may\"?\n\n> +optionally be written with escape sequences: `\\a` for bell, `\\b` for\n> +backspace, `\\f` for form feed, `\\n` for line feed, `\\r` for carriage\n> +return, `\\t` for horizontal tab, and `\\v` for vertical tab. Any byte can\n> +be written with 3-digit octal codes (e.g., `\\033`).\n\nSeparating the escaped characters into two classes (mandatory LF and\nBackSlash, and others) is probably a good idea to clarify the\ndescription. Nicely done.\n\n> A `` must use UNIX-style directory separators (forward slash `/`)\n> and must be in canonical form. That is it must not:\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 13f98e6688..5cde8f8d01 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -3189,8 +3189,9 @@ test_path_eol_success () {\n> \t'\n> }\n> \n> -test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> -test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n> +test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n\nIt is annoying to see these changes (and the same for the next\nhunk). Would it make a lot of damage to existing lines in this file\nif we just say \"do not align with extra spaces in between strings\"?\nIf so, that is a good reason to keep doing things this way, but if I\nrecall correctly, these test_path_eol/space_success are what this\nseries added to the file, so if we stop such alignment from the get-go,\nit may be alright.\n\n> +test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n> \n> #\n> # Valid paths before a space: filecopy (source) and filerename (source).\n> @@ -3256,8 +3257,9 @@ test_path_space_success () {\n> \t'\n> }\n> \n> -test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> -test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n> +test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n> +test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n> +test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n> \n> #\n> # Test a single commit change with an invalid path. Run it with all occurrences\n"},{"id":"492734","messageId":"xmqqwmp5rqz1.fsf@gitster.g","threadId":"61174","inReplyTo":"dc67464b6aec46c5b6ac0d9cb8549d48a35c5353.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 7/8] fast-import: forbid escaped NUL in paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-10T18:51:30Z","receivedAt":"2024-04-10T18:51:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> NUL cannot appear in paths. Even disregarding filesystem path\n> limitations, the tree object format delimits with NUL, so such a path\n> cannot be encoded by Git.\n>\n> When a quoted path is unquoted, it could possibly contain NUL from\n> \"\\000\". Forbid it so it isn't truncated.\n>\n> fast-import still has other issues with NUL, but those will be addressed\n> later.\n\nLater meaning outside the series, as 8/8 is not about that? Not a\ncomplaint, and if the way I interpreted is correct, then there is no\nneed to update the above statement. Just double-checking.\n\n> Signed-off-by: Thalia Archibald \n> ---\n> Documentation/git-fast-import.txt | 1 +\n> builtin/fast-import.c | 2 ++\n> t/t9300-fast-import.sh | 1 +\n> 3 files changed, 4 insertions(+)\n>\n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index db53b50268..edda30f90c 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -660,6 +660,7 @@ and must be in canonical form. That is it must not:\n> \n> The root of the tree can be represented by an empty string as ``.\n> \n> +`` cannot contain NUL, either literally or escaped as `\\000`.\n\nOK.\n\n> +\t\tif (strlen(sb->buf) != sb->len)\n> +\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n\nNice. !memchr(sb->buf, ch, sb->len) would be more general solution\nif we were looking for ch that is not NUL, but for checking NUL,\nwhat you wrote is the most natural to read.\n\n> \t} else {\n> \t\tif (include_spaces)\n> \t\t\t*endp = p + strlen(p);\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index 5cde8f8d01..1e68426852 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n> @@ -3300,6 +3300,7 @@ test_path_base_fail () {\n> \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n> \ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n> \ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n> +\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n> }\n> test_path_eol_quoted_fail () {\n> \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n"},{"id":"492735","messageId":"xmqqr0fdrpl1.fsf@gitster.g","threadId":"61174","inReplyTo":"5e02d887bc93a4212499c3bd13b36c401cc19081.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 8/8] fast-import: make comments more precise","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-10T19:21:30Z","receivedAt":"2024-04-10T19:21:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> The former is somewhat imprecise. The latter became out of sync with the\n> behavior in e814c39c2f (fast-import: refactor parsing of spaces,\n> 2014-06-18).\n>\n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 6 +++---\n> 1 file changed, 3 insertions(+), 3 deletions(-)\n\nThanks for being careful. Looking good.\n\n>\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 98096b6fa7..fd23a00150 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2210,7 +2210,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n> *\n> * idnum ::= ':' bigint;\n> *\n> - * Return the first character after the value in *endptr.\n> + * Update *endptr to point to the first character after the value.\n> *\n> * Complain if the following character is not what is expected,\n> * either a space or end of the string.\n> @@ -2243,8 +2243,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n> }\n> \n> /*\n> - * Parse the mark reference, demanding a trailing space. Return a\n> - * pointer to the space.\n> + * Parse the mark reference, demanding a trailing space. Update *p to\n> + * point to the first character after the space.\n> */\n> static uintmax_t parse_mark_ref_space(const char **p)\n> {\n"},{"id":"492748","messageId":"FEF264E3-B671-4F0C-AE41-FC8EE44DBE28@archibald.dev","threadId":"61174","inReplyTo":"xmqqjzl5t6mn.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-10T22:50:21Z","receivedAt":"2024-04-10T22:50:33Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 10, 2024, at 11:28, Junio C Hamano wrote:\n> Thalia Archibald writes:\n>> \n>> -test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n>> -test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n>> +test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n>> +test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n>> +test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n> \n> It is annoying to see these changes (and the same for the next\n> hunk). Would it make a lot of damage to existing lines in this file\n> if we just say \"do not align with extra spaces in between strings\"?\n> If so, that is a good reason to keep doing things this way, but if I\n> recall correctly, these test_path_eol/space_success are what this\n> series added to the file, so if we stop such alignment from the get-go,\n> it may be alright.\n> \n>> -test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n>> -test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n>> +test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n>> +test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n>> +test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' ‘hi.c'\n\nIs it a style problem, that you prefer parameters to not be aligned? I\nthink it reads nicer this way, especially because there are quotes\nwithin quotes and spaces at the starts and ends of strings, which can\nlead to reinterpreting the boundaries of the strings on a less-careful\nread through. They’re like a table of tests. But ultimately, it should\nbe the Git style that prevails not mine, so if that’s it, I’ll change\nit.\n\nOr I could preemptively align them according to the final alignment in\nthis series. I expect there wouldn't be many changes to these tests\nlater, so it should be stable.\n\nI expected more pushback with 3/8, where 9 tests were indented to place\nthem inside loops in order to test them with multiple values for root,\nso it seems not to be purely about whitespace changes in diffs.\n\nIn any case, it’s not a big deal and I’m happy to go with your\ndirection.\n\nThalia\n"},{"id":"492753","messageId":"xmqqy19kqxbd.fsf@gitster.g","threadId":"61174","inReplyTo":"FEF264E3-B671-4F0C-AE41-FC8EE44DBE28@archibald.dev","subject":"Re: [PATCH v3 6/8] fast-import: document C-style escapes for paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-11T05:32:06Z","receivedAt":"2024-04-11T05:32:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> I expected more pushback with 3/8, where 9 tests were indented to place\n> them inside loops in order to test them with multiple values for root,\n> so it seems not to be purely about whitespace changes in diffs.\n\nWell, if I read it, I may have (or not have) comments on the step,\nbut because Patrick started from front, I was reading backwards from\nthe end of the series, and I didn't reach 3/8 ;-)\n\n\n"},{"id":"492778","messageId":"Zhep8lEsMa0aEpy_@tanuki","threadId":"61174","inReplyTo":"xmqqy19kqxbd.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] fast-import: document C-style escapes for paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:14:26Z","receivedAt":"2024-04-11T09:14:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 10:32:06PM -0700, Junio C Hamano wrote:\n> Thalia Archibald writes:\n> \n> > I expected more pushback with 3/8, where 9 tests were indented to place\n> > them inside loops in order to test them with multiple values for root,\n> > so it seems not to be purely about whitespace changes in diffs.\n> \n> Well, if I read it, I may have (or not have) comments on the step,\n> but because Patrick started from front, I was reading backwards from\n> the end of the series, and I didn't reach 3/8 ;-)\n\nI wasn't all that happy with that conversion indeed. But I also couldn't\nreally think of a nicer way to handle this. While we could've just not\nreindented the tests at all, I kind of doubt that this would be the\nwisest decisions.\n\nSo I just didn't complain :)\n\nPatrick\n"},{"id":"492798","messageId":"xmqq5xwnu182.fsf@gitster.g","threadId":"61174","inReplyTo":"2e78690023f653436ec2eba0c5fb9dfa6bb62624.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 5/8] fast-import: improve documentation for unquoted paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-11T19:51:41Z","receivedAt":"2024-04-11T19:51:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> It describes what cannot be in an unquoted path, but not what it is.\n> Reframe it as a definition of unquoted paths. The requirement that it\n> not start with `\"` is the core element that implies the rest.\n\nThe other is that a path with LF in it cannot be written unquoted,\nwhich should be treated the same way as ones that begin with dq in\nthis explanation, I think.\n\n> The restriction that the source paths of filecopy and filerename cannot\n> contain SP is only stated in their respective sections. Restate it in\n> the section.\n\nElsewhere later in the series we clarify that NUL cannot appear in a\npath, so the above looks perfect at this point in the series.\n\n> Signed-off-by: Thalia Archibald \n> ---\n> Documentation/git-fast-import.txt | 19 ++++++++++++-------\n> 1 file changed, 12 insertions(+), 7 deletions(-)\n>\n> diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\n> index b2607366b9..f26d7a8571 100644\n> --- a/Documentation/git-fast-import.txt\n> +++ b/Documentation/git-fast-import.txt\n> @@ -630,18 +630,23 @@ in octal. Git only supports the following modes:\n> In both formats `` is the complete path of the file to be added\n> (if not already existing) or modified (if already existing).\n> \n> -A `` string must use UNIX-style directory separators (forward\n> -slash `/`), may contain any byte other than `LF`, and must not\n> -start with double quote (`\"`).\n> \n> -A path can use C-style string quoting; this is accepted in all cases\n> -and mandatory if the filename starts with double quote or contains\n> -`LF`. In C-style quoting, the complete name should be surrounded with\n\n> +A `` can be written as unquoted bytes or a C-style quoted string:\n\nIf it is followed by two-bullet-point enumeration (one for unquoted,\nthe other for quoted), then sentence ending in a colon here is\nperfectly fine, but we are not doing so in the next two paragraphs,\nso let's give it a normal full-stop (period). And make it a\nparagraph on its own, which you did correctly.\n\n> +When a `` does not start with double quote (`\"`), it is an\n> +unquoted string and is parsed as literal bytes without any escape\n> +sequences. However, if the filename contains `LF` or starts with double\n> +quote, it must be written as a quoted string. Additionally, the source\n> +`` in `filecopy` or `filerename` must be quoted if it contains SP.\n\nAs the description for in filecopy and filerename refers back\nto this description, this \"Additionally\" is a good clarification to\nhave here. Nicely done.\n\n> +A `` can use C-style string quoting; this is accepted in all cases\n> +and mandatory in the cases where the filename cannot be represented as\n> +an unquoted string. In C-style quoting, the complete name should be surrounded with\n\nI somehow think the early part (before \"In C-style quoting\") is\nredundant and unnecessary, as we started the section with \"as\nunquoted bytes or a C-style quoted string.\" As the previous\nparagraph about unquoted path begins with \"When a does not\nstart with a double quote\", I would have expected this paragraph to\nbegin like so:\n\n\tWhen a `` begins with a double quote (`\"`), it is a\n\tC-style quoted string, where the complete name is enclosed\n\tin a pair of double quotes, and ...\n\n> double quotes, and any `LF`, backslash, or double quote characters\n> must be escaped by preceding them with a backslash (e.g.,\n> `\"path/with\\n, \\\\ and \\\" in it\"`).\n> \n> -The value of `` must be in canonical form. That is it must not:\n> +A `` must use UNIX-style directory separators (forward slash `/`)\n> +and must be in canonical form. That is it must not:\n> \n> * contain an empty directory component (e.g. `foo//bar` is invalid),\n> * end with a directory separator (e.g. `foo/` is invalid),\n\nThanks.\n"},{"id":"492799","messageId":"xmqq1q7bu14z.fsf@gitster.g","threadId":"61174","inReplyTo":"1cef05e59a806069cc921511329dea67eee80dba.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 4/8] fast-import: remove dead strbuf","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-11T19:53:32Z","receivedAt":"2024-04-11T19:53:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> The strbuf in `note_change_n` is to copy the remainder of `p` before\n> potentially invalidating it when reading the next line. However, `p` is\n> not used after that point. It has been unused since the function was\n> created in a8dd2e7d2b (fast-import: Add support for importing commit\n> notes, 2009-10-09) and looks to be a fossil from adapting\n> `file_change_m`. Remove it.\n>\n> Signed-off-by: Thalia Archibald \n> ---\n> builtin/fast-import.c | 5 -----\n> 1 file changed, 5 deletions(-)\n\nLosing code that is not used is always good ;-)\n\n\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 0da7e8a5a5..7a398dc975 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2444,7 +2444,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n> \n> static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n> {\n> -\tstatic struct strbuf uq = STRBUF_INIT;\n> \tstruct object_entry *oe;\n> \tstruct branch *s;\n> \tstruct object_id oid, commit_oid;\n> @@ -2509,10 +2508,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n> \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n> \n> \tif (inline_data) {\n> -\t\tif (p != uq.buf) {\n> -\t\t\tstrbuf_addstr(&uq, p);\n> -\t\t\tp = uq.buf;\n> -\t\t}\n> \t\tread_next_command();\n> \t\tparse_and_store_blob(&last_blob, &oid, 0);\n> \t} else if (oe) {\n"},{"id":"492801","messageId":"xmqqwmp3smal.fsf@gitster.g","threadId":"61174","inReplyTo":"39879d0a6627b62f685b38b9afcfe19e4c14c38e.1712741871.git.thalia@archibald.dev","subject":"Re: [PATCH v3 3/8] fast-import: allow unquoted empty path for root","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-11T19:59:30Z","receivedAt":"2024-04-11T19:59:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> For most of this feature's lifetime, the documentation has not\n> prescribed the use of quoted empty strings. In e5959106d6\n> (Documentation/fast-import: put explanation of M 040000 \"\" in\n> context, 2011-01-15), its documentation was changed from “`` may\n> also be an empty string (`\"\"`) to specify the root of the tree” to “The\n> root of the tree can be represented by an empty string as ``”.\n>\n> Thus, we can assume that some front-ends have depended on this behavior.\n\nIf I were writing this, I would say \"must\" instead of \"can\", as\notherwise it would probably be a good idea if we could tighten it.\n\nOf course no need to reroll just to update this.\n\n> diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\n> index de2f1304e8..13f98e6688 100755\n> --- a/t/t9300-fast-import.sh\n> +++ b/t/t9300-fast-import.sh\n\nThe changes needed to test both a C_quoted empty string and an\nunquoted empty string are so small (when viewed with \"show -w\")\nand pleasant. Nicely done.\n"},{"id":"492845","messageId":"cover.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712741870.git.thalia@archibald.dev","subject":"[PATCH v4 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:01:41Z","receivedAt":"2024-04-12T08:01:57Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"> fast-import has subtle differences in how it parses file paths between each\n> occurrence of in the grammar. Many errors are suppressed or not checked,\n> which could lead to silent data corruption. A particularly bad case is when a\n> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> supported), it would be treated as literal bytes instead of a quoted string.\n>\n> Bring path parsing into line with the documented behavior and improve\n> documentation to fill in missing details.\n\nUpdated to address Junio's review comments. Thanks!\n\nThis round needed no code changes, so is probably ready, if the documentation\nchanges look good.\n\nChanges since v3:\n* Reword documentation paragraphs on path quoting.\n* Fix minor typo in commit message.\n\nThalia\n\n\nThalia Archibald (8):\n fast-import: tighten path unquoting\n fast-import: directly use strbufs for paths\n fast-import: allow unquoted empty path for root\n fast-import: remove dead strbuf\n fast-import: improve documentation for path quoting\n fast-import: document C-style escapes for paths\n fast-import: forbid escaped NUL in paths\n fast-import: make comments more precise\n\n Documentation/git-fast-import.txt | 31 +-\n builtin/fast-import.c | 158 ++++----\n t/t9300-fast-import.sh | 624 +++++++++++++++++++++---------\n 3 files changed, 551 insertions(+), 262 deletions(-)\n\nRange-diff against v3:\n1: d9ab0c6a75 = 1: d6ea8aca46 fast-import: tighten path unquoting\n2: 696ca27bb7 = 2: 9499f34aae fast-import: directly use strbufs for paths\n3: 39879d0a66 ! 3: 9b1e6b80f5 fast-import: allow unquoted empty path for root\n @@ Commit message\n also be an empty string (`\"\"`) to specify the root of the tree” to “The\n root of the tree can be represented by an empty string as ``”.\n \n - Thus, we can assume that some front-ends have depended on this behavior.\n + Thus, we should assume that some front-ends have depended on this\n + behavior.\n \n Remove this restriction for the destination paths of filecopy and\n filerename and change tests targeting the root to test `\"\"` and ``.\n4: 1cef05e59a = 4: 1a2b0dc616 fast-import: remove dead strbuf\n5: 2e78690023 ! 5: fb0d870d53 fast-import: improve documentation for unquoted paths\n @@ Metadata\n Author: Thalia Archibald \n \n ## Commit message ##\n - fast-import: improve documentation for unquoted paths\n + fast-import: improve documentation for path quoting\n \n - It describes what cannot be in an unquoted path, but not what it is.\n - Reframe it as a definition of unquoted paths. The requirement that it\n - not start with `\"` is the core element that implies the rest.\n + It describes what characters cannot be in an unquoted path, but not\n + their semantics. Reframe it as a definition of unquoted paths. From the\n + perspective of the parser, whether it starts with `\"` is what defines\n + whether it will parse it as quoted or unquoted.\n +\n + The restrictions on characters in unquoted paths (with starting-\", LF,\n + and spaces) are explained in the quoted paragraph. Move it to the\n + unquoted paragraph and reword.\n \n The restriction that the source paths of filecopy and filerename cannot\n contain SP is only stated in their respective sections. Restate it in\n @@ Documentation/git-fast-import.txt: in octal. Git only supports the following mo\n -A `` string must use UNIX-style directory separators (forward\n -slash `/`), may contain any byte other than `LF`, and must not\n -start with double quote (`\"`).\n -+A `` can be written as unquoted bytes or a C-style quoted string:\n ++A `` can be written as unquoted bytes or a C-style quoted string.\n \n -A path can use C-style string quoting; this is accepted in all cases\n -and mandatory if the filename starts with double quote or contains\n -`LF`. In C-style quoting, the complete name should be surrounded with\n -+When a `` does not start with double quote (`\"`), it is an\n +-double quotes, and any `LF`, backslash, or double quote characters\n +-must be escaped by preceding them with a backslash (e.g.,\n +-`\"path/with\\n, \\\\ and \\\" in it\"`).\n ++When a `` does not start with a double quote (`\"`), it is an\n +unquoted string and is parsed as literal bytes without any escape\n +sequences. However, if the filename contains `LF` or starts with double\n -+quote, it must be written as a quoted string. Additionally, the source\n -+`` in `filecopy` or `filerename` must be quoted if it contains SP.\n -+\n -+A `` can use C-style string quoting; this is accepted in all cases\n -+and mandatory in the cases where the filename cannot be represented as\n -+an unquoted string. In C-style quoting, the complete name should be surrounded with\n - double quotes, and any `LF`, backslash, or double quote characters\n - must be escaped by preceding them with a backslash (e.g.,\n - `\"path/with\\n, \\\\ and \\\" in it\"`).\n ++quote, it cannot be represented as an unquoted string and must be\n ++quoted. Additionally, the source `` in `filecopy` or `filerename`\n ++must be quoted if it contains SP.\n \n -The value of `` must be in canonical form. That is it must not:\n ++When a `` starts with a double quote (`\"`), it is a C-style quoted\n ++string, where the complete filename is enclosed in a pair of double\n ++quotes and escape sequences are used. Certain characters must be escaped\n ++by preceding them with a backslash: `LF` is written as `\\n`, backslash\n ++as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n ++quoted strings.\n ++\n +A `` must use UNIX-style directory separators (forward slash `/`)\n -+and must be in canonical form. That is it must not:\n ++and its value must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n6: 1b07ddffe0 ! 6: 4b6017ded8 fast-import: document C-style escapes for paths\n @@ Commit message\n Signed-off-by: Thalia Archibald \n \n ## Documentation/git-fast-import.txt ##\n -@@ Documentation/git-fast-import.txt: quote, it must be written as a quoted string. Additionally, the source\n - \n - A `` can use C-style string quoting; this is accepted in all cases\n - and mandatory in the cases where the filename cannot be represented as\n --an unquoted string. In C-style quoting, the complete name should be surrounded with\n --double quotes, and any `LF`, backslash, or double quote characters\n --must be escaped by preceding them with a backslash (e.g.,\n --`\"path/with\\n, \\\\ and \\\" in it\"`).\n -+an unquoted string. In C-style quoting, the complete filename is\n -+surrounded with double quote (`\"`) and certain characters must be\n -+escaped by preceding them with a backslash: `LF` is written as `\\n`,\n -+backslash as `\\\\`, and double quote as `\\\"`. Some characters may may\n -+optionally be written with escape sequences: `\\a` for bell, `\\b` for\n -+backspace, `\\f` for form feed, `\\n` for line feed, `\\r` for carriage\n -+return, `\\t` for horizontal tab, and `\\v` for vertical tab. Any byte can\n -+be written with 3-digit octal codes (e.g., `\\033`).\n +@@ Documentation/git-fast-import.txt: When a `` starts with a double quote (`\"`), it is a C-style quoted\n + string, where the complete filename is enclosed in a pair of double\n + quotes and escape sequences are used. Certain characters must be escaped\n + by preceding them with a backslash: `LF` is written as `\\n`, backslash\n +-as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n ++as `\\\\`, and double quote as `\\\"`. Some characters may optionally be\n ++written with escape sequences: `\\a` for bell, `\\b` for backspace, `\\f`\n ++for form feed, `\\n` for line feed, `\\r` for carriage return, `\\t` for\n ++horizontal tab, and `\\v` for vertical tab. Any byte can be written with\n ++3-digit octal codes (e.g., `\\033`). All filenames can be represented as\n + quoted strings.\n \n A `` must use UNIX-style directory separators (forward slash `/`)\n - and must be in canonical form. That is it must not:\n \n ## t/t9300-fast-import.sh ##\n @@ t/t9300-fast-import.sh: test_path_eol_success () {\n7: dc67464b6a = 7: 5b464f4b01 fast-import: forbid escaped NUL in paths\n8: 5e02d887bc = 8: 6eb66fce45 fast-import: make comments more precise\n-- \n2.44.0\n\n"},{"id":"492846","messageId":"9499f34aae0620a659c054a934bc1caf6700d40a.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 2/8] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:04Z","receivedAt":"2024-04-12T08:03:15Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Previously, one case would not write the path to the strbuf: when the\npath is unquoted and at the end of the string. It was essentially\ncopy-on-write. However, with the logic simplification of the previous\ncommit, this case was eliminated and the strbuf is always populated.\n\nDirectly use the strbufs now instead of an alias.\n\nSince this already changes all the lines that use the strbufs, rename\nthem from `uq` to be more descriptive. That they are unquoted is not\ntheir most important property, so name them after what they carry.\n\nAdditionally, `file_change_m` no longer needs to copy the path before\nreading inline data.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 64 ++++++++++++++++++-------------------------\n 1 file changed, 27 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex ce9231afe6..8f6312fbaf 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2307,7 +2307,7 @@ static void parse_path_space(struct strbuf *sb, const char *p,\n \n static void file_change_m(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2344,13 +2344,12 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n \n \t/* Git does not track empty, non-toplevel directories. */\n-\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n-\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n+\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n+\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n \t\treturn;\n \t}\n \n@@ -2371,10 +2370,6 @@ static void file_change_m(const char *p, struct branch *b)\n \t\tif (S_ISDIR(mode))\n \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n \t\t\t\tcommand_buf.buf);\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\twhile (read_next_command() != EOF) {\n \t\t\tconst char *v;\n \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n@@ -2400,55 +2395,51 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*p) {\n+\tif (!*path.buf) {\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n+\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *s, *d;\n-\tstatic struct strbuf s_uq = STRBUF_INIT;\n-\tstatic struct strbuf d_uq = STRBUF_INIT;\n+\tstatic struct strbuf source = STRBUF_INIT;\n+\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n \n-\tstrbuf_reset(&s_uq);\n-\tparse_path_space(&s_uq, p, &p, \"source\");\n-\ts = s_uq.buf;\n+\tstrbuf_reset(&source);\n+\tparse_path_space(&source, p, &p, \"source\");\n \n \tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\tstrbuf_reset(&d_uq);\n-\tparse_path_eol(&d_uq, p, \"dest\");\n-\td = d_uq.buf;\n+\tstrbuf_reset(&dest);\n+\tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n-\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n \telse\n-\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n-\t\tdie(\"Path %s not in branch\", s);\n-\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n+\t\tdie(\"Path %s not in branch\", source.buf);\n+\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, d,\n+\ttree_content_set(&b->branch_tree, dest.buf,\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n@@ -3176,7 +3167,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3193,10 +3184,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_get(root, p, &leaf, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\n \t * until it is saved. Save, for simplicity.\n@@ -3204,7 +3194,7 @@ static void parse_ls(const char *p, struct branch *b)\n \tif (S_ISDIR(leaf.versions[1].mode))\n \t\tstore_tree(&leaf);\n \n-\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n \tif (leaf.tree)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n-- \n2.44.0\n\n\n"},{"id":"492847","messageId":"9b1e6b80f50180d213284849db53fecfdb1ebf3c.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 3/8] fast-import: allow unquoted empty path for root","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:13Z","receivedAt":"2024-04-12T08:03:27Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Ever since filerename was added in f39a946a1f (Support wholesale\ndirectory renames in fast-import, 2007-07-09) and filecopy in b6f3481bb4\n(Teach fast-import to recursively copy files/directories, 2007-07-15),\nboth have produced an error when the destination path is empty. Later,\nwhen support for targeting the root directory with an empty string was\nadded in 2794ad5244 (fast-import: Allow filemodify to set the root,\n2010-10-10), this had the effect of allowing the quoted empty string\n(`\"\"`), but forbidding its unquoted variant (``). This seems to have\nbeen intended as simple data validation for parsing two paths, rather\nthan a syntax restriction, because it was not extended to the other\noperations.\n\nAll other occurrences of paths (in filemodify, filedelete, the source of\nfilecopy and filerename, and ls) allow both.\n\nFor most of this feature's lifetime, the documentation has not\nprescribed the use of quoted empty strings. In e5959106d6\n(Documentation/fast-import: put explanation of M 040000 \"\" in\ncontext, 2011-01-15), its documentation was changed from “`` may\nalso be an empty string (`\"\"`) to specify the root of the tree” to “The\nroot of the tree can be represented by an empty string as ``”.\n\nThus, we should assume that some front-ends have depended on this\nbehavior.\n\nRemove this restriction for the destination paths of filecopy and\nfilerename and change tests targeting the root to test `\"\"` and ``.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 3 -\n t/t9300-fast-import.sh | 363 +++++++++++++++++++++--------------------\n 2 files changed, 190 insertions(+), 176 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 8f6312fbaf..0da7e8a5a5 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2419,9 +2419,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n \tstrbuf_reset(&source);\n \tparse_path_space(&source, p, &p, \"source\");\n-\n-\tif (!*p)\n-\t\tdie(\"Missing dest: %s\", command_buf.buf);\n \tstrbuf_reset(&dest);\n \tparse_path_eol(&dest, p, \"dest\");\n \ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex de2f1304e8..13f98e6688 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1059,30 +1059,33 @@ test_expect_success 'M: rename subdirectory to new subdirectory' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'M: rename root to subdirectory' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/M4\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/M4\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n-\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n-\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n-\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n-\tEOF\n-\tgit fast-import actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n+\t\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n+\t\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n+\t\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n+\t\tEOF\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series N\n@@ -1259,49 +1262,52 @@ test_expect_success PIPE 'N: empty directory reads as missing' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'N: copy root directory by tree hash' '\n-\tcat >expect <<-EOF &&\n-\t:100755 000000 $newf $zero D\tfile3/newf\n-\t:100644 000000 $oldf $zero D\tfile3/oldf\n-\tEOF\n-\troot=$(git rev-parse refs/heads/branch^0^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N6\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 000000 $newf $zero D\tfile3/newf\n+\t\t:100644 000000 $oldf $zero D\tfile3/oldf\n+\t\tEOF\n+\t\troot_tree=$(git rev-parse refs/heads/branch^0^{tree}) &&\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N6\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tM 040000 $root_tree $root\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n \n-test_expect_success 'N: copy root by path' '\n-\tcat >expect <<-EOF &&\n-\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n-\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n-\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n-\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n-\tEOF\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N-copy-root-path\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n+\t\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n+\t\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n+\t\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n+\t\tEOF\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N-copy-root-path\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tC $root oldroot\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n test_expect_success 'N: delete directory by copying' '\n \tcat >expect <<-\\EOF &&\n@@ -1431,98 +1437,102 @@ test_expect_success 'N: reject foo/ syntax in ls argument' '\n \tINPUT_END\n '\n \n-test_expect_success 'N: copy to root by id and modify' '\n-\techo \"hello, world\" >expect.foo &&\n-\techo hello >expect.bar &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N7\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.foo &&\n+\t\techo hello >expect.bar &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N7\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.foo &&\n-\tgit show N8:foo/bar >actual.bar &&\n-\ttest_cmp expect.foo actual.foo &&\n-\ttest_cmp expect.bar actual.bar\n-'\n+\t\tM 040000 $tree $root\n+\t\tM 644 inline foo/foo\n+\t\tdata <actual.foo &&\n+\t\tgit show N8:foo/bar >actual.bar &&\n+\t\ttest_cmp expect.foo actual.foo &&\n+\t\ttest_cmp expect.bar actual.bar\n+\t'\n \n-test_expect_success 'N: extract subtree' '\n-\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N9\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/N9\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect.baz &&\n-\techo hello, world >expect.qux &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N10\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.baz &&\n+\t\techo hello, world >expect.qux &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N10\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.baz &&\n-\tgit show N11:bar/qux >actual.qux &&\n-\tgit show N11:bar/quux >actual.quux &&\n-\ttest_cmp expect.baz actual.baz &&\n-\ttest_cmp expect.qux actual.qux &&\n-\ttest_cmp expect.qux actual.quux'\n+\t\tM 040000 $tree $root\n+\t\tM 100644 inline foo/bar/qux\n+\t\tdata <actual.baz &&\n+\t\tgit show N11:bar/qux >actual.qux &&\n+\t\tgit show N11:bar/quux >actual.quux &&\n+\t\ttest_cmp expect.baz actual.baz &&\n+\t\ttest_cmp expect.qux actual.qux &&\n+\t\ttest_cmp expect.qux actual.quux\n+\t'\n+done\n \n ###\n ### series O\n@@ -3067,6 +3077,7 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n # There are two sorts of ways a path can be parsed, depending on whether it is\n # the last field on the line. Additionally, ls without a has a special\n # case. Test every occurrence of in the grammar against every error case.\n+# Paths for the root (empty strings) are tested elsewhere.\n #\n \n #\n@@ -3321,16 +3332,19 @@ test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n ###\n # Setup is carried over from series S.\n \n-test_expect_success 'T: ls root tree' '\n-\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n-\t040000 tree $(git rev-parse S^{tree})\tZ\n-\tEOF\n-\tsha1=$(git rev-parse --verify S) &&\n-\tgit fast-import --import-marks=marks <<-EOF >actual &&\n-\tls $sha1 \"\"\n-\tEOF\n-\ttest_cmp expect actual\n-'\n+for root in '\"\"' ''\n+do\n+\ttest_expect_success \"T: ls root ($root) tree\" '\n+\t\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n+\t\t040000 tree $(git rev-parse S^{tree})\tZ\n+\t\tEOF\n+\t\tsha1=$(git rev-parse --verify S) &&\n+\t\tgit fast-import --import-marks=marks <<-EOF >actual &&\n+\t\tls $sha1 $root\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n \n test_expect_success 'T: delete branch' '\n \tgit branch to-delete &&\n@@ -3432,30 +3446,33 @@ test_expect_success 'U: validate directory delete result' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'U: filedelete root succeeds' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/U\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/U-delete-root\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 000000 $f7id $ZERO_OID D\thello.c\n-\tEOF\n+\ttest_expect_success \"U: validate root ($root) delete result\" '\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 000000 $f7id $ZERO_OID D\thello.c\n+\t\tEOF\n \n-\tgit diff-tree -M -r U^1 U >actual &&\n+\t\tgit diff-tree -M -r U U-delete-root >actual &&\n \n-\tcompare_diff_raw expect actual\n-'\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series V (checkpoint)\n-- \n2.44.0\n\n\n"},{"id":"492848","messageId":"1a2b0dc6162c754fa63525c017801b2cf25fead3.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 4/8] fast-import: remove dead strbuf","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:22Z","receivedAt":"2024-04-12T08:03:37Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The strbuf in `note_change_n` is to copy the remainder of `p` before\npotentially invalidating it when reading the next line. However, `p` is\nnot used after that point. It has been unused since the function was\ncreated in a8dd2e7d2b (fast-import: Add support for importing commit\nnotes, 2009-10-09) and looks to be a fossil from adapting\n`file_change_m`. Remove it.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 0da7e8a5a5..7a398dc975 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2444,7 +2444,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct branch *s;\n \tstruct object_id oid, commit_oid;\n@@ -2509,10 +2508,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\tread_next_command();\n \t\tparse_and_store_blob(&last_blob, &oid, 0);\n \t} else if (oe) {\n-- \n2.44.0\n\n\n"},{"id":"492849","messageId":"fb0d870d530d320d5ef9825d2128b92f2e2ee478.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 5/8] fast-import: improve documentation for path quoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:30Z","receivedAt":"2024-04-12T08:03:40Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"It describes what characters cannot be in an unquoted path, but not\ntheir semantics. Reframe it as a definition of unquoted paths. From the\nperspective of the parser, whether it starts with `\"` is what defines\nwhether it will parse it as quoted or unquoted.\n\nThe restrictions on characters in unquoted paths (with starting-\", LF,\nand spaces) are explained in the quoted paragraph. Move it to the\nunquoted paragraph and reword.\n\nThe restriction that the source paths of filecopy and filerename cannot\ncontain SP is only stated in their respective sections. Restate it in\nthe section.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 26 ++++++++++++++++----------\n 1 file changed, 16 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex b2607366b9..1882758b8a 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -630,18 +630,24 @@ in octal. Git only supports the following modes:\n In both formats `` is the complete path of the file to be added\n (if not already existing) or modified (if already existing).\n \n-A `` string must use UNIX-style directory separators (forward\n-slash `/`), may contain any byte other than `LF`, and must not\n-start with double quote (`\"`).\n+A `` can be written as unquoted bytes or a C-style quoted string.\n \n-A path can use C-style string quoting; this is accepted in all cases\n-and mandatory if the filename starts with double quote or contains\n-`LF`. In C-style quoting, the complete name should be surrounded with\n-double quotes, and any `LF`, backslash, or double quote characters\n-must be escaped by preceding them with a backslash (e.g.,\n-`\"path/with\\n, \\\\ and \\\" in it\"`).\n+When a `` does not start with a double quote (`\"`), it is an\n+unquoted string and is parsed as literal bytes without any escape\n+sequences. However, if the filename contains `LF` or starts with double\n+quote, it cannot be represented as an unquoted string and must be\n+quoted. Additionally, the source `` in `filecopy` or `filerename`\n+must be quoted if it contains SP.\n \n-The value of `` must be in canonical form. That is it must not:\n+When a `` starts with a double quote (`\"`), it is a C-style quoted\n+string, where the complete filename is enclosed in a pair of double\n+quotes and escape sequences are used. Certain characters must be escaped\n+by preceding them with a backslash: `LF` is written as `\\n`, backslash\n+as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n+quoted strings.\n+\n+A `` must use UNIX-style directory separators (forward slash `/`)\n+and its value must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n-- \n2.44.0\n\n\n"},{"id":"492850","messageId":"4b6017ded8ec434fc6d7ac90964b59e40f6dd561.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 6/8] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:38Z","receivedAt":"2024-04-12T08:03:47Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Simply saying “C-style” string quoting is imprecise, as only a subset of\nC escapes are supported. Document the exact escapes.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 6 +++++-\n t/t9300-fast-import.sh | 10 ++++++----\n 2 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 1882758b8a..c6082c3b49 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -643,7 +643,11 @@ When a `` starts with a double quote (`\"`), it is a C-style quoted\n string, where the complete filename is enclosed in a pair of double\n quotes and escape sequences are used. Certain characters must be escaped\n by preceding them with a backslash: `LF` is written as `\\n`, backslash\n-as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n+as `\\\\`, and double quote as `\\\"`. Some characters may optionally be\n+written with escape sequences: `\\a` for bell, `\\b` for backspace, `\\f`\n+for form feed, `\\n` for line feed, `\\r` for carriage return, `\\t` for\n+horizontal tab, and `\\v` for vertical tab. Any byte can be written with\n+3-digit octal codes (e.g., `\\033`). All filenames can be represented as\n quoted strings.\n \n A `` must use UNIX-style directory separators (forward slash `/`)\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 13f98e6688..5cde8f8d01 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3189,8 +3189,9 @@ test_path_eol_success () {\n \t'\n }\n \n-test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Valid paths before a space: filecopy (source) and filerename (source).\n@@ -3256,8 +3257,9 @@ test_path_space_success () {\n \t'\n }\n \n-test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Test a single commit change with an invalid path. Run it with all occurrences\n-- \n2.44.0\n\n\n"},{"id":"492851","messageId":"5b464f4b014760d0d9ca3413dacaa7e48f22291f.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 7/8] fast-import: forbid escaped NUL in paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:47Z","receivedAt":"2024-04-12T08:04:00Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"NUL cannot appear in paths. Even disregarding filesystem path\nlimitations, the tree object format delimits with NUL, so such a path\ncannot be encoded by Git.\n\nWhen a quoted path is unquoted, it could possibly contain NUL from\n\"\\000\". Forbid it so it isn't truncated.\n\nfast-import still has other issues with NUL, but those will be addressed\nlater.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 1 +\n builtin/fast-import.c | 2 ++\n t/t9300-fast-import.sh | 1 +\n 3 files changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex c6082c3b49..8b6dde45f1 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -661,6 +661,7 @@ and its value must be in canonical form. That is it must not:\n \n The root of the tree can be represented by an empty string as ``.\n \n+`` cannot contain NUL, either literally or escaped as `\\000`.\n It is recommended that `` always be encoded using UTF-8.\n \n `filedelete`\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 7a398dc975..98096b6fa7 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2269,6 +2269,8 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t\tif (strlen(sb->buf) != sb->len)\n+\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n \t} else {\n \t\tif (include_spaces)\n \t\t\t*endp = p + strlen(p);\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 5cde8f8d01..1e68426852 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3300,6 +3300,7 @@ test_path_base_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n \ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n \ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n+\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n-- \n2.44.0\n\n\n"},{"id":"492852","messageId":"6eb66fce45f16a87fa01e3e38d85c9e32131b0d4.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 8/8] fast-import: make comments more precise","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:03:55Z","receivedAt":"2024-04-12T08:04:04Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The former is somewhat imprecise. The latter became out of sync with the\nbehavior in e814c39c2f (fast-import: refactor parsing of spaces,\n2014-06-18).\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 98096b6fa7..fd23a00150 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2210,7 +2210,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n *\n * idnum ::= ':' bigint;\n *\n- * Return the first character after the value in *endptr.\n+ * Update *endptr to point to the first character after the value.\n *\n * Complain if the following character is not what is expected,\n * either a space or end of the string.\n@@ -2243,8 +2243,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n }\n \n /*\n- * Parse the mark reference, demanding a trailing space. Return a\n- * pointer to the space.\n+ * Parse the mark reference, demanding a trailing space. Update *p to\n+ * point to the first character after the space.\n */\n static uintmax_t parse_mark_ref_space(const char **p)\n {\n-- \n2.44.0\n\n\n"},{"id":"492853","messageId":"d6ea8aca465eaed4b18b546b5d484bc98fdaa377.1712907684.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v4 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-12T08:02:56Z","receivedAt":"2024-04-12T08:11:56Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Path parsing in fast-import is inconsistent and many unquoting errors\nare suppressed or not checked.\n\n appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n filecopy ::= 'C' SP SP LF\n filerename ::= 'R' SP SP LF\n ls ::= 'ls' SP SP LF\n ls-commit ::= 'ls' SP LF\n\nand fast-import.c parses them in five different ways:\n\n1. For filemodify and filedelete:\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes to the end of\n the line (including any number of SP).\n2. For filecopy (source) and filerename (source):\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes up to, but not\n including, the next SP.\n3. For filecopy (dest) and filerename (dest):\n Like 1., but an unquoted empty string is forbidden.\n4. For ls:\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n5. For ls-commit:\n Unquote and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\nIn the first three, any errors from trying to unquote a string are\nsuppressed, so a quoted string that contains invalid escapes would be\ninterpreted as literal bytes. For example, `\"\\xff\"` would fail to\nunquote (because hex escapes are not supported), and it would instead be\ninterpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\ncertainly not intended. Some front-ends erroneously use their language's\nstandard quoting routine instead of matching Git's, which could silently\nintroduce escapes that would be incorrectly parsed due to this and lead\nto data corruption.\n\nThe documentation states “To use a source path that contains SP the path\nmust be quoted.”, so it is expected that some implementations depend on\nspaces being allowed in paths in the final position. Thus we have two\ndocumented ways to parse paths, so simplify the implementation to that.\n\nNow we have:\n\n1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n\n2. `parse_path_space` for filecopy (source) and filerename (source):\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes up to, but not including, the\n next SP. It must be followed by SP.\n\nThere remain two special cases: The dest in filecopy and rename\ncannot be an unquoted empty string (this will be addressed subsequently)\nand in ls-commit must be quoted to disambiguate it from ls.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 104 ++++++++++-------\n t/t9300-fast-import.sh | 258 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 318 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 782bda007c..ce9231afe6 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2258,10 +2258,56 @@ static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n \n+/*\n+ * Parse the path string into the strbuf. It may be quoted with escape sequences\n+ * or unquoted without escape sequences. When unquoted, it may only contain a\n+ * space if `include_spaces` is nonzero.\n+ */\n+static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n+\t\tint include_spaces, const char *field)\n+{\n+\tif (*p == '\"') {\n+\t\tif (unquote_c_style(sb, p, endp))\n+\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t} else {\n+\t\tif (include_spaces)\n+\t\t\t*endp = p + strlen(p);\n+\t\telse\n+\t\t\t*endp = strchrnul(p, ' ');\n+\t\tstrbuf_add(sb, p, *endp - p);\n+\t}\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and complain if this is not the end of\n+ * the string. It may contain spaces even when unquoted.\n+ */\n+static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n+{\n+\tconst char *end;\n+\n+\tparse_path(sb, p, &end, 1, field);\n+\tif (*end)\n+\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and ensure it is followed by a space.\n+ * It may not contain spaces when unquoted. Update *endp to point to the first\n+ * character after the space.\n+ */\n+static void parse_path_space(struct strbuf *sb, const char *p,\n+\t\tconst char **endp, const char *field)\n+{\n+\tparse_path(sb, p, endp, 0, field);\n+\tif (**endp != ' ')\n+\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n+\t(*endp)++;\n+}\n+\n static void file_change_m(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2299,11 +2345,8 @@ static void file_change_m(const char *p, struct branch *b)\n \t}\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n@@ -2367,48 +2410,29 @@ static void file_change_m(const char *p, struct branch *b)\n static void file_change_d(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n }\n \n-static void file_change_cr(const char *s, struct branch *b, int rename)\n+static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *d;\n+\tconst char *s, *d;\n \tstatic struct strbuf s_uq = STRBUF_INIT;\n \tstatic struct strbuf d_uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&s_uq);\n-\tif (!unquote_c_style(&s_uq, s, &endp)) {\n-\t\tif (*endp != ' ')\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t} else {\n-\t\tendp = strchr(s, ' ');\n-\t\tif (!endp)\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t\tstrbuf_add(&s_uq, s, endp - s);\n-\t}\n+\tparse_path_space(&s_uq, p, &p, \"source\");\n \ts = s_uq.buf;\n \n-\tendp++;\n-\tif (!*endp)\n+\tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\n-\td = endp;\n \tstrbuf_reset(&d_uq);\n-\tif (!unquote_c_style(&d_uq, d, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n-\t\td = d_uq.buf;\n-\t}\n+\tparse_path_eol(&d_uq, p, \"dest\");\n+\td = d_uq.buf;\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n@@ -3152,6 +3176,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n+\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3168,16 +3193,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tif (*p == '\"') {\n-\t\tstatic struct strbuf uq = STRBUF_INIT;\n-\t\tconst char *endp;\n-\t\tstrbuf_reset(&uq);\n-\t\tif (unquote_c_style(&uq, p, &endp))\n-\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tstrbuf_reset(&uq);\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 60e30fed3c..de2f1304e8 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -2142,6 +2142,7 @@ test_expect_success 'Q: deny note on empty branch' '\n \tEOF\n \ttest_must_fail git fast-import has a special\n+# case. Test every occurrence of in the grammar against every error case.\n+#\n+\n+#\n+# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n+# filerename (dest), and ls.\n+#\n+# commit :301 from root -- modify hello.c (for setup)\n+# commit :302 from :301 -- modify $path\n+# commit :303 from :302 -- delete $path\n+# commit :304 from :301 -- copy hello.c $path\n+# commit :305 from :301 -- rename hello.c $path\n+# ls :305 $path\n+#\n+test_path_eol_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths at EOL with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_m.exp &&\n+\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n+\t\ttest_cmp tree_m.exp tree_m.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n+\t\tgit ls-tree $commit_d >tree_d.out &&\n+\t\ttest_cmp tree_d.exp tree_d.out &&\n+\n+\t\t(\n+\t\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" &&\n+\t\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n+\t\t) | sort >tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\ttest_cmp out tree_r.exp\n+\t'\n+}\n+\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+\n+#\n+# Valid paths before a space: filecopy (source) and filerename (source).\n+#\n+# commit :301 from root -- modify $path (for setup)\n+# commit :302 from :301 -- copy $path hello2.c\n+# commit :303 from :301 -- rename $path hello2.c\n+#\n+test_path_space_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths before space with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-space\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out\n+\t'\n+}\n+\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+\n+#\n+# Test a single commit change with an invalid path. Run it with all occurrences\n+# of in the grammar against all error kinds.\n+#\n+test_path_fail () {\n+\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n+\ttest_expect_success \"S: $change with $what must fail\" '\n+\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :1\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <, the must be quoted.\n+test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n+\n ###\n ### series T (ls)\n ###\n-- \n2.44.0\n\n\n"},{"id":"492877","messageId":"xmqqle5io7zf.fsf@gitster.g","threadId":"61174","inReplyTo":"d6ea8aca465eaed4b18b546b5d484bc98fdaa377.1712907684.git.thalia@archibald.dev","subject":"Re: [PATCH v4 1/8] fast-import: tighten path unquoting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-12T16:34:28Z","receivedAt":"2024-04-12T16:34:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n> index 782bda007c..ce9231afe6 100644\n> --- a/builtin/fast-import.c\n> +++ b/builtin/fast-import.c\n> @@ -2258,10 +2258,56 @@ static uintmax_t parse_mark_ref_space(const char **p)\n> \treturn mark;\n> }\n> \n> +/*\n> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n> + * or unquoted without escape sequences. When unquoted, it may only contain a\n> + * space if `include_spaces` is nonzero.\n> + */\n\nIt took me three reads to understand the last sentence. It would\nhave been easier if it were written as \"it may contain a space only\nif ...\". I'd also named it \"allow_unquoted_spaces\"---it is not like\nthis function includes extra spaces on top of whatever was given.\n\n> +static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n> +\t\tint include_spaces, const char *field)\n> +{\n> +\tif (*p == '\"') {\n> +\t\tif (unquote_c_style(sb, p, endp))\n> +\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n> +\t} else {\n> +\t\tif (include_spaces)\n> +\t\t\t*endp = p + strlen(p);\n> +\t\telse\n> +\t\t\t*endp = strchrnul(p, ' ');\n> +\t\tstrbuf_add(sb, p, *endp - p);\n> +\t}\n> +}\n\nA very straigtht-forward implementation. Makes sense.\n\n> +/*\n> + * Parse the path string into the strbuf, and complain if this is not the end of\n> + * the string. It may contain spaces even when unquoted.\n> + */\n> +static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n> +{\n> +\tconst char *end;\n> +\n> +\tparse_path(sb, p, &end, 1, field);\n> +\tif (*end)\n> +\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n> +}\n\nOK.\n\n> +/*\n> + * Parse the path string into the strbuf, and ensure it is followed by a space.\n> + * It may not contain spaces when unquoted. Update *endp to point to the first\n> + * character after the space.\n> + */\n> +static void parse_path_space(struct strbuf *sb, const char *p,\n> +\t\tconst char **endp, const char *field)\n> +{\n> +\tparse_path(sb, p, endp, 0, field);\n> +\tif (**endp != ' ')\n> +\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n> +\t(*endp)++;\n> +}\n\nOK.\n\nThe updated callers that use the above helper functions do read a\nlot more easily, while filling the gaps in the original\nimplementation. Very nicely done.\n\nThanks.\n\n"},{"id":"492899","messageId":"41A4023D-E468-4D63-9881-57392D06D852@archibald.dev","threadId":"61174","inReplyTo":"xmqqle5io7zf.fsf@gitster.g","subject":"Re: [PATCH v4 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-13T00:07:38Z","receivedAt":"2024-04-13T00:07:49Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"On Apr 12, 2024, at 09:34, Junio C Hamano wrote:\n> Thalia Archibald writes:\n>> diff --git a/builtin/fast-import.c b/builtin/fast-import.c\n>> index 782bda007c..ce9231afe6 100644\n>> --- a/builtin/fast-import.c\n>> +++ b/builtin/fast-import.c\n>> @@ -2258,10 +2258,56 @@ static uintmax_t parse_mark_ref_space(const char **p)\n>> return mark;\n>> }\n>> \n>> +/*\n>> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n>> + * or unquoted without escape sequences. When unquoted, it may only contain a\n>> + * space if `include_spaces` is nonzero.\n>> + */\n> \n> It took me three reads to understand the last sentence. It would\n> have been easier if it were written as \"it may contain a space only\n> if ...\". I'd also named it \"allow_unquoted_spaces\"---it is not like\n> this function includes extra spaces on top of whatever was given.\n\nPatrick commented on this earlier too:\n\n> On Mar 28, 2024, at 01:21, Patrick Steinhardt wrote:\n>> \n>> On Fri, Mar 22, 2024 at 12:03:18AM +0000, Thalia Archibald wrote:\n>>> +/*\n>>> + * Parse the path string into the strbuf. It may be quoted with escape sequences\n>>> + * or unquoted without escape sequences. When unquoted, it may only contain a\n>>> + * space if `allow_spaces` is nonzero.\n>>> + */\n>>> +static void parse_path(struct strbuf *sb, const char *p, const char **endp, int allow_spaces, const char *field)\n>>> +{\n>>> + strbuf_reset(sb);\n>>> + if (*p == '\"') {\n>>> + if (unquote_c_style(sb, p, endp))\n>>> + die(\"Invalid %s: %s\", field, command_buf.buf);\n>>> + } else {\n>>> + if (allow_spaces)\n>>> + *endp = p + strlen(p);\n>> \n>> I wonder whether `stop_at_unquoted_space` might be more telling. It's\n>> not like we disallow spaces here, it's that we treat them as the\n>> separator to the next field.\n> \n> I agree, but I’d rather something shorter, so I’ve changed it to `include_spaces`.\n\nWith all that in mind, I think Patrick is right that the best way to\nthink of this is that space functions as a field separator, conditional\non this flag. In practice, that leads to restrictions on whether you\ncan write paths that contain spaces without quotes.\n\nAs to naming, `allow_spaces` and `include_spaces` are problematic for\nthe reasons you both have pointed out. I think `stop_at_unquoted_space`\nis problematic, because that’s not where it stops when quoted, but\nrather at the close quote. I think that `include_unquoted_spaces` is\ngood, because it describes that spaces are included in this field when\nit is an unquoted string. `allow_unquoted_spaces` implies that its an\nerror to have a space, but no such error is raised here.\n\nHow’s this change? I’ve reworded the relevant sentence and specified any\n“it”s and replaced the “when unquoted, …” qualifier with “unquoted\nstrings may …” to reduce ambiguity.\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex fd23a00150..2070c78c56 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2259,12 +2259,13 @@ static uintmax_t parse_mark_ref_space(const char **p)\n}\n\n/*\n- * Parse the path string into the strbuf. It may be quoted with escape sequences\n- * or unquoted without escape sequences. When unquoted, it may only contain a\n- * space if `include_spaces` is nonzero.\n+ * Parse the path string into the strbuf. The path can either be quoted with\n+ * escape sequences or unquoted without escape sequences. Unquoted strings may\n+ * contain spaces only if `include_unquoted_spaces` is nonzero; otherwise, it\n+ * stops parsing at the first space.\n */\nstatic void parse_path(struct strbuf *sb, const char *p, const char **endp,\n-\t\tint include_spaces, const char *field)\n+\t\tint include_unquoted_spaces, const char *field)\n{\n\tif (*p == '\"') {\n\t\tif (unquote_c_style(sb, p, endp))\n@@ -2272,7 +2273,7 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n\t\tif (strlen(sb->buf) != sb->len)\n\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n\t} else {\n-\t\tif (include_spaces)\n+\t\tif (include_unquoted_spaces)\n\t\t\t*endp = p + strlen(p);\n\t\telse\n\t\t\t*endp = strchrnul(p, ' ');\n@@ -2282,7 +2283,7 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n\n/*\n * Parse the path string into the strbuf, and complain if this is not the end of\n- * the string. It may contain spaces even when unquoted.\n+ * the string. Unquoted strings may contain spaces.\n */\nstatic void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n{\n@@ -2295,7 +2296,7 @@ static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n\n/*\n * Parse the path string into the strbuf, and ensure it is followed by a space.\n- * It may not contain spaces when unquoted. Update *endp to point to the first\n+ * Unquoted strings may not contain spaces. Update *endp to point to the first\n * character after the space.\n */\nstatic void parse_path_space(struct strbuf *sb, const char *p,\n\n\nThalia\n"},{"id":"492907","messageId":"xmqq34rpglk1.fsf@gitster.g","threadId":"61174","inReplyTo":"41A4023D-E468-4D63-9881-57392D06D852@archibald.dev","subject":"Re: [PATCH v4 1/8] fast-import: tighten path unquoting","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-13T18:33:02Z","receivedAt":"2024-04-13T18:33:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thalia Archibald writes:\n\n> As to naming, `allow_spaces` and `include_spaces` are problematic for\n> the reasons you both have pointed out. I think `stop_at_unquoted_space`\n> is problematic, because that’s not where it stops when quoted, but\n> rather at the close quote. I think that `include_unquoted_spaces` is\n> good, because it describes that spaces are included in this field when\n> it is an unquoted string. `allow_unquoted_spaces` implies that its an\n> error to have a space, but no such error is raised here.\n\nOK, so the bit tells the function if we are dealing with the last\nfield on the input line, because unquoted side needs to know when\nto stop reading the path.\n\n\tstatic void parse_path(... int is_last_field ...)\n\t{\n\t\tif (*p == '\"') {\n\t\t\t... handling of a quoted path is unchanged\n\t\t\t... regardless of where on the line it apears\n\t\t} else {\n\t\t\t/*\n\t\t\t * unless we know this is the last field,\n\t\t\t * an unquoted SP is the end of this field.\n\t\t\t */\n\t\t\t*endp = is_last_field \n ? p + strlen(p)\n\t\t\t : strchrnul(p, ' ');\n\t\t}\n\t}\n\nAnother way to look at it is that we are telling the function if a\nspace in an unquoted path is part of the path or not.\n\nThe distinction matters only if we require, in some record type, a\npath field that is the last on the line to be quoted when it has a\nSP in it, in which case, \"is_last_field\" is a wrong name, and we\nneed to say something like space_is_end_of_field_if_unquoted (or we\ncan reverse the polarity to say unquoted_space_is_part_of_the_path,\ninclude_unqouted_space, etc.). But if not, I find that \"we normally\nstop at SP when unquoted but the last field is a special case\"\nsomewhat more natural. I do not feel too strongly, though.\n\nThanks.\n\n\n"},{"id":"492909","messageId":"cover.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1712907684.git.thalia@archibald.dev","subject":"[PATCH v5 0/8] fast-import: tighten parsing of paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:11:32Z","receivedAt":"2024-04-14T01:11:39Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"> fast-import has subtle differences in how it parses file paths between each\n> occurrence of in the grammar. Many errors are suppressed or not checked,\n> which could lead to silent data corruption. A particularly bad case is when a\n> front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> supported), it would be treated as literal bytes instead of a quoted string.\n>\n> Bring path parsing into line with the documented behavior and improve\n> documentation to fill in missing details.\n\nChanges since v4:\n* Refine C comments and parameter name.\n\nThalia\n\n\nThalia Archibald (8):\n fast-import: tighten path unquoting\n fast-import: directly use strbufs for paths\n fast-import: allow unquoted empty path for root\n fast-import: remove dead strbuf\n fast-import: improve documentation for path quoting\n fast-import: document C-style escapes for paths\n fast-import: forbid escaped NUL in paths\n fast-import: make comments more precise\n\n Documentation/git-fast-import.txt | 31 +-\n builtin/fast-import.c | 162 ++++----\n t/t9300-fast-import.sh | 624 +++++++++++++++++++++---------\n 3 files changed, 555 insertions(+), 262 deletions(-)\n\nRange-diff against v4:\n1: d6ea8aca46 ! 1: 2c18fe5fe9 fast-import: tighten path unquoting\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n }\n \n +/*\n -+ * Parse the path string into the strbuf. It may be quoted with escape sequences\n -+ * or unquoted without escape sequences. When unquoted, it may only contain a\n -+ * space if `include_spaces` is nonzero.\n ++ * Parse the path string into the strbuf. The path can either be quoted with\n ++ * escape sequences or unquoted without escape sequences. Unquoted strings may\n ++ * contain spaces only if `is_last_field` is nonzero; otherwise, it stops\n ++ * parsing at the first space.\n + */\n +static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n -+\t\tint include_spaces, const char *field)\n ++\t\tint is_last_field, const char *field)\n +{\n +\tif (*p == '\"') {\n +\t\tif (unquote_c_style(sb, p, endp))\n +\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n +\t} else {\n -+\t\tif (include_spaces)\n -+\t\t\t*endp = p + strlen(p);\n -+\t\telse\n -+\t\t\t*endp = strchrnul(p, ' ');\n ++\t\t/*\n ++\t\t * Unless we are parsing the last field of a line,\n ++\t\t * SP is the end of this field.\n ++\t\t */\n ++\t\t*endp = is_last_field\n ++\t\t\t? p + strlen(p)\n ++\t\t\t: strchrnul(p, ' ');\n +\t\tstrbuf_add(sb, p, *endp - p);\n +\t}\n +}\n +\n +/*\n + * Parse the path string into the strbuf, and complain if this is not the end of\n -+ * the string. It may contain spaces even when unquoted.\n ++ * the string. Unquoted strings may contain spaces.\n + */\n +static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n +{\n @@ builtin/fast-import.c: static uintmax_t parse_mark_ref_space(const char **p)\n +\n +/*\n + * Parse the path string into the strbuf, and ensure it is followed by a space.\n -+ * It may not contain spaces when unquoted. Update *endp to point to the first\n ++ * Unquoted strings may not contain spaces. Update *endp to point to the first\n + * character after the space.\n + */\n +static void parse_path_space(struct strbuf *sb, const char *p,\n2: 9499f34aae = 2: 4e9f3aa52c fast-import: directly use strbufs for paths\n3: 9b1e6b80f5 = 3: cae5764cec fast-import: allow unquoted empty path for root\n4: 1a2b0dc616 = 4: 96ff70895a fast-import: remove dead strbuf\n5: fb0d870d53 = 5: e1a1b0395d fast-import: improve documentation for path quoting\n6: 4b6017ded8 = 6: 08e6fb37be fast-import: document C-style escapes for paths\n7: 5b464f4b01 = 7: a01d0a1b25 fast-import: forbid escaped NUL in paths\n8: 6eb66fce45 = 8: 65d7896e39 fast-import: make comments more precise\n-- \n2.44.0\n\n\n"},{"id":"492910","messageId":"4e9f3aa52cf1e5169d74142772e693c7b487c39e.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 2/8] fast-import: directly use strbufs for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:11:44Z","receivedAt":"2024-04-14T01:11:51Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Previously, one case would not write the path to the strbuf: when the\npath is unquoted and at the end of the string. It was essentially\ncopy-on-write. However, with the logic simplification of the previous\ncommit, this case was eliminated and the strbuf is always populated.\n\nDirectly use the strbufs now instead of an alias.\n\nSince this already changes all the lines that use the strbufs, rename\nthem from `uq` to be more descriptive. That they are unquoted is not\ntheir most important property, so name them after what they carry.\n\nAdditionally, `file_change_m` no longer needs to copy the path before\nreading inline data.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 64 ++++++++++++++++++-------------------------\n 1 file changed, 27 insertions(+), 37 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 8eba89689b..765429a66c 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2311,7 +2311,7 @@ static void parse_path_space(struct strbuf *sb, const char *p,\n \n static void file_change_m(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2348,13 +2348,12 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n \t}\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n \n \t/* Git does not track empty, non-toplevel directories. */\n-\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n-\t\ttree_content_remove(&b->branch_tree, p, NULL, 0);\n+\tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *path.buf) {\n+\t\ttree_content_remove(&b->branch_tree, path.buf, NULL, 0);\n \t\treturn;\n \t}\n \n@@ -2375,10 +2374,6 @@ static void file_change_m(const char *p, struct branch *b)\n \t\tif (S_ISDIR(mode))\n \t\t\tdie(\"Directories cannot be specified 'inline': %s\",\n \t\t\t\tcommand_buf.buf);\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\twhile (read_next_command() != EOF) {\n \t\t\tconst char *v;\n \t\t\tif (skip_prefix(command_buf.buf, \"cat-blob \", &v))\n@@ -2404,55 +2399,51 @@ static void file_change_m(const char *p, struct branch *b)\n \t\t\t\tcommand_buf.buf);\n \t}\n \n-\tif (!*p) {\n+\tif (!*path.buf) {\n \t\ttree_content_replace(&b->branch_tree, &oid, mode, NULL);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, p, &oid, mode, NULL);\n+\ttree_content_set(&b->branch_tree, path.buf, &oid, mode, NULL);\n }\n \n static void file_change_d(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_remove(&b->branch_tree, p, NULL, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_remove(&b->branch_tree, path.buf, NULL, 1);\n }\n \n static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *s, *d;\n-\tstatic struct strbuf s_uq = STRBUF_INIT;\n-\tstatic struct strbuf d_uq = STRBUF_INIT;\n+\tstatic struct strbuf source = STRBUF_INIT;\n+\tstatic struct strbuf dest = STRBUF_INIT;\n \tstruct tree_entry leaf;\n \n-\tstrbuf_reset(&s_uq);\n-\tparse_path_space(&s_uq, p, &p, \"source\");\n-\ts = s_uq.buf;\n+\tstrbuf_reset(&source);\n+\tparse_path_space(&source, p, &p, \"source\");\n \n \tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\tstrbuf_reset(&d_uq);\n-\tparse_path_eol(&d_uq, p, \"dest\");\n-\td = d_uq.buf;\n+\tstrbuf_reset(&dest);\n+\tparse_path_eol(&dest, p, \"dest\");\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n-\t\ttree_content_remove(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_remove(&b->branch_tree, source.buf, &leaf, 1);\n \telse\n-\t\ttree_content_get(&b->branch_tree, s, &leaf, 1);\n+\t\ttree_content_get(&b->branch_tree, source.buf, &leaf, 1);\n \tif (!leaf.versions[1].mode)\n-\t\tdie(\"Path %s not in branch\", s);\n-\tif (!*d) {\t/* C \"path/to/subdir\" \"\" */\n+\t\tdie(\"Path %s not in branch\", source.buf);\n+\tif (!*dest.buf) {\t/* C \"path/to/subdir\" \"\" */\n \t\ttree_content_replace(&b->branch_tree,\n \t\t\t&leaf.versions[1].oid,\n \t\t\tleaf.versions[1].mode,\n \t\t\tleaf.tree);\n \t\treturn;\n \t}\n-\ttree_content_set(&b->branch_tree, d,\n+\ttree_content_set(&b->branch_tree, dest.buf,\n \t\t&leaf.versions[1].oid,\n \t\tleaf.versions[1].mode,\n \t\tleaf.tree);\n@@ -3180,7 +3171,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n+\tstatic struct strbuf path = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3197,10 +3188,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tstrbuf_reset(&uq);\n-\tparse_path_eol(&uq, p, \"path\");\n-\tp = uq.buf;\n-\ttree_content_get(root, p, &leaf, 1);\n+\tstrbuf_reset(&path);\n+\tparse_path_eol(&path, p, \"path\");\n+\ttree_content_get(root, path.buf, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\n \t * until it is saved. Save, for simplicity.\n@@ -3208,7 +3198,7 @@ static void parse_ls(const char *p, struct branch *b)\n \tif (S_ISDIR(leaf.versions[1].mode))\n \t\tstore_tree(&leaf);\n \n-\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, p);\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].oid.hash, path.buf);\n \tif (leaf.tree)\n \t\trelease_tree_content_recursive(leaf.tree);\n \tif (!b || root != &b->branch_tree)\n-- \n2.44.0\n\n\n"},{"id":"492911","messageId":"2c18fe5fe9b67a0650f72be7a7196644fe6e2131.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 1/8] fast-import: tighten path unquoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:11:37Z","receivedAt":"2024-04-14T01:11:52Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Path parsing in fast-import is inconsistent and many unquoting errors\nare suppressed or not checked.\n\n appears in the grammar in these places:\n\n filemodify ::= 'M' SP ( | 'inline') SP LF\n filedelete ::= 'D' SP LF\n filecopy ::= 'C' SP SP LF\n filerename ::= 'R' SP SP LF\n ls ::= 'ls' SP SP LF\n ls-commit ::= 'ls' SP LF\n\nand fast-import.c parses them in five different ways:\n\n1. For filemodify and filedelete:\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes to the end of\n the line (including any number of SP).\n2. For filecopy (source) and filerename (source):\n Try to unquote . If it unquotes without errors, use the\n unquoted version; otherwise, treat it as literal bytes up to, but not\n including, the next SP.\n3. For filecopy (dest) and filerename (dest):\n Like 1., but an unquoted empty string is forbidden.\n4. For ls:\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n5. For ls-commit:\n Unquote and report parse errors.\n (It must start with `\"` to disambiguate from ls.)\n\nIn the first three, any errors from trying to unquote a string are\nsuppressed, so a quoted string that contains invalid escapes would be\ninterpreted as literal bytes. For example, `\"\\xff\"` would fail to\nunquote (because hex escapes are not supported), and it would instead be\ninterpreted as the byte sequence '\"', '\\\\', 'x', 'f', 'f', '\"', which is\ncertainly not intended. Some front-ends erroneously use their language's\nstandard quoting routine instead of matching Git's, which could silently\nintroduce escapes that would be incorrectly parsed due to this and lead\nto data corruption.\n\nThe documentation states “To use a source path that contains SP the path\nmust be quoted.”, so it is expected that some implementations depend on\nspaces being allowed in paths in the final position. Thus we have two\ndocumented ways to parse paths, so simplify the implementation to that.\n\nNow we have:\n\n1. `parse_path_eol` for filemodify, filedelete, filecopy (dest),\n filerename (dest), ls, and ls-commit:\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes to the end of the line\n (including any number of SP).\n\n2. `parse_path_space` for filecopy (source) and filerename (source):\n\n If starts with `\"`, unquote it and report parse errors;\n otherwise, treat it as literal bytes up to, but not including, the\n next SP. It must be followed by SP.\n\nThere remain two special cases: The dest in filecopy and rename\ncannot be an unquoted empty string (this will be addressed subsequently)\nand in ls-commit must be quoted to disambiguate it from ls.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 108 ++++++++++-------\n t/t9300-fast-import.sh | 258 ++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 322 insertions(+), 44 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 782bda007c..8eba89689b 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2258,10 +2258,60 @@ static uintmax_t parse_mark_ref_space(const char **p)\n \treturn mark;\n }\n \n+/*\n+ * Parse the path string into the strbuf. The path can either be quoted with\n+ * escape sequences or unquoted without escape sequences. Unquoted strings may\n+ * contain spaces only if `is_last_field` is nonzero; otherwise, it stops\n+ * parsing at the first space.\n+ */\n+static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n+\t\tint is_last_field, const char *field)\n+{\n+\tif (*p == '\"') {\n+\t\tif (unquote_c_style(sb, p, endp))\n+\t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t} else {\n+\t\t/*\n+\t\t * Unless we are parsing the last field of a line,\n+\t\t * SP is the end of this field.\n+\t\t */\n+\t\t*endp = is_last_field\n+\t\t\t? p + strlen(p)\n+\t\t\t: strchrnul(p, ' ');\n+\t\tstrbuf_add(sb, p, *endp - p);\n+\t}\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and complain if this is not the end of\n+ * the string. Unquoted strings may contain spaces.\n+ */\n+static void parse_path_eol(struct strbuf *sb, const char *p, const char *field)\n+{\n+\tconst char *end;\n+\n+\tparse_path(sb, p, &end, 1, field);\n+\tif (*end)\n+\t\tdie(\"Garbage after %s: %s\", field, command_buf.buf);\n+}\n+\n+/*\n+ * Parse the path string into the strbuf, and ensure it is followed by a space.\n+ * Unquoted strings may not contain spaces. Update *endp to point to the first\n+ * character after the space.\n+ */\n+static void parse_path_space(struct strbuf *sb, const char *p,\n+\t\tconst char **endp, const char *field)\n+{\n+\tparse_path(sb, p, endp, 0, field);\n+\tif (**endp != ' ')\n+\t\tdie(\"Missing space after %s: %s\", field, command_buf.buf);\n+\t(*endp)++;\n+}\n+\n static void file_change_m(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct object_entry *oe;\n \tstruct object_id oid;\n \tuint16_t mode, inline_data = 0;\n@@ -2299,11 +2349,8 @@ static void file_change_m(const char *p, struct branch *b)\n \t}\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \n \t/* Git does not track empty, non-toplevel directories. */\n \tif (S_ISDIR(mode) && is_empty_tree_oid(&oid) && *p) {\n@@ -2367,48 +2414,29 @@ static void file_change_m(const char *p, struct branch *b)\n static void file_change_d(const char *p, struct branch *b)\n {\n \tstatic struct strbuf uq = STRBUF_INIT;\n-\tconst char *endp;\n \n \tstrbuf_reset(&uq);\n-\tif (!unquote_c_style(&uq, p, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_remove(&b->branch_tree, p, NULL, 1);\n }\n \n-static void file_change_cr(const char *s, struct branch *b, int rename)\n+static void file_change_cr(const char *p, struct branch *b, int rename)\n {\n-\tconst char *d;\n+\tconst char *s, *d;\n \tstatic struct strbuf s_uq = STRBUF_INIT;\n \tstatic struct strbuf d_uq = STRBUF_INIT;\n-\tconst char *endp;\n \tstruct tree_entry leaf;\n \n \tstrbuf_reset(&s_uq);\n-\tif (!unquote_c_style(&s_uq, s, &endp)) {\n-\t\tif (*endp != ' ')\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t} else {\n-\t\tendp = strchr(s, ' ');\n-\t\tif (!endp)\n-\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n-\t\tstrbuf_add(&s_uq, s, endp - s);\n-\t}\n+\tparse_path_space(&s_uq, p, &p, \"source\");\n \ts = s_uq.buf;\n \n-\tendp++;\n-\tif (!*endp)\n+\tif (!*p)\n \t\tdie(\"Missing dest: %s\", command_buf.buf);\n-\n-\td = endp;\n \tstrbuf_reset(&d_uq);\n-\tif (!unquote_c_style(&d_uq, d, &endp)) {\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after dest in: %s\", command_buf.buf);\n-\t\td = d_uq.buf;\n-\t}\n+\tparse_path_eol(&d_uq, p, \"dest\");\n+\td = d_uq.buf;\n \n \tmemset(&leaf, 0, sizeof(leaf));\n \tif (rename)\n@@ -3152,6 +3180,7 @@ static void print_ls(int mode, const unsigned char *hash, const char *path)\n \n static void parse_ls(const char *p, struct branch *b)\n {\n+\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct tree_entry *root = NULL;\n \tstruct tree_entry leaf = {NULL};\n \n@@ -3168,16 +3197,9 @@ static void parse_ls(const char *p, struct branch *b)\n \t\t\troot->versions[1].mode = S_IFDIR;\n \t\tload_tree(root);\n \t}\n-\tif (*p == '\"') {\n-\t\tstatic struct strbuf uq = STRBUF_INIT;\n-\t\tconst char *endp;\n-\t\tstrbuf_reset(&uq);\n-\t\tif (unquote_c_style(&uq, p, &endp))\n-\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n-\t\tif (*endp)\n-\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n-\t\tp = uq.buf;\n-\t}\n+\tstrbuf_reset(&uq);\n+\tparse_path_eol(&uq, p, \"path\");\n+\tp = uq.buf;\n \ttree_content_get(root, p, &leaf, 1);\n \t/*\n \t * A directory in preparation would have a sha1 of zero\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 60e30fed3c..de2f1304e8 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -2142,6 +2142,7 @@ test_expect_success 'Q: deny note on empty branch' '\n \tEOF\n \ttest_must_fail git fast-import has a special\n+# case. Test every occurrence of in the grammar against every error case.\n+#\n+\n+#\n+# Valid paths at the end of a line: filemodify, filedelete, filecopy (dest),\n+# filerename (dest), and ls.\n+#\n+# commit :301 from root -- modify hello.c (for setup)\n+# commit :302 from :301 -- modify $path\n+# commit :303 from :302 -- delete $path\n+# commit :304 from :301 -- copy hello.c $path\n+# commit :305 from :301 -- rename hello.c $path\n+# ls :305 $path\n+#\n+test_path_eol_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths at EOL with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-eol\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF >out 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_m.exp &&\n+\t\tgit ls-tree $commit_m | sort >tree_m.out &&\n+\t\ttest_cmp tree_m.exp tree_m.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\thello.c\\n\" >tree_d.exp &&\n+\t\tgit ls-tree $commit_d >tree_d.out &&\n+\t\ttest_cmp tree_d.exp tree_d.out &&\n+\n+\t\t(\n+\t\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" &&\n+\t\t\tprintf \"100644 blob $blob1\\thello.c\\n\"\n+\t\t) | sort >tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob1\\t$unquoted_path\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out &&\n+\n+\t\ttest_cmp out tree_r.exp\n+\t'\n+}\n+\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+\n+#\n+# Valid paths before a space: filecopy (source) and filerename (source).\n+#\n+# commit :301 from root -- modify $path (for setup)\n+# commit :302 from :301 -- copy $path hello2.c\n+# commit :303 from :301 -- rename $path hello2.c\n+#\n+test_path_space_success () {\n+\tlocal test=\"$1\" path=\"$2\" unquoted_path=\"$3\"\n+\ttest_expect_success \"S: paths before space with $test must work\" '\n+\t\ttest_when_finished \"git branch -D S-path-space\" &&\n+\n+\t\tgit fast-import --export-marks=marks.out <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :401\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <tree_c.exp &&\n+\t\tgit ls-tree $commit_c | sort >tree_c.out &&\n+\t\ttest_cmp tree_c.exp tree_c.out &&\n+\n+\t\tprintf \"100644 blob $blob\\thello2.c\\n\" >tree_r.exp &&\n+\t\tgit ls-tree $commit_r >tree_r.out &&\n+\t\ttest_cmp tree_r.exp tree_r.out\n+\t'\n+}\n+\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+\n+#\n+# Test a single commit change with an invalid path. Run it with all occurrences\n+# of in the grammar against all error kinds.\n+#\n+test_path_fail () {\n+\tlocal change=\"$1\" what=\"$2\" prefix=\"$3\" path=\"$4\" suffix=\"$5\" err_grep=\"$6\"\n+\ttest_expect_success \"S: $change with $what must fail\" '\n+\t\ttest_must_fail git fast-import <<-EOF 2>err &&\n+\t\tblob\n+\t\tmark :1\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <, the must be quoted.\n+test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n+\n ###\n ### series T (ls)\n ###\n-- \n2.44.0\n\n\n"},{"id":"492912","messageId":"cae5764cec349d295035f2b910a5a00f12cb7908.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 3/8] fast-import: allow unquoted empty path for root","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:11:52Z","receivedAt":"2024-04-14T01:12:06Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Ever since filerename was added in f39a946a1f (Support wholesale\ndirectory renames in fast-import, 2007-07-09) and filecopy in b6f3481bb4\n(Teach fast-import to recursively copy files/directories, 2007-07-15),\nboth have produced an error when the destination path is empty. Later,\nwhen support for targeting the root directory with an empty string was\nadded in 2794ad5244 (fast-import: Allow filemodify to set the root,\n2010-10-10), this had the effect of allowing the quoted empty string\n(`\"\"`), but forbidding its unquoted variant (``). This seems to have\nbeen intended as simple data validation for parsing two paths, rather\nthan a syntax restriction, because it was not extended to the other\noperations.\n\nAll other occurrences of paths (in filemodify, filedelete, the source of\nfilecopy and filerename, and ls) allow both.\n\nFor most of this feature's lifetime, the documentation has not\nprescribed the use of quoted empty strings. In e5959106d6\n(Documentation/fast-import: put explanation of M 040000 \"\" in\ncontext, 2011-01-15), its documentation was changed from “`` may\nalso be an empty string (`\"\"`) to specify the root of the tree” to “The\nroot of the tree can be represented by an empty string as ``”.\n\nThus, we should assume that some front-ends have depended on this\nbehavior.\n\nRemove this restriction for the destination paths of filecopy and\nfilerename and change tests targeting the root to test `\"\"` and ``.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 3 -\n t/t9300-fast-import.sh | 363 +++++++++++++++++++++--------------------\n 2 files changed, 190 insertions(+), 176 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 765429a66c..c8a1e3ef3d 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2423,9 +2423,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n \tstrbuf_reset(&source);\n \tparse_path_space(&source, p, &p, \"source\");\n-\n-\tif (!*p)\n-\t\tdie(\"Missing dest: %s\", command_buf.buf);\n \tstrbuf_reset(&dest);\n \tparse_path_eol(&dest, p, \"dest\");\n \ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex de2f1304e8..13f98e6688 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1059,30 +1059,33 @@ test_expect_success 'M: rename subdirectory to new subdirectory' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'M: rename root to subdirectory' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/M4\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/M4\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n-\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n-\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n-\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n-\tEOF\n-\tgit fast-import actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 100644 $oldf $oldf R100\tfile2/oldf\tsub/file2/oldf\n+\t\t:100755 100755 $f4id $f4id R100\tfile4\tsub/file4\n+\t\t:100755 100755 $newf $newf R100\ti/am/new/to/you\tsub/i/am/new/to/you\n+\t\t:100755 100755 $f6id $f6id R100\tnewdir/exec.sh\tsub/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id R100\tnewdir/interesting\tsub/newdir/interesting\n+\t\tEOF\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series N\n@@ -1259,49 +1262,52 @@ test_expect_success PIPE 'N: empty directory reads as missing' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'N: copy root directory by tree hash' '\n-\tcat >expect <<-EOF &&\n-\t:100755 000000 $newf $zero D\tfile3/newf\n-\t:100644 000000 $oldf $zero D\tfile3/oldf\n-\tEOF\n-\troot=$(git rev-parse refs/heads/branch^0^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N6\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 000000 $newf $zero D\tfile3/newf\n+\t\t:100644 000000 $oldf $zero D\tfile3/oldf\n+\t\tEOF\n+\t\troot_tree=$(git rev-parse refs/heads/branch^0^{tree}) &&\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N6\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tM 040000 $root_tree $root\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n \n-test_expect_success 'N: copy root by path' '\n-\tcat >expect <<-EOF &&\n-\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n-\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n-\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n-\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n-\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n-\tEOF\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N-copy-root-path\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect <<-EOF &&\n+\t\t:100755 100755 $newf $newf C100\tfile2/newf\toldroot/file2/newf\n+\t\t:100644 100644 $oldf $oldf C100\tfile2/oldf\toldroot/file2/oldf\n+\t\t:100755 100755 $f4id $f4id C100\tfile4\toldroot/file4\n+\t\t:100755 100755 $f6id $f6id C100\tnewdir/exec.sh\toldroot/newdir/exec.sh\n+\t\t:100644 100644 $f5id $f5id C100\tnewdir/interesting\toldroot/newdir/interesting\n+\t\tEOF\n+\t\tcat >input <<-INPUT_END &&\n+\t\tcommit refs/heads/N-copy-root-path\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <actual &&\n-\tcompare_diff_raw expect actual\n-'\n+\t\tfrom refs/heads/branch^0\n+\t\tC $root oldroot\n+\t\tINPUT_END\n+\t\tgit fast-import actual &&\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n test_expect_success 'N: delete directory by copying' '\n \tcat >expect <<-\\EOF &&\n@@ -1431,98 +1437,102 @@ test_expect_success 'N: reject foo/ syntax in ls argument' '\n \tINPUT_END\n '\n \n-test_expect_success 'N: copy to root by id and modify' '\n-\techo \"hello, world\" >expect.foo &&\n-\techo hello >expect.bar &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N7\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.foo &&\n+\t\techo hello >expect.bar &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N7\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.foo &&\n-\tgit show N8:foo/bar >actual.bar &&\n-\ttest_cmp expect.foo actual.foo &&\n-\ttest_cmp expect.bar actual.bar\n-'\n+\t\tM 040000 $tree $root\n+\t\tM 644 inline foo/foo\n+\t\tdata <actual.foo &&\n+\t\tgit show N8:foo/bar >actual.bar &&\n+\t\ttest_cmp expect.foo actual.foo &&\n+\t\ttest_cmp expect.bar actual.bar\n+\t'\n \n-test_expect_success 'N: extract subtree' '\n-\tbranch=$(git rev-parse --verify refs/heads/branch^{tree}) &&\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/N9\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/N9\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect.baz &&\n-\techo hello, world >expect.qux &&\n-\tgit fast-import <<-SETUP_END &&\n-\tcommit refs/heads/N10\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <expect.baz &&\n+\t\techo hello, world >expect.qux &&\n+\t\tgit fast-import <<-SETUP_END &&\n+\t\tcommit refs/heads/N10\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata < $GIT_COMMITTER_DATE\n-\tdata < $GIT_COMMITTER_DATE\n+\t\tdata <actual.baz &&\n-\tgit show N11:bar/qux >actual.qux &&\n-\tgit show N11:bar/quux >actual.quux &&\n-\ttest_cmp expect.baz actual.baz &&\n-\ttest_cmp expect.qux actual.qux &&\n-\ttest_cmp expect.qux actual.quux'\n+\t\tM 040000 $tree $root\n+\t\tM 100644 inline foo/bar/qux\n+\t\tdata <actual.baz &&\n+\t\tgit show N11:bar/qux >actual.qux &&\n+\t\tgit show N11:bar/quux >actual.quux &&\n+\t\ttest_cmp expect.baz actual.baz &&\n+\t\ttest_cmp expect.qux actual.qux &&\n+\t\ttest_cmp expect.qux actual.quux\n+\t'\n+done\n \n ###\n ### series O\n@@ -3067,6 +3077,7 @@ test_expect_success 'S: ls with garbage after sha1 must fail' '\n # There are two sorts of ways a path can be parsed, depending on whether it is\n # the last field on the line. Additionally, ls without a has a special\n # case. Test every occurrence of in the grammar against every error case.\n+# Paths for the root (empty strings) are tested elsewhere.\n #\n \n #\n@@ -3321,16 +3332,19 @@ test_path_eol_quoted_fail 'ls (without dataref in commit)' 'ls ' path\n ###\n # Setup is carried over from series S.\n \n-test_expect_success 'T: ls root tree' '\n-\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n-\t040000 tree $(git rev-parse S^{tree})\tZ\n-\tEOF\n-\tsha1=$(git rev-parse --verify S) &&\n-\tgit fast-import --import-marks=marks <<-EOF >actual &&\n-\tls $sha1 \"\"\n-\tEOF\n-\ttest_cmp expect actual\n-'\n+for root in '\"\"' ''\n+do\n+\ttest_expect_success \"T: ls root ($root) tree\" '\n+\t\tsed -e \"s/Z\\$//\" >expect <<-EOF &&\n+\t\t040000 tree $(git rev-parse S^{tree})\tZ\n+\t\tEOF\n+\t\tsha1=$(git rev-parse --verify S) &&\n+\t\tgit fast-import --import-marks=marks <<-EOF >actual &&\n+\t\tls $sha1 $root\n+\t\tEOF\n+\t\ttest_cmp expect actual\n+\t'\n+done\n \n test_expect_success 'T: delete branch' '\n \tgit branch to-delete &&\n@@ -3432,30 +3446,33 @@ test_expect_success 'U: validate directory delete result' '\n \tcompare_diff_raw expect actual\n '\n \n-test_expect_success 'U: filedelete root succeeds' '\n-\tcat >input <<-INPUT_END &&\n-\tcommit refs/heads/U\n-\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n-\tdata <input <<-INPUT_END &&\n+\t\tcommit refs/heads/U-delete-root\n+\t\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\t\tdata <expect <<-EOF &&\n-\t:100644 000000 $f7id $ZERO_OID D\thello.c\n-\tEOF\n+\ttest_expect_success \"U: validate root ($root) delete result\" '\n+\t\tcat >expect <<-EOF &&\n+\t\t:100644 000000 $f7id $ZERO_OID D\thello.c\n+\t\tEOF\n \n-\tgit diff-tree -M -r U^1 U >actual &&\n+\t\tgit diff-tree -M -r U U-delete-root >actual &&\n \n-\tcompare_diff_raw expect actual\n-'\n+\t\tcompare_diff_raw expect actual\n+\t'\n+done\n \n ###\n ### series V (checkpoint)\n-- \n2.44.0\n\n\n"},{"id":"492913","messageId":"96ff70895a272d3b2836058be8e189c9ff563908.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 4/8] fast-import: remove dead strbuf","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:11:59Z","receivedAt":"2024-04-14T01:12:10Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The strbuf in `note_change_n` is to copy the remainder of `p` before\npotentially invalidating it when reading the next line. However, `p` is\nnot used after that point. It has been unused since the function was\ncreated in a8dd2e7d2b (fast-import: Add support for importing commit\nnotes, 2009-10-09) and looks to be a fossil from adapting\n`file_change_m`. Remove it.\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex c8a1e3ef3d..832d0055f9 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2448,7 +2448,6 @@ static void file_change_cr(const char *p, struct branch *b, int rename)\n \n static void note_change_n(const char *p, struct branch *b, unsigned char *old_fanout)\n {\n-\tstatic struct strbuf uq = STRBUF_INIT;\n \tstruct object_entry *oe;\n \tstruct branch *s;\n \tstruct object_id oid, commit_oid;\n@@ -2513,10 +2512,6 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa\n \t\tdie(\"Invalid ref name or SHA1 expression: %s\", p);\n \n \tif (inline_data) {\n-\t\tif (p != uq.buf) {\n-\t\t\tstrbuf_addstr(&uq, p);\n-\t\t\tp = uq.buf;\n-\t\t}\n \t\tread_next_command();\n \t\tparse_and_store_blob(&last_blob, &oid, 0);\n \t} else if (oe) {\n-- \n2.44.0\n\n\n"},{"id":"492914","messageId":"e1a1b0395d0679f386b07c6dfeec4f481ccc7420.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 5/8] fast-import: improve documentation for path quoting","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:12:04Z","receivedAt":"2024-04-14T01:12:12Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"It describes what characters cannot be in an unquoted path, but not\ntheir semantics. Reframe it as a definition of unquoted paths. From the\nperspective of the parser, whether it starts with `\"` is what defines\nwhether it will parse it as quoted or unquoted.\n\nThe restrictions on characters in unquoted paths (with starting-\", LF,\nand spaces) are explained in the quoted paragraph. Move it to the\nunquoted paragraph and reword.\n\nThe restriction that the source paths of filecopy and filerename cannot\ncontain SP is only stated in their respective sections. Restate it in\nthe section.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 26 ++++++++++++++++----------\n 1 file changed, 16 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex b2607366b9..1882758b8a 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -630,18 +630,24 @@ in octal. Git only supports the following modes:\n In both formats `` is the complete path of the file to be added\n (if not already existing) or modified (if already existing).\n \n-A `` string must use UNIX-style directory separators (forward\n-slash `/`), may contain any byte other than `LF`, and must not\n-start with double quote (`\"`).\n+A `` can be written as unquoted bytes or a C-style quoted string.\n \n-A path can use C-style string quoting; this is accepted in all cases\n-and mandatory if the filename starts with double quote or contains\n-`LF`. In C-style quoting, the complete name should be surrounded with\n-double quotes, and any `LF`, backslash, or double quote characters\n-must be escaped by preceding them with a backslash (e.g.,\n-`\"path/with\\n, \\\\ and \\\" in it\"`).\n+When a `` does not start with a double quote (`\"`), it is an\n+unquoted string and is parsed as literal bytes without any escape\n+sequences. However, if the filename contains `LF` or starts with double\n+quote, it cannot be represented as an unquoted string and must be\n+quoted. Additionally, the source `` in `filecopy` or `filerename`\n+must be quoted if it contains SP.\n \n-The value of `` must be in canonical form. That is it must not:\n+When a `` starts with a double quote (`\"`), it is a C-style quoted\n+string, where the complete filename is enclosed in a pair of double\n+quotes and escape sequences are used. Certain characters must be escaped\n+by preceding them with a backslash: `LF` is written as `\\n`, backslash\n+as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n+quoted strings.\n+\n+A `` must use UNIX-style directory separators (forward slash `/`)\n+and its value must be in canonical form. That is it must not:\n \n * contain an empty directory component (e.g. `foo//bar` is invalid),\n * end with a directory separator (e.g. `foo/` is invalid),\n-- \n2.44.0\n\n\n"},{"id":"492915","messageId":"08e6fb37beb806f42ce66dc89f51c2d59865a4fa.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 6/8] fast-import: document C-style escapes for paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:12:12Z","receivedAt":"2024-04-14T01:12:23Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"Simply saying “C-style” string quoting is imprecise, as only a subset of\nC escapes are supported. Document the exact escapes.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 6 +++++-\n t/t9300-fast-import.sh | 10 ++++++----\n 2 files changed, 11 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 1882758b8a..c6082c3b49 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -643,7 +643,11 @@ When a `` starts with a double quote (`\"`), it is a C-style quoted\n string, where the complete filename is enclosed in a pair of double\n quotes and escape sequences are used. Certain characters must be escaped\n by preceding them with a backslash: `LF` is written as `\\n`, backslash\n-as `\\\\`, and double quote as `\\\"`. All filenames can be represented as\n+as `\\\\`, and double quote as `\\\"`. Some characters may optionally be\n+written with escape sequences: `\\a` for bell, `\\b` for backspace, `\\f`\n+for form feed, `\\n` for line feed, `\\r` for carriage return, `\\t` for\n+horizontal tab, and `\\v` for vertical tab. Any byte can be written with\n+3-digit octal codes (e.g., `\\033`). All filenames can be represented as\n quoted strings.\n \n A `` must use UNIX-style directory separators (forward slash `/`)\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 13f98e6688..5cde8f8d01 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3189,8 +3189,9 @@ test_path_eol_success () {\n \t'\n }\n \n-test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_eol_success 'unquoted spaces' ' hello world.c ' ' hello world.c '\n+test_path_eol_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Valid paths before a space: filecopy (source) and filerename (source).\n@@ -3256,8 +3257,9 @@ test_path_space_success () {\n \t'\n }\n \n-test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n-test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'quoted spaces' '\" hello world.c \"' ' hello world.c '\n+test_path_space_success 'no unquoted spaces' 'hello_world.c' 'hello_world.c'\n+test_path_space_success 'octal escapes' '\"\\150\\151\\056\\143\"' 'hi.c'\n \n #\n # Test a single commit change with an invalid path. Run it with all occurrences\n-- \n2.44.0\n\n\n"},{"id":"492916","messageId":"a01d0a1b25be19cb3b472a12a648ace5337648d4.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 7/8] fast-import: forbid escaped NUL in paths","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:12:19Z","receivedAt":"2024-04-14T01:12:26Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"NUL cannot appear in paths. Even disregarding filesystem path\nlimitations, the tree object format delimits with NUL, so such a path\ncannot be encoded by Git.\n\nWhen a quoted path is unquoted, it could possibly contain NUL from\n\"\\000\". Forbid it so it isn't truncated.\n\nfast-import still has other issues with NUL, but those will be addressed\nlater.\n\nSigned-off-by: Thalia Archibald \n---\n Documentation/git-fast-import.txt | 1 +\n builtin/fast-import.c | 2 ++\n t/t9300-fast-import.sh | 1 +\n 3 files changed, 4 insertions(+)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex c6082c3b49..8b6dde45f1 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -661,6 +661,7 @@ and its value must be in canonical form. That is it must not:\n \n The root of the tree can be represented by an empty string as ``.\n \n+`` cannot contain NUL, either literally or escaped as `\\000`.\n It is recommended that `` always be encoded using UTF-8.\n \n `filedelete`\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 832d0055f9..419ffdcdb5 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2270,6 +2270,8 @@ static void parse_path(struct strbuf *sb, const char *p, const char **endp,\n \tif (*p == '\"') {\n \t\tif (unquote_c_style(sb, p, endp))\n \t\t\tdie(\"Invalid %s: %s\", field, command_buf.buf);\n+\t\tif (strlen(sb->buf) != sb->len)\n+\t\t\tdie(\"NUL in %s: %s\", field, command_buf.buf);\n \t} else {\n \t\t/*\n \t\t * Unless we are parsing the last field of a line,\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 5cde8f8d01..1e68426852 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -3300,6 +3300,7 @@ test_path_base_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\" suffix=\"$4\"\n \ttest_path_fail \"$change\" 'unclosed \" in '\"$field\" \"$prefix\" '\"hello.c' \"$suffix\" \"Invalid $field\"\n \ttest_path_fail \"$change\" \"invalid escape in quoted $field\" \"$prefix\" '\"hello\\xff\"' \"$suffix\" \"Invalid $field\"\n+\ttest_path_fail \"$change\" \"escaped NUL in quoted $field\" \"$prefix\" '\"hello\\000\"' \"$suffix\" \"NUL in $field\"\n }\n test_path_eol_quoted_fail () {\n \tlocal change=\"$1\" prefix=\"$2\" field=\"$3\"\n-- \n2.44.0\n\n\n"},{"id":"492917","messageId":"65d7896e399a64670a71fc3fbde6e39dd0dc299d.1713056559.git.thalia@archibald.dev","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"[PATCH v5 8/8] fast-import: make comments more precise","fromName":"Thalia Archibald","fromEmail":"thalia@archibald.dev","sentAt":"2024-04-14T01:12:27Z","receivedAt":"2024-04-14T01:12:40Z","isPatch":true,"sender":{"key":"thalia@archibald.dev","avatar":"https://avatars.githubusercontent.com/u/8071784?v=4"},"body":"The former is somewhat imprecise. The latter became out of sync with the\nbehavior in e814c39c2f (fast-import: refactor parsing of spaces,\n2014-06-18).\n\nSigned-off-by: Thalia Archibald \n---\n builtin/fast-import.c | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 419ffdcdb5..dc5a9d32dd 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -2210,7 +2210,7 @@ static int parse_mapped_oid_hex(const char *hex, struct object_id *oid, const ch\n *\n * idnum ::= ':' bigint;\n *\n- * Return the first character after the value in *endptr.\n+ * Update *endptr to point to the first character after the value.\n *\n * Complain if the following character is not what is expected,\n * either a space or end of the string.\n@@ -2243,8 +2243,8 @@ static uintmax_t parse_mark_ref_eol(const char *p)\n }\n \n /*\n- * Parse the mark reference, demanding a trailing space. Return a\n- * pointer to the space.\n+ * Parse the mark reference, demanding a trailing space. Update *p to\n+ * point to the first character after the space.\n */\n static uintmax_t parse_mark_ref_space(const char **p)\n {\n-- \n2.44.0\n\n\n"},{"id":"492929","messageId":"ZhzR3QiuzfQ8LYgy@tanuki","threadId":"61174","inReplyTo":"cover.1713056559.git.thalia@archibald.dev","subject":"Re: [PATCH v5 0/8] fast-import: tighten parsing of paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-15T07:06:05Z","receivedAt":"2024-04-15T07:06:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Apr 14, 2024 at 01:11:32AM +0000, Thalia Archibald wrote:\n> > fast-import has subtle differences in how it parses file paths between each\n> > occurrence of in the grammar. Many errors are suppressed or not checked,\n> > which could lead to silent data corruption. A particularly bad case is when a\n> > front-end sent escapes that Git doesn't recognize (e.g., hex escapes are not\n> > supported), it would be treated as literal bytes instead of a quoted string.\n> >\n> > Bring path parsing into line with the documented behavior and improve\n> > documentation to fill in missing details.\n> \n> Changes since v4:\n> * Refine C comments and parameter name.\n> \n> Thalia\n\nI had another cursory read of this patch series that relied on the range\ndiffs for most of the part. In any way, this version looks good to me.\nThanks!\n\nPatrick\n"},{"id":"492951","messageId":"xmqqpluqpnav.fsf@gitster.g","threadId":"61174","inReplyTo":"ZhzR3QiuzfQ8LYgy@tanuki","subject":"Re: [PATCH v5 0/8] fast-import: tighten parsing of paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-15T17:07:20Z","receivedAt":"2024-04-15T17:10:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I had another cursory read of this patch series that relied on the range\n> diffs for most of the part. In any way, this version looks good to me.\n> Thanks!\n\nLikewise. Let's mark it for 'next'.\n"}]}