{"thread":{"id":"26574","subject":"[PATCH v4] fast-import: add 'ls' command","startedAt":"2011-02-22T09:04:31Z","lastAt":"2011-02-22T09:12:43Z","messageCount":2,"participants":["Jonathan Nieder"],"isPatch":true,"patchVersion":4,"patchTotal":null},"messages":[{"id":"161868","messageId":"20110222090431.GA24337@elie","threadId":"26574","inReplyTo":null,"subject":"[PATCH v4] fast-import: add 'ls' command","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-22T09:04:31Z","receivedAt":"2011-02-22T09:04:31Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"From: David Barr \nDate: Thu, 2 Dec 2010 21:40:20 +1100\n\nLazy fast-import frontend authors that want to rely on the backend to\nkeep track of the content of the imported trees _almost_ have what\nthey need in the 'cat-blob' command (v1.7.4-rc0~30^2~3, 2010-11-28).\nBut it is not quite enough, since\n\n (1) cat-blob can be used to retrieve the content of files, but\n not their mode, and\n\n (2) using cat-blob requires the frontend to keep track of a name\n (mark number or object id) for each blob to be retrieved\n\nIntroduce an 'ls' command to complement cat-blob and take care of the\nremaining needs. The 'ls' command finds what is at a given path\nwithin a given tree-ish (tag, commit, or tree):\n\n\t'ls' SP SP LF\n\nor in fast-import's active commit:\n\n\t'ls' SP LF\n\nThe response is a single line sent through the cat-blob channel,\nimitating ls-tree output. So for example:\n\n\tFE> ls :1 Documentation\n\tgfi> 040000 tree 9e6c2b599341d28a2a375f8207507e0a2a627fe9\tDocumentation\n\tFE> ls 9e6c2b599341d28a2a375f8207507e0a2a627fe9 git-fast-import.txt\n\tgfi> 100644 blob 4f92954396e3f0f97e75b6838a5635b583708870\tgit-fast-import.txt\n\tFE> ls :1 RelNotes\n\tgfi> 120000 blob b942e499449d97aeb50c73ca2bdc1c6e6d528743\tRelNotes\n\tFE> cat-blob b942e499449d97aeb50c73ca2bdc1c6e6d528743\n\tgfi> b942e499449d97aeb50c73ca2bdc1c6e6d528743 blob 32\n\tgfi> Documentation/RelNotes/1.7.4.txt\n\nThe most interesting parts of the reply are the first word, which is\na 6-digit octal mode (regular file, executable, symlink, directory,\nor submodule), and the part from the second space to the tab, which is\na that can be used in later cat-blob, ls, and filemodify (M)\ncommands to refer to the content (blob, tree, or commit) at that path.\n\nIf there is nothing there, the response is \"missing some/path\".\n\nThe intent is for this command to be used to read files from the\nactive commit, so a frontend can apply patches to them, and to copy\nfiles and directories from previous revisions.\n\nFor example, proposed updates to svn-fe use this command in place of\nits internal representation of the repository directory structure.\nThis simplifies the frontend a great deal and means support for\nresuming an import in a separate fast-import run (i.e., incremental\nimport) is basically free.\n\nSigned-off-by: David Barr \nSigned-off-by: Jonathan Nieder \nImproved-by: Junio C Hamano \nImproved-by: Sverre Rabbelier \n---\nHan-Wen Nienhuys wrote[1], on 2007-02-18:\n\n> BTW, I just had a brainwave\n>\n> Wouldn't it make sense to build a git-fast-export, which\n> can be used to interrogate a git-repository: a GIT tool\n> could run a single git command,\n>\n> git-fast-export\n>\n> and communicate with that process in a similar fashion, eg.\n\nI wrote[2], on 2010-06-30:\n\n> if we have the assumption that copies from old revs are not that common\n> then we can modify the tree that used to represent rN in place;\n> when there is a copy from an old rev, fetch from the git object db (or\n> wherever the objects are destined to go)\n\nIt's time.\n\nThanks for all your help. Thoughts, improvements welcome as always.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/39339/focus=40075\n[2] http://colabti.org/irclogger/irclogger_log/git-devel?date=2010-06-30#l972\n\n Documentation/git-fast-import.txt | 63 ++++++++++++++-\n fast-import.c | 162 ++++++++++++++++++++++++++++++++++++-\n t/t9300-fast-import.sh | 92 +++++++++++++++++++--\n 3 files changed, 303 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex c3a2766..e1b7a0f 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -196,7 +196,8 @@ especially when a higher level language such as Perl, Python or\n Ruby is being used.\n \n fast-import is very strict about its input. Where we say SP below we mean\n-*exactly* one space. Likewise LF means one (and only one) linefeed.\n+*exactly* one space. Likewise LF means one (and only one) linefeed\n+and HT one (and only one) horizontal tab.\n Supplying additional whitespace characters will cause unexpected\n results, such as branch names or file names with leading or trailing\n spaces in their name, or early termination of fast-import when it encounters\n@@ -334,6 +335,11 @@ and control the current import process. More detailed discussion\n \tformat to the file descriptor set with `--cat-blob-fd` or\n \t`stdout` if unspecified.\n \n+`ls`::\n+\tCauses fast-import to print a line describing a directory\n+\tentry in 'ls-tree' format to the file descriptor set with\n+\t`--cat-blob-fd` or `stdout` if unspecified.\n+\n `feature`::\n \tRequire that fast-import supports the specified feature, or\n \tabort if it does not.\n@@ -919,6 +925,55 @@ This command can be used anywhere in the stream that comments are\n accepted. In particular, the `cat-blob` command can be used in the\n middle of a commit but not in the middle of a `data` command.\n \n+`ls`\n+~~~~\n+Prints information about the object at a path to a file descriptor\n+previously arranged with the `--cat-blob-fd` argument. This allows\n+printing a blob from the active commit (with `cat-blob`) or copying a\n+blob or tree from a previous commit for use in the current one (with\n+`filemodify`).\n+\n+The `ls` command can be used anywhere in the stream that comments are\n+accepted, including the middle of a commit.\n+\n+Reading from the active commit::\n+\tThis form can only be used in the middle of a `commit`.\n+\tThe path names a directory entry within fast-import's\n+\tactive commit. The path must be quoted in this case.\n++\n+....\n+\t'ls' SP LF\n+....\n+\n+Reading from a named tree::\n+\tThe `` can be a mark reference (`:`) or the\n+\tfull 40-byte SHA-1 of a Git tag, commit, or tree object,\n+\tpreexisting or waiting to be written.\n+\tThe path is relative to the top level of the tree\n+\tnamed by ``.\n++\n+....\n+\t'ls' SP SP LF\n+....\n+\n+See `filemodify` above for a detailed description of ``.\n+\n+Output uses the same format as `git ls-tree {litdd} `:\n+\n+====\n+\t SP ('blob' | 'tree' | 'commit') SP HT LF\n+====\n+\n+The represents the blob, tree, or commit object at \n+and can be used in later 'cat-blob', 'filemodify', or 'ls' commands.\n+\n+If there is no file or subtree at that path, 'git fast-import' will\n+instead report\n+\n+====\n+\tmissing SP LF\n+====\n+\n `feature`\n ~~~~~~~~~\n Require that fast-import supports the specified feature, or abort if\n@@ -946,8 +1001,10 @@ import-marks::\n \tany \"feature import-marks\" command in the stream.\n \n cat-blob::\n-\tIgnored. Versions of fast-import not supporting the\n-\t\"cat-blob\" command will exit with a message indicating so.\n+ls::\n+\tRequire that the backend support the 'cat-blob' or 'ls' command.\n+\tVersions of fast-import not supporting the specified command\n+\twill exit with a message indicating so.\n \tThis lets the import error out early with a clear message,\n \trather than wasting time on the early part of an import\n \tbefore the unsupported command is detected.\ndiff --git a/fast-import.c b/fast-import.c\nindex 3886a1b..6c37b84 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -24,10 +24,12 @@ Format of STDIN stream:\n commit_msg\n ('from' sp committish lf)?\n ('merge' sp committish lf)*\n- file_change*\n+ (file_change | ls)*\n lf?;\n commit_msg ::= data;\n \n+ ls ::= 'ls' sp '\"' quoted(path) '\"' lf;\n+\n file_change ::= file_clr\n | file_del\n | file_rnm\n@@ -132,7 +134,7 @@ Format of STDIN stream:\n ts ::= # time since the epoch in seconds, ascii base10 notation;\n tz ::= # GIT style timezone;\n \n- # note: comments and cat requests may appear anywhere\n+ # note: comments, ls and cat requests may appear anywhere\n # in the input, except within a data command. Any form\n # of the data command always escapes the related input\n # from comment processing.\n@@ -141,7 +143,9 @@ Format of STDIN stream:\n # must be the first character on that line (an lf\n # preceded it).\n #\n+\n cat_blob ::= 'cat-blob' sp (hexsha1 | idnum) lf;\n+ ls_tree ::= 'ls' sp (hexsha1 | idnum) sp path_str lf;\n \n comment ::= '#' not_lf* lf;\n not_lf ::= # Any byte that is not ASCII newline (LF);\n@@ -374,6 +378,7 @@ static int cat_blob_fd = STDOUT_FILENO;\n \n static void parse_argv(void);\n static void parse_cat_blob(void);\n+static void parse_ls(struct branch *b);\n \n static void write_branch_report(FILE *rpt, struct branch *b)\n {\n@@ -2614,6 +2619,8 @@ static void parse_new_commit(void)\n \t\t\tnote_change_n(b, prev_fanout);\n \t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n \t\t\tfile_change_deleteall(b);\n+\t\telse if (!prefixcmp(command_buf.buf, \"ls \"))\n+\t\t\tparse_ls(b);\n \t\telse {\n \t\t\tunread_command_buf = 1;\n \t\t\tbreak;\n@@ -2837,6 +2844,153 @@ static void parse_cat_blob(void)\n \tcat_blob(oe, sha1);\n }\n \n+static struct object_entry *dereference(struct object_entry *oe,\n+\t\t\t\t\tunsigned char sha1[20])\n+{\n+\tunsigned long size;\n+\tvoid *buf = NULL;\n+\tif (!oe) {\n+\t\tenum object_type type = sha1_object_info(sha1, NULL);\n+\t\tif (type < 0)\n+\t\t\tdie(\"object not found: %s\", sha1_to_hex(sha1));\n+\t\t/* cache it! */\n+\t\toe = insert_object(sha1);\n+\t\toe->type = type;\n+\t\toe->pack_id = MAX_PACK_ID;\n+\t\toe->idx.offset = 1;\n+\t}\n+\tswitch (oe->type) {\n+\tcase OBJ_TREE:\t/* easy case. */\n+\t\treturn oe;\n+\tcase OBJ_COMMIT:\n+\tcase OBJ_TAG:\n+\t\tbreak;\n+\tdefault:\n+\t\tdie(\"Not a treeish: %s\", command_buf.buf);\n+\t}\n+\n+\tif (oe->pack_id != MAX_PACK_ID) {\t/* in a pack being written */\n+\t\tbuf = gfi_unpack_entry(oe, &size);\n+\t} else {\n+\t\tenum object_type unused;\n+\t\tbuf = read_sha1_file(sha1, &unused, &size);\n+\t}\n+\tif (!buf)\n+\t\tdie(\"Can't load object %s\", sha1_to_hex(sha1));\n+\n+\t/* Peel one layer. */\n+\tswitch (oe->type) {\n+\tcase OBJ_TAG:\n+\t\tif (size < 40 + strlen(\"object \") ||\n+\t\t get_sha1_hex(buf + strlen(\"object \"), sha1))\n+\t\t\tdie(\"Invalid SHA1 in tag: %s\", command_buf.buf);\n+\t\tbreak;\n+\tcase OBJ_COMMIT:\n+\t\tif (size < 40 + strlen(\"tree \") ||\n+\t\t get_sha1_hex(buf + strlen(\"tree \"), sha1))\n+\t\t\tdie(\"Invalid SHA1 in commit: %s\", command_buf.buf);\n+\t}\n+\n+\tfree(buf);\n+\treturn find_object(sha1);\n+}\n+\n+static struct object_entry *parse_treeish_dataref(const char **p)\n+{\n+\tunsigned char sha1[20];\n+\tstruct object_entry *e;\n+\n+\tif (**p == ':') {\t/* */\n+\t\tchar *endptr;\n+\t\te = find_mark(strtoumax(*p + 1, &endptr, 10));\n+\t\tif (endptr == *p + 1)\n+\t\t\tdie(\"Invalid mark: %s\", command_buf.buf);\n+\t\tif (!e)\n+\t\t\tdie(\"Unknown mark: %s\", command_buf.buf);\n+\t\t*p = endptr;\n+\t\thashcpy(sha1, e->idx.sha1);\n+\t} else {\t/* */\n+\t\tif (get_sha1_hex(*p, sha1))\n+\t\t\tdie(\"Invalid SHA1: %s\", command_buf.buf);\n+\t\te = find_object(sha1);\n+\t\t*p += 40;\n+\t}\n+\n+\twhile (!e || e->type != OBJ_TREE)\n+\t\te = dereference(e, sha1);\n+\treturn e;\n+}\n+\n+static void print_ls(int mode, const unsigned char *sha1, const char *path)\n+{\n+\tstatic struct strbuf line = STRBUF_INIT;\n+\n+\t/* See show_tree(). */\n+\tconst char *type =\n+\t\tS_ISGITLINK(mode) ? commit_type :\n+\t\tS_ISDIR(mode) ? tree_type :\n+\t\tblob_type;\n+\n+\tif (!mode) {\n+\t\t/* missing SP path LF */\n+\t\tstrbuf_reset(&line);\n+\t\tstrbuf_addstr(&line, \"missing \");\n+\t\tquote_c_style(path, &line, NULL, 0);\n+\t\tstrbuf_addch(&line, '\\n');\n+\t} else {\n+\t\t/* mode SP type SP object_name TAB path LF */\n+\t\tstrbuf_reset(&line);\n+\t\tstrbuf_addf(&line, \"%06o %s %s\\t\",\n+\t\t\t\tmode, type, sha1_to_hex(sha1));\n+\t\tquote_c_style(path, &line, NULL, 0);\n+\t\tstrbuf_addch(&line, '\\n');\n+\t}\n+\tcat_blob_write(line.buf, line.len);\n+}\n+\n+static void parse_ls(struct branch *b)\n+{\n+\tconst char *p;\n+\tstruct tree_entry *root = NULL;\n+\tstruct tree_entry leaf = {0};\n+\n+\t/* ls SP ( SP)? */\n+\tp = command_buf.buf + strlen(\"ls \");\n+\tif (*p == '\"') {\n+\t\tif (!b)\n+\t\t\tdie(\"Not in a commit: %s\", command_buf.buf);\n+\t\troot = &b->branch_tree;\n+\t} else {\n+\t\tstruct object_entry *e = parse_treeish_dataref(&p);\n+\t\troot = new_tree_entry();\n+\t\thashcpy(root->versions[1].sha1, e->idx.sha1);\n+\t\tload_tree(root);\n+\t\tif (*p++ != ' ')\n+\t\t\tdie(\"Missing space after tree-ish: %s\", command_buf.buf);\n+\t}\n+\tif (*p == '\"') {\n+\t\tstatic struct strbuf uq = STRBUF_INIT;\n+\t\tconst char *endp;\n+\t\tstrbuf_reset(&uq);\n+\t\tif (unquote_c_style(&uq, p, &endp))\n+\t\t\tdie(\"Invalid path: %s\", command_buf.buf);\n+\t\tif (*endp)\n+\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n+\t\tp = uq.buf;\n+\t}\n+\ttree_content_get(root, p, &leaf);\n+\t/*\n+\t * A directory in preparation would have a sha1 of zero\n+\t * until it is saved. Save, for simplicity.\n+\t */\n+\tif (S_ISDIR(leaf.versions[1].mode))\n+\t\tstore_tree(&leaf);\n+\n+\tprint_ls(leaf.versions[1].mode, leaf.versions[1].sha1, p);\n+\tif (!b || root != &b->branch_tree)\n+\t\trelease_tree_entry(root);\n+}\n+\n static void checkpoint(void)\n {\n \tcheckpoint_requested = 0;\n@@ -3001,7 +3155,7 @@ static int parse_one_feature(const char *feature, int from_stream)\n \t\trelative_marks_paths = 0;\n \t} else if (!prefixcmp(feature, \"force\")) {\n \t\tforce_update = 1;\n-\t} else if (!strcmp(feature, \"notes\")) {\n+\t} else if (!strcmp(feature, \"notes\") || !strcmp(feature, \"ls\")) {\n \t\t; /* do nothing; we have the feature */\n \t} else {\n \t\treturn 0;\n@@ -3142,6 +3296,8 @@ int main(int argc, const char **argv)\n \twhile (read_next_command() != EOF) {\n \t\tif (!strcmp(\"blob\", command_buf.buf))\n \t\t\tparse_new_blob();\n+\t\telse if (!prefixcmp(command_buf.buf, \"ls \"))\n+\t\t\tparse_ls(NULL);\n \t\telse if (!prefixcmp(command_buf.buf, \"commit \"))\n \t\t\tparse_new_commit();\n \t\telse if (!prefixcmp(command_buf.buf, \"tag \"))\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 52ac0e5..6b1ba6c 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -42,6 +42,14 @@ echo \"$@\"'\n \n >empty\n \n+test_expect_success 'setup: have pipes?' '\n+\trm -f frob &&\n+\tif mkfifo frob\n+\tthen\n+\t\ttest_set_prereq PIPE\n+\tfi\n+'\n+\n ###\n ### series A\n ###\n@@ -898,6 +906,77 @@ test_expect_success \\\n \t git diff-tree -C --find-copies-harder -r N4^ N4 >actual &&\n \t compare_diff_raw expect actual'\n \n+test_expect_success PIPE 'N: read and copy directory' '\n+\tcat >expect <<-\\EOF\n+\t:100755 100755 f1fb5da718392694d0076d677d6d0e364c79b0bc f1fb5da718392694d0076d677d6d0e364c79b0bc C100\tfile2/newf\tfile3/newf\n+\t:100644 100644 7123f7f44e39be127c5eb701e5968176ee9d78b1 7123f7f44e39be127c5eb701e5968176ee9d78b1 C100\tfile2/oldf\tfile3/oldf\n+\tEOF\n+\tgit update-ref -d refs/heads/N4 &&\n+\trm -f backflow &&\n+\tmkfifo backflow &&\n+\t(\n+\t\texec $GIT_COMMITTER_DATE\n+\t\tdata <backflow &&\n+\tgit diff-tree -C --find-copies-harder -r N4^ N4 >actual &&\n+\tcompare_diff_raw expect actual\n+'\n+\n+test_expect_success PIPE 'N: empty directory reads as missing' '\n+\tcat <<-\\EOF >expect &&\n+\tOBJNAME\n+\t:000000 100644 OBJNAME OBJNAME A\tunrelated\n+\tEOF\n+\techo \"missing src\" >expect.response &&\n+\tgit update-ref -d refs/heads/read-empty &&\n+\trm -f backflow &&\n+\tmkfifo backflow &&\n+\t(\n+\t\texec $GIT_COMMITTER_DATE\n+\t\tdata <response &&\n+\t\tcat <<-\\EOF\n+\t\tD dst1\n+\t\tD dst2\n+\t\tEOF\n+\t) |\n+\tgit fast-import --cat-blob-fd=3 3>backflow &&\n+\ttest_cmp expect.response response &&\n+\tgit rev-list read-empty |\n+\tgit diff-tree -r --root --stdin |\n+\tsed \"s/$_x40/OBJNAME/g\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success \\\n \t'N: copy root directory by tree hash' \\\n \t'cat >expect <<-\\EOF &&\n@@ -1861,6 +1940,11 @@ test_expect_success 'R: feature no-relative-marks should be honoured' '\n test_cmp marks.new non-relative.out\n '\n \n+test_expect_success 'R: feature ls supported' '\n+\techo \"feature ls\" |\n+\tgit fast-import\n+'\n+\n test_expect_success 'R: feature cat-blob supported' '\n \techo \"feature cat-blob\" |\n \tgit fast-import\n@@ -1986,14 +2070,6 @@ test_expect_success 'R: print two blobs to stdout' '\n \ttest_cmp expect actual\n '\n \n-test_expect_success 'setup: have pipes?' '\n-\trm -f frob &&\n-\tif mkfifo frob\n-\tthen\n-\t\ttest_set_prereq PIPE\n-\tfi\n-'\n-\n test_expect_success PIPE 'R: copy using cat-file' '\n \texpect_id=$(git hash-object big) &&\n \texpect_len=$(wc -c [Subject: [PATCH v4] fast-import: add 'ls' command]\n\nForgot to mention: v3 is at\nhttp://thread.gmane.org/gmane.comp.version-control.git/166575\nfor those who like context.\n\nSorry for the noise.\n"}]}