{"thread":{"id":"25425","subject":"[PATCH/RFC] Add support for subversion dump format v3","startedAt":"2010-10-12T13:40:38Z","lastAt":"2010-10-12T13:40:40Z","messageCount":3,"participants":["David Barr"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"153304","messageId":"1286890845-4911-1-git-send-email-david.barr@cordelta.com","threadId":"25425","inReplyTo":null,"subject":"[PATCH/RFC] Add support for subversion dump format v3","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-10-12T13:40:38Z","receivedAt":"2010-10-12T13:40:38Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"This series follows Jonathan Nieder's svn diff applier series.\n\nPatches 1 to 4 add the required infrastructure to fast-import.\nThe primary feature is the addition of the cat-file command to\nfast-import. This allows access to objects written to the\nthe current pack prior to a checkpoint and is critical to\nretrieving full-texts to drive the diff applier.\n\nPatch 5 adds the basic parsing necessary to process the v3 format.\n\nPatch 6 adds logic around decoding prop deltas.\n\nPatch 7 integrates svn-fe with svn-da to decode text deltas.\nIt was primarily authored by Jonathan but inspired by Ram and\ncompleted by myself.\nThis is probably the most sprawling of the series and likely the\nheaviest target of review.\n\n\n Documentation/git-fast-import.txt |   49 +++++++++\n contrib/svn-fe/svn-fe.txt         |    6 +-\n fast-import.c                     |  147 +++++++++++++++++++++++++-\n t/t9010-svn-fe.sh                 |    6 +-\n t/t9300-fast-import.sh            |  210 +++++++++++++++++++++++++++++++++++++\n test-svn-fe.c                     |    3 +-\n vcs-svn/fast_export.c             |  187 ++++++++++++++++++++++++++++++++-\n vcs-svn/fast_export.h             |   13 ++-\n vcs-svn/repo_tree.c               |   37 +++++++-\n vcs-svn/repo_tree.h               |    3 +\n vcs-svn/svndiff.c                 |   11 +--\n vcs-svn/svndiff.h                 |    3 +-\n vcs-svn/svndump.c                 |   99 ++++++++++++++++--\n 13 files changed, 739 insertions(+), 35 deletions(-)\n"},{"id":"153306","messageId":"1286890845-4911-2-git-send-email-david.barr@cordelta.com","threadId":"25425","inReplyTo":"1286890845-4911-1-git-send-email-david.barr@cordelta.com","subject":"[PATCH 1/7] Teach fast-import to print the id of each imported commit","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-10-12T13:40:39Z","receivedAt":"2010-10-12T13:40:39Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"From: Jonathan Nieder <jrnieder@gmail.com>\n\nFor the svn importer, it would be useful to build a map from\nsubversion revision numbers to git commits as the import takes place.\nThis is particularly relevant because the subversion api sometimes\nrepresents as \"copy this directory from this revision\", and the\nimporter needs to be able to access the corresponding trees.  So\n(optionally) print each commit id when the corresponding object is\nwritten.\n\nUnfortunately when each commit object is written, it is not yet\naccessible to the caller until a checkpoint has finished.  A later\npatch will teach fast-import to directly pass on the relevant data on\nrequest, using the same channel.\n\nCc: Shawn O. Pearce <spearce@spearce.org>\nCc: Sverre Rabbelier <srabbelier@gmail.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n Documentation/git-fast-import.txt |   13 +++++++++++++\n fast-import.c                     |   18 ++++++++++++++++++\n t/t9300-fast-import.sh            |   36 ++++++++++++++++++++++++++++++++++++\n 3 files changed, 67 insertions(+), 0 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex 966ba4f..e217635 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -92,6 +92,18 @@ OPTIONS\n \t--(no-)-relative-marks= with the --(import|export)-marks=\n \toptions.\n \n+--report-fd=<fd>::\n+\tPrint the 40-character object name for each commit to\n+\tthe specified file descriptor before writing it to the\n+\tpack.  This information may be useful if the importer\n+\tneeds to maintain a map from revisions in the source\n+\trepository to commit ids in the target repository\n+\tduring the import.\n++\n+The described objects are not necessarily accessible\n+using standard git plumbing tools until a little while\n+after the next checkpoint.\n+\n --export-pack-edges=<file>::\n \tAfter creating a packfile, print a line of data to\n \t<file> listing the filename of the packfile and the last\n@@ -896,6 +908,7 @@ The following features are currently supported:\n * date-format\n * import-marks\n * export-marks\n+* report-fd\n * relative-marks\n * no-relative-marks\n * force\ndiff --git a/fast-import.c b/fast-import.c\nindex 2317b0f..ef0cee7 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -361,6 +361,9 @@ static uintmax_t next_mark;\n static struct strbuf new_data = STRBUF_INIT;\n static int seen_data_command;\n \n+/* Where to report commits */\n+static int report_fd = -1;\n+\n static void parse_argv(void);\n \n static void write_branch_report(FILE *rpt, struct branch *b)\n@@ -2571,6 +2574,11 @@ static void parse_new_commit(void)\n \n \tif (!store_object(OBJ_COMMIT, &new_data, NULL, b->sha1, next_mark))\n \t\tb->pack_id = pack_id;\n+\tif (report_fd != -1) {\n+\t\tchar *buf = sha1_to_hex(b->sha1);\n+\t\tbuf[40] = '\\n';\n+\t\twrite_or_die(report_fd, buf, 41);\n+\t}\n \tb->last_commit = object_count_by_type[OBJ_COMMIT];\n }\n \n@@ -2755,6 +2763,14 @@ static void option_export_marks(const char *marks)\n \tsafe_create_leading_directories_const(export_marks_file);\n }\n \n+static void option_report_fd(const char *fd)\n+{\n+\tunsigned long n = strtoul(fd, NULL, 0);\n+\tif (n > (unsigned long) INT_MAX)\n+\t\tdie(\"--report-fd cannot exceed %d\", INT_MAX);\n+\treport_fd = (int) n;\n+}\n+\n static void option_export_pack_edges(const char *edges)\n {\n \tif (pack_edges)\n@@ -2808,6 +2824,8 @@ static int parse_one_feature(const char *feature, int from_stream)\n \t\toption_import_marks(feature + 13, from_stream);\n \t} else if (!prefixcmp(feature, \"export-marks=\")) {\n \t\toption_export_marks(feature + 13);\n+\t} else if (!prefixcmp(feature, \"report-fd=\")) {\n+\t\toption_report_fd(feature + strlen(\"report-fd=\"));\n \t} else if (!prefixcmp(feature, \"relative-marks\")) {\n \t\trelative_marks_paths = 1;\n \t} else if (!prefixcmp(feature, \"no-relative-marks\")) {\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 7c05920..4371308 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1611,6 +1611,42 @@ test_expect_success 'R: feature no-relative-marks should be honoured' '\n     test_cmp marks.new non-relative.out\n '\n \n+test_expect_success 'setup: have pipes?' '\n+\ttest_when_finished \"rm -f frob\" &&\n+\tif mkfifo frob\n+\tthen\n+\t\ttest_set_prereq PIPE\n+\tfi\n+'\n+\n+test_expect_success PIPE 'R: fast-import --report-fd' '\n+\tcat >frontend <<-\\FRONTEND_END &&\n+\t#!/bin/sh\n+\tcat <<EOF &&\n+\tcommit refs/heads/printed\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tdata <<COMMIT\n+\tto be printed\n+\tCOMMIT\n+\n+\tfrom refs/heads/master\n+\tD file3\n+\n+\tEOF\n+\n+\tread cid <&3 &&\n+\techo \"$cid\" >received\n+\tEOF\n+\tFRONTEND_END\n+\n+\tmkfifo commits &&\n+\ttest_when_finished \"rm -f commits\" &&\n+\tsh frontend 3<commits |\n+\tgit fast-import --report-fd=3 3>commits &&\n+\tgit rev-parse --verify printed >real &&\n+\ttest_cmp real received\n+'\n+\n cat >input << EOF\n option git quiet\n blob\n-- \n1.7.3.1\n"},{"id":"153305","messageId":"1286890845-4911-3-git-send-email-david.barr@cordelta.com","threadId":"25425","inReplyTo":"1286890845-4911-1-git-send-email-david.barr@cordelta.com","subject":"[PATCH 2/7] fast-import: Let importers retrieve the objects being written","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-10-12T13:40:40Z","receivedAt":"2010-10-12T13:40:40Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"From: Jonathan Nieder <jrnieder@gmail.com>\n\nAs the description of the \"progress\" option in git-fast-import.1\nhints, there is no convenient way to immediately access the objects\nwritten to a new repository through fast-import.  Until a checkpoint\nhas been started and finishes writing the pack index, any new blobs,\ntrees, and commits will not be accessible using standard git tools.\n\nSo introduce another way: a \"cat\" command introduced in the command\nstream requests for fast-import to print an object to the same\nreport-fd stream used to report commits being written.\n\nThe output uses the same format as \"git cat-file --batch\".\n\nLike cat-file --batch, this does not provide an option to dereference\nobjects to a type of the requestor's choosing.  Tags are presented\nas tags, commits as commits, and trees as trees.\n\nObjects can be specified by path within a tree as well, using a\n\n cat TREE \"PATH\"\n\nsyntax.  With this syntax, also, the tree can only be specified by\n:n marker or 40-digit tree id.\n\nCc: Shawn O. Pearce <spearce@spearce.org>\nCc: David Barr <david.barr@cordelta.com>\nCc: Ramkumar Ramachandra <artagnon@gmail.com>\nHelped-by: Sverre Rabbelier <srabbelier@gmail.com>\nSigned-off-by: Jonathan Nieder <jrnieder@gmail.com>\n---\n Documentation/git-fast-import.txt |   34 +++++++++++-\n fast-import.c                     |  108 +++++++++++++++++++++++++++++++++++++\n t/t9300-fast-import.sh            |  102 ++++++++++++++++++++++++++++++++++\n 3 files changed, 243 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt\nindex e217635..2cf48f5 100644\n--- a/Documentation/git-fast-import.txt\n+++ b/Documentation/git-fast-import.txt\n@@ -102,7 +102,8 @@ OPTIONS\n +\n The described objects are not necessarily accessible\n using standard git plumbing tools until a little while\n-after the next checkpoint.\n+after the next checkpoint.  To request access to the\n+objects before then, use `cat` lines in the command stream.\n \n --export-pack-edges=<file>::\n \tAfter creating a packfile, print a line of data to\n@@ -332,6 +333,10 @@ and control the current import process.  More detailed discussion\n \tstandard output.  This command is optional and is not needed\n \tto perform an import.\n \n+`cat`::\n+\tCauses fast-import to print an object in 'cat-file --batch'\n+\tformat to the file descriptor set with \"feature report-fd\".\n+\n `feature`::\n \tRequire that fast-import supports the specified feature, or\n \tabort if it does not.\n@@ -888,6 +893,33 @@ Placing a `progress` command immediately after a `checkpoint` will\n inform the reader when the `checkpoint` has been completed and it\n can safely access the refs that fast-import updated.\n \n+`cat`\n+~~~~~\n+Causes fast-import to print an object to a file descriptor\n+previously arranged with the `--report-fd` option.  The command\n+otherwise has no impact on the current import; its main purpose is to\n+retrieve objects that may be in fast-import's memory but not\n+accessible from the target repository a little quicker than by the\n+method suggested by the description of the `progress` option.\n+\n+....\n+\t'cat' SP <dataref> LF\n+....\n+\n+The `<dataref>` can be either a mark reference (`:<idnum>`)\n+set previously, or a full 40-byte SHA-1 of any Git object,\n+preexisting or ready to be written.\n+\n+If `<dataref>` refers to a tree object, it may be followed by\n+a path within that tree to retrieve a subtree or blob.\n+\n+....\n+\t'cat' SP <treeref> SP <path> LF\n+....\n+\n+A `<path>` string should be surrounded with quotation marks and\n+use C-style escaping.\n+\n `feature`\n ~~~~~~~~~\n Require that fast-import supports the specified feature, or abort if\ndiff --git a/fast-import.c b/fast-import.c\nindex ef0cee7..099f63e 100644\n--- a/fast-import.c\n+++ b/fast-import.c\n@@ -55,6 +55,9 @@ Format of STDIN stream:\n     ('from' sp committish lf)?\n     lf?;\n \n+  cat_request ::= 'cat' sp (hexsha1 | idnum) lf\n+    | 'cat' sp hexsha1 sp path_str lf;\n+\n   checkpoint ::= 'checkpoint' lf\n     lf?;\n \n@@ -2688,6 +2691,109 @@ static void parse_reset_branch(void)\n \t\tunread_command_buf = 1;\n }\n \n+static void quoted_path_sha1(unsigned char sha1[20], struct tree_entry *root,\n+\t\t\t\tconst char *path, const char *line)\n+{\n+\tstruct strbuf uq = STRBUF_INIT;\n+\tstruct tree_entry leaf = {0};\n+\tconst char *x;\n+\n+\tif (unquote_c_style(&uq, path, &x))\n+\t\tdie(\"Invalid path: %s\", line);\n+\tif (*x)\n+\t\tdie(\"Garbage after path: %s\", line);\n+\ttree_content_get(root, uq.buf, &leaf);\n+\tif (!leaf.versions[1].mode)\n+\t\tdie(\"Path %s not in branch\", uq.buf);\n+\thashcpy(sha1, leaf.versions[1].sha1);\n+}\n+\n+static void sendreport(const char *buf, unsigned long size)\n+{\n+\tif (write_in_full(report_fd, buf, size) != size)\n+\t\tdie_errno(\"Write to frontend failed\");\n+}\n+\n+static void cat_object(struct object_entry *oe, unsigned char sha1[20])\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\tunsigned long size;\n+\tenum object_type type = 0;\n+\tchar *buf;\n+\n+\tif (report_fd < 0)\n+\t\tdie(\"Internal error: bad report_fd %d\", report_fd);\n+\tif (oe && oe->pack_id != MAX_PACK_ID) {\n+\t\ttype = oe->type;\n+\t\tbuf = gfi_unpack_entry(oe, &size);\n+\t} else {\n+\t\tbuf = read_sha1_file(sha1, &type, &size);\n+\t}\n+\tif (!buf)\n+\t\tdie(\"Can't read object %s\", sha1_to_hex(sha1));\n+\n+\t/*\n+\t * Output based on batch_one_object() from cat-file.c.\n+\t */\n+\tif (type <= 0) {\n+\t\tstrbuf_reset(&line);\n+\t\tstrbuf_addf(&line, \"%s missing\\n\", sha1_to_hex(sha1));\n+\t\tif (write_in_full(report_fd, line.buf, line.len) != line.len)\n+\t\t\tdie_errno(\"Write to frontend failed 1\");\n+\t}\n+\tstrbuf_reset(&line);\n+\tstrbuf_addf(&line, \"%s %s %lu\\n\", sha1_to_hex(sha1),\n+\t\t\t\t\t\ttypename(type), size);\n+\tsendreport(line.buf, line.len);\n+\tsendreport(buf, size);\n+\tsendreport(\"\\n\", 1);\n+\tfree(buf);\n+}\n+\n+\n+static void parse_cat_request(void)\n+{\n+\tconst char *p;\n+\tstruct object_entry *oe = oe;\n+\tunsigned char sha1[20];\n+\tstruct tree_entry root = {0};\n+\n+\t/* cat SP <object> */\n+\tp = command_buf.buf + strlen(\"cat \");\n+\tif (report_fd < 0)\n+\t\tdie(\"The cat command requires the report-fd feature.\");\n+\tif (*p == ':') {\n+\t\tchar *x;\n+\t\toe = find_mark(strtoumax(p + 1, &x, 10));\n+\t\tif (x == p + 1)\n+\t\t\tdie(\"Invalid mark: %s\", command_buf.buf);\n+\t\tif (!oe)\n+\t\t\tdie(\"Unknown mark: %s\", command_buf.buf);\n+\t\tp = x;\n+\t\thashcpy(sha1, oe->idx.sha1);\n+\t} else {\n+\t\tif (get_sha1_hex(p, sha1))\n+\t\t\tdie(\"Invalid SHA1: %s\", command_buf.buf);\n+\t\tp += 40;\n+\t\tif (!*p)\n+\t\t\toe = find_object(sha1);\n+\t}\n+\n+\t/* [ SP \"<path>\" ] */\n+\tif (*p) {\n+\t\tif (*p++ != ' ')\n+\t\t\tdie(\"Missing space after SHA1: %s\", command_buf.buf);\n+\n+\t\t/* cat <tree> \"<path>\" form. */\n+\t\thashcpy(root.versions[1].sha1, sha1);\n+\t\tload_tree(&root);\n+\t\tquoted_path_sha1(sha1, &root, p, command_buf.buf);\n+\t\toe = find_object(sha1);\n+\t}\n+\n+\tcat_object(oe, sha1);\n+}\n+\n static void parse_checkpoint(void)\n {\n \tif (object_count) {\n@@ -2971,6 +3077,8 @@ int main(int argc, const char **argv)\n \t\t\tparse_new_tag();\n \t\telse if (!prefixcmp(command_buf.buf, \"reset \"))\n \t\t\tparse_reset_branch();\n+\t\telse if (!prefixcmp(command_buf.buf, \"cat \"))\n+\t\t\tparse_cat_request();\n \t\telse if (!strcmp(\"checkpoint\", command_buf.buf))\n \t\t\tparse_checkpoint();\n \t\telse if (!prefixcmp(command_buf.buf, \"progress \"))\ndiff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh\nindex 4371308..8155b85 100755\n--- a/t/t9300-fast-import.sh\n+++ b/t/t9300-fast-import.sh\n@@ -1647,6 +1647,108 @@ test_expect_success PIPE 'R: fast-import --report-fd' '\n \ttest_cmp real received\n '\n \n+test_expect_success PIPE 'R: report-fd: can feed back printed tree' '\n+\tcat >frontend <<-\\FRONTEND_END &&\n+\t#!/bin/sh\n+\tcat <<EOF &&\n+\tcommit refs/heads/printed\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tdata <<COMMIT\n+\tto be printed\n+\tCOMMIT\n+\n+\tfrom refs/heads/master\n+\tD file3\n+\n+\tEOF\n+\n+\tread commit_id <&3 &&\n+\techo \"$commit_id\" >printed &&\n+\techo \"$commit_id commit\" >expect.response &&\n+\techo \"cat $commit_id\" &&\n+\tread cid2 type size <&3 &&\n+\techo \"$cid2 $type\" >response &&\n+\tdd if=/dev/stdin of=commit bs=1 count=$size <&3 &&\n+\tread newline <&3 &&\n+\tread tree tree_id <commit &&\n+\n+\tcat <<EOF &&\n+\tcommit refs/heads/printed\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tdata <<COMMIT\n+\tto be printed\n+\tCOMMIT\n+\n+\tfrom refs/heads/printed^0\n+\tM 040000 $tree_id old\n+\n+\tEOF\n+\tread cid <&3\n+\tFRONTEND_END\n+\n+\tmkfifo commits &&\n+\ttest_when_finished \"rm -f commits\" &&\n+\tsh frontend 3<commits |\n+\tgit fast-import --report-fd=3 3>commits &&\n+\tgit rev-parse printed^ >expect.printed &&\n+\tgit cat-file commit printed^ >expect.commit &&\n+\n+\ttest_cmp expect.printed printed &&\n+\ttest_cmp expect.response response &&\n+\ttest_cmp expect.commit commit\n+'\n+\n+test_expect_success PIPE 'R: report-fd: can feed back printed blob' '\n+\tfile6_id=$(echo \"$file6_data\" | git hash-object --stdin) &&\n+\tcat >expect <<-EOF &&\n+\t:100755 100644 $file6_id $file6_id C100\tnewdir/exec.sh\tfile6\n+\tEOF\n+\n+\tcat >frontend <<-\\FRONTEND_END &&\n+\t#!/bin/sh\n+\n+\tbranch=$(git rev-parse --verify refs/heads/branch) &&\n+\tcat <<EOF &&\n+\tfeature report-fd=3\n+\tcat $branch\n+\tEOF\n+\n+\tread commit_id type size <&3 &&\n+\tdd if=/dev/stdin of=commit bs=1 count=$size <&3 &&\n+\tread newline <&3 &&\n+\tread tree tree_id <commit &&\n+\n+\techo \"cat $tree_id \\\"newdir/exec.sh\\\"\" &&\n+\tread blob_id type size <&3 &&\n+\tdd if=/dev/stdin of=blob bs=1 count=$size <&3 &&\n+\tread newline <&3 &&\n+\n+\tcat <<EOF &&\n+\tcommit refs/heads/copyblob\n+\tcommitter $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE\n+\tdata <<COMMIT\n+\tcopy file6 to top level\n+\tCOMMIT\n+\n+\tfrom refs/heads/branch^0\n+\tM 644 inline \"file6\"\n+\tdata $size\n+\tEOF\n+\tcat blob &&\n+\techo &&\n+\techo &&\n+\n+\tread cid <&3\n+\tFRONTEND_END\n+\n+\tmkfifo commits &&\n+\ttest_when_finished \"rm -f commits\" &&\n+\tsh frontend 3<commits |\n+\tgit fast-import 3>commits &&\n+\tgit diff-tree -C --find-copies-harder -r copyblob^ copyblob >actual &&\n+\tcompare_diff_raw expect actual\n+'\n+\n cat >input << EOF\n option git quiet\n blob\n-- \n1.7.3.1\n"}]}