{"thread":{"id":"28113","subject":"[PATCH v3 00/10] vcs-svn,svn-fe add a couple of options","startedAt":"2011-08-16T09:54:45Z","lastAt":"2011-08-16T09:54:55Z","messageCount":11,"participants":["Dmitry Ivankov"],"isPatch":true,"patchVersion":3,"patchTotal":10},"messages":[{"id":"173576","messageId":"1313488495-2203-1-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":null,"subject":"[PATCH v3 00/10] vcs-svn,svn-fe add a couple of options","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:45Z","receivedAt":"2011-08-16T09:54:45Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"This is a next iteration of [1].\n\nMost Noticeable changes since [1] are:\n1) squash [2/11] and [4/11] into [3/10] \"use parse-options\"\n2) replace [11/11] with a lesser patch that just adds api for\nwriting notes in vcs-svn/ but doesn't yet uses it. So that the\nseries is more stable, while exact notes usage can be adjusted\nand played with a bit more separately.\n3) applied most of suggestions from [1] with minor edits\n\nThe patch base is svn-fe branch at git://repo.or.cz/git/jrn.git\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/177025\n\nDmitry Ivankov (10):\n svn-fe: add man target to Makefile\n svn-fe: add EXTLIBS needed for parse-options\n svn-fe,test-svn-fe: use parse-options\n vcs-svn: make svndump_init parameters a struct\n vcs-svn: move commit parameters logic to svndump.c\n vcs-svn,svn-fe: allow to specify dump destination ref\n vcs-svn,svn-fe: convert REPORT_FILENO to an option\n vcs-svn,svn-fe: allow to disable 'progress' lines\n vcs-svn,svn-fe: add --incremental option\n vcs-svn: add fast_export_note to create notes\n\n contrib/svn-fe/Makefile | 18 ++++----\n contrib/svn-fe/svn-fe.c | 45 ++++++++++++++++--\n contrib/svn-fe/svn-fe.txt | 38 +++++++++++++--\n t/t9010-svn-fe.sh | 118 ++++++++++++++++++++++++++++++++++++++-------\n test-svn-fe.c | 57 ++++++++++++++++------\n vcs-svn/fast_export.c | 60 ++++++++++-------------\n vcs-svn/fast_export.h | 10 ++--\n vcs-svn/svndump.c | 79 ++++++++++++++++++++++++------\n vcs-svn/svndump.h | 14 +++++-\n 9 files changed, 332 insertions(+), 107 deletions(-)\n\n-- \n1.7.3.4\n"},{"id":"173577","messageId":"1313488495-2203-2-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 01/10] svn-fe: add man target to Makefile","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:46Z","receivedAt":"2011-08-16T09:54:46Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"There already is a svn-fe.1 target. But 'man' being a standard\ntarget is easier to discover or type. It can also be reused if\nmore manpages arise here.\n\nSigned-off-by: Dmitry Ivankov \n---\n contrib/svn-fe/Makefile | 4 +++-\n 1 files changed, 3 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile\nindex 360d8da..bc03a3e 100644\n--- a/contrib/svn-fe/Makefile\n+++ b/contrib/svn-fe/Makefile\n@@ -33,6 +33,8 @@ ifndef V\n endif\n endif\n \n+man: svn-fe.1\n+\n svn-fe$X: svn-fe.o $(VCSSVN_LIB) $(GIT_LIB)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ svn-fe.o \\\n \t\t$(ALL_LDFLAGS) $(LIBS)\n@@ -60,4 +62,4 @@ svn-fe.1: svn-fe.txt\n clean:\n \t$(RM) svn-fe$X svn-fe.o svn-fe.html svn-fe.xml svn-fe.1\n \n-.PHONY: all clean FORCE\n+.PHONY: all man clean FORCE\n-- \n1.7.3.4\n"},{"id":"173579","messageId":"1313488495-2203-3-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 02/10] svn-fe: add EXTLIBS needed for parse-options","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:47Z","receivedAt":"2011-08-16T09:54:47Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"Currently parse-options.o pull quite a big bunch of dependencies\nthat are neither pulled in by svn-fe Makefile nor included in libgit.a.\n\nUse a temporary hack: put hardcoded EXTLIBS, this may not work in all\nsetups because /Makefile logic is not repeated. The list of extlibs\nis likely to be exhaustive, but one may need to adjust it.\n\n-lcrypto is needed for SHA-1 routines unless NO_OPENSSL or BLK_SHA1\nis set, -lpcre is for grep if USE_LIBPCRE is set, and -lz is needed\nthroughout.\n\nIn the future, none of these should be needed, after a little\nrearranging to ensure that parse-options.o has no references to\ntranslation units that need to access the object db.\n\nSigned-off-by: Dmitry Ivankov \nSigned-off-by: Jonathan Nieder \n---\n contrib/svn-fe/Makefile | 12 +++++-------\n 1 files changed, 5 insertions(+), 7 deletions(-)\n\ndiff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile\nindex bc03a3e..8b12df1 100644\n--- a/contrib/svn-fe/Makefile\n+++ b/contrib/svn-fe/Makefile\n@@ -8,11 +8,12 @@ CFLAGS = -g -O2 -Wall\n LDFLAGS =\n ALL_CFLAGS = $(CFLAGS)\n ALL_LDFLAGS = $(LDFLAGS)\n-EXTLIBS =\n+EXTLIBS = -lssl -lcrypto -lpcre -lz -lpthread\n \n GIT_LIB = ../../libgit.a\n VCSSVN_LIB = ../../vcs-svn/lib.a\n-LIBS = $(VCSSVN_LIB) $(GIT_LIB) $(EXTLIBS)\n+XDIFF_LIB = ../../xdiff/lib.a\n+LIBS = $(VCSSVN_LIB) $(GIT_LIB) $(XDIFF_LIB) $(EXTLIBS)\n \n QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir\n QUIET_SUBDIR1 =\n@@ -53,11 +54,8 @@ svn-fe.1: svn-fe.txt\n \t\t../contrib/svn-fe/$@\n \t$(MV) ../../Documentation/svn-fe.1 .\n \n-../../vcs-svn/lib.a: FORCE\n-\t$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) vcs-svn/lib.a\n-\n-../../libgit.a: FORCE\n-\t$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) libgit.a\n+$(VCSSVN_LIB) $(GIT_LIB) $(XDIFF_LIB): ../../%.a: FORCE\n+\t$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $*.a\n \n clean:\n \t$(RM) svn-fe$X svn-fe.o svn-fe.html svn-fe.xml svn-fe.1\n-- \n1.7.3.4\n"},{"id":"173578","messageId":"1313488495-2203-4-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 03/10] svn-fe,test-svn-fe: use parse-options","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:48Z","receivedAt":"2011-08-16T09:54:48Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"There was custom options parsing. As more options arise it will\nbe easier to add and document new options with parse-options api.\n\nUse parse-options api in svn-fe and test-svn-fe. This implies adding\nhelp messages. And by the way clarify the \"url\" parameter meaning,\nrenaming it to git-svn-id-url and updating svn-fe.txt. Also allow a\n--git-svn-id-url=url way of specifying it.\n\n$ svn-fe --git-svn-id-url=url\ndoes the same thing as\n$ svn-fe url\ni.e., url is used to generate git-svn-id: lines, if url is set.\n\nSigned-off-by: Dmitry Ivankov \nReviewed-by: Jonathan Nieder \n---\n contrib/svn-fe/Makefile | 2 +-\n contrib/svn-fe/svn-fe.c | 32 +++++++++++++++++++++++++++++---\n contrib/svn-fe/svn-fe.txt | 17 +++++++++++++----\n test-svn-fe.c | 43 +++++++++++++++++++++++++++++--------------\n 4 files changed, 72 insertions(+), 22 deletions(-)\n\ndiff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile\nindex 8b12df1..15ba24d 100644\n--- a/contrib/svn-fe/Makefile\n+++ b/contrib/svn-fe/Makefile\n@@ -41,7 +41,7 @@ svn-fe$X: svn-fe.o $(VCSSVN_LIB) $(GIT_LIB)\n \t\t$(ALL_LDFLAGS) $(LIBS)\n \n svn-fe.o: svn-fe.c ../../vcs-svn/svndump.h\n-\t$(QUIET_CC)$(CC) -I../../vcs-svn -o $*.o -c $(ALL_CFLAGS) $<\n+\t$(QUIET_CC)$(CC) -I../../vcs-svn -I../.. -o $*.o -c $(ALL_CFLAGS) $<\n \n svn-fe.html: svn-fe.txt\n \t$(QUIET_SUBDIR0)../../Documentation $(QUIET_SUBDIR1) \\\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex 35db24f..a95e72f 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -3,14 +3,40 @@\n * You may freely use, modify, distribute, and relicense it.\n */\n \n-#include \n+#include \"git-compat-util.h\"\n+#include \"parse-options.h\"\n #include \"svndump.h\"\n \n-int main(int argc, char **argv)\n+static const char * const svn_fe_usage[] = {\n+\t\"svn-fe [options] [git-svn-id-url] < dump | fast-import-backend\",\n+\tNULL\n+};\n+\n+static const char *url;\n+\n+static struct option svn_fe_options[] = {\n+\tOPT_STRING(0, \"git-svn-id-url\", &url, \"url\",\n+\t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n+\tOPT_END()\n+};\n+\n+int main(int argc, const char **argv)\n {\n+\targc = parse_options(argc, argv, NULL, svn_fe_options,\n+\t\t\t\t\t\tsvn_fe_usage, 0);\n+\tif (argc > 1)\n+\t\tusage_with_options(svn_fe_usage, svn_fe_options);\n+\n+\tif (argc == 1) {\n+\t\tif (url)\n+\t\t\tusage_msg_opt(\"git-svn-id-url is set twice: as a \"\n+\t\t\t\t\t\"--parameter and as a [parameter]\",\n+\t\t\t\t\tsvn_fe_usage, svn_fe_options);\n+\t\turl = argv[0];\n+\t}\n \tif (svndump_init(NULL))\n \t\treturn 1;\n-\tsvndump_read((argc > 1) ? argv[1] : NULL);\n+\tsvndump_read(url);\n \tsvndump_deinit();\n \tsvndump_reset();\n \treturn 0;\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex 2dd27ce..8c6d347 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n [verse]\n mkfifo backchannel &&\n svnadmin dump --deltas REPO |\n-\tsvn-fe [url] 3backchannel\n \n DESCRIPTION\n@@ -25,6 +25,14 @@ command.\n Note: this tool is very young. The details of its commandline\n interface may change in backward incompatible ways.\n \n+OPTIONS\n+-------\n+\n+--git-svn-id-url=::\n+\tUrl to be used in git-svn-id: lines in git-svn\n+\tmetadata lines format. See NOTES for more detailed\n+\tdescription.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\n@@ -50,9 +58,10 @@ user \n as committer, where 'user' is the value of the `svn:author` property\n and 'UUID' the repository's identifier.\n \n-To support incremental imports, 'svn-fe' puts a `git-svn-id` line at\n-the end of each commit log message if passed an url on the command\n-line. This line has the form `git-svn-id: URL@REVNO UUID`.\n+'svn-fe' can be used in preparing a repository for 'git-svn' as follows.\n+If `git-svn-id-url` is specified, 'svn-fe' will put `git-svn-id` line at\n+the end of each commit log message.\n+This line has the form `git-svn-id: URL@REVNO UUID`.\n \n The resulting repository will generally require further processing\n to put each project in its own repository and to separate the history\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex 332a5f7..c10d3ca 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -3,28 +3,39 @@\n */\n \n #include \"git-compat-util.h\"\n+#include \"parse-options.h\"\n #include \"vcs-svn/svndump.h\"\n #include \"vcs-svn/svndiff.h\"\n #include \"vcs-svn/sliding_window.h\"\n #include \"vcs-svn/line_buffer.h\"\n \n-static const char test_svnfe_usage[] =\n-\t\"test-svn-fe ( | [-d] )\";\n+static const char * const test_svnfe_usage[] = {\n+\t\"test-svn-fe \",\n+\t\"test-svn-fe -d \",\n+\tNULL\n+};\n \n-static int apply_delta(int argc, char *argv[])\n+static int delta_test;\n+\n+static struct option test_svnfe_options[] = {\n+\tOPT_SET_INT('d', \"apply-delta\", &delta_test, \"test apply_delta\", 1),\n+\tOPT_END()\n+};\n+\n+static int apply_delta(int argc, const char *argv[])\n {\n \tstruct line_buffer preimage = LINE_BUFFER_INIT;\n \tstruct line_buffer delta = LINE_BUFFER_INIT;\n \tstruct sliding_view preimage_view = SLIDING_VIEW_INIT(&preimage, -1);\n \n-\tif (argc != 5)\n-\t\tusage(test_svnfe_usage);\n+\tif (argc != 3)\n+\t\tusage_with_options(test_svnfe_usage, test_svnfe_options);\n \n-\tif (buffer_init(&preimage, argv[2]))\n+\tif (buffer_init(&preimage, argv[0]))\n \t\tdie_errno(\"cannot open preimage\");\n-\tif (buffer_init(&delta, argv[3]))\n+\tif (buffer_init(&delta, argv[1]))\n \t\tdie_errno(\"cannot open delta\");\n-\tif (svndiff0_apply(&delta, (off_t) strtoull(argv[4], NULL, 0),\n+\tif (svndiff0_apply(&delta, (off_t) strtoull(argv[2], NULL, 0),\n \t\t\t\t\t&preimage_view, stdout))\n \t\treturn 1;\n \tif (buffer_deinit(&preimage))\n@@ -37,10 +48,16 @@ static int apply_delta(int argc, char *argv[])\n \treturn 0;\n }\n \n-int main(int argc, char *argv[])\n+int main(int argc, const char *argv[])\n {\n-\tif (argc == 2) {\n-\t\tif (svndump_init(argv[1]))\n+\targc = parse_options(argc, argv, NULL, test_svnfe_options,\n+\t\t\t\t\t\ttest_svnfe_usage, 0);\n+\n+\tif (delta_test)\n+\t\treturn apply_delta(argc, argv);\n+\n+\tif (argc == 1) {\n+\t\tif (svndump_init(argv[0]))\n \t\t\treturn 1;\n \t\tsvndump_read(NULL);\n \t\tsvndump_deinit();\n@@ -48,7 +65,5 @@ int main(int argc, char *argv[])\n \t\treturn 0;\n \t}\n \n-\tif (argc >= 2 && !strcmp(argv[1], \"-d\"))\n-\t\treturn apply_delta(argc, argv);\n-\tusage(test_svnfe_usage);\n+\tusage_with_options(test_svnfe_usage, test_svnfe_options);\n }\n-- \n1.7.3.4\n"},{"id":"173582","messageId":"1313488495-2203-5-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 04/10] vcs-svn: make svndump_init parameters a struct","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:49Z","receivedAt":"2011-08-16T09:54:49Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svndump_init takes a dumpfile parameter and svndump_read takes url\nparameter. Internally url is stored in dump_ctx that is in fact\nreset in svndump_init, but then is reset once more in svndump_read.\nIt'd be better to make url a svndump_init parameter and avoid this\ndouble reset.\n\nA bunch of new svndump parameters are going to be introduced. So wrap\nthem all to a svndump_options struct to make adding new ones smooth\nand easy.\n\nThe usage changes like following.\nBefore:\n\tif (svndump_init(dumpfile))\n\t\tdie(\"svndump_init failed\");\n\tsvndump_read(url);\nAfter:\n\tstruct svndump_args opts;\n\tmemset(&opts, 0, sizeof(opts));\n\topts.url = url;\n\topts.filename = dumpfile;\n\tif (svndump_init(&opts))\n\t\tdie(\"svndump_init failed\");\n\tsvndump_read();\n\nSigned-off-by: Dmitry Ivankov \n---\n contrib/svn-fe/svn-fe.c | 12 ++++++------\n test-svn-fe.c | 7 +++++--\n vcs-svn/svndump.c | 12 ++++++------\n vcs-svn/svndump.h | 11 +++++++++--\n 4 files changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex a95e72f..0165c3f 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -12,10 +12,10 @@ static const char * const svn_fe_usage[] = {\n \tNULL\n };\n \n-static const char *url;\n+static struct svndump_options options;\n \n static struct option svn_fe_options[] = {\n-\tOPT_STRING(0, \"git-svn-id-url\", &url, \"url\",\n+\tOPT_STRING(0, \"git-svn-id-url\", &options.git_svn_url, \"url\",\n \t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n \tOPT_END()\n };\n@@ -28,15 +28,15 @@ int main(int argc, const char **argv)\n \t\tusage_with_options(svn_fe_usage, svn_fe_options);\n \n \tif (argc == 1) {\n-\t\tif (url)\n+\t\tif (options.git_svn_url)\n \t\t\tusage_msg_opt(\"git-svn-id-url is set twice: as a \"\n \t\t\t\t\t\"--parameter and as a [parameter]\",\n \t\t\t\t\tsvn_fe_usage, svn_fe_options);\n-\t\turl = argv[0];\n+\t\toptions.git_svn_url = argv[0];\n \t}\n-\tif (svndump_init(NULL))\n+\tif (svndump_init(&options))\n \t\treturn 1;\n-\tsvndump_read(url);\n+\tsvndump_read();\n \tsvndump_deinit();\n \tsvndump_reset();\n \treturn 0;\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex c10d3ca..0dd0657 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -15,6 +15,8 @@ static const char * const test_svnfe_usage[] = {\n \tNULL\n };\n \n+static struct svndump_options options;\n+\n static int delta_test;\n \n static struct option test_svnfe_options[] = {\n@@ -57,9 +59,10 @@ int main(int argc, const char *argv[])\n \t\treturn apply_delta(argc, argv);\n \n \tif (argc == 1) {\n-\t\tif (svndump_init(argv[0]))\n+\t\toptions.dumpfile = argv[0];\n+\t\tif (svndump_init(&options))\n \t\t\treturn 1;\n-\t\tsvndump_read(NULL);\n+\t\tsvndump_read();\n \t\tsvndump_deinit();\n \t\tsvndump_reset();\n \t\treturn 0;\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex b1f4161..5cdf6b8 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -12,6 +12,7 @@\n #include \"fast_export.h\"\n #include \"line_buffer.h\"\n #include \"strbuf.h\"\n+#include \"svndump.h\"\n \n /*\n * Compare start of string to literal of equal length;\n@@ -313,14 +314,13 @@ static void end_revision(void)\n \t\tfast_export_end_commit(rev_ctx.revision);\n }\n \n-void svndump_read(const char *url)\n+void svndump_read(void)\n {\n \tchar *val;\n \tchar *t;\n \tuint32_t active_ctx = DUMP_CTX;\n \tuint32_t len;\n \n-\treset_dump_ctx(url);\n \twhile ((t = buffer_read_line(&input))) {\n \t\tval = strchr(t, ':');\n \t\tif (!val)\n@@ -455,10 +455,10 @@ void svndump_read(const char *url)\n \t\tend_revision();\n }\n \n-int svndump_init(const char *filename)\n+int svndump_init(const struct svndump_options *o)\n {\n-\tif (buffer_init(&input, filename))\n-\t\treturn error(\"cannot open %s: %s\", filename, strerror(errno));\n+\tif (buffer_init(&input, o->dumpfile))\n+\t\treturn error(\"cannot open %s: %s\", o->dumpfile, strerror(errno));\n \tfast_export_init(REPORT_FILENO);\n \tstrbuf_init(&dump_ctx.uuid, 4096);\n \tstrbuf_init(&dump_ctx.url, 4096);\n@@ -466,7 +466,7 @@ int svndump_init(const char *filename)\n \tstrbuf_init(&rev_ctx.author, 4096);\n \tstrbuf_init(&node_ctx.src, 4096);\n \tstrbuf_init(&node_ctx.dst, 4096);\n-\treset_dump_ctx(NULL);\n+\treset_dump_ctx(o->git_svn_url);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \treturn 0;\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex df9ceb0..db39dfe 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -1,8 +1,15 @@\n #ifndef SVNDUMP_H_\n #define SVNDUMP_H_\n \n-int svndump_init(const char *filename);\n-void svndump_read(const char *url);\n+struct svndump_options {\n+\t/*\n+\t * dumpfile is opened in svndump_init and is read in svndump_read.\n+\t */\n+\tconst char *dumpfile, *git_svn_url;\n+};\n+\n+int svndump_init(const struct svndump_options *o);\n+void svndump_read(void);\n void svndump_deinit(void);\n void svndump_reset(void);\n \n-- \n1.7.3.4\n"},{"id":"173580","messageId":"1313488495-2203-6-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 05/10] vcs-svn: move commit parameters logic to svndump.c","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:50Z","receivedAt":"2011-08-16T09:54:50Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"fast_export.c had logic to set up commit ref, author name, email,\nparent commit, import mark and git-svn-id: line based on both it's\nown state (current import batch history) and the arguments passed.\n\nDo separate the layers: make fast_export focus on producing the\nfast-import stream, applying the deltas but not on svn-fe specific\nlogic. svndump now is responsible for choosing commit parents, marks,\nref name. Making it possible to generate incremental streams, produce\nstream for several branches at a time, customize progress lines generation\nand adding new logic becomes easier.\n\nfast_export API changes:\n- make fast_export_begin_commit to be more intuitive by using a set of\n parameters closer to what gets written to fast-import.\n\n- rename fast_export_end_commit to fast_export_progress as it does only\n a progress line generation. fast_export_end_commit can be reintroduced\n once the need will arise.\n\n- git-svn-id line is now a caller concern. If it is needed, it should be\n simply appended to the log message.\n\n- author_name and author_email are now generated by the caller.\n\n- ref name to be updated with the commit is now a parameter rather than\n a fixed \"refs/heads/master\".\n\nThe caller now may have to setup temporary buffers for author identity,\nref names, etc. This is a small additional per-commit cost to arrange\nand/or copy them. Though, it's only per-commit rather that per-path\nand might be worth the gain in readablity.\n\nSigned-off-by: Dmitry Ivankov \nSigned-off-by: Jonathan Nieder \n---\n vcs-svn/fast_export.c | 47 +++++++++++++----------------------------------\n vcs-svn/fast_export.h | 8 ++++----\n vcs-svn/svndump.c | 44 +++++++++++++++++++++++++++++++++++++++-----\n 3 files changed, 56 insertions(+), 43 deletions(-)\n\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 19d7c34..3dfccd2 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -13,9 +13,6 @@\n #include \"sliding_window.h\"\n #include \"line_buffer.h\"\n \n-#define MAX_GITSVN_LINE_LEN 4096\n-\n-static uint32_t first_commit_done;\n static struct line_buffer postimage = LINE_BUFFER_INIT;\n static struct line_buffer report_buffer = LINE_BUFFER_INIT;\n \n@@ -31,7 +28,6 @@ static int init_postimage(void)\n \n void fast_export_init(int fd)\n {\n-\tfirst_commit_done = 0;\n \tif (buffer_fdinit(&report_buffer, fd))\n \t\tdie_errno(\"cannot read from file descriptor %d\", fd);\n }\n@@ -73,40 +69,23 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n \tputchar('\\n');\n }\n \n-static char gitsvnline[MAX_GITSVN_LINE_LEN];\n-void fast_export_begin_commit(uint32_t revision, const char *author,\n-\t\t\tconst struct strbuf *log,\n-\t\t\tconst char *uuid, const char *url,\n-\t\t\tunsigned long timestamp)\n+void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,\n+\t\t\tconst char *author_name, const char *author_email,\n+\t\t\tconst struct strbuf *log, unsigned long timestamp)\n {\n-\tstatic const struct strbuf empty = STRBUF_INIT;\n-\tif (!log)\n-\t\tlog = ∅\n-\tif (*uuid && *url) {\n-\t\tsnprintf(gitsvnline, MAX_GITSVN_LINE_LEN,\n-\t\t\t\t\"\\n\\ngit-svn-id: %s@%\"PRIu32\" %s\\n\",\n-\t\t\t\t url, revision, uuid);\n-\t} else {\n-\t\t*gitsvnline = '\\0';\n-\t}\n-\tprintf(\"commit refs/heads/master\\n\");\n-\tprintf(\"mark :%\"PRIu32\"\\n\", revision);\n-\tprintf(\"committer %s <%s@%s> %ld +0000\\n\",\n-\t\t *author ? author : \"nobody\",\n-\t\t *author ? author : \"nobody\",\n-\t\t *uuid ? uuid : \"local\", timestamp);\n-\tprintf(\"data %\"PRIuMAX\"\\n\",\n-\t\t(uintmax_t) (log->len + strlen(gitsvnline)));\n+\tprintf(\"commit %s\\n\", ref);\n+\tif (mark)\n+\t\tprintf(\"mark :%\"PRIu32\"\\n\", mark);\n+\tprintf(\"committer %s <%s> %ld +0000\\n\",\n+\t\tauthor_name, author_email, timestamp);\n+\tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t) log->len);\n \tfwrite(log->buf, log->len, 1, stdout);\n-\tprintf(\"%s\\n\", gitsvnline);\n-\tif (!first_commit_done) {\n-\t\tif (revision > 1)\n-\t\t\tprintf(\"from :%\"PRIu32\"\\n\", revision - 1);\n-\t\tfirst_commit_done = 1;\n-\t}\n+\tputchar('\\n');\n+\tif (from && *from)\n+\t\tprintf(\"from %s\\n\", from);\n }\n \n-void fast_export_end_commit(uint32_t revision)\n+void fast_export_progress(uint32_t revision)\n {\n \tprintf(\"progress Imported commit %\"PRIu32\".\\n\\n\", revision);\n }\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex 43d05b6..bf58880 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -10,10 +10,10 @@ void fast_export_reset(void);\n \n void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n-void fast_export_begin_commit(uint32_t revision, const char *author,\n-\t\t\tconst struct strbuf *log, const char *uuid,\n-\t\t\tconst char *url, unsigned long timestamp);\n-void fast_export_end_commit(uint32_t revision);\n+void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,\n+\t\t\tconst char *author_name, const char *author_email,\n+\t\t\tconst struct strbuf *log, unsigned long timestamp);\n+void fast_export_progress(uint32_t revision);\n void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input);\n void fast_export_blob_delta(uint32_t mode,\n \t\t\tuint32_t old_mode, const char *old_data,\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 5cdf6b8..28d84c9 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -37,6 +37,8 @@\n #define LENGTH_UNKNOWN (~0)\n #define DATE_RFC2822_LEN 31\n \n+#define MAX_GITSVN_LINE_LEN 4096\n+\n static struct line_buffer input = LINE_BUFFER_INIT;\n \n static struct {\n@@ -54,6 +56,7 @@ static struct {\n static struct {\n \tuint32_t version;\n \tstruct strbuf uuid, url;\n+\tint first_commit_done;\n } dump_ctx;\n \n static void reset_node_ctx(char *fname)\n@@ -86,6 +89,7 @@ static void reset_dump_ctx(const char *url)\n \t\tstrbuf_addstr(&dump_ctx.url, url);\n \tdump_ctx.version = 1;\n \tstrbuf_reset(&dump_ctx.uuid);\n+\tdump_ctx.first_commit_done = 0;\n }\n \n static void handle_property(const struct strbuf *key_buf,\n@@ -299,19 +303,49 @@ static void handle_node(void)\n \t\t\t\tnode_ctx.textLength, &input);\n }\n \n+static void add_metadata_trailer(struct strbuf *buf)\n+{\n+\tif (*dump_ctx.uuid.buf && *dump_ctx.url.buf)\n+\t\tstrbuf_addf(buf, \"\\n\\ngit-svn-id: %s@%\"PRIu32\" %s\\n\",\n+\t\t\t dump_ctx.url.buf, rev_ctx.revision, dump_ctx.uuid.buf);\n+}\n+\n static void begin_revision(void)\n {\n+\tstatic struct strbuf email;\n+\tconst char *author;\n+\tuint32_t prev;\n+\tchar buf[32];\n+\n \tif (!rev_ctx.revision)\t/* revision 0 gets no git commit. */\n \t\treturn;\n-\tfast_export_begin_commit(rev_ctx.revision, rev_ctx.author.buf,\n-\t\t&rev_ctx.log, dump_ctx.uuid.buf, dump_ctx.url.buf,\n-\t\trev_ctx.timestamp);\n+\tprev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;\n+\tif (prev)\n+\t\tsnprintf(buf, 32, \":%\"PRIu32, prev);\n+\telse\n+\t\t*buf = 0;\n+\tauthor = *rev_ctx.author.buf ? rev_ctx.author.buf : \"nobody\";\n+\n+\tstrbuf_reset(&email);\n+\tstrbuf_addstr(&email, author);\n+\tstrbuf_addch(&email, '@');\n+\tif (*dump_ctx.uuid.buf)\n+\t\tstrbuf_addstr(&email, dump_ctx.uuid.buf);\n+\telse\n+\t\tstrbuf_addstr(&email, \"local\");\n+\n+\tadd_metadata_trailer(&rev_ctx.log);\n+\n+\tfast_export_begin_commit(\"refs/heads/master\", rev_ctx.revision, buf,\n+\t\tauthor, email.buf, &rev_ctx.log, rev_ctx.timestamp);\n }\n \n static void end_revision(void)\n {\n-\tif (rev_ctx.revision)\n-\t\tfast_export_end_commit(rev_ctx.revision);\n+\tif (rev_ctx.revision) {\n+\t\tfast_export_progress(rev_ctx.revision);\n+\t\tdump_ctx.first_commit_done = 1;\n+\t}\n }\n \n void svndump_read(void)\n-- \n1.7.3.4\n"},{"id":"173581","messageId":"1313488495-2203-7-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 06/10] vcs-svn,svn-fe: allow to specify dump destination ref","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:51Z","receivedAt":"2011-08-16T09:54:51Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svn-fe produces fast-import stream for a fixed refs/heads/master ref.\nIt is usually desired to write to a different ref. In a remote helper\nit would be a ref in private namespace. If svn-fe is used by someone\ndirectly it'll be more safe to remind where the commits can go. And\nin both cases it may be needed to import from two repos and hence to\ndifferent refs.\n\nAdd a destination ref parameter to vcs-svn/, a corresponding parameter\nto svn-fe and a simple test for it.\n\n$ svn-fe --ref=refs/heads/master ...\nis an explicit way to stay with the default destination.\n\nSigned-off-by: Dmitry Ivankov \nSigned-off-by: Jonathan Nieder \n---\n contrib/svn-fe/svn-fe.c | 2 ++\n contrib/svn-fe/svn-fe.txt | 4 ++++\n t/t9010-svn-fe.sh | 42 ++++++++++++++++++++++++++----------------\n test-svn-fe.c | 4 +++-\n vcs-svn/svndump.c | 16 +++++++++++-----\n vcs-svn/svndump.h | 1 +\n 6 files changed, 47 insertions(+), 22 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex 0165c3f..9dd8336 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -17,6 +17,8 @@ static struct svndump_options options;\n static struct option svn_fe_options[] = {\n \tOPT_STRING(0, \"git-svn-id-url\", &options.git_svn_url, \"url\",\n \t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n+\tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\n+\t\t\"write to instead of refs/heads/master\"),\n \tOPT_END()\n };\n \ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex 8c6d347..0d19475 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -33,6 +33,10 @@ OPTIONS\n \tmetadata lines format. See NOTES for more detailed\n \tdescription.\n \n+--ref=::\n+\tRef to be written by the generated stream.\n+\tDefault is refs/heads/master.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\ndiff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh\nindex b7eed24..b45527e 100755\n--- a/t/t9010-svn-fe.sh\n+++ b/t/t9010-svn-fe.sh\n@@ -20,9 +20,10 @@ try_dump () {\n \tinput=$1 &&\n \tmaybe_fail_svnfe=${2:+test_$2} &&\n \tmaybe_fail_fi=${3:+test_$3} &&\n+\targs=${4:-} &&\n \n \t{\n-\t\t$maybe_fail_svnfe test-svn-fe \"$input\" >stream 3stream 3backflow &&\n \twait $!\n@@ -54,6 +55,22 @@ text_no_props () {\n \n >empty\n \n+cat >emptyprop.dump <<\\EOF\n+SVN-fs-dump-format-version: 3\n+\n+Revision-number: 1\n+Prop-content-length: 10\n+Content-length: 10\n+\n+PROPS-END\n+\n+Revision-number: 2\n+Prop-content-length: 10\n+Content-length: 10\n+\n+PROPS-END\n+EOF\n+\n test_expect_success 'setup: have pipes?' '\n \trm -f frob &&\n \tif mkfifo frob\n@@ -97,26 +114,19 @@ test_expect_failure PIPE 'empty revision' '\n test_expect_success PIPE 'empty properties' '\n \treinit_git &&\n \tprintf \"rev : %s\\n\" \"\" \"\" >expect &&\n-\tcat >emptyprop.dump <<-\\EOF &&\n-\tSVN-fs-dump-format-version: 3\n-\n-\tRevision-number: 1\n-\tProp-content-length: 10\n-\tContent-length: 10\n-\n-\tPROPS-END\n-\n-\tRevision-number: 2\n-\tProp-content-length: 10\n-\tContent-length: 10\n-\n-\tPROPS-END\n-\tEOF\n \ttry_dump emptyprop.dump &&\n \tgit log -p --format=\"rev <%an, %ae>: %s\" HEAD >actual &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success PIPE 'import to notmaster ref' '\n+\treinit_git &&\n+\ttry_dump emptyprop.dump \"\" \"\" \"--ref=refs/heads/notmaster\" &&\n+\n+\tgit rev-parse --verify notmaster &&\n+\ttest_must_fail git rev-parse --verify master\n+'\n+\n test_expect_success PIPE 'author name and commit message' '\n \treinit_git &&\n \techo \"\" >expect.author &&\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex 0dd0657..fddd3e8 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -10,7 +10,7 @@\n #include \"vcs-svn/line_buffer.h\"\n \n static const char * const test_svnfe_usage[] = {\n-\t\"test-svn-fe \",\n+\t\"test-svn-fe [options] \",\n \t\"test-svn-fe -d \",\n \tNULL\n };\n@@ -21,6 +21,8 @@ static int delta_test;\n \n static struct option test_svnfe_options[] = {\n \tOPT_SET_INT('d', \"apply-delta\", &delta_test, \"test apply_delta\", 1),\n+\tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\n+\t\t\"write to instead of refs/heads/master\"),\n \tOPT_END()\n };\n \ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 28d84c9..3a64708 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -57,6 +57,7 @@ static struct {\n \tuint32_t version;\n \tstruct strbuf uuid, url;\n \tint first_commit_done;\n+\tstruct strbuf ref_name;\n } dump_ctx;\n \n static void reset_node_ctx(char *fname)\n@@ -82,7 +83,7 @@ static void reset_rev_ctx(uint32_t revision)\n \tstrbuf_reset(&rev_ctx.author);\n }\n \n-static void reset_dump_ctx(const char *url)\n+static void reset_dump_ctx(const char *url, const char *dst_ref)\n {\n \tstrbuf_reset(&dump_ctx.url);\n \tif (url)\n@@ -90,6 +91,8 @@ static void reset_dump_ctx(const char *url)\n \tdump_ctx.version = 1;\n \tstrbuf_reset(&dump_ctx.uuid);\n \tdump_ctx.first_commit_done = 0;\n+\tstrbuf_reset(&dump_ctx.ref_name);\n+\tstrbuf_addstr(&dump_ctx.ref_name, dst_ref);\n }\n \n static void handle_property(const struct strbuf *key_buf,\n@@ -336,8 +339,8 @@ static void begin_revision(void)\n \n \tadd_metadata_trailer(&rev_ctx.log);\n \n-\tfast_export_begin_commit(\"refs/heads/master\", rev_ctx.revision, buf,\n-\t\tauthor, email.buf, &rev_ctx.log, rev_ctx.timestamp);\n+\tfast_export_begin_commit(dump_ctx.ref_name.buf, rev_ctx.revision, buf,\n+\t\t\t\tauthor, email.buf, &rev_ctx.log, rev_ctx.timestamp);\n }\n \n static void end_revision(void)\n@@ -491,6 +494,9 @@ void svndump_read(void)\n \n int svndump_init(const struct svndump_options *o)\n {\n+\tconst char *ref = o->ref;\n+\tif (!ref)\n+\t\tref = \"refs/heads/master\";\n \tif (buffer_init(&input, o->dumpfile))\n \t\treturn error(\"cannot open %s: %s\", o->dumpfile, strerror(errno));\n \tfast_export_init(REPORT_FILENO);\n@@ -500,7 +506,7 @@ int svndump_init(const struct svndump_options *o)\n \tstrbuf_init(&rev_ctx.author, 4096);\n \tstrbuf_init(&node_ctx.src, 4096);\n \tstrbuf_init(&node_ctx.dst, 4096);\n-\treset_dump_ctx(o->git_svn_url);\n+\treset_dump_ctx(o->git_svn_url, ref);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \treturn 0;\n@@ -509,7 +515,7 @@ int svndump_init(const struct svndump_options *o)\n void svndump_deinit(void)\n {\n \tfast_export_deinit();\n-\treset_dump_ctx(NULL);\n+\treset_dump_ctx(NULL, \"\");\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \tstrbuf_release(&rev_ctx.log);\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex db39dfe..0b01ccd 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -6,6 +6,7 @@ struct svndump_options {\n \t * dumpfile is opened in svndump_init and is read in svndump_read.\n \t */\n \tconst char *dumpfile, *git_svn_url;\n+\tconst char *ref;\n };\n \n int svndump_init(const struct svndump_options *o);\n-- \n1.7.3.4\n"},{"id":"173585","messageId":"1313488495-2203-8-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 07/10] vcs-svn,svn-fe: convert REPORT_FILENO to an option","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:52Z","receivedAt":"2011-08-16T09:54:52Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"svn-fe needs to read fast-import's responses to \"ls\" and \"cat-blob\".\nThese come through a file descriptor number 3.\n\nDescriptor number 3 isn't a standard of any kind and thus there is\nno reason to hardcode it. Moreover it may require noticeable effort\nto setup this descriptor, if number 3 is already taken for example.\n\nAdd an option for this file descriptor number in vcs-svn/ and svn-fe,\nadd a simple test for it.\n\nTo be used like following:\n$ svn-fe --read-blob-fd=7 ... 7\n---\n contrib/svn-fe/svn-fe.c | 3 ++\n contrib/svn-fe/svn-fe.txt | 9 ++++++-\n t/t9010-svn-fe.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++-\n test-svn-fe.c | 3 ++\n vcs-svn/svndump.c | 4 +--\n vcs-svn/svndump.h | 1 +\n 6 files changed, 66 insertions(+), 6 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex 9dd8336..a4fd7f5 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -19,11 +19,14 @@ static struct option svn_fe_options[] = {\n \t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n \tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\n \t\t\"write to instead of refs/heads/master\"),\n+\tOPT_INTEGER(0, \"read-blob-fd\", &options.backflow_fd,\n+\t\t\"read blobs and trees from this fd instead of 3\"),\n \tOPT_END()\n };\n \n int main(int argc, const char **argv)\n {\n+\toptions.backflow_fd = 3;\n \targc = parse_options(argc, argv, NULL, svn_fe_options,\n \t\t\t\t\t\tsvn_fe_usage, 0);\n \tif (argc > 1)\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex 0d19475..7bfc5a6 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -8,9 +8,9 @@ svn-fe - convert an SVN \"dumpfile\" to a fast-import stream\n SYNOPSIS\n --------\n [verse]\n-mkfifo backchannel &&\n+mkfifo backchannel && fd=3 &&\n svnadmin dump --deltas REPO |\n-\tsvn-fe [options] [git-svn-id-url] 3backchannel\n \n DESCRIPTION\n@@ -37,6 +37,11 @@ OPTIONS\n \tRef to be written by the generated stream.\n \tDefault is refs/heads/master.\n \n+--read-blob-fd=::\n+\tInteger number of file descriptor from which\n+\tresponses to 'ls' and 'cat-blob' requests will come.\n+\tDefault is fd=3.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\ndiff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh\nindex b45527e..efb4925 100755\n--- a/t/t9010-svn-fe.sh\n+++ b/t/t9010-svn-fe.sh\n@@ -21,9 +21,10 @@ try_dump () {\n \tmaybe_fail_svnfe=${2:+test_$2} &&\n \tmaybe_fail_fi=${3:+test_$3} &&\n \targs=${4:-} &&\n+\tfd=${5:-3} &&\n \n \t{\n-\t\t$maybe_fail_svnfe test-svn-fe $args \"$input\" >stream 3stream $fdbackflow &&\n \twait $!\n@@ -1121,4 +1122,53 @@ test_expect_success SVNREPO,PIPE 't9135/svn.dump' '\n \t)\n '\n \n+test_expect_success PIPE 'use different backflow fd' '\n+\treinit_git &&\n+\techo hi >hi &&\n+\t{\n+\t\tproperties \\\n+\t\t\tsvn:author author@example.com \\\n+\t\t\tsvn:date \"1999-02-01T00:01:002.000000Z\" \\\n+\t\t\tsvn:log \"add directory with some files in it\" &&\n+\t\techo PROPS-END\n+\t} >props &&\n+\t{\n+\t\techo Prop-content-length: $(wc -c revprops &&\n+\t{\n+\t\tcat <<-EOF &&\n+\t\tSVN-fs-dump-format-version: 3\n+\n+\t\tRevision-number: 1\n+\t\tEOF\n+\t\tcat revprops &&\n+\t\tcat <<-EOF &&\n+\t\tNode-path: directory\n+\t\tNode-kind: dir\n+\t\tNode-action: add\n+\t\tNode-path: directory/somefile\n+\t\tNode-kind: file\n+\t\tNode-action: add\n+\t\tEOF\n+\t\ttext_no_props hi &&\n+\n+\t\techo \"Revision-number: 2\" &&\n+\t\tcat revprops &&\n+\t\tcat <<-\\EOF\n+\t\tNode-path: otherfile\n+\t\tNode-kind: file\n+\t\tNode-action: add\n+\t\tNode-copyfrom-rev: 1\n+\t\tNode-copyfrom-path: directory/somefile\n+\t\tEOF\n+\t} >directory.dump &&\n+\ttry_dump directory.dump \"\" \"\" \"--read-blob-fd=7\" 7 &&\n+\n+\tgit checkout HEAD otherfile &&\n+\ttest_cmp hi otherfile\n+'\n+\n test_done\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex fddd3e8..568e47e 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -23,6 +23,8 @@ static struct option test_svnfe_options[] = {\n \tOPT_SET_INT('d', \"apply-delta\", &delta_test, \"test apply_delta\", 1),\n \tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\n \t\t\"write to instead of refs/heads/master\"),\n+\tOPT_INTEGER(0, \"read-blob-fd\", &options.backflow_fd,\n+\t\t\"read blobs and trees from this fd instead of 3\"),\n \tOPT_END()\n };\n \n@@ -54,6 +56,7 @@ static int apply_delta(int argc, const char *argv[])\n \n int main(int argc, const char *argv[])\n {\n+\toptions.backflow_fd = 3;\n \targc = parse_options(argc, argv, NULL, test_svnfe_options,\n \t\t\t\t\t\ttest_svnfe_usage, 0);\n \ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 3a64708..847d733 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -20,8 +20,6 @@\n */\n #define constcmp(s, ref) memcmp(s, ref, sizeof(ref) - 1)\n \n-#define REPORT_FILENO 3\n-\n #define NODEACT_REPLACE 4\n #define NODEACT_DELETE 3\n #define NODEACT_ADD 2\n@@ -499,7 +497,7 @@ int svndump_init(const struct svndump_options *o)\n \t\tref = \"refs/heads/master\";\n \tif (buffer_init(&input, o->dumpfile))\n \t\treturn error(\"cannot open %s: %s\", o->dumpfile, strerror(errno));\n-\tfast_export_init(REPORT_FILENO);\n+\tfast_export_init(o->backflow_fd);\n \tstrbuf_init(&dump_ctx.uuid, 4096);\n \tstrbuf_init(&dump_ctx.url, 4096);\n \tstrbuf_init(&rev_ctx.log, 4096);\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex 0b01ccd..8de9999 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -7,6 +7,7 @@ struct svndump_options {\n \t */\n \tconst char *dumpfile, *git_svn_url;\n \tconst char *ref;\n+\tint backflow_fd;\n };\n \n int svndump_init(const struct svndump_options *o);\n-- \n1.7.3.4\n"},{"id":"173583","messageId":"1313488495-2203-9-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 08/10] vcs-svn,svn-fe: allow to disable 'progress' lines","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:53Z","receivedAt":"2011-08-16T09:54:53Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"vcs-svn/ writes a progress line after each processed revision. It\nis too noisy for big imports. That's a stress for a terminal and\nany other output can be lost or scrolled away among these lines.\nIf svn-fe is invoked by a remote helper the import stream with\nprogress lines in it will go directly to the git fast-import which\nalways prints every progress line met in the stream.\n\nFor now just add a switch to turn progress lines off:\n$ svn-fe --no-progress ...\n\nSigned-off-by: Dmitry Ivankov \n---\n contrib/svn-fe/svn-fe.c | 3 +++\n contrib/svn-fe/svn-fe.txt | 4 ++++\n test-svn-fe.c | 1 +\n vcs-svn/svndump.c | 6 +++++-\n vcs-svn/svndump.h | 1 +\n 5 files changed, 14 insertions(+), 1 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex a4fd7f5..6aedba2 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -15,6 +15,9 @@ static const char * const svn_fe_usage[] = {\n static struct svndump_options options;\n \n static struct option svn_fe_options[] = {\n+\t{ OPTION_SET_INT, 0, \"progress\", &options.progress,\n+\t\tNULL, \"don't write a progress line after each commit\",\n+\t\tPARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },\n \tOPT_STRING(0, \"git-svn-id-url\", &options.git_svn_url, \"url\",\n \t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n \tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex 7bfc5a6..abeebfe 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -42,6 +42,10 @@ OPTIONS\n \tresponses to 'ls' and 'cat-blob' requests will come.\n \tDefault is fd=3.\n \n+--[no-]progress::\n+\tWrite 'progress' lines to fast-import stream. These\n+\tcan be displayed by fast-import.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex 568e47e..e51a9bb 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -57,6 +57,7 @@ static int apply_delta(int argc, const char *argv[])\n int main(int argc, const char *argv[])\n {\n \toptions.backflow_fd = 3;\n+\toptions.progress = 1;\n \targc = parse_options(argc, argv, NULL, test_svnfe_options,\n \t\t\t\t\t\ttest_svnfe_usage, 0);\n \ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 847d733..ec68649 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -37,6 +37,8 @@\n \n #define MAX_GITSVN_LINE_LEN 4096\n \n+static int print_progress;\n+\n static struct line_buffer input = LINE_BUFFER_INIT;\n \n static struct {\n@@ -344,7 +346,8 @@ static void begin_revision(void)\n static void end_revision(void)\n {\n \tif (rev_ctx.revision) {\n-\t\tfast_export_progress(rev_ctx.revision);\n+\t\tif (print_progress)\n+\t\t\tfast_export_progress(rev_ctx.revision);\n \t\tdump_ctx.first_commit_done = 1;\n \t}\n }\n@@ -497,6 +500,7 @@ int svndump_init(const struct svndump_options *o)\n \t\tref = \"refs/heads/master\";\n \tif (buffer_init(&input, o->dumpfile))\n \t\treturn error(\"cannot open %s: %s\", o->dumpfile, strerror(errno));\n+\tprint_progress = o->progress;\n \tfast_export_init(o->backflow_fd);\n \tstrbuf_init(&dump_ctx.uuid, 4096);\n \tstrbuf_init(&dump_ctx.url, 4096);\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex 8de9999..de54969 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -8,6 +8,7 @@ struct svndump_options {\n \tconst char *dumpfile, *git_svn_url;\n \tconst char *ref;\n \tint backflow_fd;\n+\tint progress;\n };\n \n int svndump_init(const struct svndump_options *o);\n-- \n1.7.3.4\n"},{"id":"173584","messageId":"1313488495-2203-10-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 09/10] vcs-svn,svn-fe: add --incremental option","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:54Z","receivedAt":"2011-08-16T09:54:54Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"This option is to make svn-fe write commits on top of the existing ref\ninstead of overwriting it. More precise, the first commit's parent is\nset to be :(first_revision_in_current_dump - 1) mark.\n\nPrerequisite is to (re)use import marks (from previous imports). It is\nsafe to use this option on a svn dump that starts with r0/r1. The svn\ndump itself should be incremental too.\n\nSigned-off-by: Dmitry Ivankov \n---\n contrib/svn-fe/svn-fe.c | 3 +++\n contrib/svn-fe/svn-fe.txt | 6 ++++++\n t/t9010-svn-fe.sh | 26 +++++++++++++++++++++++++-\n test-svn-fe.c | 3 +++\n vcs-svn/svndump.c | 13 +++++++++----\n vcs-svn/svndump.h | 2 +-\n 6 files changed, 47 insertions(+), 6 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex 6aedba2..9e0be17 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -18,6 +18,9 @@ static struct option svn_fe_options[] = {\n \t{ OPTION_SET_INT, 0, \"progress\", &options.progress,\n \t\tNULL, \"don't write a progress line after each commit\",\n \t\tPARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },\n+\tOPT_SET_INT(0, \"incremental\", &options.incremental,\n+\t\t\"resume export, requires marks and incremental dump\",\n+\t\t1),\n \tOPT_STRING(0, \"git-svn-id-url\", &options.git_svn_url, \"url\",\n \t\t\"add git-svn-id line to log messages, imitating git-svn\"),\n \tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex abeebfe..f13648c 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -46,6 +46,12 @@ OPTIONS\n \tWrite 'progress' lines to fast-import stream. These\n \tcan be displayed by fast-import.\n \n+--incremental::\n+\tIf the first revision in dump has number greater than\n+\t1, make :(revision - 1) it's parent. For this to work\n+\tfast-import must be supplied with import-marks file\n+\tand the dump must be incremental.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\ndiff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh\nindex efb4925..845e227 100755\n--- a/t/t9010-svn-fe.sh\n+++ b/t/t9010-svn-fe.sh\n@@ -22,11 +22,12 @@ try_dump () {\n \tmaybe_fail_fi=${3:+test_$3} &&\n \targs=${4:-} &&\n \tfd=${5:-3} &&\n+\tfi_args=${6:-} &&\n \n \t{\n \t\teval \"$maybe_fail_svnfe test-svn-fe $args \"$input\" >stream $fdbackflow &&\n+\teval \"$maybe_fail_fi git fast-import $fi_args --cat-blob-fd=3 backflow\" &&\n \twait $!\n }\n \n@@ -71,6 +72,15 @@ Content-length: 10\n \n PROPS-END\n EOF\n+cat >moreempty.dump <<-EOF &&\n+SVN-fs-dump-format-version: 3\n+\n+Revision-number: 3\n+Prop-content-length: 10\n+Content-length: 10\n+\n+PROPS-END\n+EOF\n \n test_expect_success 'setup: have pipes?' '\n \trm -f frob &&\n@@ -1171,4 +1181,18 @@ test_expect_success PIPE 'use different backflow fd' '\n \ttest_cmp hi otherfile\n '\n \n+test_expect_success PIPE 'incremental import' '\n+\treinit_git &&\n+\t>./marks &&\n+\n+\ttry_dump emptyprop.dump \"\" \"\" \"--incremental\" \"\" \"--export-marks=./marks\" &&\n+\ttest_line_count = 2 ./marks &&\n+\n+\ttry_dump moreempty.dump \"\" \"\" \"--incremental\" \"\" \"--import-marks=./marks --export-marks=./marks\" &&\n+\ttest_line_count = 3 ./marks &&\n+\n+\tgit log --format=oneline >history &&\n+\ttest_line_count = 3 ./history\n+'\n+\n test_done\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex e51a9bb..206827e 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -23,6 +23,9 @@ static struct option test_svnfe_options[] = {\n \tOPT_SET_INT('d', \"apply-delta\", &delta_test, \"test apply_delta\", 1),\n \tOPT_STRING(0, \"ref\", &options.ref, \"refname\",\n \t\t\"write to instead of refs/heads/master\"),\n+\tOPT_SET_INT(0, \"incremental\", &options.incremental,\n+\t\t\"resume export, requires marks and incremental dump\",\n+\t\t1),\n \tOPT_INTEGER(0, \"read-blob-fd\", &options.backflow_fd,\n \t\t\"read blobs and trees from this fd instead of 3\"),\n \tOPT_END()\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex ec68649..75ea49f 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -58,6 +58,7 @@ static struct {\n \tstruct strbuf uuid, url;\n \tint first_commit_done;\n \tstruct strbuf ref_name;\n+\tint incremental;\n } dump_ctx;\n \n static void reset_node_ctx(char *fname)\n@@ -83,7 +84,7 @@ static void reset_rev_ctx(uint32_t revision)\n \tstrbuf_reset(&rev_ctx.author);\n }\n \n-static void reset_dump_ctx(const char *url, const char *dst_ref)\n+static void reset_dump_ctx(const char *url, const char *dst_ref, int incremental)\n {\n \tstrbuf_reset(&dump_ctx.url);\n \tif (url)\n@@ -93,6 +94,7 @@ static void reset_dump_ctx(const char *url, const char *dst_ref)\n \tdump_ctx.first_commit_done = 0;\n \tstrbuf_reset(&dump_ctx.ref_name);\n \tstrbuf_addstr(&dump_ctx.ref_name, dst_ref);\n+\tdump_ctx.incremental = incremental;\n }\n \n static void handle_property(const struct strbuf *key_buf,\n@@ -322,7 +324,10 @@ static void begin_revision(void)\n \n \tif (!rev_ctx.revision)\t/* revision 0 gets no git commit. */\n \t\treturn;\n-\tprev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;\n+\tif (dump_ctx.incremental)\n+\t\tprev = rev_ctx.revision - 1;\n+\telse\n+\t\tprev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;\n \tif (prev)\n \t\tsnprintf(buf, 32, \":%\"PRIu32, prev);\n \telse\n@@ -508,7 +513,7 @@ int svndump_init(const struct svndump_options *o)\n \tstrbuf_init(&rev_ctx.author, 4096);\n \tstrbuf_init(&node_ctx.src, 4096);\n \tstrbuf_init(&node_ctx.dst, 4096);\n-\treset_dump_ctx(o->git_svn_url, ref);\n+\treset_dump_ctx(o->git_svn_url, ref, o->incremental);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \treturn 0;\n@@ -517,7 +522,7 @@ int svndump_init(const struct svndump_options *o)\n void svndump_deinit(void)\n {\n \tfast_export_deinit();\n-\treset_dump_ctx(NULL, \"\");\n+\treset_dump_ctx(NULL, \"\", 0);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \tstrbuf_release(&rev_ctx.log);\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex de54969..32062e4 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -8,7 +8,7 @@ struct svndump_options {\n \tconst char *dumpfile, *git_svn_url;\n \tconst char *ref;\n \tint backflow_fd;\n-\tint progress;\n+\tint progress, incremental;\n };\n \n int svndump_init(const struct svndump_options *o);\n-- \n1.7.3.4\n"},{"id":"173586","messageId":"1313488495-2203-11-git-send-email-divanorama@gmail.com","threadId":"28113","inReplyTo":"1313488495-2203-1-git-send-email-divanorama@gmail.com","subject":"[PATCH v3 10/10] vcs-svn: add fast_export_note to create notes","fromName":"Dmitry Ivankov","fromEmail":"divanorama@gmail.com","sentAt":"2011-08-16T09:54:55Z","receivedAt":"2011-08-16T09:54:55Z","isPatch":true,"sender":{"key":"divanorama@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158999?v=4"},"body":"fast_export lacked a method to writes notes to fast-import stream.\nAdd two new functions fast_export_note which is similar to\nfast_export_modify. And also add fast_export_buf_to_data to be able\nto write inline blobs that don't come from a line_buffer or from delta\napplication.\n\nTo be used like this:\nfast_export_begin_commit(\"refs/notes/somenotes\", ...)\n\nfast_export_note(\"refs/heads/master\", \"inline\")\nfast_export_buf_to_data(&data)\nor maybe\nfast_export_note(\"refs/heads/master\", sha1)\n\nSigned-off-by: Dmitry Ivankov \n---\n vcs-svn/fast_export.c | 11 +++++++++++\n vcs-svn/fast_export.h | 2 ++\n 2 files changed, 13 insertions(+), 0 deletions(-)\n\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 3dfccd2..07220b0 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -69,6 +69,10 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n \tputchar('\\n');\n }\n \n+void fast_export_note(const char *committish, const char *dataref) {\n+\tprintf(\"N %s %s\\n\", dataref, committish);\n+}\n+\n void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,\n \t\t\tconst char *author_name, const char *author_email,\n \t\t\tconst struct strbuf *log, unsigned long timestamp)\n@@ -206,6 +210,13 @@ static long apply_delta(off_t len, struct line_buffer *input,\n \treturn ret;\n }\n \n+void fast_export_buf_to_data(const struct strbuf *data)\n+{\n+\tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t)data->len);\n+\tfwrite(data->buf, data->len, 1, stdout);\n+\tfputc('\\n', stdout);\n+}\n+\n void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input)\n {\n \tif (mode == REPO_MODE_LNK) {\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex bf58880..d17e5d3 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -10,10 +10,12 @@ void fast_export_reset(void);\n \n void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n+void fast_export_note(const char *committish, const char *dataref);\n void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,\n \t\t\tconst char *author_name, const char *author_email,\n \t\t\tconst struct strbuf *log, unsigned long timestamp);\n void fast_export_progress(uint32_t revision);\n+void fast_export_buf_to_data(const struct strbuf *data);\n void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input);\n void fast_export_blob_delta(uint32_t mode,\n \t\t\tuint32_t old_mode, const char *old_data,\n-- \n1.7.3.4\n"}]}