{"thread":{"id":"31371","subject":"[PATCH v7 00/16] GSOC remote-svn","startedAt":"2012-08-28T08:49:34Z","lastAt":"2012-08-28T18:01:56Z","messageCount":26,"participants":["Florian Achleitner","Junio C Hamano"],"isPatch":true,"patchVersion":7,"patchTotal":16},"messages":[{"id":"197963","messageId":"1346143790-23491-1-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":null,"subject":"[PATCH v7 00/16] GSOC remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:34Z","receivedAt":"2012-08-28T08:49:34Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Reroll includes fixups by Ramsey. Thanks!\nDiff:\n- Add missing dependency to rule in Makefile.\n- improve compatibility of integer types.\n- t9020-*.sh: remove excess slash in urls that makes python on windows \n interpret it as a network path.\n- t9020-*.sh: skip if python isn't available.\n- replace getline() in remote-testsvn.c. There are platforms that don't provide\n this function.\n\n[PATCH v7 01/16] Implement a remote helper for svn in C\n[PATCH v7 02/16] Add git-remote-testsvn to Makefile\n[PATCH v7 03/16] Add svndump_init_fd to allow reading dumps from\n[PATCH v7 04/16] Add argv_array_detach and argv_array_free_detached\n[PATCH v7 05/16] Connect fast-import to the remote-helper via pipe,\n[PATCH v7 06/16] Add documentation for the 'bidi-import' capability\n[PATCH v7 07/16] When debug==1, start fast-import with \"--stats\"\n[PATCH v7 08/16] remote-svn, vcs-svn: Enable fetching to private\n[PATCH v7 09/16] Allow reading svn dumps from files via file:// urls\n[PATCH v7 10/16] vcs-svn: add fast_export_note to create notes\n[PATCH v7 11/16] Create a note for every imported commit containing\n[PATCH v7 12/16] remote-svn: Activate import/export-marks for\n[PATCH v7 13/16] remote-svn: add incremental import\n[PATCH v7 14/16] Add a svnrdump-simulator replaying a dump file for\n[PATCH v7 15/16] remote-svn: add marks-file regeneration\n[PATCH v7 16/16] Add a test script for remote-svn\n"},{"id":"197973","messageId":"1346143790-23491-2-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-1-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 01/16] Implement a remote helper for svn in C","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:35Z","receivedAt":"2012-08-28T08:49:35Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Enable basic fetching from subversion repositories. When processing\nremote URLs starting with testsvn::, git invokes this remote-helper.\nIt starts svnrdump to extract revisions from the subversion repository\nin the 'dump file format', and converts them to a git-fast-import stream\nusing the functions of vcs-svn/.\n\nImported refs are created in a private namespace at\nrefs/svn//master. The revision history is imported\nlinearly (no branch detection) and completely, i.e. from revision 0 to\nHEAD.\n\nThe 'bidi-import' capability is used. The remote-helper expects data\nfrom fast-import on its stdin. It buffers a batch of 'import' command\nlines in a string_list before starting to process them.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n remote-testsvn.c | 174 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 174 insertions(+)\n create mode 100644 remote-testsvn.c\n\ndiff --git a/remote-testsvn.c b/remote-testsvn.c\nnew file mode 100644\nindex 0000000..ebe803b\n--- /dev/null\n+++ b/remote-testsvn.c\n@@ -0,0 +1,174 @@\n+#include \"cache.h\"\n+#include \"remote.h\"\n+#include \"strbuf.h\"\n+#include \"url.h\"\n+#include \"exec_cmd.h\"\n+#include \"run-command.h\"\n+#include \"vcs-svn/svndump.h\"\n+#include \"notes.h\"\n+#include \"argv-array.h\"\n+\n+static const char *url;\n+static const char *private_ref;\n+static const char *remote_ref = \"refs/heads/master\";\n+\n+static int cmd_capabilities(const char *line);\n+static int cmd_import(const char *line);\n+static int cmd_list(const char *line);\n+\n+typedef int (*input_command_handler)(const char *);\n+struct input_command_entry {\n+\tconst char *name;\n+\tinput_command_handler fn;\n+\tunsigned char batchable;\t/* whether the command starts or is part of a batch */\n+};\n+\n+static const struct input_command_entry input_command_list[] = {\n+\t{ \"capabilities\", cmd_capabilities, 0 },\n+\t{ \"import\", cmd_import, 1 },\n+\t{ \"list\", cmd_list, 0 },\n+\t{ NULL, NULL }\n+};\n+\n+static int cmd_capabilities(const char *line) {\n+\tprintf(\"import\\n\");\n+\tprintf(\"bidi-import\\n\");\n+\tprintf(\"refspec %s:%s\\n\\n\", remote_ref, private_ref);\n+\tfflush(stdout);\n+\treturn 0;\n+}\n+\n+static void terminate_batch(void)\n+{\n+\t/* terminate a current batch's fast-import stream */\n+\tprintf(\"done\\n\");\n+\tfflush(stdout);\n+}\n+\n+static int cmd_import(const char *line)\n+{\n+\tint code;\n+\tint dumpin_fd;\n+\tunsigned int startrev = 0;\n+\tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n+\tstruct child_process svndump_proc;\n+\n+\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n+\tsvndump_proc.out = -1;\n+\targv_array_push(&svndump_argv, \"svnrdump\");\n+\targv_array_push(&svndump_argv, \"dump\");\n+\targv_array_push(&svndump_argv, url);\n+\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n+\tsvndump_proc.argv = svndump_argv.argv;\n+\n+\tcode = start_command(&svndump_proc);\n+\tif (code)\n+\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n+\tdumpin_fd = svndump_proc.out;\n+\n+\tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n+\tsvndump_read(url, private_ref);\n+\tsvndump_deinit();\n+\tsvndump_reset();\n+\n+\tclose(dumpin_fd);\n+\tcode = finish_command(&svndump_proc);\n+\tif (code)\n+\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n+\targv_array_clear(&svndump_argv);\n+\n+\treturn 0;\n+}\n+\n+static int cmd_list(const char *line)\n+{\n+\tprintf(\"? %s\\n\\n\", remote_ref);\n+\tfflush(stdout);\n+\treturn 0;\n+}\n+\n+static int do_command(struct strbuf *line)\n+{\n+\tconst struct input_command_entry *p = input_command_list;\n+\tstatic struct string_list batchlines = STRING_LIST_INIT_DUP;\n+\tstatic const struct input_command_entry *batch_cmd;\n+\t/*\n+\t * commands can be grouped together in a batch.\n+\t * Batches are ended by \\n. If no batch is active the program ends.\n+\t * During a batch all lines are buffered and passed to the handler function\n+\t * when the batch is terminated.\n+\t */\n+\tif (line->len == 0) {\n+\t\tif (batch_cmd) {\n+\t\t\tstruct string_list_item *item;\n+\t\t\tfor_each_string_list_item(item, &batchlines)\n+\t\t\t\tbatch_cmd->fn(item->string);\n+\t\t\tterminate_batch();\n+\t\t\tbatch_cmd = NULL;\n+\t\t\tstring_list_clear(&batchlines, 0);\n+\t\t\treturn 0;\t/* end of the batch, continue reading other commands. */\n+\t\t}\n+\t\treturn 1;\t/* end of command stream, quit */\n+\t}\n+\tif (batch_cmd) {\n+\t\tif (prefixcmp(batch_cmd->name, line->buf))\n+\t\t\tdie(\"Active %s batch interrupted by %s\", batch_cmd->name, line->buf);\n+\t\t/* buffer batch lines */\n+\t\tstring_list_append(&batchlines, line->buf);\n+\t\treturn 0;\n+\t}\n+\n+\tfor (p = input_command_list; p->name; p++) {\n+\t\tif (!prefixcmp(line->buf, p->name) && (strlen(p->name) == line->len ||\n+\t\t\t\tline->buf[strlen(p->name)] == ' ')) {\n+\t\t\tif (p->batchable) {\n+\t\t\t\tbatch_cmd = p;\n+\t\t\t\tstring_list_append(&batchlines, line->buf);\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t\treturn p->fn(line->buf);\n+\t\t}\n+\t}\n+\tdie(\"Unknown command '%s'\\n\", line->buf);\n+\treturn 0;\n+}\n+\n+int main(int argc, const char **argv)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tstatic struct remote *remote;\n+\tconst char *url_in;\n+\n+\tgit_extract_argv0_path(argv[0]);\n+\tsetup_git_directory();\n+\tif (argc < 2 || argc > 3) {\n+\t\tusage(\"git-remote-svn []\");\n+\t\treturn 1;\n+\t}\n+\n+\tremote = remote_get(argv[1]);\n+\turl_in = (argc == 3) ? argv[2] : remote->url[0];\n+\n+\tend_url_with_slash(&buf, url_in);\n+\turl = strbuf_detach(&buf, NULL);\n+\n+\tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n+\tprivate_ref = strbuf_detach(&buf, NULL);\n+\n+\twhile(1) {\n+\t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF) {\n+\t\t\tif (ferror(stdin))\n+\t\t\t\tdie(\"Error reading command stream\");\n+\t\t\telse\n+\t\t\t\tdie(\"Unexpected end of command stream\");\n+\t\t}\n+\t\tif (do_command(&buf))\n+\t\t\tbreak;\n+\t\tstrbuf_reset(&buf);\n+\t}\n+\n+\tstrbuf_release(&buf);\n+\tfree((void*)url);\n+\tfree((void*)private_ref);\n+\treturn 0;\n+}\n-- \n1.7.9.5\n"},{"id":"197971","messageId":"1346143790-23491-3-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-2-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 02/16] Add git-remote-testsvn to Makefile","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:36Z","receivedAt":"2012-08-28T08:49:36Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"The link-rule is a copy of the standard git$X rule but adds VCSSVN_LIB.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n Makefile | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 66e8216..1b09454 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -477,6 +477,7 @@ PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n PROGRAM_OBJS += upload-pack.o\n+PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n X =\n@@ -2352,6 +2353,10 @@ git-http-push$X: revision.o http.o http-push.o GIT-LDFLAGS $(GITLIBS)\n \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n \t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n \n+git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n+\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n+\t$(VCSSVN_LIB)\n+\n $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n \t$(QUIET_LNCP)$(RM) $@ && \\\n \tln $< $@ 2>/dev/null || \\\n-- \n1.7.9.5\n"},{"id":"197968","messageId":"1346143790-23491-4-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-3-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 03/16] Add svndump_init_fd to allow reading dumps from arbitrary FDs","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:37Z","receivedAt":"2012-08-28T08:49:37Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"The existing function only allows reading from a filename or from\nstdin. Allow passing of a FD and an additional FD for the back report\npipe. This allows us to retrieve the name of the pipe in the caller.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n vcs-svn/svndump.c | 22 ++++++++++++++++++----\n vcs-svn/svndump.h | 1 +\n 2 files changed, 19 insertions(+), 4 deletions(-)\n\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 2b168ae..d81a078 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -468,11 +468,9 @@ void svndump_read(const char *url)\n \t\tend_revision();\n }\n \n-int svndump_init(const char *filename)\n+static void init(int report_fd)\n {\n-\tif (buffer_init(&input, filename))\n-\t\treturn error(\"cannot open %s: %s\", filename, strerror(errno));\n-\tfast_export_init(REPORT_FILENO);\n+\tfast_export_init(report_fd);\n \tstrbuf_init(&dump_ctx.uuid, 4096);\n \tstrbuf_init(&dump_ctx.url, 4096);\n \tstrbuf_init(&rev_ctx.log, 4096);\n@@ -482,6 +480,22 @@ int svndump_init(const char *filename)\n \treset_dump_ctx(NULL);\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n+\treturn;\n+}\n+\n+int svndump_init(const char *filename)\n+{\n+\tif (buffer_init(&input, filename))\n+\t\treturn error(\"cannot open %s: %s\", filename ? filename : \"NULL\", strerror(errno));\n+\tinit(REPORT_FILENO);\n+\treturn 0;\n+}\n+\n+int svndump_init_fd(int in_fd, int back_fd)\n+{\n+\tif(buffer_fdinit(&input, xdup(in_fd)))\n+\t\treturn error(\"cannot open fd %d: %s\", in_fd, strerror(errno));\n+\tinit(xdup(back_fd));\n \treturn 0;\n }\n \ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex df9ceb0..acb5b47 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -2,6 +2,7 @@\n #define SVNDUMP_H_\n \n int svndump_init(const char *filename);\n+int svndump_init_fd(int in_fd, int back_fd);\n void svndump_read(const char *url);\n void svndump_deinit(void);\n void svndump_reset(void);\n-- \n1.7.9.5\n"},{"id":"197977","messageId":"1346143790-23491-5-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-4-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 04/16] Add argv_array_detach and argv_array_free_detached","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:38Z","receivedAt":"2012-08-28T08:49:38Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Allow detaching of ownership of the argv_array's contents and add a\nfunction to free those detached argv_arrays later.\n\nThis makes it possible to use argv_array efficiently with the exiting\nstruct child_process which only contains a member char **argv.\n\nAdd to documentation.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n Documentation/technical/api-argv-array.txt | 8 ++++++++\n argv-array.c | 20 ++++++++++++++++++++\n argv-array.h | 2 ++\n 3 files changed, 30 insertions(+)\n\ndiff --git a/Documentation/technical/api-argv-array.txt b/Documentation/technical/api-argv-array.txt\nindex 1b7d8f1..6b97d6d 100644\n--- a/Documentation/technical/api-argv-array.txt\n+++ b/Documentation/technical/api-argv-array.txt\n@@ -49,3 +49,11 @@ Functions\n `argv_array_clear`::\n \tFree all memory associated with the array and return it to the\n \tinitial, empty state.\n+\n+`argv_array_detach`::\n+\tDetach the argv array from the `struct argv_array`, transfering\n+\townership of the allocated array and strings.\n+\n+`argv_array_free_detached`::\n+\tFree the memory allocated by a `struct argv_array` that was later\n+\tdetached and is now no longer needed.\ndiff --git a/argv-array.c b/argv-array.c\nindex 0b5f889..aab50d6 100644\n--- a/argv-array.c\n+++ b/argv-array.c\n@@ -59,3 +59,23 @@ void argv_array_clear(struct argv_array *array)\n \t}\n \targv_array_init(array);\n }\n+\n+const char **argv_array_detach(struct argv_array *array, int *argc)\n+{\n+\tconst char **argv =\n+\t\tarray->argv == empty_argv || array->argc == 0 ? NULL : array->argv;\n+\tif (argc)\n+\t\t*argc = array->argc;\n+\targv_array_init(array);\n+\treturn argv;\n+}\n+\n+void argv_array_free_detached(const char **argv)\n+{\n+\tif (argv) {\n+\t\tint i;\n+\t\tfor (i = 0; argv[i]; i++)\n+\t\t\tfree((char **)argv[i]);\n+\t\tfree(argv);\n+\t}\n+}\ndiff --git a/argv-array.h b/argv-array.h\nindex b93a69c..b3ef351 100644\n--- a/argv-array.h\n+++ b/argv-array.h\n@@ -17,5 +17,7 @@ __attribute__((format (printf,2,3)))\n void argv_array_pushf(struct argv_array *, const char *fmt, ...);\n void argv_array_pushl(struct argv_array *, ...);\n void argv_array_clear(struct argv_array *);\n+const char **argv_array_detach(struct argv_array *array, int *argc);\n+void argv_array_free_detached(const char **argv);\n \n #endif /* ARGV_ARRAY_H */\n-- \n1.7.9.5\n"},{"id":"197976","messageId":"1346143790-23491-6-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-5-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 05/16] Connect fast-import to the remote-helper via pipe, adding 'bidi-import' capability","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:39Z","receivedAt":"2012-08-28T08:49:39Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"The fast-import commands 'cat-blob' and 'ls' can be used by\nremote-helpers to retrieve information about blobs and trees that\nalready exist in fast-import's memory. This requires a channel from\nfast-import to the remote-helper.\n\nremote-helpers that use these features shall advertise the new\n'bidi-import' capability to signal that they require the communication\nchannel. When forking fast-import in transport-helper.c connect it to\na dup of the remote-helper's stdin-pipe. The additional file\ndescriptor is passed to fast-import via its command line\n(--cat-blob-fd). It follows that git and fast-import are connected to\nthe remote-helpers's stdin.\n\nBecause git can send multiple commands to the remote-helper on it's\nstdin, it is required that helpers that advertise 'bidi-import' buffer\nall input commands until the batch of 'import' commands is ended by a\nnewline before sending data to fast-import. This is to prevent mixing\ncommands and fast-import responses on the helper's stdin.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n transport-helper.c | 44 +++++++++++++++++++++++++++++++-------------\n 1 file changed, 31 insertions(+), 13 deletions(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex cfe0988..3523f1f 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -10,6 +10,7 @@\n #include \"string-list.h\"\n #include \"thread-utils.h\"\n #include \"sigchain.h\"\n+#include \"argv-array.h\"\n \n static int debug;\n \n@@ -19,6 +20,7 @@ struct helper_data {\n \tFILE *out;\n \tunsigned fetch : 1,\n \t\timport : 1,\n+\t\tbidi_import : 1,\n \t\texport : 1,\n \t\toption : 1,\n \t\tpush : 1,\n@@ -101,6 +103,7 @@ static void do_take_over(struct transport *transport)\n static struct child_process *get_helper(struct transport *transport)\n {\n \tstruct helper_data *data = transport->data;\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct child_process *helper;\n \tconst char **refspecs = NULL;\n@@ -122,11 +125,10 @@ static struct child_process *get_helper(struct transport *transport)\n \thelper->in = -1;\n \thelper->out = -1;\n \thelper->err = 0;\n-\thelper->argv = xcalloc(4, sizeof(*helper->argv));\n-\tstrbuf_addf(&buf, \"git-remote-%s\", data->name);\n-\thelper->argv[0] = strbuf_detach(&buf, NULL);\n-\thelper->argv[1] = transport->remote->name;\n-\thelper->argv[2] = remove_ext_force(transport->url);\n+\targv_array_pushf(&argv, \"git-remote-%s\", data->name);\n+\targv_array_push(&argv, transport->remote->name);\n+\targv_array_push(&argv, remove_ext_force(transport->url));\n+\thelper->argv = argv_array_detach(&argv, NULL);\n \thelper->git_cmd = 0;\n \thelper->silent_exec_failure = 1;\n \n@@ -178,6 +180,8 @@ static struct child_process *get_helper(struct transport *transport)\n \t\t\tdata->push = 1;\n \t\telse if (!strcmp(capname, \"import\"))\n \t\t\tdata->import = 1;\n+\t\telse if (!strcmp(capname, \"bidi-import\"))\n+\t\t\tdata->bidi_import = 1;\n \t\telse if (!strcmp(capname, \"export\"))\n \t\t\tdata->export = 1;\n \t\telse if (!data->refspecs && !prefixcmp(capname, \"refspec \")) {\n@@ -241,8 +245,7 @@ static int disconnect_helper(struct transport *transport)\n \t\tclose(data->helper->out);\n \t\tfclose(data->out);\n \t\tres = finish_command(data->helper);\n-\t\tfree((char *)data->helper->argv[0]);\n-\t\tfree(data->helper->argv);\n+\t\targv_array_free_detached(data->helper->argv);\n \t\tfree(data->helper);\n \t\tdata->helper = NULL;\n \t}\n@@ -376,14 +379,23 @@ static int fetch_with_fetch(struct transport *transport,\n static int get_importer(struct transport *transport, struct child_process *fastimport)\n {\n \tstruct child_process *helper = get_helper(transport);\n+\tstruct helper_data *data = transport->data;\n+\tstruct argv_array argv = ARGV_ARRAY_INIT;\n+\tint cat_blob_fd, code;\n \tmemset(fastimport, 0, sizeof(*fastimport));\n \tfastimport->in = helper->out;\n-\tfastimport->argv = xcalloc(5, sizeof(*fastimport->argv));\n-\tfastimport->argv[0] = \"fast-import\";\n-\tfastimport->argv[1] = \"--quiet\";\n+\targv_array_push(&argv, \"fast-import\");\n+\targv_array_push(&argv, \"--quiet\");\n \n+\tif (data->bidi_import) {\n+\t\tcat_blob_fd = xdup(helper->in);\n+\t\targv_array_pushf(&argv, \"--cat-blob-fd=%d\", cat_blob_fd);\n+\t}\n+\tfastimport->argv = argv.argv;\n \tfastimport->git_cmd = 1;\n-\treturn start_command(fastimport);\n+\n+\tcode = start_command(fastimport);\n+\treturn code;\n }\n \n static int get_exporter(struct transport *transport,\n@@ -438,11 +450,17 @@ static int fetch_with_import(struct transport *transport,\n \t}\n \n \twrite_constant(data->helper->in, \"\\n\");\n+\t/*\n+\t * remote-helpers that advertise the bidi-import capability are required to\n+\t * buffer the complete batch of import commands until this newline before\n+\t * sending data to fast-import.\n+\t * These helpers read back data from fast-import on their stdin, which could\n+\t * be mixed with import commands, otherwise.\n+\t */\n \n \tif (finish_command(&fastimport))\n \t\tdie(\"Error while running fast-import\");\n-\tfree(fastimport.argv);\n-\tfastimport.argv = NULL;\n+\targv_array_free_detached(fastimport.argv);\n \n \t/*\n \t * The fast-import stream of a remote helper that advertises\n-- \n1.7.9.5\n"},{"id":"197966","messageId":"1346143790-23491-7-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-6-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 06/16] Add documentation for the 'bidi-import' capability of remote-helpers","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:40Z","receivedAt":"2012-08-28T08:49:40Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Signed-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-remote-helpers.txt | 21 ++++++++++++++++++++-\n 1 file changed, 20 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt\nindex f5836e4..5ce4cda 100644\n--- a/Documentation/git-remote-helpers.txt\n+++ b/Documentation/git-remote-helpers.txt\n@@ -98,6 +98,20 @@ advertised with this capability must cover all refs reported by\n the list command. If no 'refspec' capability is advertised,\n there is an implied `refspec *:*`.\n \n+'bidi-import'::\n+\tThe fast-import commands 'cat-blob' and 'ls' can be used by remote-helpers\n+\tto retrieve information about blobs and trees that already exist in\n+\tfast-import's memory. This requires a channel from fast-import to the\n+\tremote-helper.\n+\tIf it is advertised in addition to \"import\", git establishes a pipe from\n+\tfast-import to the remote-helper's stdin.\n+\tIt follows that git and fast-import are both connected to the\n+\tremote-helper's stdin. Because git can send multiple commands to\n+\tthe remote-helper it is required that helpers that use 'bidi-import'\n+\tbuffer all 'import' commands of a batch before sending data to fast-import.\n+\tThis is to prevent mixing commands and fast-import responses on the\n+\thelper's stdin.\n+\n Capabilities for Pushing\n ~~~~~~~~~~~~~~~~~~~~~~~~\n 'connect'::\n@@ -286,7 +300,12 @@ terminated with a blank line. For each batch of 'import', the remote\n helper should produce a fast-import stream terminated by a 'done'\n command.\n +\n-Supported if the helper has the \"import\" capability.\n+Note that if the 'bidi-import' capability is used the complete batch\n+sequence has to be buffered before starting to send data to fast-import\n+to prevent mixing of commands and fast-import responses on the helper's\n+stdin.\n++\n+Supported if the helper has the 'import' capability.\n \n 'connect' ::\n \tConnects to given service. Standard input and standard output\n-- \n1.7.9.5\n"},{"id":"197964","messageId":"1346143790-23491-8-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-7-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 07/16] When debug==1, start fast-import with \"--stats\" instead of \"--quiet\"","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:41Z","receivedAt":"2012-08-28T08:49:41Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"fast-import prints statistics that could be interesting to the\ndeveloper of remote helpers.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n transport-helper.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 3523f1f..4713b69 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -385,7 +385,7 @@ static int get_importer(struct transport *transport, struct child_process *fasti\n \tmemset(fastimport, 0, sizeof(*fastimport));\n \tfastimport->in = helper->out;\n \targv_array_push(&argv, \"fast-import\");\n-\targv_array_push(&argv, \"--quiet\");\n+\targv_array_push(&argv, debug ? \"--stats\" : \"--quiet\");\n \n \tif (data->bidi_import) {\n \t\tcat_blob_fd = xdup(helper->in);\n-- \n1.7.9.5\n"},{"id":"197975","messageId":"1346143790-23491-9-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-8-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 08/16] remote-svn, vcs-svn: Enable fetching to private refs","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:42Z","receivedAt":"2012-08-28T08:49:42Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"The reference to update by the fast-import stream is hard-coded. When\nfetching from a remote the remote-helper shall update refs in a\nprivate namespace, i.e. a private subdir of refs/. This namespace is\ndefined by the 'refspec' capability, that the remote-helper advertises\nas a reply to the 'capabilities' command.\n\nExtend svndump and fast-export to allow passing the target ref.\nUpdate svn-fe to be compatible.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n contrib/svn-fe/svn-fe.c | 2 +-\n test-svn-fe.c | 2 +-\n vcs-svn/fast_export.c | 4 ++--\n vcs-svn/fast_export.h | 2 +-\n vcs-svn/svndump.c | 14 +++++++-------\n vcs-svn/svndump.h | 2 +-\n 6 files changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex 35db24f..c796cc0 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -10,7 +10,7 @@ int main(int argc, char **argv)\n {\n \tif (svndump_init(NULL))\n \t\treturn 1;\n-\tsvndump_read((argc > 1) ? argv[1] : NULL);\n+\tsvndump_read((argc > 1) ? argv[1] : NULL, \"refs/heads/master\");\n \tsvndump_deinit();\n \tsvndump_reset();\n \treturn 0;\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex 83633a2..cb0d80f 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -40,7 +40,7 @@ int main(int argc, char *argv[])\n \tif (argc == 2) {\n \t\tif (svndump_init(argv[1]))\n \t\t\treturn 1;\n-\t\tsvndump_read(NULL);\n+\t\tsvndump_read(NULL, \"refs/heads/master\");\n \t\tsvndump_deinit();\n \t\tsvndump_reset();\n \t\treturn 0;\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 1f04697..11f8f94 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -72,7 +72,7 @@ static char gitsvnline[MAX_GITSVN_LINE_LEN];\n void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log,\n \t\t\tconst char *uuid, const char *url,\n-\t\t\tunsigned long timestamp)\n+\t\t\tunsigned long timestamp, const char *local_ref)\n {\n \tstatic const struct strbuf empty = STRBUF_INIT;\n \tif (!log)\n@@ -84,7 +84,7 @@ void fast_export_begin_commit(uint32_t revision, const char *author,\n \t} else {\n \t\t*gitsvnline = '\\0';\n \t}\n-\tprintf(\"commit refs/heads/master\\n\");\n+\tprintf(\"commit %s\\n\", local_ref);\n \tprintf(\"mark :%\"PRIu32\"\\n\", revision);\n \tprintf(\"committer %s <%s@%s> %ld +0000\\n\",\n \t\t *author ? author : \"nobody\",\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex 8823aca..17eb13b 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -11,7 +11,7 @@ void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log, const char *uuid,\n-\t\t\tconst char *url, unsigned long timestamp);\n+\t\t\tconst char *url, unsigned long timestamp, const char *local_ref);\n void fast_export_end_commit(uint32_t revision);\n void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input);\n void fast_export_blob_delta(uint32_t mode,\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex d81a078..288bb42 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -299,22 +299,22 @@ static void handle_node(void)\n \t\t\t\tnode_ctx.text_length, &input);\n }\n \n-static void begin_revision(void)\n+static void begin_revision(const char *remote_ref)\n {\n \tif (!rev_ctx.revision)\t/* revision 0 gets no git commit. */\n \t\treturn;\n \tfast_export_begin_commit(rev_ctx.revision, rev_ctx.author.buf,\n \t\t&rev_ctx.log, dump_ctx.uuid.buf, dump_ctx.url.buf,\n-\t\trev_ctx.timestamp);\n+\t\trev_ctx.timestamp, remote_ref);\n }\n \n-static void end_revision(void)\n+static void end_revision()\n {\n \tif (rev_ctx.revision)\n \t\tfast_export_end_commit(rev_ctx.revision);\n }\n \n-void svndump_read(const char *url)\n+void svndump_read(const char *url, const char *local_ref)\n {\n \tchar *val;\n \tchar *t;\n@@ -353,7 +353,7 @@ void svndump_read(const char *url)\n \t\t\tif (active_ctx == NODE_CTX)\n \t\t\t\thandle_node();\n \t\t\tif (active_ctx == REV_CTX)\n-\t\t\t\tbegin_revision();\n+\t\t\t\tbegin_revision(local_ref);\n \t\t\tif (active_ctx != DUMP_CTX)\n \t\t\t\tend_revision();\n \t\t\tactive_ctx = REV_CTX;\n@@ -366,7 +366,7 @@ void svndump_read(const char *url)\n \t\t\t\tif (active_ctx == NODE_CTX)\n \t\t\t\t\thandle_node();\n \t\t\t\tif (active_ctx == REV_CTX)\n-\t\t\t\t\tbegin_revision();\n+\t\t\t\t\tbegin_revision(local_ref);\n \t\t\t\tactive_ctx = NODE_CTX;\n \t\t\t\treset_node_ctx(val);\n \t\t\t\tbreak;\n@@ -463,7 +463,7 @@ void svndump_read(const char *url)\n \tif (active_ctx == NODE_CTX)\n \t\thandle_node();\n \tif (active_ctx == REV_CTX)\n-\t\tbegin_revision();\n+\t\tbegin_revision(local_ref);\n \tif (active_ctx != DUMP_CTX)\n \t\tend_revision();\n }\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex acb5b47..febeecb 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -3,7 +3,7 @@\n \n int svndump_init(const char *filename);\n int svndump_init_fd(int in_fd, int back_fd);\n-void svndump_read(const char *url);\n+void svndump_read(const char *url, const char *local_ref);\n void svndump_deinit(void);\n void svndump_reset(void);\n \n-- \n1.7.9.5\n"},{"id":"197970","messageId":"1346143790-23491-10-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-9-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 09/16] Allow reading svn dumps from files via file:// urls","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:43Z","receivedAt":"2012-08-28T08:49:43Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"For testing as well as for importing large, already available dumps,\nit's useful to bypass svnrdump and replay the svndump from a file\ndirectly.\n\nAdd support for file:// urls in the remote url, e.g.\n\n svn::file:///path/to/dump\n\nWhen the remote helper finds an url starting with file:// it tries to\nopen that file instead of invoking svnrdump.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n remote-testsvn.c | 55 +++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 36 insertions(+), 19 deletions(-)\n\ndiff --git a/remote-testsvn.c b/remote-testsvn.c\nindex ebe803b..2b9d151 100644\n--- a/remote-testsvn.c\n+++ b/remote-testsvn.c\n@@ -9,6 +9,7 @@\n #include \"argv-array.h\"\n \n static const char *url;\n+static int dump_from_file;\n static const char *private_ref;\n static const char *remote_ref = \"refs/heads/master\";\n \n@@ -53,29 +54,38 @@ static int cmd_import(const char *line)\n \tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n \tstruct child_process svndump_proc;\n \n-\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n-\tsvndump_proc.out = -1;\n-\targv_array_push(&svndump_argv, \"svnrdump\");\n-\targv_array_push(&svndump_argv, \"dump\");\n-\targv_array_push(&svndump_argv, url);\n-\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n-\tsvndump_proc.argv = svndump_argv.argv;\n-\n-\tcode = start_command(&svndump_proc);\n-\tif (code)\n-\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n-\tdumpin_fd = svndump_proc.out;\n-\n+\tif (dump_from_file) {\n+\t\tdumpin_fd = open(url, O_RDONLY);\n+\t\tif(dumpin_fd < 0) {\n+\t\t\tdie_errno(\"Couldn't open svn dump file %s.\", url);\n+\t\t}\n+\t}\n+\telse {\n+\t\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n+\t\tsvndump_proc.out = -1;\n+\t\targv_array_push(&svndump_argv, \"svnrdump\");\n+\t\targv_array_push(&svndump_argv, \"dump\");\n+\t\targv_array_push(&svndump_argv, url);\n+\t\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n+\t\tsvndump_proc.argv = svndump_argv.argv;\n+\n+\t\tcode = start_command(&svndump_proc);\n+\t\tif (code)\n+\t\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n+\t\tdumpin_fd = svndump_proc.out;\n+\t}\n \tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n \tsvndump_read(url, private_ref);\n \tsvndump_deinit();\n \tsvndump_reset();\n \n \tclose(dumpin_fd);\n-\tcode = finish_command(&svndump_proc);\n-\tif (code)\n-\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n-\targv_array_clear(&svndump_argv);\n+\tif(!dump_from_file) {\n+\t\tcode = finish_command(&svndump_proc);\n+\t\tif (code)\n+\t\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n+\t\targv_array_clear(&svndump_argv);\n+\t}\n \n \treturn 0;\n }\n@@ -149,8 +159,15 @@ int main(int argc, const char **argv)\n \tremote = remote_get(argv[1]);\n \turl_in = (argc == 3) ? argv[2] : remote->url[0];\n \n-\tend_url_with_slash(&buf, url_in);\n-\turl = strbuf_detach(&buf, NULL);\n+\tif (!prefixcmp(url_in, \"file://\")) {\n+\t\tdump_from_file = 1;\n+\t\turl = url_decode(url_in + sizeof(\"file://\")-1);\n+\t}\n+\telse {\n+\t\tdump_from_file = 0;\n+\t\tend_url_with_slash(&buf, url_in);\n+\t\turl = strbuf_detach(&buf, NULL);\n+\t}\n \n \tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n \tprivate_ref = strbuf_detach(&buf, NULL);\n-- \n1.7.9.5\n"},{"id":"197965","messageId":"1346143790-23491-11-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-10-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 10/16] vcs-svn: add fast_export_note to create notes","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:44Z","receivedAt":"2012-08-28T08:49:44Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"From: Dmitry Ivankov \n\nfast_export lacked a method to writes notes to fast-import stream.\nAdd two new functions fast_export_note which is similar to\nfast_export_modify. And also add fast_export_buf_to_data to be able to\nwrite inline blobs that don't come from a line_buffer or from delta\napplication.\n\nTo be used like this:\n\n fast_export_begin_commit(\"refs/notes/somenotes\", ...)\n fast_export_note(\"refs/heads/master\", \"inline\")\n fast_export_buf_to_data(&data)\n\nor maybe\n\n fast_export_note(\"refs/heads/master\", sha1)\n\nSigned-off-by: Dmitry Ivankov \nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n vcs-svn/fast_export.c | 12 ++++++++++++\n vcs-svn/fast_export.h | 2 ++\n 2 files changed, 14 insertions(+)\n\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 11f8f94..1ecae4b 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -68,6 +68,11 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n \tputchar('\\n');\n }\n \n+void fast_export_note(const char *committish, const char *dataref)\n+{\n+\tprintf(\"N %s %s\\n\", dataref, committish);\n+}\n+\n static char gitsvnline[MAX_GITSVN_LINE_LEN];\n void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log,\n@@ -222,6 +227,13 @@ static long apply_delta(off_t len, struct line_buffer *input,\n \treturn ret;\n }\n \n+void fast_export_buf_to_data(const struct strbuf *data)\n+{\n+\tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t)data->len);\n+\tfwrite(data->buf, data->len, 1, stdout);\n+\tfputc('\\n', stdout);\n+}\n+\n void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input)\n {\n \tassert(len >= 0);\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex 17eb13b..9b32f1e 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -9,11 +9,13 @@ void fast_export_deinit(void);\n \n void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n+void fast_export_note(const char *committish, const char *dataref);\n void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log, const char *uuid,\n \t\t\tconst char *url, unsigned long timestamp, const char *local_ref);\n void fast_export_end_commit(uint32_t revision);\n void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input);\n+void fast_export_buf_to_data(const struct strbuf *data);\n void fast_export_blob_delta(uint32_t mode,\n \t\t\tuint32_t old_mode, const char *old_data,\n \t\t\toff_t len, struct line_buffer *input);\n-- \n1.7.9.5\n"},{"id":"197969","messageId":"1346143790-23491-12-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-11-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 11/16] Create a note for every imported commit containing svn metadata","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:45Z","receivedAt":"2012-08-28T08:49:45Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"To provide metadata from svn dumps for further processing, e.g.\nbranch detection, attach a note to each imported commit that stores\nadditional information. The notes are currently hard-coded in\nrefs/notes/svn/revs. Currently the following lines from the svn dump\nare directly accumulated in the note. This can be refined as needed.\n\n - \"Revision-number\"\n - \"Node-path\"\n - \"Node-kind\"\n - \"Node-action\"\n - \"Node-copyfrom-path\"\n - \"Node-copyfrom-rev\"\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n vcs-svn/fast_export.c | 14 ++++++++++++--\n vcs-svn/fast_export.h | 2 ++\n vcs-svn/svndump.c | 21 +++++++++++++++++++--\n 3 files changed, 33 insertions(+), 4 deletions(-)\n\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 1ecae4b..df51c59 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -3,8 +3,7 @@\n * See LICENSE for details.\n */\n \n-#include \"git-compat-util.h\"\n-#include \"strbuf.h\"\n+#include \"cache.h\"\n #include \"quote.h\"\n #include \"fast_export.h\"\n #include \"repo_tree.h\"\n@@ -68,6 +67,17 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n \tputchar('\\n');\n }\n \n+void fast_export_begin_note(uint32_t revision, const char *author,\n+\t\tconst char *log, unsigned long timestamp)\n+{\n+\tsize_t loglen = strlen(log);\n+\tprintf(\"commit refs/notes/svn/revs\\n\");\n+\tprintf(\"committer %s <%s@%s> %ld +0000\\n\", author, author, \"local\", timestamp);\n+\tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t)loglen);\n+\tfwrite(log, loglen, 1, stdout);\n+\tfputc('\\n', stdout);\n+}\n+\n void fast_export_note(const char *committish, const char *dataref)\n {\n \tprintf(\"N %s %s\\n\", dataref, committish);\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex 9b32f1e..c2f6f11 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -10,6 +10,8 @@ void fast_export_deinit(void);\n void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n void fast_export_note(const char *committish, const char *dataref);\n+void fast_export_begin_note(uint32_t revision, const char *author,\n+\t\tconst char *log, unsigned long timestamp);\n void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log, const char *uuid,\n \t\t\tconst char *url, unsigned long timestamp, const char *local_ref);\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 288bb42..cd65b51 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -48,7 +48,7 @@ static struct {\n static struct {\n \tuint32_t revision;\n \tunsigned long timestamp;\n-\tstruct strbuf log, author;\n+\tstruct strbuf log, author, note;\n } rev_ctx;\n \n static struct {\n@@ -77,6 +77,7 @@ static void reset_rev_ctx(uint32_t revision)\n \trev_ctx.timestamp = 0;\n \tstrbuf_reset(&rev_ctx.log);\n \tstrbuf_reset(&rev_ctx.author);\n+\tstrbuf_reset(&rev_ctx.note);\n }\n \n static void reset_dump_ctx(const char *url)\n@@ -310,8 +311,15 @@ static void begin_revision(const char *remote_ref)\n \n static void end_revision()\n {\n-\tif (rev_ctx.revision)\n+\tstruct strbuf mark = STRBUF_INIT;\n+\tif (rev_ctx.revision) {\n \t\tfast_export_end_commit(rev_ctx.revision);\n+\t\tfast_export_begin_note(rev_ctx.revision, \"remote-svn\",\n+\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp);\n+\t\tstrbuf_addf(&mark, \":%\"PRIu32, rev_ctx.revision);\n+\t\tfast_export_note(mark.buf, \"inline\");\n+\t\tfast_export_buf_to_data(&rev_ctx.note);\n+\t}\n }\n \n void svndump_read(const char *url, const char *local_ref)\n@@ -358,6 +366,7 @@ void svndump_read(const char *url, const char *local_ref)\n \t\t\t\tend_revision();\n \t\t\tactive_ctx = REV_CTX;\n \t\t\treset_rev_ctx(atoi(val));\n+\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Node-path\"):\n \t\t\tif (constcmp(t, \"Node-\"))\n@@ -369,10 +378,12 @@ void svndump_read(const char *url, const char *local_ref)\n \t\t\t\t\tbegin_revision(local_ref);\n \t\t\t\tactive_ctx = NODE_CTX;\n \t\t\t\treset_node_ctx(val);\n+\t\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tif (constcmp(t + strlen(\"Node-\"), \"kind\"))\n \t\t\t\tcontinue;\n+\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tif (!strcmp(val, \"dir\"))\n \t\t\t\tnode_ctx.type = REPO_MODE_DIR;\n \t\t\telse if (!strcmp(val, \"file\"))\n@@ -383,6 +394,7 @@ void svndump_read(const char *url, const char *local_ref)\n \t\tcase sizeof(\"Node-action\"):\n \t\t\tif (constcmp(t, \"Node-action\"))\n \t\t\t\tcontinue;\n+\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tif (!strcmp(val, \"delete\")) {\n \t\t\t\tnode_ctx.action = NODEACT_DELETE;\n \t\t\t} else if (!strcmp(val, \"add\")) {\n@@ -401,11 +413,13 @@ void svndump_read(const char *url, const char *local_ref)\n \t\t\t\tcontinue;\n \t\t\tstrbuf_reset(&node_ctx.src);\n \t\t\tstrbuf_addstr(&node_ctx.src, val);\n+\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Node-copyfrom-rev\"):\n \t\t\tif (constcmp(t, \"Node-copyfrom-rev\"))\n \t\t\t\tcontinue;\n \t\t\tnode_ctx.srcRev = atoi(val);\n+\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Text-content-length\"):\n \t\t\tif (constcmp(t, \"Text\") && constcmp(t, \"Prop\"))\n@@ -475,6 +489,7 @@ static void init(int report_fd)\n \tstrbuf_init(&dump_ctx.url, 4096);\n \tstrbuf_init(&rev_ctx.log, 4096);\n \tstrbuf_init(&rev_ctx.author, 4096);\n+\tstrbuf_init(&rev_ctx.note, 4096);\n \tstrbuf_init(&node_ctx.src, 4096);\n \tstrbuf_init(&node_ctx.dst, 4096);\n \treset_dump_ctx(NULL);\n@@ -506,6 +521,8 @@ void svndump_deinit(void)\n \treset_rev_ctx(0);\n \treset_node_ctx(NULL);\n \tstrbuf_release(&rev_ctx.log);\n+\tstrbuf_release(&rev_ctx.author);\n+\tstrbuf_release(&rev_ctx.note);\n \tstrbuf_release(&node_ctx.src);\n \tstrbuf_release(&node_ctx.dst);\n \tif (buffer_deinit(&input))\n-- \n1.7.9.5\n"},{"id":"197967","messageId":"1346143790-23491-13-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-12-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 12/16] remote-svn: Activate import/export-marks for fast-import","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:46Z","receivedAt":"2012-08-28T08:49:46Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Enable import and export of a marks file by sending the appropriate\nfeature commands to fast-import before sending data.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n remote-testsvn.c | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/remote-testsvn.c b/remote-testsvn.c\nindex 2b9d151..b6e7968 100644\n--- a/remote-testsvn.c\n+++ b/remote-testsvn.c\n@@ -12,6 +12,7 @@ static const char *url;\n static int dump_from_file;\n static const char *private_ref;\n static const char *remote_ref = \"refs/heads/master\";\n+static const char *marksfilename;\n \n static int cmd_capabilities(const char *line);\n static int cmd_import(const char *line);\n@@ -74,6 +75,10 @@ static int cmd_import(const char *line)\n \t\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n \t\tdumpin_fd = svndump_proc.out;\n \t}\n+\t/* setup marks file import/export */\n+\tprintf(\"feature import-marks-if-exists=%s\\n\"\n+\t\t\t\"feature export-marks=%s\\n\", marksfilename, marksfilename);\n+\n \tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n \tsvndump_read(url, private_ref);\n \tsvndump_deinit();\n@@ -172,6 +177,10 @@ int main(int argc, const char **argv)\n \tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n \tprivate_ref = strbuf_detach(&buf, NULL);\n \n+\tstrbuf_addf(&buf, \"%s/info/fast-import/remote-svn/%s.marks\",\n+\t\tget_git_dir(), remote->name);\n+\tmarksfilename = strbuf_detach(&buf, NULL);\n+\n \twhile(1) {\n \t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF) {\n \t\t\tif (ferror(stdin))\n@@ -187,5 +196,6 @@ int main(int argc, const char **argv)\n \tstrbuf_release(&buf);\n \tfree((void*)url);\n \tfree((void*)private_ref);\n+\tfree((void*)marksfilename);\n \treturn 0;\n }\n-- \n1.7.9.5\n"},{"id":"197972","messageId":"1346143790-23491-14-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-13-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 13/16] remote-svn: add incremental import","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:47Z","receivedAt":"2012-08-28T08:49:47Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Search for a note attached to the ref to update and read it's\n'Revision-number:'-line. Start import from the next svn revision.\n\nIf there is no next revision in the svn repo, svnrdump terminates with\na message on stderr an non-zero return value. This looks a little\nweird, but there is no other way to know whether there is a new\nrevision in the svn repo.\n\nOn the start of an incremental import, the parent of the first commit\nin the fast-import stream is set to the branch name to update. All\nfollowing commits specify their parent by a mark number. Previous mark\nfiles are currently not reused.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n contrib/svn-fe/svn-fe.c | 3 ++-\n remote-testsvn.c | 67 ++++++++++++++++++++++++++++++++++++++++++++---\n test-svn-fe.c | 2 +-\n vcs-svn/fast_export.c | 10 +++++--\n vcs-svn/fast_export.h | 6 ++---\n vcs-svn/svndump.c | 10 +++----\n vcs-svn/svndump.h | 2 +-\n 7 files changed, 84 insertions(+), 16 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\nindex c796cc0..f363505 100644\n--- a/contrib/svn-fe/svn-fe.c\n+++ b/contrib/svn-fe/svn-fe.c\n@@ -10,7 +10,8 @@ int main(int argc, char **argv)\n {\n \tif (svndump_init(NULL))\n \t\treturn 1;\n-\tsvndump_read((argc > 1) ? argv[1] : NULL, \"refs/heads/master\");\n+\tsvndump_read((argc > 1) ? argv[1] : NULL, \"refs/heads/master\",\n+\t\t\t\"refs/notes/svn/revs\");\n \tsvndump_deinit();\n \tsvndump_reset();\n \treturn 0;\ndiff --git a/remote-testsvn.c b/remote-testsvn.c\nindex b6e7968..e90d221 100644\n--- a/remote-testsvn.c\n+++ b/remote-testsvn.c\n@@ -12,7 +12,8 @@ static const char *url;\n static int dump_from_file;\n static const char *private_ref;\n static const char *remote_ref = \"refs/heads/master\";\n-static const char *marksfilename;\n+static const char *marksfilename, *notes_ref;\n+struct rev_note { unsigned int rev_nr; };\n \n static int cmd_capabilities(const char *line);\n static int cmd_import(const char *line);\n@@ -47,14 +48,70 @@ static void terminate_batch(void)\n \tfflush(stdout);\n }\n \n+/* NOTE: 'ref' refers to a git reference, while 'rev' refers to a svn revision. */\n+static char *read_ref_note(const unsigned char sha1[20]) {\n+\tconst unsigned char *note_sha1;\n+\tchar *msg = NULL;\n+\tunsigned long msglen;\n+\tenum object_type type;\n+\tinit_notes(NULL, notes_ref, NULL, 0);\n+\tif(\t(note_sha1 = get_note(NULL, sha1)) == NULL ||\n+\t\t\t!(msg = read_sha1_file(note_sha1, &type, &msglen)) ||\n+\t\t\t!msglen || type != OBJ_BLOB) {\n+\t\tfree(msg);\n+\t\treturn NULL;\n+\t}\n+\tfree_notes(NULL);\n+\treturn msg;\n+}\n+\n+static int parse_rev_note(const char *msg, struct rev_note *res) {\n+\tconst char *key, *value, *end;\n+\tsize_t len;\n+\twhile(*msg) {\n+\t\tend = strchr(msg, '\\n');\n+\t\tlen = end ? end - msg : strlen(msg);\n+\n+\t\tkey = \"Revision-number: \";\n+\t\tif(!prefixcmp(msg, key)) {\n+\t\t\tlong i;\n+\t\t\tvalue = msg + strlen(key);\n+\t\t\ti = atol(value);\n+\t\t\tif(i < 0 || i > UINT32_MAX)\n+\t\t\t\treturn 1;\n+\t\t\tres->rev_nr = i;\n+\t\t}\n+\t\tmsg += len + 1;\n+\t}\n+\treturn 0;\n+}\n+\n static int cmd_import(const char *line)\n {\n \tint code;\n \tint dumpin_fd;\n-\tunsigned int startrev = 0;\n+\tchar *note_msg;\n+\tunsigned char head_sha1[20];\n+\tunsigned int startrev;\n \tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n \tstruct child_process svndump_proc;\n \n+\tif(read_ref(private_ref, head_sha1))\n+\t\tstartrev = 0;\n+\telse {\n+\t\tnote_msg = read_ref_note(head_sha1);\n+\t\tif(note_msg == NULL) {\n+\t\t\twarning(\"No note found for %s.\", private_ref);\n+\t\t\tstartrev = 0;\n+\t\t}\n+\t\telse {\n+\t\t\tstruct rev_note note = { 0 };\n+\t\t\tparse_rev_note(note_msg, ¬e);\n+\t\t\tstartrev = note.rev_nr + 1;\n+\t\t\tfree(note_msg);\n+\t\t}\n+\t}\n+\n \tif (dump_from_file) {\n \t\tdumpin_fd = open(url, O_RDONLY);\n \t\tif(dumpin_fd < 0) {\n@@ -80,7 +137,7 @@ static int cmd_import(const char *line)\n \t\t\t\"feature export-marks=%s\\n\", marksfilename, marksfilename);\n \n \tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n-\tsvndump_read(url, private_ref);\n+\tsvndump_read(url, private_ref, notes_ref);\n \tsvndump_deinit();\n \tsvndump_reset();\n \n@@ -177,6 +234,9 @@ int main(int argc, const char **argv)\n \tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n \tprivate_ref = strbuf_detach(&buf, NULL);\n \n+\tstrbuf_addf(&buf, \"refs/notes/%s/revs\", remote->name);\n+\tnotes_ref = strbuf_detach(&buf, NULL);\n+\n \tstrbuf_addf(&buf, \"%s/info/fast-import/remote-svn/%s.marks\",\n \t\tget_git_dir(), remote->name);\n \tmarksfilename = strbuf_detach(&buf, NULL);\n@@ -196,6 +256,7 @@ int main(int argc, const char **argv)\n \tstrbuf_release(&buf);\n \tfree((void*)url);\n \tfree((void*)private_ref);\n+\tfree((void*)notes_ref);\n \tfree((void*)marksfilename);\n \treturn 0;\n }\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex cb0d80f..0f2d9a4 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -40,7 +40,7 @@ int main(int argc, char *argv[])\n \tif (argc == 2) {\n \t\tif (svndump_init(argv[1]))\n \t\t\treturn 1;\n-\t\tsvndump_read(NULL, \"refs/heads/master\");\n+\t\tsvndump_read(NULL, \"refs/heads/master\", \"refs/notes/svn/revs\");\n \t\tsvndump_deinit();\n \t\tsvndump_reset();\n \t\treturn 0;\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex df51c59..f2b23c8 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -68,13 +68,19 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n }\n \n void fast_export_begin_note(uint32_t revision, const char *author,\n-\t\tconst char *log, unsigned long timestamp)\n+\t\tconst char *log, unsigned long timestamp, const char *note_ref)\n {\n+\tstatic int firstnote = 1;\n \tsize_t loglen = strlen(log);\n-\tprintf(\"commit refs/notes/svn/revs\\n\");\n+\tprintf(\"commit %s\\n\", note_ref);\n \tprintf(\"committer %s <%s@%s> %ld +0000\\n\", author, author, \"local\", timestamp);\n \tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t)loglen);\n \tfwrite(log, loglen, 1, stdout);\n+\tif (firstnote) {\n+\t\tif (revision > 1)\n+\t\t\tprintf(\"from %s^0\", note_ref);\n+\t\tfirstnote = 0;\n+\t}\n \tfputc('\\n', stdout);\n }\n \ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex c2f6f11..c8b5adb 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -11,10 +11,10 @@ void fast_export_delete(const char *path);\n void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n void fast_export_note(const char *committish, const char *dataref);\n void fast_export_begin_note(uint32_t revision, const char *author,\n-\t\tconst char *log, unsigned long timestamp);\n+\t\tconst char *log, unsigned long timestamp, const char *note_ref);\n void fast_export_begin_commit(uint32_t revision, const char *author,\n-\t\t\tconst struct strbuf *log, const char *uuid,\n-\t\t\tconst char *url, unsigned long timestamp, const char *local_ref);\n+\t\t\tconst struct strbuf *log, const char *uuid,const char *url,\n+\t\t\tunsigned long timestamp, const char *local_ref);\n void fast_export_end_commit(uint32_t revision);\n void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input);\n void fast_export_buf_to_data(const struct strbuf *data);\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex cd65b51..31d1d83 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -309,20 +309,20 @@ static void begin_revision(const char *remote_ref)\n \t\trev_ctx.timestamp, remote_ref);\n }\n \n-static void end_revision()\n+static void end_revision(const char *note_ref)\n {\n \tstruct strbuf mark = STRBUF_INIT;\n \tif (rev_ctx.revision) {\n \t\tfast_export_end_commit(rev_ctx.revision);\n \t\tfast_export_begin_note(rev_ctx.revision, \"remote-svn\",\n-\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp);\n+\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp, note_ref);\n \t\tstrbuf_addf(&mark, \":%\"PRIu32, rev_ctx.revision);\n \t\tfast_export_note(mark.buf, \"inline\");\n \t\tfast_export_buf_to_data(&rev_ctx.note);\n \t}\n }\n \n-void svndump_read(const char *url, const char *local_ref)\n+void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n {\n \tchar *val;\n \tchar *t;\n@@ -363,7 +363,7 @@ void svndump_read(const char *url, const char *local_ref)\n \t\t\tif (active_ctx == REV_CTX)\n \t\t\t\tbegin_revision(local_ref);\n \t\t\tif (active_ctx != DUMP_CTX)\n-\t\t\t\tend_revision();\n+\t\t\t\tend_revision(notes_ref);\n \t\t\tactive_ctx = REV_CTX;\n \t\t\treset_rev_ctx(atoi(val));\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n@@ -479,7 +479,7 @@ void svndump_read(const char *url, const char *local_ref)\n \tif (active_ctx == REV_CTX)\n \t\tbegin_revision(local_ref);\n \tif (active_ctx != DUMP_CTX)\n-\t\tend_revision();\n+\t\tend_revision(notes_ref);\n }\n \n static void init(int report_fd)\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex febeecb..b8eb129 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -3,7 +3,7 @@\n \n int svndump_init(const char *filename);\n int svndump_init_fd(int in_fd, int back_fd);\n-void svndump_read(const char *url, const char *local_ref);\n+void svndump_read(const char *url, const char *local_ref, const char *notes_ref);\n void svndump_deinit(void);\n void svndump_reset(void);\n \n-- \n1.7.9.5\n"},{"id":"197974","messageId":"1346143790-23491-15-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-14-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 14/16] Add a svnrdump-simulator replaying a dump file for testing","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:48Z","receivedAt":"2012-08-28T08:49:48Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"To ease testing without depending on a reachable svn server, this\ncompact python script mimics parts of svnrdumps behaviour. It\nrequires the remote url to start with sim://.\n\nStart and end revisions are evaluated. If the requested revision\ndoesn't exist, as it is the case with incremental imports, if no new\ncommit was added, it returns 1 (like svnrdump).\n\nTo allow using the same dump file for simulating multiple incremental\nimports, the highest revision can be limited by setting the environment\nvariable SVNRMAX to that value. This simulates the situation where\nhigher revs don't exist yet.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n contrib/svn-fe/svnrdump_sim.py | 53 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n create mode 100755 contrib/svn-fe/svnrdump_sim.py\n\ndiff --git a/contrib/svn-fe/svnrdump_sim.py b/contrib/svn-fe/svnrdump_sim.py\nnew file mode 100755\nindex 0000000..1cfac4a\n--- /dev/null\n+++ b/contrib/svn-fe/svnrdump_sim.py\n@@ -0,0 +1,53 @@\n+#!/usr/bin/python\n+\"\"\"\n+Simulates svnrdump by replaying an existing dump from a file, taking care\n+of the specified revision range.\n+To simulate incremental imports the environment variable SVNRMAX can be set\n+to the highest revision that should be available.\n+\"\"\"\n+import sys, os\n+\n+\n+def getrevlimit():\n+ var = 'SVNRMAX'\n+ if os.environ.has_key(var):\n+ return os.environ[var]\n+ return None\n+\n+def writedump(url, lower, upper):\n+ if url.startswith('sim://'):\n+ filename = url[6:]\n+ if filename[-1] == '/': filename = filename[:-1] #remove terminating slash\n+ else:\n+ raise ValueError('sim:// url required')\n+ f = open(filename, 'r');\n+ state = 'header'\n+ wroterev = False\n+ while(True):\n+ l = f.readline()\n+ if l == '': break\n+ if state == 'header' and l.startswith('Revision-number: '):\n+ state = 'prefix'\n+ if state == 'prefix' and l == 'Revision-number: %s\\n' % lower:\n+ state = 'selection'\n+ if not upper == 'HEAD' and state == 'selection' and l == 'Revision-number: %s\\n' % upper:\n+ break;\n+\n+ if state == 'header' or state == 'selection':\n+ if state == 'selection': wroterev = True\n+ sys.stdout.write(l)\n+ return wroterev\n+\n+if __name__ == \"__main__\":\n+ if not (len(sys.argv) in (3, 4, 5)):\n+ print \"usage: %s dump URL -rLOWER:UPPER\"\n+ sys.exit(1)\n+ if not sys.argv[1] == 'dump': raise NotImplementedError('only \"dump\" is suppported.')\n+ url = sys.argv[2]\n+ r = ('0', 'HEAD')\n+ if len(sys.argv) == 4 and sys.argv[3][0:2] == '-r':\n+ r = sys.argv[3][2:].lstrip().split(':')\n+ if not getrevlimit() is None: r[1] = getrevlimit()\n+ if writedump(url, r[0], r[1]): ret = 0\n+ else: ret = 1\n+ sys.exit(ret)\n-- \n1.7.9.5\n"},{"id":"197978","messageId":"1346143790-23491-16-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-15-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 15/16] remote-svn: add marks-file regeneration","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:49Z","receivedAt":"2012-08-28T08:49:49Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"fast-import mark files are stored outside the object database and are\ntherefore not fetched and can be lost somehow else. marks provide a\nsvn revision --> git sha1 mapping, while the notes that are attached\nto each commit when it is imported provide a git sha1 --> svn revision\nmapping.\n\nIf the marks file is not available or not plausible, regenerate it by\nwalking through the notes tree. , i.e. The plausibility check tests\nif the highest revision in the marks file matches the revision of the\ntop ref. It doesn't ensure that the mark file is completely correct.\nThis could only be done with an effort equal to unconditional\nregeneration.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n remote-testsvn.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n\ndiff --git a/remote-testsvn.c b/remote-testsvn.c\nindex e90d221..d0b81d5 100644\n--- a/remote-testsvn.c\n+++ b/remote-testsvn.c\n@@ -86,6 +86,73 @@ static int parse_rev_note(const char *msg, struct rev_note *res) {\n \treturn 0;\n }\n \n+static int note2mark_cb(const unsigned char *object_sha1,\n+\t\tconst unsigned char *note_sha1, char *note_path,\n+\t\tvoid *cb_data) {\n+\tFILE *file = (FILE *)cb_data;\n+\tchar *msg;\n+\tunsigned long msglen;\n+\tenum object_type type;\n+\tstruct rev_note note;\n+\tif (!(msg = read_sha1_file(note_sha1, &type, &msglen)) ||\n+\t\t\t!msglen || type != OBJ_BLOB) {\n+\t\tfree(msg);\n+\t\treturn 1;\n+\t}\n+\tif (parse_rev_note(msg, ¬e))\n+\t\treturn 2;\n+\tif (fprintf(file, \":%d %s\\n\", note.rev_nr, sha1_to_hex(object_sha1)) < 1)\n+\t\treturn 3;\n+\treturn 0;\n+}\n+\n+static void regenerate_marks(void)\n+{\n+\tint ret;\n+\tFILE *marksfile;\n+\tmarksfile = fopen(marksfilename, \"w+\");\n+\tif (!marksfile)\n+\t\tdie_errno(\"Couldn't create mark file %s.\", marksfilename);\n+\tret = for_each_note(NULL, 0, note2mark_cb, marksfile);\n+\tif (ret)\n+\t\tdie(\"Regeneration of marks failed, returned %d.\", ret);\n+\tfclose(marksfile);\n+}\n+\n+static void check_or_regenerate_marks(int latestrev) {\n+\tFILE *marksfile;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tint found = 0;\n+\n+\tif (latestrev < 1)\n+\t\treturn;\n+\n+\tinit_notes(NULL, notes_ref, NULL, 0);\n+\tmarksfile = fopen(marksfilename, \"r\");\n+\tif (!marksfile) {\n+\t\tregenerate_marks();\n+\t\tmarksfile = fopen(marksfilename, \"r\");\n+\t\tif (!marksfile)\n+\t\t\tdie_errno(\"cannot read marks file %s!\", marksfilename);\n+\t\tfclose(marksfile);\n+\t} else {\n+\t\tstrbuf_addf(&sb, \":%d \", latestrev);\n+\t\twhile (strbuf_getline(&line, marksfile, '\\n') != EOF) {\n+\t\t\tif (!prefixcmp(line.buf, sb.buf)) {\n+\t\t\t\tfound++;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\tfclose(marksfile);\n+\t\tif (!found)\n+\t\t\tregenerate_marks();\n+\t}\n+\tfree_notes(NULL);\n+\tstrbuf_release(&sb);\n+\tstrbuf_release(&line);\n+}\n+\n static int cmd_import(const char *line)\n {\n \tint code;\n@@ -111,6 +178,7 @@ static int cmd_import(const char *line)\n \t\t\tfree(note_msg);\n \t\t}\n \t}\n+\tcheck_or_regenerate_marks(startrev - 1);\n \n \tif (dump_from_file) {\n \t\tdumpin_fd = open(url, O_RDONLY);\n-- \n1.7.9.5\n"},{"id":"197979","messageId":"1346143790-23491-17-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31371","inReplyTo":"1346143790-23491-16-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[PATCH v7 16/16] Add a test script for remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:49:50Z","receivedAt":"2012-08-28T08:49:50Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Use svnrdump_sim.py to emulate svnrdump without an svn server.\nTests fetching, incremental fetching, fetching from file://,\nand the regeneration of fast-import's marks file.\n\nSigned-off-by: Florian Achleitner \nSigned-off-by: Junio C Hamano \n---\n t/t9020-remote-svn.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 82 insertions(+)\n create mode 100755 t/t9020-remote-svn.sh\n\ndiff --git a/t/t9020-remote-svn.sh b/t/t9020-remote-svn.sh\nnew file mode 100755\nindex 0000000..e6ed4ca\n--- /dev/null\n+++ b/t/t9020-remote-svn.sh\n@@ -0,0 +1,82 @@\n+#!/bin/sh\n+\n+test_description='tests remote-svn'\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq PYTHON\n+then\n+\tskip_all='skipping remote-svn tests, python not available'\n+\ttest_done\n+fi\n+\n+# We override svnrdump by placing a symlink to the svnrdump-emulator in .\n+export PATH=\"$HOME:$PATH\"\n+ln -sf $GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py \"$HOME/svnrdump\"\n+\n+init_git () {\n+\trm -fr .git &&\n+\tgit init &&\n+\t#git remote add svnsim testsvn::sim:///$TEST_DIRECTORY/t9020/example.svnrdump\n+\t# let's reuse an exisiting dump file!?\n+\tgit remote add svnsim testsvn::sim://$TEST_DIRECTORY/t9154/svn.dump\n+\tgit remote add svnfile testsvn::file://$TEST_DIRECTORY/t9154/svn.dump\n+}\n+\n+if test -e \"$GIT_BUILD_DIR/git-remote-testsvn\"\n+then\n+\ttest_set_prereq REMOTE_SVN\n+fi\n+\n+test_debug '\n+\tgit --version\n+\twhich git\n+\twhich svnrdump\n+'\n+\n+test_expect_success REMOTE_SVN 'simple fetch' '\n+\tinit_git &&\n+\tgit fetch svnsim &&\n+\ttest_cmp .git/refs/svn/svnsim/master .git/refs/remotes/svnsim/master &&\n+\tcp .git/refs/remotes/svnsim/master master.good\n+'\n+\n+test_debug '\n+\tcat .git/refs/svn/svnsim/master\n+\tcat .git/refs/remotes/svnsim/master\n+'\n+\n+test_expect_success REMOTE_SVN 'repeated fetch, nothing shall change' '\n+\tgit fetch svnsim &&\n+\ttest_cmp master.good .git/refs/remotes/svnsim/master\n+'\n+\n+test_expect_success REMOTE_SVN 'fetch from a file:// url gives the same result' '\n+\tgit fetch svnfile\n+'\n+\n+test_expect_failure REMOTE_SVN 'the sha1 differ because the git-svn-id line in the commit msg contains the url' '\n+\ttest_cmp .git/refs/remotes/svnfile/master .git/refs/remotes/svnsim/master\n+'\n+\n+test_expect_success REMOTE_SVN 'mark-file regeneration' '\n+\t# filter out any other marks, that can not be regenerated. Only up to 3 digit revisions are allowed here\n+\tgrep \":[0-9]\\{1,3\\} \" .git/info/fast-import/remote-svn/svnsim.marks > .git/info/fast-import/remote-svn/svnsim.marks.old &&\n+\trm .git/info/fast-import/remote-svn/svnsim.marks &&\n+\tgit fetch svnsim &&\n+\ttest_cmp .git/info/fast-import/remote-svn/svnsim.marks.old .git/info/fast-import/remote-svn/svnsim.marks\n+'\n+\n+test_expect_success REMOTE_SVN 'incremental imports must lead to the same head' '\n+\texport SVNRMAX=3 &&\n+\tinit_git &&\n+\tgit fetch svnsim &&\n+\ttest_cmp .git/refs/svn/svnsim/master .git/refs/remotes/svnsim/master &&\n+\tunset SVNRMAX &&\n+\tgit fetch svnsim &&\n+\ttest_cmp master.good .git/refs/remotes/svnsim/master\n+'\n+\n+test_debug 'git branch -a'\n+\n+test_done\n-- \n1.7.9.5\n"},{"id":"197980","messageId":"2052320.5WmLkH0W9f@flobuntu","threadId":"31371","inReplyTo":"1346143790-23491-1-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 00/16] GSOC remote-svn","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-28T08:59:58Z","receivedAt":"2012-08-28T08:59:58Z","isPatch":true,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Tuesday 28 August 2012 10:49:34 Florian Achleitner wrote:\n> Reroll includes fixups by Ramsey. Thanks!\n> Diff:\n> [..]\n> - improve compatibility of integer types.\n> [..]\n\nThis line is wrong in this series. Just delete it. Sorry.\n"},{"id":"197993","messageId":"7v4nnn5538.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-3-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 02/16] Add git-remote-testsvn to Makefile","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T16:55:23Z","receivedAt":"2012-08-28T16:55:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> The link-rule is a copy of the standard git$X rule but adds VCSSVN_LIB.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> Makefile | 5 +++++\n> 1 file changed, 5 insertions(+)\n>\n> diff --git a/Makefile b/Makefile\n> index 66e8216..1b09454 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -477,6 +477,7 @@ PROGRAM_OBJS += sh-i18n--envsubst.o\n> PROGRAM_OBJS += shell.o\n> PROGRAM_OBJS += show-index.o\n> PROGRAM_OBJS += upload-pack.o\n> +PROGRAM_OBJS += remote-testsvn.o\n> \n> # Binary suffix, set to .exe for Windows builds\n> X =\n> @@ -2352,6 +2353,10 @@ git-http-push$X: revision.o http.o http-push.o GIT-LDFLAGS $(GITLIBS)\n> \t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \\\n> \t\t$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)\n> \n> +git-remote-testsvn$X: remote-testsvn.o GIT-LDFLAGS $(GITLIBS) $(VCSSVN_LIB)\n> +\t$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS) \\\n> +\t$(VCSSVN_LIB)\n> +\n> $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)\n> \t$(QUIET_LNCP)$(RM) $@ && \\\n> \tln $< $@ 2>/dev/null || \\\n\nI'd squash in a change to add this executable to .gitignore\n(attached) to this patch. Unless you have objections to it, or\nsuggestions to do it better, there is no need to resend.\n\nThanks.\n\n .gitignore | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git i/.gitignore w/.gitignore\nindex bb5c91e..51294d6 100644\n--- i/.gitignore\n+++ w/.gitignore\n@@ -125,6 +125,7 @@\n /git-remote-fd\n /git-remote-ext\n /git-remote-testgit\n+/git-remote-testsvn\n /git-repack\n /git-replace\n /git-repo-config\n"},{"id":"197999","messageId":"7vipc23neh.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-4-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 03/16] Add svndump_init_fd to allow reading dumps from arbitrary FDs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:02:35Z","receivedAt":"2012-08-28T17:02:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> The existing function only allows reading from a filename or from\n> stdin. Allow passing of a FD and an additional FD for the back report\n> pipe. This allows us to retrieve the name of the pipe in the caller.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> vcs-svn/svndump.c | 22 ++++++++++++++++++----\n> vcs-svn/svndump.h | 1 +\n> 2 files changed, 19 insertions(+), 4 deletions(-)\n>\n> diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\n> index 2b168ae..d81a078 100644\n> --- a/vcs-svn/svndump.c\n> +++ b/vcs-svn/svndump.c\n> @@ -468,11 +468,9 @@ void svndump_read(const char *url)\n> \t\tend_revision();\n> }\n> \n> -int svndump_init(const char *filename)\n> +static void init(int report_fd)\n> {\n> -\tif (buffer_init(&input, filename))\n> -\t\treturn error(\"cannot open %s: %s\", filename, strerror(errno));\n> -\tfast_export_init(REPORT_FILENO);\n> +\tfast_export_init(report_fd);\n> \tstrbuf_init(&dump_ctx.uuid, 4096);\n> \tstrbuf_init(&dump_ctx.url, 4096);\n> \tstrbuf_init(&rev_ctx.log, 4096);\n> @@ -482,6 +480,22 @@ int svndump_init(const char *filename)\n> \treset_dump_ctx(NULL);\n> \treset_rev_ctx(0);\n> \treset_node_ctx(NULL);\n> +\treturn;\n> +}\n> +\n> +int svndump_init(const char *filename)\n> +{\n> +\tif (buffer_init(&input, filename))\n> +\t\treturn error(\"cannot open %s: %s\", filename ? filename : \"NULL\", strerror(errno));\n> +\tinit(REPORT_FILENO);\n> +\treturn 0;\n> +}\n> +\n> +int svndump_init_fd(int in_fd, int back_fd)\n> +{\n> +\tif(buffer_fdinit(&input, xdup(in_fd)))\n\nStyle:\n\n\tif (buffer_fdinit(&input, xdup(in_fd))\n\n> +\t\treturn error(\"cannot open fd %d: %s\", in_fd, strerror(errno));\n> +\tinit(xdup(back_fd));\n> \treturn 0;\n> }\n> \n> diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\n> index df9ceb0..acb5b47 100644\n> --- a/vcs-svn/svndump.h\n> +++ b/vcs-svn/svndump.h\n> @@ -2,6 +2,7 @@\n> #define SVNDUMP_H_\n> \n> int svndump_init(const char *filename);\n> +int svndump_init_fd(int in_fd, int back_fd);\n> void svndump_read(const char *url);\n> void svndump_deinit(void);\n> void svndump_reset(void);\n"},{"id":"198000","messageId":"7vd32a3nee.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-10-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 09/16] Allow reading svn dumps from files via file:// urls","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:06:20Z","receivedAt":"2012-08-28T17:06:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> For testing as well as for importing large, already available dumps,\n> it's useful to bypass svnrdump and replay the svndump from a file\n> directly.\n>\n> Add support for file:// urls in the remote url, e.g.\n>\n> svn::file:///path/to/dump\n>\n> When the remote helper finds an url starting with file:// it tries to\n> open that file instead of invoking svnrdump.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> remote-testsvn.c | 55 +++++++++++++++++++++++++++++++++++-------------------\n> 1 file changed, 36 insertions(+), 19 deletions(-)\n>\n> diff --git a/remote-testsvn.c b/remote-testsvn.c\n> index ebe803b..2b9d151 100644\n> --- a/remote-testsvn.c\n> +++ b/remote-testsvn.c\n> @@ -9,6 +9,7 @@\n> #include \"argv-array.h\"\n> \n> static const char *url;\n> +static int dump_from_file;\n> static const char *private_ref;\n> static const char *remote_ref = \"refs/heads/master\";\n> \n> @@ -53,29 +54,38 @@ static int cmd_import(const char *line)\n> \tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n> \tstruct child_process svndump_proc;\n> \n> -\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n> -\tsvndump_proc.out = -1;\n> -\targv_array_push(&svndump_argv, \"svnrdump\");\n> -\targv_array_push(&svndump_argv, \"dump\");\n> -\targv_array_push(&svndump_argv, url);\n> -\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n> -\tsvndump_proc.argv = svndump_argv.argv;\n> -\n> -\tcode = start_command(&svndump_proc);\n> -\tif (code)\n> -\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n> -\tdumpin_fd = svndump_proc.out;\n> -\n> +\tif (dump_from_file) {\n> +\t\tdumpin_fd = open(url, O_RDONLY);\n> +\t\tif(dumpin_fd < 0) {\n> +\t\t\tdie_errno(\"Couldn't open svn dump file %s.\", url);\n> +\t\t}\n> +\t}\n> +\telse {\n> +\t\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n\n\nStyle:\n\n\tif (dump_from_file) {\n\t\tdumpin_fd = open(url, O_RDONLY);\n\t\tif (dumpin_fd < 0)\n \tdie_errno(\"Couldn't...\", url);\n\t} else {\n \tmemset(&svndump_proc, ...);\n\t\t...\n\t}\n\n> +\t\tsvndump_proc.out = -1;\n> +\t\targv_array_push(&svndump_argv, \"svnrdump\");\n> +\t\targv_array_push(&svndump_argv, \"dump\");\n> +\t\targv_array_push(&svndump_argv, url);\n> +\t\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n> +\t\tsvndump_proc.argv = svndump_argv.argv;\n> +\n> +\t\tcode = start_command(&svndump_proc);\n> +\t\tif (code)\n> +\t\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n> +\t\tdumpin_fd = svndump_proc.out;\n> +\t}\n> \tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n> \tsvndump_read(url, private_ref);\n> \tsvndump_deinit();\n> \tsvndump_reset();\n> \n> \tclose(dumpin_fd);\n> -\tcode = finish_command(&svndump_proc);\n> -\tif (code)\n> -\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n> -\targv_array_clear(&svndump_argv);\n> +\tif(!dump_from_file) {\n\nStyle:\n\n\tif (!dump_from_file) {\n\n> +\t\tcode = finish_command(&svndump_proc);\n> +\t\tif (code)\n> +\t\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n> +\t\targv_array_clear(&svndump_argv);\n> +\t}\n> \n> \treturn 0;\n> }\n> @@ -149,8 +159,15 @@ int main(int argc, const char **argv)\n> \tremote = remote_get(argv[1]);\n> \turl_in = (argc == 3) ? argv[2] : remote->url[0];\n> \n> -\tend_url_with_slash(&buf, url_in);\n> -\turl = strbuf_detach(&buf, NULL);\n> +\tif (!prefixcmp(url_in, \"file://\")) {\n> +\t\tdump_from_file = 1;\n> +\t\turl = url_decode(url_in + sizeof(\"file://\")-1);\n> +\t}\n> +\telse {\n\nStyle:\n\n\tif (...) {\n\t\t...\n } else {\n\t\t...\n\t}\n\n> +\t\tdump_from_file = 0;\n> +\t\tend_url_with_slash(&buf, url_in);\n> +\t\turl = strbuf_detach(&buf, NULL);\n> +\t}\n> \n> \tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n> \tprivate_ref = strbuf_detach(&buf, NULL);\n"},{"id":"198001","messageId":"7v7gsi3nec.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-2-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 01/16] Implement a remote helper for svn in C","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:08:17Z","receivedAt":"2012-08-28T17:08:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> Enable basic fetching from subversion repositories. When processing\n> remote URLs starting with testsvn::, git invokes this remote-helper.\n> It starts svnrdump to extract revisions from the subversion repository\n> in the 'dump file format', and converts them to a git-fast-import stream\n> using the functions of vcs-svn/.\n>\n> Imported refs are created in a private namespace at\n> refs/svn//master. The revision history is imported\n> linearly (no branch detection) and completely, i.e. from revision 0 to\n> HEAD.\n>\n> The 'bidi-import' capability is used. The remote-helper expects data\n> from fast-import on its stdin. It buffers a batch of 'import' command\n> lines in a string_list before starting to process them.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> remote-testsvn.c | 174 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 174 insertions(+)\n> create mode 100644 remote-testsvn.c\n>\n> diff --git a/remote-testsvn.c b/remote-testsvn.c\n> new file mode 100644\n> index 0000000..ebe803b\n> --- /dev/null\n> +++ b/remote-testsvn.c\n> @@ -0,0 +1,174 @@\n> +#include \"cache.h\"\n> +#include \"remote.h\"\n> +#include \"strbuf.h\"\n> +#include \"url.h\"\n> +#include \"exec_cmd.h\"\n> +#include \"run-command.h\"\n> +#include \"vcs-svn/svndump.h\"\n> +#include \"notes.h\"\n> +#include \"argv-array.h\"\n> +\n> +static const char *url;\n> +static const char *private_ref;\n> +static const char *remote_ref = \"refs/heads/master\";\n> +\n> +static int cmd_capabilities(const char *line);\n> +static int cmd_import(const char *line);\n> +static int cmd_list(const char *line);\n> +\n> +typedef int (*input_command_handler)(const char *);\n> +struct input_command_entry {\n> +\tconst char *name;\n> +\tinput_command_handler fn;\n> +\tunsigned char batchable;\t/* whether the command starts or is part of a batch */\n> +};\n> +\n> +static const struct input_command_entry input_command_list[] = {\n> +\t{ \"capabilities\", cmd_capabilities, 0 },\n> +\t{ \"import\", cmd_import, 1 },\n> +\t{ \"list\", cmd_list, 0 },\n> +\t{ NULL, NULL }\n> +};\n> +\n> +static int cmd_capabilities(const char *line) {\n\nStyle:\n\n\tstatic int cmd_capabilities(const char *line)\n {\n\n> +\tprintf(\"import\\n\");\n> +\tprintf(\"bidi-import\\n\");\n> +\tprintf(\"refspec %s:%s\\n\\n\", remote_ref, private_ref);\n> +\tfflush(stdout);\n> +\treturn 0;\n> +}\n> +\n> +static void terminate_batch(void)\n> +{\n> +\t/* terminate a current batch's fast-import stream */\n> +\tprintf(\"done\\n\");\n> +\tfflush(stdout);\n> +}\n> +\n> +static int cmd_import(const char *line)\n> +{\n> +\tint code;\n> +\tint dumpin_fd;\n> +\tunsigned int startrev = 0;\n> +\tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n> +\tstruct child_process svndump_proc;\n> +\n> +\tmemset(&svndump_proc, 0, sizeof(struct child_process));\n> +\tsvndump_proc.out = -1;\n> +\targv_array_push(&svndump_argv, \"svnrdump\");\n> +\targv_array_push(&svndump_argv, \"dump\");\n> +\targv_array_push(&svndump_argv, url);\n> +\targv_array_pushf(&svndump_argv, \"-r%u:HEAD\", startrev);\n> +\tsvndump_proc.argv = svndump_argv.argv;\n> +\n> +\tcode = start_command(&svndump_proc);\n> +\tif (code)\n> +\t\tdie(\"Unable to start %s, code %d\", svndump_proc.argv[0], code);\n> +\tdumpin_fd = svndump_proc.out;\n> +\n> +\tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n> +\tsvndump_read(url, private_ref);\n> +\tsvndump_deinit();\n> +\tsvndump_reset();\n> +\n> +\tclose(dumpin_fd);\n> +\tcode = finish_command(&svndump_proc);\n> +\tif (code)\n> +\t\twarning(\"%s, returned %d\", svndump_proc.argv[0], code);\n> +\targv_array_clear(&svndump_argv);\n> +\n> +\treturn 0;\n> +}\n> +\n> +static int cmd_list(const char *line)\n> +{\n> +\tprintf(\"? %s\\n\\n\", remote_ref);\n> +\tfflush(stdout);\n> +\treturn 0;\n> +}\n> +\n> +static int do_command(struct strbuf *line)\n> +{\n> +\tconst struct input_command_entry *p = input_command_list;\n> +\tstatic struct string_list batchlines = STRING_LIST_INIT_DUP;\n> +\tstatic const struct input_command_entry *batch_cmd;\n> +\t/*\n> +\t * commands can be grouped together in a batch.\n> +\t * Batches are ended by \\n. If no batch is active the program ends.\n> +\t * During a batch all lines are buffered and passed to the handler function\n> +\t * when the batch is terminated.\n> +\t */\n> +\tif (line->len == 0) {\n> +\t\tif (batch_cmd) {\n> +\t\t\tstruct string_list_item *item;\n> +\t\t\tfor_each_string_list_item(item, &batchlines)\n> +\t\t\t\tbatch_cmd->fn(item->string);\n> +\t\t\tterminate_batch();\n> +\t\t\tbatch_cmd = NULL;\n> +\t\t\tstring_list_clear(&batchlines, 0);\n> +\t\t\treturn 0;\t/* end of the batch, continue reading other commands. */\n> +\t\t}\n> +\t\treturn 1;\t/* end of command stream, quit */\n> +\t}\n> +\tif (batch_cmd) {\n> +\t\tif (prefixcmp(batch_cmd->name, line->buf))\n> +\t\t\tdie(\"Active %s batch interrupted by %s\", batch_cmd->name, line->buf);\n> +\t\t/* buffer batch lines */\n> +\t\tstring_list_append(&batchlines, line->buf);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tfor (p = input_command_list; p->name; p++) {\n> +\t\tif (!prefixcmp(line->buf, p->name) && (strlen(p->name) == line->len ||\n> +\t\t\t\tline->buf[strlen(p->name)] == ' ')) {\n> +\t\t\tif (p->batchable) {\n> +\t\t\t\tbatch_cmd = p;\n> +\t\t\t\tstring_list_append(&batchlines, line->buf);\n> +\t\t\t\treturn 0;\n> +\t\t\t}\n> +\t\t\treturn p->fn(line->buf);\n> +\t\t}\n> +\t}\n> +\tdie(\"Unknown command '%s'\\n\", line->buf);\n> +\treturn 0;\n> +}\n> +\n> +int main(int argc, const char **argv)\n> +{\n> +\tstruct strbuf buf = STRBUF_INIT;\n> +\tstatic struct remote *remote;\n> +\tconst char *url_in;\n> +\n> +\tgit_extract_argv0_path(argv[0]);\n> +\tsetup_git_directory();\n> +\tif (argc < 2 || argc > 3) {\n> +\t\tusage(\"git-remote-svn []\");\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tremote = remote_get(argv[1]);\n> +\turl_in = (argc == 3) ? argv[2] : remote->url[0];\n> +\n> +\tend_url_with_slash(&buf, url_in);\n> +\turl = strbuf_detach(&buf, NULL);\n> +\n> +\tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n> +\tprivate_ref = strbuf_detach(&buf, NULL);\n> +\n> +\twhile(1) {\n\nStyle:\n\n\twhile (1) {\n\n> +\t\tif (strbuf_getline(&buf, stdin, '\\n') == EOF) {\n> +\t\t\tif (ferror(stdin))\n> +\t\t\t\tdie(\"Error reading command stream\");\n> +\t\t\telse\n> +\t\t\t\tdie(\"Unexpected end of command stream\");\n> +\t\t}\n> +\t\tif (do_command(&buf))\n> +\t\t\tbreak;\n> +\t\tstrbuf_reset(&buf);\n> +\t}\n> +\n> +\tstrbuf_release(&buf);\n> +\tfree((void*)url);\n> +\tfree((void*)private_ref);\n\nStyle:\n\n\tfree((void *)url);\n\tfree((void *)private_ref);\n\n[PATCH 12/16] adds another instance of this.\n\nHaving said that, it might be cleaner to prepare two strbuf\ndedicated for these variables, borrow url and private_ref out of\nthem while they are used as the program level globals, and use\nstrbuf_release() at the end.\n\n> +\treturn 0;\n> +}\n"},{"id":"198005","messageId":"7vvcg228tr.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-9-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 08/16] remote-svn, vcs-svn: Enable fetching to private refs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:53:06Z","receivedAt":"2012-08-28T17:53:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> The reference to update by the fast-import stream is hard-coded. When\n> fetching from a remote the remote-helper shall update refs in a\n> private namespace, i.e. a private subdir of refs/. This namespace is\n> defined by the 'refspec' capability, that the remote-helper advertises\n> as a reply to the 'capabilities' command.\n>\n> Extend svndump and fast-export to allow passing the target ref.\n> Update svn-fe to be compatible.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> ...\n> diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\n> index d81a078..288bb42 100644\n> --- a/vcs-svn/svndump.c\n> +++ b/vcs-svn/svndump.c\n> @@ -299,22 +299,22 @@ static void handle_node(void)\n> \t\t\t\tnode_ctx.text_length, &input);\n> }\n> \n> -static void begin_revision(void)\n> +static void begin_revision(const char *remote_ref)\n> {\n> \tif (!rev_ctx.revision)\t/* revision 0 gets no git commit. */\n> \t\treturn;\n> \tfast_export_begin_commit(rev_ctx.revision, rev_ctx.author.buf,\n> \t\t&rev_ctx.log, dump_ctx.uuid.buf, dump_ctx.url.buf,\n> -\t\trev_ctx.timestamp);\n> +\t\trev_ctx.timestamp, remote_ref);\n> }\n> \n> -static void end_revision(void)\n> +static void end_revision()\n\nDon't.\n\n> {\n> \tif (rev_ctx.revision)\n> \t\tfast_export_end_commit(rev_ctx.revision);\n> }\n"},{"id":"198002","messageId":"7v1uiq3nea.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-14-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 13/16] remote-svn: add incremental import","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:54:07Z","receivedAt":"2012-08-28T17:54:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> Search for a note attached to the ref to update and read it's\n> 'Revision-number:'-line. Start import from the next svn revision.\n>\n> If there is no next revision in the svn repo, svnrdump terminates with\n> a message on stderr an non-zero return value. This looks a little\n> weird, but there is no other way to know whether there is a new\n> revision in the svn repo.\n>\n> On the start of an incremental import, the parent of the first commit\n> in the fast-import stream is set to the branch name to update. All\n> following commits specify their parent by a mark number. Previous mark\n> files are currently not reused.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> contrib/svn-fe/svn-fe.c | 3 ++-\n> remote-testsvn.c | 67 ++++++++++++++++++++++++++++++++++++++++++++---\n> test-svn-fe.c | 2 +-\n> vcs-svn/fast_export.c | 10 +++++--\n> vcs-svn/fast_export.h | 6 ++---\n> vcs-svn/svndump.c | 10 +++----\n> vcs-svn/svndump.h | 2 +-\n> 7 files changed, 84 insertions(+), 16 deletions(-)\n>\n> diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c\n> index c796cc0..f363505 100644\n> --- a/contrib/svn-fe/svn-fe.c\n> +++ b/contrib/svn-fe/svn-fe.c\n> @@ -10,7 +10,8 @@ int main(int argc, char **argv)\n> {\n> \tif (svndump_init(NULL))\n> \t\treturn 1;\n> -\tsvndump_read((argc > 1) ? argv[1] : NULL, \"refs/heads/master\");\n> +\tsvndump_read((argc > 1) ? argv[1] : NULL, \"refs/heads/master\",\n> +\t\t\t\"refs/notes/svn/revs\");\n> \tsvndump_deinit();\n> \tsvndump_reset();\n> \treturn 0;\n> diff --git a/remote-testsvn.c b/remote-testsvn.c\n> index b6e7968..e90d221 100644\n> --- a/remote-testsvn.c\n> +++ b/remote-testsvn.c\n> @@ -12,7 +12,8 @@ static const char *url;\n> static int dump_from_file;\n> static const char *private_ref;\n> static const char *remote_ref = \"refs/heads/master\";\n> -static const char *marksfilename;\n> +static const char *marksfilename, *notes_ref;\n> +struct rev_note { unsigned int rev_nr; };\n> \n> static int cmd_capabilities(const char *line);\n> static int cmd_import(const char *line);\n> @@ -47,14 +48,70 @@ static void terminate_batch(void)\n> \tfflush(stdout);\n> }\n> \n> +/* NOTE: 'ref' refers to a git reference, while 'rev' refers to a svn revision. */\n> +static char *read_ref_note(const unsigned char sha1[20]) {\n\nStyle:\n\n\tstatic char *read_ref_note(const unsigned char sha1[20])\n {\n\n> +\tconst unsigned char *note_sha1;\n> +\tchar *msg = NULL;\n> +\tunsigned long msglen;\n> +\tenum object_type type;\n> +\tinit_notes(NULL, notes_ref, NULL, 0);\n> +\tif(\t(note_sha1 = get_note(NULL, sha1)) == NULL ||\n> +\t\t\t!(msg = read_sha1_file(note_sha1, &type, &msglen)) ||\n> +\t\t\t!msglen || type != OBJ_BLOB) {\n> +\t\tfree(msg);\n> +\t\treturn NULL;\n> +\t}\n> +\tfree_notes(NULL);\n> +\treturn msg;\n> +}\n\nStyle:\n\n\tif (!(note_sha1 = get_note(NULL, sha1)) ||\n\t !(msg = read_sha1_file(note_sha1, &type, &msglen)) ||\n\t !msglen ||\n type != OBJ_BLOB) {\n\t\t...\n\nBut a bigger question is if any of these cases is a non-error.\n\nIt may be perfectly normal so get_note() that returns NULL may be a\nnormal condition, but is it something you want to silently ignore if\nread_sha1_file() did not give you anything when called with a\nnote_sha1 that ought to be valid? How about the case where you got\nmsg but msglen is zero? Is it an error? Is it normal and the\ncaller wants to see a note that happens to be an empty string? How\nabout the case where the note returned was ot a blob? Is it\nsomething you want to silently ignore, or is it an error?\n\nHaving multiple assingments inside a conditional, and chaining them\ntogether with \"||\", lets you code lazily, and the resulting code\nlike the above _appear_ concise, but in order to prepare the code to\nanswer these questions sensibly, it is often a good habit to avoid\nthe appearance of conciseness that hides the lack of thought (e.g.\nthe code is hiding the reason why it does not call free_notes() when\nyou do have note_sah1 but found a note that is of undesired object\ntype, and the reader cannot tell if it is done deliberately).\n\nIt is far more preferrable to see this written like:\n\n\tif (!(note_sha1 = get_note(NULL, sha1)))\n \treturn NULL; /* no notes - nothing to return */\n\tmsg = read_sha1_file(note_sha1, &type, &msglen);\n\tif (!msg) {\n\t\terror(\"cannot read notes for ...\");\n } else if (type != OBJ_BLOB) {\n\t\tfree(msg); /* something we cannot use */\n\t\tmsg = NULL;\n\t} ... you may have more else if clauses here ...\n free_notes(NULL);\n return msg;\n\nAlso, I am not sure if you want to silently ignore OBJ_BLOB here.\nIf I understand correctly, you are not reading from a random notes\ntree, but from a notes tree that was populated by an earlier\nincarnation of your process, no? If asking for a note in that notes\ntree yields a note that you do not recognize, shouldn't you treat it\nas an error and raise a big red flag? The same discussion goes for\nignoring an empty msg. If you never produce an empty msg, and if\nnobody else is supposed to add random stuff to that notes tree,\nshouldn't you treat it as an indication that something fishy is\ngoing on if you read an empty msg?\n\n> +static int parse_rev_note(const char *msg, struct rev_note *res) {\n\nStyle.\n\n\tstatic int parse_rev_note(const char *msg, struct rev_note *res)\n\t{\n\n> +\tconst char *key, *value, *end;\n> +\tsize_t len;\n> +\twhile(*msg) {\n\nStyle.\n\n while (*msg) {\n\n> +\t\tend = strchr(msg, '\\n');\n> +\t\tlen = end ? end - msg : strlen(msg);\n> +\n> +\t\tkey = \"Revision-number: \";\n> +\t\tif(!prefixcmp(msg, key)) {\n\nStyle.\n\n\t\tif (!prefixcmp(msg, key)) {\n\n> +\t\t\tlong i;\n> +\t\t\tvalue = msg + strlen(key);\n> +\t\t\ti = atol(value);\n> +\t\t\tif(i < 0 || i > UINT32_MAX)\n\nStyle.\n\t\tif (i < 0 || ...)\n\nMore importantly, if you are parsing text that is supposed to be a\nformat known to you and not human generated, you should avoid using\natoi & atol when parsing numbers; use strtol or strtoul instead, as\nthey allow you much better error handling.\n\n> +\t\t\t\treturn 1;\n\nIs it signaling an error to the caller? The usual convention is to\nuse negative value for such a purpose.\n\n> +\t\t\tres->rev_nr = i;\n> +\t\t}\n> +\t\tmsg += len + 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int cmd_import(const char *line)\n> {\n> \tint code;\n> \tint dumpin_fd;\n> -\tunsigned int startrev = 0;\n> +\tchar *note_msg;\n> +\tunsigned char head_sha1[20];\n> +\tunsigned int startrev;\n> \tstruct argv_array svndump_argv = ARGV_ARRAY_INIT;\n> \tstruct child_process svndump_proc;\n> \n> +\tif(read_ref(private_ref, head_sha1))\n\nStyle.\n\n> +\t\tstartrev = 0;\n> +\telse {\n> +\t\tnote_msg = read_ref_note(head_sha1);\n> +\t\tif(note_msg == NULL) {\n> +\t\t\twarning(\"No note found for %s.\", private_ref);\n> +\t\t\tstartrev = 0;\n> +\t\t}\n> +\t\telse {\n\nStyle.\n\n> +\t\t\tstruct rev_note note = { 0 };\n> +\t\t\tparse_rev_note(note_msg, ¬e);\n> +\t\t\tstartrev = note.rev_nr + 1;\n> +\t\t\tfree(note_msg);\n> +\t\t}\n> +\t}\n> +\n> \tif (dump_from_file) {\n> \t\tdumpin_fd = open(url, O_RDONLY);\n> \t\tif(dumpin_fd < 0) {\n> @@ -80,7 +137,7 @@ static int cmd_import(const char *line)\n> \t\t\t\"feature export-marks=%s\\n\", marksfilename, marksfilename);\n> \n> \tsvndump_init_fd(dumpin_fd, STDIN_FILENO);\n> -\tsvndump_read(url, private_ref);\n> +\tsvndump_read(url, private_ref, notes_ref);\n> \tsvndump_deinit();\n> \tsvndump_reset();\n> \n> @@ -177,6 +234,9 @@ int main(int argc, const char **argv)\n> \tstrbuf_addf(&buf, \"refs/svn/%s/master\", remote->name);\n> \tprivate_ref = strbuf_detach(&buf, NULL);\n> \n> +\tstrbuf_addf(&buf, \"refs/notes/%s/revs\", remote->name);\n> +\tnotes_ref = strbuf_detach(&buf, NULL);\n> +\n> \tstrbuf_addf(&buf, \"%s/info/fast-import/remote-svn/%s.marks\",\n> \t\tget_git_dir(), remote->name);\n> \tmarksfilename = strbuf_detach(&buf, NULL);\n> @@ -196,6 +256,7 @@ int main(int argc, const char **argv)\n> \tstrbuf_release(&buf);\n> \tfree((void*)url);\n> \tfree((void*)private_ref);\n> +\tfree((void*)notes_ref);\n> \tfree((void*)marksfilename);\n> \treturn 0;\n> }\n> diff --git a/test-svn-fe.c b/test-svn-fe.c\n> index cb0d80f..0f2d9a4 100644\n> --- a/test-svn-fe.c\n> +++ b/test-svn-fe.c\n> @@ -40,7 +40,7 @@ int main(int argc, char *argv[])\n> \tif (argc == 2) {\n> \t\tif (svndump_init(argv[1]))\n> \t\t\treturn 1;\n> -\t\tsvndump_read(NULL, \"refs/heads/master\");\n> +\t\tsvndump_read(NULL, \"refs/heads/master\", \"refs/notes/svn/revs\");\n> \t\tsvndump_deinit();\n> \t\tsvndump_reset();\n> \t\treturn 0;\n> diff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\n> index df51c59..f2b23c8 100644\n> --- a/vcs-svn/fast_export.c\n> +++ b/vcs-svn/fast_export.c\n> @@ -68,13 +68,19 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)\n> }\n> \n> void fast_export_begin_note(uint32_t revision, const char *author,\n> -\t\tconst char *log, unsigned long timestamp)\n> +\t\tconst char *log, unsigned long timestamp, const char *note_ref)\n> {\n> +\tstatic int firstnote = 1;\n> \tsize_t loglen = strlen(log);\n> -\tprintf(\"commit refs/notes/svn/revs\\n\");\n> +\tprintf(\"commit %s\\n\", note_ref);\n> \tprintf(\"committer %s <%s@%s> %ld +0000\\n\", author, author, \"local\", timestamp);\n> \tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t)loglen);\n> \tfwrite(log, loglen, 1, stdout);\n> +\tif (firstnote) {\n> +\t\tif (revision > 1)\n> +\t\t\tprintf(\"from %s^0\", note_ref);\n> +\t\tfirstnote = 0;\n> +\t}\n> \tfputc('\\n', stdout);\n> }\n> \n> diff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\n> index c2f6f11..c8b5adb 100644\n> --- a/vcs-svn/fast_export.h\n> +++ b/vcs-svn/fast_export.h\n> @@ -11,10 +11,10 @@ void fast_export_delete(const char *path);\n> void fast_export_modify(const char *path, uint32_t mode, const char *dataref);\n> void fast_export_note(const char *committish, const char *dataref);\n> void fast_export_begin_note(uint32_t revision, const char *author,\n> -\t\tconst char *log, unsigned long timestamp);\n> +\t\tconst char *log, unsigned long timestamp, const char *note_ref);\n> void fast_export_begin_commit(uint32_t revision, const char *author,\n> -\t\t\tconst struct strbuf *log, const char *uuid,\n> -\t\t\tconst char *url, unsigned long timestamp, const char *local_ref);\n> +\t\t\tconst struct strbuf *log, const char *uuid,const char *url,\n> +\t\t\tunsigned long timestamp, const char *local_ref);\n> void fast_export_end_commit(uint32_t revision);\n> void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input);\n> void fast_export_buf_to_data(const struct strbuf *data);\n> diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\n> index cd65b51..31d1d83 100644\n> --- a/vcs-svn/svndump.c\n> +++ b/vcs-svn/svndump.c\n> @@ -309,20 +309,20 @@ static void begin_revision(const char *remote_ref)\n> \t\trev_ctx.timestamp, remote_ref);\n> }\n> \n> -static void end_revision()\n\n7e11902 (vcs-svn: add a comment before each commit, 2011-01-04)\nadded this as\n\n\tstatic void end_revision(void)\n\nbut it degenerated to pre-ANSI definition at \"[PATCH 8/16] Enable\nfetching to private refs\", which needs to be fixed by losing that\nhunk.\n\n> +static void end_revision(const char *note_ref)\n> {\n> \tstruct strbuf mark = STRBUF_INIT;\n> \tif (rev_ctx.revision) {\n> \t\tfast_export_end_commit(rev_ctx.revision);\n> \t\tfast_export_begin_note(rev_ctx.revision, \"remote-svn\",\n> -\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp);\n> +\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp, note_ref);\n> \t\tstrbuf_addf(&mark, \":%\"PRIu32, rev_ctx.revision);\n> \t\tfast_export_note(mark.buf, \"inline\");\n> \t\tfast_export_buf_to_data(&rev_ctx.note);\n> \t}\n> }\n> \n> -void svndump_read(const char *url, const char *local_ref)\n> +void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n> {\n> \tchar *val;\n> \tchar *t;\n> @@ -363,7 +363,7 @@ void svndump_read(const char *url, const char *local_ref)\n> \t\t\tif (active_ctx == REV_CTX)\n> \t\t\t\tbegin_revision(local_ref);\n> \t\t\tif (active_ctx != DUMP_CTX)\n> -\t\t\t\tend_revision();\n> +\t\t\t\tend_revision(notes_ref);\n> \t\t\tactive_ctx = REV_CTX;\n> \t\t\treset_rev_ctx(atoi(val));\n> \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n> @@ -479,7 +479,7 @@ void svndump_read(const char *url, const char *local_ref)\n> \tif (active_ctx == REV_CTX)\n> \t\tbegin_revision(local_ref);\n> \tif (active_ctx != DUMP_CTX)\n> -\t\tend_revision();\n> +\t\tend_revision(notes_ref);\n> }\n> \n> static void init(int report_fd)\n> diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\n> index febeecb..b8eb129 100644\n> --- a/vcs-svn/svndump.h\n> +++ b/vcs-svn/svndump.h\n> @@ -3,7 +3,7 @@\n> \n> int svndump_init(const char *filename);\n> int svndump_init_fd(int in_fd, int back_fd);\n> -void svndump_read(const char *url, const char *local_ref);\n> +void svndump_read(const char *url, const char *local_ref, const char *notes_ref);\n> void svndump_deinit(void);\n> void svndump_reset(void);\n"},{"id":"198004","messageId":"7vpq6a28tp.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-16-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 15/16] remote-svn: add marks-file regeneration","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T17:59:42Z","receivedAt":"2012-08-28T17:59:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> fast-import mark files are stored outside the object database and are\n> therefore not fetched and can be lost somehow else. marks provide a\n> svn revision --> git sha1 mapping, while the notes that are attached\n> to each commit when it is imported provide a git sha1 --> svn revision\n> mapping.\n>\n> If the marks file is not available or not plausible, regenerate it by\n> walking through the notes tree. , i.e. The plausibility check tests\n> if the highest revision in the marks file matches the revision of the\n> top ref. It doesn't ensure that the mark file is completely correct.\n> This could only be done with an effort equal to unconditional\n> regeneration.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> remote-testsvn.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 68 insertions(+)\n>\n> diff --git a/remote-testsvn.c b/remote-testsvn.c\n> index e90d221..d0b81d5 100644\n> --- a/remote-testsvn.c\n> +++ b/remote-testsvn.c\n> @@ -86,6 +86,73 @@ static int parse_rev_note(const char *msg, struct rev_note *res) {\n> \treturn 0;\n> }\n> \n> +static int note2mark_cb(const unsigned char *object_sha1,\n> +\t\tconst unsigned char *note_sha1, char *note_path,\n> +\t\tvoid *cb_data) {\n> +\tFILE *file = (FILE *)cb_data;\n> +\tchar *msg;\n> +\tunsigned long msglen;\n> +\tenum object_type type;\n> +\tstruct rev_note note;\n> +\tif (!(msg = read_sha1_file(note_sha1, &type, &msglen)) ||\n> +\t\t\t!msglen || type != OBJ_BLOB) {\n> +\t\tfree(msg);\n> +\t\treturn 1;\n> +\t}\n\nThe same comments as an earlier patch in the series applies here,\nregarding chained assignments in coditional, whether each case is an\nerror that needs to be reported, and the sign of the error return\nvalue.\n\n> +\tif (parse_rev_note(msg, ¬e))\n> +\t\treturn 2;\n> +\tif (fprintf(file, \":%d %s\\n\", note.rev_nr, sha1_to_hex(object_sha1)) < 1)\n> +\t\treturn 3;\n> +\treturn 0;\n> +}\n> +\n> +static void regenerate_marks(void)\n> +{\n> +\tint ret;\n> +\tFILE *marksfile;\n> +\tmarksfile = fopen(marksfilename, \"w+\");\n> +\tif (!marksfile)\n> +\t\tdie_errno(\"Couldn't create mark file %s.\", marksfilename);\n> +\tret = for_each_note(NULL, 0, note2mark_cb, marksfile);\n> +\tif (ret)\n> +\t\tdie(\"Regeneration of marks failed, returned %d.\", ret);\n> +\tfclose(marksfile);\n> +}\n> +\n> +static void check_or_regenerate_marks(int latestrev) {\n\nStyle.\n\n> +\tFILE *marksfile;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tstruct strbuf line = STRBUF_INIT;\n> +\tint found = 0;\n> +\n> +\tif (latestrev < 1)\n> +\t\treturn;\n\nIt's more pleasant to read to have a blank line between the end of\ndecls and the first statement, like this function does. Please fix\ntwo functions that appear before this function in this file to match.\n\n> +\tinit_notes(NULL, notes_ref, NULL, 0);\n> +\tmarksfile = fopen(marksfilename, \"r\");\n> +\tif (!marksfile) {\n> +\t\tregenerate_marks();\n> +\t\tmarksfile = fopen(marksfilename, \"r\");\n> +\t\tif (!marksfile)\n> +\t\t\tdie_errno(\"cannot read marks file %s!\", marksfilename);\n> +\t\tfclose(marksfile);\n> +\t} else {\n> +\t\tstrbuf_addf(&sb, \":%d \", latestrev);\n> +\t\twhile (strbuf_getline(&line, marksfile, '\\n') != EOF) {\n> +\t\t\tif (!prefixcmp(line.buf, sb.buf)) {\n> +\t\t\t\tfound++;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t}\n> +\t\tfclose(marksfile);\n> +\t\tif (!found)\n> +\t\t\tregenerate_marks();\n> +\t}\n> +\tfree_notes(NULL);\n> +\tstrbuf_release(&sb);\n> +\tstrbuf_release(&line);\n> +}\n> +\n> static int cmd_import(const char *line)\n> {\n> \tint code;\n> @@ -111,6 +178,7 @@ static int cmd_import(const char *line)\n> \t\t\tfree(note_msg);\n> \t\t}\n> \t}\n> +\tcheck_or_regenerate_marks(startrev - 1);\n> \n> \tif (dump_from_file) {\n> \t\tdumpin_fd = open(url, O_RDONLY);\n"},{"id":"198003","messageId":"7vk3wi28tn.fsf@alter.siamese.dyndns.org","threadId":"31371","inReplyTo":"1346143790-23491-17-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [PATCH v7 16/16] Add a test script for remote-svn","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-28T18:01:56Z","receivedAt":"2012-08-28T18:01:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner writes:\n\n> Use svnrdump_sim.py to emulate svnrdump without an svn server.\n> Tests fetching, incremental fetching, fetching from file://,\n> and the regeneration of fast-import's marks file.\n>\n> Signed-off-by: Florian Achleitner \n> Signed-off-by: Junio C Hamano \n> ---\n> t/t9020-remote-svn.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 82 insertions(+)\n> create mode 100755 t/t9020-remote-svn.sh\n>\n> diff --git a/t/t9020-remote-svn.sh b/t/t9020-remote-svn.sh\n> new file mode 100755\n> index 0000000..e6ed4ca\n> --- /dev/null\n> +++ b/t/t9020-remote-svn.sh\n> @@ -0,0 +1,82 @@\n> +#!/bin/sh\n> +\n> +test_description='tests remote-svn'\n> +\n> +. ./test-lib.sh\n> +\n> +if ! test_have_prereq PYTHON\n> +then\n> +\tskip_all='skipping remote-svn tests, python not available'\n> +\ttest_done\n> +fi\n> +\n> +# We override svnrdump by placing a symlink to the svnrdump-emulator in .\n> +export PATH=\"$HOME:$PATH\"\n> +ln -sf $GIT_BUILD_DIR/contrib/svn-fe/svnrdump_sim.py \"$HOME/svnrdump\"\n> +\n> +init_git () {\n> +\trm -fr .git &&\n> +\tgit init &&\n> +\t#git remote add svnsim testsvn::sim:///$TEST_DIRECTORY/t9020/example.svnrdump\n> +\t# let's reuse an exisiting dump file!?\n> +\tgit remote add svnsim testsvn::sim://$TEST_DIRECTORY/t9154/svn.dump\n> +\tgit remote add svnfile testsvn::file://$TEST_DIRECTORY/t9154/svn.dump\n> +}\n> +\n> +if test -e \"$GIT_BUILD_DIR/git-remote-testsvn\"\n> +then\n> +\ttest_set_prereq REMOTE_SVN\n> +fi\n> +\n> +test_debug '\n> +\tgit --version\n> +\twhich git\n> +\twhich svnrdump\n> +'\n> +\n> +test_expect_success REMOTE_SVN 'simple fetch' '\n> +\tinit_git &&\n> +\tgit fetch svnsim &&\n> +\ttest_cmp .git/refs/svn/svnsim/master .git/refs/remotes/svnsim/master &&\n> +\tcp .git/refs/remotes/svnsim/master master.good\n> +'\n> +\n> +test_debug '\n> +\tcat .git/refs/svn/svnsim/master\n> +\tcat .git/refs/remotes/svnsim/master\n> +'\n> +\n> +test_expect_success REMOTE_SVN 'repeated fetch, nothing shall change' '\n> +\tgit fetch svnsim &&\n> +\ttest_cmp master.good .git/refs/remotes/svnsim/master\n> +'\n> +\n> +test_expect_success REMOTE_SVN 'fetch from a file:// url gives the same result' '\n> +\tgit fetch svnfile\n> +'\n> +\n> +test_expect_failure REMOTE_SVN 'the sha1 differ because the git-svn-id line in the commit msg contains the url' '\n> +\ttest_cmp .git/refs/remotes/svnfile/master .git/refs/remotes/svnsim/master\n> +'\n> +\n> +test_expect_success REMOTE_SVN 'mark-file regeneration' '\n> +\t# filter out any other marks, that can not be regenerated. Only up to 3 digit revisions are allowed here\n> +\tgrep \":[0-9]\\{1,3\\} \" .git/info/fast-import/remote-svn/svnsim.marks > .git/info/fast-import/remote-svn/svnsim.marks.old &&\n> +\trm .git/info/fast-import/remote-svn/svnsim.marks &&\n> +\tgit fetch svnsim &&\n> +\ttest_cmp .git/info/fast-import/remote-svn/svnsim.marks.old .git/info/fast-import/remote-svn/svnsim.marks\n> +'\n\nCould these loooooong lines be made a bit more manageable? It is\ngetting extremely hard to follow.\n\n\n> +test_expect_success REMOTE_SVN 'incremental imports must lead to the same head' '\n> +\texport SVNRMAX=3 &&\n> +\tinit_git &&\n> +\tgit fetch svnsim &&\n> +\ttest_cmp .git/refs/svn/svnsim/master .git/refs/remotes/svnsim/master &&\n> +\tunset SVNRMAX &&\n> +\tgit fetch svnsim &&\n> +\ttest_cmp master.good .git/refs/remotes/svnsim/master\n> +'\n> +\n> +test_debug 'git branch -a'\n> +\n> +test_done\n \n"}]}