{"thread":{"id":"31274","subject":"[RFC 1/5] GSOC: prepare svndump for branch detection","startedAt":"2012-08-17T20:40:05Z","lastAt":"2012-08-20T17:43:53Z","messageCount":12,"participants":["Florian Achleitner","David Michael Barr","Junio C Hamano","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"197225","messageId":"1345236010-1648-1-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":null,"subject":"[RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:05Z","receivedAt":"2012-08-17T20:40:05Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Hi!\n\nThis patch series should prepare vcs-svn/svndump.* for branch \ndetection. When starting with this feature I found that the existing\nfunctions are not yet appropriate for that.\nThese rewrites the node handling part of svndump.c, it is very\ninvasive. The logic in handle_node is not simple, I hope that I \nunderstood every case the existing code tries to adress.\nAt least it doesn't break an existing testcase.\n\nThe series applies on top of:\n[PATCH/RFC v4 16/16] Add a test script for remote-svn.\nI could also rebase it onto master if you think it makes sense.\n\nFlorian\n\n [RFC 1/5] vcs-svn: Add sha1 calculaton to fast_export and\n [RFC 2/5] svndump: move struct definitions to .h.\n [RFC 3/5] vcs-svn/svndump: restructure node_ctx, rev_ctx handling\n [RFC 4/5] vcs-svn/svndump: rewrite handle_node(),\n [RFC 5/5] vcs-svn: remove repo_tree\n"},{"id":"197226","messageId":"1345236010-1648-2-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-1-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[RFC 1/5] vcs-svn: Add sha1 calculaton to fast_export and line_buffer.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:06Z","receivedAt":"2012-08-17T20:40:06Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"The functions fast_export_blob_delta, fast_export_data are used to\nfeed the data of blobs into fast-import. Both use line_buffer's\nbuffer_copy_bytes to write the preprocessed data to the stream.\n\nAllow for calculating the resulting blob's sha1 by adding and\nadding an addtional argument to fast_export_blob_delta and\nfast_export_data that -- if not NULL -- is set to the sha1\nof the new blob.\n\nBecuase these two functions are only used in vcs-svn/, replace\nthe existing ones.\n\nThe calculated sha1 can be used to refer to the blob later\nin the fast-import command stream.\nE.g.:\nadd a blob with the 'blob' command and use it later\ninside a 'commit' command.\n\nSigned-off-by: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>\n---\n test-line-buffer.c    |    2 +-\n vcs-svn/fast_export.c |   35 +++++++++++++++++++++++++++++++----\n vcs-svn/fast_export.h |    5 +++--\n vcs-svn/line_buffer.c |    7 ++++---\n vcs-svn/line_buffer.h |    4 ++--\n vcs-svn/svndump.c     |    4 ++--\n 6 files changed, 43 insertions(+), 14 deletions(-)\n\ndiff --git a/test-line-buffer.c b/test-line-buffer.c\nindex ef1d7ba..0c126b2 100644\n--- a/test-line-buffer.c\n+++ b/test-line-buffer.c\n@@ -29,7 +29,7 @@ static void handle_command(const char *command, const char *arg, struct line_buf\n \t\t}\n \tcase 'c':\n \t\tif (!prefixcmp(command, \"copy \")) {\n-\t\t\tbuffer_copy_bytes(buf, strtouint32(arg));\n+\t\t\tbuffer_copy_bytes(buf, strtouint32(arg), NULL);\n \t\t\treturn;\n \t\t}\n \tcase 's':\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 872ef81..c780d32 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -250,8 +250,11 @@ void fast_export_buf_to_data(const struct strbuf *data)\n \tfputc('\\n', stdout);\n }\n \n-void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input)\n+void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input,\n+\t\tunsigned char sha1[20])\n {\n+\tgit_SHA_CTX sha1_ctx;\n+\tstruct strbuf sb = STRBUF_INIT;\n \tassert(len >= 0);\n \tif (mode == REPO_MODE_LNK) {\n \t\t/* svn symlink blobs start with \"link \" */\n@@ -261,10 +264,21 @@ void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input)\n \t\tif (buffer_skip_bytes(input, 5) != 5)\n \t\t\tdie_short_read(input);\n \t}\n+\tif (sha1) {\n+\t\t/* don't forget to inlude the header in the sha1 */\n+\t\tgit_SHA1_Init(&sha1_ctx);\n+\t\tstrbuf_addf(&sb, \"blob %\" PRIuMAX, (uintmax_t) len);\n+\t\tgit_SHA1_Update(&sha1_ctx, sb.buf, sb.len + 1);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\n \tprintf(\"data %\"PRIuMAX\"\\n\", (uintmax_t) len);\n-\tif (buffer_copy_bytes(input, len) != len)\n+\tif (buffer_copy_bytes(input, len, sha1 ? &sha1_ctx : NULL) != len)\n \t\tdie_short_read(input);\n \tfputc('\\n', stdout);\n+\n+\tif (sha1)\n+\t\tgit_SHA1_Final(sha1, &sha1_ctx);\n }\n \n static int parse_ls_response(const char *response, uint32_t *mode,\n@@ -323,9 +337,11 @@ int fast_export_ls(const char *path, uint32_t *mode, struct strbuf *dataref)\n \n void fast_export_blob_delta(uint32_t mode,\n \t\t\t\tuint32_t old_mode, const char *old_data,\n-\t\t\t\toff_t len, struct line_buffer *input)\n+\t\t\t\toff_t len, struct line_buffer *input, unsigned char sha1[20])\n {\n \tlong postimage_len;\n+\tgit_SHA_CTX sha1_ctx;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \tassert(len >= 0);\n \tpostimage_len = apply_delta(len, input, old_data, old_mode);\n@@ -333,7 +349,18 @@ void fast_export_blob_delta(uint32_t mode,\n \t\tbuffer_skip_bytes(&postimage, strlen(\"link \"));\n \t\tpostimage_len -= strlen(\"link \");\n \t}\n+\tif (sha1) {\n+\t\t/* don't forget to inlude the header in the sha1 */\n+\t\tgit_SHA1_Init(&sha1_ctx);\n+\t\tstrbuf_addf(&sb, \"blob %\" PRIuMAX, (uintmax_t) postimage_len);\n+\t\tgit_SHA1_Update(&sha1_ctx, sb.buf, sb.len + 1);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\n \tprintf(\"data %ld\\n\", postimage_len);\n-\tbuffer_copy_bytes(&postimage, postimage_len);\n+\tbuffer_copy_bytes(&postimage, postimage_len, sha1 ? &sha1_ctx : NULL);\n \tfputc('\\n', stdout);\n+\n+\tif (sha1)\n+\t\tgit_SHA1_Final(sha1, &sha1_ctx);\n }\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nindex c8b5adb..f89e1b1 100644\n--- a/vcs-svn/fast_export.h\n+++ b/vcs-svn/fast_export.h\n@@ -16,11 +16,12 @@ void fast_export_begin_commit(uint32_t revision, const char *author,\n \t\t\tconst struct strbuf *log, const char *uuid,const char *url,\n \t\t\tunsigned long timestamp, const char *local_ref);\n void fast_export_end_commit(uint32_t revision);\n-void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input);\n+void fast_export_data(uint32_t mode, off_t len, struct line_buffer *input,\n+\t\tunsigned char sha1[20]);\n void fast_export_buf_to_data(const struct strbuf *data);\n void fast_export_blob_delta(uint32_t mode,\n \t\t\tuint32_t old_mode, const char *old_data,\n-\t\t\toff_t len, struct line_buffer *input);\n+\t\t\toff_t len, struct line_buffer *input, unsigned char sha1[20]);\n \n /* If there is no such file at that rev, returns -1, errno == ENOENT. */\n int fast_export_ls_rev(uint32_t rev, const char *path,\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nindex 57cc1ce..e9c1b7f 100644\n--- a/vcs-svn/line_buffer.c\n+++ b/vcs-svn/line_buffer.c\n@@ -3,9 +3,8 @@\n  * See LICENSE for details.\n  */\n \n-#include \"git-compat-util.h\"\n+#include \"cache.h\"\n #include \"line_buffer.h\"\n-#include \"strbuf.h\"\n \n #define COPY_BUFFER_LEN 4096\n \n@@ -97,7 +96,7 @@ size_t buffer_read_binary(struct line_buffer *buf,\n \treturn strbuf_fread(sb, size, buf->infile);\n }\n \n-off_t buffer_copy_bytes(struct line_buffer *buf, off_t nbytes)\n+off_t buffer_copy_bytes(struct line_buffer *buf, off_t nbytes, git_SHA_CTX *sha1_ctx)\n {\n \tchar byte_buffer[COPY_BUFFER_LEN];\n \toff_t done = 0;\n@@ -107,6 +106,8 @@ off_t buffer_copy_bytes(struct line_buffer *buf, off_t nbytes)\n \t\tin = fread(byte_buffer, 1, in, buf->infile);\n \t\tdone += in;\n \t\tfwrite(byte_buffer, 1, in, stdout);\n+\t\tif (sha1_ctx)\n+\t\t\tgit_SHA1_Update(sha1_ctx, byte_buffer, in);\n \t\tif (ferror(stdout))\n \t\t\treturn done + buffer_skip_bytes(buf, nbytes - done);\n \t}\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nindex ee23b4f..5b57cdd 100644\n--- a/vcs-svn/line_buffer.h\n+++ b/vcs-svn/line_buffer.h\n@@ -1,7 +1,7 @@\n #ifndef LINE_BUFFER_H_\n #define LINE_BUFFER_H_\n \n-#include \"strbuf.h\"\n+#include \"cache.h\"\n \n #define LINE_BUFFER_LEN 10000\n \n@@ -24,7 +24,7 @@ char *buffer_read_line(struct line_buffer *buf);\n int buffer_read_char(struct line_buffer *buf);\n size_t buffer_read_binary(struct line_buffer *buf, struct strbuf *sb, size_t len);\n /* Returns number of bytes read (not necessarily written). */\n-off_t buffer_copy_bytes(struct line_buffer *buf, off_t len);\n+off_t buffer_copy_bytes(struct line_buffer *buf, off_t len, git_SHA_CTX *sha1_ctx);\n off_t buffer_skip_bytes(struct line_buffer *buf, off_t len);\n \n #endif\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 31d1d83..7592c8e 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -292,12 +292,12 @@ static void handle_node(void)\n \t}\n \tif (!node_ctx.text_delta) {\n \t\tfast_export_modify(node_ctx.dst.buf, node_ctx.type, \"inline\");\n-\t\tfast_export_data(node_ctx.type, node_ctx.text_length, &input);\n+\t\tfast_export_data(node_ctx.type, node_ctx.text_length, &input, NULL);\n \t\treturn;\n \t}\n \tfast_export_modify(node_ctx.dst.buf, node_ctx.type, \"inline\");\n \tfast_export_blob_delta(node_ctx.type, old_mode, old_data,\n-\t\t\t\tnode_ctx.text_length, &input);\n+\t\t\t\tnode_ctx.text_length, &input, NULL);\n }\n \n static void begin_revision(const char *remote_ref)\n-- \n1.7.9.5\n"},{"id":"197227","messageId":"1345236010-1648-3-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-2-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[RFC 2/5] svndump: move struct definitions to .h.","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:07Z","receivedAt":"2012-08-17T20:40:07Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"This is a preparation for adding branch detection code in seperate\n.c files.\nExternal branch detection functions will need to use these.\n---\n vcs-svn/svndump.c |   19 +++----------------\n vcs-svn/svndump.h |   19 +++++++++++++++++++\n 2 files changed, 22 insertions(+), 16 deletions(-)\n\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 7592c8e..296be8c 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -38,23 +38,10 @@\n \n static struct line_buffer input = LINE_BUFFER_INIT;\n \n-static struct {\n-\tuint32_t action, srcRev, type;\n-\toff_t prop_length, text_length;\n-\tstruct strbuf src, dst;\n-\tuint32_t text_delta, prop_delta;\n-} node_ctx;\n+static struct node_ctx_t node_ctx;\n+static struct rev_ctx_t rev_ctx;\n+static struct dump_ctx_t dump_ctx;\n \n-static struct {\n-\tuint32_t revision;\n-\tunsigned long timestamp;\n-\tstruct strbuf log, author, note;\n-} rev_ctx;\n-\n-static struct {\n-\tuint32_t version;\n-\tstruct strbuf uuid, url;\n-} dump_ctx;\n \n static void reset_node_ctx(char *fname)\n {\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex b8eb129..d545453 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -1,5 +1,6 @@\n #ifndef SVNDUMP_H_\n #define SVNDUMP_H_\n+#include \"cache.h\"\n \n int svndump_init(const char *filename);\n int svndump_init_fd(int in_fd, int back_fd);\n@@ -7,4 +8,22 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n void svndump_deinit(void);\n void svndump_reset(void);\n \n+struct node_ctx_t {\n+\tuint32_t action, srcRev, type;\n+\toff_t prop_length, text_length;\n+\tstruct strbuf src, dst;\n+\tuint32_t text_delta, prop_delta;\n+};\n+\n+struct rev_ctx_t {\n+\tuint32_t revision;\n+\tunsigned long timestamp;\n+\tstruct strbuf log, author, note;\n+};\n+\n+struct dump_ctx_t {\n+\tuint32_t version;\n+\tstruct strbuf uuid, url;\n+};\n+\n #endif\n-- \n1.7.9.5\n"},{"id":"197228","messageId":"1345236010-1648-4-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-3-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[RFC 3/5] vcs-svn/svndump: restructure node_ctx, rev_ctx handling","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:08Z","receivedAt":"2012-08-17T20:40:08Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"As a preparation for handling branches in svndumps, make rev_ctx\nand node_ctx more flexible.\n\nAdd the object to work on to the arguments of reset_*_ctx() and to\nhandle_node() to allow for multiple *_ctx objects.\n\nConvert the static global node_ctx to a linked list ofsuch objects\nto be able to accumulate all Node data of a revision in memory\nbefore processing it.\n\nSigned-off-by: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>\n---\n vcs-svn/svndump.c |  207 +++++++++++++++++++++++++++++++----------------------\n vcs-svn/svndump.h |    2 +\n 2 files changed, 124 insertions(+), 85 deletions(-)\n\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 296be8c..2fca9f8 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -38,42 +38,81 @@\n \n static struct line_buffer input = LINE_BUFFER_INIT;\n \n-static struct node_ctx_t node_ctx;\n+static struct node_ctx_t *node_ctx;\n static struct rev_ctx_t rev_ctx;\n static struct dump_ctx_t dump_ctx;\n+static const char *current_ref;\n \n+static struct node_ctx_t *node_list, *node_list_tail;\n \n-static void reset_node_ctx(char *fname)\n+static struct node_ctx_t *new_node_ctx(char *fname)\n {\n-\tnode_ctx.type = 0;\n-\tnode_ctx.action = NODEACT_UNKNOWN;\n-\tnode_ctx.prop_length = -1;\n-\tnode_ctx.text_length = -1;\n-\tstrbuf_reset(&node_ctx.src);\n-\tnode_ctx.srcRev = 0;\n-\tstrbuf_reset(&node_ctx.dst);\n+\tstruct node_ctx_t *node = xmalloc(sizeof(struct node_ctx_t));\n+\ttrace_printf(\"new_node_ctx %p\\n\", node);\n+\tnode->type = 0;\n+\tnode->action = NODEACT_UNKNOWN;\n+\tnode->prop_length = -1;\n+\tnode->text_length = -1;\n+\tstrbuf_init(&node->src, 4096);\n+\tnode->srcRev = 0;\n+\tstrbuf_init(&node->dst, 4096);\n \tif (fname)\n-\t\tstrbuf_addstr(&node_ctx.dst, fname);\n-\tnode_ctx.text_delta = 0;\n-\tnode_ctx.prop_delta = 0;\n+\t\tstrbuf_addstr(&node->dst, fname);\n+\tnode->text_delta = 0;\n+\tnode->prop_delta = 0;\n+\tnode->dataref = NULL;\n+\tnode->next = NULL;\n+\treturn node;\n }\n \n-static void reset_rev_ctx(uint32_t revision)\n+static void free_node_ctx(struct node_ctx_t *node)\n {\n-\trev_ctx.revision = revision;\n-\trev_ctx.timestamp = 0;\n-\tstrbuf_reset(&rev_ctx.log);\n-\tstrbuf_reset(&rev_ctx.author);\n-\tstrbuf_reset(&rev_ctx.note);\n+\ttrace_printf(\"free_node_ctx %p\\n\", node);\n+\tstrbuf_release(&node->src);\n+\tstrbuf_release(&node->dst);\n+\tfree((char*)node->dataref);\n+\tfree(node);\n }\n \n-static void reset_dump_ctx(const char *url)\n+static void free_node_list()\n {\n-\tstrbuf_reset(&dump_ctx.url);\n+\tstruct node_ctx_t *p = node_list, *n;\n+\ttrace_printf(\"free_node_list head %p tail %p\\n\", node_list, node_list_tail);\n+\twhile (p) {\n+\t\tn = p->next;\n+\t\tfree_node_ctx(p);\n+\t\tp = n;\n+\t}\n+\tnode_list = node_list_tail = NULL;\n+}\n+\n+static void append_node_list(struct node_ctx_t *n)\n+{\n+\ttrace_printf(\"append_node_list %p head %p tail %p\\n\", n, node_list, node_list_tail);\n+\tif (!node_list)\n+\t\tnode_list = node_list_tail = n;\n+\telse {\n+\t\tnode_list_tail->next = n;\n+\t\tnode_list_tail = n;\n+\t}\n+}\n+\n+static void reset_rev_ctx(struct rev_ctx_t *rev, uint32_t revision)\n+{\n+\trev->revision = revision;\n+\trev->timestamp = 0;\n+\tstrbuf_reset(&rev->log);\n+\tstrbuf_reset(&rev->author);\n+\tstrbuf_reset(&rev->note);\n+}\n+\n+static void reset_dump_ctx(struct dump_ctx_t *dump, const char *url)\n+{\n+\tstrbuf_reset(&dump->url);\n \tif (url)\n-\t\tstrbuf_addstr(&dump_ctx.url, url);\n-\tdump_ctx.version = 1;\n-\tstrbuf_reset(&dump_ctx.uuid);\n+\t\tstrbuf_addstr(&dump->url, url);\n+\tdump->version = 1;\n+\tstrbuf_reset(&dump->uuid);\n }\n \n static void handle_property(const struct strbuf *key_buf,\n@@ -121,11 +160,11 @@ static void handle_property(const struct strbuf *key_buf,\n \t\t\tdie(\"invalid dump: sets type twice\");\n \t\t}\n \t\tif (!val) {\n-\t\t\tnode_ctx.type = REPO_MODE_BLB;\n+\t\t\tnode_ctx->type = REPO_MODE_BLB;\n \t\t\treturn;\n \t\t}\n \t\t*type_set = 1;\n-\t\tnode_ctx.type = keylen == strlen(\"svn:executable\") ?\n+\t\tnode_ctx->type = keylen == strlen(\"svn:executable\") ?\n \t\t\t\tREPO_MODE_EXE :\n \t\t\t\tREPO_MODE_LNK;\n \t}\n@@ -193,11 +232,11 @@ static void read_props(void)\n \t}\n }\n \n-static void handle_node(void)\n+static void handle_node(struct node_ctx_t *node)\n {\n-\tconst uint32_t type = node_ctx.type;\n-\tconst int have_props = node_ctx.prop_length != -1;\n-\tconst int have_text = node_ctx.text_length != -1;\n+\tconst uint32_t type = node->type;\n+\tconst int have_props = node->prop_length != -1;\n+\tconst int have_text = node->text_length != -1;\n \t/*\n \t * Old text for this node:\n \t *  NULL\t- directory or bug\n@@ -208,21 +247,21 @@ static void handle_node(void)\n \tconst char *old_data = NULL;\n \tuint32_t old_mode = REPO_MODE_BLB;\n \n-\tif (node_ctx.action == NODEACT_DELETE) {\n-\t\tif (have_text || have_props || node_ctx.srcRev)\n+\tif (node->action == NODEACT_DELETE) {\n+\t\tif (have_text || have_props || node->srcRev)\n \t\t\tdie(\"invalid dump: deletion node has \"\n \t\t\t\t\"copyfrom info, text, or properties\");\n-\t\trepo_delete(node_ctx.dst.buf);\n+\t\trepo_delete(node->dst.buf);\n \t\treturn;\n \t}\n-\tif (node_ctx.action == NODEACT_REPLACE) {\n-\t\trepo_delete(node_ctx.dst.buf);\n-\t\tnode_ctx.action = NODEACT_ADD;\n+\tif (node->action == NODEACT_REPLACE) {\n+\t\trepo_delete(node->dst.buf);\n+\t\tnode->action = NODEACT_ADD;\n \t}\n-\tif (node_ctx.srcRev) {\n-\t\trepo_copy(node_ctx.srcRev, node_ctx.src.buf, node_ctx.dst.buf);\n-\t\tif (node_ctx.action == NODEACT_ADD)\n-\t\t\tnode_ctx.action = NODEACT_CHANGE;\n+\tif (node->srcRev) {\n+\t\trepo_copy(node->srcRev, node->src.buf, node->dst.buf);\n+\t\tif (node->action == NODEACT_ADD)\n+\t\t\tnode->action = NODEACT_CHANGE;\n \t}\n \tif (have_text && type == REPO_MODE_DIR)\n \t\tdie(\"invalid dump: directories cannot have text attached\");\n@@ -230,20 +269,20 @@ static void handle_node(void)\n \t/*\n \t * Find old content (old_data) and decide on the new mode.\n \t */\n-\tif (node_ctx.action == NODEACT_CHANGE && !*node_ctx.dst.buf) {\n+\tif (node->action == NODEACT_CHANGE && !*node->dst.buf) {\n \t\tif (type != REPO_MODE_DIR)\n \t\t\tdie(\"invalid dump: root of tree is not a regular file\");\n \t\told_data = NULL;\n-\t} else if (node_ctx.action == NODEACT_CHANGE) {\n+\t} else if (node->action == NODEACT_CHANGE) {\n \t\tuint32_t mode;\n-\t\told_data = repo_read_path(node_ctx.dst.buf, &mode);\n+\t\told_data = repo_read_path(node->dst.buf, &mode); /* malloced buffer */\n \t\tif (mode == REPO_MODE_DIR && type != REPO_MODE_DIR)\n \t\t\tdie(\"invalid dump: cannot modify a directory into a file\");\n \t\tif (mode != REPO_MODE_DIR && type == REPO_MODE_DIR)\n \t\t\tdie(\"invalid dump: cannot modify a file into a directory\");\n-\t\tnode_ctx.type = mode;\n+\t\tnode->type = mode;\n \t\told_mode = mode;\n-\t} else if (node_ctx.action == NODEACT_ADD) {\n+\t} else if (node->action == NODEACT_ADD) {\n \t\tif (type == REPO_MODE_DIR)\n \t\t\told_data = NULL;\n \t\telse if (have_text)\n@@ -258,9 +297,9 @@ static void handle_node(void)\n \t * Adjust mode to reflect properties.\n \t */\n \tif (have_props) {\n-\t\tif (!node_ctx.prop_delta)\n-\t\t\tnode_ctx.type = type;\n-\t\tif (node_ctx.prop_length)\n+\t\tif (!node->prop_delta)\n+\t\t\tnode->type = type;\n+\t\tif (node->prop_length)\n \t\t\tread_props();\n \t}\n \n@@ -274,17 +313,17 @@ static void handle_node(void)\n \t\t/* For the fast_export_* functions, NULL means empty. */\n \t\told_data = NULL;\n \tif (!have_text) {\n-\t\tfast_export_modify(node_ctx.dst.buf, node_ctx.type, old_data);\n+\t\tfast_export_modify(node->dst.buf, node->type, old_data);\n \t\treturn;\n \t}\n-\tif (!node_ctx.text_delta) {\n-\t\tfast_export_modify(node_ctx.dst.buf, node_ctx.type, \"inline\");\n-\t\tfast_export_data(node_ctx.type, node_ctx.text_length, &input, NULL);\n+\tif (!node->text_delta) {\n+\t\tfast_export_modify(node->dst.buf, node->type, \"inline\");\n+\t\tfast_export_data(node->type, node->text_length, &input, NULL);\n \t\treturn;\n \t}\n-\tfast_export_modify(node_ctx.dst.buf, node_ctx.type, \"inline\");\n-\tfast_export_blob_delta(node_ctx.type, old_mode, old_data,\n-\t\t\t\tnode_ctx.text_length, &input, NULL);\n+\tfast_export_modify(node->dst.buf, node->type, \"inline\");\n+\tfast_export_blob_delta(node->type, old_mode, old_data,\n+\t\t\t\tnode->text_length, &input, NULL);\n }\n \n static void begin_revision(const char *remote_ref)\n@@ -316,7 +355,7 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \tuint32_t active_ctx = DUMP_CTX;\n \tuint32_t len;\n \n-\treset_dump_ctx(url);\n+\treset_dump_ctx(&dump_ctx, url);\n \twhile ((t = buffer_read_line(&input))) {\n \t\tval = strchr(t, ':');\n \t\tif (!val)\n@@ -346,13 +385,13 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\tif (constcmp(t, \"Revision-number\"))\n \t\t\t\tcontinue;\n \t\t\tif (active_ctx == NODE_CTX)\n-\t\t\t\thandle_node();\n+\t\t\t\thandle_node(node_ctx);\n \t\t\tif (active_ctx == REV_CTX)\n \t\t\t\tbegin_revision(local_ref);\n \t\t\tif (active_ctx != DUMP_CTX)\n \t\t\t\tend_revision(notes_ref);\n \t\t\tactive_ctx = REV_CTX;\n-\t\t\treset_rev_ctx(atoi(val));\n+\t\t\treset_rev_ctx(&rev_ctx, atoi(val));\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Node-path\"):\n@@ -360,11 +399,11 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\t\tcontinue;\n \t\t\tif (!constcmp(t + strlen(\"Node-\"), \"path\")) {\n \t\t\t\tif (active_ctx == NODE_CTX)\n-\t\t\t\t\thandle_node();\n+\t\t\t\t\thandle_node(node_ctx);\n \t\t\t\tif (active_ctx == REV_CTX)\n \t\t\t\t\tbegin_revision(local_ref);\n \t\t\t\tactive_ctx = NODE_CTX;\n-\t\t\t\treset_node_ctx(val);\n+\t\t\t\tnode_ctx = new_node_ctx(val);\n \t\t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\t\tbreak;\n \t\t\t}\n@@ -372,9 +411,9 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\t\tcontinue;\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tif (!strcmp(val, \"dir\"))\n-\t\t\t\tnode_ctx.type = REPO_MODE_DIR;\n+\t\t\t\tnode_ctx->type = REPO_MODE_DIR;\n \t\t\telse if (!strcmp(val, \"file\"))\n-\t\t\t\tnode_ctx.type = REPO_MODE_BLB;\n+\t\t\t\tnode_ctx->type = REPO_MODE_BLB;\n \t\t\telse\n \t\t\t\tfprintf(stderr, \"Unknown node-kind: %s\\n\", val);\n \t\t\tbreak;\n@@ -383,29 +422,29 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\t\tcontinue;\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tif (!strcmp(val, \"delete\")) {\n-\t\t\t\tnode_ctx.action = NODEACT_DELETE;\n+\t\t\t\tnode_ctx->action = NODEACT_DELETE;\n \t\t\t} else if (!strcmp(val, \"add\")) {\n-\t\t\t\tnode_ctx.action = NODEACT_ADD;\n+\t\t\t\tnode_ctx->action = NODEACT_ADD;\n \t\t\t} else if (!strcmp(val, \"change\")) {\n-\t\t\t\tnode_ctx.action = NODEACT_CHANGE;\n+\t\t\t\tnode_ctx->action = NODEACT_CHANGE;\n \t\t\t} else if (!strcmp(val, \"replace\")) {\n-\t\t\t\tnode_ctx.action = NODEACT_REPLACE;\n+\t\t\t\tnode_ctx->action = NODEACT_REPLACE;\n \t\t\t} else {\n \t\t\t\tfprintf(stderr, \"Unknown node-action: %s\\n\", val);\n-\t\t\t\tnode_ctx.action = NODEACT_UNKNOWN;\n+\t\t\t\tnode_ctx->action = NODEACT_UNKNOWN;\n \t\t\t}\n \t\t\tbreak;\n \t\tcase sizeof(\"Node-copyfrom-path\"):\n \t\t\tif (constcmp(t, \"Node-copyfrom-path\"))\n \t\t\t\tcontinue;\n-\t\t\tstrbuf_reset(&node_ctx.src);\n-\t\t\tstrbuf_addstr(&node_ctx.src, val);\n+\t\t\tstrbuf_reset(&node_ctx->src);\n+\t\t\tstrbuf_addstr(&node_ctx->src, val);\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Node-copyfrom-rev\"):\n \t\t\tif (constcmp(t, \"Node-copyfrom-rev\"))\n \t\t\t\tcontinue;\n-\t\t\tnode_ctx.srcRev = atoi(val);\n+\t\t\tnode_ctx->srcRev = atoi(val);\n \t\t\tstrbuf_addf(&rev_ctx.note, \"%s\\n\", t);\n \t\t\tbreak;\n \t\tcase sizeof(\"Text-content-length\"):\n@@ -424,19 +463,19 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\t\t\tdie(\"unrepresentable length in dump: %s\", val);\n \n \t\t\t\tif (*t == 'T')\n-\t\t\t\t\tnode_ctx.text_length = (off_t) len;\n+\t\t\t\t\tnode_ctx->text_length = (off_t) len;\n \t\t\t\telse\n-\t\t\t\t\tnode_ctx.prop_length = (off_t) len;\n+\t\t\t\t\tnode_ctx->prop_length = (off_t) len;\n \t\t\t\tbreak;\n \t\t\t}\n \t\tcase sizeof(\"Text-delta\"):\n \t\t\tif (!constcmp(t, \"Text-delta\")) {\n-\t\t\t\tnode_ctx.text_delta = !strcmp(val, \"true\");\n+\t\t\t\tnode_ctx->text_delta = !strcmp(val, \"true\");\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\tif (constcmp(t, \"Prop-delta\"))\n \t\t\t\tcontinue;\n-\t\t\tnode_ctx.prop_delta = !strcmp(val, \"true\");\n+\t\t\tnode_ctx->prop_delta = !strcmp(val, \"true\");\n \t\t\tbreak;\n \t\tcase sizeof(\"Content-length\"):\n \t\t\tif (constcmp(t, \"Content-length\"))\n@@ -450,7 +489,7 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \t\t\tif (active_ctx == REV_CTX) {\n \t\t\t\tread_props();\n \t\t\t} else if (active_ctx == NODE_CTX) {\n-\t\t\t\thandle_node();\n+\t\t\t\thandle_node(node_ctx);\n \t\t\t\tactive_ctx = INTERNODE_CTX;\n \t\t\t} else {\n \t\t\t\tfprintf(stderr, \"Unexpected content length header: %\"PRIu32\"\\n\", len);\n@@ -462,7 +501,7 @@ void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n \tif (buffer_ferror(&input))\n \t\tdie_short_read();\n \tif (active_ctx == NODE_CTX)\n-\t\thandle_node();\n+\t\thandle_node(node_ctx);\n \tif (active_ctx == REV_CTX)\n \t\tbegin_revision(local_ref);\n \tif (active_ctx != DUMP_CTX)\n@@ -477,11 +516,10 @@ static void init(int report_fd)\n \tstrbuf_init(&rev_ctx.log, 4096);\n \tstrbuf_init(&rev_ctx.author, 4096);\n \tstrbuf_init(&rev_ctx.note, 4096);\n-\tstrbuf_init(&node_ctx.src, 4096);\n-\tstrbuf_init(&node_ctx.dst, 4096);\n-\treset_dump_ctx(NULL);\n-\treset_rev_ctx(0);\n-\treset_node_ctx(NULL);\n+\treset_dump_ctx(&dump_ctx, NULL);\n+\treset_rev_ctx(&rev_ctx, 0);\n+\tnode_ctx = new_node_ctx(NULL);\n+\tnode_list = node_list_tail = NULL;\n \treturn;\n }\n \n@@ -504,14 +542,12 @@ int svndump_init_fd(int in_fd, int back_fd)\n void svndump_deinit(void)\n {\n \tfast_export_deinit();\n-\treset_dump_ctx(NULL);\n-\treset_rev_ctx(0);\n-\treset_node_ctx(NULL);\n+\treset_dump_ctx(&dump_ctx, NULL);\n+\treset_rev_ctx(&rev_ctx, 0);\n \tstrbuf_release(&rev_ctx.log);\n \tstrbuf_release(&rev_ctx.author);\n \tstrbuf_release(&rev_ctx.note);\n-\tstrbuf_release(&node_ctx.src);\n-\tstrbuf_release(&node_ctx.dst);\n+\tfree_node_list();\n \tif (buffer_deinit(&input))\n \t\tfprintf(stderr, \"Input error\\n\");\n \tif (ferror(stdout))\n@@ -524,4 +560,5 @@ void svndump_reset(void)\n \tstrbuf_release(&dump_ctx.url);\n \tstrbuf_release(&rev_ctx.log);\n \tstrbuf_release(&rev_ctx.author);\n+\tfree_node_list();\n }\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nindex d545453..29794df 100644\n--- a/vcs-svn/svndump.h\n+++ b/vcs-svn/svndump.h\n@@ -13,6 +13,8 @@ struct node_ctx_t {\n \toff_t prop_length, text_length;\n \tstruct strbuf src, dst;\n \tuint32_t text_delta, prop_delta;\n+\tconst char *dataref;\n+\tstruct node_ctx_t *next;\n };\n \n struct rev_ctx_t {\n-- \n1.7.9.5\n"},{"id":"197230","messageId":"1345236010-1648-5-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-4-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[RFC 4/5] vcs-svn/svndump: rewrite handle_node(), begin|end_revision()","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:09Z","receivedAt":"2012-08-17T20:40:09Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Split the decision of what to do and actually doing it in\nhandle_node() to allow for detection of branches from svn nodes.\nSplit it into handle_node() and apply_node().\n\nsvn dumps are structured in revisions, which contain multiple nodes.\nNodes represent operations on data. Currently the function\nhandle_node() strongly mixes the interpretation of the node data\nwith the output of processed data to fast-import.\n\nIn a fast-import stream a commit object requires a branch name to\nwhich the new commit is added at its beginning.\n\nWe want to detect branches in svn. This can only be done by analyzing\nnode operations, like copyfrom. This conflicts with the current\nimplementation, where at the beginning of each new revision in the svn\ndump, a new commit on a hard-coded git branch is created, before even\nreading the first node.\n\nTo allow analyzing the nodes before deciding on which branch the commit\nwill be placed, store the node metadata of one complete revision, and\ncreate a commit from it, when it ends.\n\nEach node can have file data appended. It's desirable to not store the\nactual file data, as it is unbounded.\nfast-import has a 'blob' command that allows writing blobs, independent\nof commits. Use this feature instead of sending data inline and send\nthe actual file data immediately when it is read in.\n\nUse the previously added SHA1 calculation feature of fast_export_data\nand fast_export_blob_delta to retrieve the SHA1 of the written blob\nand reference it later. fast-import's marks can not be used for that,\nbecause they are already used for marking commits, where the mark\nnumber corresponds to exactly one svn revision.\n\nChange handle_node() to interpret the node data, store it in a node_ctx,\nsend blobs to fast-import, and append the new node_ctx to the list of\nnode_ctx.\nDo this until the end of a revision.\n\nJust clear the list of note_ctx in begin_revision().\n\nAt end_revision() all node metadata is available in the node_ctx list.\nFuture's branch detectors can decide what branches are to be changed.\nThen, call apply_node() for each of them to actually create a commit\nand change/add/delete files according to the node_ctx using the already\nadded blobs.\n\nThis can also be used to create commits if the node metadata does not\ncome from a svndump, but is stored in e.g. notes, for later branch\ndetection.\n\nSigned-off-by: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>\n---\n vcs-svn/svndump.c |  165 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 107 insertions(+), 58 deletions(-)\n\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nindex 2fca9f8..6feedd9 100644\n--- a/vcs-svn/svndump.c\n+++ b/vcs-svn/svndump.c\n@@ -48,7 +48,6 @@ static struct node_ctx_t *node_list, *node_list_tail;\n static struct node_ctx_t *new_node_ctx(char *fname)\n {\n \tstruct node_ctx_t *node = xmalloc(sizeof(struct node_ctx_t));\n-\ttrace_printf(\"new_node_ctx %p\\n\", node);\n \tnode->type = 0;\n \tnode->action = NODEACT_UNKNOWN;\n \tnode->prop_length = -1;\n@@ -67,7 +66,6 @@ static struct node_ctx_t *new_node_ctx(char *fname)\n \n static void free_node_ctx(struct node_ctx_t *node)\n {\n-\ttrace_printf(\"free_node_ctx %p\\n\", node);\n \tstrbuf_release(&node->src);\n \tstrbuf_release(&node->dst);\n \tfree((char*)node->dataref);\n@@ -77,7 +75,6 @@ static void free_node_ctx(struct node_ctx_t *node)\n static void free_node_list()\n {\n \tstruct node_ctx_t *p = node_list, *n;\n-\ttrace_printf(\"free_node_list head %p tail %p\\n\", node_list, node_list_tail);\n \twhile (p) {\n \t\tn = p->next;\n \t\tfree_node_ctx(p);\n@@ -88,7 +85,6 @@ static void free_node_list()\n \n static void append_node_list(struct node_ctx_t *n)\n {\n-\ttrace_printf(\"append_node_list %p head %p tail %p\\n\", n, node_list, node_list_tail);\n \tif (!node_list)\n \t\tnode_list = node_list_tail = n;\n \telse {\n@@ -246,23 +242,10 @@ static void handle_node(struct node_ctx_t *node)\n \tstatic const char *const empty_blob = \"::empty::\";\n \tconst char *old_data = NULL;\n \tuint32_t old_mode = REPO_MODE_BLB;\n+\tunsigned char data_sha1[20];\n+\tstruct strbuf sb = STRBUF_INIT;\n+\n \n-\tif (node->action == NODEACT_DELETE) {\n-\t\tif (have_text || have_props || node->srcRev)\n-\t\t\tdie(\"invalid dump: deletion node has \"\n-\t\t\t\t\"copyfrom info, text, or properties\");\n-\t\trepo_delete(node->dst.buf);\n-\t\treturn;\n-\t}\n-\tif (node->action == NODEACT_REPLACE) {\n-\t\trepo_delete(node->dst.buf);\n-\t\tnode->action = NODEACT_ADD;\n-\t}\n-\tif (node->srcRev) {\n-\t\trepo_copy(node->srcRev, node->src.buf, node->dst.buf);\n-\t\tif (node->action == NODEACT_ADD)\n-\t\t\tnode->action = NODEACT_CHANGE;\n-\t}\n \tif (have_text && type == REPO_MODE_DIR)\n \t\tdie(\"invalid dump: directories cannot have text attached\");\n \n@@ -270,28 +253,61 @@ static void handle_node(struct node_ctx_t *node)\n \t * Find old content (old_data) and decide on the new mode.\n \t */\n \tif (node->action == NODEACT_CHANGE && !*node->dst.buf) {\n+\t\t/*\n+\t\t * changes the root of the tree (empty dst), e.g. adding properties.\n+\t\t * see 9e8c5321\n+\t\t */\n \t\tif (type != REPO_MODE_DIR)\n-\t\t\tdie(\"invalid dump: root of tree is not a regular file\");\n+\t\t\tdie(\"invalid dump: root of tree is not a directory\");\n \t\told_data = NULL;\n \t} else if (node->action == NODEACT_CHANGE) {\n \t\tuint32_t mode;\n-\t\told_data = repo_read_path(node->dst.buf, &mode); /* malloced buffer */\n+\t\tif (fast_export_ls_rev(rev_ctx.revision - 1, node->dst.buf, &mode, &sb)) {\n+\t\t\tif (errno != ENOENT)\n+\t\t\t\tdie_errno(\"BUG: unexpected fast_export_ls error\");\n+\t\t\t/* Treat missing paths as directories. */\n+\t\t\tmode = REPO_MODE_DIR;\n+\t\t\told_data = NULL;\n+\t\t} else\n+\t\t\told_data = strbuf_detach(&sb, NULL);\n \t\tif (mode == REPO_MODE_DIR && type != REPO_MODE_DIR)\n-\t\t\tdie(\"invalid dump: cannot modify a directory into a file\");\n+\t\t\tdie(\"invalid dump: cannot modify a directory into a file: \"\n+\t\t\t\t\t\"%s. old_data %s\", node->dst.buf, old_data);\n \t\tif (mode != REPO_MODE_DIR && type == REPO_MODE_DIR)\n-\t\t\tdie(\"invalid dump: cannot modify a file into a directory\");\n+\t\t\tdie(\"invalid dump: cannot modify a file into a directory: %s\",\n+\t\t\t\t\tnode->dst.buf);\n \t\tnode->type = mode;\n \t\told_mode = mode;\n-\t} else if (node->action == NODEACT_ADD) {\n-\t\tif (type == REPO_MODE_DIR)\n+\t} else if (node->action == NODEACT_ADD || node->action == NODEACT_REPLACE) {\n+\t\tif (node->srcRev) {\t/* was copied */\n+\t\t\t/* read dataref and mode from src blob */\n+\t\t\tstrbuf_reset(&sb);\n+\t\t\tif (fast_export_ls_rev(node->srcRev, node->src.buf, &node->type, &sb)) {\n+\t\t\t\tif (errno != ENOENT)\n+\t\t\t\t\tdie_errno(\"BUG: unexpected fast_export_ls_rev error\");\n+\t\t\t\t/*\n+\t\t\t\t * if the dataref is not available, it may be a copy of an empty\n+\t\t\t\t * dir. We delete the target, and write it when the first file\n+\t\t\t\t * is added.\n+\t\t\t\t */\n+\t\t\t\tnode->action = NODEACT_DELETE;\n+\t\t\t\tnode->dataref = NULL;\n+\t\t\t} else\n+\t\t\t\tnode->dataref = strbuf_detach(&sb, NULL);\n+\t\t\told_data = NULL;\n+\t\t} else if (type == REPO_MODE_DIR)\n \t\t\told_data = NULL;\n \t\telse if (have_text)\n \t\t\told_data = empty_blob;\n \t\telse\n \t\t\tdie(\"invalid dump: adds node without text\");\n-\t} else {\n+\t} else if (node->action == NODEACT_DELETE) {\n+\t\told_data = empty_blob;\n+\t\tif (have_text || have_props || node->srcRev)\n+\t\t\tdie(\"invalid dump: deletion node has \"\n+\t\t\t\t\"copyfrom info, text, or properties\");\n+\t} else\n \t\tdie(\"invalid dump: Node-path block lacks Node-action\");\n-\t}\n \n \t/*\n \t * Adjust mode to reflect properties.\n@@ -304,48 +320,81 @@ static void handle_node(struct node_ctx_t *node)\n \t}\n \n \t/*\n-\t * Save the result.\n+\t * Send the data and save the node_ctx.\n \t */\n-\tif (type == REPO_MODE_DIR)\t/* directories are not tracked. */\n-\t\treturn;\n-\tassert(old_data);\n-\tif (old_data == empty_blob)\n-\t\t/* For the fast_export_* functions, NULL means empty. */\n-\t\told_data = NULL;\n-\tif (!have_text) {\n-\t\tfast_export_modify(node->dst.buf, node->type, old_data);\n-\t\treturn;\n+\tif (type != REPO_MODE_DIR) {\t/* directories are not tracked. */\n+\t\tassert(old_data);\n+\t\tif (old_data == empty_blob)\n+\t\t\t/* For the fast_export_* functions, NULL means empty. */\n+\t\t\told_data = NULL;\n+\t\tif (!have_text)\n+\t\t\tnode->dataref = old_data;\n+\t\telse {\n+\t\t\tif (!node->text_delta) {\n+\t\t\t\tprintf(\"blob\\n\");\n+\t\t\t\tfast_export_data(node->type, node->text_length, &input, data_sha1);\n+\t\t\t}\n+\t\t\telse {\n+\t\t\t\tprintf(\"blob\\n\");\n+\t\t\t\tfast_export_blob_delta(node->type, old_mode, old_data,\n+\t\t\t\t\t\tnode->text_length, &input, data_sha1);\n+\t\t\t}\n+\n+\t\t\tstrbuf_addstr(&sb, sha1_to_hex(data_sha1));\n+\t\t\tnode->dataref = sb.buf;\n+\t\t}\n \t}\n-\tif (!node->text_delta) {\n-\t\tfast_export_modify(node->dst.buf, node->type, \"inline\");\n-\t\tfast_export_data(node->type, node->text_length, &input, NULL);\n+\tappend_node_list(node);\n+}\n+\n+static void apply_node(struct node_ctx_t *node)\n+{\n+\tif (node->action == NODEACT_DELETE) {\n+\t\tfast_export_delete(node->dst.buf);\n \t\treturn;\n \t}\n-\tfast_export_modify(node->dst.buf, node->type, \"inline\");\n-\tfast_export_blob_delta(node->type, old_mode, old_data,\n-\t\t\t\tnode->text_length, &input, NULL);\n+\tif (node->action == NODEACT_REPLACE)\n+\t\tfast_export_delete(node->dst.buf);\n+\t/*\n+\t * apply the previously sent node-data to a commit\n+\t */\n+\tif (node->dataref)\n+\t\tfast_export_modify(node->dst.buf, node->type, node->dataref);\n+}\n+\n+\n+static void apply_node_list()\n+{\n+\tstruct node_ctx_t *p = node_list, *n;\n+\twhile (p) {\n+\t\tn = p->next;\n+\t\tapply_node(p);\n+\t\tp = n;\n+\t}\n+}\n+\n+static void begin_revision(const char *remote_ref_)\n+{\n+\tcurrent_ref = remote_ref_;\n+\tfree_node_list();\n }\n \n-static void begin_revision(const char *remote_ref)\n+static void end_revision(const char *note_ref)\n {\n+\tstruct strbuf mark = STRBUF_INIT;\n \tif (!rev_ctx.revision)\t/* revision 0 gets no git commit. */\n \t\treturn;\n \tfast_export_begin_commit(rev_ctx.revision, rev_ctx.author.buf,\n \t\t&rev_ctx.log, dump_ctx.uuid.buf, dump_ctx.url.buf,\n-\t\trev_ctx.timestamp, remote_ref);\n-}\n+\t\trev_ctx.timestamp, current_ref);\n+\tapply_node_list();\n+\tfast_export_end_commit(rev_ctx.revision);\n \n-static void end_revision(const char *note_ref)\n-{\n-\tstruct strbuf mark = STRBUF_INIT;\n-\tif (rev_ctx.revision) {\n-\t\tfast_export_end_commit(rev_ctx.revision);\n-\t\tfast_export_begin_note(rev_ctx.revision, \"remote-svn\",\n-\t\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp, note_ref);\n-\t\tstrbuf_addf(&mark, \":%\"PRIu32, rev_ctx.revision);\n-\t\tfast_export_note(mark.buf, \"inline\");\n-\t\tfast_export_buf_to_data(&rev_ctx.note);\n-\t}\n+\tfast_export_begin_note(rev_ctx.revision, \"remote-svn\",\n+\t\t\t\"Note created by remote-svn.\", rev_ctx.timestamp, note_ref);\n+\tstrbuf_addf(&mark, \":%\"PRIu32, rev_ctx.revision);\n+\tfast_export_note(mark.buf, \"inline\");\n+\tfast_export_buf_to_data(&rev_ctx.note);\n }\n \n void svndump_read(const char *url, const char *local_ref, const char *notes_ref)\n-- \n1.7.9.5\n"},{"id":"197229","messageId":"1345236010-1648-6-git-send-email-florian.achleitner.2.6.31@gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-5-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"[RFC 5/5] vcs-svn: remove repo_tree","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-17T20:40:10Z","receivedAt":"2012-08-17T20:40:10Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"Rewritten svndump.c left only very little functionlity in repo_tree.c\nwhich could easily be inlined. Let's remove it.\n\nSigned-off-by: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>\n---\n Makefile            |    1 -\n vcs-svn/repo_tree.c |   48 ------------------------------------------------\n vcs-svn/repo_tree.h |   14 --------------\n 3 files changed, 63 deletions(-)\n delete mode 100644 vcs-svn/repo_tree.c\n\ndiff --git a/Makefile b/Makefile\nindex 6b0c961..aea877d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2187,7 +2187,6 @@ XDIFF_OBJS += xdiff/xhistogram.o\n \n VCSSVN_OBJS += vcs-svn/line_buffer.o\n VCSSVN_OBJS += vcs-svn/sliding_window.o\n-VCSSVN_OBJS += vcs-svn/repo_tree.o\n VCSSVN_OBJS += vcs-svn/fast_export.o\n VCSSVN_OBJS += vcs-svn/svndiff.o\n VCSSVN_OBJS += vcs-svn/svndump.o\ndiff --git a/vcs-svn/repo_tree.c b/vcs-svn/repo_tree.c\ndeleted file mode 100644\nindex 67d27f0..0000000\n--- a/vcs-svn/repo_tree.c\n+++ /dev/null\n@@ -1,48 +0,0 @@\n-/*\n- * Licensed under a two-clause BSD-style license.\n- * See LICENSE for details.\n- */\n-\n-#include \"git-compat-util.h\"\n-#include \"strbuf.h\"\n-#include \"repo_tree.h\"\n-#include \"fast_export.h\"\n-\n-const char *repo_read_path(const char *path, uint32_t *mode_out)\n-{\n-\tint err;\n-\tstatic struct strbuf buf = STRBUF_INIT;\n-\n-\tstrbuf_reset(&buf);\n-\terr = fast_export_ls(path, mode_out, &buf);\n-\tif (err) {\n-\t\tif (errno != ENOENT)\n-\t\t\tdie_errno(\"BUG: unexpected fast_export_ls error\");\n-\t\t/* Treat missing paths as directories. */\n-\t\t*mode_out = REPO_MODE_DIR;\n-\t\treturn NULL;\n-\t}\n-\treturn buf.buf;\n-}\n-\n-void repo_copy(uint32_t revision, const char *src, const char *dst)\n-{\n-\tint err;\n-\tuint32_t mode;\n-\tstatic struct strbuf data = STRBUF_INIT;\n-\n-\tstrbuf_reset(&data);\n-\terr = fast_export_ls_rev(revision, src, &mode, &data);\n-\tif (err) {\n-\t\tif (errno != ENOENT)\n-\t\t\tdie_errno(\"BUG: unexpected fast_export_ls_rev error\");\n-\t\tfast_export_delete(dst);\n-\t\treturn;\n-\t}\n-\tfast_export_modify(dst, mode, data.buf);\n-}\n-\n-void repo_delete(const char *path)\n-{\n-\tfast_export_delete(path);\n-}\ndiff --git a/vcs-svn/repo_tree.h b/vcs-svn/repo_tree.h\nindex 889c6a3..6d0f51e 100644\n--- a/vcs-svn/repo_tree.h\n+++ b/vcs-svn/repo_tree.h\n@@ -1,23 +1,9 @@\n #ifndef REPO_TREE_H_\n #define REPO_TREE_H_\n \n-struct strbuf;\n-\n #define REPO_MODE_DIR 0040000\n #define REPO_MODE_BLB 0100644\n #define REPO_MODE_EXE 0100755\n #define REPO_MODE_LNK 0120000\n \n-uint32_t next_blob_mark(void);\n-void repo_copy(uint32_t revision, const char *src, const char *dst);\n-void repo_add(const char *path, uint32_t mode, uint32_t blob_mark);\n-const char *repo_read_path(const char *path, uint32_t *mode_out);\n-void repo_delete(const char *path);\n-void repo_commit(uint32_t revision, const char *author,\n-\t\tconst struct strbuf *log, const char *uuid, const char *url,\n-\t\tlong unsigned timestamp);\n-void repo_diff(uint32_t r1, uint32_t r2);\n-void repo_init(void);\n-void repo_reset(void);\n-\n #endif\n-- \n1.7.9.5\n"},{"id":"197253","messageId":"CACPE+fvkTNNHXbFTwhoH7=aQKoc9YqtfTBOkJDugUogni0sYww@mail.gmail.com","threadId":"31274","inReplyTo":"1345236010-1648-1-git-send-email-florian.achleitner.2.6.31@gmail.com","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"David Michael Barr","fromEmail":"b@rr-dav.id.au","sentAt":"2012-08-18T18:37:35Z","receivedAt":"2012-08-18T18:37:35Z","isPatch":false,"sender":{"key":"b@rr-dav.id.au","avatar":"https://gravatar.com/avatar/1c0f0df262aa1749c478ee3586cef5da6d58382c06cb220882b7ef9b93cbec6f?d=mp&s=160"},"body":"On Sat, Aug 18, 2012 at 6:40 AM, Florian Achleitner\n<florian.achleitner.2.6.31@gmail.com> wrote:\n> Hi!\n>\n> This patch series should prepare vcs-svn/svndump.* for branch\n> detection. When starting with this feature I found that the existing\n> functions are not yet appropriate for that.\n> These rewrites the node handling part of svndump.c, it is very\n> invasive. The logic in handle_node is not simple, I hope that I\n> understood every case the existing code tries to adress.\n> At least it doesn't break an existing testcase.\n>\n> The series applies on top of:\n> [PATCH/RFC v4 16/16] Add a test script for remote-svn.\n> I could also rebase it onto master if you think it makes sense.\n>\n> Florian\n>\n>  [RFC 1/5] vcs-svn: Add sha1 calculaton to fast_export and\n\nThis change makes me uncomfortable.\nWe are doubling up on hashing with fast-import.\nThis introduces git-specific logic into vcs-svn.\n\n>  [RFC 2/5] svndump: move struct definitions to .h.\n>  [RFC 3/5] vcs-svn/svndump: restructure node_ctx, rev_ctx handling\n>  [RFC 4/5] vcs-svn/svndump: rewrite handle_node(),\n>  [RFC 5/5] vcs-svn: remove repo_tree\n\nI haven't read the rest of the series yet but I expect\nit is less controversial than the first patch.\n\n--\nDavid Michael Barr\n"},{"id":"197324","messageId":"5679616.CtEd2rlvlO@flomedio","threadId":"31274","inReplyTo":"CACPE+fvkTNNHXbFTwhoH7=aQKoc9YqtfTBOkJDugUogni0sYww@mail.gmail.com","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-20T06:26:04Z","receivedAt":"2012-08-20T06:26:04Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Sunday 19 August 2012 04:37:35 David Michael Barr wrote:\n> On Sat, Aug 18, 2012 at 6:40 AM, Florian Achleitner\n> \n> <florian.achleitner.2.6.31@gmail.com> wrote:\n> > Hi!\n> > \n> > This patch series should prepare vcs-svn/svndump.* for branch\n> > detection. When starting with this feature I found that the existing\n> > functions are not yet appropriate for that.\n> > These rewrites the node handling part of svndump.c, it is very\n> > invasive. The logic in handle_node is not simple, I hope that I\n> > understood every case the existing code tries to adress.\n> > At least it doesn't break an existing testcase.\n> > \n> > The series applies on top of:\n> > [PATCH/RFC v4 16/16] Add a test script for remote-svn.\n> > I could also rebase it onto master if you think it makes sense.\n> > \n> > Florian\n> > \n> >  [RFC 1/5] vcs-svn: Add sha1 calculaton to fast_export and\n> \n> This change makes me uncomfortable.\n> We are doubling up on hashing with fast-import.\n> This introduces git-specific logic into vcs-svn.\n\nYou might need to read the rest of the series to see why I did this.\nShort version: For fast-import, I seperated sending data from the commits, it \nis sent using the 'blob' command.\nYou have two choices of referencing that blobs later, by using a mark, or by \ngiving their sha1. Marks are already used for marking commits, and there is \nonly one \"mark namespace\". So I couldn't use marks to reference the blobs in  \na nice way. This allows for referencing them by their sha1.\n\n> \n> >  [RFC 2/5] svndump: move struct definitions to .h.\n> >  [RFC 3/5] vcs-svn/svndump: restructure node_ctx, rev_ctx handling\n> >  [RFC 4/5] vcs-svn/svndump: rewrite handle_node(),\n> >  [RFC 5/5] vcs-svn: remove repo_tree\n> \n> I haven't read the rest of the series yet but I expect\n> it is less controversial than the first patch.\n\nHm.. I'm not sure ;)\n> \n> --\n> David Michael Barr\n\nFlorian \n"},{"id":"197327","messageId":"7v7gsum4oc.fsf@alter.siamese.dyndns.org","threadId":"31274","inReplyTo":"5679616.CtEd2rlvlO@flomedio","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-20T06:57:23Z","receivedAt":"2012-08-20T06:57:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Florian Achleitner <florian.achleitner.2.6.31@gmail.com> writes:\n\n>> This change makes me uncomfortable.\n>> We are doubling up on hashing with fast-import.\n>> This introduces git-specific logic into vcs-svn.\n\nIIUC, vcs-svn/fast-export is meant to produce a stream in the\nfast-import format, and that format is meant to be VCS agnostic,\nit would need a careful thinking to add anything Git specific to\nit.  If you make other people's importers unable to read from you\nbecause you tell them the contents of blob in Git's terms, that is\nnot very good.\n\n> You have two choices of referencing that blobs later, by using a mark, or by \n> giving their sha1. Marks are already used for marking commits, and there is\n> only one \"mark namespace\". So I couldn't use marks to reference the blobs in  \n> a nice way. This allows for referencing them by their sha1.\n\nSurely you can, by using even and odd numbers (or modulo 4 if you\nmay later want to mark trees and tags as well, but I doubt that is\nneeded), no?\n"},{"id":"197406","messageId":"1701463.olu8UUX9CV@flomedio","threadId":"31274","inReplyTo":"7v7gsum4oc.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-20T13:35:53Z","receivedAt":"2012-08-20T13:35:53Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Sunday 19 August 2012 23:57:23 Junio C Hamano wrote:\n> Florian Achleitner <florian.achleitner.2.6.31@gmail.com> writes:\n> >> This change makes me uncomfortable.\n> >> We are doubling up on hashing with fast-import.\n> >> This introduces git-specific logic into vcs-svn.\n> \n> IIUC, vcs-svn/fast-export is meant to produce a stream in the\n> fast-import format, and that format is meant to be VCS agnostic,\n> it would need a careful thinking to add anything Git specific to\n> it.  If you make other people's importers unable to read from you\n> because you tell them the contents of blob in Git's terms, that is\n> not very good.\n\nGood point.\n\n> \n> > You have two choices of referencing that blobs later, by using a mark, or\n> > by giving their sha1. Marks are already used for marking commits, and\n> > there is only one \"mark namespace\". So I couldn't use marks to reference\n> > the blobs in a nice way. This allows for referencing them by their sha1.\n> \n> Surely you can, by using even and odd numbers (or modulo 4 if you\n> may later want to mark trees and tags as well, but I doubt that is\n> needed), no?\n\nCurrently, the mark number is equal to the svn revision number the commit \ncorresponds to. I didn't want to break that, but not mandatory. We could also \nsplit the mark namespace by reserving one or more of the most significant bits \nas a type specifier. \nI'll develop a marks-based version ..\n"},{"id":"197421","messageId":"20120820164530.GD168@mannheim-rule.local","threadId":"31274","inReplyTo":"1701463.olu8UUX9CV@flomedio","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2012-08-20T16:45:30Z","receivedAt":"2012-08-20T16:45:30Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Florian Achleitner wrote:\n\n> Currently, the mark number is equal to the svn revision number the commit\n> corresponds to. I didn't want to break that, but not mandatory. We could also\n> split the mark namespace by reserving one or more of the most significant bits\n> as a type specifier.\n> I'll develop a marks-based version ..\n\nHave we already exhausted possibilities that don't involve changing\nvcs-svn/ code quite so much?  One possibility mentioned before was to\npost-process the stream that svn-fe produces, which seemed appealing\nfrom a debuggability point of view.\n\nCurious,\nJonathan\n"},{"id":"197428","messageId":"3221500.GV11OifKyt@flomedio","threadId":"31274","inReplyTo":"20120820164530.GD168@mannheim-rule.local","subject":"Re: [RFC 1/5] GSOC: prepare svndump for branch detection","fromName":"Florian Achleitner","fromEmail":"florian.achleitner.2.6.31@gmail.com","sentAt":"2012-08-20T17:43:53Z","receivedAt":"2012-08-20T17:43:53Z","isPatch":false,"sender":{"key":"florian.achleitner.2.6.31@gmail.com","avatar":"https://avatars.githubusercontent.com/u/880777?v=4"},"body":"On Monday 20 August 2012 09:45:30 Jonathan Nieder wrote:\n> Florian Achleitner wrote:\n> > Currently, the mark number is equal to the svn revision number the commit\n> > corresponds to. I didn't want to break that, but not mandatory. We could\n> > also split the mark namespace by reserving one or more of the most\n> > significant bits as a type specifier.\n> > I'll develop a marks-based version ..\n> \n> Have we already exhausted possibilities that don't involve changing\n> vcs-svn/ code quite so much?  One possibility mentioned before was to\n> post-process the stream that svn-fe produces, which seemed appealing\n> from a debuggability point of view.\n> \n\nDo you mean like another program in the pipe, that translates the fast-import \nstream produced by svn-fe into another fast-import stream?\nsvnrdump | svn-fe | svnbranchdetect | git-fast-import ?\n\nMy two previous ideas were meant like this:\n1. Import everything into git and detect branches on the stuff in git, or\n2. detect branches as it imports.\n\nBoth require to create commits for their work. So the idea behind these \npatches is to split the creation of commits from the creation of data. So that \nthe data can be sent immediatly as it is coming in from svnrdump, and \ntherefore save memory by not buffering it. \n\nAnd create the commits later. Either all linear and splitting it into branches \nlater which requires creating commits but not data, or creating branched \ncommits immediatly. This requires to inspect all  node data before starting a \ncommit.\n\nAnyways it's just an idea..\n\n> Curious,\n> Jonathan\n\nHope that helps,\nFlorian\n"}]}