{"thread":{"id":"24005","subject":"[PATCH 0/6] Merge David's SVN exporter","startedAt":"2010-06-04T13:41:05Z","lastAt":"2010-06-07T14:35:48Z","messageCount":15,"participants":["Ramkumar Ramachandra","Jonathan Nieder","David Michael Barr","Erik Faye-Lund"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"142973","messageId":"1275658871-1473-1-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":null,"subject":"[PATCH 0/6] Merge David's SVN exporter","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:05Z","receivedAt":"2010-06-04T13:41:05Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nSorry about the mess- The authorship is corrected in this series.\n\n-- Ram\n\nDavid Barr (5):\n  Add memory pool library\n  Add library for string-specific memory pool\n  Add stream helper library\n  Add infrastructure to write revisions in fast-export format\n  Add SVN dump parser\n\nJason Evans (1):\n  Add cpp macro implementation of treaps\n\n vcs-svn/fast_export.c |   69 ++++++++++\n vcs-svn/fast_export.h |   14 ++\n vcs-svn/line_buffer.c |  129 ++++++++++++++++++\n vcs-svn/line_buffer.h |   14 ++\n vcs-svn/obj_pool.h    |   98 ++++++++++++++\n vcs-svn/repo_tree.c   |  353 +++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/repo_tree.h   |   27 ++++\n vcs-svn/string_pool.c |  110 +++++++++++++++\n vcs-svn/string_pool.h |   14 ++\n vcs-svn/svndump.c     |  294 ++++++++++++++++++++++++++++++++++++++++\n vcs-svn/svndump.h     |    7 +\n vcs-svn/trp.h         |  118 ++++++++++++++++\n vcs-svn/trp.txt       |   62 +++++++++\n 13 files changed, 1309 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/fast_export.c\n create mode 100644 vcs-svn/fast_export.h\n create mode 100644 vcs-svn/line_buffer.c\n create mode 100644 vcs-svn/line_buffer.h\n create mode 100644 vcs-svn/obj_pool.h\n create mode 100644 vcs-svn/repo_tree.c\n create mode 100644 vcs-svn/repo_tree.h\n create mode 100644 vcs-svn/string_pool.c\n create mode 100644 vcs-svn/string_pool.h\n create mode 100644 vcs-svn/svndump.c\n create mode 100644 vcs-svn/svndump.h\n create mode 100644 vcs-svn/trp.h\n create mode 100644 vcs-svn/trp.txt\n"},{"id":"142974","messageId":"1275658871-1473-2-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 1/6] Add memory pool library","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:06Z","receivedAt":"2010-06-04T13:41:06Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: David Barr <david.barr@cordelta.com>\n\nAdd a memory pool library implemented using cpp macros. The library\nprovides macros that can be used to create a type-specific memory pool\nAPI.\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/obj_pool.h |   98 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 98 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/obj_pool.h\n\ndiff --git a/vcs-svn/obj_pool.h b/vcs-svn/obj_pool.h\nnew file mode 100644\nindex 0000000..84c8321\n--- /dev/null\n+++ b/vcs-svn/obj_pool.h\n@@ -0,0 +1,98 @@\n+#ifndef OBJ_POOL_H_\n+#define OBJ_POOL_H_\n+\n+#include \"git-compat-util.h\"\n+\n+/*\n+ * The obj_pool_gen() macro generates a type-specific memory pool\n+ * implementation.\n+ *\n+ * Arguments:\n+ *\n+ *   pre              : Prefix for generated functions (ex: string_).\n+ *   obj_t            : Type for treap data structure (ex: char).\n+ *   intial_capacity  : The initial size of the memory pool (ex: 4096).\n+ *\n+ */\n+#define obj_pool_gen(pre, obj_t, initial_capacity) \\\n+static struct { \\\n+\tuint32_t size; \\\n+\tuint32_t capacity; \\\n+\tobj_t *base; \\\n+        FILE *file; \\\n+} pre##_pool = { 0, 0, NULL, NULL}; \\\n+static void pre##_init(void) \\\n+{ \\\n+\tstruct stat st; \\\n+\tsize_t ps = sysconf (_SC_PAGESIZE); \\\n+\t/* Touch binary file before opening read/write */ \\\n+\tpre##_pool.file = fopen(#pre \".bin\", \"a\"); \\\n+\tfclose(pre##_pool.file); \\\n+\t/* Open, check size, compute capacity */ \\\n+\tpre##_pool.file = fopen(#pre \".bin\", \"r+\"); \\\n+\tfstat(fileno(pre##_pool.file), &st); \\\n+\tpre##_pool.size = st.st_size / sizeof(obj_t); \\\n+\tpre##_pool.capacity = ((st.st_size + ps - 1) & ~(ps - 1)) / sizeof(obj_t); \\\n+\tif (pre##_pool.capacity < initial_capacity) \\\n+\t\tpre##_pool.capacity = initial_capacity; \\\n+\t/* Truncate to calculated capacity and map to VM */ \\\n+\tftruncate(fileno(pre##_pool.file), pre##_pool.capacity * sizeof(obj_t)); \\\n+\tpre##_pool.base = mmap(0, pre##_pool.capacity * sizeof(obj_t), \\\n+\t\t\t\tPROT_READ | PROT_WRITE, MAP_SHARED, \\\n+\t\t\t\tfileno(pre##_pool.file), 0); \\\n+} \\\n+static uint32_t pre##_alloc(uint32_t count) \\\n+{ \\\n+\tuint32_t offset; \\\n+\tif (pre##_pool.size + count > pre##_pool.capacity) { \\\n+\t\tif (NULL == pre##_pool.base) \\\n+\t\t\tpre##_init(); \\\n+\t\tfsync(fileno(pre##_pool.file)); \\\n+\t\tmunmap(pre##_pool.base, \\\n+\t\t\tpre##_pool.capacity * sizeof(obj_t)); \\\n+\t\tpre##_pool.base = NULL; \\\n+\t\twhile (pre##_pool.size + count > pre##_pool.capacity) \\\n+\t\t\tif (pre##_pool.capacity) \\\n+\t\t\t\tpre##_pool.capacity *= 2; \\\n+\t\t\telse \\\n+\t\t\t\tpre##_pool.capacity = initial_capacity; \\\n+\t\tftruncate(fileno(pre##_pool.file), \\\n+\t\t\t\tpre##_pool.capacity * sizeof(obj_t)); \\\n+\t\tpre##_pool.base = \\\n+\t\t\tmmap(0, pre##_pool.capacity * sizeof(obj_t), \\\n+\t\t\t\tPROT_READ | PROT_WRITE, MAP_SHARED, \\\n+\t\t\t\tfileno(pre##_pool.file), 0); \\\n+\t} \\\n+\toffset = pre##_pool.size; \\\n+\tpre##_pool.size += count; \\\n+\treturn offset; \\\n+} \\\n+static void pre##_free(uint32_t count) \\\n+{ \\\n+\tpre##_pool.size -= count; \\\n+} \\\n+static uint32_t pre##_offset(obj_t *obj) \\\n+{ \\\n+\treturn obj == NULL ? ~0 : obj - pre##_pool.base; \\\n+} \\\n+static obj_t *pre##_pointer(uint32_t offset) \\\n+{ \\\n+\treturn offset >= pre##_pool.size ? NULL : &pre##_pool.base[offset]; \\\n+} \\\n+static void pre##_reset(void) \\\n+{ \\\n+\tif (pre##_pool.base) { \\\n+\t\tfsync(fileno(pre##_pool.file)); \\\n+\t\tmunmap(pre##_pool.base, \\\n+\t\t\tpre##_pool.capacity * sizeof(obj_t)); \\\n+\t\tftruncate(fileno(pre##_pool.file), \\\n+\t\t\t\tpre##_pool.size * sizeof(obj_t)); \\\n+\t\tfclose(pre##_pool.file); \\\n+\t} \\\n+\tpre##_pool.base = NULL; \\\n+\tpre##_pool.size = 0; \\\n+\tpre##_pool.capacity = 0; \\\n+\tpre##_pool.file = NULL; \\\n+}\n+\n+#endif\n-- \n1.7.1\n"},{"id":"142975","messageId":"1275658871-1473-3-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 2/6] Add cpp macro implementation of treaps","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:07Z","receivedAt":"2010-06-04T13:41:07Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: Jason Evans <jasone@canonware.com>\n\nThe implementation exposes an API to generate type-specific treap\nimplmentation and various functions to operate on it. It uses\nobj_pool.h to store memory nodes in a treap.\n\nTreaps provide a memory-efficient binary search tree structure.\nInsertion/deletion/search are about as about as fast in the average\ncase as red-black trees and the chances of worst-case behavior are\nvanishingly small, thanks to (pseudo-)randomness.  That is a small\nprice to pay, given that treaps are much simpler to implement.\n\n[db: Altered to reference nodes by offset from a common base pointer]\n[db: Bob Jenkins' hashing implementation dropped for Knuth's]\n[db: Methods unnecessary for search and insert dropped]\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/trp.h   |  118 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/trp.txt |   62 +++++++++++++++++++++++++++++\n 2 files changed, 180 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/trp.h\n create mode 100644 vcs-svn/trp.txt\n\ndiff --git a/vcs-svn/trp.h b/vcs-svn/trp.h\nnew file mode 100644\nindex 0000000..c90f5c3\n--- /dev/null\n+++ b/vcs-svn/trp.h\n@@ -0,0 +1,118 @@\n+/*\n+ * cpp macro implementation of treaps.\n+ *\n+ * Usage:\n+ *   #include <stdint.h>\n+ *   #include <trp.h>\n+ *   trp_gen(...)\n+ */\n+\n+#ifndef TRP_H_\n+#define TRP_H_\n+\n+/* Node structure. */\n+struct trp_node {\n+\tuint32_t trpn_left;\n+\tuint32_t trpn_right;\n+};\n+\n+/* Root structure. */\n+struct trp_root {\n+\tuint32_t trp_root;\n+};\n+\n+/* Pointer/Offset conversion */\n+#define trpn_pointer(a_base, a_offset) (a_base##_pointer(a_offset))\n+#define trpn_offset(a_base, a_pointer) (a_base##_offset(a_pointer))\n+\n+/* Left accessors. */\n+#define trp_left_get(a_base, a_field, a_node) \\\n+\ttrpn_pointer(a_base, (a_node)->a_field.trpn_left)\n+#define trp_left_set(a_base, a_field, a_node, a_left) \\\n+\t(a_node)->a_field.trpn_left = trpn_offset(a_base, a_left)\n+\n+/* Right accessors. */\n+#define trp_right_get(a_base, a_field, a_node) \\\n+\ttrpn_pointer(a_base, (a_node)->a_field.trpn_right)\n+#define trp_right_set(a_base, a_field, a_node, a_right) \\\n+\t(a_node)->a_field.trpn_right = trpn_offset(a_base, a_right)\n+\n+/* Priority accessors. */\n+#define KNUTH_GOLDEN_RATIO_32BIT 2654435761u\n+#define trp_prio_get(a_node) \\\n+\t(KNUTH_GOLDEN_RATIO_32BIT*(uint32_t)(uintptr_t)(a_node))\n+\n+/* Node initializer. */\n+#define trp_node_new(a_base, a_field, a_node) \\\n+\ttrp_left_set(a_base, a_field, (a_node), NULL); \\\n+\ttrp_right_set(a_base, a_field, (a_node), NULL)\n+\n+/* Internal utility macros. */\n+#define trpn_rotate_left(a_base, a_field, a_node, r_node) \\\n+\tdo { (r_node) = trp_right_get(a_base, a_field, (a_node)); \\\n+\ttrp_right_set(a_base, a_field, (a_node), \\\n+\t\ttrp_left_get(a_base, a_field, (r_node))); \\\n+\ttrp_left_set(a_base, a_field, (r_node), (a_node)); } while(0)\n+\n+#define trpn_rotate_right(a_base, a_field, a_node, r_node) \\\n+\tdo { (r_node) = trp_left_get(a_base, a_field, (a_node)); \\\n+\ttrp_left_set(a_base, a_field, (a_node), \\\n+\t\ttrp_right_get(a_base, a_field, (r_node))); \\\n+\ttrp_right_set(a_base, a_field, (r_node), (a_node)); } while(0)\n+\n+#define trp_gen(a_attr, a_pre, a_type, a_field, a_base, a_cmp) \\\n+a_attr a_type *a_pre##psearch(struct trp_root *treap, a_type *key) \\\n+{ \\\n+\ta_type *ret; \\\n+\ta_type *tnode = trpn_pointer(a_base, treap->trp_root); \\\n+\tret = NULL; \\\n+\twhile (tnode != NULL) { \\\n+\t\tint cmp = (a_cmp)(key, tnode); \\\n+\t\tif (cmp < 0) \\\n+\t\t\ttnode = trp_left_get(a_base, a_field, tnode); \\\n+\t\telse if (cmp > 0) { \\\n+\t\t\tret = tnode; \\\n+\t\t\ttnode = trp_right_get(a_base, a_field, tnode); \\\n+\t\t} else { \\\n+\t\t\tret = tnode; \\\n+\t\t\tbreak; \\\n+\t\t} \\\n+\t} \\\n+\treturn (ret); \\\n+} \\\n+a_attr a_type *a_pre##insert_recurse(a_type *cur_node, a_type *ins_node) \\\n+{ \\\n+\tif (cur_node == NULL) \\\n+\t\treturn (ins_node); \\\n+\telse { \\\n+\t\ta_type *ret; \\\n+\t\tint cmp = a_cmp(ins_node, cur_node); \\\n+\t\tif (cmp < 0) { \\\n+\t\t\ta_type *left = a_pre##insert_recurse( \\\n+\t\t\t\ttrp_left_get(a_base, a_field, cur_node), ins_node); \\\n+\t\t\ttrp_left_set(a_base, a_field, cur_node, left); \\\n+\t\t\tif (trp_prio_get(left) < trp_prio_get(cur_node)) \\\n+\t\t\t\ttrpn_rotate_right(a_base, a_field, cur_node, ret); \\\n+\t\t\telse \\\n+\t\t\t\tret = cur_node; \\\n+\t\t} else { \\\n+\t\t\ta_type *right = a_pre##insert_recurse( \\\n+\t\t\t\ttrp_right_get(a_base, a_field, cur_node), ins_node); \\\n+\t\t\ttrp_right_set(a_base, a_field, cur_node, right); \\\n+\t\t\tif (trp_prio_get(right) < trp_prio_get(cur_node)) \\\n+\t\t\t\ttrpn_rotate_left(a_base, a_field, cur_node, ret); \\\n+\t\t\telse \\\n+\t\t\t\tret = cur_node; \\\n+\t\t} \\\n+\t\treturn (ret); \\\n+\t} \\\n+} \\\n+a_attr void a_pre##insert(struct trp_root *treap, a_type *node) \\\n+{ \\\n+\ttrp_node_new(a_base, a_field, node); \\\n+\ttreap->trp_root = trpn_offset(a_base, a_pre##insert_recurse( \\\n+\t\t\t\t\t      trpn_pointer(a_base, treap->trp_root), \\\n+\t\t\t\t\t      node)); \\\n+}\n+\n+#endif\ndiff --git a/vcs-svn/trp.txt b/vcs-svn/trp.txt\nnew file mode 100644\nindex 0000000..7cf9b40\n--- /dev/null\n+++ b/vcs-svn/trp.txt\n@@ -0,0 +1,62 @@\n+TODO: Update this documentation to match the changes to trp.h\n+\n+The trp_gen() macro generates a type-specific treap implementation,\n+based on the above cpp macros.\n+\n+Arguments:\n+\n+  a_attr     : Function attribute for generated functions (ex: static).\n+  a_pre      : Prefix for generated functions (ex: treap_).\n+  a_t_type   : Type for treap data structure (ex: treap_t).\n+  a_type     : Type for treap node data structure (ex: treap_node_t).\n+  a_field    : Name of treap node linkage (ex: treap_link).\n+  a_base     : Expression for the base pointer from which nodes are offset.\n+  a_cmp      : Node comparison function name, with the following prototype:\n+                 int (a_cmp *)(a_type *a_node, a_type *a_other);\n+                                       ^^^^^^\n+                                    or a_key\n+               Interpretation of comparision function return values:\n+                 -1 : a_node <  a_other\n+                  0 : a_node == a_other\n+                  1 : a_node >  a_other\n+               In all cases, the a_node or a_key macro argument is the first\n+               argument to the comparison function, which makes it possible\n+               to write comparison functions that treat the first argument\n+               specially.\n+\n+Assuming the following setup:\n+\n+  typedef struct ex_node_s ex_node_t;\n+  struct ex_node_s {\n+      trp_node(ex_node_t) ex_link;\n+  };\n+  typedef trp(ex_node_t) ex_t;\n+  static ex_node_t ex_base[MAX_NODES];\n+  trp_gen(static, ex_, ex_t, ex_node_t, ex_link, ex_base, ex_cmp)\n+\n+The following API is generated:\n+\n+  static void\n+  ex_new(ex_t *treap);\n+      Description: Initialize a treap structure.\n+      Args:\n+        treap: Pointer to an uninitialized treap object.\n+\n+  static ex_node_t *\n+  ex_psearch(ex_t *treap, ex_node_t *key);\n+      Description: Search for node that matches key.  If no match is found,\n+                   return what would be key's successor/predecessor, were\n+                   key in treap.\n+      Args:\n+        treap: Pointer to a initialized treap object.\n+        key  : Search key.\n+      Ret: Node in treap that matches key, or if no match, hypothetical\n+           node's successor/predecessor (NULL if no successor/predecessor).\n+\n+  static void\n+  ex_insert(ex_t *treap, ex_node_t *node);\n+      Description: Insert node into treap.\n+      Args:\n+        treap: Pointer to a initialized treap object.\n+        node : Node to be inserted into treap.\n+\n-- \n1.7.1\n"},{"id":"142976","messageId":"1275658871-1473-4-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 3/6] Add library for string-specific memory pool","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:08Z","receivedAt":"2010-06-04T13:41:08Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: David Barr <david.barr@cordelta.com>\n\nThis library uses the macros in the obj_pool.h and trp.h to create a\nmemory pool for strings and expose an API for handling them.\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/string_pool.c |  110 +++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/string_pool.h |   14 ++++++\n 2 files changed, 124 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/string_pool.c\n create mode 100644 vcs-svn/string_pool.h\n\ndiff --git a/vcs-svn/string_pool.c b/vcs-svn/string_pool.c\nnew file mode 100644\nindex 0000000..cfcf127\n--- /dev/null\n+++ b/vcs-svn/string_pool.c\n@@ -0,0 +1,110 @@\n+#include \"git-compat-util.h\"\n+\n+#include \"trp.h\"\n+#include \"obj_pool.h\"\n+#include \"string_pool.h\"\n+\n+typedef struct node_s node_t;\n+static struct trp_root tree = { ~0 };\n+\n+struct node_s {\n+\tuint32_t offset;\n+\tstruct trp_node children;\n+};\n+\n+/* Create two memory pools: one for node_t, and another for strings */\n+obj_pool_gen(node, node_t, 4096);\n+obj_pool_gen(string, char, 4096);\n+\n+static char *node_value(node_t *node)\n+{\n+\treturn node ? string_pointer(node->offset) : NULL;\n+}\n+\n+static int node_value_cmp(node_t *a, node_t *b)\n+{\n+\treturn strcmp(node_value(a), node_value(b));\n+}\n+\n+static int node_indentity_cmp(node_t *a, node_t *b)\n+{\n+\tint r = node_value_cmp(a, b);\n+\treturn r ? r : (((uintptr_t) a) > ((uintptr_t) b))\n+\t\t- (((uintptr_t) a) < ((uintptr_t) b));\n+}\n+\n+/* Build a Treap from the node_s structure (a trp_node w/ offset) */\n+trp_gen(static, tree_, node_t, children, node, node_indentity_cmp);\n+\n+char *pool_fetch(uint32_t entry)\n+{\n+\treturn node_value(node_pointer(entry));\n+}\n+\n+uint32_t pool_intern(char *key)\n+{\n+\t/* Canonicalize key */\n+\tnode_t *match = NULL;\n+\tuint32_t key_len;\n+\tif (key == NULL)\n+\t\treturn ~0;\n+\tkey_len = strlen(key) + 1;\n+\tnode_t *node = node_pointer(node_alloc(1));\n+\tnode->offset = string_alloc(key_len);\n+\tstrcpy(node_value(node), key);\n+\tmatch = tree_psearch(&tree, node);\n+\tif (!match || node_value_cmp(node, match)) {\n+\t\ttree_insert(&tree, node);\n+\t} else {\n+\t\tnode_free(1);\n+\t\tstring_free(key_len);\n+\t\tnode = match;\n+\t}\n+\treturn node_offset(node);\n+}\n+\n+uint32_t pool_tok_r(char *str, const char *delim, char **saveptr)\n+{\n+\tchar *token = strtok_r(str, delim, saveptr);\n+\treturn token ? pool_intern(token) : ~0;\n+}\n+\n+void pool_print_seq(uint32_t len, uint32_t *seq, char delim, FILE *stream)\n+{\n+\tuint32_t i;\n+\tfor (i = 0; i < len && ~seq[i]; i++) {\n+\t\tfputs(pool_fetch(seq[i]), stream);\n+\t\tif (i < len - 1 && ~seq[i + 1])\n+\t\t\tfputc(delim, stream);\n+\t}\n+}\n+\n+uint32_t pool_tok_seq(uint32_t max, uint32_t *seq, char *delim, char *str)\n+{\n+\tchar *context = NULL;\n+\tuint32_t length = 0, token = str ? pool_tok_r(str, delim, &context) : ~0;\n+\twhile (length < max) {\n+\t\tseq[length++] = token;\n+\t\tif (token == ~0)\n+\t\t\tbreak;\n+\t\ttoken = pool_tok_r(NULL, delim, &context);\n+\t}\n+\tseq[length ? length - 1 : 0] = ~0;\n+\treturn length;\n+}\n+\n+void pool_init(void)\n+{\n+\tuint32_t node;\n+\tnode_init();\n+\tstring_init();\n+\tfor (node = 0; node < node_pool.size; node++) {\n+\t\ttree_insert(&tree, node_pointer(node));\n+\t}\n+}\n+\n+void pool_reset(void)\n+{\n+\tnode_reset();\n+\tstring_reset();\n+}\ndiff --git a/vcs-svn/string_pool.h b/vcs-svn/string_pool.h\nnew file mode 100644\nindex 0000000..e2cc447\n--- /dev/null\n+++ b/vcs-svn/string_pool.h\n@@ -0,0 +1,14 @@\n+#ifndef STRING_POOL_H_\n+#define\tSTRING_POOL_H_\n+\n+#include \"git-compat-util.h\"\n+\n+uint32_t pool_intern(char *key);\n+char *pool_fetch(uint32_t entry);\n+uint32_t pool_tok_r(char *str, const char *delim, char **saveptr);\n+void pool_print_seq(uint32_t len, uint32_t *seq, char delim, FILE *stream);\n+uint32_t pool_tok_seq(uint32_t max, uint32_t *seq, char *delim, char *str);\n+void pool_init(void);\n+void pool_reset(void);\n+\n+#endif\n-- \n1.7.1\n"},{"id":"142978","messageId":"1275658871-1473-5-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 4/6] Add stream helper library","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:09Z","receivedAt":"2010-06-04T13:41:09Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: David Barr <david.barr@cordelta.com>\n\nThis library provides facilities to read streams into buffers. It\nmaintains a couple of static buffers and provides an API to use them.\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/line_buffer.c |  129 +++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/line_buffer.h |   14 +++++\n 2 files changed, 143 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/line_buffer.c\n create mode 100644 vcs-svn/line_buffer.h\n\ndiff --git a/vcs-svn/line_buffer.c b/vcs-svn/line_buffer.c\nnew file mode 100644\nindex 0000000..740676d\n--- /dev/null\n+++ b/vcs-svn/line_buffer.c\n@@ -0,0 +1,129 @@\n+#include \"git-compat-util.h\"\n+\n+#include \"line_buffer.h\"\n+#include \"obj_pool.h\"\n+\n+#define LINE_BUFFER_LEN 10000\n+#define COPY_BUFFER_LEN 4096\n+\n+/* Create memory pool for char sequence of known length */\n+obj_pool_gen(blob, char, 4096);\n+\n+static char line_buffer[LINE_BUFFER_LEN];\n+static char byte_buffer[COPY_BUFFER_LEN];\n+static uint32_t line_buffer_len = 0;\n+static uint32_t line_len = 0;\n+static FILE *infile;\n+\n+int buffer_init(char *filename)\n+{\n+\tinfile = fopen(filename, \"r\");\n+\tif (!infile)\n+\t\treturn 1;\n+\treturn 0;\n+}\n+\n+int buffer_deinit()\n+{\n+\tfclose(infile);\n+\treturn 0;\n+}\n+\n+char *buffer_read_line(void)\n+{\n+\tchar *end;\n+\tuint32_t n_read;\n+\n+\tif (line_len) {\n+\t\tmemmove(line_buffer, &line_buffer[line_len],\n+\t\t\tline_buffer_len - line_len);\n+\t\tline_buffer_len -= line_len;\n+\t\tline_len = 0;\n+\t}\n+\n+\tend = memchr(line_buffer, '\\n', line_buffer_len);\n+\twhile (line_buffer_len < LINE_BUFFER_LEN - 1 &&\n+\t       !feof(infile) && ferror(infile) && NULL == end) {\n+\t\tn_read = fread(&line_buffer[line_buffer_len], 1,\n+\t\t\t       LINE_BUFFER_LEN - 1 - line_buffer_len,\n+\t\t\t       infile);\n+\t\tend = memchr(&line_buffer[line_buffer_len], '\\n', n_read);\n+\t\tline_buffer_len += n_read;\n+\t}\n+\n+\tif (ferror(infile))\n+\t\treturn NULL;\n+\n+\tif (end != NULL) {\n+\t\tline_len = end - line_buffer;\n+\t\tline_buffer[line_len++] = '\\0';\n+\t} else {\n+\t\tline_len = line_buffer_len;\n+\t\tline_buffer[line_buffer_len] = '\\0';\n+\t}\n+\n+\tif (line_len == 0)\n+\t\treturn NULL;\n+\n+\treturn line_buffer;\n+}\n+\n+char *buffer_read_string(uint32_t len)\n+{\n+\tchar *s;\n+\tblob_free(blob_pool.size);\n+\ts = blob_pointer(blob_alloc(len + 1));\n+\tuint32_t offset = 0;\n+\tif (line_buffer_len > line_len) {\n+\t\toffset = line_buffer_len - line_len;\n+\t\tif (offset > len)\n+\t\t\toffset = len;\n+\t\tmemcpy(s, &line_buffer[line_len], offset);\n+\t\tline_len += offset;\n+\t}\n+\tif (offset < len)\n+\t\toffset += fread(&s[offset], 1, len - offset, infile);\n+\ts[offset] = '\\0';\n+\treturn s;\n+}\n+\n+void buffer_copy_bytes(uint32_t len)\n+{\n+\tuint32_t in;\n+\tif (line_buffer_len > line_len) {\n+\t\tin = line_buffer_len - line_len;\n+\t\tif (in > len)\n+\t\t\tin = len;\n+\t\tfwrite(&line_buffer[line_len], 1, in, stdout);\n+\t\tlen -= in;\n+\t\tline_len += in;\n+\t}\n+\twhile (len > 0 && !feof(infile)) {\n+\t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n+\t\tin = fread(byte_buffer, 1, in, infile);\n+\t\tlen -= in;\n+\t\tfwrite(byte_buffer, 1, in, stdout);\n+\t}\n+}\n+\n+void buffer_skip_bytes(uint32_t len)\n+{\n+\tuint32_t in;\n+\tif (line_buffer_len > line_len) {\n+\t\tin = line_buffer_len - line_len;\n+\t\tif (in > len)\n+\t\t\tin = len;\n+\t\tline_len += in;\n+\t\tlen -= in;\n+\t}\n+\twhile (len > 0 && !feof(infile) && !ferror(infile)) {\n+\t\tin = len < COPY_BUFFER_LEN ? len : COPY_BUFFER_LEN;\n+\t\tin = fread(byte_buffer, 1, in, infile);\n+\t\tlen -= in;\n+\t}\n+}\n+\n+void buffer_reset(void)\n+{\n+\tblob_reset();\n+}\ndiff --git a/vcs-svn/line_buffer.h b/vcs-svn/line_buffer.h\nnew file mode 100644\nindex 0000000..a6c42d7\n--- /dev/null\n+++ b/vcs-svn/line_buffer.h\n@@ -0,0 +1,14 @@\n+#ifndef LINE_BUFFER_H_\n+#define LINE_BUFFER_H_\n+\n+#include <stdint.h>\n+\n+int buffer_init(char *filename);\n+int buffer_deinit(void);\n+char *buffer_read_line(void);\n+char *buffer_read_string(uint32_t len);\n+void buffer_copy_bytes(uint32_t len);\n+void buffer_skip_bytes(uint32_t len);\n+void buffer_reset(void);\n+\n+#endif\n-- \n1.7.1\n"},{"id":"142977","messageId":"1275658871-1473-6-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 5/6] Add infrastructure to write revisions in fast-export format","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:10Z","receivedAt":"2010-06-04T13:41:10Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: David Barr <david.barr@cordelta.com>\n\nrepo_tree maintains the exporter's state and provides a facility to\nto call fast_export, which then writes objects to stdout suitable for\nconsumption by git-fast-import.\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/fast_export.c |   69 ++++++++++\n vcs-svn/fast_export.h |   14 ++\n vcs-svn/repo_tree.c   |  353 +++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/repo_tree.h   |   27 ++++\n 4 files changed, 463 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/fast_export.c\n create mode 100644 vcs-svn/fast_export.h\n create mode 100644 vcs-svn/repo_tree.c\n create mode 100644 vcs-svn/repo_tree.h\n\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nnew file mode 100644\nindex 0000000..e5eb409\n--- /dev/null\n+++ b/vcs-svn/fast_export.c\n@@ -0,0 +1,69 @@\n+#include \"git-compat-util.h\"\n+\n+#include \"fast_export.h\"\n+#include \"line_buffer.h\"\n+#include \"repo_tree.h\"\n+#include \"string_pool.h\"\n+\n+#define MAX_GITSVN_LINE_LEN 4096\n+\n+static uint32_t first_commit_done;\n+\n+void fast_export_delete(uint32_t depth, uint32_t *path)\n+{\n+\tputchar('D');\n+\tputchar(' ');\n+\tpool_print_seq(depth, path, '/', stdout);\n+\tputchar('\\n');\n+}\n+\n+void fast_export_modify(uint32_t depth, uint32_t *path, uint32_t mode,\n+                        uint32_t mark)\n+{\n+\tprintf(\"M %06o :%d \", mode, mark);\n+\tpool_print_seq(depth, path, '/', stdout);\n+\tputchar('\\n');\n+}\n+\n+static char gitsvnline[MAX_GITSVN_LINE_LEN];\n+void fast_export_commit(uint32_t revision, uint32_t author, char *log,\n+                        uint32_t uuid, uint32_t url,\n+                        unsigned long timestamp)\n+{\n+\tif (!log)\n+\t\tlog = \"\";\n+\tif (~uuid && ~url) {\n+\t\tsnprintf(gitsvnline, MAX_GITSVN_LINE_LEN, \"\\n\\ngit-svn-id: %s@%d %s\\n\",\n+\t\t\t\t pool_fetch(url), revision, pool_fetch(uuid));\n+\t} else {\n+\t\t*gitsvnline = '\\0';\n+\t}\n+\tprintf(\"commit refs/heads/master\\n\");\n+\tprintf(\"committer %s <%s@%s> %ld +0000\\n\",\n+\t\t   ~author ? pool_fetch(author) : \"nobody\",\n+\t\t   ~author ? pool_fetch(author) : \"nobody\",\n+\t\t   ~uuid ? pool_fetch(uuid) : \"local\", timestamp);\n+\tprintf(\"data %zd\\n%s%s\\n\",\n+\t\t   strlen(log) + strlen(gitsvnline), log, gitsvnline);\n+\tif (!first_commit_done) {\n+\t\tif (revision > 1)\n+\t\t\tprintf(\"from refs/heads/master^0\\n\");\n+\t\tfirst_commit_done = 1;\n+\t}\n+\trepo_diff(revision - 1, revision);\n+\tfputc('\\n', stdout);\n+\n+\tprintf(\"progress Imported commit %d.\\n\\n\", revision);\n+}\n+\n+void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len)\n+{\n+\tif (mode == REPO_MODE_LNK) {\n+\t\t/* svn symlink blobs start with \"link \" */\n+\t\tbuffer_skip_bytes(5);\n+\t\tlen -= 5;\n+\t}\n+\tprintf(\"blob\\nmark :%d\\ndata %d\\n\", mark, len);\n+\tbuffer_copy_bytes(len);\n+\tfputc('\\n', stdout);\n+}\ndiff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h\nnew file mode 100644\nindex 0000000..e9ea3ed\n--- /dev/null\n+++ b/vcs-svn/fast_export.h\n@@ -0,0 +1,14 @@\n+#ifndef FAST_EXPORT_H_\n+#define FAST_EXPORT_H_\n+\n+#include <stdint.h>\n+#include <time.h>\n+\n+void fast_export_delete(uint32_t depth, uint32_t *path);\n+void fast_export_modify(uint32_t depth, uint32_t *path, uint32_t mode,\n+                        uint32_t mark);\n+void fast_export_commit(uint32_t revision, uint32_t author, char *log,\n+                        uint32_t uuid, uint32_t url, unsigned long timestamp);\n+void fast_export_blob(uint32_t mode, uint32_t mark, uint32_t len);\n+\n+#endif\ndiff --git a/vcs-svn/repo_tree.c b/vcs-svn/repo_tree.c\nnew file mode 100644\nindex 0000000..5dfffb8\n--- /dev/null\n+++ b/vcs-svn/repo_tree.c\n@@ -0,0 +1,353 @@\n+#include \"git-compat-util.h\"\n+\n+#include \"string_pool.h\"\n+#include \"repo_tree.h\"\n+#include \"obj_pool.h\"\n+#include \"fast_export.h\"\n+\n+struct repo_dirent {\n+\tuint32_t name_offset;\n+\tuint32_t mode;\n+\tuint32_t content_offset;\n+};\n+\n+struct repo_dir {\n+\tuint32_t size;\n+\tuint32_t first_offset;\n+};\n+\n+struct repo_commit {\n+\tuint32_t mark;\n+\tuint32_t root_dir_offset;\n+};\n+\n+/* Generate memory pools for commit, dir and dirent */\n+obj_pool_gen(commit, struct repo_commit, 4096);\n+obj_pool_gen(dir, struct repo_dir, 4096);\n+obj_pool_gen(dirent, struct repo_dirent, 4096);\n+\n+static uint32_t num_dirs_saved;\n+static uint32_t num_dirents_saved;\n+static uint32_t active_commit;\n+static uint32_t _mark;\n+\n+uint32_t next_blob_mark(void)\n+{\n+\treturn _mark++;\n+}\n+\n+static struct repo_dir *repo_commit_root_dir(struct repo_commit *commit)\n+{\n+\treturn dir_pointer(commit->root_dir_offset);\n+}\n+\n+static struct repo_dirent *repo_first_dirent(struct repo_dir *dir)\n+{\n+\treturn dirent_pointer(dir->first_offset);\n+}\n+\n+static int repo_dirent_name_cmp(const void *a, const void *b)\n+{\n+\tconst struct repo_dirent *dirent1 = a, *dirent2 = b;\n+\tuint32_t a_offset = dirent1->name_offset;\n+\tuint32_t b_offset = dirent2->name_offset;\n+\treturn (a_offset > b_offset) - (a_offset < b_offset);\n+}\n+\n+static struct repo_dirent *repo_dirent_by_name(struct repo_dir *dir,\n+                                          uint32_t name_offset)\n+{\n+\tstruct repo_dirent key;\n+\tif (dir == NULL || dir->size == 0)\n+\t\treturn NULL;\n+\tkey.name_offset = name_offset;\n+\treturn bsearch(&key, repo_first_dirent(dir), dir->size,\n+\t\t\t\t   sizeof(struct repo_dirent), repo_dirent_name_cmp);\n+}\n+\n+static int repo_dirent_is_dir(struct repo_dirent *dirent)\n+{\n+\treturn dirent != NULL && dirent->mode == REPO_MODE_DIR;\n+}\n+\n+static struct repo_dir *repo_dir_from_dirent(struct repo_dirent *dirent)\n+{\n+\tif (!repo_dirent_is_dir(dirent))\n+\t\treturn NULL;\n+\treturn dir_pointer(dirent->content_offset);\n+}\n+\n+static uint32_t dir_with_dirents_alloc(uint32_t size)\n+{\n+\tuint32_t offset = dir_alloc(1);\n+\tdir_pointer(offset)->size = size;\n+\tdir_pointer(offset)->first_offset = dirent_alloc(size);\n+\treturn offset;\n+}\n+\n+static struct repo_dir *repo_clone_dir(struct repo_dir *orig_dir, uint32_t padding)\n+{\n+\tuint32_t orig_o, new_o, dirent_o;\n+\torig_o = dir_offset(orig_dir);\n+\tif (orig_o < num_dirs_saved) {\n+\t\tnew_o = dir_with_dirents_alloc(orig_dir->size + padding);\n+\t\torig_dir = dir_pointer(orig_o);\n+\t\tdirent_o = dir_pointer(new_o)->first_offset;\n+\t} else {\n+\t\tif (padding == 0)\n+\t\t\treturn orig_dir;\n+\t\tnew_o = orig_o;\n+\t\tdirent_o = dirent_alloc(orig_dir->size + padding);\n+\t}\n+\tmemcpy(dirent_pointer(dirent_o), repo_first_dirent(orig_dir),\n+\t\t   orig_dir->size * sizeof(struct repo_dirent));\n+\tdir_pointer(new_o)->size = orig_dir->size + padding;\n+\tdir_pointer(new_o)->first_offset = dirent_o;\n+\treturn dir_pointer(new_o);\n+}\n+\n+static struct repo_dirent *repo_read_dirent(uint32_t revision, uint32_t *path)\n+{\n+\tuint32_t name = 0;\n+\tstruct repo_dir *dir = NULL;\n+\tstruct repo_dirent *dirent = NULL;\n+\tdir = repo_commit_root_dir(commit_pointer(revision));\n+\twhile (~(name = *path++)) {\n+\t\tdirent = repo_dirent_by_name(dir, name);\n+\t\tif (dirent == NULL) {\n+\t\t\treturn NULL;\n+\t\t} else if (repo_dirent_is_dir(dirent)) {\n+\t\t\tdir = repo_dir_from_dirent(dirent);\n+\t\t} else {\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn dirent;\n+}\n+\n+static void\n+repo_write_dirent(uint32_t *path, uint32_t mode, uint32_t content_offset,\n+                  uint32_t del)\n+{\n+\tuint32_t name, revision, dirent_o = ~0, dir_o = ~0, parent_dir_o = ~0;\n+\tstruct repo_dir *dir;\n+\tstruct repo_dirent *dirent = NULL;\n+\trevision = active_commit;\n+\tdir = repo_commit_root_dir(commit_pointer(revision));\n+\tdir = repo_clone_dir(dir, 0);\n+\tcommit_pointer(revision)->root_dir_offset = dir_offset(dir);\n+\twhile (~(name = *path++)) {\n+\t\tparent_dir_o = dir_offset(dir);\n+\t\tdirent = repo_dirent_by_name(dir, name);\n+\t\tif (dirent == NULL) {\n+\t\t\tdir = repo_clone_dir(dir, 1);\n+\t\t\tdirent = &repo_first_dirent(dir)[dir->size - 1];\n+\t\t\tdirent->name_offset = name;\n+\t\t\tdirent->mode = REPO_MODE_DIR;\n+\t\t\tqsort(repo_first_dirent(dir), dir->size,\n+\t\t\t\t  sizeof(struct repo_dirent), repo_dirent_name_cmp);\n+\t\t\tdirent = repo_dirent_by_name(dir, name);\n+\t\t\tdir_o = dir_with_dirents_alloc(0);\n+\t\t\tdirent->content_offset = dir_o;\n+\t\t\tdir = dir_pointer(dir_o);\n+\t\t} else if ((dir = repo_dir_from_dirent(dirent))) {\n+\t\t\tdirent_o = dirent_offset(dirent);\n+\t\t\tdir = repo_clone_dir(dir, 0);\n+\t\t\tif (dirent_o != ~0)\n+\t\t\t\tdirent_pointer(dirent_o)->content_offset = dir_offset(dir);\n+\t\t} else {\n+\t\t\tdirent->mode = REPO_MODE_DIR;\n+\t\t\tdirent_o = dirent_offset(dirent);\n+\t\t\tdir_o = dir_with_dirents_alloc(0);\n+\t\t\tdirent = dirent_pointer(dirent_o);\n+\t\t\tdir = dir_pointer(dir_o);\n+\t\t\tdirent->content_offset = dir_o;\n+\t\t}\n+\t}\n+\tif (dirent) {\n+\t\tdirent->mode = mode;\n+\t\tdirent->content_offset = content_offset;\n+\t\tif (del && ~parent_dir_o) {\n+\t\t\tdirent->name_offset = ~0;\n+\t\t\tdir = dir_pointer(parent_dir_o);\n+\t\t\tqsort(repo_first_dirent(dir), dir->size,\n+\t\t\t\t  sizeof(struct repo_dirent), repo_dirent_name_cmp);\n+\t\t\tdir->size--;\n+\t\t}\n+\t}\n+}\n+\n+uint32_t repo_copy(uint32_t revision, uint32_t *src, uint32_t *dst)\n+{\n+\tuint32_t mode = 0, content_offset = 0;\n+\tstruct repo_dirent *src_dirent;\n+\tsrc_dirent = repo_read_dirent(revision, src);\n+\tif (src_dirent != NULL) {\n+\t\tmode = src_dirent->mode;\n+\t\tcontent_offset = src_dirent->content_offset;\n+\t\trepo_write_dirent(dst, mode, content_offset, 0);\n+\t}\n+\treturn mode;\n+}\n+\n+void repo_add(uint32_t *path, uint32_t mode, uint32_t blob_mark)\n+{\n+\trepo_write_dirent(path, mode, blob_mark, 0);\n+}\n+\n+uint32_t repo_replace(uint32_t *path, uint32_t blob_mark)\n+{\n+\tuint32_t mode = 0;\n+\tstruct repo_dirent *src_dirent;\n+\tsrc_dirent = repo_read_dirent(active_commit, path);\n+\tif (src_dirent != NULL) {\n+\t\tmode = src_dirent->mode;\n+\t\trepo_write_dirent(path, mode, blob_mark, 0);\n+\t}\n+\treturn mode;\n+}\n+\n+void repo_modify(uint32_t *path, uint32_t mode, uint32_t blob_mark)\n+{\n+\tstruct repo_dirent *src_dirent;\n+\tsrc_dirent = repo_read_dirent(active_commit, path);\n+\tif (src_dirent != NULL && blob_mark == 0) {\n+\t\tblob_mark = src_dirent->content_offset;\n+\t}\n+\trepo_write_dirent(path, mode, blob_mark, 0);\n+}\n+\n+void repo_delete(uint32_t *path)\n+{\n+\trepo_write_dirent(path, 0, 0, 1);\n+}\n+\n+static void repo_git_add_r(uint32_t depth, uint32_t *path, struct repo_dir *dir);\n+\n+static void repo_git_add(uint32_t depth, uint32_t *path, struct repo_dirent *dirent)\n+{\n+\tif (repo_dirent_is_dir(dirent)) {\n+\t\trepo_git_add_r(depth, path, repo_dir_from_dirent(dirent));\n+\t} else {\n+\t\tfast_export_modify(depth, path, dirent->mode, dirent->content_offset);\n+\t}\n+}\n+\n+static void repo_git_add_r(uint32_t depth, uint32_t *path, struct repo_dir *dir)\n+{\n+\tuint32_t o;\n+\tstruct repo_dirent *de;\n+\tde = repo_first_dirent(dir);\n+\tfor (o = 0; o < dir->size; o++) {\n+\t\tpath[depth] = de[o].name_offset;\n+\t\trepo_git_add(depth + 1, path, &de[o]);\n+\t}\n+}\n+\n+static void repo_diff_r(uint32_t depth, uint32_t *path, struct repo_dir *dir1,\n+\t\t\tstruct repo_dir *dir2)\n+{\n+\tstruct repo_dirent *de1, *de2, *max_de1, *max_de2;\n+\tde1 = repo_first_dirent(dir1);\n+\tde2 = repo_first_dirent(dir2);\n+\tmax_de1 = &de1[dir1->size];\n+\tmax_de2 = &de2[dir2->size];\n+\n+\twhile (de1 < max_de1 && de2 < max_de2) {\n+\t\tif (de1->name_offset < de2->name_offset) {\n+\t\t\tpath[depth] = (de1++)->name_offset;\n+\t\t\tfast_export_delete(depth + 1, path);\n+\t\t} else if (de1->name_offset > de2->name_offset) {\n+\t\t\tpath[depth] = de2->name_offset;\n+\t\t\trepo_git_add(depth + 1, path, de2++);\n+\t\t} else {\n+\t\t\tpath[depth] = de1->name_offset;\n+\t\t\tif (de1->mode != de2->mode ||\n+\t\t\t\tde1->content_offset != de2->content_offset) {\n+\t\t\t\tif (repo_dirent_is_dir(de1) && repo_dirent_is_dir(de2)) {\n+\t\t\t\t\trepo_diff_r(depth + 1, path,\n+\t\t\t\t\t\t\t\trepo_dir_from_dirent(de1),\n+\t\t\t\t\t\t\t\trepo_dir_from_dirent(de2));\n+\t\t\t\t} else {\n+\t\t\t\t\tif (repo_dirent_is_dir(de1) != repo_dirent_is_dir(de2)) {\n+\t\t\t\t\t\tfast_export_delete(depth + 1, path);\n+\t\t\t\t\t}\n+\t\t\t\t\trepo_git_add(depth + 1, path, de2);\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tde1++;\n+\t\t\tde2++;\n+\t\t}\n+\t}\n+\twhile (de1 < max_de1) {\n+\t\tpath[depth] = (de1++)->name_offset;\n+\t\tfast_export_delete(depth + 1, path);\n+\t}\n+\twhile (de2 < max_de2) {\n+\t\tpath[depth] = de2->name_offset;\n+\t\trepo_git_add(depth + 1, path, de2++);\n+\t}\n+}\n+\n+static uint32_t path_stack[REPO_MAX_PATH_DEPTH];\n+\n+void repo_diff(uint32_t r1, uint32_t r2)\n+{\n+\trepo_diff_r(0,\n+\t            path_stack,\n+\t            repo_commit_root_dir(commit_pointer(r1)),\n+\t            repo_commit_root_dir(commit_pointer(r2)));\n+}\n+\n+void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,\n+                 uint32_t url, unsigned long timestamp)\n+{\n+\tfast_export_commit(revision, author, log, uuid, url, timestamp);\n+\tnum_dirs_saved = dir_pool.size;\n+\tnum_dirents_saved = dirent_pool.size;\n+\tactive_commit = commit_alloc(1);\n+\tcommit_pointer(active_commit)->root_dir_offset =\n+\t\tcommit_pointer(active_commit - 1)->root_dir_offset;\n+}\n+\n+static void mark_init(void)\n+{\n+\tuint32_t i;\n+\t_mark = 0;\n+\tfor (i = 0; i < dirent_pool.size; i++)\n+\t\tif (!repo_dirent_is_dir(dirent_pointer(i)) &&\n+\t\t\tdirent_pointer(i)->content_offset > _mark)\n+\t\t\t_mark = dirent_pointer(i)->content_offset;\n+\t_mark++;\n+}\n+\n+void repo_init() {\n+\tpool_init();\n+\tcommit_init();\n+\tdir_init();\n+\tdirent_init();\n+\tmark_init();\n+\tnum_dirs_saved = dir_pool.size;\n+\tnum_dirents_saved = dirent_pool.size;\n+\tactive_commit = commit_pool.size - 1;\n+\tif (active_commit == -1) {\n+\t\tcommit_alloc(2);\n+\t\t/* Create empty tree for commit 0. */\n+\t\tcommit_pointer(0)->root_dir_offset =\n+\t\t\tdir_with_dirents_alloc(0);\n+\t\t/* Preallocate commit 1, ready for changes. */\n+\t\tcommit_pointer(1)->root_dir_offset =\n+\t\t\tcommit_pointer(0)->root_dir_offset;\n+\t\tactive_commit = 1;\n+\t\tnum_dirs_saved = dir_pool.size;\n+\t\tnum_dirents_saved = dirent_pool.size;\n+\t}\n+}\n+\n+void repo_reset(void)\n+{\n+\tpool_reset();\n+\tcommit_reset();\n+\tdir_reset();\n+\tdirent_reset();\n+}\ndiff --git a/vcs-svn/repo_tree.h b/vcs-svn/repo_tree.h\nnew file mode 100644\nindex 0000000..c49b65a\n--- /dev/null\n+++ b/vcs-svn/repo_tree.h\n@@ -0,0 +1,27 @@\n+#ifndef REPO_TREE_H_\n+#define REPO_TREE_H_\n+\n+#include <stdint.h>\n+#include <time.h>\n+\n+#define REPO_MODE_DIR 0040000\n+#define REPO_MODE_BLB 0100644\n+#define REPO_MODE_EXE 0100755\n+#define REPO_MODE_LNK 0120000\n+\n+#define REPO_MAX_PATH_LEN 4096\n+#define REPO_MAX_PATH_DEPTH 1000\n+\n+uint32_t next_blob_mark(void);\n+uint32_t repo_copy(uint32_t revision, uint32_t *src, uint32_t *dst);\n+void repo_add(uint32_t *path, uint32_t mode, uint32_t blob_mark);\n+uint32_t repo_replace(uint32_t *path, uint32_t blob_mark);\n+void repo_modify(uint32_t *path, uint32_t mode, uint32_t blob_mark);\n+void repo_delete(uint32_t *path);\n+void repo_commit(uint32_t revision, uint32_t author, char *log, uint32_t uuid,\n+                 uint32_t url, long unsigned timestamp);\n+void repo_diff(uint32_t r1, uint32_t r2);\n+void repo_init(void);\n+void repo_reset(void);\n+\n+#endif\n-- \n1.7.1\n"},{"id":"142979","messageId":"1275658871-1473-7-git-send-email-artagnon@gmail.com","threadId":"24005","inReplyTo":"1275658871-1473-1-git-send-email-artagnon@gmail.com","subject":"[PATCH 6/6] Add SVN dump parser","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-04T13:41:11Z","receivedAt":"2010-06-04T13:41:11Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"From: David Barr <david.barr@cordelta.com>\n\nsvndump parses data that is in SVN dumpfile format produced by\n`svnadmin dump` with the help of line_buffer, and uses repo_tree and\nfast_export to emit a git fast-import stream.\n\nSigned-off-by: David Barr <david.barr@cordelta.com>\nSigned-off-by: Ramkumar Ramachandra <artagnon@gmail.com>\n---\n vcs-svn/svndump.c |  294 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n vcs-svn/svndump.h |    7 ++\n 2 files changed, 301 insertions(+), 0 deletions(-)\n create mode 100644 vcs-svn/svndump.c\n create mode 100644 vcs-svn/svndump.h\n\ndiff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c\nnew file mode 100644\nindex 0000000..9ee1246\n--- /dev/null\n+++ b/vcs-svn/svndump.c\n@@ -0,0 +1,294 @@\n+/*\n+ * Parse and rearrange a svnadmin dump.\n+ * Create the dump with:\n+ * svnadmin dump --incremental -r<startrev>:<endrev> <repository> >outfile\n+ */\n+\n+#include \"cache.h\"\n+#include \"git-compat-util.h\"\n+\n+#include \"repo_tree.h\"\n+#include \"fast_export.h\"\n+#include \"line_buffer.h\"\n+#include \"obj_pool.h\"\n+#include \"string_pool.h\"\n+\n+#define NODEACT_REPLACE 4\n+#define NODEACT_DELETE 3\n+#define NODEACT_ADD 2\n+#define NODEACT_CHANGE 1\n+#define NODEACT_UNKNOWN 0\n+\n+#define DUMP_CTX 0\n+#define REV_CTX  1\n+#define NODE_CTX 2\n+\n+#define LENGTH_UNKNOWN (~0)\n+#define DATE_RFC2822_LEN 31\n+\n+/* Create memory pool for log messages */\n+obj_pool_gen(log, char, 4096);\n+\n+static char* log_copy(uint32_t length, char *log)\n+{\n+\tchar *buffer;\n+\tlog_free(log_pool.size);\n+\tbuffer = log_pointer(log_alloc(length));\n+\tstrncpy(buffer, log, length);\n+\treturn buffer;\n+}\n+\n+static struct {\n+\tuint32_t action, propLength, textLength, srcRev, srcMode, mark, type;\n+\tuint32_t src[REPO_MAX_PATH_DEPTH], dst[REPO_MAX_PATH_DEPTH];\n+} node_ctx;\n+\n+static struct {\n+\tuint32_t revision, author;\n+\tunsigned long timestamp;\n+\tchar *log;\n+} rev_ctx;\n+\n+static struct {\n+\tuint32_t uuid, url;\n+} dump_ctx;\n+\n+static struct {\n+\tuint32_t svn_log, svn_author, svn_date, svn_executable, svn_special, uuid,\n+\t\trevision_number, node_path, node_kind, node_action,\n+\t\tnode_copyfrom_path, node_copyfrom_rev, text_content_length,\n+\t\tprop_content_length, content_length;\n+} keys;\n+\n+static void reset_node_ctx(char *fname)\n+{\n+\tnode_ctx.type = 0;\n+\tnode_ctx.action = NODEACT_UNKNOWN;\n+\tnode_ctx.propLength = LENGTH_UNKNOWN;\n+\tnode_ctx.textLength = LENGTH_UNKNOWN;\n+\tnode_ctx.src[0] = ~0;\n+\tnode_ctx.srcRev = 0;\n+\tnode_ctx.srcMode = 0;\n+\tpool_tok_seq(REPO_MAX_PATH_DEPTH, node_ctx.dst, \"/\", fname);\n+\tnode_ctx.mark = 0;\n+}\n+\n+static void reset_rev_ctx(uint32_t revision)\n+{\n+\trev_ctx.revision = revision;\n+\trev_ctx.timestamp = \"\";\n+\trev_ctx.log = NULL;\n+\trev_ctx.author = ~0;\n+}\n+\n+static void reset_dump_ctx(uint32_t url)\n+{\n+\tdump_ctx.url = url;\n+\tdump_ctx.uuid = ~0;\n+}\n+\n+static void init_keys(void)\n+{\n+\tkeys.svn_log = pool_intern(\"svn:log\");\n+\tkeys.svn_author = pool_intern(\"svn:author\");\n+\tkeys.svn_date = pool_intern(\"svn:date\");\n+\tkeys.svn_executable = pool_intern(\"svn:executable\");\n+\tkeys.svn_special = pool_intern(\"svn:special\");\n+\tkeys.uuid = pool_intern(\"UUID\");\n+\tkeys.revision_number = pool_intern(\"Revision-number\");\n+\tkeys.node_path = pool_intern(\"Node-path\");\n+\tkeys.node_kind = pool_intern(\"Node-kind\");\n+\tkeys.node_action = pool_intern(\"Node-action\");\n+\tkeys.node_copyfrom_path = pool_intern(\"Node-copyfrom-path\");\n+\tkeys.node_copyfrom_rev = pool_intern(\"Node-copyfrom-rev\");\n+\tkeys.text_content_length = pool_intern(\"Text-content-length\");\n+\tkeys.prop_content_length = pool_intern(\"Prop-content-length\");\n+\tkeys.content_length = pool_intern(\"Content-length\");\n+}\n+\n+static void read_props(void)\n+{\n+\tuint32_t len;\n+\tuint32_t key = ~0;\n+\tchar buffer[27];\n+\tchar *val = NULL;\n+\tchar *t;\n+\twhile ((t = buffer_read_line()) && strcmp(t, \"PROPS-END\")) {\n+\t\tif (!strncmp(t, \"K \", 2)) {\n+\t\t\tlen = atoi(&t[2]);\n+\t\t\tkey = pool_intern(buffer_read_string(len));\n+\t\t\tbuffer_read_line();\n+\t\t} else if (!strncmp(t, \"V \", 2)) {\n+\t\t\tlen = atoi(&t[2]);\n+\t\t\tval = buffer_read_string(len);\n+\t\t\tif (key == keys.svn_log) {\n+\t\t\t\t/* Value length excludes terminating nul. */\n+\t\t\t\trev_ctx.log = log_copy(len + 1, val);\n+\t\t\t} else if (key == keys.svn_author) {\n+\t\t\t\trev_ctx.author = pool_intern(val);\n+\t\t\t} else if (key == keys.svn_date) {\n+\t\t\t\tif (parse_date(val, buffer, sizeof(buffer)) > 0)\n+\t\t\t\t\trev_ctx.timestamp = strtoul(buffer, NULL, 0);\n+\t\t\t\telse\n+\t\t\t\t\tfprintf(stderr, \"Invalid timestamp: %s\", val);\n+\t\t\t} else if (key == keys.svn_executable) {\n+\t\t\t\tnode_ctx.type = REPO_MODE_EXE;\n+\t\t\t} else if (key == keys.svn_special) {\n+\t\t\t\tnode_ctx.type = REPO_MODE_LNK;\n+\t\t\t}\n+\t\t\tkey = ~0;\n+\t\t\tbuffer_read_line();\n+\t\t}\n+\t}\n+}\n+\n+static void handle_node(void)\n+{\n+\tif (node_ctx.propLength != LENGTH_UNKNOWN && node_ctx.propLength) {\n+\t\tread_props();\n+\t}\n+\n+\tif (node_ctx.srcRev) {\n+\t\tnode_ctx.srcMode = repo_copy(node_ctx.srcRev, node_ctx.src, node_ctx.dst);\n+\t}\n+\n+\tif (node_ctx.textLength != LENGTH_UNKNOWN &&\n+\t\tnode_ctx.type != REPO_MODE_DIR) {\n+\t\tnode_ctx.mark = next_blob_mark();\n+\t}\n+\n+\tif (node_ctx.action == NODEACT_DELETE) {\n+\t\trepo_delete(node_ctx.dst);\n+\t} else if (node_ctx.action == NODEACT_CHANGE ||\n+\t\t\t   node_ctx.action == NODEACT_REPLACE) {\n+\t\tif (node_ctx.action == NODEACT_REPLACE &&\n+\t\t\tnode_ctx.type == REPO_MODE_DIR) {\n+\t\t\trepo_replace(node_ctx.dst, node_ctx.mark);\n+\t\t} else if (node_ctx.propLength != LENGTH_UNKNOWN ) {\n+\t\t\trepo_modify(node_ctx.dst, node_ctx.type, node_ctx.mark);\n+\t\t} else if (node_ctx.textLength != LENGTH_UNKNOWN) {\n+\t\t\tnode_ctx.srcMode = repo_replace(node_ctx.dst, node_ctx.mark);\n+\t\t}\n+\t} else if (node_ctx.action == NODEACT_ADD) {\n+\t\tif (node_ctx.srcRev &&\n+\t\t\tnode_ctx.propLength == LENGTH_UNKNOWN &&\n+\t\t\tnode_ctx.textLength != LENGTH_UNKNOWN) {\n+\t\t\tnode_ctx.srcMode = repo_replace(node_ctx.dst, node_ctx.mark);\n+\t\t} else if ((node_ctx.type == REPO_MODE_DIR && !node_ctx.srcRev) ||\n+\t\t\t\t   node_ctx.textLength != LENGTH_UNKNOWN){\n+\t\t\trepo_add(node_ctx.dst, node_ctx.type, node_ctx.mark);\n+\t\t}\n+\t}\n+\n+\tif (node_ctx.propLength == LENGTH_UNKNOWN && node_ctx.srcMode) {\n+\t\tnode_ctx.type = node_ctx.srcMode;\n+\t}\n+\n+\tif (node_ctx.mark) {\n+\t\tfast_export_blob(node_ctx.type, node_ctx.mark, node_ctx.textLength);\n+\t} else if (node_ctx.textLength != LENGTH_UNKNOWN) {\n+\t\tbuffer_skip_bytes(node_ctx.textLength);\n+\t}\n+}\n+\n+static void handle_revision(void)\n+{\n+\tif (rev_ctx.revision)\n+\t\trepo_commit(rev_ctx.revision, rev_ctx.author, rev_ctx.log,\n+\t\t            dump_ctx.uuid, dump_ctx.url, rev_ctx.timestamp);\n+}\n+\n+void svndump_read(uint32_t url)\n+{\n+\tchar *val;\n+\tchar *t;\n+\tuint32_t active_ctx = DUMP_CTX;\n+\tuint32_t len;\n+\tuint32_t key;\n+\n+\treset_dump_ctx(url);\n+\twhile ((t = buffer_read_line())) {\n+\t\tval = strstr(t, \": \");\n+\t\tif (!val) continue;\n+\t\t*val++ = '\\0';\n+\t\t*val++ = '\\0';\n+\t\tkey = pool_intern(t);\n+\n+\t\tif(key == keys.uuid) {\n+\t\t\tdump_ctx.uuid = pool_intern(val);\n+\t\t} else if (key == keys.revision_number) {\n+\t\t\tif (active_ctx == NODE_CTX) handle_node();\n+\t\t\tif (active_ctx != DUMP_CTX) handle_revision();\n+\t\t\tactive_ctx = REV_CTX;\n+\t\t\treset_rev_ctx(atoi(val));\n+\t\t} else if (key == keys.node_path) {\n+\t\t\tif (active_ctx == NODE_CTX)\n+\t\t\t\thandle_node();\n+\t\t\tactive_ctx = NODE_CTX;\n+\t\t\treset_node_ctx(val);\n+\t\t} else if (key == keys.node_kind) {\n+\t\t\tif (!strcmp(val, \"dir\")) {\n+\t\t\t\tnode_ctx.type = REPO_MODE_DIR;\n+\t\t\t} else if (!strcmp(val, \"file\")) {\n+\t\t\t\tnode_ctx.type = REPO_MODE_BLB;\n+\t\t\t} else {\n+\t\t\t\tfprintf(stderr, \"Unknown node-kind: %s\\n\", val);\n+\t\t\t}\n+\t\t} else if (key == keys.node_action) {\n+\t\t\tif (!strcmp(val, \"delete\")) {\n+\t\t\t\tnode_ctx.action = NODEACT_DELETE;\n+\t\t\t} else if (!strcmp(val, \"add\")) {\n+\t\t\t\tnode_ctx.action = NODEACT_ADD;\n+\t\t\t} else if (!strcmp(val, \"change\")) {\n+\t\t\t\tnode_ctx.action = NODEACT_CHANGE;\n+\t\t\t} else if (!strcmp(val, \"replace\")) {\n+\t\t\t\tnode_ctx.action = NODEACT_REPLACE;\n+\t\t\t} else {\n+\t\t\t\tfprintf(stderr, \"Unknown node-action: %s\\n\", val);\n+\t\t\t\tnode_ctx.action = NODEACT_UNKNOWN;\n+\t\t\t}\n+\t\t} else if (key == keys.node_copyfrom_path) {\n+\t\t\tpool_tok_seq(REPO_MAX_PATH_DEPTH, node_ctx.src, \"/\", val);\n+\t\t} else if (key == keys.node_copyfrom_rev) {\n+\t\t\tnode_ctx.srcRev = atoi(val);\n+\t\t} else if (key == keys.text_content_length) {\n+\t\t\tnode_ctx.textLength = atoi(val);\n+\t\t} else if (key == keys.prop_content_length) {\n+\t\t\tnode_ctx.propLength = atoi(val);\n+\t\t} else if (key == keys.content_length) {\n+\t\t\tlen = atoi(val);\n+\t\t\tbuffer_read_line();\n+\t\t\tif (active_ctx == REV_CTX) {\n+\t\t\t\tread_props();\n+\t\t\t} else if (active_ctx == NODE_CTX) {\n+\t\t\t\thandle_node();\n+\t\t\t\tactive_ctx = REV_CTX;\n+\t\t\t} else {\n+\t\t\t\tfprintf(stderr, \"Unexpected content length header: %d\\n\", len);\n+\t\t\t\tbuffer_skip_bytes(len);\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (active_ctx == NODE_CTX) handle_node();\n+\tif (active_ctx != DUMP_CTX) handle_revision();\n+}\n+\n+static void svndump_init(void)\n+{\n+\tlog_init();\n+\trepo_init();\n+\treset_dump_ctx(~0);\n+\treset_rev_ctx(0);\n+\treset_node_ctx(NULL);\n+\tinit_keys();\n+}\n+\n+void svndump_reset(void)\n+{\n+\tlog_reset();\n+\tbuffer_reset();\n+\trepo_reset();\n+\treset_dump_ctx(~0);\n+\treset_rev_ctx(0);\n+\treset_node_ctx(NULL);\n+}\ndiff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h\nnew file mode 100644\nindex 0000000..e205f1f\n--- /dev/null\n+++ b/vcs-svn/svndump.h\n@@ -0,0 +1,7 @@\n+#ifndef SVNDUMP_H_\n+#define SVNDUMP_H_\n+\n+void svndump_read(char *url);\n+void svndump_reset(void);\n+\n+#endif\n-- \n1.7.1\n"},{"id":"142997","messageId":"20100604182928.GB20909@progeny.tock","threadId":"24005","inReplyTo":"1275658871-1473-2-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 1/6] Add memory pool library","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-06-04T18:29:28Z","receivedAt":"2010-06-04T18:29:28Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> Add a memory pool library implemented using cpp macros. The library\n> provides macros that can be used to create a type-specific memory pool\n> API.\n\nMore details:\n\n  The memory pool library is distinguished from the existing specialized\n  allocators in alloc.c by using a contiguous block for all allocations.\n  This means that on one hand, long-lived pointers have to be written as\n  offsets, since the base address changes as the pool grows, but on the\n  other hand, the entire pool can be easily written to the file system.\n  This allows the memory pool to persist between runs of an application.\n  \n  For svn-fe, such a facility is useful because each svn revision can\n  copy trees and files from any previous revision.  Therefore the\n  relevant information for all revisions has to persist somehow to\n  support incremental runs.\n  \n  The current implementation is backed by the file system using mmap().\n\nThanks for the explanations, David.\n\n> +#define obj_pool_gen(pre, obj_t, initial_capacity) \\\n> +static struct { \\\n> +\tuint32_t size; \\\n> +\tuint32_t capacity; \\\n> +\tobj_t *base; \\\n> +        FILE *file; \\\n\nWhitespace damaged.\n\n> +} pre##_pool = { 0, 0, NULL, NULL}; \\\n> +static void pre##_init(void) \\\n> +{ \\\n> +\tstruct stat st; \\\n> +\tsize_t ps = sysconf (_SC_PAGESIZE); \\\n\nProbably an ignorant question, but why?  I do not think the win32 mmap\nemulation in git currently supports sysconf().\n\n> +\t/* Touch binary file before opening read/write */ \\\n> +\tpre##_pool.file = fopen(#pre \".bin\", \"a\"); \\\n> +\tfclose(pre##_pool.file); \\\n> +\t/* Open, check size, compute capacity */ \\\n> +\tpre##_pool.file = fopen(#pre \".bin\", \"r+\"); \\\n\nMaybe\n\n\t..._pool.file = fopen(..., \"a+\");\n\trewind(..._pool.file);\n\nwould be simpler.  But given that there is no buffered I/O going on\nfor this file, why not just\n\n\t..._pool.file = open(..., O_RDWR);\n\n?\n\n> +\t/* Truncate to calculated capacity and map to VM */ \\\n> +\tftruncate(fileno(pre##_pool.file), pre##_pool.capacity * sizeof(obj_t)); \\\n\nReasonable.\n\n> +\tpre##_pool.base = mmap(0, pre##_pool.capacity * sizeof(obj_t), \\\n> +\t\t\t\tPROT_READ | PROT_WRITE, MAP_SHARED, \\\n> +\t\t\t\tfileno(pre##_pool.file), 0); \\\n\nThis is the first use of MAP_SHARED in git.  I wouldn’t be surprised if\nthe win32 mmap emulation does not support it.\n\n> +} \\\n> +static uint32_t pre##_alloc(uint32_t count) \\\n> +{ \\\n> +\tuint32_t offset; \\\n> +\tif (pre##_pool.size + count > pre##_pool.capacity) { \\\n> +\t\tif (NULL == pre##_pool.base) \\\n> +\t\t\tpre##_init(); \\\n> +\t\tfsync(fileno(pre##_pool.file)); \\\n\nNecessary?  (Maybe yes: we are about to truncate the file, so the\nfsync() may be intended to force the data to be committed before\nthe metadata.)  Portable?  (I suspect the fsync() should go after\nthe munmap().)\n\n[...]\n> +static void pre##_reset(void) \\\n> +{ \\\n> +\tif (pre##_pool.base) { \\\n> +\t\tfsync(fileno(pre##_pool.file)); \\\n\nSame question.\n\n> +\t\tmunmap(pre##_pool.base, \\\n> +\t\t\tpre##_pool.capacity * sizeof(obj_t)); \\\n> +\t\tftruncate(fileno(pre##_pool.file), \\\n> +\t\t\t\tpre##_pool.size * sizeof(obj_t)); \\\n> +\t\tfclose(pre##_pool.file); \\\n\nBecause of these portability concerns, I’d rather use the old version\nuntil the incremental parser is ready.\n\nHope that helps,\nJonathan\n"},{"id":"142998","messageId":"20100604183525.GA21295@progeny.tock","threadId":"24005","inReplyTo":"1275658871-1473-5-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 4/6] Add stream helper library","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-06-04T18:35:25Z","receivedAt":"2010-06-04T18:35:25Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> +static uint32_t line_buffer_len = 0;\n> +static uint32_t line_len = 0;\n\nI still don’t understand why this needs to read ahead.\n\nI can take a stab at cleaning it up...\n\n> --- /dev/null\n> +++ b/vcs-svn/line_buffer.h\n> @@ -0,0 +1,14 @@\n> +#ifndef LINE_BUFFER_H_\n> +#define LINE_BUFFER_H_\n> +\n> +#include <stdint.h>\n\nShould use git-compat-util.h instead.\n\nJonathan\n"},{"id":"142999","messageId":"20100604190222.GB21295@progeny.tock","threadId":"24005","inReplyTo":"1275658871-1473-6-git-send-email-artagnon@gmail.com","subject":"Re: [PATCH 5/6] Add infrastructure to write revisions in fast-export format","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-06-04T19:02:22Z","receivedAt":"2010-06-04T19:02:22Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Ram,\n\nThanks for the style cleanup.\n\nRamkumar Ramachandra wrote:\n\n> --- /dev/null\n> +++ b/vcs-svn/repo_tree.c\n[...]\n> +static struct repo_dir *repo_clone_dir(struct repo_dir *orig_dir, uint32_t padding)\n> +{\n> +\tuint32_t orig_o, new_o, dirent_o;\n> +\torig_o = dir_offset(orig_dir);\n> +\tif (orig_o < num_dirs_saved) {\n> +\t\tnew_o = dir_with_dirents_alloc(orig_dir->size + padding);\n> +\t\torig_dir = dir_pointer(orig_o);\n> +\t\tdirent_o = dir_pointer(new_o)->first_offset;\n> +\t} else {\n> +\t\tif (padding == 0)\n> +\t\t\treturn orig_dir;\n> +\t\tnew_o = orig_o;\n> +\t\tdirent_o = dirent_alloc(orig_dir->size + padding);\n> +\t}\n> +\tmemcpy(dirent_pointer(dirent_o), repo_first_dirent(orig_dir),\n> +\t\t   orig_dir->size * sizeof(struct repo_dirent));\n> +\tdir_pointer(new_o)->size = orig_dir->size + padding;\n> +\tdir_pointer(new_o)->first_offset = dirent_o;\n> +\treturn dir_pointer(new_o);\n> +}\n\nThis function expands a list of directory entries, copying it if\nalready committed as part of a previous revision.\n\nThe big downside is that as David mentioned it does not scale well with\nthe size of the directory and number of expansions.  But a patch is in\nthe pipeline to fix that.  I do not think it should hold the series up.\n\n> +\n> +static struct repo_dirent *repo_read_dirent(uint32_t revision, uint32_t *path)\n> +{\n> +\tuint32_t name = 0;\n> +\tstruct repo_dir *dir = NULL;\n> +\tstruct repo_dirent *dirent = NULL;\n> +\tdir = repo_commit_root_dir(commit_pointer(revision));\n> +\twhile (~(name = *path++)) {\n> +\t\tdirent = repo_dirent_by_name(dir, name);\n> +\t\tif (dirent == NULL) {\n> +\t\t\treturn NULL;\n> +\t\t} else if (repo_dirent_is_dir(dirent)) {\n> +\t\t\tdir = repo_dir_from_dirent(dirent);\n> +\t\t} else {\n> +\t\t\tbreak;\n> +\t\t}\n\nStyle: would probably be clearer to write:\n\n\twhile (~(name = *path++)) {\n\t\tdirent = repo_dirent_by_name(dir, name);\n\t\tif (!dirent || !repo_dirent_is_dir(dirent))\n\t\t\tbreak;\n\t\tdir = repo_dir_from_dirent(dirent);\n\t}\n\ni.e., fewer unnecessary braces, and dealing with the exceptional cases\nseparately from the normal case.\n\nAs before, I wonder about the error cases.  Might it make sense to\nreport the error if someone tries to copy a nonexistent directory\nfrom a previous revision?\n\n> +static void\n> +repo_write_dirent(uint32_t *path, uint32_t mode, uint32_t content_offset,\n> +                  uint32_t del)\n\nFunction is too long for my taste (I realize this is a matter of\ntaste).  The innermost blocks would make sense as functions in their\nown right.\n\n[...]\n> +static void repo_diff_r(uint32_t depth, uint32_t *path, struct repo_dir *dir1,\n> +\t\t\tstruct repo_dir *dir2)\n[...]\n> +\twhile (de1 < max_de1 && de2 < max_de2) {\n> +\t\tif (de1->name_offset < de2->name_offset) {\n> +\t\t\tpath[depth] = (de1++)->name_offset;\n> +\t\t\tfast_export_delete(depth + 1, path);\n> +\t\t} else if (de1->name_offset > de2->name_offset) {\n> +\t\t\tpath[depth] = de2->name_offset;\n> +\t\t\trepo_git_add(depth + 1, path, de2++);\n> +\t\t} else {\n> +\t\t\tpath[depth] = de1->name_offset;\n> +\t\t\tif (de1->mode != de2->mode ||\n> +\t\t\t\tde1->content_offset != de2->content_offset) {\n> +\t\t\t\tif (repo_dirent_is_dir(de1) && repo_dirent_is_dir(de2)) {\n\nMy 80-column terminal is suffering.  Why not use the common\n\n\twhile (...) {\n\t\tif (exceptional case) {\n\t\t\t....\n\t\t\tcontinue;\n\t\t} else if (exceptional case) {\n\t\t\t...\n\t\t\tcontinue;\n\t\t}\n\t\tpath[depth] = de1->name_offset;\n\t\tif (easy case) {\n\t\t\t...\n\t\t\tcontinue;\n\t\t}\n\t\tusual case\n\t}\n\npattern?\n\n> +#define REPO_MAX_PATH_LEN 4096\n> +#define REPO_MAX_PATH_DEPTH 1000\n\nThese limits are not checked; is the caller supposed to check them\nitself?  Does svn obey them?\n\nJonathan\n"},{"id":"143152","messageId":"AANLkTilqFNqGj8rq3fHZld6ddkHLp_usQSQKgQqeAt0X@mail.gmail.com","threadId":"24005","inReplyTo":"20100604190222.GB21295@progeny.tock","subject":"Re: [PATCH 5/6] Add infrastructure to write revisions in fast-export format","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-07T12:35:30Z","receivedAt":"2010-06-07T12:35:30Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Jonathan,\n\nJonathan Nieder wrote:\n> The big downside is that as David mentioned it does not scale well with\n> the size of the directory and number of expansions.  But a patch is in\n> the pipeline to fix that.  I do not think it should hold the series up.\n\nRight. The `dirents` branch is now merged.\n\n> Style: would probably be clearer to write:\n>\n>        while (~(name = *path++)) {\n>                dirent = repo_dirent_by_name(dir, name);\n>                if (!dirent || !repo_dirent_is_dir(dirent))\n>                        break;\n>                dir = repo_dir_from_dirent(dirent);\n>        }\n>\n> i.e., fewer unnecessary braces, and dealing with the exceptional cases\n> separately from the normal case.\n\nThis was (probably unintentionally) re-factored by David when merging\nin his `dirents` branch.\n\n> As before, I wonder about the error cases.  Might it make sense to\n> report the error if someone tries to copy a nonexistent directory\n> from a previous revision?\n\n\n> Function is too long for my taste (I realize this is a matter of\n> taste).  The innermost blocks would make sense as functions in their\n> own right.\n\nThis came up in your previous review- I tried to split it up into more\nfunctions, but due to the number of local variables, it turned out to\nbe a mess and I had to revert. Would you like to try re-factoring it?\n\n> My 80-column terminal is suffering.  Why not use the common\n> pattern?\n\nRe-factored (again, probably unintentionally) by David during the merge.\n\n> These limits are not checked; is the caller supposed to check them\n> itself?  Does svn obey them?\n\nI asked David too, and as far as we know, these limits are pretty\narbitrary. They're no enforced by SVN or any specific filesystem. We\ncan discuss about having less arbitrary limits and checking it in\nsvndump.c (while parsing the dump).\n\n-- Ram\n"},{"id":"143155","messageId":"AANLkTimzuJXjX4OJUkmQtLwjGDgHBaZUHSGA0xUUi6kY@mail.gmail.com","threadId":"24005","inReplyTo":"20100604182928.GB20909@progeny.tock","subject":"Re: [PATCH 1/6] Add memory pool library","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-07T13:28:13Z","receivedAt":"2010-06-07T13:28:13Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Jonathan,\n\nJonathan Nieder wrote:\n>  The memory pool library is distinguished from the existing specialized\n>  allocators in alloc.c by using a contiguous block for all allocations.\n>  This means that on one hand, long-lived pointers have to be written as\n>  offsets, since the base address changes as the pool grows, but on the\n>  other hand, the entire pool can be easily written to the file system.\n>  This allows the memory pool to persist between runs of an application.\n>\n>  For svn-fe, such a facility is useful because each svn revision can\n>  copy trees and files from any previous revision.  Therefore the\n>  relevant information for all revisions has to persist somehow to\n>  support incremental runs.\n>\n>  The current implementation is backed by the file system using mmap().\n>\n> Thanks for the explanations, David.\n\nThanks. Added to commit message: should be fine the next time I post the series.\n\n> Whitespace damaged.\n\nFixed in b180ad7 (pushed few seconds ago).\n\n> Probably an ignorant question, but why?  I do not think the win32 mmap\n> emulation in git currently supports sysconf().\n\nI'm not sure why the pool capacity should be dependent on the page size.\n\n>        ..._pool.file = open(..., O_RDWR);\n\nFixed in b180ad7 (pushed few seconds ago).\n\n> This is the first use of MAP_SHARED in git.  I wouldn’t be surprised if\n> the win32 mmap emulation does not support it.\n\nYou're right; I just checked. compat/mmap.c:7 clearly specifies that\nonly MAP_PRIVATE is supported.\n\tif (start != NULL || !(flags & MAP_PRIVATE))\n\t\tdie(\"Invalid usage of mmap when built with NO_MMAP\");\n\nWhat can we do about this? Should we attempt to implement MAP_SHARED?\n\n> Necessary?  (Maybe yes: we are about to truncate the file, so the\n> fsync() may be intended to force the data to be committed before\n> the metadata.)  Portable?  (I suspect the fsync() should go after\n> the munmap().)\n\nYou want to flush changes *after* unmapping the pages of memory from\nthe file? I can't find any notes on portability relating to this- can\nyou clarify?\n\n> Because of these portability concerns, I’d rather use the old version\n> until the incremental parser is ready.\n\nRolling back that many changes can be more painful than actually\nfixing these few concerns (especially large number of change after\n`dirents` was merged in). It means more delay in getting the exporter\nmerged.\n\n-- Ram\n"},{"id":"143156","messageId":"B78F4B61-E006-47CD-9186-EF81BE5CDAC7@cordelta.com","threadId":"24005","inReplyTo":"AANLkTilqFNqGj8rq3fHZld6ddkHLp_usQSQKgQqeAt0X@mail.gmail.com","subject":"Re: [PATCH 5/6] Add infrastructure to write revisions in fast-export format","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-06-07T13:36:21Z","receivedAt":"2010-06-07T13:36:21Z","isPatch":true,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi Ram,\n\n>> Style: would probably be clearer to write:\n>> \n>>        while (~(name = *path++)) {\n>>                dirent = repo_dirent_by_name(dir, name);\n>>                if (!dirent || !repo_dirent_is_dir(dirent))\n>>                        break;\n>>                dir = repo_dir_from_dirent(dirent);\n>>        }\n>> \n>> i.e., fewer unnecessary braces, and dealing with the exceptional cases\n>> separately from the normal case.\n> \n> This was (probably unintentionally) re-factored by David when merging\n> in his `dirents` branch.\n\nOh no, it was intentional.\n\n>> My 80-column terminal is suffering.  Why not use the common\n>> pattern?\n> \n> Re-factored (again, probably unintentionally) by David during the merge.\n\nAs above, totally intentional.\n\n@Jonathan: Thanks for the review, I took it into account when refactoring dirents.\n\n>> These limits are not checked; is the caller supposed to check them\n>> itself?  Does svn obey them?\n> \n> I asked David too, and as far as we know, these limits are pretty\n> arbitrary. They're no enforced by SVN or any specific filesystem. We\n> can discuss about having less arbitrary limits and checking it in\n> svndump.c (while parsing the dump).\n\nI am indeed in favour of less arbitrary limits.\nThe current magic numbers were chosen as common practical bounds.\n"},{"id":"143160","messageId":"AANLkTimd_l0uKHTqgS211fv30hxd-k0pR55YoY4xCua1@mail.gmail.com","threadId":"24005","inReplyTo":"AANLkTimzuJXjX4OJUkmQtLwjGDgHBaZUHSGA0xUUi6kY@mail.gmail.com","subject":"Re: [PATCH 1/6] Add memory pool library","fromName":"Erik Faye-Lund","fromEmail":"kusmabite@googlemail.com","sentAt":"2010-06-07T14:00:18Z","receivedAt":"2010-06-07T14:00:18Z","isPatch":true,"sender":{"key":"kusmabite@gmail.com","avatar":"https://avatars.githubusercontent.com/u/47073?v=4"},"body":"On Mon, Jun 7, 2010 at 3:28 PM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:\n>> Probably an ignorant question, but why?  I do not think the win32 mmap\n>> emulation in git currently supports sysconf().\n>\n> I'm not sure why the pool capacity should be dependent on the page size.\n>\n\nMy guess would be that it's a performance-thing.\n\nAdding a win32-implementation of sysconf for this purpose isn't\ntricky, but perhaps it's a little bit overkill. Can't we just use some\nlarge power of two number? Hopefully, we could chose a size that is a\nmultiple of the page-size on all systems...\n\nAnyway, this is roughly what a Windows-implementation of sysconf would\nlook like (not tested - only checked that it compiles):\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex f90a114..526f0f9 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -1473,3 +1473,13 @@ struct dirent *mingw_readdir(DIR *dir)\n \treturn (struct dirent*)&dir->dd_dir;\n }\n #endif // !NO_MINGW_REPLACE_READDIR\n+\n+long mingw_sysconf(int name)\n+{\n+\tSYSTEM_INFO si;\n+\tif (name != _SC_PAGESIZE)\n+\t\treturn errno = EINVAL,\n+\t\t    error(\"sysconf emulation only supports _SC_PAGESIZE\");\n+\tGetSystemInfo(&si);\n+\treturn si.dwPageSize;\n+}\ndiff --git a/compat/mingw.h b/compat/mingw.h\nindex 6c6cbda..db28b0d 100644\n--- a/compat/mingw.h\n+++ b/compat/mingw.h\n@@ -322,3 +322,7 @@ struct dirent *mingw_readdir(DIR *dir);\n  * Used by Pthread API implementation for Windows\n  */\n extern int err_win_to_posix(DWORD winerr);\n+\n+#define _SC_PAGESIZE 0\n+long mingw_sysconf(int name);\n+#define sysconf mingw_sysconf\n\n-- \nErik \"kusma\" Faye-Lund\n"},{"id":"143162","messageId":"AANLkTilorbTSy0Va7wX2BauiiVlAEUdzDi87DNF-SGNP@mail.gmail.com","threadId":"24005","inReplyTo":"AANLkTimd_l0uKHTqgS211fv30hxd-k0pR55YoY4xCua1@mail.gmail.com","subject":"Re: [PATCH 1/6] Add memory pool library","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-07T14:35:48Z","receivedAt":"2010-06-07T14:35:48Z","isPatch":true,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Erik,\n\nErik Faye-Lund wrote:\n> On Mon, Jun 7, 2010 at 3:28 PM, Ramkumar Ramachandra <artagnon@gmail.com> wrote:\n> Adding a win32-implementation of sysconf for this purpose isn't\n> tricky, but perhaps it's a little bit overkill. Can't we just use some\n> large power of two number? Hopefully, we could chose a size that is a\n> multiple of the page-size on all systems...\n\nUntil an alternative is ready, I'll just use a large power of two and\nroll out another revision.\n\n> Anyway, this is roughly what a Windows-implementation of sysconf would\n> look like (not tested - only checked that it compiles):\n\nThanks for implementing this. Unfortunately, I don't have a Windows\nbox to test this- why don't you send a patch to the list and sign it\noff? While I agree that just implementing sysconf for _SC_PAGESIZE is\nan overkill, others can test and extend the code.\n\n-- Ram\n"}]}