{"thread":{"id":"37387","subject":"[PATCH] teach fast-export an --anonymize option","startedAt":"2014-08-21T07:01:30Z","lastAt":"2014-08-31T15:53:51Z","messageCount":21,"participants":["Jeff King","Junio C Hamano","Duy Nguyen","Philip Oakley","Ramsay Jones","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"248042","messageId":"20140821070130.GA15930@peff.net","threadId":"37387","inReplyTo":null,"subject":"[PATCH] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-21T07:01:30Z","receivedAt":"2014-08-21T07:01:30Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Sometimes users want to report a bug they experience on\ntheir repository, but they are not at liberty to share the\ncontents of the repository. It would be useful if they could\nproduce a repository that has a similar shape to its history\nand tree, but without leaking any information. This\n\"anonymized\" repository could then be shared with developers\n(assuming it still replicates the original problem).\n\nThis patch implements an \"--anonymize\" option to\nfast-export, which generates a stream that can recreate such\na repository. Producing a single stream makes it easy for\nthe caller to verify that they are not leaking any useful\ninformation. You can get an overview of what will be shared\nby running a command like:\n\n git fast-export --anonymize --all |\n perl -pe 's/\\d+/X/g' |\n sort -u |\n less\n\nwhich will show every unique line we generate, modulo any\nnumbers (each anonymized token is assigned a number, like\n\"User 0\", and we replace it consistently in the output).\n\nIn addition to anonymizing, this produces test cases that\nare relatively small (compared to the original repository)\nand fast to generate (compared to using filter-branch, or\nmodifying the output of fast-export yourself). Here are\nnumbers for git.git:\n\n $ time git fast-export --anonymize --all \\\n --tag-of-filtered-object=drop >output\n real 0m2.883s\n user 0m2.828s\n sys 0m0.052s\n\n $ gzip output\n $ ls -lh output.gz | awk '{print $5}'\n 2.9M\n\nSigned-off-by: Jeff King \n---\nI haven't used this for anything real yet. It was a fun exercise, and I\ndo think it should work in practice. I'd be curious to hear a success\nreport of somebody actually debugging something with this.\n\nIn theory we could anonymize in a reversible way (e.g., by encrypting\neach token with a key, and then not sharing the key), but it's a lot\nmore complicated and I don't think it buys us much. The one thing I'd\nreally like is to be able to test packing on an anonymized repository,\nbut two objects which delta well together will not have their encrypted\ncontents delta (unless you use something weak like ECB mode, in which\ncase the contents are not really as anonymized as you would hope).\n\nI think most interesting cases involve things like commit traversal, and\nthat should still work here, even with made-up contents. Some weird\ncases involving trees would not work if they depend on the filenames\n(e.g., things that impact sort order). We could allow finer-grained\ncontrol, like \"--anonymize=commits,blobs\" if somebody was OK sharing\ntheir filenames. I did not go that far here, but it should be pretty\neasy to build on top.\n\n Documentation/git-fast-export.txt | 6 +\n builtin/fast-export.c | 280 ++++++++++++++++++++++++++++++++++++--\n t/t9351-fast-export-anonymize.sh | 117 ++++++++++++++++\n 3 files changed, 392 insertions(+), 11 deletions(-)\n create mode 100755 t/t9351-fast-export-anonymize.sh\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 221506b..0ec7cad 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -105,6 +105,12 @@ marks the same across runs.\n \tin the commit (as opposed to just listing the files which are\n \tdifferent from the commit's first parent).\n \n+--anonymize::\n+\tReplace all paths, blob contents, commit and tag messages,\n+\tnames, and email addresses in the output with anonymized data,\n+\twhile still retaining the shape of history and of the stored\n+\ttree.\n+\n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n \tbe specified.\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 92b4624..acd2838 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"quote.h\"\n #include \"remote.h\"\n+#include \"blob.h\"\n \n static const char *fast_export_usage[] = {\n \tN_(\"git fast-export [rev-list-opts]\"),\n@@ -34,6 +35,7 @@ static int full_tree;\n static struct string_list extra_refs = STRING_LIST_INIT_NODUP;\n static struct refspec *refspecs;\n static int refspecs_nr;\n+static int anonymize;\n \n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -81,6 +83,76 @@ static int has_unshown_parent(struct commit *commit)\n \treturn 0;\n }\n \n+struct anonymized_entry {\n+\tstruct hashmap_entry hash;\n+\tconst char *orig;\n+\tsize_t orig_len;\n+\tconst char *anon;\n+\tsize_t anon_len;\n+};\n+\n+static int anonymized_entry_cmp(const void *va, const void *vb,\n+\t\t\t\tconst void *data)\n+{\n+\tconst struct anonymized_entry *a = va, *b = vb;\n+\treturn a->orig_len != b->orig_len ||\n+\t\tmemcmp(a->orig, b->orig, a->orig_len);\n+}\n+\n+/*\n+ * Basically keep a cache of X->Y so that we can repeatedly replace\n+ * the same anonymized string with another. The actual generation\n+ * is farmed out to the generate function.\n+ */\n+static const char *anonymize_mem(struct hashmap *map,\n+\t\t\t\t char *(*generate)(const char *, size_t *),\n+\t\t\t\t const char *orig, size_t *len)\n+{\n+\tstruct anonymized_entry key, *ret;\n+\n+\tif (!map->cmpfn)\n+\t\thashmap_init(map, anonymized_entry_cmp, 0);\n+\n+\thashmap_entry_init(&key, memhash(orig, *len));\n+\tkey.orig = orig;\n+\tkey.orig_len = *len;\n+\tret = hashmap_get(map, &key, NULL);\n+\n+\tif (!ret) {\n+\t\tret = xmalloc(sizeof(*ret));\n+\t\thashmap_entry_init(&ret->hash, key.hash.hash);\n+\t\tret->orig = xstrdup(orig);\n+\t\tret->orig_len = *len;\n+\t\tret->anon = generate(orig, len);\n+\t\tret->anon_len = *len;\n+\t\thashmap_put(map, ret);\n+\t}\n+\n+\t*len = ret->anon_len;\n+\treturn ret->anon;\n+}\n+\n+/*\n+ * We anonymize each component of a path individually,\n+ * so that paths a/b and a/c will share a common root.\n+ * The paths are cached via anonymize_mem so that repeated\n+ * lookups for \"a\" will yield the same value.\n+ */\n+static void anonymize_path(struct strbuf *out, const char *path,\n+\t\t\t struct hashmap *map,\n+\t\t\t char *(*generate)(const char *, size_t *))\n+{\n+\twhile (*path) {\n+\t\tconst char *end_of_component = strchrnul(path, '/');\n+\t\tsize_t len = end_of_component - path;\n+\t\tconst char *c = anonymize_mem(map, generate, path, &len);\n+\t\tstrbuf_add(out, c, len);\n+\t\tpath = end_of_component;\n+\t\tif (*path)\n+\t\t\tstrbuf_addch(out, *path++);\n+\t}\n+}\n+\n /* Since intptr_t is C99, we do not use it here */\n static inline uint32_t *mark_to_ptr(uint32_t mark)\n {\n@@ -119,6 +191,26 @@ static void show_progress(void)\n \t\tprintf(\"progress %d objects\\n\", counter);\n }\n \n+/*\n+ * Ideally we would want some transformation of the blob data here\n+ * that is unreversible, but would still be the same size and have\n+ * the same data relationship to other blobs (so that we get the same\n+ * delta and packing behavior as the original). But the first and last\n+ * requirements there are probably mutually exclusive, so let's take\n+ * the easy way out for now, and just generate arbitrary content.\n+ *\n+ * There's no need to cache this result with anonymize_mem, since\n+ * we already handle blob content caching with marks.\n+ */\n+static char *anonymize_blob(unsigned long *size)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"anonymous blob %d\", counter++);\n+\t*size = out.len;\n+\treturn strbuf_detach(&out, NULL);\n+}\n+\n static void export_blob(const unsigned char *sha1)\n {\n \tunsigned long size;\n@@ -137,12 +229,19 @@ static void export_blob(const unsigned char *sha1)\n \tif (object && object->flags & SHOWN)\n \t\treturn;\n \n-\tbuf = read_sha1_file(sha1, &type, &size);\n-\tif (!buf)\n-\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n-\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n-\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n-\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\tif (anonymize) {\n+\t\tbuf = anonymize_blob(&size);\n+\t\tobject = (struct object *)lookup_blob(sha1);\n+\t\teaten = 0;\n+\t} else {\n+\t\tbuf = read_sha1_file(sha1, &type, &size);\n+\t\tif (!buf)\n+\t\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n+\t\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n+\t\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n+\t\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\t}\n+\n \tif (!object)\n \t\tdie(\"Could not read blob %s\", sha1_to_hex(sha1));\n \n@@ -190,7 +289,7 @@ static int depth_first(const void *a_, const void *b_)\n \treturn (a->status == 'R') - (b->status == 'R');\n }\n \n-static void print_path(const char *path)\n+static void print_path_1(const char *path)\n {\n \tint need_quote = quote_c_style(path, NULL, NULL, 0);\n \tif (need_quote)\n@@ -201,6 +300,28 @@ static void print_path(const char *path)\n \t\tprintf(\"%s\", path);\n }\n \n+static char *anonymize_path_component(const char *path, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"path%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static void print_path(const char *path)\n+{\n+\tif (!anonymize)\n+\t\tprint_path_1(path);\n+\telse {\n+\t\tstatic struct hashmap paths;\n+\t\tstatic struct strbuf anon = STRBUF_INIT;\n+\n+\t\tanonymize_path(&anon, path, &paths, anonymize_path_component);\n+\t\tprint_path_1(anon.buf);\n+\t\tstrbuf_reset(&anon);\n+\t}\n+}\n+\n static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t struct diff_options *options, void *data)\n {\n@@ -241,7 +362,9 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\tcase DIFF_STATUS_ADDED:\n \t\t\t/*\n \t\t\t * Links refer to objects in another repositories;\n-\t\t\t * output the SHA-1 verbatim.\n+\t\t\t * output the SHA-1 verbatim. We don't anonymize\n+\t\t\t * these at all; they are not reversible and\n+\t\t\t * probably not a big deal to share.\n \t\t\t */\n \t\t\tif (no_data || S_ISGITLINK(spec->mode))\n \t\t\t\tprintf(\"M %06o %s \", spec->mode,\n@@ -279,6 +402,109 @@ static const char *find_encoding(const char *begin, const char *end)\n \treturn bol;\n }\n \n+static char *anonymize_ref_component(const char *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"ref%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static const char *anonymize_refname(const char *refname)\n+{\n+\t/*\n+\t * If any of these prefixes is found, we will leave it intact\n+\t * so that tags remain tags and so forth. We also leave \"master\"\n+\t * as a special case, since it does not reveal anything interesting.\n+\t */\n+\tstatic const char *prefixes[] = {\n+\t\t\"refs/heads/master\",\n+\t\t\"refs/heads/\",\n+\t\t\"refs/tags/\",\n+\t\t\"refs/remotes/\",\n+\t\t\"refs/\"\n+\t};\n+\tstatic struct hashmap refs;\n+\tstatic struct strbuf anon = STRBUF_INIT;\n+\tint i;\n+\n+\tstrbuf_reset(&anon);\n+\tfor (i = 0; i < ARRAY_SIZE(prefixes); i++) {\n+\t\tif (skip_prefix(refname, prefixes[i], &refname)) {\n+\t\t\tstrbuf_addstr(&anon, prefixes[i]);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tanonymize_path(&anon, refname, &refs, anonymize_ref_component);\n+\treturn anon.buf;\n+}\n+\n+/*\n+ * We do not even bother to cache commit messages, as they are unlikely\n+ * to be repeated verbatim, and it is not that interesting when they are.\n+ */\n+static char *anonymize_commit_message(const char *old)\n+{\n+\tstatic int counter;\n+\treturn xstrfmt(\"subject %d\\n\\nbody\\n\", counter++);\n+}\n+\n+static struct hashmap idents;\n+static char *anonymize_ident(const char *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"User %d \", counter, counter);\n+\tcounter++;\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+/*\n+ * Our strategy here is to anonymize the names and email addresses,\n+ * but keep timestamps intact, as they influence things like traversal\n+ * order (and by themselves should not be too revealing).\n+ */\n+static void anonymize_ident_line(const char **beg, const char **end)\n+{\n+\tstatic struct strbuf buffers[] = { STRBUF_INIT, STRBUF_INIT };\n+\tstatic unsigned which_buffer;\n+\n+\tstruct strbuf *out;\n+\tstruct ident_split split;\n+\tconst char *end_of_header;\n+\n+\tout = &buffers[which_buffer++];\n+\twhich_buffer %= ARRAY_SIZE(buffers);\n+\tstrbuf_reset(out);\n+\n+\t/* skip \"committer\", \"author\", \"tagger\", etc */\n+\tend_of_header = strchr(*beg, ' ');\n+\tif (!end_of_header)\n+\t\tdie(\"BUG: malformed line fed to anonymize_ident_line: %.*s\",\n+\t\t (int)(*end - *beg), *beg);\n+\tend_of_header++;\n+\tstrbuf_add(out, *beg, end_of_header - *beg);\n+\n+\tif (!split_ident_line(&split, end_of_header, *end - end_of_header) &&\n+\t split.date_begin) {\n+\t\tconst char *ident;\n+\t\tsize_t len;\n+\n+\t\tlen = split.mail_end - split.name_begin;\n+\t\tident = anonymize_mem(&idents, anonymize_ident,\n+\t\t\t\t split.name_begin, &len);\n+\t\tstrbuf_add(out, ident, len);\n+\t\tstrbuf_addch(out, ' ');\n+\t\tstrbuf_add(out, split.date_begin, split.tz_end - split.date_begin);\n+\t} else {\n+\t\tstrbuf_addstr(out, \"Malformed Ident 0 -0000\");\n+\t}\n+\n+\t*beg = out->buf;\n+\t*end = out->buf + out->len;\n+}\n+\n static void handle_commit(struct commit *commit, struct rev_info *rev)\n {\n \tint saved_output_format = rev->diffopt.output_format;\n@@ -287,6 +513,7 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tconst char *encoding, *message;\n \tchar *reencoded = NULL;\n \tstruct commit_list *p;\n+\tconst char *refname;\n \tint i;\n \n \trev->diffopt.output_format = DIFF_FORMAT_CALLBACK;\n@@ -326,13 +553,22 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \t\tif (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))\n \t\t\texport_blob(diff_queued_diff.queue[i]->two->sha1);\n \n+\trefname = commit->util;\n+\tif (anonymize) {\n+\t\trefname = anonymize_refname(refname);\n+\t\tanonymize_ident_line(&committer, &committer_end);\n+\t\tanonymize_ident_line(&author, &author_end);\n+\t}\n+\n \tmark_next_object(&commit->object);\n-\tif (!is_encoding_utf8(encoding))\n+\tif (anonymize)\n+\t\treencoded = anonymize_commit_message(message);\n+\telse if (!is_encoding_utf8(encoding))\n \t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n \tif (!commit->parents)\n-\t\tprintf(\"reset %s\\n\", (const char*)commit->util);\n+\t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n%.*s\\n%.*s\\ndata %u\\n%s\",\n-\t (const char *)commit->util, last_idnum,\n+\t refname, last_idnum,\n \t (int)(author_end - author), author,\n \t (int)(committer_end - committer), committer,\n \t (unsigned)(reencoded\n@@ -363,6 +599,14 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tshow_progress();\n }\n \n+static char *anonymize_tag(const char *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"tag message %d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n static void handle_tail(struct object_array *commits, struct rev_info *revs)\n {\n \tstruct commit *commit;\n@@ -419,6 +663,17 @@ static void handle_tag(const char *name, struct tag *tag)\n \t} else {\n \t\ttagger++;\n \t\ttagger_end = strchrnul(tagger, '\\n');\n+\t\tif (anonymize)\n+\t\t\tanonymize_ident_line(&tagger, &tagger_end);\n+\t}\n+\n+\tif (anonymize) {\n+\t\tname = anonymize_refname(name);\n+\t\tif (message) {\n+\t\t\tstatic struct hashmap tags;\n+\t\t\tmessage = anonymize_mem(&tags, anonymize_tag,\n+\t\t\t\t\t\tmessage, &message_size);\n+\t\t}\n \t}\n \n \t/* handle signed tags */\n@@ -584,6 +839,8 @@ static void handle_tags_and_duplicates(void)\n \t\t\thandle_tag(name, (struct tag *)object);\n \t\t\tbreak;\n \t\tcase OBJ_COMMIT:\n+\t\t\tif (anonymize)\n+\t\t\t\tname = anonymize_refname(name);\n \t\t\t/* create refs pointing to already seen commits */\n \t\t\tcommit = (struct commit *)object;\n \t\t\tprintf(\"reset %s\\nfrom :%d\\n\\n\", name,\n@@ -719,6 +976,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"no-data\", &no_data, N_(\"Skip output of blob data\")),\n \t\tOPT_STRING_LIST(0, \"refspec\", &refspecs_list, N_(\"refspec\"),\n \t\t\t N_(\"Apply refspec to exported refs\")),\n+\t\tOPT_BOOL(0, \"anonymize\", &anonymize, N_(\"anonymize output\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\nnew file mode 100755\nindex 0000000..f76ffe4\n--- /dev/null\n+++ b/t/t9351-fast-export-anonymize.sh\n@@ -0,0 +1,117 @@\n+#!/bin/sh\n+\n+test_description='basic tests for fast-export --anonymize'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup simple repo' '\n+\ttest_commit base &&\n+\ttest_commit foo &&\n+\tgit checkout -b other HEAD^ &&\n+\tmkdir subdir &&\n+\ttest_commit subdir/bar &&\n+\ttest_commit subdir/xyzzy &&\n+\tgit tag -m \"annotated tag\" mytag\n+'\n+\n+test_expect_success 'export anonymized stream' '\n+\tgit fast-export --anonymize --all >stream\n+'\n+\n+# this also covers commit messages\n+test_expect_success 'stream omits path names' '\n+\t! fgrep base stream &&\n+\t! fgrep foo stream &&\n+\t! fgrep subdir stream &&\n+\t! fgrep bar stream &&\n+\t! fgrep xyzzy stream\n+'\n+\n+test_expect_success 'stream allows master as refname' '\n+\tfgrep master stream\n+'\n+\n+test_expect_success 'stream omits other refnames' '\n+\t! fgrep other stream\n+'\n+\n+test_expect_success 'stream omits identities' '\n+\t! fgrep \"$GIT_COMMITTER_NAME\" stream &&\n+\t! fgrep \"$GIT_COMMITTER_EMAIL\" stream &&\n+\t! fgrep \"$GIT_AUTHOR_NAME\" stream &&\n+\t! fgrep \"$GIT_AUTHOR_EMAIL\" stream\n+'\n+\n+test_expect_success 'stream omits tag message' '\n+\t! fgrep \"annotated tag\" stream\n+'\n+\n+# NOTE: we chdir to the new, anonymized repository\n+# after this. All further tests should assume this.\n+test_expect_success 'import stream to new repository' '\n+\tgit init new &&\n+\tcd new &&\n+\tgit fast-import <../stream\n+'\n+\n+test_expect_success 'result has two branches' '\n+\tgit for-each-ref --format=\"%(refname)\" refs/heads >branches &&\n+\ttest_line_count = 2 branches &&\n+\tother_branch=$(grep -v refs/heads/master branches)\n+'\n+\n+test_expect_success 'repo has original shape' '\n+\tcat >expect <<-\\EOF &&\n+\t> subject 3\n+\t> subject 2\n+\t< subject 1\n+\t- subject 0\n+\tEOF\n+\tgit log --format=\"%m %s\" --left-right --boundary \\\n+\t\tmaster...$other_branch >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'root tree has original shape' '\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\ttree\n+\tEOF\n+\tgit ls-tree $other_branch >root &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'paths in subdir ended up in one tree' '\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\tblob\n+\tEOF\n+\ttree=$(grep tree root | cut -f2) &&\n+\tgit ls-tree $other_branch:$tree >tree &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'tag points to branch tip' '\n+\tgit rev-parse $other_branch >expect &&\n+\tgit for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'idents are shared' '\n+\tgit log --all --format=\"%an <%ae>\" >authors &&\n+\tsort -u authors >unique &&\n+\ttest_line_count = 1 unique &&\n+\tgit log --all --format=\"%cn <%ce>\" >committers &&\n+\tsort -u committers >unique &&\n+\ttest_line_count = 1 unique &&\n+\t! test_cmp authors committers\n+'\n+\n+test_expect_success 'commit timestamps are retained' '\n+\tgit log --all --format=\"%ct\" >timestamps &&\n+\tsort -u timestamps >unique &&\n+\ttest_line_count = 4 unique\n+'\n+\n+test_done\n-- \n2.1.0.346.ga0367b9\n"},{"id":"248082","messageId":"xmqqmwaxr44x.fsf@gitster.dls.corp.google.com","threadId":"37387","inReplyTo":"20140821070130.GA15930@peff.net","subject":"Re: [PATCH] teach fast-export an --anonymize option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-08-21T20:15:10Z","receivedAt":"2014-08-21T20:15:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> +/*\n> + * We anonymize each component of a path individually,\n> + * so that paths a/b and a/c will share a common root.\n> + * The paths are cached via anonymize_mem so that repeated\n> + * lookups for \"a\" will yield the same value.\n> + */\n> +static void anonymize_path(struct strbuf *out, const char *path,\n> +\t\t\t struct hashmap *map,\n> +\t\t\t char *(*generate)(const char *, size_t *))\n> +{\n> +\twhile (*path) {\n> +\t\tconst char *end_of_component = strchrnul(path, '/');\n> +\t\tsize_t len = end_of_component - path;\n> +\t\tconst char *c = anonymize_mem(map, generate, path, &len);\n> +\t\tstrbuf_add(out, c, len);\n> +\t\tpath = end_of_component;\n> +\t\tif (*path)\n> +\t\t\tstrbuf_addch(out, *path++);\n> +\t}\n> +}\n\nDo two paths sort the same way before and after anonymisation? For\nexample, if generate() works as a simple substitution, it should map\na character that sorts before (or after) '/' with another that also\nsorts before (or after) '/' for us to be able to diagnose an error\nthat comes from D/F sort order confusion.\n"},{"id":"248083","messageId":"xmqqiollqzel.fsf@gitster.dls.corp.google.com","threadId":"37387","inReplyTo":"20140821070130.GA15930@peff.net","subject":"Re: [PATCH] teach fast-export an --anonymize option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-08-21T21:57:22Z","receivedAt":"2014-08-21T21:57:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> +--anonymize::\n> +\tReplace all paths, blob contents, commit and tag messages,\n> +\tnames, and email addresses in the output with anonymized data,\n> +\twhile still retaining the shape of history and of the stored\n> +\ttree.\n\nSometimes branch names can contain codenames the project may prefer\nto hide from the general public, so they may need to be anonymised\nas well.\n"},{"id":"248087","messageId":"20140821224150.GA21105@peff.net","threadId":"37387","inReplyTo":"xmqqmwaxr44x.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-21T22:41:50Z","receivedAt":"2014-08-21T22:41:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 21, 2014 at 01:15:10PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > +/*\n> > + * We anonymize each component of a path individually,\n> > + * so that paths a/b and a/c will share a common root.\n> > + * The paths are cached via anonymize_mem so that repeated\n> > + * lookups for \"a\" will yield the same value.\n> > + */\n> > +static void anonymize_path(struct strbuf *out, const char *path,\n> > +\t\t\t struct hashmap *map,\n> > +\t\t\t char *(*generate)(const char *, size_t *))\n> > +{\n> > +\twhile (*path) {\n> > +\t\tconst char *end_of_component = strchrnul(path, '/');\n> > +\t\tsize_t len = end_of_component - path;\n> > +\t\tconst char *c = anonymize_mem(map, generate, path, &len);\n> > +\t\tstrbuf_add(out, c, len);\n> > +\t\tpath = end_of_component;\n> > +\t\tif (*path)\n> > +\t\t\tstrbuf_addch(out, *path++);\n> > +\t}\n> > +}\n> \n> Do two paths sort the same way before and after anonymisation? For\n> example, if generate() works as a simple substitution, it should map\n> a character that sorts before (or after) '/' with another that also\n> sorts before (or after) '/' for us to be able to diagnose an error\n> that comes from D/F sort order confusion.\n\nNo, the sort order is totally lost. I'd be afraid that a general scheme\nwould end up leaking information about what was in the filenames. It\nmight be acceptable to leak some information here, though, if it adds to\nthe realism of the result.\n\nI tried here to lay the basic infrastructure and do the simplest thing\nthat might work, so we could evaluate proposals like that independently\n(and also because I didn't come up with a clever enough algorithm to do\nwhat you're asking). Patches welcome on top. :)\n\n-Peff\n"},{"id":"248088","messageId":"20140821224910.GB21105@peff.net","threadId":"37387","inReplyTo":"xmqqiollqzel.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-21T22:49:10Z","receivedAt":"2014-08-21T22:49:10Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 21, 2014 at 02:57:22PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> > +--anonymize::\n> > +\tReplace all paths, blob contents, commit and tag messages,\n> > +\tnames, and email addresses in the output with anonymized data,\n> > +\twhile still retaining the shape of history and of the stored\n> > +\ttree.\n> \n> Sometimes branch names can contain codenames the project may prefer\n> to hide from the general public, so they may need to be anonymised\n> as well.\n\nYes, I do anonymize them (and check it in the tests). See\nanonymize_refname. I just forgot to include it in the list. Trivial\nsquashable patch is below.\n\nThe few things I don't anonymize are:\n\n 1. ref prefixes. We see the same distribution of refs/heads vs\n refs/tags, etc.\n\n 2. refs/heads/master is left untouched, for convenience (and because\n it's not really a secret). The implementation is lazy, though, and\n would leave \"refs/heads/master-supersecret\", as well. I can tighten\n that if we really want to be careful.\n\n 3. gitlinks are left untouched, since sha1s cannot be reversed. This\n could leak some information (if your private repo points to a\n public, I can find out you have it as submodule). I doubt it\n matters, but we can also scramble the sha1s.\n\n---\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 0ec7cad..52831fa 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -106,10 +106,10 @@ marks the same across runs.\n \tdifferent from the commit's first parent).\n \n --anonymize::\n-\tReplace all paths, blob contents, commit and tag messages,\n-\tnames, and email addresses in the output with anonymized data,\n-\twhile still retaining the shape of history and of the stored\n-\ttree.\n+\tReplace all refnames, paths, blob contents, commit and tag\n+\tmessages, names, and email addresses in the output with\n+\tanonymized data, while still retaining the shape of history and\n+\tof the stored tree.\n \n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n"},{"id":"248089","messageId":"20140821232100.GA27849@peff.net","threadId":"37387","inReplyTo":"20140821224910.GB21105@peff.net","subject":"[PATCH v2] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-21T23:21:00Z","receivedAt":"2014-08-21T23:21:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 21, 2014 at 06:49:10PM -0400, Jeff King wrote:\n\n> The few things I don't anonymize are:\n> \n> 1. ref prefixes. We see the same distribution of refs/heads vs\n> refs/tags, etc.\n> \n> 2. refs/heads/master is left untouched, for convenience (and because\n> it's not really a secret). The implementation is lazy, though, and\n> would leave \"refs/heads/master-supersecret\", as well. I can tighten\n> that if we really want to be careful.\n> \n> 3. gitlinks are left untouched, since sha1s cannot be reversed. This\n> could leak some information (if your private repo points to a\n> public, I can find out you have it as submodule). I doubt it\n> matters, but we can also scramble the sha1s.\n\nHere's a re-roll that addresses the latter two. I don't think any are a\nbig deal, but it's much easier to say \"it's handled\" than try to figure\nout whether and when it's important.\n\nThis also includes the documentation update I sent earlier. The\ninterdiff is a bit noisy, as I also converted the anonymize_mem function\nto take void pointers (since it doesn't know or care what it's storing,\nand this makes storing unsigned chars for sha1s easier).\n\n-- >8 --\nSubject: teach fast-export an --anonymize option\n\nSometimes users want to report a bug they experience on\ntheir repository, but they are not at liberty to share the\ncontents of the repository. It would be useful if they could\nproduce a repository that has a similar shape to its history\nand tree, but without leaking any information. This\n\"anonymized\" repository could then be shared with developers\n(assuming it still replicates the original problem).\n\nThis patch implements an \"--anonymize\" option to\nfast-export, which generates a stream that can recreate such\na repository. Producing a single stream makes it easy for\nthe caller to verify that they are not leaking any useful\ninformation. You can get an overview of what will be shared\nby running a command like:\n\n git fast-export --anonymize --all |\n perl -pe 's/\\d+/X/g' |\n sort -u |\n less\n\nwhich will show every unique line we generate, modulo any\nnumbers (each anonymized token is assigned a number, like\n\"User 0\", and we replace it consistently in the output).\n\nIn addition to anonymizing, this produces test cases that\nare relatively small (compared to the original repository)\nand fast to generate (compared to using filter-branch, or\nmodifying the output of fast-export yourself). Here are\nnumbers for git.git:\n\n $ time git fast-export --anonymize --all \\\n --tag-of-filtered-object=drop >output\n real 0m2.883s\n user 0m2.828s\n sys 0m0.052s\n\n $ gzip output\n $ ls -lh output.gz | awk '{print $5}'\n 2.9M\n\nSigned-off-by: Jeff King \n---\n Documentation/git-fast-export.txt | 6 +\n builtin/fast-export.c | 300 ++++++++++++++++++++++++++++++++++++--\n t/t9351-fast-export-anonymize.sh | 117 +++++++++++++++\n 3 files changed, 412 insertions(+), 11 deletions(-)\n create mode 100755 t/t9351-fast-export-anonymize.sh\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 221506b..52831fa 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -105,6 +105,12 @@ marks the same across runs.\n \tin the commit (as opposed to just listing the files which are\n \tdifferent from the commit's first parent).\n \n+--anonymize::\n+\tReplace all refnames, paths, blob contents, commit and tag\n+\tmessages, names, and email addresses in the output with\n+\tanonymized data, while still retaining the shape of history and\n+\tof the stored tree.\n+\n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n \tbe specified.\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 92b4624..b8182c2 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"quote.h\"\n #include \"remote.h\"\n+#include \"blob.h\"\n \n static const char *fast_export_usage[] = {\n \tN_(\"git fast-export [rev-list-opts]\"),\n@@ -34,6 +35,7 @@ static int full_tree;\n static struct string_list extra_refs = STRING_LIST_INIT_NODUP;\n static struct refspec *refspecs;\n static int refspecs_nr;\n+static int anonymize;\n \n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -81,6 +83,76 @@ static int has_unshown_parent(struct commit *commit)\n \treturn 0;\n }\n \n+struct anonymized_entry {\n+\tstruct hashmap_entry hash;\n+\tconst char *orig;\n+\tsize_t orig_len;\n+\tconst char *anon;\n+\tsize_t anon_len;\n+};\n+\n+static int anonymized_entry_cmp(const void *va, const void *vb,\n+\t\t\t\tconst void *data)\n+{\n+\tconst struct anonymized_entry *a = va, *b = vb;\n+\treturn a->orig_len != b->orig_len ||\n+\t\tmemcmp(a->orig, b->orig, a->orig_len);\n+}\n+\n+/*\n+ * Basically keep a cache of X->Y so that we can repeatedly replace\n+ * the same anonymized string with another. The actual generation\n+ * is farmed out to the generate function.\n+ */\n+static const void *anonymize_mem(struct hashmap *map,\n+\t\t\t\t void *(*generate)(const void *, size_t *),\n+\t\t\t\t const void *orig, size_t *len)\n+{\n+\tstruct anonymized_entry key, *ret;\n+\n+\tif (!map->cmpfn)\n+\t\thashmap_init(map, anonymized_entry_cmp, 0);\n+\n+\thashmap_entry_init(&key, memhash(orig, *len));\n+\tkey.orig = orig;\n+\tkey.orig_len = *len;\n+\tret = hashmap_get(map, &key, NULL);\n+\n+\tif (!ret) {\n+\t\tret = xmalloc(sizeof(*ret));\n+\t\thashmap_entry_init(&ret->hash, key.hash.hash);\n+\t\tret->orig = xstrdup(orig);\n+\t\tret->orig_len = *len;\n+\t\tret->anon = generate(orig, len);\n+\t\tret->anon_len = *len;\n+\t\thashmap_put(map, ret);\n+\t}\n+\n+\t*len = ret->anon_len;\n+\treturn ret->anon;\n+}\n+\n+/*\n+ * We anonymize each component of a path individually,\n+ * so that paths a/b and a/c will share a common root.\n+ * The paths are cached via anonymize_mem so that repeated\n+ * lookups for \"a\" will yield the same value.\n+ */\n+static void anonymize_path(struct strbuf *out, const char *path,\n+\t\t\t struct hashmap *map,\n+\t\t\t void *(*generate)(const void *, size_t *))\n+{\n+\twhile (*path) {\n+\t\tconst char *end_of_component = strchrnul(path, '/');\n+\t\tsize_t len = end_of_component - path;\n+\t\tconst char *c = anonymize_mem(map, generate, path, &len);\n+\t\tstrbuf_add(out, c, len);\n+\t\tpath = end_of_component;\n+\t\tif (*path)\n+\t\t\tstrbuf_addch(out, *path++);\n+\t}\n+}\n+\n /* Since intptr_t is C99, we do not use it here */\n static inline uint32_t *mark_to_ptr(uint32_t mark)\n {\n@@ -119,6 +191,26 @@ static void show_progress(void)\n \t\tprintf(\"progress %d objects\\n\", counter);\n }\n \n+/*\n+ * Ideally we would want some transformation of the blob data here\n+ * that is unreversible, but would still be the same size and have\n+ * the same data relationship to other blobs (so that we get the same\n+ * delta and packing behavior as the original). But the first and last\n+ * requirements there are probably mutually exclusive, so let's take\n+ * the easy way out for now, and just generate arbitrary content.\n+ *\n+ * There's no need to cache this result with anonymize_mem, since\n+ * we already handle blob content caching with marks.\n+ */\n+static char *anonymize_blob(unsigned long *size)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"anonymous blob %d\", counter++);\n+\t*size = out.len;\n+\treturn strbuf_detach(&out, NULL);\n+}\n+\n static void export_blob(const unsigned char *sha1)\n {\n \tunsigned long size;\n@@ -137,12 +229,19 @@ static void export_blob(const unsigned char *sha1)\n \tif (object && object->flags & SHOWN)\n \t\treturn;\n \n-\tbuf = read_sha1_file(sha1, &type, &size);\n-\tif (!buf)\n-\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n-\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n-\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n-\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\tif (anonymize) {\n+\t\tbuf = anonymize_blob(&size);\n+\t\tobject = (struct object *)lookup_blob(sha1);\n+\t\teaten = 0;\n+\t} else {\n+\t\tbuf = read_sha1_file(sha1, &type, &size);\n+\t\tif (!buf)\n+\t\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n+\t\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n+\t\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n+\t\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\t}\n+\n \tif (!object)\n \t\tdie(\"Could not read blob %s\", sha1_to_hex(sha1));\n \n@@ -190,7 +289,7 @@ static int depth_first(const void *a_, const void *b_)\n \treturn (a->status == 'R') - (b->status == 'R');\n }\n \n-static void print_path(const char *path)\n+static void print_path_1(const char *path)\n {\n \tint need_quote = quote_c_style(path, NULL, NULL, 0);\n \tif (need_quote)\n@@ -201,6 +300,43 @@ static void print_path(const char *path)\n \t\tprintf(\"%s\", path);\n }\n \n+static void *anonymize_path_component(const void *path, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"path%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static void print_path(const char *path)\n+{\n+\tif (!anonymize)\n+\t\tprint_path_1(path);\n+\telse {\n+\t\tstatic struct hashmap paths;\n+\t\tstatic struct strbuf anon = STRBUF_INIT;\n+\n+\t\tanonymize_path(&anon, path, &paths, anonymize_path_component);\n+\t\tprint_path_1(anon.buf);\n+\t\tstrbuf_reset(&anon);\n+\t}\n+}\n+\n+static void *generate_fake_sha1(const void *old, size_t *len)\n+{\n+\tstatic uint32_t counter = 1; /* avoid null sha1 */\n+\tunsigned char *out = xcalloc(20, 1);\n+\tput_be32(out + 16, counter++);\n+\treturn out;\n+}\n+\n+static const unsigned char *anonymize_sha1(const unsigned char *sha1)\n+{\n+\tstatic struct hashmap sha1s;\n+\tsize_t len = 20;\n+\treturn anonymize_mem(&sha1s, generate_fake_sha1, sha1, &len);\n+}\n+\n static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t struct diff_options *options, void *data)\n {\n@@ -245,7 +381,9 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t */\n \t\t\tif (no_data || S_ISGITLINK(spec->mode))\n \t\t\t\tprintf(\"M %06o %s \", spec->mode,\n-\t\t\t\t sha1_to_hex(spec->sha1));\n+\t\t\t\t sha1_to_hex(anonymize ?\n+\t\t\t\t\t\t anonymize_sha1(spec->sha1) :\n+\t\t\t\t\t\t spec->sha1));\n \t\t\telse {\n \t\t\t\tstruct object *object = lookup_object(spec->sha1);\n \t\t\t\tprintf(\"M %06o :%d \", spec->mode,\n@@ -279,6 +417,114 @@ static const char *find_encoding(const char *begin, const char *end)\n \treturn bol;\n }\n \n+static void *anonymize_ref_component(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"ref%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static const char *anonymize_refname(const char *refname)\n+{\n+\t/*\n+\t * If any of these prefixes is found, we will leave it intact\n+\t * so that tags remain tags and so forth.\n+\t */\n+\tstatic const char *prefixes[] = {\n+\t\t\"refs/heads/\",\n+\t\t\"refs/tags/\",\n+\t\t\"refs/remotes/\",\n+\t\t\"refs/\"\n+\t};\n+\tstatic struct hashmap refs;\n+\tstatic struct strbuf anon = STRBUF_INIT;\n+\tint i;\n+\n+\t/*\n+\t * We also leave \"master\" as a special case, since it does not reveal\n+\t * anything interesting.\n+\t */\n+\tif (!strcmp(refname, \"refs/heads/master\"))\n+\t\treturn refname;\n+\n+\tstrbuf_reset(&anon);\n+\tfor (i = 0; i < ARRAY_SIZE(prefixes); i++) {\n+\t\tif (skip_prefix(refname, prefixes[i], &refname)) {\n+\t\t\tstrbuf_addstr(&anon, prefixes[i]);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tanonymize_path(&anon, refname, &refs, anonymize_ref_component);\n+\treturn anon.buf;\n+}\n+\n+/*\n+ * We do not even bother to cache commit messages, as they are unlikely\n+ * to be repeated verbatim, and it is not that interesting when they are.\n+ */\n+static char *anonymize_commit_message(const char *old)\n+{\n+\tstatic int counter;\n+\treturn xstrfmt(\"subject %d\\n\\nbody\\n\", counter++);\n+}\n+\n+static struct hashmap idents;\n+static void *anonymize_ident(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"User %d \", counter, counter);\n+\tcounter++;\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+/*\n+ * Our strategy here is to anonymize the names and email addresses,\n+ * but keep timestamps intact, as they influence things like traversal\n+ * order (and by themselves should not be too revealing).\n+ */\n+static void anonymize_ident_line(const char **beg, const char **end)\n+{\n+\tstatic struct strbuf buffers[] = { STRBUF_INIT, STRBUF_INIT };\n+\tstatic unsigned which_buffer;\n+\n+\tstruct strbuf *out;\n+\tstruct ident_split split;\n+\tconst char *end_of_header;\n+\n+\tout = &buffers[which_buffer++];\n+\twhich_buffer %= ARRAY_SIZE(buffers);\n+\tstrbuf_reset(out);\n+\n+\t/* skip \"committer\", \"author\", \"tagger\", etc */\n+\tend_of_header = strchr(*beg, ' ');\n+\tif (!end_of_header)\n+\t\tdie(\"BUG: malformed line fed to anonymize_ident_line: %.*s\",\n+\t\t (int)(*end - *beg), *beg);\n+\tend_of_header++;\n+\tstrbuf_add(out, *beg, end_of_header - *beg);\n+\n+\tif (!split_ident_line(&split, end_of_header, *end - end_of_header) &&\n+\t split.date_begin) {\n+\t\tconst char *ident;\n+\t\tsize_t len;\n+\n+\t\tlen = split.mail_end - split.name_begin;\n+\t\tident = anonymize_mem(&idents, anonymize_ident,\n+\t\t\t\t split.name_begin, &len);\n+\t\tstrbuf_add(out, ident, len);\n+\t\tstrbuf_addch(out, ' ');\n+\t\tstrbuf_add(out, split.date_begin, split.tz_end - split.date_begin);\n+\t} else {\n+\t\tstrbuf_addstr(out, \"Malformed Ident 0 -0000\");\n+\t}\n+\n+\t*beg = out->buf;\n+\t*end = out->buf + out->len;\n+}\n+\n static void handle_commit(struct commit *commit, struct rev_info *rev)\n {\n \tint saved_output_format = rev->diffopt.output_format;\n@@ -287,6 +533,7 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tconst char *encoding, *message;\n \tchar *reencoded = NULL;\n \tstruct commit_list *p;\n+\tconst char *refname;\n \tint i;\n \n \trev->diffopt.output_format = DIFF_FORMAT_CALLBACK;\n@@ -326,13 +573,22 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \t\tif (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))\n \t\t\texport_blob(diff_queued_diff.queue[i]->two->sha1);\n \n+\trefname = commit->util;\n+\tif (anonymize) {\n+\t\trefname = anonymize_refname(refname);\n+\t\tanonymize_ident_line(&committer, &committer_end);\n+\t\tanonymize_ident_line(&author, &author_end);\n+\t}\n+\n \tmark_next_object(&commit->object);\n-\tif (!is_encoding_utf8(encoding))\n+\tif (anonymize)\n+\t\treencoded = anonymize_commit_message(message);\n+\telse if (!is_encoding_utf8(encoding))\n \t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n \tif (!commit->parents)\n-\t\tprintf(\"reset %s\\n\", (const char*)commit->util);\n+\t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n%.*s\\n%.*s\\ndata %u\\n%s\",\n-\t (const char *)commit->util, last_idnum,\n+\t refname, last_idnum,\n \t (int)(author_end - author), author,\n \t (int)(committer_end - committer), committer,\n \t (unsigned)(reencoded\n@@ -363,6 +619,14 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tshow_progress();\n }\n \n+static void *anonymize_tag(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"tag message %d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n static void handle_tail(struct object_array *commits, struct rev_info *revs)\n {\n \tstruct commit *commit;\n@@ -419,6 +683,17 @@ static void handle_tag(const char *name, struct tag *tag)\n \t} else {\n \t\ttagger++;\n \t\ttagger_end = strchrnul(tagger, '\\n');\n+\t\tif (anonymize)\n+\t\t\tanonymize_ident_line(&tagger, &tagger_end);\n+\t}\n+\n+\tif (anonymize) {\n+\t\tname = anonymize_refname(name);\n+\t\tif (message) {\n+\t\t\tstatic struct hashmap tags;\n+\t\t\tmessage = anonymize_mem(&tags, anonymize_tag,\n+\t\t\t\t\t\tmessage, &message_size);\n+\t\t}\n \t}\n \n \t/* handle signed tags */\n@@ -584,6 +859,8 @@ static void handle_tags_and_duplicates(void)\n \t\t\thandle_tag(name, (struct tag *)object);\n \t\t\tbreak;\n \t\tcase OBJ_COMMIT:\n+\t\t\tif (anonymize)\n+\t\t\t\tname = anonymize_refname(name);\n \t\t\t/* create refs pointing to already seen commits */\n \t\t\tcommit = (struct commit *)object;\n \t\t\tprintf(\"reset %s\\nfrom :%d\\n\\n\", name,\n@@ -719,6 +996,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"no-data\", &no_data, N_(\"Skip output of blob data\")),\n \t\tOPT_STRING_LIST(0, \"refspec\", &refspecs_list, N_(\"refspec\"),\n \t\t\t N_(\"Apply refspec to exported refs\")),\n+\t\tOPT_BOOL(0, \"anonymize\", &anonymize, N_(\"anonymize output\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\nnew file mode 100755\nindex 0000000..f76ffe4\n--- /dev/null\n+++ b/t/t9351-fast-export-anonymize.sh\n@@ -0,0 +1,117 @@\n+#!/bin/sh\n+\n+test_description='basic tests for fast-export --anonymize'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup simple repo' '\n+\ttest_commit base &&\n+\ttest_commit foo &&\n+\tgit checkout -b other HEAD^ &&\n+\tmkdir subdir &&\n+\ttest_commit subdir/bar &&\n+\ttest_commit subdir/xyzzy &&\n+\tgit tag -m \"annotated tag\" mytag\n+'\n+\n+test_expect_success 'export anonymized stream' '\n+\tgit fast-export --anonymize --all >stream\n+'\n+\n+# this also covers commit messages\n+test_expect_success 'stream omits path names' '\n+\t! fgrep base stream &&\n+\t! fgrep foo stream &&\n+\t! fgrep subdir stream &&\n+\t! fgrep bar stream &&\n+\t! fgrep xyzzy stream\n+'\n+\n+test_expect_success 'stream allows master as refname' '\n+\tfgrep master stream\n+'\n+\n+test_expect_success 'stream omits other refnames' '\n+\t! fgrep other stream\n+'\n+\n+test_expect_success 'stream omits identities' '\n+\t! fgrep \"$GIT_COMMITTER_NAME\" stream &&\n+\t! fgrep \"$GIT_COMMITTER_EMAIL\" stream &&\n+\t! fgrep \"$GIT_AUTHOR_NAME\" stream &&\n+\t! fgrep \"$GIT_AUTHOR_EMAIL\" stream\n+'\n+\n+test_expect_success 'stream omits tag message' '\n+\t! fgrep \"annotated tag\" stream\n+'\n+\n+# NOTE: we chdir to the new, anonymized repository\n+# after this. All further tests should assume this.\n+test_expect_success 'import stream to new repository' '\n+\tgit init new &&\n+\tcd new &&\n+\tgit fast-import <../stream\n+'\n+\n+test_expect_success 'result has two branches' '\n+\tgit for-each-ref --format=\"%(refname)\" refs/heads >branches &&\n+\ttest_line_count = 2 branches &&\n+\tother_branch=$(grep -v refs/heads/master branches)\n+'\n+\n+test_expect_success 'repo has original shape' '\n+\tcat >expect <<-\\EOF &&\n+\t> subject 3\n+\t> subject 2\n+\t< subject 1\n+\t- subject 0\n+\tEOF\n+\tgit log --format=\"%m %s\" --left-right --boundary \\\n+\t\tmaster...$other_branch >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'root tree has original shape' '\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\ttree\n+\tEOF\n+\tgit ls-tree $other_branch >root &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'paths in subdir ended up in one tree' '\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\tblob\n+\tEOF\n+\ttree=$(grep tree root | cut -f2) &&\n+\tgit ls-tree $other_branch:$tree >tree &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'tag points to branch tip' '\n+\tgit rev-parse $other_branch >expect &&\n+\tgit for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'idents are shared' '\n+\tgit log --all --format=\"%an <%ae>\" >authors &&\n+\tsort -u authors >unique &&\n+\ttest_line_count = 1 unique &&\n+\tgit log --all --format=\"%cn <%ce>\" >committers &&\n+\tsort -u committers >unique &&\n+\ttest_line_count = 1 unique &&\n+\t! test_cmp authors committers\n+'\n+\n+test_expect_success 'commit timestamps are retained' '\n+\tgit log --all --format=\"%ct\" >timestamps &&\n+\tsort -u timestamps >unique &&\n+\ttest_line_count = 4 unique\n+'\n+\n+test_done\n-- \n2.1.0.346.ga0367b9\n"},{"id":"248110","messageId":"CACsJy8DHAeaohbt8TpD=Sr8GvF3rp9D4-vePq_EbZQWVVWcGnw@mail.gmail.com","threadId":"37387","inReplyTo":"20140821232100.GA27849@peff.net","subject":"Re: [PATCH v2] teach fast-export an --anonymize option","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-08-22T13:06:16Z","receivedAt":"2014-08-22T13:06:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Aug 22, 2014 at 6:21 AM, Jeff King wrote:\n> -- >8 --\n> Subject: teach fast-export an --anonymize option\n>\n> Sometimes users want to report a bug they experience on\n> their repository, but they are not at liberty to share the\n> contents of the repository. It would be useful if they could\n> produce a repository that has a similar shape to its history\n> and tree, but without leaking any information. This\n> \"anonymized\" repository could then be shared with developers\n> (assuming it still replicates the original problem).\n\nThis is cool. Thanks Jeff. Steven could you try this with the repo\nthat failed shallow clone --no-single-branch the other day?\n\n>\n> This patch implements an \"--anonymize\" option to\n> fast-export, which generates a stream that can recreate such\n> a repository. Producing a single stream makes it easy for\n> the caller to verify that they are not leaking any useful\n> information. You can get an overview of what will be shared\n> by running a command like:\n>\n> git fast-export --anonymize --all |\n> perl -pe 's/\\d+/X/g' |\n> sort -u |\n> less\n>\n> which will show every unique line we generate, modulo any\n> numbers (each anonymized token is assigned a number, like\n> \"User 0\", and we replace it consistently in the output).\n>\n> In addition to anonymizing, this produces test cases that\n> are relatively small (compared to the original repository)\n> and fast to generate (compared to using filter-branch, or\n> modifying the output of fast-export yourself). Here are\n> numbers for git.git:\n>\n> $ time git fast-export --anonymize --all \\\n> --tag-of-filtered-object=drop >output\n> real 0m2.883s\n> user 0m2.828s\n> sys 0m0.052s\n>\n> $ gzip output\n> $ ls -lh output.gz | awk '{print $5}'\n> 2.9M\n>\n> Signed-off-by: Jeff King \n> ---\n> Documentation/git-fast-export.txt | 6 +\n> builtin/fast-export.c | 300 ++++++++++++++++++++++++++++++++++++--\n> t/t9351-fast-export-anonymize.sh | 117 +++++++++++++++\n> 3 files changed, 412 insertions(+), 11 deletions(-)\n> create mode 100755 t/t9351-fast-export-anonymize.sh\n>\n> diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\n> index 221506b..52831fa 100644\n> --- a/Documentation/git-fast-export.txt\n> +++ b/Documentation/git-fast-export.txt\n> @@ -105,6 +105,12 @@ marks the same across runs.\n> in the commit (as opposed to just listing the files which are\n> different from the commit's first parent).\n>\n> +--anonymize::\n> + Replace all refnames, paths, blob contents, commit and tag\n> + messages, names, and email addresses in the output with\n> + anonymized data, while still retaining the shape of history and\n> + of the stored tree.\n> +\n> --refspec::\n> Apply the specified refspec to each ref exported. Multiple of them can\n> be specified.\n> diff --git a/builtin/fast-export.c b/builtin/fast-export.c\n> index 92b4624..b8182c2 100644\n> --- a/builtin/fast-export.c\n> +++ b/builtin/fast-export.c\n> @@ -18,6 +18,7 @@\n> #include \"parse-options.h\"\n> #include \"quote.h\"\n> #include \"remote.h\"\n> +#include \"blob.h\"\n>\n> static const char *fast_export_usage[] = {\n> N_(\"git fast-export [rev-list-opts]\"),\n> @@ -34,6 +35,7 @@ static int full_tree;\n> static struct string_list extra_refs = STRING_LIST_INIT_NODUP;\n> static struct refspec *refspecs;\n> static int refspecs_nr;\n> +static int anonymize;\n>\n> static int parse_opt_signed_tag_mode(const struct option *opt,\n> const char *arg, int unset)\n> @@ -81,6 +83,76 @@ static int has_unshown_parent(struct commit *commit)\n> return 0;\n> }\n>\n> +struct anonymized_entry {\n> + struct hashmap_entry hash;\n> + const char *orig;\n> + size_t orig_len;\n> + const char *anon;\n> + size_t anon_len;\n> +};\n> +\n> +static int anonymized_entry_cmp(const void *va, const void *vb,\n> + const void *data)\n> +{\n> + const struct anonymized_entry *a = va, *b = vb;\n> + return a->orig_len != b->orig_len ||\n> + memcmp(a->orig, b->orig, a->orig_len);\n> +}\n> +\n> +/*\n> + * Basically keep a cache of X->Y so that we can repeatedly replace\n> + * the same anonymized string with another. The actual generation\n> + * is farmed out to the generate function.\n> + */\n> +static const void *anonymize_mem(struct hashmap *map,\n> + void *(*generate)(const void *, size_t *),\n> + const void *orig, size_t *len)\n> +{\n> + struct anonymized_entry key, *ret;\n> +\n> + if (!map->cmpfn)\n> + hashmap_init(map, anonymized_entry_cmp, 0);\n> +\n> + hashmap_entry_init(&key, memhash(orig, *len));\n> + key.orig = orig;\n> + key.orig_len = *len;\n> + ret = hashmap_get(map, &key, NULL);\n> +\n> + if (!ret) {\n> + ret = xmalloc(sizeof(*ret));\n> + hashmap_entry_init(&ret->hash, key.hash.hash);\n> + ret->orig = xstrdup(orig);\n> + ret->orig_len = *len;\n> + ret->anon = generate(orig, len);\n> + ret->anon_len = *len;\n> + hashmap_put(map, ret);\n> + }\n> +\n> + *len = ret->anon_len;\n> + return ret->anon;\n> +}\n> +\n> +/*\n> + * We anonymize each component of a path individually,\n> + * so that paths a/b and a/c will share a common root.\n> + * The paths are cached via anonymize_mem so that repeated\n> + * lookups for \"a\" will yield the same value.\n> + */\n> +static void anonymize_path(struct strbuf *out, const char *path,\n> + struct hashmap *map,\n> + void *(*generate)(const void *, size_t *))\n> +{\n> + while (*path) {\n> + const char *end_of_component = strchrnul(path, '/');\n> + size_t len = end_of_component - path;\n> + const char *c = anonymize_mem(map, generate, path, &len);\n> + strbuf_add(out, c, len);\n> + path = end_of_component;\n> + if (*path)\n> + strbuf_addch(out, *path++);\n> + }\n> +}\n> +\n> /* Since intptr_t is C99, we do not use it here */\n> static inline uint32_t *mark_to_ptr(uint32_t mark)\n> {\n> @@ -119,6 +191,26 @@ static void show_progress(void)\n> printf(\"progress %d objects\\n\", counter);\n> }\n>\n> +/*\n> + * Ideally we would want some transformation of the blob data here\n> + * that is unreversible, but would still be the same size and have\n> + * the same data relationship to other blobs (so that we get the same\n> + * delta and packing behavior as the original). But the first and last\n> + * requirements there are probably mutually exclusive, so let's take\n> + * the easy way out for now, and just generate arbitrary content.\n> + *\n> + * There's no need to cache this result with anonymize_mem, since\n> + * we already handle blob content caching with marks.\n> + */\n> +static char *anonymize_blob(unsigned long *size)\n> +{\n> + static int counter;\n> + struct strbuf out = STRBUF_INIT;\n> + strbuf_addf(&out, \"anonymous blob %d\", counter++);\n> + *size = out.len;\n> + return strbuf_detach(&out, NULL);\n> +}\n> +\n> static void export_blob(const unsigned char *sha1)\n> {\n> unsigned long size;\n> @@ -137,12 +229,19 @@ static void export_blob(const unsigned char *sha1)\n> if (object && object->flags & SHOWN)\n> return;\n>\n> - buf = read_sha1_file(sha1, &type, &size);\n> - if (!buf)\n> - die (\"Could not read blob %s\", sha1_to_hex(sha1));\n> - if (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n> - die(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n> - object = parse_object_buffer(sha1, type, size, buf, &eaten);\n> + if (anonymize) {\n> + buf = anonymize_blob(&size);\n> + object = (struct object *)lookup_blob(sha1);\n> + eaten = 0;\n> + } else {\n> + buf = read_sha1_file(sha1, &type, &size);\n> + if (!buf)\n> + die (\"Could not read blob %s\", sha1_to_hex(sha1));\n> + if (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n> + die(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n> + object = parse_object_buffer(sha1, type, size, buf, &eaten);\n> + }\n> +\n> if (!object)\n> die(\"Could not read blob %s\", sha1_to_hex(sha1));\n>\n> @@ -190,7 +289,7 @@ static int depth_first(const void *a_, const void *b_)\n> return (a->status == 'R') - (b->status == 'R');\n> }\n>\n> -static void print_path(const char *path)\n> +static void print_path_1(const char *path)\n> {\n> int need_quote = quote_c_style(path, NULL, NULL, 0);\n> if (need_quote)\n> @@ -201,6 +300,43 @@ static void print_path(const char *path)\n> printf(\"%s\", path);\n> }\n>\n> +static void *anonymize_path_component(const void *path, size_t *len)\n> +{\n> + static int counter;\n> + struct strbuf out = STRBUF_INIT;\n> + strbuf_addf(&out, \"path%d\", counter++);\n> + return strbuf_detach(&out, len);\n> +}\n> +\n> +static void print_path(const char *path)\n> +{\n> + if (!anonymize)\n> + print_path_1(path);\n> + else {\n> + static struct hashmap paths;\n> + static struct strbuf anon = STRBUF_INIT;\n> +\n> + anonymize_path(&anon, path, &paths, anonymize_path_component);\n> + print_path_1(anon.buf);\n> + strbuf_reset(&anon);\n> + }\n> +}\n> +\n> +static void *generate_fake_sha1(const void *old, size_t *len)\n> +{\n> + static uint32_t counter = 1; /* avoid null sha1 */\n> + unsigned char *out = xcalloc(20, 1);\n> + put_be32(out + 16, counter++);\n> + return out;\n> +}\n> +\n> +static const unsigned char *anonymize_sha1(const unsigned char *sha1)\n> +{\n> + static struct hashmap sha1s;\n> + size_t len = 20;\n> + return anonymize_mem(&sha1s, generate_fake_sha1, sha1, &len);\n> +}\n> +\n> static void show_filemodify(struct diff_queue_struct *q,\n> struct diff_options *options, void *data)\n> {\n> @@ -245,7 +381,9 @@ static void show_filemodify(struct diff_queue_struct *q,\n> */\n> if (no_data || S_ISGITLINK(spec->mode))\n> printf(\"M %06o %s \", spec->mode,\n> - sha1_to_hex(spec->sha1));\n> + sha1_to_hex(anonymize ?\n> + anonymize_sha1(spec->sha1) :\n> + spec->sha1));\n> else {\n> struct object *object = lookup_object(spec->sha1);\n> printf(\"M %06o :%d \", spec->mode,\n> @@ -279,6 +417,114 @@ static const char *find_encoding(const char *begin, const char *end)\n> return bol;\n> }\n>\n> +static void *anonymize_ref_component(const void *old, size_t *len)\n> +{\n> + static int counter;\n> + struct strbuf out = STRBUF_INIT;\n> + strbuf_addf(&out, \"ref%d\", counter++);\n> + return strbuf_detach(&out, len);\n> +}\n> +\n> +static const char *anonymize_refname(const char *refname)\n> +{\n> + /*\n> + * If any of these prefixes is found, we will leave it intact\n> + * so that tags remain tags and so forth.\n> + */\n> + static const char *prefixes[] = {\n> + \"refs/heads/\",\n> + \"refs/tags/\",\n> + \"refs/remotes/\",\n> + \"refs/\"\n> + };\n> + static struct hashmap refs;\n> + static struct strbuf anon = STRBUF_INIT;\n> + int i;\n> +\n> + /*\n> + * We also leave \"master\" as a special case, since it does not reveal\n> + * anything interesting.\n> + */\n> + if (!strcmp(refname, \"refs/heads/master\"))\n> + return refname;\n> +\n> + strbuf_reset(&anon);\n> + for (i = 0; i < ARRAY_SIZE(prefixes); i++) {\n> + if (skip_prefix(refname, prefixes[i], &refname)) {\n> + strbuf_addstr(&anon, prefixes[i]);\n> + break;\n> + }\n> + }\n> +\n> + anonymize_path(&anon, refname, &refs, anonymize_ref_component);\n> + return anon.buf;\n> +}\n> +\n> +/*\n> + * We do not even bother to cache commit messages, as they are unlikely\n> + * to be repeated verbatim, and it is not that interesting when they are.\n> + */\n> +static char *anonymize_commit_message(const char *old)\n> +{\n> + static int counter;\n> + return xstrfmt(\"subject %d\\n\\nbody\\n\", counter++);\n> +}\n> +\n> +static struct hashmap idents;\n> +static void *anonymize_ident(const void *old, size_t *len)\n> +{\n> + static int counter;\n> + struct strbuf out = STRBUF_INIT;\n> + strbuf_addf(&out, \"User %d \", counter, counter);\n> + counter++;\n> + return strbuf_detach(&out, len);\n> +}\n> +\n> +/*\n> + * Our strategy here is to anonymize the names and email addresses,\n> + * but keep timestamps intact, as they influence things like traversal\n> + * order (and by themselves should not be too revealing).\n> + */\n> +static void anonymize_ident_line(const char **beg, const char **end)\n> +{\n> + static struct strbuf buffers[] = { STRBUF_INIT, STRBUF_INIT };\n> + static unsigned which_buffer;\n> +\n> + struct strbuf *out;\n> + struct ident_split split;\n> + const char *end_of_header;\n> +\n> + out = &buffers[which_buffer++];\n> + which_buffer %= ARRAY_SIZE(buffers);\n> + strbuf_reset(out);\n> +\n> + /* skip \"committer\", \"author\", \"tagger\", etc */\n> + end_of_header = strchr(*beg, ' ');\n> + if (!end_of_header)\n> + die(\"BUG: malformed line fed to anonymize_ident_line: %.*s\",\n> + (int)(*end - *beg), *beg);\n> + end_of_header++;\n> + strbuf_add(out, *beg, end_of_header - *beg);\n> +\n> + if (!split_ident_line(&split, end_of_header, *end - end_of_header) &&\n> + split.date_begin) {\n> + const char *ident;\n> + size_t len;\n> +\n> + len = split.mail_end - split.name_begin;\n> + ident = anonymize_mem(&idents, anonymize_ident,\n> + split.name_begin, &len);\n> + strbuf_add(out, ident, len);\n> + strbuf_addch(out, ' ');\n> + strbuf_add(out, split.date_begin, split.tz_end - split.date_begin);\n> + } else {\n> + strbuf_addstr(out, \"Malformed Ident 0 -0000\");\n> + }\n> +\n> + *beg = out->buf;\n> + *end = out->buf + out->len;\n> +}\n> +\n> static void handle_commit(struct commit *commit, struct rev_info *rev)\n> {\n> int saved_output_format = rev->diffopt.output_format;\n> @@ -287,6 +533,7 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n> const char *encoding, *message;\n> char *reencoded = NULL;\n> struct commit_list *p;\n> + const char *refname;\n> int i;\n>\n> rev->diffopt.output_format = DIFF_FORMAT_CALLBACK;\n> @@ -326,13 +573,22 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n> if (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))\n> export_blob(diff_queued_diff.queue[i]->two->sha1);\n>\n> + refname = commit->util;\n> + if (anonymize) {\n> + refname = anonymize_refname(refname);\n> + anonymize_ident_line(&committer, &committer_end);\n> + anonymize_ident_line(&author, &author_end);\n> + }\n> +\n> mark_next_object(&commit->object);\n> - if (!is_encoding_utf8(encoding))\n> + if (anonymize)\n> + reencoded = anonymize_commit_message(message);\n> + else if (!is_encoding_utf8(encoding))\n> reencoded = reencode_string(message, \"UTF-8\", encoding);\n> if (!commit->parents)\n> - printf(\"reset %s\\n\", (const char*)commit->util);\n> + printf(\"reset %s\\n\", refname);\n> printf(\"commit %s\\nmark :%\"PRIu32\"\\n%.*s\\n%.*s\\ndata %u\\n%s\",\n> - (const char *)commit->util, last_idnum,\n> + refname, last_idnum,\n> (int)(author_end - author), author,\n> (int)(committer_end - committer), committer,\n> (unsigned)(reencoded\n> @@ -363,6 +619,14 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n> show_progress();\n> }\n>\n> +static void *anonymize_tag(const void *old, size_t *len)\n> +{\n> + static int counter;\n> + struct strbuf out = STRBUF_INIT;\n> + strbuf_addf(&out, \"tag message %d\", counter++);\n> + return strbuf_detach(&out, len);\n> +}\n> +\n> static void handle_tail(struct object_array *commits, struct rev_info *revs)\n> {\n> struct commit *commit;\n> @@ -419,6 +683,17 @@ static void handle_tag(const char *name, struct tag *tag)\n> } else {\n> tagger++;\n> tagger_end = strchrnul(tagger, '\\n');\n> + if (anonymize)\n> + anonymize_ident_line(&tagger, &tagger_end);\n> + }\n> +\n> + if (anonymize) {\n> + name = anonymize_refname(name);\n> + if (message) {\n> + static struct hashmap tags;\n> + message = anonymize_mem(&tags, anonymize_tag,\n> + message, &message_size);\n> + }\n> }\n>\n> /* handle signed tags */\n> @@ -584,6 +859,8 @@ static void handle_tags_and_duplicates(void)\n> handle_tag(name, (struct tag *)object);\n> break;\n> case OBJ_COMMIT:\n> + if (anonymize)\n> + name = anonymize_refname(name);\n> /* create refs pointing to already seen commits */\n> commit = (struct commit *)object;\n> printf(\"reset %s\\nfrom :%d\\n\\n\", name,\n> @@ -719,6 +996,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n> OPT_BOOL(0, \"no-data\", &no_data, N_(\"Skip output of blob data\")),\n> OPT_STRING_LIST(0, \"refspec\", &refspecs_list, N_(\"refspec\"),\n> N_(\"Apply refspec to exported refs\")),\n> + OPT_BOOL(0, \"anonymize\", &anonymize, N_(\"anonymize output\")),\n> OPT_END()\n> };\n>\n> diff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\n> new file mode 100755\n> index 0000000..f76ffe4\n> --- /dev/null\n> +++ b/t/t9351-fast-export-anonymize.sh\n> @@ -0,0 +1,117 @@\n> +#!/bin/sh\n> +\n> +test_description='basic tests for fast-export --anonymize'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup simple repo' '\n> + test_commit base &&\n> + test_commit foo &&\n> + git checkout -b other HEAD^ &&\n> + mkdir subdir &&\n> + test_commit subdir/bar &&\n> + test_commit subdir/xyzzy &&\n> + git tag -m \"annotated tag\" mytag\n> +'\n> +\n> +test_expect_success 'export anonymized stream' '\n> + git fast-export --anonymize --all >stream\n> +'\n> +\n> +# this also covers commit messages\n> +test_expect_success 'stream omits path names' '\n> + ! fgrep base stream &&\n> + ! fgrep foo stream &&\n> + ! fgrep subdir stream &&\n> + ! fgrep bar stream &&\n> + ! fgrep xyzzy stream\n> +'\n> +\n> +test_expect_success 'stream allows master as refname' '\n> + fgrep master stream\n> +'\n> +\n> +test_expect_success 'stream omits other refnames' '\n> + ! fgrep other stream\n> +'\n> +\n> +test_expect_success 'stream omits identities' '\n> + ! fgrep \"$GIT_COMMITTER_NAME\" stream &&\n> + ! fgrep \"$GIT_COMMITTER_EMAIL\" stream &&\n> + ! fgrep \"$GIT_AUTHOR_NAME\" stream &&\n> + ! fgrep \"$GIT_AUTHOR_EMAIL\" stream\n> +'\n> +\n> +test_expect_success 'stream omits tag message' '\n> + ! fgrep \"annotated tag\" stream\n> +'\n> +\n> +# NOTE: we chdir to the new, anonymized repository\n> +# after this. All further tests should assume this.\n> +test_expect_success 'import stream to new repository' '\n> + git init new &&\n> + cd new &&\n> + git fast-import <../stream\n> +'\n> +\n> +test_expect_success 'result has two branches' '\n> + git for-each-ref --format=\"%(refname)\" refs/heads >branches &&\n> + test_line_count = 2 branches &&\n> + other_branch=$(grep -v refs/heads/master branches)\n> +'\n> +\n> +test_expect_success 'repo has original shape' '\n> + cat >expect <<-\\EOF &&\n> + > subject 3\n> + > subject 2\n> + < subject 1\n> + - subject 0\n> + EOF\n> + git log --format=\"%m %s\" --left-right --boundary \\\n> + master...$other_branch >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'root tree has original shape' '\n> + cat >expect <<-\\EOF &&\n> + blob\n> + tree\n> + EOF\n> + git ls-tree $other_branch >root &&\n> + cut -d\" \" -f2 actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'paths in subdir ended up in one tree' '\n> + cat >expect <<-\\EOF &&\n> + blob\n> + blob\n> + EOF\n> + tree=$(grep tree root | cut -f2) &&\n> + git ls-tree $other_branch:$tree >tree &&\n> + cut -d\" \" -f2 actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'tag points to branch tip' '\n> + git rev-parse $other_branch >expect &&\n> + git for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'idents are shared' '\n> + git log --all --format=\"%an <%ae>\" >authors &&\n> + sort -u authors >unique &&\n> + test_line_count = 1 unique &&\n> + git log --all --format=\"%cn <%ce>\" >committers &&\n> + sort -u committers >unique &&\n> + test_line_count = 1 unique &&\n> + ! test_cmp authors committers\n> +'\n> +\n> +test_expect_success 'commit timestamps are retained' '\n> + git log --all --format=\"%ct\" >timestamps &&\n> + sort -u timestamps >unique &&\n> + test_line_count = 4 unique\n> +'\n> +\n> +test_done\n> --\n> 2.1.0.346.ga0367b9\n>\n\n\n\n-- \nDuy\n"},{"id":"248134","messageId":"9E69746EFF0048FF84631A9794206566@PhilipOakley","threadId":"37387","inReplyTo":"20140821232100.GA27849@peff.net","subject":"Re: [PATCH v2] teach fast-export an --anonymize option","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2014-08-22T18:39:59Z","receivedAt":"2014-08-22T18:39:59Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jeff King\" : Friday, August 22, 2014 12:21 AM\n> On Thu, Aug 21, 2014 at 06:49:10PM -0400, Jeff King wrote:\n>\n>> The few things I don't anonymize are:\n>>\n>> 1. ref prefixes. We see the same distribution of refs/heads vs\n>> refs/tags, etc.\n>>\n>> 2. refs/heads/master is left untouched, for convenience (and \n>> because\n>> it's not really a secret). The implementation is lazy, though, \n>> and\n>> would leave \"refs/heads/master-supersecret\", as well. I can \n>> tighten\n>> that if we really want to be careful.\n>>\n>> 3. gitlinks are left untouched, since sha1s cannot be reversed. \n>> This\n>> could leak some information (if your private repo points to a\n>> public, I can find out you have it as submodule). I doubt it\n>> matters, but we can also scramble the sha1s.\n>\n> Here's a re-roll that addresses the latter two. I don't think any are \n> a\n> big deal, but it's much easier to say \"it's handled\" than try to \n> figure\n> out whether and when it's important.\n>\n> This also includes the documentation update I sent earlier. The\n> interdiff is a bit noisy, as I also converted the anonymize_mem \n> function\n> to take void pointers (since it doesn't know or care what it's \n> storing,\n> and this makes storing unsigned chars for sha1s easier).\n>\n\nJust a bit of bikeshedding for future improvements..\n\nThe .gitignore is another potential user problem area that may benefit \nform not being anonymised when problems strike. For example, there's a \ncurrent problem on the git-users list \nhttps://groups.google.com/forum/#!topic/git-users/JJFIEsI5HRQ about \"git \nclean vs git status re .gitignore\", which would then also beg questions \nabout retaining file extensions/suffixes (.txt, .o, .c, etc).\n\nI've had a similar problem with an over zealous file compare routine \nwhere the same too much vs too little was an issue.\n\nOne thought is that the user should be able to, as an option, select the \nnumber of initial characters retained from filenames, and similarly, the \noption to retain the file extension, and possibly directory names, such \nthat the full .gitignore still works in most cases, and the sort order \nworks (as far as it goes on number of characters).\n\nAll things for future improvers to consider.\n\nPhilip\n\n> -- >8 --\n> Subject: teach fast-export an --anonymize option\n>\n> Sometimes users want to report a bug they experience on\n> their repository, but they are not at liberty to share the\n> contents of the repository. It would be useful if they could\n> produce a repository that has a similar shape to its history\n> and tree, but without leaking any information. This\n> \"anonymized\" repository could then be shared with developers\n> (assuming it still replicates the original problem).\n>\n> This patch implements an \"--anonymize\" option to\n> fast-export, which generates a stream that can recreate such\n> a repository. Producing a single stream makes it easy for\n> the caller to verify that they are not leaking any useful\n> information. You can get an overview of what will be shared\n> by running a command like:\n>\n> git fast-export --anonymize --all |\n> perl -pe 's/\\d+/X/g' |\n> sort -u |\n> less\n>\n> which will show every unique line we generate, modulo any\n> numbers (each anonymized token is assigned a number, like\n> \"User 0\", and we replace it consistently in the output).\n>\n> In addition to anonymizing, this produces test cases that\n> are relatively small (compared to the original repository)\n> and fast to generate (compared to using filter-branch, or\n> modifying the output of fast-export yourself). Here are\n> numbers for git.git:\n>\n> $ time git fast-export --anonymize --all \\\n> --tag-of-filtered-object=drop >output\n> real 0m2.883s\n> user 0m2.828s\n> sys 0m0.052s\n>\n> $ gzip output\n> $ ls -lh output.gz | awk '{print $5}'\n> 2.9M\n>\n> Signed-off-by: Jeff King \n> ---\n[...] \n"},{"id":"248178","messageId":"20140823061932.GB18256@peff.net","threadId":"37387","inReplyTo":"9E69746EFF0048FF84631A9794206566@PhilipOakley","subject":"Re: [PATCH v2] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-23T06:19:32Z","receivedAt":"2014-08-23T06:19:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 22, 2014 at 07:39:59PM +0100, Philip Oakley wrote:\n\n> Just a bit of bikeshedding for future improvements..\n> \n> The .gitignore is another potential user problem area that may benefit form\n> not being anonymised when problems strike.\n\nThanks, I had meant to mention some implications for .gitmodules here,\nbut forgot about .gitignore (and .gitattributes!).\n\nFor any git-specific files like this, we have two challenges:\n\n 1. We've munged their filenames (so .gitignore is probably path123\n now).\n\n 2. We'll have munged their contents. So even if we left the file as\n .gitignore, it will have junk in it.\n\nFixing (1) is pretty easy. I structured all of the anonymizing functions\nto take the old values, even though most of them just throw it away\nentirely (which is a good way to be sure you're not leaking anything!).\nBut we could pass through a few specific ones.\n\nHowever, that doesn't help us if the contents are still munged (in fact\nit's worse, because git will be annoyed that your .gitmodules file\ncontains unparseable crap). So how do we munge those files?\n\nIt depends on the individual file, I think, and what the user wants to\nprotect.\n\nFor .gitignore and .gitattributes, we can translate the pathnames\ncontained in the file. But that doesn't work in the general case,\nbecause the file could have wildcards or other non-literal syntax.\n\nFor .gitmodules, I think it's all-or-nothing. Either the user is OK\nsharing the URLs of their submodules or not (we could munge _just_ the\nURLs, but it's not like the result would be remotely functional).\n\nSo while we might be able to get some things working on the .gitignore\nside, I kind of think the simplest way forward is just adding finer\ngranularity for the user. Let them say \"my filenames are OK to share\nbecause they're part of the problem, but just make sure you hide my\ncommit messages and file contents\".\n\nAnd then if you're not munging filenames, we would turn off .gitignore\nand .gitattributes munging. The implementation is not too hard.\nexport_blob does not have the path of the blob, but we generate the list\nof blobs to export from a diff, so we can feed the path that way. That\ntechnically misses a case where you have a blob at path \"X\", we\nanonymize it, and then you later move it to \".gitignore\", which would\nnot be anonymized. But that is unlikely enough that it is probably not\nworth worrying about.\n\n> For example, there's a current\n> problem on the git-users list\n> https://groups.google.com/forum/#!topic/git-users/JJFIEsI5HRQ about \"git\n> clean vs git status re .gitignore\", which would then also beg questions\n> about retaining file extensions/suffixes (.txt, .o, .c, etc).\n\nYeah, I think retaining extensions would be a reasonable option (and you\nwould probably use it with an option to retain .gitattributes or\n.gitignore whole if you were confident that those files did not have\nanything private and just used extension wildcards).\n\n> One thought is that the user should be able to, as an option, select the\n> number of initial characters retained from filenames, and similarly, the\n> option to retain the file extension, and possibly directory names, such that\n> the full .gitignore still works in most cases, and the sort order works (as\n> far as it goes on number of characters).\n\nYeah, those all seem reasonable.\n\n> All things for future improvers to consider.\n\nAgreed. I wanted to go through your list not because I want to implement\nany of those things right now, but because I wanted to make sure that\nthere was nothing in my approach that would preclude us from building\nthose things later. And I don't think there is (and I'd be happy if\nsomebody else felt like building them on top, now or later).\n\n-Peff\n"},{"id":"248406","messageId":"xmqq8um9gbwh.fsf@gitster.dls.corp.google.com","threadId":"37387","inReplyTo":"20140821232100.GA27849@peff.net","subject":"Re: [PATCH v2] teach fast-export an --anonymize option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-08-27T16:01:02Z","receivedAt":"2014-08-27T16:01:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> diff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\n> new file mode 100755\n> index 0000000..f76ffe4\n> --- /dev/null\n> +++ b/t/t9351-fast-export-anonymize.sh\n> @@ -0,0 +1,117 @@\n> +#!/bin/sh\n> +\n> +test_description='basic tests for fast-export --anonymize'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup simple repo' '\n> +\ttest_commit base &&\n> +\ttest_commit foo &&\n> +\tgit checkout -b other HEAD^ &&\n> +\tmkdir subdir &&\n> +\ttest_commit subdir/bar &&\n> +\ttest_commit subdir/xyzzy &&\n> +\tgit tag -m \"annotated tag\" mytag\n> +'\n> +\n> +test_expect_success 'export anonymized stream' '\n> +\tgit fast-export --anonymize --all >stream\n> +'\n> +\n> +# this also covers commit messages\n> +test_expect_success 'stream omits path names' '\n> +\t! fgrep base stream &&\n> +\t! fgrep foo stream &&\n> +\t! fgrep subdir stream &&\n> +\t! fgrep bar stream &&\n> +\t! fgrep xyzzy stream\n> +'\n\nI know there are a few isolated places that already use \"fgrep\", but\nlet's not spread the disease. Neither \"fgrep\" nor \"egrep\" appears in\nPOSIX and they can easily be spelled more portably as \"grep -F\" and\n\"grep -E\", respectively.\n\n> +test_expect_success 'stream allows master as refname' '\n> +\tfgrep master stream\n> +'\n> +\n> +test_expect_success 'stream omits other refnames' '\n> +\t! fgrep other stream\n> +'\n\nWhat should happen to mytag?\n\n> +\n> +test_expect_success 'stream omits identities' '\n> +\t! fgrep \"$GIT_COMMITTER_NAME\" stream &&\n> +\t! fgrep \"$GIT_COMMITTER_EMAIL\" stream &&\n> +\t! fgrep \"$GIT_AUTHOR_NAME\" stream &&\n> +\t! fgrep \"$GIT_AUTHOR_EMAIL\" stream\n> +'\n> +\n> +test_expect_success 'stream omits tag message' '\n> +\t! fgrep \"annotated tag\" stream\n> +'\n> +\n> +# NOTE: we chdir to the new, anonymized repository\n> +# after this. All further tests should assume this.\n> +test_expect_success 'import stream to new repository' '\n> +\tgit init new &&\n> +\tcd new &&\n> +\tgit fast-import <../stream\n> +'\n> +\n> +test_expect_success 'result has two branches' '\n> +\tgit for-each-ref --format=\"%(refname)\" refs/heads >branches &&\n> +\ttest_line_count = 2 branches &&\n> +\tother_branch=$(grep -v refs/heads/master branches)\n> +'\n> +\n> +test_expect_success 'repo has original shape' '\n> +\tcat >expect <<-\\EOF &&\n> +\t> subject 3\n> +\t> subject 2\n> +\t< subject 1\n> +\t- subject 0\n> +\tEOF\n> +\tgit log --format=\"%m %s\" --left-right --boundary \\\n> +\t\tmaster...$other_branch >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nYuck and Hmph. Doing a shape-preserving conversion is very\nimportant, but I wonder if we can we verify without having to cast a\nparticular rewrite rule in stone. We know we want to preserve\nrelative order of committer timestamps (to reproduce bugs that\ndepend on the traversal order), and it may be OK to reuse the\nexactly the same committer timestamps from the original, in which\ncase we can make sure that we create the original history with\nappropriate \"test_tick\"s (I think test_commit does that for us) and\nuse \"%ct\" instead of \"%s\" here, perhaps? That way we can later\nchange the rewrite rules of commit object payload without having to\nadjust this test.\n\n> +\n> +test_expect_success 'root tree has original shape' '\n> +\tcat >expect <<-\\EOF &&\n> +\tblob\n> +\ttree\n> +\tEOF\n> +\tgit ls-tree $other_branch >root &&\n> +\tcut -d\" \" -f2 actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'paths in subdir ended up in one tree' '\n> +\tcat >expect <<-\\EOF &&\n> +\tblob\n> +\tblob\n> +\tEOF\n> +\ttree=$(grep tree root | cut -f2) &&\n> +\tgit ls-tree $other_branch:$tree >tree &&\n> +\tcut -d\" \" -f2 actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'tag points to branch tip' '\n> +\tgit rev-parse $other_branch >expect &&\n> +\tgit for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nI notice you haven't checked how many tags you have in the\nrepository, unlike the number of branches which you counted\nearlier.\n\n> +test_expect_success 'idents are shared' '\n> +\tgit log --all --format=\"%an <%ae>\" >authors &&\n> +\tsort -u authors >unique &&\n> +\ttest_line_count = 1 unique &&\n> +\tgit log --all --format=\"%cn <%ce>\" >committers &&\n> +\tsort -u committers >unique &&\n> +\ttest_line_count = 1 unique &&\n> +\t! test_cmp authors committers\n> +'\n\nTwo commits by the same author must convert to two commits by the\nsame anonymized author, but that is not tested here; the history\nmade in 'setup a simple repo' step is a bit too simple to do that\nanyway, though ;-).\n\n> +test_expect_success 'commit timestamps are retained' '\n> +\tgit log --all --format=\"%ct\" >timestamps &&\n> +\tsort -u timestamps >unique &&\n> +\ttest_line_count = 4 unique\n> +'\n> +\n> +test_done\n"},{"id":"248413","messageId":"20140827165854.GC1432@peff.net","threadId":"37387","inReplyTo":"xmqq8um9gbwh.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v2] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-27T16:58:54Z","receivedAt":"2014-08-27T16:58:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Aug 27, 2014 at 09:01:02AM -0700, Junio C Hamano wrote:\n\n> > +# this also covers commit messages\n> > +test_expect_success 'stream omits path names' '\n> > +\t! fgrep base stream &&\n> > +\t! fgrep foo stream &&\n> > +\t! fgrep subdir stream &&\n> > +\t! fgrep bar stream &&\n> > +\t! fgrep xyzzy stream\n> > +'\n> \n> I know there are a few isolated places that already use \"fgrep\", but\n> let's not spread the disease. Neither \"fgrep\" nor \"egrep\" appears in\n> POSIX and they can easily be spelled more portably as \"grep -F\" and\n> \"grep -E\", respectively.s\n\nI actually specifically used \"fgrep\" rather than \"grep -F\" because some\nplatforms have the former but not the latter. It has been a while since\ncommit 8753941, but I think Solaris was such a platform (and maybe AIX,\ntoo, based on [1]).\n\n[1] http://article.gmane.org/gmane.comp.version-control.git/97076\n\nThey could actually just be \"grep\" in this case, since we know the\npatterns do not have any metacharacters. I was thinking that our\n$GIT_COMMITTER_NAME (grepped below) did, but it is \"C O Mitter\", not \"C.\nO. Mitter\".\n\nI'll switch that in the re-roll.\n\n> > +test_expect_success 'stream allows master as refname' '\n> > +\tfgrep master stream\n> > +'\n> > +\n> > +test_expect_success 'stream omits other refnames' '\n> > +\t! fgrep other stream\n> > +'\n> \n> What should happen to mytag?\n\nI added the tag test at the very end, and you can see that it did not\nget as much attention. :) We should check \"! grep mytag stream\" here.\nWill be in the re-roll.\n\n> > +test_expect_success 'repo has original shape' '\n> > +\tcat >expect <<-\\EOF &&\n> > +\t> subject 3\n> > +\t> subject 2\n> > +\t< subject 1\n> > +\t- subject 0\n> > +\tEOF\n> > +\tgit log --format=\"%m %s\" --left-right --boundary \\\n> > +\t\tmaster...$other_branch >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> \n> Yuck and Hmph. Doing a shape-preserving conversion is very\n> important, but I wonder if we can we verify without having to cast a\n> particular rewrite rule in stone. We know we want to preserve\n> relative order of committer timestamps (to reproduce bugs that\n> depend on the traversal order), and it may be OK to reuse the\n> exactly the same committer timestamps from the original, in which\n> case we can make sure that we create the original history with\n> appropriate \"test_tick\"s (I think test_commit does that for us) and\n> use \"%ct\" instead of \"%s\" here, perhaps? That way we can later\n> change the rewrite rules of commit object payload without having to\n> adjust this test.\n\nYeah, everything is lost except the shape and committer timestamps. So\nour choices are basically \"%m\" or \"%m %ct\". I think the latter is\nprobably the least bad choice. Will switch.\n\nThere's a potential problem with picking the same branches in the same\norder. We can cheat a little here, though, because \"master\" retains its\nsame name. Since there's only one other branch, it's always the other\none we want.\n\nThe trees are not necessarily so lucky. We check that the root tree ends\nup with a blob and a tree. But the anonymization does not necessarily\nhave to preserve their order (and it probably wouldn't under many\nschemes). I think we can get away with just sorting the type list.\n\n> > +test_expect_success 'tag points to branch tip' '\n> > +\tgit rev-parse $other_branch >expect &&\n> > +\tgit for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n> > +\ttest_cmp expect actual\n> > +'\n> \n> I notice you haven't checked how many tags you have in the\n> repository, unlike the number of branches which you counted\n> earlier.\n\nYes, because test_commit makes a bunch of extraneous tags, which I\nthought made the test a little brittle.\n\n> > +test_expect_success 'idents are shared' '\n> > +\tgit log --all --format=\"%an <%ae>\" >authors &&\n> > +\tsort -u authors >unique &&\n> > +\ttest_line_count = 1 unique &&\n> > +\tgit log --all --format=\"%cn <%ce>\" >committers &&\n> > +\tsort -u committers >unique &&\n> > +\ttest_line_count = 1 unique &&\n> > +\t! test_cmp authors committers\n> > +'\n> \n> Two commits by the same author must convert to two commits by the\n> same anonymized author, but that is not tested here; the history\n> made in 'setup a simple repo' step is a bit too simple to do that\n> anyway, though ;-).\n\nI think we do check that. The commits are all by the same author, so the\nfact that \"sort -u authors\" has one line means that they all got\nanonymized to the same author. What we don't check is that two\n_different_ authors get different idents. I check that the two idents\nget different values (authors != committers), but it is not clear from a\nblackbox test that it is because the anonymization is working (it might\nbe because authors and committers come from a different pool of strings,\nthough I know having written the code that that is not the case).\n\nI also do not test that the same ident as an author and as a committer\nends up the same. Honestly, I didn't really feel that it was worth much\nbother. The shape of history and the tree is the most interesting thing\nhere, and the primary thing about idents is that we wipe them.\n\nMaybe it was just me being lazy, though.\n\n> > +test_expect_success 'commit timestamps are retained' '\n> > +\tgit log --all --format=\"%ct\" >timestamps &&\n> > +\tsort -u timestamps >unique &&\n> > +\ttest_line_count = 4 unique\n> > +'\n\nI think we can drop this one if we check %ct in the graph-shape test\nabove. It's redundant.\n\nI'll send v3 in a minute. Here's the interdiff.\n\ndiff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\nindex f76ffe4..897dc50 100755\n--- a/t/t9351-fast-export-anonymize.sh\n+++ b/t/t9351-fast-export-anonymize.sh\n@@ -19,30 +19,31 @@ test_expect_success 'export anonymized stream' '\n \n # this also covers commit messages\n test_expect_success 'stream omits path names' '\n-\t! fgrep base stream &&\n-\t! fgrep foo stream &&\n-\t! fgrep subdir stream &&\n-\t! fgrep bar stream &&\n-\t! fgrep xyzzy stream\n+\t! grep base stream &&\n+\t! grep foo stream &&\n+\t! grep subdir stream &&\n+\t! grep bar stream &&\n+\t! grep xyzzy stream\n '\n \n test_expect_success 'stream allows master as refname' '\n-\tfgrep master stream\n+\tgrep master stream\n '\n \n test_expect_success 'stream omits other refnames' '\n-\t! fgrep other stream\n+\t! grep other stream &&\n+\t! grep mytag stream\n '\n \n test_expect_success 'stream omits identities' '\n-\t! fgrep \"$GIT_COMMITTER_NAME\" stream &&\n-\t! fgrep \"$GIT_COMMITTER_EMAIL\" stream &&\n-\t! fgrep \"$GIT_AUTHOR_NAME\" stream &&\n-\t! fgrep \"$GIT_AUTHOR_EMAIL\" stream\n+\t! grep \"$GIT_COMMITTER_NAME\" stream &&\n+\t! grep \"$GIT_COMMITTER_EMAIL\" stream &&\n+\t! grep \"$GIT_AUTHOR_NAME\" stream &&\n+\t! grep \"$GIT_AUTHOR_EMAIL\" stream\n '\n \n test_expect_success 'stream omits tag message' '\n-\t! fgrep \"annotated tag\" stream\n+\t! grep \"annotated tag\" stream\n '\n \n # NOTE: we chdir to the new, anonymized repository\n@@ -59,25 +60,25 @@ test_expect_success 'result has two branches' '\n \tother_branch=$(grep -v refs/heads/master branches)\n '\n \n-test_expect_success 'repo has original shape' '\n-\tcat >expect <<-\\EOF &&\n-\t> subject 3\n-\t> subject 2\n-\t< subject 1\n-\t- subject 0\n-\tEOF\n-\tgit log --format=\"%m %s\" --left-right --boundary \\\n-\t\tmaster...$other_branch >actual &&\n+test_expect_success 'repo has original shape and timestamps' '\n+\tshape () {\n+\t\tgit log --format=\"%m %ct\" --left-right --boundary \"$@\"\n+\t} &&\n+\t(cd .. && shape master...other) >expect &&\n+\tshape master...$other_branch >actual &&\n \ttest_cmp expect actual\n '\n \n test_expect_success 'root tree has original shape' '\n+\t# the output entries are not necessarily in the same\n+\t# order, but we know at least that we will have one tree\n+\t# and one blob, so just check the sorted order\n \tcat >expect <<-\\EOF &&\n \tblob\n \ttree\n \tEOF\n \tgit ls-tree $other_branch >root &&\n-\tcut -d\" \" -f2 actual &&\n+\tcut -d\" \" -f2 actual &&\n \ttest_cmp expect actual\n '\n \n@@ -108,10 +109,4 @@ test_expect_success 'idents are shared' '\n \t! test_cmp authors committers\n '\n \n-test_expect_success 'commit timestamps are retained' '\n-\tgit log --all --format=\"%ct\" >timestamps &&\n-\tsort -u timestamps >unique &&\n-\ttest_line_count = 4 unique\n-'\n-\n test_done\n"},{"id":"248414","messageId":"20140827170127.GA6138@peff.net","threadId":"37387","inReplyTo":"20140827165854.GC1432@peff.net","subject":"[PATCH v3] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-27T17:01:28Z","receivedAt":"2014-08-27T17:01:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"Sometimes users want to report a bug they experience on\ntheir repository, but they are not at liberty to share the\ncontents of the repository. It would be useful if they could\nproduce a repository that has a similar shape to its history\nand tree, but without leaking any information. This\n\"anonymized\" repository could then be shared with developers\n(assuming it still replicates the original problem).\n\nThis patch implements an \"--anonymize\" option to\nfast-export, which generates a stream that can recreate such\na repository. Producing a single stream makes it easy for\nthe caller to verify that they are not leaking any useful\ninformation. You can get an overview of what will be shared\nby running a command like:\n\n git fast-export --anonymize --all |\n perl -pe 's/\\d+/X/g' |\n sort -u |\n less\n\nwhich will show every unique line we generate, modulo any\nnumbers (each anonymized token is assigned a number, like\n\"User 0\", and we replace it consistently in the output).\n\nIn addition to anonymizing, this produces test cases that\nare relatively small (compared to the original repository)\nand fast to generate (compared to using filter-branch, or\nmodifying the output of fast-export yourself). Here are\nnumbers for git.git:\n\n $ time git fast-export --anonymize --all \\\n --tag-of-filtered-object=drop >output\n real 0m2.883s\n user 0m2.828s\n sys 0m0.052s\n\n $ gzip output\n $ ls -lh output.gz | awk '{print $5}'\n 2.9M\n\nSigned-off-by: Jeff King \n---\nThis has all of the test tweaks I responded about. The only thing I\ndidn't do is beef up the ident tests, but I'm not really sure it's worth\nthe trouble.\n\n Documentation/git-fast-export.txt | 6 +\n builtin/fast-export.c | 300 ++++++++++++++++++++++++++++++++++++--\n t/t9351-fast-export-anonymize.sh | 112 ++++++++++++++\n 3 files changed, 407 insertions(+), 11 deletions(-)\n create mode 100755 t/t9351-fast-export-anonymize.sh\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 221506b..52831fa 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -105,6 +105,12 @@ marks the same across runs.\n \tin the commit (as opposed to just listing the files which are\n \tdifferent from the commit's first parent).\n \n+--anonymize::\n+\tReplace all refnames, paths, blob contents, commit and tag\n+\tmessages, names, and email addresses in the output with\n+\tanonymized data, while still retaining the shape of history and\n+\tof the stored tree.\n+\n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n \tbe specified.\ndiff --git a/builtin/fast-export.c b/builtin/fast-export.c\nindex 92b4624..b8182c2 100644\n--- a/builtin/fast-export.c\n+++ b/builtin/fast-export.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"quote.h\"\n #include \"remote.h\"\n+#include \"blob.h\"\n \n static const char *fast_export_usage[] = {\n \tN_(\"git fast-export [rev-list-opts]\"),\n@@ -34,6 +35,7 @@ static int full_tree;\n static struct string_list extra_refs = STRING_LIST_INIT_NODUP;\n static struct refspec *refspecs;\n static int refspecs_nr;\n+static int anonymize;\n \n static int parse_opt_signed_tag_mode(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -81,6 +83,76 @@ static int has_unshown_parent(struct commit *commit)\n \treturn 0;\n }\n \n+struct anonymized_entry {\n+\tstruct hashmap_entry hash;\n+\tconst char *orig;\n+\tsize_t orig_len;\n+\tconst char *anon;\n+\tsize_t anon_len;\n+};\n+\n+static int anonymized_entry_cmp(const void *va, const void *vb,\n+\t\t\t\tconst void *data)\n+{\n+\tconst struct anonymized_entry *a = va, *b = vb;\n+\treturn a->orig_len != b->orig_len ||\n+\t\tmemcmp(a->orig, b->orig, a->orig_len);\n+}\n+\n+/*\n+ * Basically keep a cache of X->Y so that we can repeatedly replace\n+ * the same anonymized string with another. The actual generation\n+ * is farmed out to the generate function.\n+ */\n+static const void *anonymize_mem(struct hashmap *map,\n+\t\t\t\t void *(*generate)(const void *, size_t *),\n+\t\t\t\t const void *orig, size_t *len)\n+{\n+\tstruct anonymized_entry key, *ret;\n+\n+\tif (!map->cmpfn)\n+\t\thashmap_init(map, anonymized_entry_cmp, 0);\n+\n+\thashmap_entry_init(&key, memhash(orig, *len));\n+\tkey.orig = orig;\n+\tkey.orig_len = *len;\n+\tret = hashmap_get(map, &key, NULL);\n+\n+\tif (!ret) {\n+\t\tret = xmalloc(sizeof(*ret));\n+\t\thashmap_entry_init(&ret->hash, key.hash.hash);\n+\t\tret->orig = xstrdup(orig);\n+\t\tret->orig_len = *len;\n+\t\tret->anon = generate(orig, len);\n+\t\tret->anon_len = *len;\n+\t\thashmap_put(map, ret);\n+\t}\n+\n+\t*len = ret->anon_len;\n+\treturn ret->anon;\n+}\n+\n+/*\n+ * We anonymize each component of a path individually,\n+ * so that paths a/b and a/c will share a common root.\n+ * The paths are cached via anonymize_mem so that repeated\n+ * lookups for \"a\" will yield the same value.\n+ */\n+static void anonymize_path(struct strbuf *out, const char *path,\n+\t\t\t struct hashmap *map,\n+\t\t\t void *(*generate)(const void *, size_t *))\n+{\n+\twhile (*path) {\n+\t\tconst char *end_of_component = strchrnul(path, '/');\n+\t\tsize_t len = end_of_component - path;\n+\t\tconst char *c = anonymize_mem(map, generate, path, &len);\n+\t\tstrbuf_add(out, c, len);\n+\t\tpath = end_of_component;\n+\t\tif (*path)\n+\t\t\tstrbuf_addch(out, *path++);\n+\t}\n+}\n+\n /* Since intptr_t is C99, we do not use it here */\n static inline uint32_t *mark_to_ptr(uint32_t mark)\n {\n@@ -119,6 +191,26 @@ static void show_progress(void)\n \t\tprintf(\"progress %d objects\\n\", counter);\n }\n \n+/*\n+ * Ideally we would want some transformation of the blob data here\n+ * that is unreversible, but would still be the same size and have\n+ * the same data relationship to other blobs (so that we get the same\n+ * delta and packing behavior as the original). But the first and last\n+ * requirements there are probably mutually exclusive, so let's take\n+ * the easy way out for now, and just generate arbitrary content.\n+ *\n+ * There's no need to cache this result with anonymize_mem, since\n+ * we already handle blob content caching with marks.\n+ */\n+static char *anonymize_blob(unsigned long *size)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"anonymous blob %d\", counter++);\n+\t*size = out.len;\n+\treturn strbuf_detach(&out, NULL);\n+}\n+\n static void export_blob(const unsigned char *sha1)\n {\n \tunsigned long size;\n@@ -137,12 +229,19 @@ static void export_blob(const unsigned char *sha1)\n \tif (object && object->flags & SHOWN)\n \t\treturn;\n \n-\tbuf = read_sha1_file(sha1, &type, &size);\n-\tif (!buf)\n-\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n-\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n-\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n-\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\tif (anonymize) {\n+\t\tbuf = anonymize_blob(&size);\n+\t\tobject = (struct object *)lookup_blob(sha1);\n+\t\teaten = 0;\n+\t} else {\n+\t\tbuf = read_sha1_file(sha1, &type, &size);\n+\t\tif (!buf)\n+\t\t\tdie (\"Could not read blob %s\", sha1_to_hex(sha1));\n+\t\tif (check_sha1_signature(sha1, buf, size, typename(type)) < 0)\n+\t\t\tdie(\"sha1 mismatch in blob %s\", sha1_to_hex(sha1));\n+\t\tobject = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\t}\n+\n \tif (!object)\n \t\tdie(\"Could not read blob %s\", sha1_to_hex(sha1));\n \n@@ -190,7 +289,7 @@ static int depth_first(const void *a_, const void *b_)\n \treturn (a->status == 'R') - (b->status == 'R');\n }\n \n-static void print_path(const char *path)\n+static void print_path_1(const char *path)\n {\n \tint need_quote = quote_c_style(path, NULL, NULL, 0);\n \tif (need_quote)\n@@ -201,6 +300,43 @@ static void print_path(const char *path)\n \t\tprintf(\"%s\", path);\n }\n \n+static void *anonymize_path_component(const void *path, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"path%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static void print_path(const char *path)\n+{\n+\tif (!anonymize)\n+\t\tprint_path_1(path);\n+\telse {\n+\t\tstatic struct hashmap paths;\n+\t\tstatic struct strbuf anon = STRBUF_INIT;\n+\n+\t\tanonymize_path(&anon, path, &paths, anonymize_path_component);\n+\t\tprint_path_1(anon.buf);\n+\t\tstrbuf_reset(&anon);\n+\t}\n+}\n+\n+static void *generate_fake_sha1(const void *old, size_t *len)\n+{\n+\tstatic uint32_t counter = 1; /* avoid null sha1 */\n+\tunsigned char *out = xcalloc(20, 1);\n+\tput_be32(out + 16, counter++);\n+\treturn out;\n+}\n+\n+static const unsigned char *anonymize_sha1(const unsigned char *sha1)\n+{\n+\tstatic struct hashmap sha1s;\n+\tsize_t len = 20;\n+\treturn anonymize_mem(&sha1s, generate_fake_sha1, sha1, &len);\n+}\n+\n static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t struct diff_options *options, void *data)\n {\n@@ -245,7 +381,9 @@ static void show_filemodify(struct diff_queue_struct *q,\n \t\t\t */\n \t\t\tif (no_data || S_ISGITLINK(spec->mode))\n \t\t\t\tprintf(\"M %06o %s \", spec->mode,\n-\t\t\t\t sha1_to_hex(spec->sha1));\n+\t\t\t\t sha1_to_hex(anonymize ?\n+\t\t\t\t\t\t anonymize_sha1(spec->sha1) :\n+\t\t\t\t\t\t spec->sha1));\n \t\t\telse {\n \t\t\t\tstruct object *object = lookup_object(spec->sha1);\n \t\t\t\tprintf(\"M %06o :%d \", spec->mode,\n@@ -279,6 +417,114 @@ static const char *find_encoding(const char *begin, const char *end)\n \treturn bol;\n }\n \n+static void *anonymize_ref_component(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"ref%d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+static const char *anonymize_refname(const char *refname)\n+{\n+\t/*\n+\t * If any of these prefixes is found, we will leave it intact\n+\t * so that tags remain tags and so forth.\n+\t */\n+\tstatic const char *prefixes[] = {\n+\t\t\"refs/heads/\",\n+\t\t\"refs/tags/\",\n+\t\t\"refs/remotes/\",\n+\t\t\"refs/\"\n+\t};\n+\tstatic struct hashmap refs;\n+\tstatic struct strbuf anon = STRBUF_INIT;\n+\tint i;\n+\n+\t/*\n+\t * We also leave \"master\" as a special case, since it does not reveal\n+\t * anything interesting.\n+\t */\n+\tif (!strcmp(refname, \"refs/heads/master\"))\n+\t\treturn refname;\n+\n+\tstrbuf_reset(&anon);\n+\tfor (i = 0; i < ARRAY_SIZE(prefixes); i++) {\n+\t\tif (skip_prefix(refname, prefixes[i], &refname)) {\n+\t\t\tstrbuf_addstr(&anon, prefixes[i]);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tanonymize_path(&anon, refname, &refs, anonymize_ref_component);\n+\treturn anon.buf;\n+}\n+\n+/*\n+ * We do not even bother to cache commit messages, as they are unlikely\n+ * to be repeated verbatim, and it is not that interesting when they are.\n+ */\n+static char *anonymize_commit_message(const char *old)\n+{\n+\tstatic int counter;\n+\treturn xstrfmt(\"subject %d\\n\\nbody\\n\", counter++);\n+}\n+\n+static struct hashmap idents;\n+static void *anonymize_ident(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"User %d \", counter, counter);\n+\tcounter++;\n+\treturn strbuf_detach(&out, len);\n+}\n+\n+/*\n+ * Our strategy here is to anonymize the names and email addresses,\n+ * but keep timestamps intact, as they influence things like traversal\n+ * order (and by themselves should not be too revealing).\n+ */\n+static void anonymize_ident_line(const char **beg, const char **end)\n+{\n+\tstatic struct strbuf buffers[] = { STRBUF_INIT, STRBUF_INIT };\n+\tstatic unsigned which_buffer;\n+\n+\tstruct strbuf *out;\n+\tstruct ident_split split;\n+\tconst char *end_of_header;\n+\n+\tout = &buffers[which_buffer++];\n+\twhich_buffer %= ARRAY_SIZE(buffers);\n+\tstrbuf_reset(out);\n+\n+\t/* skip \"committer\", \"author\", \"tagger\", etc */\n+\tend_of_header = strchr(*beg, ' ');\n+\tif (!end_of_header)\n+\t\tdie(\"BUG: malformed line fed to anonymize_ident_line: %.*s\",\n+\t\t (int)(*end - *beg), *beg);\n+\tend_of_header++;\n+\tstrbuf_add(out, *beg, end_of_header - *beg);\n+\n+\tif (!split_ident_line(&split, end_of_header, *end - end_of_header) &&\n+\t split.date_begin) {\n+\t\tconst char *ident;\n+\t\tsize_t len;\n+\n+\t\tlen = split.mail_end - split.name_begin;\n+\t\tident = anonymize_mem(&idents, anonymize_ident,\n+\t\t\t\t split.name_begin, &len);\n+\t\tstrbuf_add(out, ident, len);\n+\t\tstrbuf_addch(out, ' ');\n+\t\tstrbuf_add(out, split.date_begin, split.tz_end - split.date_begin);\n+\t} else {\n+\t\tstrbuf_addstr(out, \"Malformed Ident 0 -0000\");\n+\t}\n+\n+\t*beg = out->buf;\n+\t*end = out->buf + out->len;\n+}\n+\n static void handle_commit(struct commit *commit, struct rev_info *rev)\n {\n \tint saved_output_format = rev->diffopt.output_format;\n@@ -287,6 +533,7 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tconst char *encoding, *message;\n \tchar *reencoded = NULL;\n \tstruct commit_list *p;\n+\tconst char *refname;\n \tint i;\n \n \trev->diffopt.output_format = DIFF_FORMAT_CALLBACK;\n@@ -326,13 +573,22 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \t\tif (!S_ISGITLINK(diff_queued_diff.queue[i]->two->mode))\n \t\t\texport_blob(diff_queued_diff.queue[i]->two->sha1);\n \n+\trefname = commit->util;\n+\tif (anonymize) {\n+\t\trefname = anonymize_refname(refname);\n+\t\tanonymize_ident_line(&committer, &committer_end);\n+\t\tanonymize_ident_line(&author, &author_end);\n+\t}\n+\n \tmark_next_object(&commit->object);\n-\tif (!is_encoding_utf8(encoding))\n+\tif (anonymize)\n+\t\treencoded = anonymize_commit_message(message);\n+\telse if (!is_encoding_utf8(encoding))\n \t\treencoded = reencode_string(message, \"UTF-8\", encoding);\n \tif (!commit->parents)\n-\t\tprintf(\"reset %s\\n\", (const char*)commit->util);\n+\t\tprintf(\"reset %s\\n\", refname);\n \tprintf(\"commit %s\\nmark :%\"PRIu32\"\\n%.*s\\n%.*s\\ndata %u\\n%s\",\n-\t (const char *)commit->util, last_idnum,\n+\t refname, last_idnum,\n \t (int)(author_end - author), author,\n \t (int)(committer_end - committer), committer,\n \t (unsigned)(reencoded\n@@ -363,6 +619,14 @@ static void handle_commit(struct commit *commit, struct rev_info *rev)\n \tshow_progress();\n }\n \n+static void *anonymize_tag(const void *old, size_t *len)\n+{\n+\tstatic int counter;\n+\tstruct strbuf out = STRBUF_INIT;\n+\tstrbuf_addf(&out, \"tag message %d\", counter++);\n+\treturn strbuf_detach(&out, len);\n+}\n+\n static void handle_tail(struct object_array *commits, struct rev_info *revs)\n {\n \tstruct commit *commit;\n@@ -419,6 +683,17 @@ static void handle_tag(const char *name, struct tag *tag)\n \t} else {\n \t\ttagger++;\n \t\ttagger_end = strchrnul(tagger, '\\n');\n+\t\tif (anonymize)\n+\t\t\tanonymize_ident_line(&tagger, &tagger_end);\n+\t}\n+\n+\tif (anonymize) {\n+\t\tname = anonymize_refname(name);\n+\t\tif (message) {\n+\t\t\tstatic struct hashmap tags;\n+\t\t\tmessage = anonymize_mem(&tags, anonymize_tag,\n+\t\t\t\t\t\tmessage, &message_size);\n+\t\t}\n \t}\n \n \t/* handle signed tags */\n@@ -584,6 +859,8 @@ static void handle_tags_and_duplicates(void)\n \t\t\thandle_tag(name, (struct tag *)object);\n \t\t\tbreak;\n \t\tcase OBJ_COMMIT:\n+\t\t\tif (anonymize)\n+\t\t\t\tname = anonymize_refname(name);\n \t\t\t/* create refs pointing to already seen commits */\n \t\t\tcommit = (struct commit *)object;\n \t\t\tprintf(\"reset %s\\nfrom :%d\\n\\n\", name,\n@@ -719,6 +996,7 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"no-data\", &no_data, N_(\"Skip output of blob data\")),\n \t\tOPT_STRING_LIST(0, \"refspec\", &refspecs_list, N_(\"refspec\"),\n \t\t\t N_(\"Apply refspec to exported refs\")),\n+\t\tOPT_BOOL(0, \"anonymize\", &anonymize, N_(\"anonymize output\")),\n \t\tOPT_END()\n \t};\n \ndiff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh\nnew file mode 100755\nindex 0000000..897dc50\n--- /dev/null\n+++ b/t/t9351-fast-export-anonymize.sh\n@@ -0,0 +1,112 @@\n+#!/bin/sh\n+\n+test_description='basic tests for fast-export --anonymize'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup simple repo' '\n+\ttest_commit base &&\n+\ttest_commit foo &&\n+\tgit checkout -b other HEAD^ &&\n+\tmkdir subdir &&\n+\ttest_commit subdir/bar &&\n+\ttest_commit subdir/xyzzy &&\n+\tgit tag -m \"annotated tag\" mytag\n+'\n+\n+test_expect_success 'export anonymized stream' '\n+\tgit fast-export --anonymize --all >stream\n+'\n+\n+# this also covers commit messages\n+test_expect_success 'stream omits path names' '\n+\t! grep base stream &&\n+\t! grep foo stream &&\n+\t! grep subdir stream &&\n+\t! grep bar stream &&\n+\t! grep xyzzy stream\n+'\n+\n+test_expect_success 'stream allows master as refname' '\n+\tgrep master stream\n+'\n+\n+test_expect_success 'stream omits other refnames' '\n+\t! grep other stream &&\n+\t! grep mytag stream\n+'\n+\n+test_expect_success 'stream omits identities' '\n+\t! grep \"$GIT_COMMITTER_NAME\" stream &&\n+\t! grep \"$GIT_COMMITTER_EMAIL\" stream &&\n+\t! grep \"$GIT_AUTHOR_NAME\" stream &&\n+\t! grep \"$GIT_AUTHOR_EMAIL\" stream\n+'\n+\n+test_expect_success 'stream omits tag message' '\n+\t! grep \"annotated tag\" stream\n+'\n+\n+# NOTE: we chdir to the new, anonymized repository\n+# after this. All further tests should assume this.\n+test_expect_success 'import stream to new repository' '\n+\tgit init new &&\n+\tcd new &&\n+\tgit fast-import <../stream\n+'\n+\n+test_expect_success 'result has two branches' '\n+\tgit for-each-ref --format=\"%(refname)\" refs/heads >branches &&\n+\ttest_line_count = 2 branches &&\n+\tother_branch=$(grep -v refs/heads/master branches)\n+'\n+\n+test_expect_success 'repo has original shape and timestamps' '\n+\tshape () {\n+\t\tgit log --format=\"%m %ct\" --left-right --boundary \"$@\"\n+\t} &&\n+\t(cd .. && shape master...other) >expect &&\n+\tshape master...$other_branch >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'root tree has original shape' '\n+\t# the output entries are not necessarily in the same\n+\t# order, but we know at least that we will have one tree\n+\t# and one blob, so just check the sorted order\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\ttree\n+\tEOF\n+\tgit ls-tree $other_branch >root &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'paths in subdir ended up in one tree' '\n+\tcat >expect <<-\\EOF &&\n+\tblob\n+\tblob\n+\tEOF\n+\ttree=$(grep tree root | cut -f2) &&\n+\tgit ls-tree $other_branch:$tree >tree &&\n+\tcut -d\" \" -f2 actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'tag points to branch tip' '\n+\tgit rev-parse $other_branch >expect &&\n+\tgit for-each-ref --format=\"%(*objectname)\" | grep . >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'idents are shared' '\n+\tgit log --all --format=\"%an <%ae>\" >authors &&\n+\tsort -u authors >unique &&\n+\ttest_line_count = 1 unique &&\n+\tgit log --all --format=\"%cn <%ce>\" >committers &&\n+\tsort -u committers >unique &&\n+\ttest_line_count = 1 unique &&\n+\t! test_cmp authors committers\n+'\n+\n+test_done\n-- \n2.1.0.346.ga0367b9\n"},{"id":"248477","messageId":"CACsJy8B3gFC7kLf-cLhAk3BgQ+v427rMXWHTqjU4LYP3NQte7Q@mail.gmail.com","threadId":"37387","inReplyTo":"20140827170127.GA6138@peff.net","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-08-28T10:30:44Z","receivedAt":"2014-08-28T10:30:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 28, 2014 at 12:01 AM, Jeff King wrote:\n> You can get an overview of what will be shared\n> by running a command like:\n>\n> git fast-export --anonymize --all |\n> perl -pe 's/\\d+/X/g' |\n> sort -u |\n> less\n>\n> which will show every unique line we generate, modulo any\n> numbers (each anonymized token is assigned a number, like\n> \"User 0\", and we replace it consistently in the output).\n\nI feel like this should be part of git-fast-export.txt, just to\nincrease the user's confidence in the tool (and I don't expect most\nusers to read this commit message).\n-- \nDuy\n"},{"id":"248480","messageId":"20140828123257.GA18642@peff.net","threadId":"37387","inReplyTo":"CACsJy8B3gFC7kLf-cLhAk3BgQ+v427rMXWHTqjU4LYP3NQte7Q@mail.gmail.com","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-28T12:32:58Z","receivedAt":"2014-08-28T12:32:58Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 28, 2014 at 05:30:44PM +0700, Duy Nguyen wrote:\n\n> On Thu, Aug 28, 2014 at 12:01 AM, Jeff King wrote:\n> > You can get an overview of what will be shared\n> > by running a command like:\n> >\n> > git fast-export --anonymize --all |\n> > perl -pe 's/\\d+/X/g' |\n> > sort -u |\n> > less\n> >\n> > which will show every unique line we generate, modulo any\n> > numbers (each anonymized token is assigned a number, like\n> > \"User 0\", and we replace it consistently in the output).\n> \n> I feel like this should be part of git-fast-export.txt, just to\n> increase the user's confidence in the tool (and I don't expect most\n> users to read this commit message).\n\nHmph. Whenever I say \"I think this patch is done\", suddenly the comments\nstart pouring in. :)\n\nI think you are right, though, and we could stand to explain\nthe feature a little more in the documentation in general.\nHow about this patch on top (or squashed in):\n\n-- >8 --\nSubject: docs/fast-export: explain --anonymize more completely\n\nThe original commit made mention of this option, but not why\none might want it or how they might use it. Let's try to be\na little more thorough, and also explain how to confirm that\nthe output really is anonymous.\n\nSigned-off-by: Jeff King \n---\n Documentation/git-fast-export.txt | 63 ++++++++++++++++++++++++++++++++++++---\n 1 file changed, 59 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\nindex 52831fa..dbe9a46 100644\n--- a/Documentation/git-fast-export.txt\n+++ b/Documentation/git-fast-export.txt\n@@ -106,10 +106,9 @@ marks the same across runs.\n \tdifferent from the commit's first parent).\n \n --anonymize::\n-\tReplace all refnames, paths, blob contents, commit and tag\n-\tmessages, names, and email addresses in the output with\n-\tanonymized data, while still retaining the shape of history and\n-\tof the stored tree.\n+\tAnonymize the contents of the repository while still retaining\n+\tthe shape of the history and stored tree. See the section on\n+\t`ANONYMIZING` below.\n \n --refspec::\n \tApply the specified refspec to each ref exported. Multiple of them can\n@@ -147,6 +146,62 @@ referenced by that revision range contains the string\n 'refs/heads/master'.\n \n \n+ANONYMIZING\n+-----------\n+\n+If the `--anonymize` option is given, git will attempt to remove all\n+identifying information from the repository while still retaining enough\n+of the original tree and history patterns to reproduce some bugs. The\n+goal is that a git bug which is found on a private repository will\n+persist in the anonymized repository, and the latter can be shared with\n+git developers to help solve the bug.\n+\n+With this option, git will replace all refnames, paths, blob contents,\n+commit and tag messages, names, and email addresses in the output with\n+anonymized data. Two instances of the same string will be replaced\n+equivalently (e.g., two commits with the same author will have the same\n+anonymized author in the output, but bear no resemblance to the original\n+author string). The relationship between commits, branches, and tags is\n+retained, as well as the commit timestamps (but the commit messages and\n+refnames bear no resemblance to the originals). The relative makeup of\n+the tree is retained (e.g., if you have a root tree with 10 files and 3\n+trees, so will the output), but their names and the contents of the\n+files will be replaced.\n+\n+If you think you have found a git bug, you can start by exporting an\n+anonymized stream of the whole repository:\n+\n+---------------------------------------------------\n+$ git fast-export --anonymize --all >anon-stream\n+---------------------------------------------------\n+\n+Then confirm that the bug persists in a repository created from that\n+stream (many bugs will not, as they really do depend on the exact\n+repository contents):\n+\n+---------------------------------------------------\n+$ git init anon-repo\n+$ cd anon-repo\n+$ git fast-import <../anon-stream\n+$ ... test your bug ...\n+---------------------------------------------------\n+\n+If the anonymized repository shows the bug, it may be worth sharing\n+`anon-stream` along with a regular bug report. Note that the anonymized\n+stream compresses very well, so gzipping it is encouraged. If you want\n+to examine the stream to see that it does not contain any private data,\n+you can peruse it directly before sending. You may also want to try:\n+\n+---------------------------------------------------\n+$ perl -pe 's/\\d+/X/g' On Thu, Aug 28, 2014 at 05:30:44PM +0700, Duy Nguyen wrote:\n> \n>> On Thu, Aug 28, 2014 at 12:01 AM, Jeff King wrote:\n>>> You can get an overview of what will be shared\n>>> by running a command like:\n>>>\n>>> git fast-export --anonymize --all |\n>>> perl -pe 's/\\d+/X/g' |\n>>> sort -u |\n>>> less\n>>>\n>>> which will show every unique line we generate, modulo any\n>>> numbers (each anonymized token is assigned a number, like\n>>> \"User 0\", and we replace it consistently in the output).\n>>\n>> I feel like this should be part of git-fast-export.txt, just to\n>> increase the user's confidence in the tool (and I don't expect most\n>> users to read this commit message).\n> \n> Hmph. Whenever I say \"I think this patch is done\", suddenly the comments\n> start pouring in. :)\n\n:-D\n\n> I think you are right, though, and we could stand to explain\n> the feature a little more in the documentation in general.\n> How about this patch on top (or squashed in):\n> \n> -- >8 --\n> Subject: docs/fast-export: explain --anonymize more completely\n> \n> The original commit made mention of this option, but not why\n> one might want it or how they might use it. Let's try to be\n> a little more thorough, and also explain how to confirm that\n> the output really is anonymous.\n> \n> Signed-off-by: Jeff King \n> ---\n> Documentation/git-fast-export.txt | 63 ++++++++++++++++++++++++++++++++++++---\n> 1 file changed, 59 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\n> index 52831fa..dbe9a46 100644\n> --- a/Documentation/git-fast-export.txt\n> +++ b/Documentation/git-fast-export.txt\n> @@ -106,10 +106,9 @@ marks the same across runs.\n> \tdifferent from the commit's first parent).\n> \n> --anonymize::\n> -\tReplace all refnames, paths, blob contents, commit and tag\n> -\tmessages, names, and email addresses in the output with\n> -\tanonymized data, while still retaining the shape of history and\n> -\tof the stored tree.\n> +\tAnonymize the contents of the repository while still retaining\n> +\tthe shape of the history and stored tree. See the section on\n> +\t`ANONYMIZING` below.\n> \n> --refspec::\n> \tApply the specified refspec to each ref exported. Multiple of them can\n> @@ -147,6 +146,62 @@ referenced by that revision range contains the string\n> 'refs/heads/master'.\n> \n> \n> +ANONYMIZING\n> +-----------\n> +\n> +If the `--anonymize` option is given, git will attempt to remove all\n> +identifying information from the repository while still retaining enough\n> +of the original tree and history patterns to reproduce some bugs. The\n> +goal is that a git bug which is found on a private repository will\n\ns/goal/hope/ ;-)\n\n> +persist in the anonymized repository, and the latter can be shared with\n> +git developers to help solve the bug.\n> +\n> +With this option, git will replace all refnames, paths, blob contents,\n> +commit and tag messages, names, and email addresses in the output with\n> +anonymized data. Two instances of the same string will be replaced\n> +equivalently (e.g., two commits with the same author will have the same\n> +anonymized author in the output, but bear no resemblance to the original\n> +author string). The relationship between commits, branches, and tags is\n> +retained, as well as the commit timestamps (but the commit messages and\n> +refnames bear no resemblance to the originals). The relative makeup of\n> +the tree is retained (e.g., if you have a root tree with 10 files and 3\n> +trees, so will the output), but their names and the contents of the\n> +files will be replaced.\n> +\n> +If you think you have found a git bug, you can start by exporting an\n> +anonymized stream of the whole repository:\n> +\n> +---------------------------------------------------\n> +$ git fast-export --anonymize --all >anon-stream\n> +---------------------------------------------------\n> +\n> +Then confirm that the bug persists in a repository created from that\n> +stream (many bugs will not, as they really do depend on the exact\n> +repository contents):\n\nDumb question (I have not even read the patch, so please just ignore me\nif this is indeed dumb!): Is the map of \navailable to the user while he attempts to confirm that the bug is still\npresent?\n\nFor example, if I anonymized git.git, and did 'git branch -v' (say), how\neasy would it be for me to recognise which branch was 'next'?\n\nATB,\nRamsay Jones\n"},{"id":"248508","messageId":"xmqqwq9sa3h8.fsf@gitster.dls.corp.google.com","threadId":"37387","inReplyTo":"20140828123257.GA18642@peff.net","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-08-28T18:11:47Z","receivedAt":"2014-08-28T18:11:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Subject: docs/fast-export: explain --anonymize more completely\n>\n> The original commit made mention of this option, but not why\n> one might want it or how they might use it. Let's try to be\n> a little more thorough, and also explain how to confirm that\n> the output really is anonymous.\n>\n> Signed-off-by: Jeff King \n> ---\n> Documentation/git-fast-export.txt | 63 ++++++++++++++++++++++++++++++++++++---\n> 1 file changed, 59 insertions(+), 4 deletions(-)\n>\n> diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt\n> index 52831fa..dbe9a46 100644\n> --- a/Documentation/git-fast-export.txt\n> +++ b/Documentation/git-fast-export.txt\n> @@ -106,10 +106,9 @@ marks the same across runs.\n> \tdifferent from the commit's first parent).\n> \n> --anonymize::\n> -\tReplace all refnames, paths, blob contents, commit and tag\n> -\tmessages, names, and email addresses in the output with\n> -\tanonymized data, while still retaining the shape of history and\n> -\tof the stored tree.\n> +\tAnonymize the contents of the repository while still retaining\n> +\tthe shape of the history and stored tree. See the section on\n> +\t`ANONYMIZING` below.\n\nTechnically s/tree/trees/, I would think. For a repository with\nmultiple branches, perhaps s/history/histories/, too, but I would\nnot insist on that ;-).\n\n> +ANONYMIZING\n> +-----------\n> +\n> +If the `--anonymize` option is given, git will attempt to remove all\n> +identifying information from the repository while still retaining enough\n> +of the original tree and history patterns to reproduce some bugs. The\n> +goal is that a git bug which is found on a private repository will\n> +persist in the anonymized repository, and the latter can be shared with\n> +git developers to help solve the bug.\n> +\n> +With this option, git will replace all refnames, paths, blob contents,\n> +commit and tag messages, names, and email addresses in the output with\n> +anonymized data. Two instances of the same string will be replaced\n> +equivalently (e.g., two commits with the same author will have the same\n> +anonymized author in the output, but bear no resemblance to the original\n> +author string). The relationship between commits, branches, and tags is\n> +retained, as well as the commit timestamps (but the commit messages and\n> +refnames bear no resemblance to the originals). The relative makeup of\n> +the tree is retained (e.g., if you have a root tree with 10 files and 3\n> +trees, so will the output), but their names and the contents of the\n> +files will be replaced.\n\nWhile I do not think I or anybody who would ask other people to use\nthis option would be confused, the phrase \"the same string\" may risk\nunnecessary worries from those who are asked to trust this option.\n\nI am not yet convinced that it is unlikely for the reader to read\nthe above and imagine that the anonymiser may go word by word,\nreplacing \"the same string\" with the same anonymised gibberish\n(which would be susceptible to old-school cryptoanalysis\ntechniques).\n\nAmong the ones that listed, refnames, blob contents, commit messages\nand tag messages are converted as a single \"string\" and I wish I\ncould think of phrasing to stress that point somehow.\n\nEach path component in paths is converted as a single \"string\", so\nwe can read from two anonymised paths if they refer to blobs in the\nsame directory in the original. This is a good thing, of course,\nbut it shows that among those listed in \"refnames, paths, blob\ncontents, ...\" in a flat sentence, some are treated as a single\ntoken for replacement but not others, and it is hard to tell for a\nreader which one is which, unless the reader knows the internals of\nGit, i.e. what kind of things we as the debuggers-of-Git would want\nto preserve.\n\nIsn't the unit for human identity anonymisation even more coarse?\nIf it is not should it?\n\nIn other words, do \"Junio C Hamano \" and \"Junio C\nHamano \" map to one gibberish human readable name\nwith two gibberish e-mail addresses, or 2 \"User$n \"? Is the\nfact that this organization seems to allocate two e-mails to each\ndeveloper something this organization may want to hide from the\npublic (and something we as the Git debuggers would not benefit from\nknowing)?\n"},{"id":"248510","messageId":"xmqqlhq8a20e.fsf@gitster.dls.corp.google.com","threadId":"37387","inReplyTo":"53FF5CD7.8040603@ramsay1.demon.co.uk","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-08-28T18:43:29Z","receivedAt":"2014-08-28T18:43:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramsay Jones writes:\n\n> Dumb question (I have not even read the patch, so please just ignore me\n> if this is indeed dumb!): Is the map of \n> available to the user while he attempts to confirm that the bug is still\n> present?\n>\n> For example, if I anonymized git.git, and did 'git branch -v' (say), how\n> easy would it be for me to recognise which branch was 'next'?\n\nIt is not dumb but actually is a very good point.\n\nThere needs an easy way for the reporting user to turn an\nobservation such as \"When I do 'git log master..next' I see this one\nextraneous commit shown\" into a corresponding statement to accompany\nthe anonymised output. The user needs it to make sure that the\nsymptom reproduces in the anonymised repository in order to decide\nif it is even worthwhile to send the output for analysis in the\nfirst place.\n"},{"id":"248512","messageId":"20140828185040.GC26855@peff.net","threadId":"37387","inReplyTo":"53FF5CD7.8040603@ramsay1.demon.co.uk","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-28T18:50:41Z","receivedAt":"2014-08-28T18:50:41Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 28, 2014 at 05:46:15PM +0100, Ramsay Jones wrote:\n\n> Dumb question (I have not even read the patch, so please just ignore me\n> if this is indeed dumb!): Is the map of \n> available to the user while he attempts to confirm that the bug is still\n> present?\n\nNo, it's not.\n\n> For example, if I anonymized git.git, and did 'git branch -v' (say), how\n> easy would it be for me to recognise which branch was 'next'?\n\nYou can't, really. The simplest thing would be to pare down your\nrepository to the minimum number of branches before anonymizing.\n\nIt might make sense to have an option to dump the maps we've stored to a\nseparate file (in theory, you could even load them back in and do an\nincremental anonymized export[1]). I think I'd rather wait on\nimplementing that until we see more real-world use cases (but as always,\nI'm happy to review if somebody wants to pick it up).\n\n-Peff\n\n[1] Incremental anonymization is not something I think is worth\n supporting by itself. However, there may be some value in being able\n to anonymize two similar repositories using the same mappings. For\n instance, a repository and its clone.\n"},{"id":"248513","messageId":"20140828190450.GD26855@peff.net","threadId":"37387","inReplyTo":"xmqqwq9sa3h8.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-28T19:04:50Z","receivedAt":"2014-08-28T19:04:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Aug 28, 2014 at 11:11:47AM -0700, Junio C Hamano wrote:\n\n> > +\tAnonymize the contents of the repository while still retaining\n> > +\tthe shape of the history and stored tree. See the section on\n> > +\t`ANONYMIZING` below.\n> \n> Technically s/tree/trees/, I would think. For a repository with\n> multiple branches, perhaps s/history/histories/, too, but I would\n> not insist on that ;-).\n\nSure, I think both of those are fine (I meant \"tree\" here to refer to\nthe general notion of a set of paths over time, not a particular tree\nobject).\n\n> > +With this option, git will replace all refnames, paths, blob contents,\n> > +commit and tag messages, names, and email addresses in the output with\n> > +anonymized data. Two instances of the same string will be replaced\n> > +equivalently (e.g., two commits with the same author will have the same\n> > +anonymized author in the output, but bear no resemblance to the original\n> > +author string). The relationship between commits, branches, and tags is\n> > +retained, as well as the commit timestamps (but the commit messages and\n> > +refnames bear no resemblance to the originals). The relative makeup of\n> > +the tree is retained (e.g., if you have a root tree with 10 files and 3\n> > +trees, so will the output), but their names and the contents of the\n> > +files will be replaced.\n> \n> While I do not think I or anybody who would ask other people to use\n> this option would be confused, the phrase \"the same string\" may risk\n> unnecessary worries from those who are asked to trust this option.\n> \n> I am not yet convinced that it is unlikely for the reader to read\n> the above and imagine that the anonymiser may go word by word,\n> replacing \"the same string\" with the same anonymised gibberish\n> (which would be susceptible to old-school cryptoanalysis\n> techniques).\n\nI tried to use phrases like \"bears no resemblance\" to indicate that the\nmapping was not leaking information. Does it bear a separate paragraph\nexplaining the transformation (I was trying to avoid that because it is\nnecessarily intimately linked with the particular implementation\nchosen).\n\n> Among the ones that listed, refnames, blob contents, commit messages\n> and tag messages are converted as a single \"string\" and I wish I\n> could think of phrasing to stress that point somehow.\n\nMaybe a separate paragraph like:\n\n Note that the replacement strings are chosen with no input from the\n original strings. There is no cryptography or other tricks involved,\n but rather we make up a new string like \"message 123\", replace a\n particular commit message with it, and then use the mapping between\n the two for the rest of the output. Thus, no information about the\n original commit message is leaked, and only the internal mapping\n (which is not part of the output stream) could reverse the\n transformation.\n\n> Each path component in paths is converted as a single \"string\", so\n> we can read from two anonymised paths if they refer to blobs in the\n> same directory in the original. This is a good thing, of course,\n> but it shows that among those listed in \"refnames, paths, blob\n> contents, ...\" in a flat sentence, some are treated as a single\n> token for replacement but not others, and it is hard to tell for a\n> reader which one is which, unless the reader knows the internals of\n> Git, i.e. what kind of things we as the debuggers-of-Git would want\n> to preserve.\n\nYes, I was really trying not to get into those details, because I do not\nthink they matter to most callers and are subject to change as we come\nup with better heuristics. I do not even want to promise an\nimplementation like \"no tricky cryptography\" above, because we may think\nof a more interesting way to transform components.\n\n> Isn't the unit for human identity anonymisation even more coarse?\n> If it is not should it?\n> \n> In other words, do \"Junio C Hamano \" and \"Junio C\n> Hamano \" map to one gibberish human readable name\n> with two gibberish e-mail addresses, or 2 \"User$n \"? Is the\n> fact that this organization seems to allocate two e-mails to each\n> developer something this organization may want to hide from the\n> public (and something we as the Git debuggers would not benefit from\n> knowing)?\n\nThe ident mapping takes a single \"Name \" string and converts it\ninto a \"User X \" string. So no, we are not leaking\nthe fact that one name has multiple emails. I actually started down that\npath, but gave it up, as it could produce entries like \"User 3\n\" which were downright confusing. Plus I did not\nthink that would be a useful thing for debuggers to know, and replacing\nthe whole string is simpler (I also entertained the idea of just\nblanking _all_ idents; what I expect to be of primary use here is the\nhistory shape, and I doubt that a bug would be triggered by the pattern\nof usernames but not their actual content).\n\n-Peff\n"},{"id":"248678","messageId":"CAPig+cSKfxAsr+RQp8N8GAOe9fa_6kGPi_-X99Vs-2bYbBoC8A@mail.gmail.com","threadId":"37387","inReplyTo":"20140828123257.GA18642@peff.net","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2014-08-31T10:34:08Z","receivedAt":"2014-08-31T10:34:08Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Thu, Aug 28, 2014 at 8:32 AM, Jeff King wrote:\n> On Thu, Aug 28, 2014 at 05:30:44PM +0700, Duy Nguyen wrote:\n>\n>> On Thu, Aug 28, 2014 at 12:01 AM, Jeff King wrote:\n>> > You can get an overview of what will be shared\n>> > by running a command like:\n>> >\n>> > git fast-export --anonymize --all |\n>> > perl -pe 's/\\d+/X/g' |\n>> > sort -u |\n>> > less\n>> >\n>> > which will show every unique line we generate, modulo any\n>> > numbers (each anonymized token is assigned a number, like\n>> > \"User 0\", and we replace it consistently in the output).\n>>\n>> I feel like this should be part of git-fast-export.txt, just to\n>> increase the user's confidence in the tool (and I don't expect most\n>> users to read this commit message).\n>\n> Hmph. Whenever I say \"I think this patch is done\", suddenly the comments\n> start pouring in. :)\n\nConsidering that the value of --anonymize is not yet known, is such an\ninvasive change to fast-export.c warranted? Would it make sense\ninstead to provide \"anonymize\" functionality as a contrib/ script or a\ndistinct git-anonymize-foo command which accepts a fast-import stream\nas input and anonymizes it as output?\n"},{"id":"248687","messageId":"20140831155350.GE17449@peff.net","threadId":"37387","inReplyTo":"CAPig+cSKfxAsr+RQp8N8GAOe9fa_6kGPi_-X99Vs-2bYbBoC8A@mail.gmail.com","subject":"Re: [PATCH v3] teach fast-export an --anonymize option","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-08-31T15:53:51Z","receivedAt":"2014-08-31T15:53:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Aug 31, 2014 at 06:34:08AM -0400, Eric Sunshine wrote:\n\n> >> I feel like this should be part of git-fast-export.txt, just to\n> >> increase the user's confidence in the tool (and I don't expect most\n> >> users to read this commit message).\n> >\n> > Hmph. Whenever I say \"I think this patch is done\", suddenly the comments\n> > start pouring in. :)\n> \n> Considering that the value of --anonymize is not yet known, is such an\n> invasive change to fast-export.c warranted? Would it make sense\n> instead to provide \"anonymize\" functionality as a contrib/ script or a\n> distinct git-anonymize-foo command which accepts a fast-import stream\n> as input and anonymizes it as output?\n\nI considered that, but there's a non-trivial amount of work in the\nparsing of the stream (I had originally thought to just ship a perl\nscript to operate on the stream). And while there's a fair bit of code\nadded to fast-export.c, none of it is ever called unless --anonymize is\nset.\n\nSo while I am not 100% sure that the idea is a good one, I do not think\nit is hurting the current fast-export in any meaningful way. Two things\nwe could do to minimize that are:\n\n 1. Move the anonymization code into a separate C file to keep the\n fast-export source a little more pristine. I avoided doing this\n just because the interfaces to the functions are fairly tailored to\n what fast-export wants.\n\n 2. Have a separate git-anonymize command which is basically running\n \"git fast-export --anonymize\" under the hood. This avoids polluting\n fast-export from the user's perspective (they do not need to care\n that it is running fast-export under the hood).\n\n-Peff\n"}]}