git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 09/10] fast-export: allow seeding the anonymized mapping

From
Jeff King <peff@peff.net>
Date
Jun 23, 2020, 15:25 UTC
Message-ID
<20200623152505.GI1435482@coredump.intra.peff.net>
In-Reply-To
<20200623152436.GA50925@coredump.intra.peff.net>

After you anonymize a repository, it can be hard to find which commits correspond between the original and the result, and thus hard to reproduce commands that triggered bugs in the original.

Let's make it possible to seed the anonymization map. This lets users either:

  - mark names to be retained as-is, if they don't consider them secret
    (in which case their original commands would just work)
  - map names to new values, which lets them adapt the reproduction
    recipe to the new names without revealing the originals

The implementation is fairly straight-forward. We already store each anonymized token in a hashmap (so that the same token appearing twice is converted to the same result). We can just introduce a new "seed" hashmap which is consulted first.

This does make a few more promises to the user about how we'll anonymize things (e.g., token-splitting pathnames). But it's unlikely that we'd want to change those rules, even if the actual anonymization of a single token changes. And it makes things much easier for the user, who can unblind only a directory name without having to specify each path within it.

One alternative to this approach would be to anonymize as we see fit, and then dump the whole refname and pathname mappings to a file. This does work, but it's a bit awkward to use (you have to manually dig the items you care about out of the mapping).

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-fast-export.txt | 24 ++++++++++++++++
 builtin/fast-export.c             | 47 ++++++++++++++++++++++++++++++-
 t/t9351-fast-export-anonymize.sh  | 11 +++++++-
 3 files changed, 80 insertions(+), 2 deletions(-)
diff --git a/Documentation/git-fast-export.txt b/Documentation/git-fast-export.txt
index e8950de3ba..2d7b62e835 100644
--- a/Documentation/git-fast-export.txt
+++ b/Documentation/git-fast-export.txt
@@ -119,6 +119,11 @@ by keeping the marks the same across runs.
 	the shape of the history and stored tree.  See the section on
 	`ANONYMIZING` below.
 
+--seed-anonymized=<from>[:<to>]::
+	Convert token `<from>` to `<to>` in the anonymized output. If
+	`<to>` is omitted, map `<from>` to itself (i.e., do not
+	anonymize it). See the section on `ANONYMIZING` below.
+
 --reference-excluded-parents::
 	By default, running a command such as `git fast-export
 	master~5..master` will not include the commit master{tilde}5
@@ -238,6 +243,25 @@ collapse "User 0", "User 1", etc into "User X"). This produces a much
 smaller output, and it is usually easy to quickly confirm that there is
 no private data in the stream.
 
+Reproducing some bugs may require referencing particular commits or
+paths, which becomes challenging after refnames and paths have been
+anonymized. You can ask for a particular token to be left as-is or
+mapped to a new value. For example, if you have a bug which reproduces
+with `git rev-list mybranch -- foo.c`, you can run:
+
+---------------------------------------------------
+$ git fast-export --anonymize --all \
+      --seed-anonymized=foo.c:secret.c \
+      --seed-anonymized=mybranch \
+      >stream
+---------------------------------------------------
+
+After importing the stream, you can then run `git rev-list mybranch --
+secret.c` in the anonymized repository.
+
+Note that paths and refnames are split into tokens at slash boundaries.
+The command above would anonymize `subdir/foo.c` as something like
+`path123/secret.c`.
 
 LIMITATIONS
 -----------
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index 1cbca5b4b4..ef82497bbf 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -45,6 +45,7 @@ static struct string_list extra_refs = STRING_LIST_INIT_NODUP;
 static struct string_list tag_refs = STRING_LIST_INIT_NODUP;
 static struct refspec refspecs = REFSPEC_INIT_FETCH;
 static int anonymize;
+static struct hashmap anonymized_seeds;
 static struct revision_sources revision_sources;
 
 static int parse_opt_signed_tag_mode(const struct option *opt,
@@ -168,8 +169,18 @@ static const char *anonymize_str(struct hashmap *map,
 	hashmap_entry_init(&key.hash, memhash(orig, len));
 	key.orig = orig;
 	key.orig_len = len;
-	ret = hashmap_get_entry(map, &key, hash, &key);
 
+	/* First check if it's a token the user configured manually... */
+	if (anonymized_seeds.cmpfn)
+		ret = hashmap_get_entry(&anonymized_seeds, &key, hash, &key);
+	else
+		ret = NULL;
+
+	/* ...otherwise check if we've already seen it in this context... */
+	if (!ret)
+		ret = hashmap_get_entry(map, &key, hash, &key);
+
+	/* ...and finally generate a new mapping if necessary */
 	if (!ret) {
 		FLEX_ALLOC_MEM(ret, orig, orig, len);
 		hashmap_entry_init(&ret->hash, key.hash.hash);
@@ -1147,6 +1158,37 @@ static void handle_deletes(void)
 	}
 }
 
+static char *anonymize_seed(void *data)
+{
+	return xstrdup(data);
+}
+
+static int parse_opt_seed_anonymized(const struct option *opt,
+				     const char *arg, int unset)
+{
+	struct hashmap *map = opt->value;
+	const char *delim, *value;
+	size_t keylen;
+
+	BUG_ON_OPT_NEG(unset);
+
+	delim = strchr(arg, ':');
+	if (delim) {
+		keylen = delim - arg;
+		value = delim + 1;
+	} else {
+		keylen = strlen(arg);
+		value = arg;
+	}
+
+	if (!keylen || !*value)
+		return error(_("--seed-anonymized token cannot be empty"));
+
+	anonymize_str(map, anonymize_seed, arg, keylen, (void *)value);
+
+	return 0;
+}
+
 int cmd_fast_export(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
@@ -1188,6 +1230,9 @@ int cmd_fast_export(int argc, const char **argv, const char *prefix)
 		OPT_STRING_LIST(0, "refspec", &refspecs_list, N_("refspec"),
 			     N_("Apply refspec to exported refs")),
 		OPT_BOOL(0, "anonymize", &anonymize, N_("anonymize output")),
+		OPT_CALLBACK_F(0, "seed-anonymized", &anonymized_seeds, N_("from:to"),
+			       N_("convert <from> to <to> in anonymized output"),
+			       PARSE_OPT_NONEG, parse_opt_seed_anonymized),
 		OPT_BOOL(0, "reference-excluded-parents",
 			 &reference_excluded_commits, N_("Reference parents which are not in fast-export stream by object id")),
 		OPT_BOOL(0, "show-original-ids", &show_original_ids,
diff --git a/t/t9351-fast-export-anonymize.sh b/t/t9351-fast-export-anonymize.sh
index dc5d75cd19..d84eec9bab 100755
--- a/t/t9351-fast-export-anonymize.sh
+++ b/t/t9351-fast-export-anonymize.sh
@@ -6,6 +6,7 @@ test_description='basic tests for fast-export --anonymize'
 test_expect_success 'setup simple repo' '
 	test_commit base &&
 	test_commit foo &&
+	test_commit retain-me &&
 	git checkout -b other HEAD^ &&
 	mkdir subdir &&
 	test_commit subdir/bar &&
@@ -18,7 +19,10 @@ test_expect_success 'setup simple repo' '
 '
 
 test_expect_success 'export anonymized stream' '
-	git fast-export --anonymize --all >stream
+	git fast-export --anonymize --all \
+		--seed-anonymized=retain-me \
+		--seed-anonymized=xyzzy:custom-name \
+		>stream
 '
 
 # this also covers commit messages
@@ -30,6 +34,11 @@ test_expect_success 'stream omits path names' '
 	! grep xyzzy stream
 '
 
+test_expect_success 'stream contains user-specified names' '
+	grep retain-me stream &&
+	grep custom-name stream
+'
+
 test_expect_success 'stream omits gitlink oids' '
 	# avoid relying on the whole oid to remain hash-agnostic; this is
 	# plenty to be unique within our test case

-- 
2.27.0.517.gbc32778fa3
Previous: Junio C HamanoNext: Eric Sunshine
Message 40 of 64 in “fast-export: allow dumping anonymization mappings”
  1. 0/3 fast-export: allow dumping anonymization mappingsJeff King, Jun 19, 2020
  2. 1/3 fast-export: allow dumping the refname mappingJeff King, Jun 19, 2020
  3. Eric SunshineJun 19, 2020
  4. Jeff KingJun 19, 2020
  5. Eric SunshineJun 19, 2020
  6. Jeff KingJun 19, 2020
  7. Eric SunshineJun 19, 2020
  8. Jeff KingJun 22, 2020
  9. Junio C HamanoJun 19, 2020
  10. Jeff KingJun 22, 2020
  11. 2/3 fast-export: anonymize "master" refnameJeff King, Jun 19, 2020
  12. 3/3 fast-export: allow dumping the path mappingJeff King, Jun 19, 2020
  13. Eric SunshineJun 19, 2020
  14. Junio C HamanoJun 19, 2020
  15. Jeff KingJun 22, 2020
  16. Johannes SchindelinJun 19, 2020
  17. Junio C HamanoJun 22, 2020
  18. 0/4 fast-export: allow dumping anonymization mappingsJeff King, Jun 22, 2020
  19. 1/4 fast-export: allow dumping the refname mappingJeff King, Jun 22, 2020
  20. 2/4 fast-export: anonymize "master" refnameJeff King, Jun 22, 2020
  21. 3/4 fast-export: refactor path printing to not rely on stdoutJeff King, Jun 22, 2020
  22. 4/4 fast-export: allow dumping the path mappingJeff King, Jun 22, 2020
  23. 0/10 fast-export: allow seeding the anonymized mappingJeff King, Jun 23, 2020
  24. 01/10 t9351: derive anonymized tree checks from original repoJeff King, Jun 23, 2020
  25. 02/10 fast-export: use xmemdupz() for anonymizing oidsJeff King, Jun 23, 2020
  26. 03/10 fast-export: store anonymized oids as hex stringsJeff King, Jun 23, 2020
  27. SZEDER GáborJun 24, 2020
  28. Jeff KingJun 24, 2020
  29. Jeff KingJun 25, 2020
  30. SZEDER GáborJun 25, 2020
  31. Jeff KingJun 25, 2020
  32. Johannes SchindelinJun 29, 2020
  33. Jeff KingJun 30, 2020
  34. 04/10 fast-export: tighten anonymize_mem() interface to handle only stringsJeff King, Jun 23, 2020
  35. 05/10 fast-export: stop storing lengths in anonymized hashmapsJeff King, Jun 23, 2020
  36. 06/10 fast-export: use a flex array to store anonymized entriesJeff King, Jun 23, 2020
  37. 07/10 fast-export: move global "idents" anonymize hashmap into functionJeff King, Jun 23, 2020
  38. 08/10 fast-export: add a "data" callback parameter to anonymize_str()Jeff King, Jun 23, 2020
  39. Junio C HamanoJun 24, 2020
  40. 09/10 fast-export: allow seeding the anonymized mappingJeff King, Jun 23, 2020
  41. Eric SunshineJun 23, 2020
  42. Jeff KingJun 23, 2020
  43. Eric SunshineJun 23, 2020
  44. Jeff KingJun 24, 2020
  45. Eric SunshineJun 23, 2020
  46. Jeff KingJun 23, 2020
  47. Eric SunshineJun 23, 2020
  48. Jeff KingJun 24, 2020
  49. 10/10 fast-export: anonymize "master" refnameJeff King, Jun 23, 2020
  50. Junio C HamanoJun 23, 2020
  51. Jeff KingJun 23, 2020
  52. 0/11 fast-export: allow seeding the anonymized mappingJeff King, Jun 25, 2020
  53. 01/11 t9351: derive anonymized tree checks from original repoJeff King, Jun 25, 2020
  54. 02/11 fast-export: use xmemdupz() for anonymizing oidsJeff King, Jun 25, 2020
  55. 03/11 fast-export: store anonymized oids as hex stringsJeff King, Jun 25, 2020
  56. 05/11 fast-export: stop storing lengths in anonymized hashmapsJeff King, Jun 25, 2020
  57. 06/11 fast-export: use a flex array to store anonymized entriesJeff King, Jun 25, 2020
  58. 04/11 fast-export: tighten anonymize_mem() interface to handle only stringsJeff King, Jun 25, 2020
  59. 07/11 fast-export: move global "idents" anonymize hashmap into functionJeff King, Jun 25, 2020
  60. 08/11 fast-export: add a "data" callback parameter to anonymize_str()Jeff King, Jun 25, 2020
  61. 09/11 fast-export: allow seeding the anonymized mappingJeff King, Jun 25, 2020
  62. 11/11 fast-export: use local array to store anonymized oidJeff King, Jun 25, 2020
  63. 10/11 fast-export: anonymize "master" refnameJeff King, Jun 25, 2020
  64. Junio C HamanoJun 25, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.