{"thread":{"id":"50012","subject":"[PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","startedAt":"2018-12-11T23:49:29Z","lastAt":"2018-12-18T16:51:06Z","messageCount":17,"participants":["Tejun Heo","Junio C Hamano","Stefan Xenos"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"365133","messageId":"20181211234909.2855638-1-tj@kernel.org","threadId":"50012","inReplyTo":null,"subject":"[PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:04Z","receivedAt":"2018-12-11T23:49:29Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hello,\n\nSome trailers refer to other commits. Let's call them xrefs\n(cross-references). For example, a cherry pick trailer points to the\nsource commit. It is sometimes useful to build a reverse map of these\nxrefs - ie. source -> cherry-pick instead of cherry-pick -> source.\n\nThis, e.g, can answer which releases a commit ended up in on\nrepositories which cherry-picks fixes back from the development\nbranch. Let's say the repository looks like the following.\n\n\t D'---E'' release-B\n\t /\n\tC' E' release-D\n / /\n A---B---C---D---E master\n\nwhere the followings cherry-picks took place.\n\n C -> C'\n D -> D'\n E -> E' -> E''\n\nUsing the reverse-mapping in this patchset, we can get the following\nannotated log which succinctly shows which commit ended up where.\n\n $ git log --notes=xref-cherry-picks --oneline | git name-rev --name-only --stdin\n 4b165af commit E\n Notes (xref-cherry-picks):\n Cherry-picked-to: release-D\n Cherry-picked-to: release-B\n\n 82bf1f3 commit D\n Notes (xref-cherry-picks):\n Cherry-picked-to: release-B~1\n\n 364eccf commit C\n Notes (xref-cherry-picks):\n Cherry-picked-to: release-B~2\n\n ed3adf3 commit B\n ddd1bf2 commit A\n\nThis patchset implements generic trailer cross-reference\nreverse-mapping and the necessary hooks and samples to keep\ncherry-pick reverse-maps up-to-date.\n\ndiffstat follows. Thanks.\n\n Documentation/git-cherry-pick.txt | 5 \n Documentation/git-fetch.txt | 5 \n Documentation/git-notes.txt | 8 \n Documentation/git-reverse-trailer-xrefs.txt | 145 +++++++++++++++\n Documentation/githooks.txt | 23 ++\n Makefile | 1 \n builtin.h | 1 \n builtin/fetch.c | 72 +++++++\n builtin/reverse-trailer-xrefs.c | 160 +++++++++++++++++\n builtin/revert.c | 14 +\n git.c | 1 \n notes-merge.c | 9 \n notes-utils.c | 2 \n notes-utils.h | 3 \n notes.c | 260 +++++++++++++++++++++++++++-\n notes.h | 9 \n t/t3321-notes-xref-cherry-picks.sh | 124 +++++++++++++\n templates/hooks--post-cherry-pick.sample | 8 \n templates/hooks--post-fetch.sample | 30 +++\n trailer.c | 105 +++++++++++\n trailer.h | 38 ++++\n 21 files changed, 1015 insertions(+), 8 deletions(-)\n\n--\ntejun\n\n"},{"id":"365134","messageId":"20181211234909.2855638-2-tj@kernel.org","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"[PATCH 1/5] trailer: Implement a helper to reverse-map trailer xrefs","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:05Z","receivedAt":"2018-12-11T23:49:33Z","isPatch":true,"sender":{"key":"tj@kernel.org","avatar":null},"body":"From: Tejun Heo \n\nSome trailers refer to other commits. Let's call them xrefs\n(cross-references). For example, a cherry pick trailer points to the\nsource commit. It is sometimes useful to build a reverse map of these\nxrefs - ie. source -> cherry-pick instead of cherry-pick -> source.\n\nThis patch implements trailer helpers to enable such reverse maps.\n\nThe helpers can process arbitrary trailers and once the xrefs are\nrecorded, the reverse-mapping can be iterated per source commit using\ntrailer_rev_xrefs_for_each().\n\nSigned-off-by: Tejun Heo \n---\n trailer.c | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n trailer.h | 38 ++++++++++++++++++++\n 2 files changed, 143 insertions(+)\n\ndiff --git a/trailer.c b/trailer.c\nindex 0796f326b..15744600b 100644\n--- a/trailer.c\n+++ b/trailer.c\n@@ -2,6 +2,7 @@\n #include \"config.h\"\n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"object-store.h\"\n #include \"commit.h\"\n #include \"tempfile.h\"\n #include \"trailer.h\"\n@@ -1170,3 +1171,107 @@ void format_trailers_from_commit(struct strbuf *out, const char *msg,\n \tformat_trailer_info(out, &info, opts);\n \ttrailer_info_release(&info);\n }\n+\n+implement_static_commit_slab(trailer_rxrefs_slab, struct object_array *);\n+\n+static struct object_array *get_trailer_rxrefs(\n+\t\t\tstruct trailer_rev_xrefs *rxrefs, struct commit *commit)\n+{\n+\tstruct object_array **slot =\n+\t\ttrailer_rxrefs_slab_peek(&rxrefs->slab, commit);\n+\n+\treturn slot ? *slot : NULL;\n+}\n+\n+static struct object_array *get_create_trailer_rxrefs(\n+\t\t\tstruct trailer_rev_xrefs *rxrefs, struct commit *commit)\n+{\n+\tstruct object_array **slot =\n+\t\ttrailer_rxrefs_slab_at(&rxrefs->slab, commit);\n+\n+\tif (*slot)\n+\t\treturn *slot;\n+\n+\tadd_object_array(&commit->object, oid_to_hex(&commit->object.oid),\n+\t\t\t &rxrefs->dst_commits);\n+\t*slot = xmalloc(sizeof(struct object_array));\n+\t**slot = (struct object_array)OBJECT_ARRAY_INIT;\n+\treturn *slot;\n+}\n+\n+void trailer_rev_xrefs_init(struct trailer_rev_xrefs *rxrefs,\n+\t\t\t const char *trailer_prefix)\n+{\n+\trxrefs->trailer_prefix = xstrdup(trailer_prefix);\n+\trxrefs->trailer_prefix_len = strlen(trailer_prefix);\n+\tinit_trailer_rxrefs_slab(&rxrefs->slab);\n+\trxrefs->dst_commits = (struct object_array)OBJECT_ARRAY_INIT;\n+}\n+\n+/* record the reverse mapping of @commit's xref trailer */\n+void trailer_rev_xrefs_record(struct trailer_rev_xrefs *rxrefs,\n+\t\t\t struct commit *commit)\n+{\n+\tstruct process_trailer_options opts = PROCESS_TRAILER_OPTIONS_INIT;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *buffer;\n+\tstruct trailer_info info;\n+\tint i;\n+\n+\tbuffer = read_object_file(&commit->object.oid, &type, &size);\n+\ttrailer_info_get(&info, buffer, &opts);\n+\n+\t/* when nested, the last trailer describes the latest cherry-pick */\n+\tfor (i = info.trailer_nr - 1; i >= 0; i--) {\n+\t\tchar *line = info.trailers[i];\n+\n+\t\tif (starts_with(line, rxrefs->trailer_prefix)) {\n+\t\t\tstruct object_id dst_oid;\n+\t\t\tstruct object *dst_object;\n+\t\t\tstruct commit *dst_commit;\n+\t\t\tstruct object_array *src_objs;\n+\t\t\tchar cherry_hex[GIT_MAX_HEXSZ + 1];\n+\n+\t\t\tif (get_oid_hex(line + rxrefs->trailer_prefix_len,\n+\t\t\t\t\t&dst_oid))\n+\t\t\t\tcontinue;\n+\n+\t\t\tdst_object = parse_object(the_repository, &dst_oid);\n+\t\t\tif (!dst_object || dst_object->type != OBJ_COMMIT)\n+\t\t\t\tcontinue;\n+\n+\t\t\tdst_commit = (struct commit *)dst_object;\n+\t\t\tsrc_objs = get_create_trailer_rxrefs(rxrefs, dst_commit);\n+\n+\t\t\toid_to_hex_r(cherry_hex, &commit->object.oid);\n+\t\t\tadd_object_array(&commit->object, cherry_hex, src_objs);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree(buffer);\n+}\n+\n+void trailer_rev_xrefs_release(struct trailer_rev_xrefs *rxrefs)\n+{\n+\tclear_trailer_rxrefs_slab(&rxrefs->slab);\n+\tobject_array_clear(&rxrefs->dst_commits);\n+\tfree(rxrefs->trailer_prefix);\n+}\n+\n+void trailer_rev_xrefs_next(struct trailer_rev_xrefs *rxrefs, int *idx_p,\n+\t\t\t struct commit **dst_commit_p,\n+\t\t\t struct object_array **src_objs_p)\n+{\n+\tif (*idx_p >= rxrefs->dst_commits.nr) {\n+\t\t*dst_commit_p = NULL;\n+\t\t*src_objs_p = NULL;\n+\t\treturn;\n+\t}\n+\n+\t*dst_commit_p = (struct commit *)\n+\t\trxrefs->dst_commits.objects[*idx_p].item;\n+\t*src_objs_p = get_trailer_rxrefs(rxrefs, *dst_commit_p);\n+\t(*idx_p)++;\n+}\ndiff --git a/trailer.h b/trailer.h\nindex b99773964..07090a11f 100644\n--- a/trailer.h\n+++ b/trailer.h\n@@ -2,6 +2,8 @@\n #define TRAILER_H\n \n #include \"list.h\"\n+#include \"object.h\"\n+#include \"commit-slab.h\"\n \n struct strbuf;\n \n@@ -99,4 +101,40 @@ void trailer_info_release(struct trailer_info *info);\n void format_trailers_from_commit(struct strbuf *out, const char *msg,\n \t\t\t\t const struct process_trailer_options *opts);\n \n+/*\n+ * Helpers to reverse trailers referencing to other commits.\n+ *\n+ * Some trailers, e.g. \"(cherry picked from...)\", references other commits.\n+ * The following helpers can be used to reverse map those references. See\n+ * builtin/reverse-trailer-xrefs.c for a usage example.\n+ */\n+declare_commit_slab(trailer_rxrefs_slab, struct object_array *);\n+\n+struct trailer_rev_xrefs {\n+\tchar *trailer_prefix;\n+\tint trailer_prefix_len;\n+\tstruct trailer_rxrefs_slab slab;\n+\tstruct object_array dst_commits;\n+};\n+\n+void trailer_rev_xrefs_init(struct trailer_rev_xrefs *rxrefs,\n+\t\t\t const char *trailer_prefix);\n+void trailer_rev_xrefs_record(struct trailer_rev_xrefs *rxrefs,\n+\t\t\t struct commit *commit);\n+void trailer_rev_xrefs_release(struct trailer_rev_xrefs *rxrefs);\n+\n+void trailer_rev_xrefs_next(struct trailer_rev_xrefs *rxrefs,\n+\t\t\t int *idx_p, struct commit **dst_commit_p,\n+\t\t\t struct object_array **src_objs_p);\n+\n+/*\n+ * Iterate the recorded reverse mappings - @dst_commit was pointed to by\n+ * commits in @src_objs.\n+ */\n+#define trailer_rev_xrefs_for_each(rxrefs, idx, dst_commit, src_objs)\t\t\\\n+\tfor ((idx) = 0,\t\t\t\t\t\t\t\t\\\n+\t trailer_rev_xrefs_next(rxrefs, &(idx), &(dst_commit), &(src_objs));\\\n+\t (dst_commit);\t\t\t\t\t\t\t\\\n+\t trailer_rev_xrefs_next(rxrefs, &(idx), &(dst_commit), &(src_objs)))\n+\n #endif /* TRAILER_H */\n-- \n2.17.1\n\n"},{"id":"365135","messageId":"20181211234909.2855638-3-tj@kernel.org","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"[PATCH 2/5] notes: Implement special handlings for refs/notes/xref-","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:06Z","receivedAt":"2018-12-11T23:49:36Z","isPatch":true,"sender":{"key":"tj@kernel.org","avatar":null},"body":"From: Tejun Heo \n\nSome trailers refer to other commits. Let's call them xrefs\n(cross-references). For example, a cherry pick trailer points to the\nsource commit. It is sometimes useful to build a reverse map of these\nxrefs - ie. source -> cherry-pick instead of cherry-pick -> source.\n\nThese reverse-maps will be recorded in special notes whose refs start\nwith refs/notes/xref-. This patch implements the following special\nhandling for the xref notes.\n\n* When xref notes are appended to an existing one, both parts get\n parsed and dead or dupliate references are dropped, so that the\n merged note contains only valid and unique xrefs.\n\n* When xref notes are formatted for printing, the formatter recurses\n into each xref and prints the nested xrefs with increasing\n indentation to show the comprehensive xref chains.\n\nThe latter part will be documented by a future patch with the actual\nuse case.\n\nSigned-off-by: Tejun Heo \n---\n Documentation/git-notes.txt | 8 ++\n notes-merge.c | 9 ++\n notes-utils.c | 2 +\n notes-utils.h | 3 +-\n notes.c | 260 +++++++++++++++++++++++++++++++++++-\n notes.h | 9 ++\n 6 files changed, 288 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-notes.txt b/Documentation/git-notes.txt\nindex df2b64dbb..872919ad4 100644\n--- a/Documentation/git-notes.txt\n+++ b/Documentation/git-notes.txt\n@@ -88,6 +88,10 @@ the command can read the input given to the `post-rewrite` hook.)\n \tAppend to the notes of an existing object (defaults to HEAD).\n \tCreates a new notes object if needed.\n \n+\tIf the notes ref starts with `/refs/notes/xref-`, each\n+\tline is expected to contain `: ` and\n+\tonly lines with reachable unique commit IDs are kept.\n+\n edit::\n \tEdit the notes for a given object (defaults to HEAD).\n \n@@ -273,6 +277,10 @@ Note that if either the local or remote version contain duplicate lines\n prior to the merge, these will also be removed by this notes merge\n strategy.\n \n+\"cat_xrefs\" is similar to \"union\" but expects each line to contain\n+`: `. Only lines with reachable unique commit IDs are\n+kept.\n+\n \n EXAMPLES\n --------\ndiff --git a/notes-merge.c b/notes-merge.c\nindex bd05d50b0..3fe2389a1 100644\n--- a/notes-merge.c\n+++ b/notes-merge.c\n@@ -464,6 +464,15 @@ static int merge_one_change(struct notes_merge_options *o,\n \t\t\tdie(\"failed to concatenate notes \"\n \t\t\t \"(combine_notes_cat_sort_uniq)\");\n \t\treturn 0;\n+\tcase NOTES_MERGE_RESOLVE_CAT_XREFS:\n+\t\tif (o->verbosity >= 2)\n+\t\t\tprintf(\"Concatenating unique and valid cross-references \"\n+\t\t\t \"in local and remote notes for %s\\n\",\n+\t\t\t oid_to_hex(&p->obj));\n+\t\tif (add_note(t, &p->obj, &p->remote, combine_notes_cat_xrefs))\n+\t\t\tdie(\"failed to concatenate notes \"\n+\t\t\t \"(combine_notes_cat_xrefs)\");\n+\t\treturn 0;\n \t}\n \tdie(\"Unknown strategy (%i).\", o->strategy);\n }\ndiff --git a/notes-utils.c b/notes-utils.c\nindex 14ea03178..db6363c39 100644\n--- a/notes-utils.c\n+++ b/notes-utils.c\n@@ -70,6 +70,8 @@ int parse_notes_merge_strategy(const char *v, enum notes_merge_strategy *s)\n \t\t*s = NOTES_MERGE_RESOLVE_UNION;\n \telse if (!strcmp(v, \"cat_sort_uniq\"))\n \t\t*s = NOTES_MERGE_RESOLVE_CAT_SORT_UNIQ;\n+\telse if (!strcmp(v, \"cat_xrefs\"))\n+\t\t*s = NOTES_MERGE_RESOLVE_CAT_XREFS;\n \telse\n \t\treturn -1;\n \ndiff --git a/notes-utils.h b/notes-utils.h\nindex 540830652..b604f9b51 100644\n--- a/notes-utils.h\n+++ b/notes-utils.h\n@@ -28,7 +28,8 @@ enum notes_merge_strategy {\n \t\tNOTES_MERGE_RESOLVE_OURS,\n \t\tNOTES_MERGE_RESOLVE_THEIRS,\n \t\tNOTES_MERGE_RESOLVE_UNION,\n-\t\tNOTES_MERGE_RESOLVE_CAT_SORT_UNIQ\n+\t\tNOTES_MERGE_RESOLVE_CAT_SORT_UNIQ,\n+\t\tNOTES_MERGE_RESOLVE_CAT_XREFS,\n };\n \n struct notes_rewrite_cfg {\ndiff --git a/notes.c b/notes.c\nindex 25cdce28b..835466787 100644\n--- a/notes.c\n+++ b/notes.c\n@@ -9,6 +9,7 @@\n #include \"tree-walk.h\"\n #include \"string-list.h\"\n #include \"refs.h\"\n+#include \"hashmap.h\"\n \n /*\n * Use a non-balancing simple 16-tree structure with struct int_node as\n@@ -996,8 +997,12 @@ void init_notes(struct notes_tree *t, const char *notes_ref,\n \tif (!notes_ref)\n \t\tnotes_ref = default_notes_ref();\n \n-\tif (!combine_notes)\n-\t\tcombine_notes = combine_notes_concatenate;\n+\tif (!combine_notes) {\n+\t\tif (starts_with(notes_ref, \"refs/notes/xref-\"))\n+\t\t\tcombine_notes = combine_notes_cat_xrefs;\n+\t\telse\n+\t\t\tcombine_notes = combine_notes_concatenate;\n+\t}\n \n \tt->root = (struct int_node *) xcalloc(1, sizeof(struct int_node));\n \tt->first_non_note = NULL;\n@@ -1189,6 +1194,76 @@ void free_notes(struct notes_tree *t)\n \tmemset(t, 0, sizeof(struct notes_tree));\n }\n \n+/*\n+ * Parse a \"[TAG:]HEX\" line. @line is trimmed. If @tag_p is not NULL and\n+ * TAG exists, the string is split. Returns the pointer to OID and updates\n+ * *@tag_p to point to TAG.\n+ */\n+static char *parse_xref(char *line, char **tag_p)\n+{\n+\tchar *p, *hex;\n+\n+\t/* ltrim */\n+\twhile (isspace(*line))\n+\t\tline++;\n+\n+\tp = strchr(line, ':');\n+\tif (p) {\n+\t\tif (tag_p) {\n+\t\t\t/* split and store TAG */\n+\t\t\t*tag_p = line;\n+\t\t\t*p = '\\0';\n+\t\t}\n+\t\t/* trim whitespaces after ':' */\n+\t\tp++;\n+\t\twhile (isspace(*p))\n+\t\t\tp++;\n+\t\thex = p;\n+\t} else {\n+\t\tif (tag_p)\n+\t\t\t*tag_p = NULL;\n+\t\thex = line;\n+\t}\n+\n+\t/* rtrim */\n+\tp = hex;\n+\twhile (*p != '\\0' && !isspace(*p))\n+\t\tp++;\n+\t*p = '\\0';\n+\treturn hex;\n+}\n+\n+static void walk_xrefs(const char *tree_ref, struct object_id *from_oid,\n+\t\t struct strbuf *sb, int level)\n+{\n+\tstruct object_array xrefs = OBJECT_ARRAY_INIT;\n+\tstruct string_list lines = STRING_LIST_INIT_DUP;\n+\tint i;\n+\n+\tread_xref_note(tree_ref, from_oid, &xrefs, &lines);\n+\n+\tfor (i = 0; i < xrefs.nr; i++) {\n+\t\tchar *line = lines.items[i].string;\n+\t\tchar *tag;\n+\n+\t\tparse_xref(line, &tag);\n+\t\tstrbuf_addf(sb, \" %s%s%*s%s\\n\",\n+\t\t\t tag ?: \"\", tag ? \": \" : \"\", 2 * level, \"\",\n+\t\t\t xrefs.objects[i].name);\n+\t\tif (xrefs.objects[i].item) {\n+\t\t\tif (level < 32)\n+\t\t\t\twalk_xrefs(tree_ref, &xrefs.objects[i].item->oid,\n+\t\t\t\t\t sb, level + 1);\n+\t\t\telse\n+\t\t\t\twarning(\"xref nested deeper than %d levels, terminating walk\",\n+\t\t\t\t\tlevel);\n+\t\t}\n+\t}\n+\n+\tobject_array_clear(&xrefs);\n+\tstring_list_clear(&lines, 0);\n+}\n+\n /*\n * Fill the given strbuf with the notes associated with the given object.\n *\n@@ -1208,6 +1283,7 @@ static void format_note(struct notes_tree *t, const struct object_id *object_oid\n \tchar *msg, *msg_p;\n \tunsigned long linelen, msglen;\n \tenum object_type type;\n+\tint format_xrefs;\n \n \tif (!t)\n \t\tt = &default_notes_tree;\n@@ -1250,6 +1326,8 @@ static void format_note(struct notes_tree *t, const struct object_id *object_oid\n \t\t}\n \t}\n \n+\tformat_xrefs = !raw && starts_with(t->ref, \"refs/notes/xref-\");\n+\n \tfor (msg_p = msg; msg_p < msg + msglen; msg_p += linelen + 1) {\n \t\tlinelen = strchrnul(msg_p, '\\n') - msg_p;\n \n@@ -1257,6 +1335,14 @@ static void format_note(struct notes_tree *t, const struct object_id *object_oid\n \t\t\tstrbuf_addstr(sb, \" \");\n \t\tstrbuf_add(sb, msg_p, linelen);\n \t\tstrbuf_addch(sb, '\\n');\n+\n+\t\tif (format_xrefs) {\n+\t\t\tstruct object_id oid;\n+\n+\t\t\tmsg_p[linelen] = '\\0';\n+\t\t\tif (!get_oid_hex(parse_xref(msg_p, NULL), &oid))\n+\t\t\t\twalk_xrefs(t->ref, &oid, sb, 1);\n+\t\t}\n \t}\n \n \tfree(msg);\n@@ -1309,3 +1395,173 @@ void expand_loose_notes_ref(struct strbuf *sb)\n \t\texpand_notes_ref(sb);\n \t}\n }\n+\n+/*\n+ * Parse a cross-referencing note.\n+ *\n+ * @note contains lines of \"[TAG:]HEX\" pointing to other commits. Read the\n+ * target commits and add the objects to @result. If @result_lines is not\n+ * NULL, it should point to a strdup'ing string_list. The verbatim note\n+ * lines matching the target commits are appened to the list.\n+ */\n+static void parse_xref_note(const char *note, unsigned long size,\n+\t\t\t const struct object_id *commit_oid,\n+\t\t\t struct object_array *result,\n+\t\t\t struct string_list *result_lines)\n+{\n+\tstruct strbuf **lines, **pline;\n+\n+\tlines = strbuf_split_buf(note, size, '\\n', 0);\n+\n+\tfor (pline = lines; *pline; pline++) {\n+\t\tstruct strbuf *line = *pline;\n+\t\tconst char *target_hex;\n+\t\tstruct object_id target_oid;\n+\t\tstruct object *target_obj;\n+\n+\t\ttarget_hex = parse_xref(line->buf, NULL);\n+\t\tif (get_oid_hex(target_hex, &target_oid)) {\n+\t\t\tif (commit_oid)\n+\t\t\t\twarning(\"read invalid sha1 on %s: %s\",\n+\t\t\t\t\toid_to_hex(commit_oid), line->buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\ttarget_obj = parse_object(the_repository, &target_oid);\n+\t\tif (!target_obj || target_obj->type != OBJ_COMMIT) {\n+\t\t\tif (commit_oid)\n+\t\t\t\twarning(\"read invalid commit on %s: %s\",\n+\t\t\t\t\toid_to_hex(commit_oid), line->buf);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tadd_object_array(target_obj, target_hex, result);\n+\t\tif (result_lines) {\n+\t\t\tassert(result_lines->strdup_strings);\n+\t\t\tstring_list_append(result_lines, line->buf);\n+\t\t}\n+\t}\n+\n+\tstrbuf_list_free(lines);\n+}\n+\n+struct notes_tree_entry {\n+\tstruct hashmap_entry ent;\n+\tstruct notes_tree tree;\n+};\n+\n+static int notes_tree_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t const void *entry, const void *entry_or_key,\n+\t\t\t const void *keydata)\n+{\n+\tconst struct notes_tree_entry *e1 = entry;\n+\tconst struct notes_tree_entry *e2 = entry_or_key;\n+\n+\treturn strcmp(e1->tree.ref, e2->tree.ref);\n+}\n+\n+/*\n+ * Read and parse a cross-referencing note.\n+ *\n+ * Read the @notes_ref note of @commit_oid and parse it with\n+ * parse_xref_note().\n+ */\n+void read_xref_note(const char *notes_ref, const struct object_id *commit_oid,\n+\t\t struct object_array *result,\n+\t\t struct string_list *result_lines)\n+{\n+\tstatic struct hashmap *notes_tree_map = NULL;\n+\tunsigned hash = memhash(notes_ref, strlen(notes_ref));\n+\tstruct notes_tree_entry key, *ent;\n+\tconst struct object_id *note_oid;\n+\tunsigned long size;\n+\tenum object_type type;\n+\tchar *note;\n+\n+\tif (!notes_tree_map) {\n+\t\tnotes_tree_map = xcalloc(1, sizeof(struct hashmap));\n+\t\thashmap_init(notes_tree_map, notes_tree_cmp, NULL, 0);\n+\t}\n+\n+\thashmap_entry_init(&key.ent, hash);\n+\tkey.tree.ref = (char *)notes_ref;\n+\tent = hashmap_get(notes_tree_map, &key, NULL);\n+\tif (!ent) {\n+\t\tent = xcalloc(1, sizeof(struct notes_tree_entry));\n+\t\tinit_notes(&ent->tree, notes_ref, NULL, 0);\n+\t\thashmap_entry_init(&ent->ent, hash);\n+\t\thashmap_put(notes_tree_map, ent);\n+\t}\n+\n+\tnote_oid = get_note(&ent->tree, commit_oid);\n+\tif (!note_oid)\n+\t\treturn;\n+\n+\tnote = read_object_file(note_oid, &type, &size);\n+\tif (!size) {\n+\t\tfree(note);\n+\t\treturn;\n+\t}\n+\n+\tparse_xref_note(note, size, commit_oid, result, result_lines);\n+\tfree(note);\n+}\n+\n+/*\n+ * Combine a xref note in @new_oid into @cur_oid. Unreachable or duplicate\n+ * xrefs are dropped. This is the default combine_notes callback for\n+ * refs/notes/xref-.\n+ */\n+int combine_notes_cat_xrefs(struct object_id *cur_oid,\n+\t\t\t const struct object_id *new_oid)\n+{\n+\tchar *cur_msg = NULL, *new_msg = NULL;\n+\tunsigned long cur_len, new_len;\n+\tenum object_type cur_type, new_type;\n+\tstruct object_array xrefs = OBJECT_ARRAY_INIT;\n+\tstruct string_list lines = STRING_LIST_INIT_DUP;\n+\tstruct strbuf output = STRBUF_INIT;\n+\tint i, j, cur_nr, ret;\n+\n+\t/* read in both note blob objects */\n+\tif (!is_null_oid(new_oid))\n+\t\tnew_msg = read_object_file(new_oid, &new_type, &new_len);\n+\tif (!new_msg || !new_len || new_type != OBJ_BLOB) {\n+\t\tfree(new_msg);\n+\t\treturn 0;\n+\t}\n+\tif (!is_null_oid(cur_oid))\n+\t\tcur_msg = read_object_file(cur_oid, &cur_type, &cur_len);\n+\tif (!cur_msg || !cur_len || cur_type != OBJ_BLOB) {\n+\t\tfree(cur_msg);\n+\t\tfree(new_msg);\n+\t\toidcpy(cur_oid, new_oid);\n+\t\treturn 0;\n+\t}\n+\n+\t/* parse xrefs and de-dup */\n+\tparse_xref_note(cur_msg, cur_len, NULL, &xrefs, &lines);\n+\tcur_nr = xrefs.nr;\n+\tparse_xref_note(new_msg, new_len, NULL, &xrefs, &lines);\n+\n+\tfor (i = 0; i < cur_nr; i++)\n+\t\tfor (j = cur_nr; j < xrefs.nr; j++)\n+\t\t\tif (!strcmp(xrefs.objects[i].name,\n+\t\t\t\t xrefs.objects[j].name))\n+\t\t\t\tlines.items[j].string[0] = '\\0';\n+\n+\t/* write out the combined object */\n+\tfor (i = 0; i < lines.nr; i++)\n+\t\tif (lines.items[i].string[0] != '\\0')\n+\t\t\tstrbuf_addf(&output, \"%s\\n\", lines.items[i].string);\n+\n+\tret = write_object_file(output.buf, output.len, blob_type, cur_oid);\n+\n+\tstrbuf_release(&output);\n+\tobject_array_clear(&xrefs);\n+\tstring_list_clear(&lines, 0);\n+\tfree(cur_msg);\n+\tfree(new_msg);\n+\n+\treturn ret;\n+}\ndiff --git a/notes.h b/notes.h\nindex 414bc6855..09ca92f36 100644\n--- a/notes.h\n+++ b/notes.h\n@@ -2,6 +2,7 @@\n #define NOTES_H\n \n #include \"string-list.h\"\n+#include \"object.h\"\n \n struct object_id;\n struct strbuf;\n@@ -317,4 +318,12 @@ void expand_notes_ref(struct strbuf *sb);\n */\n void expand_loose_notes_ref(struct strbuf *sb);\n \n+/* For refs/notes/xref- */\n+void read_xref_note(const char *notes_ref, const struct object_id *commit_oid,\n+\t\t struct object_array *result,\n+\t\t struct string_list *result_lines);\n+\n+int combine_notes_cat_xrefs(struct object_id *cur_oid,\n+\t\t\t const struct object_id *new_oid);\n+\n #endif\n-- \n2.17.1\n\n"},{"id":"365136","messageId":"20181211234909.2855638-4-tj@kernel.org","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"[PATCH 3/5] notes: Implement git-reverse-trailer-xrefs","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:07Z","receivedAt":"2018-12-11T23:49:38Z","isPatch":true,"sender":{"key":"tj@kernel.org","avatar":null},"body":"From: Tejun Heo \n\nSome trailers refer to other commits. Let's call them xrefs\n(cross-references). For example, a cherry pick trailer points to the\nsource commit. It is sometimes useful to build a reverse map of these\nxrefs - ie. source -> cherry-pick instead of cherry-pick -> source.\n\nThis patch implements git-reverse-trailer-xrefs which can process a\ngiven trailer for the specified commits and generate and update the\nmatching refs/notes/xref- notes.\n\nThe command reverse-maps trailers beginning with --trailer-prefix and\nstores them in --ref notes with each line tagged with --tag. It also\nhas a refs/notes/xref-cherry-picks preset for the three params.\n\nSigned-off-by: Tejun Heo \n---\n Documentation/git-reverse-trailer-xrefs.txt | 136 +++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/reverse-trailer-xrefs.c | 160 ++++++++++++++++++++\n git.c | 1 +\n 5 files changed, 299 insertions(+)\n create mode 100644 Documentation/git-reverse-trailer-xrefs.txt\n create mode 100644 builtin/reverse-trailer-xrefs.c\n\ndiff --git a/Documentation/git-reverse-trailer-xrefs.txt b/Documentation/git-reverse-trailer-xrefs.txt\nnew file mode 100644\nindex 000000000..20d260486\n--- /dev/null\n+++ b/Documentation/git-reverse-trailer-xrefs.txt\n@@ -0,0 +1,136 @@\n+git-reverse-trailer-xrefs(1)\n+============================\n+\n+NAME\n+----\n+git-reverse-trailer-xrefs - Record reverse-map of trailer commit references into notes\n+\n+SYNOPSIS\n+--------\n+[verse]\n+`git reverse_trailer_xrefs` --xref-cherry-picks [--clear] [] [...]\n+`git reverse_trailer_xrefs` --trailer-prefix= --ref= [--tag=] [] [...]\n+`git reverse_trailer_xrefs` --ref= --clear [] [...]\n+\n+\n+DESCRIPTION\n+-----------\n+Record or clear reverse-map of trailer commit references in the\n+specified notes ref.\n+\n+Some commit trailers reference other commits. For example,\n+`git-cherry-pick -x` adds the following trailer to record the source\n+commit.\n+----------\n+(cherry picked from commit )\n+----------\n+The reverse mappings of such cross references can be useful. For\n+cherry-picks, it would allow finding all the cherry-picked commits of\n+a given source commit. `git-reverse-trailer-xrefs` can be used to\n+create and maintain such reverse mappings in notes.\n+\n+When used with `--xref-cherry-picks`, the cherry-pick trailers are\n+parsed from the specified commits and the reverse mappings are\n+recorded in the `refs/notes/xref-cherry-picks` notes of the source\n+commits in the following format.\n+----------\n+Cherry-picked-to: \n+----------\n+\n+When a note with its notes ref starting with `refs/notes/xref-` is\n+formatted to be displayed with the commit for, e.g., `git-show` or\n+`git-log`, the destination commit is followed recursively and the\n+matching notes are shown with increasing level of indentations.\n+\n+`--trailer-prefix`, `--notes` and `--tag` can be used to use a custom\n+set of trailer, notes ref and reverse mapping tag.\n+\n+OPTIONS\n+-------\n+...::\n+\tCommit-ish object names to describe. Defaults to HEAD if omitted.\n+\n+--xref-cherry-picks::\n+\tUse the preset to reverse map `git-cherry-pick -x`\n+\ttrailers. `--trailer-prefix` is set to `(cherry-picked from\n+\tcommit `, `--notes` is set to `refs/notes/xref-cherry-picks`\n+\tand `--tag` is set to `Cherry-picked-to`. This option can't be\n+\tspecified with the three preset options.\n+\n+--trailer-prefix=::\n+\tProcess trailers which start with . It is matched\n+\tcharacter-by-character and should be followed by the\n+\treferenced commit ID. When there are multiple matching\n+\ttrailers, the last one is used.\n+\n+--notes=::\n+\tThe notes ref to use for the reverse mapping. While this can\n+\tbe any notes ref, it is recommented to use ones starting with\n+\t`refs/notes/xref-` as they are recognized as cross-referencing\n+\tnotes and handled specially when updating and showing.\n+\n+--tag=::\n+\tOptional tag to use when generating reverse reference\n+\tnotes. If specified, each note line is formatted as `:\n+\t`; otherwise, ``.\n+\n+--clear::\n+\tInstead of creating reverse mapping notes, clear them from the\n+\tspecified commits.\n+\n+\n+EXAMPLES\n+--------\n+\n+Assume the following history. Development is happening in \"master\" and\n+releases are branched off and fixes are cherry-picked into them.\n+\n+------------\n+\tD'---E'' release-B\n+ /\n+ C' E' release-D\n+ / /\n+A---B---C---D---E master\n+------------\n+\n+The following cherry-picks took place.\n+\n+------------\n+C -> C'\n+D -> D'\n+E -> E' -> E''\n+------------\n+\n+The reverse mappings for all commits can be created using the\n+following command.\n+\n+------------\n+$ git reverse-trailer-xrefs --all --xref-cherry-picks\n+------------\n+\n+With the notes added, where each commit ended up can be easily\n+determined.\n+\n+------------\n+$ git log --notes=xref-cherry-picks --oneline | git name-rev --name-only --stdin\n+4b165af commit E\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-D\n+ Cherry-picked-to: release-B\n+\n+82bf1f3 commit D\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-B~1\n+\n+364eccf commit C\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-B~2\n+\n+ed3adf3 commit B\n+ddd1bf2 commit A\n+------------\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 1a44c811a..3c23ecf9d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1086,6 +1086,7 @@ BUILTIN_OBJS += builtin/multi-pack-index.o\n BUILTIN_OBJS += builtin/mv.o\n BUILTIN_OBJS += builtin/name-rev.o\n BUILTIN_OBJS += builtin/notes.o\n+BUILTIN_OBJS += builtin/reverse-trailer-xrefs.o\n BUILTIN_OBJS += builtin/pack-objects.o\n BUILTIN_OBJS += builtin/pack-redundant.o\n BUILTIN_OBJS += builtin/pack-refs.o\ndiff --git a/builtin.h b/builtin.h\nindex 6538932e9..51089e258 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -195,6 +195,7 @@ extern int cmd_multi_pack_index(int argc, const char **argv, const char *prefix)\n extern int cmd_mv(int argc, const char **argv, const char *prefix);\n extern int cmd_name_rev(int argc, const char **argv, const char *prefix);\n extern int cmd_notes(int argc, const char **argv, const char *prefix);\n+extern int cmd_reverse_trailer_xrefs(int argc, const char **argv, const char *prefix);\n extern int cmd_pack_objects(int argc, const char **argv, const char *prefix);\n extern int cmd_pack_redundant(int argc, const char **argv, const char *prefix);\n extern int cmd_patch_id(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/reverse-trailer-xrefs.c b/builtin/reverse-trailer-xrefs.c\nnew file mode 100644\nindex 000000000..0edb3b91a\n--- /dev/null\n+++ b/builtin/reverse-trailer-xrefs.c\n@@ -0,0 +1,160 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"strbuf.h\"\n+#include \"repository.h\"\n+#include \"config.h\"\n+#include \"commit.h\"\n+#include \"blob.h\"\n+#include \"notes.h\"\n+#include \"notes-utils.h\"\n+#include \"trailer.h\"\n+#include \"revision.h\"\n+#include \"list-objects.h\"\n+#include \"object-store.h\"\n+#include \"parse-options.h\"\n+\n+static const char * const reverse_trailer_xrefs_usage[] = {\n+\tN_(\"git reverse_trailer_xrefs --xref-cherry-picks [--clear] [] [...]\"),\n+\tN_(\"git reverse_trailer_xrefs --trailer-prefix= --notes= --tag= [] [...]\"),\n+\tN_(\"git reverse_trailer_xrefs --notes= --clear [] [...]\"),\n+\tNULL\n+};\n+\n+#define CHERRY_PICKED_PREFIX\t\"(cherry picked from commit \"\n+#define CHERRY_PICKS_REF\t\"refs/notes/xref-cherry-picks\"\n+#define CHERRY_PICKED_TO_TAG\t\"Cherry-picked-to\"\n+\n+static int verbose;\n+\n+static void clear_trailer_xref_note(struct commit *commit, void *data)\n+{\n+\tstruct notes_tree *tree = data;\n+\tint status;\n+\n+\tstatus = remove_note(tree, commit->object.oid.hash);\n+\n+\tif (verbose) {\n+\t\tif (status)\n+\t\t\tfprintf(stderr, \"Object %s has no note\\n\",\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t\telse\n+\t\t\tfprintf(stderr, \"Removing note for object %s\\n\",\n+\t\t\t\toid_to_hex(&commit->object.oid));\n+\t}\n+}\n+\n+static void record_trailer_xrefs(struct commit *commit, void *data)\n+{\n+\ttrailer_rev_xrefs_record(data, commit);\n+}\n+\n+static int note_trailer_xrefs(struct notes_tree *tree,\n+\t\t\t struct commit *dst_commit,\n+\t\t\t struct object_array *src_objs, const char *tag)\n+{\n+\tchar dst_hex[GIT_MAX_HEXSZ + 1];\n+\tstruct strbuf note = STRBUF_INIT;\n+\tstruct object_id note_oid;\n+\tint i, ret;\n+\n+\toid_to_hex_r(dst_hex, &dst_commit->object.oid);\n+\n+\tfor (i = 0; i < src_objs->nr; i++) {\n+\t\tconst char *hex = src_objs->objects[i].name;\n+\n+\t\tif (tag)\n+\t\t\tstrbuf_addf(¬e, \"%s: %s\\n\", tag, hex);\n+\t\telse\n+\t\t\tstrbuf_addf(¬e, \"%s\\n\", tag);\n+\t\tif (verbose)\n+\t\t\tfprintf(stderr, \"Adding note %s -> %s\\n\", dst_hex, hex);\n+\t}\n+\n+\tret = write_object_file(note.buf, note.len, blob_type, ¬e_oid);\n+\tstrbuf_release(¬e);\n+\tif (ret)\n+\t\treturn ret;\n+\n+\tret = add_note(tree, &dst_commit->object.oid, ¬e_oid, NULL);\n+\treturn ret;\n+}\n+\n+static int reverse_trailer_xrefs(struct rev_info *revs, int clear,\n+\t\t\t\t const char *trailer_prefix,\n+\t\t\t\t const char *notes_ref, const char *tag)\n+{\n+\tstruct notes_tree tree = { };\n+\tint i, ret;\n+\n+\tinit_notes(&tree, notes_ref, NULL, NOTES_INIT_WRITABLE);\n+\n+\tif (prepare_revision_walk(revs))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\tif (clear) {\n+\t\ttraverse_commit_list(revs, clear_trailer_xref_note, NULL, &tree);\n+\t} else {\n+\t\tstruct trailer_rev_xrefs rxrefs;\n+\t\tstruct commit *dst_commit;\n+\t\tstruct object_array *src_objs;\n+\n+\t\ttrailer_rev_xrefs_init(&rxrefs, trailer_prefix);\n+\t\ttraverse_commit_list(revs, record_trailer_xrefs, NULL, &rxrefs);\n+\n+\t\ttrailer_rev_xrefs_for_each(&rxrefs, i, dst_commit, src_objs) {\n+\t\t\tret = note_trailer_xrefs(&tree, dst_commit, src_objs,\n+\t\t\t\t\t\t tag);\n+\t\t\tif (ret)\n+\t\t\t\treturn ret;\n+\t\t}\n+\n+\t\ttrailer_rev_xrefs_release(&rxrefs);\n+\t}\n+\n+\tcommit_notes(&tree, \"Notes updated by 'git reverse-trailer-xrefs'\");\n+\treturn 0;\n+}\n+\n+int cmd_reverse_trailer_xrefs(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct rev_info revs;\n+\tstruct setup_revision_opt s_r_opt = {\n+\t\t.def = \"HEAD\",\n+\t\t.revarg_opt = REVARG_CANNOT_BE_FILENAME\n+\t};\n+\tint cherry = 0, clear = 0;\n+\tconst char *trailer_prefix = NULL, *notes_ref = NULL, *tag = NULL;\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"xref-cherry-picks\", &cherry, N_(\"use preset for xref-cherry-picks notes\")),\n+\t\tOPT_STRING(0, \"trailer-prefix\", &trailer_prefix, N_(\"prefix\"), N_(\"process trailers starting with \")),\n+\t\tOPT_STRING(0, \"notes\", ¬es_ref, N_(\"notes-ref\"), N_(\"update notes in \")),\n+\t\tOPT_STRING(0, \"tag\", &tag, N_(\"tag\"), N_(\"tag xref notes with \")),\n+\t\tOPT_BOOL(0, \"clear\", &clear, N_(\"clear trailer xref notes from the specified commits\")),\n+\t\tOPT__VERBOSE(&verbose, N_(\"verbose\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tgit_config(git_default_config, NULL);\n+\n+\targc = parse_options(argc, argv, prefix, options,\n+\t\t\t reverse_trailer_xrefs_usage,\n+\t\t\t PARSE_OPT_KEEP_ARGV0 | PARSE_OPT_KEEP_UNKNOWN);\n+\n+\tinit_revisions(&revs, prefix);\n+\targc = setup_revisions(argc, argv, &revs, &s_r_opt);\n+\n+\tif (cherry) {\n+\t\ttrailer_prefix = CHERRY_PICKED_PREFIX;\n+\t\tnotes_ref = CHERRY_PICKS_REF;\n+\t\ttag = CHERRY_PICKED_TO_TAG;\n+\t}\n+\n+\tif (!notes_ref || (!clear && (!trailer_prefix || !tag)))\n+\t\tdie(_(\"insufficient arguments\"));\n+\n+\tif (argc > 1)\n+\t\tdie(_(\"unrecognized argument: %s\"), argv[1]);\n+\n+\treturn reverse_trailer_xrefs(&revs, clear,\n+\t\t\t\t trailer_prefix, notes_ref, tag);\n+}\ndiff --git a/git.c b/git.c\nindex 2f604a41e..4948c8e01 100644\n--- a/git.c\n+++ b/git.c\n@@ -515,6 +515,7 @@ static struct cmd_struct commands[] = {\n \t{ \"mv\", cmd_mv, RUN_SETUP | NEED_WORK_TREE },\n \t{ \"name-rev\", cmd_name_rev, RUN_SETUP },\n \t{ \"notes\", cmd_notes, RUN_SETUP },\n+\t{ \"reverse-trailer-xrefs\", cmd_reverse_trailer_xrefs, RUN_SETUP },\n \t{ \"pack-objects\", cmd_pack_objects, RUN_SETUP },\n \t{ \"pack-redundant\", cmd_pack_redundant, RUN_SETUP | NO_PARSEOPT },\n \t{ \"pack-refs\", cmd_pack_refs, RUN_SETUP },\n-- \n2.17.1\n\n"},{"id":"365137","messageId":"20181211234909.2855638-5-tj@kernel.org","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"[PATCH 4/5] githooks: Add post-cherry-pick and post-fetch hooks","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:08Z","receivedAt":"2018-12-11T23:49:41Z","isPatch":true,"sender":{"key":"tj@kernel.org","avatar":null},"body":"From: Tejun Heo \n\n* post-cherry-pick: Called after a cherry-pick and given parameters so\n that it can tell which are the new cherry-picks.\n\n* post-fetch: Called after a fetch. Each updated ref and sha1 are fed\n on stdin.\n\nThese two hooks will be used to keep refs/notes/xref-cherry-picks\nup-to-date.\n\nSigned-off-by: Tejun Heo \n---\n Documentation/git-cherry-pick.txt | 5 +++\n Documentation/git-fetch.txt | 5 +++\n Documentation/githooks.txt | 23 ++++++++++\n builtin/fetch.c | 72 ++++++++++++++++++++++++++++---\n builtin/revert.c | 14 ++++++\n 5 files changed, 114 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-cherry-pick.txt b/Documentation/git-cherry-pick.txt\nindex d35d771fc..527cb9fea 100644\n--- a/Documentation/git-cherry-pick.txt\n+++ b/Documentation/git-cherry-pick.txt\n@@ -224,6 +224,11 @@ the working tree.\n spending extra time to avoid mistakes based on incorrectly matching\n context lines.\n \n+HOOKS\n+-----\n+This command can run `post-cherry-pick` hook. See linkgit:githooks[5]\n+for more information.\n+\n SEE ALSO\n --------\n linkgit:git-revert[1]\ndiff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt\nindex e31993559..a04c6079a 100644\n--- a/Documentation/git-fetch.txt\n+++ b/Documentation/git-fetch.txt\n@@ -290,6 +290,11 @@ fetched, making it impossible to check out that submodule later without\n having to do a fetch again. This is expected to be fixed in a future Git\n version.\n \n+HOOKS\n+-----\n+This command can run `post-fetch` hook. See linkgit:githooks[5]\n+for more information.\n+\n SEE ALSO\n --------\n linkgit:git-pull[1]\ndiff --git a/Documentation/githooks.txt b/Documentation/githooks.txt\nindex 959044347..24c122343 100644\n--- a/Documentation/githooks.txt\n+++ b/Documentation/githooks.txt\n@@ -149,6 +149,14 @@ invoked after a commit is made.\n This hook is meant primarily for notification, and cannot affect\n the outcome of `git commit`.\n \n+post-cherry-pick\n+~~~~~~~~~~~~~~~~\n+\n+This hook is invoked by linkgit:git-cherry-pick[1]. This hook is\n+called with two parameters. The first is `` and the second\n+``, where `..` describes all new\n+cherry-picked commits.\n+\n pre-rebase\n ~~~~~~~~~~\n \n@@ -191,6 +199,21 @@ save and restore any form of metadata associated with the working tree\n (e.g.: permissions/ownership, ACLS, etc). See contrib/hooks/setgitperms.perl\n for an example of how to do this.\n \n+post-fetch\n+~~~~~~~~~~\n+This hook is called by linkgit:git-fetch[1] and can be used to process\n+newly fetched commits and tags.\n+\n+Information about what was fetched is provided on the hook's standard\n+input with lines of the form:\n+\n+ SP SP SP LF\n+\n+where `` got updated from `` to `` as a\n+result of fetching ``. If a branch or tag was created,\n+`` will be 40 `0`. If a tag was pruned, `` will\n+be `(delete)` and will be 40 `0`.\n+\n pre-push\n ~~~~~~~~\n \ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex e0140327a..eac792a33 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -66,6 +66,7 @@ static struct refspec refmap = REFSPEC_INIT_FETCH;\n static struct list_objects_filter_options filter_options;\n static struct string_list server_options = STRING_LIST_INIT_DUP;\n static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP;\n+static struct strbuf post_fetch_sb = STRBUF_INIT;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -510,10 +511,24 @@ static struct ref *get_ref_map(struct remote *remote,\n \treturn ref_map;\n }\n \n+static void record_post_fetch(const char *name,\n+\t\t\t const struct object_id *old_oid,\n+\t\t\t const char *remote,\n+\t\t\t const struct object_id *new_oid)\n+{\n+\tchar old_hex[GIT_MAX_HEXSZ + 1], new_hex[GIT_MAX_HEXSZ + 1];\n+\n+\toid_to_hex_r(old_hex, old_oid);\n+\toid_to_hex_r(new_hex, new_oid);\n+\tstrbuf_addf(&post_fetch_sb, \"%s %s %s %s\\n\",\n+\t\t name, old_hex, remote ?: \"(delete)\", new_hex);\n+}\n+\n #define STORE_REF_ERROR_OTHER 1\n #define STORE_REF_ERROR_DF_CONFLICT 2\n \n static int s_update_ref(const char *action,\n+\t\t\tconst char *remote,\n \t\t\tstruct ref *ref,\n \t\t\tint check_old)\n {\n@@ -546,6 +561,7 @@ static int s_update_ref(const char *action,\n \tref_transaction_free(transaction);\n \tstrbuf_release(&err);\n \tfree(msg);\n+\trecord_post_fetch(ref->name, &ref->old_oid, remote, &ref->new_oid);\n \treturn 0;\n fail:\n \tref_transaction_free(transaction);\n@@ -726,7 +742,7 @@ static int update_local_ref(struct ref *ref,\n \t starts_with(ref->name, \"refs/tags/\")) {\n \t\tif (force || ref->force) {\n \t\t\tint r;\n-\t\t\tr = s_update_ref(\"updating tag\", ref, 0);\n+\t\t\tr = s_update_ref(\"updating tag\", remote, ref, 0);\n \t\t\tformat_display(display, r ? '!' : 't', _(\"[tag update]\"),\n \t\t\t\t r ? _(\"unable to update local ref\") : NULL,\n \t\t\t\t remote, pretty_ref, summary_width);\n@@ -766,7 +782,7 @@ static int update_local_ref(struct ref *ref,\n \t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n \t\t (recurse_submodules != RECURSE_SUBMODULES_ON))\n \t\t\tcheck_for_new_submodule_commits(&ref->new_oid);\n-\t\tr = s_update_ref(msg, ref, 0);\n+\t\tr = s_update_ref(msg, remote, ref, 0);\n \t\tformat_display(display, r ? '!' : '*', what,\n \t\t\t r ? _(\"unable to update local ref\") : NULL,\n \t\t\t remote, pretty_ref, summary_width);\n@@ -782,7 +798,7 @@ static int update_local_ref(struct ref *ref,\n \t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n \t\t (recurse_submodules != RECURSE_SUBMODULES_ON))\n \t\t\tcheck_for_new_submodule_commits(&ref->new_oid);\n-\t\tr = s_update_ref(\"fast-forward\", ref, 1);\n+\t\tr = s_update_ref(\"fast-forward\", remote, ref, 1);\n \t\tformat_display(display, r ? '!' : ' ', quickref.buf,\n \t\t\t r ? _(\"unable to update local ref\") : NULL,\n \t\t\t remote, pretty_ref, summary_width);\n@@ -797,7 +813,7 @@ static int update_local_ref(struct ref *ref,\n \t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n \t\t (recurse_submodules != RECURSE_SUBMODULES_ON))\n \t\t\tcheck_for_new_submodule_commits(&ref->new_oid);\n-\t\tr = s_update_ref(\"forced-update\", ref, 1);\n+\t\tr = s_update_ref(\"forced-update\", remote, ref, 1);\n \t\tformat_display(display, r ? '!' : '+', quickref.buf,\n \t\t\t r ? _(\"unable to update local ref\") : _(\"forced update\"),\n \t\t\t remote, pretty_ref, summary_width);\n@@ -1071,8 +1087,11 @@ static int prune_refs(struct refspec *rs, struct ref *ref_map,\n \tif (!dry_run) {\n \t\tstruct string_list refnames = STRING_LIST_INIT_NODUP;\n \n-\t\tfor (ref = stale_refs; ref; ref = ref->next)\n+\t\tfor (ref = stale_refs; ref; ref = ref->next) {\n \t\t\tstring_list_append(&refnames, ref->name);\n+\t\t\trecord_post_fetch(ref->name, &ref->old_oid,\n+\t\t\t\t\t NULL, &null_oid);\n+\t\t}\n \n \t\tresult = delete_refs(\"fetch: prune\", &refnames, 0);\n \t\tstring_list_clear(&refnames, 0);\n@@ -1561,6 +1580,47 @@ static int fetch_one(struct remote *remote, int argc, const char **argv, int pru\n \treturn exit_code;\n }\n \n+static int run_post_fetch_hook(void)\n+{\n+\tint ret = 0, x;\n+\tstruct child_process proc = CHILD_PROCESS_INIT;\n+\tconst char *argv[2];\n+\n+\tif (!(argv[0] = find_hook(\"post-fetch\")))\n+\t\treturn 0;\n+\targv[1] = NULL;\n+\n+\tproc.argv = argv;\n+\tproc.in = -1;\n+\n+\tif (start_command(&proc)) {\n+\t\tfinish_command(&proc);\n+\t\treturn -1;\n+\t}\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\tif (write_in_full(proc.in, post_fetch_sb.buf, post_fetch_sb.len) < 0) {\n+\t\t/* We do not mind if a hook does not read all refs. */\n+\t\tif (errno != EPIPE)\n+\t\t\tret = -1;\n+\t}\n+\n+\tstrbuf_release(&post_fetch_sb);\n+\n+\tx = close(proc.in);\n+\tif (!ret)\n+\t\tret = x;\n+\n+\tsigchain_pop(SIGPIPE);\n+\n+\tx = finish_command(&proc);\n+\tif (!ret)\n+\t\tret = x;\n+\n+\treturn ret;\n+}\n+\n int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -1669,6 +1729,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \tclose_all_packs(the_repository->objects);\n \n+\trun_post_fetch_hook();\n+\n \targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n \tif (verbosity < 0)\n \t\targv_array_push(&argv_gc_auto, \"--quiet\");\ndiff --git a/builtin/revert.c b/builtin/revert.c\nindex c93393c89..0b7e578cc 100644\n--- a/builtin/revert.c\n+++ b/builtin/revert.c\n@@ -8,6 +8,8 @@\n #include \"dir.h\"\n #include \"sequencer.h\"\n #include \"branch.h\"\n+#include \"refs.h\"\n+#include \"run-command.h\"\n \n /*\n * This implements the builtins revert and cherry-pick.\n@@ -223,12 +225,24 @@ int cmd_revert(int argc, const char **argv, const char *prefix)\n int cmd_cherry_pick(int argc, const char **argv, const char *prefix)\n {\n \tstruct replay_opts opts = REPLAY_OPTS_INIT;\n+\tstruct object_id old_oid, new_oid;\n+\tchar old_hex[GIT_MAX_HEXSZ + 1], new_hex[GIT_MAX_HEXSZ + 1];\n \tint res;\n \n+\tif (read_ref(\"HEAD\", &old_oid))\n+\t\tdie(_(\"failed to read HEAD, cherry-pick failed\"));\n+\n \topts.action = REPLAY_PICK;\n \tsequencer_init_config(&opts);\n \tres = run_sequencer(argc, argv, &opts);\n \tif (res < 0)\n \t\tdie(_(\"cherry-pick failed\"));\n+\n+\tif (read_ref(\"HEAD\", &new_oid))\n+\t\tdie(_(\"failed to read HEAD after cherry-pick\"));\n+\n+\toid_to_hex_r(old_hex, &old_oid);\n+\toid_to_hex_r(new_hex, &new_oid);\n+\trun_hook_le(0, \"post-cherry-pick\", old_hex, new_hex, NULL);\n \treturn res;\n }\n-- \n2.17.1\n\n"},{"id":"365138","messageId":"20181211234909.2855638-6-tj@kernel.org","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"[PATCH 5/5] notes: Implement xref-cherry-picks hooks and tests","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-11T23:49:09Z","receivedAt":"2018-12-11T23:49:45Z","isPatch":true,"sender":{"key":"tj@kernel.org","avatar":null},"body":"From: Tejun Heo \n\nAdd post-cherry-pick.sample and post-fetch.sample which, when enabled,\nwill keep refs/notes/xref-cherry-picks up-to-date as new cherry-picks\nare created and fetched. Also, add tests to verify xref-cherry-picks.\n\nSigned-off-by: Tejun Heo \n---\n Documentation/git-reverse-trailer-xrefs.txt | 9 ++\n t/t3321-notes-xref-cherry-picks.sh | 124 ++++++++++++++++++++\n templates/hooks--post-cherry-pick.sample | 8 ++\n templates/hooks--post-fetch.sample | 30 +++++\n 4 files changed, 171 insertions(+)\n create mode 100644 t/t3321-notes-xref-cherry-picks.sh\n create mode 100644 templates/hooks--post-cherry-pick.sample\n create mode 100644 templates/hooks--post-fetch.sample\n\ndiff --git a/Documentation/git-reverse-trailer-xrefs.txt b/Documentation/git-reverse-trailer-xrefs.txt\nindex 20d260486..651ecdce1 100644\n--- a/Documentation/git-reverse-trailer-xrefs.txt\n+++ b/Documentation/git-reverse-trailer-xrefs.txt\n@@ -131,6 +131,15 @@ ddd1bf2 commit A\n ------------\n \n \n+Keeping xref-cherry-picks up-to-date\n+------------------------------------\n+\n+Reverse-maps can be kept-up incrementally with hooks. For example, to\n+keep xref-cherry-picks up-to-date, `git-reverse-trailer-xrefs` should\n+be invoked on new cherry-picks and fetched commits. See\n+`hooks/post-cherry-pick.sample` and `hooks/post-fetch.sample`.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/t/t3321-notes-xref-cherry-picks.sh b/t/t3321-notes-xref-cherry-picks.sh\nnew file mode 100644\nindex 000000000..96b6731c9\n--- /dev/null\n+++ b/t/t3321-notes-xref-cherry-picks.sh\n@@ -0,0 +1,124 @@\n+#!/bin/sh\n+\n+test_description='Verify xref-cherry-picks handling\n+\n+Assume the following git repository.\n+\n+\t D*---E** release-B\n+\t /\n+\tC* E* release-D\n+ / /\n+ A---B---C---D---E master\n+\n+which contains the following cherry-picks.\n+\n+ C -> C*\n+ D -> D*\n+ E -> E* -> E**\n+\n+1. Build the above repository using `git-cherry-pick -x` with the\n+ sample post-cherry-pick hook enabled. Verify that the\n+ xref-cherry-picks notes are populated correctly.\n+\n+2. Clear the notes and rebuild them by directly running\n+ git-reverse-xref-trailers and verify the output.\n+\n+3. Run it again and check the output still agrees to verify duplicate\n+ handling.\n+\n+4. Build a cloned repository using per-branch fetches with the sample\n+ post-fetch hook enabled. Verify that the xref-cherry-picks notes\n+ are populatec correctly.\n+'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+GIT_AUTHOR_EMAIL=bogus_email_address\n+export GIT_AUTHOR_EMAIL\n+\n+test_expect_success \\\n+ 'Build repo with cherry-picks and verify xref-cherry-picks' \\\n+ 'test_create_repo main &&\n+ cd main &&\n+ mkdir -p .git/hooks &&\n+ mv .git/hooks-disabled/post-cherry-pick.sample .git/hooks/post-cherry-pick &&\n+\n+ test_tick &&\n+ echo A >> testfile &&\n+ git update-index --add testfile &&\n+ git commit -am \"A\" &&\n+ echo B >> testfile &&\n+ git commit -am \"B\" &&\n+ echo C >> testfile &&\n+ git commit -am \"C\" &&\n+ echo D >> testfile &&\n+ git commit -am \"D\" &&\n+ echo E >> testfile &&\n+ git commit -am \"E\" &&\n+\n+ test_tick &&\n+ git checkout -b release-D master^ &&\n+ git cherry-pick -x master &&\n+\n+ test_tick &&\n+ git checkout -b release-B master^^^ &&\n+ git cherry-pick -x release-D^^ &&\n+ git cherry-pick -x release-D^ &&\n+ git cherry-pick -x release-D &&\n+\n+ cat > expect <<-EOF &&\n+master E\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-D\n+ Cherry-picked-to: release-B\n+\n+master~1 D\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-B~1\n+\n+master~2 C\n+Notes (xref-cherry-picks):\n+ Cherry-picked-to: release-B~2\n+\n+master~3 B\n+master~4 A\n+EOF\n+\n+ git log --pretty=oneline --notes=xref-cherry-picks master | git name-rev --name-only --stdin > actual &&\n+ test_cmp expect actual\n+'\n+\n+test_expect_success \\\n+ 'Clear, rebuild and verify xref-cherry-picks' \\\n+ 'git reverse-trailer-xrefs --xref-cherry-picks --all --clear &&\n+ git reverse-trailer-xrefs --xref-cherry-picks --all &&\n+ git log --pretty=oneline --notes=xref-cherry-picks master | git name-rev --name-only --stdin > actual &&\n+ test_cmp expect actual\n+'\n+\n+test_expect_success \\\n+ 'Build it again to verify duplicate handling' \\\n+ 'git reverse-trailer-xrefs --xref-cherry-picks --all &&\n+ git log --pretty=oneline --notes=xref-cherry-picks master | git name-rev --name-only --stdin > actual &&\n+ test_cmp expect actual\n+'\n+\n+test_expect_success \\\n+ 'Build a clone through per-branch fetches and verify xref-cherry-picks' \\\n+ 'cd .. &&\n+ test_create_repo clone &&\n+ cd clone &&\n+ mkdir -p .git/hooks &&\n+ mv .git/hooks-disabled/post-fetch.sample .git/hooks/post-fetch &&\n+\n+ git fetch -fu ../main master:master &&\n+ git fetch -fu ../main release-D:release-D &&\n+ git fetch -fu ../main release-B:release-B &&\n+\n+ git log --pretty=oneline --notes=xref-cherry-picks master | git name-rev --name-only --stdin > actual &&\n+ test_cmp ../main/expect actual\n+'\n+\n+test_done\ndiff --git a/templates/hooks--post-cherry-pick.sample b/templates/hooks--post-cherry-pick.sample\nnew file mode 100644\nindex 000000000..3af8b5d23\n--- /dev/null\n+++ b/templates/hooks--post-cherry-pick.sample\n@@ -0,0 +1,8 @@\n+#!/bin/sh\n+#\n+# An example hook script to reverse map new cherry-picks. See\n+# git-reverse-trailer-xrefs(1) for details.\n+#\n+# To enable this hook, rename this file to \"post-cherry-pick\".\n+\n+git reverse-trailer-xrefs --xref-cherry-picks $1..$2\ndiff --git a/templates/hooks--post-fetch.sample b/templates/hooks--post-fetch.sample\nnew file mode 100644\nindex 000000000..6b98a5c10\n--- /dev/null\n+++ b/templates/hooks--post-fetch.sample\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+#\n+# An example hook script to reverse map cherry-picks in newly fetched\n+# commits. See git-reverse-trailer-xrefs(1) for details.\n+#\n+# To enable this hook, rename this file to \"post-fetch\".\n+\n+z40=0000000000000000000000000000000000000000\n+\n+while read ref old_sha remote new_sha\n+do\n+ case $ref in\n+ refs/heads/*)\n+\t;;\n+ *)\n+\tcontinue\n+ esac\n+\n+ if [ $new_sha == $z40 ]\n+ then\n+\tcontinue\n+ fi\n+\n+ if [ $old_sha != $z40 ]\n+ then\n+\tgit reverse-trailer-xrefs --xref-cherry-picks $old_sha..$new_sha\n+ else\n+\tgit reverse-trailer-xrefs --xref-cherry-picks $new_sha\n+ fi\n+done\n-- \n2.17.1\n\n"},{"id":"365154","messageId":"xmqqo99rjjcu.fsf@gitster-ct.c.googlers.com","threadId":"50012","inReplyTo":"20181211234909.2855638-1-tj@kernel.org","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-12T07:26:57Z","receivedAt":"2018-12-12T07:27:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tejun Heo writes:\n\n> Some trailers refer to other commits. Let's call them xrefs\n> (cross-references). For example, a cherry pick trailer points to the\n> source commit. It is sometimes useful to build a reverse map of these\n> xrefs - ie. source -> cherry-pick instead of cherry-pick -> source.\n>\n> This, e.g, can answer which releases a commit ended up in on\n> repositories which cherry-picks fixes back from the development\n> branch. Let's say the repository looks like the following.\n>\n> \t D'---E'' release-B\n> \t /\n> \t C' E' release-D\n> / /\n> A---B---C---D---E master\n>\n> where the followings cherry-picks took place.\n>\n> C -> C'\n> D -> D'\n> E -> E' -> E''\n>\n> Using the reverse-mapping in this patchset, we can get the following\n> annotated log which succinctly shows which commit ended up where.\n> ...\n\nThanks. A few comments, after skimming the patches enough to get\nthe general feel of the design but before deeply reading them to see\nhow complete the implementation is.\n\nThe topic introduces two new hooks: one to run immediately after\ncherry-picking so that the reverse mapping for a single src->dst\npair can be added, and another to run immediately after fetching so\nthat all new commits that have appeared over the wire can be scanned\nto see if any of them is a cherry-pick of other commits.\n\nThey are good starting points, but for a complete solution I'd\nimagine that you'd need to hook to many other places in the\nworkflow. For example, you'd need to deal with the case where a\ncommit created by cherry-picking an original is further rewritten\nwith \"commit --amend\" or \"rebase\". They may already trigger the\npost rewrite hook, so you may not have to add a new hook, but in\n[3/5], you'd need to describe in the documentaiton for the new\nreverse-trailer command full set of hooks you expect the command to\nbe used to maintain the reverse mapping, as the hooks you give them\nas examples in [5/5] are merely a part of a complete solution.\n\nIt also is not immediately obvious to me what your general strategy\nto maintain this reverse mapping is, when new ways and codepaths to\ncause new commits with \"cherry-picked-from\" trailer appear. Do we\nkeep piling on new hooks? Or is the reverse mapping information\nallowed to be a bit stale and be completed immediately before it\ngets used? A totally different approach could be to record list of\ncommits, all commits behind which have been scanned for reverse\nmapping, in the tip of the notes history, perhaps in the commit log\nmessage (which is machine generated anyway). Then, before you need\nthe up-to-date-to-the-last-second reverse mapping, you could run\n\n\tgit rev-list --all --not $these_tips_recorded\n\nand scan the commits, just like you scan what you fetched. And when\nyou update the reverse mapping notes tree, the commit to record that\nnotes update can record the tip of the above traverseal.\n\nI also wonder how this topic and the topic Stefan Xenos has been\nworking on (cf. <20181201005240.113210-1-sxenos@google.com>) can\nbenefit from each other by cross pollination. Stefan's topic also\nneeds to answer, given a commit, what other commits were created out\nof it by cherry-picking and then further amending the results, so\nthat when the original commit itself gets amended, the cherry-picked\nones that were created from the original can be easily identified\nand get updated in the same way as necessary. \n\nThe motivating workflow your topic's cover letter gives us is to\nmaintain the release branch that goes only forward, so in that\ncontext, how a commit on the release branch that was created by\ncherry-picking an original gets updated when the original commit\ngets amended would be different (I am assuming that you cannot\naffored to rebase the release branch to update a deeply buried\ncommit that was an earlier cherry-pick), but I would imagine that\nboth topics share the need for a good data structure to keep track\nof (1) the relationship between the original and copy of the\ncherry-pick and (2) the fact that the original of such a cherry-pick\nis now stale and the copy might want to get updated.\n\nThanks.\n"},{"id":"365177","messageId":"20181212145456.GQ2509588@devbig004.ftw2.facebook.com","threadId":"50012","inReplyTo":"xmqqo99rjjcu.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-12T14:54:56Z","receivedAt":"2018-12-12T14:55:02Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hello, Junio.\n\nOn Wed, Dec 12, 2018 at 04:26:57PM +0900, Junio C Hamano wrote:\n> It also is not immediately obvious to me what your general strategy\n> to maintain this reverse mapping is, when new ways and codepaths to\n> cause new commits with \"cherry-picked-from\" trailer appear. Do we\n> keep piling on new hooks? Or is the reverse mapping information\n\nSo, what I actually wanted was \"this repo now has these new commits\"\nhook, but that didn't seem to be in line with other hooks as most were\nbound to specific directly user-visible commands and actions.\n\n> allowed to be a bit stale and be completed immediately before it\n> gets used? A totally different approach could be to record list of\n> commits, all commits behind which have been scanned for reverse\n> mapping, in the tip of the notes history, perhaps in the commit log\n> message (which is machine generated anyway). Then, before you need\n> the up-to-date-to-the-last-second reverse mapping, you could run\n> \n> \tgit rev-list --all --not $these_tips_recorded\n\nWouldn't it be more useful to have repo-updated-with-these-commits\nhook instead rather than putting more logic on note handling?\n\n> and scan the commits, just like you scan what you fetched. And when\n> you update the reverse mapping notes tree, the commit to record that\n> notes update can record the tip of the above traverseal.\n> \n> I also wonder how this topic and the topic Stefan Xenos has been\n> working on (cf. <20181201005240.113210-1-sxenos@google.com>) can\n> benefit from each other by cross pollination. Stefan's topic also\n> needs to answer, given a commit, what other commits were created out\n> of it by cherry-picking and then further amending the results, so\n> that when the original commit itself gets amended, the cherry-picked\n> ones that were created from the original can be easily identified\n> and get updated in the same way as necessary. \n\nAh, I see, forward-propagating amends to cherry-picks.\n\n> The motivating workflow your topic's cover letter gives us is to\n> maintain the release branch that goes only forward, so in that\n> context, how a commit on the release branch that was created by\n> cherry-picking an original gets updated when the original commit\n> gets amended would be different (I am assuming that you cannot\n\nAt least we don't do this with our trees and most trees that I've\nworked with don't either as that would make it really confusing for\npeople to work together.\n\n> affored to rebase the release branch to update a deeply buried\n> commit that was an earlier cherry-pick), but I would imagine that\n> both topics share the need for a good data structure to keep track\n> of (1) the relationship between the original and copy of the\n> cherry-pick and (2) the fact that the original of such a cherry-pick\n> is now stale and the copy might want to get updated.\n\nAs long as we can keep the reverse rference notes consistent, wouldn't\namend propagation just consume them?\n\nThanks.\n\n-- \ntejun\n"},{"id":"365217","messageId":"xmqqefamgmey.fsf@gitster-ct.c.googlers.com","threadId":"50012","inReplyTo":"20181212145456.GQ2509588@devbig004.ftw2.facebook.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-13T03:01:25Z","receivedAt":"2018-12-13T03:01:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tejun Heo writes:\n\n>> allowed to be a bit stale and be completed immediately before it\n>> gets used? A totally different approach could be to record list of\n>> commits, all commits behind which have been scanned for reverse\n>> mapping, in the tip of the notes history, perhaps in the commit log\n>> message (which is machine generated anyway). Then, before you need\n>> the up-to-date-to-the-last-second reverse mapping, you could run\n>> \n>> \tgit rev-list --all --not $these_tips_recorded\n>\n> Wouldn't it be more useful to have repo-updated-with-these-commits\n> hook instead rather than putting more logic on note handling?\n>\n>> and scan the commits, just like you scan what you fetched. And when\n>> you update the reverse mapping notes tree, the commit to record that\n>> notes update can record the tip of the above traversal.\n\nI do not consider what you do in notes/xref-* \"more logic on note\nhandling\" in the sense that the logic is part of \"notes\" API.\n\nThe moment you decided to reserve one hierarchy in refs/notes/ and\ndesigned what the mapping recorded there means, you designed a new\ntrailer-xrefs API. It is a part of that API how blobs stored in\nyour refs/notes/xref-cherry-picks are formatted and what they mean.\nIt's the same thing---it is also part of your API how the log\nmessage for recording commits in that hierarchy is formatted and\nwhat it means.\n\n> As long as we can keep the reverse rference notes consistent, wouldn't\n> amend propagation just consume them?\n\nYes. Would that mean you do not need the notes/xref-* series we are\nseeing here, and instead (re)use what Stefan's series, which already\nneeds to have access to and record the information anyway, records?\n"},{"id":"365218","messageId":"xmqqa7lagm18.fsf@gitster-ct.c.googlers.com","threadId":"50012","inReplyTo":"xmqqefamgmey.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-13T03:09:39Z","receivedAt":"2018-12-13T03:09:44Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> As long as we can keep the reverse rference notes consistent, wouldn't\n>> amend propagation just consume them?\n>\n> Yes. Would that mean you do not need the notes/xref-* series we are\n> seeing here, and instead (re)use what Stefan's series, which already\n> needs to have access to and record the information anyway, records?\n\nJust to avoid confusion or unnecessary guessing.\n\nPlease do not take the above as \"don't do notes/xref-; instead read\nfrom the 'meta commits'\". I do not have a preference between the\ntwo proposed implementations. The important thing is that we won't\nend up with having to maintain two separate mechanisms that want to\nkeep track of essentially the same class of information. FWIW I'd\nbe perfectly fine if the unification goes the other way, as long as\ngoals of both parties are met, and for that, I'd like to see you two\nwork together, or at least be aware of what each other is doing and\nsee if cross-pollination would result in a mutually better solution.\n\nThanks\n"},{"id":"365221","messageId":"20181213034041.GR2509588@devbig004.ftw2.facebook.com","threadId":"50012","inReplyTo":"xmqqefamgmey.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-13T03:40:41Z","receivedAt":"2018-12-13T03:40:52Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hello, Junio.\n\nOn Thu, Dec 13, 2018 at 12:01:25PM +0900, Junio C Hamano wrote:\n> > Wouldn't it be more useful to have repo-updated-with-these-commits\n> > hook instead rather than putting more logic on note handling?\n> >\n> >> and scan the commits, just like you scan what you fetched. And when\n> >> you update the reverse mapping notes tree, the commit to record that\n> >> notes update can record the tip of the above traversal.\n> \n> I do not consider what you do in notes/xref-* \"more logic on note\n> handling\" in the sense that the logic is part of \"notes\" API.\n> \n> The moment you decided to reserve one hierarchy in refs/notes/ and\n> designed what the mapping recorded there means, you designed a new\n> trailer-xrefs API. It is a part of that API how blobs stored in\n> your refs/notes/xref-cherry-picks are formatted and what they mean.\n> It's the same thing---it is also part of your API how the log\n> message for recording commits in that hierarchy is formatted and\n> what it means.\n\nHmmm... I see. I still have a bit of trouble seeing why doing it that\nway is better tho. Wouldn't new-object-hook be simpler? They'll\nachieve about the same thing but one would need to keep the states in\ntwo places.\n\n> > As long as we can keep the reverse rference notes consistent, wouldn't\n> > amend propagation just consume them?\n> \n> Yes. Would that mean you do not need the notes/xref-* series we are\n> seeing here, and instead (re)use what Stefan's series, which already\n> needs to have access to and record the information anyway, records?\n\nOh yeah, for sure. Didn't know Stefan was doing something similar.\nWill continue on the other reply.\n\nThanks.\n\n-- \ntejun\n"},{"id":"365222","messageId":"20181213034606.GS2509588@devbig004.ftw2.facebook.com","threadId":"50012","inReplyTo":"xmqqa7lagm18.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-13T03:46:06Z","receivedAt":"2018-12-13T03:46:12Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hello, Junio, Stefan.\n\nOn Thu, Dec 13, 2018 at 12:09:39PM +0900, Junio C Hamano wrote:\n> Please do not take the above as \"don't do notes/xref-; instead read\n> from the 'meta commits'\". I do not have a preference between the\n> two proposed implementations. The important thing is that we won't\n> end up with having to maintain two separate mechanisms that want to\n> keep track of essentially the same class of information. FWIW I'd\n> be perfectly fine if the unification goes the other way, as long as\n> goals of both parties are met, and for that, I'd like to see you two\n> work together, or at least be aware of what each other is doing and\n> see if cross-pollination would result in a mutually better solution.\n\nSo, from my POV, the only thing I want is being able to easily tell\nwhich commit got cherry picked where. I really don't care how that's\nachieved. Stefan, if there's anything I can do to push it forward,\nlet me know and if you see anything useful in my patchset, please feel\nfree to incorporate them in any way.\n\nThanks.\n\n-- \ntejun\n"},{"id":"365226","messageId":"xmqqftv2f05j.fsf@gitster-ct.c.googlers.com","threadId":"50012","inReplyTo":"20181213034041.GR2509588@devbig004.ftw2.facebook.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-12-13T05:47:36Z","receivedAt":"2018-12-13T05:47:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tejun Heo writes:\n\n> Hmmm... I see. I still have a bit of trouble seeing why doing it that\n> way is better tho. Wouldn't new-object-hook be simpler? They'll\n> achieve about the same thing but one would need to keep the states in\n> two places.\n\nThe new-object-hook thing will not have keep the states. Whenever\nGit realizes that it created a new object, it must call that hook,\nand at that point in time, without keeping any state, it knows which\nobjects are what it has just created. So \"in two places\" is not a\nproblem at all. There is only one place (i.e. the place the sweeper\nwould record what it just did to communicate with its future\ninvocation).\n\nA new-object-hook that will always be called any time a new object\nenters the picture would be a nightmare to maintain up-to-date. One\nmissed codepath and your coverage will have holes. Having a back-up\nmechanism to sweep for new objects will give you a better chance of\nstaying correct as the system evolves, I'd think.\n\n"},{"id":"365271","messageId":"20181213161546.GV2509588@devbig004.ftw2.facebook.com","threadId":"50012","inReplyTo":"xmqqftv2f05j.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-13T16:15:46Z","receivedAt":"2018-12-13T16:15:51Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hello, Junio.\n\nOn Thu, Dec 13, 2018 at 02:47:36PM +0900, Junio C Hamano wrote:\n> Tejun Heo writes:\n> \n> > Hmmm... I see. I still have a bit of trouble seeing why doing it that\n> > way is better tho. Wouldn't new-object-hook be simpler? They'll\n> > achieve about the same thing but one would need to keep the states in\n> > two places.\n> \n> The new-object-hook thing will not have keep the states. Whenever\n> Git realizes that it created a new object, it must call that hook,\n> and at that point in time, without keeping any state, it knows which\n> objects are what it has just created. So \"in two places\" is not a\n\nYeap, that's what I was trying to say.\n\n> problem at all. There is only one place (i.e. the place the sweeper\n> would record what it just did to communicate with its future\n> invocation).\n> \n> A new-object-hook that will always be called any time a new object\n> enters the picture would be a nightmare to maintain up-to-date. One\n\nBut I didn't know this. I probably am too naive in thinking so but\nit's a bit surprising that there's no single / few chokepoints for\nrepo updates that we can attach to.\n\n> missed codepath and your coverage will have holes. Having a back-up\n> mechanism to sweep for new objects will give you a better chance of\n> staying correct as the system evolves, I'd think.\n\nThe duplicate state tracking still bothers me a bit (duplicate in the\nsense that the end results are the same) especially as this would mean\nany similar mechanism would need to track their own states too, but I\nreally don't have the full (not even close) picture here and it can\neasily be me missing something.\n\nAnyways, I'll wait for Stefan to chime in.\n\nThanks.\n\n-- \ntejun\n"},{"id":"365536","messageId":"CAPL8ZiuNgv4s1w-FKAznV5A9MKxSxW40fTPrF0Xt8Ywy6UOsVQ@mail.gmail.com","threadId":"50012","inReplyTo":"20181213034606.GS2509588@devbig004.ftw2.facebook.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Stefan Xenos","fromEmail":"sxenos@google.com","sentAt":"2018-12-18T14:40:37Z","receivedAt":"2018-12-18T14:40:54Z","isPatch":false,"sender":{"key":"sxenos@google.com","avatar":null},"body":"Heya, Tejun!\n\nThanks for getting in touch. I agree with Junio that we probably\nshouldn't be tracking the same information in two ways if we can think\nof something that does both... so let's see if we can think of\nsomething! The evolve proposal suggests having a metas/ namespace to\ntrack ongoing changes is intended as a way to track the changes a user\nis actively working on. If we were to use it to retroactively store\nevery historical cherry-pick in a large repository, I'm concerned it\nmay get too cluttered to use for its originally-intended purpose. I'm\nalso not sure how well things would perform with a huge number of\nrefs. The remainder of the proposal (using metacommits to store the\nrelationships) would work for the xref use-case, but we'd probably\nwant to tweak the way we store the root refs in order to do a good job\nwith the large number of inactive cherry-picks he probably wants. Any\ncode that is looking for cross-reference metadata could look in both\nnamespaces.\n\nConversely, if we were to tweak the xrefs proposal by adding the\nobsolescence information that evolve needs, we'd be missing a place to\nstore the user's ongoing changes, a way to push and pull specific\nchanges, a way to describe alternate histories for a commit, and a\nmechanism for preventing the commits needed by evolve from getting\ngarbage collected.\n\nAll the problems with both approaches are solve-able, though.\n\nI spent a few hours discussing this with Stefan Beller last week and I\nthink we've come up with a variation on the evolve proposal that would\ncover both our use-cases. Let me know what you think. To address the\ncluttering of the metas namespace, we could introduce a new\n\"hiddenmetas\" namespace. It would contain the same type of data\nrecorded in the metas namespace, but it would normally be hidden from\nthe user when they list their changes, etc. It would also be immune\nfrom the automatic deletions that are applied to the \"metas\"\nnamespace. From your point of view, the critical bit is that it would\ncontain information about cherry-picks. That would address the\n\"user-visible clutter\" problem. Utility methods that want to search\nfor cherry-picks would iterate over both namespaces.\n\nFor the performance problem, I think we could just live with it\ntemporarily and wait for the currently-ongoing reftable work since the\nreftable proposal would address exactly this performance issue (by\nallowing us to store and manipulate large numbers of refs\nefficiently).\n\nA nice characteristic of this approach is that it would also address a\nproblem with the evolve proposal that had concerned me: how to deal\nwith the filter-branch command, which would have created pretty much\nthe same problems regarding the large number of metadata refs for\nchanges the user isn't actively working on.\n\nIt might be nice to also consider and some alternatives, but I haven't\nhad enough time to think up more of them (and I haven't digested the\nxrefs proposal sufficiently to suggest an enhanced version of it yet).\nIf anyone else has ideas for combining the xrefs and evolve use-cases,\nhaving more alternatives to choose from is always better!\n\nIf you're okay with the \"hiddenmetas\" approach in principle, I'll\nupdate the evolve proposal doc to include it. Once we work out how the\nstorage format will work, we can coordinate our implementation work.\n\n - Stefan\n\n\n\n\nOn Wed, Dec 12, 2018 at 7:46 PM Tejun Heo wrote:\n>\n> Hello, Junio, Stefan.\n>\n> On Thu, Dec 13, 2018 at 12:09:39PM +0900, Junio C Hamano wrote:\n> > Please do not take the above as \"don't do notes/xref-; instead read\n> > from the 'meta commits'\". I do not have a preference between the\n> > two proposed implementations. The important thing is that we won't\n> > end up with having to maintain two separate mechanisms that want to\n> > keep track of essentially the same class of information. FWIW I'd\n> > be perfectly fine if the unification goes the other way, as long as\n> > goals of both parties are met, and for that, I'd like to see you two\n> > work together, or at least be aware of what each other is doing and\n> > see if cross-pollination would result in a mutually better solution.\n>\n> So, from my POV, the only thing I want is being able to easily tell\n> which commit got cherry picked where. I really don't care how that's\n> achieved. Stefan, if there's anything I can do to push it forward,\n> let me know and if you see anything useful in my patchset, please feel\n> free to incorporate them in any way.\n>\n> Thanks.\n>\n> --\n> tejun\n"},{"id":"365539","messageId":"CAPL8ZiuxzdUtkRxALBZ=LjTaKQ6d85BmB9nTQ0JPLnVRo91j8Q@mail.gmail.com","threadId":"50012","inReplyTo":"CAPL8ZiuNgv4s1w-FKAznV5A9MKxSxW40fTPrF0Xt8Ywy6UOsVQ@mail.gmail.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Stefan Xenos","fromEmail":"sxenos@google.com","sentAt":"2018-12-18T16:48:35Z","receivedAt":"2018-12-18T16:48:51Z","isPatch":false,"sender":{"key":"sxenos@google.com","avatar":null},"body":"I've just uploaded a new evolve proposal that includes a spec for the\n\"hiddenmetas\" namespace, where we can store historical cherry-pick\ninformation.\n\nOn Tue, Dec 18, 2018 at 6:40 AM Stefan Xenos wrote:\n>\n> Heya, Tejun!\n>\n> Thanks for getting in touch. I agree with Junio that we probably\n> shouldn't be tracking the same information in two ways if we can think\n> of something that does both... so let's see if we can think of\n> something! The evolve proposal suggests having a metas/ namespace to\n> track ongoing changes is intended as a way to track the changes a user\n> is actively working on. If we were to use it to retroactively store\n> every historical cherry-pick in a large repository, I'm concerned it\n> may get too cluttered to use for its originally-intended purpose. I'm\n> also not sure how well things would perform with a huge number of\n> refs. The remainder of the proposal (using metacommits to store the\n> relationships) would work for the xref use-case, but we'd probably\n> want to tweak the way we store the root refs in order to do a good job\n> with the large number of inactive cherry-picks he probably wants. Any\n> code that is looking for cross-reference metadata could look in both\n> namespaces.\n>\n> Conversely, if we were to tweak the xrefs proposal by adding the\n> obsolescence information that evolve needs, we'd be missing a place to\n> store the user's ongoing changes, a way to push and pull specific\n> changes, a way to describe alternate histories for a commit, and a\n> mechanism for preventing the commits needed by evolve from getting\n> garbage collected.\n>\n> All the problems with both approaches are solve-able, though.\n>\n> I spent a few hours discussing this with Stefan Beller last week and I\n> think we've come up with a variation on the evolve proposal that would\n> cover both our use-cases. Let me know what you think. To address the\n> cluttering of the metas namespace, we could introduce a new\n> \"hiddenmetas\" namespace. It would contain the same type of data\n> recorded in the metas namespace, but it would normally be hidden from\n> the user when they list their changes, etc. It would also be immune\n> from the automatic deletions that are applied to the \"metas\"\n> namespace. From your point of view, the critical bit is that it would\n> contain information about cherry-picks. That would address the\n> \"user-visible clutter\" problem. Utility methods that want to search\n> for cherry-picks would iterate over both namespaces.\n>\n> For the performance problem, I think we could just live with it\n> temporarily and wait for the currently-ongoing reftable work since the\n> reftable proposal would address exactly this performance issue (by\n> allowing us to store and manipulate large numbers of refs\n> efficiently).\n>\n> A nice characteristic of this approach is that it would also address a\n> problem with the evolve proposal that had concerned me: how to deal\n> with the filter-branch command, which would have created pretty much\n> the same problems regarding the large number of metadata refs for\n> changes the user isn't actively working on.\n>\n> It might be nice to also consider and some alternatives, but I haven't\n> had enough time to think up more of them (and I haven't digested the\n> xrefs proposal sufficiently to suggest an enhanced version of it yet).\n> If anyone else has ideas for combining the xrefs and evolve use-cases,\n> having more alternatives to choose from is always better!\n>\n> If you're okay with the \"hiddenmetas\" approach in principle, I'll\n> update the evolve proposal doc to include it. Once we work out how the\n> storage format will work, we can coordinate our implementation work.\n>\n> - Stefan\n>\n>\n>\n>\n> On Wed, Dec 12, 2018 at 7:46 PM Tejun Heo wrote:\n> >\n> > Hello, Junio, Stefan.\n> >\n> > On Thu, Dec 13, 2018 at 12:09:39PM +0900, Junio C Hamano wrote:\n> > > Please do not take the above as \"don't do notes/xref-; instead read\n> > > from the 'meta commits'\". I do not have a preference between the\n> > > two proposed implementations. The important thing is that we won't\n> > > end up with having to maintain two separate mechanisms that want to\n> > > keep track of essentially the same class of information. FWIW I'd\n> > > be perfectly fine if the unification goes the other way, as long as\n> > > goals of both parties are met, and for that, I'd like to see you two\n> > > work together, or at least be aware of what each other is doing and\n> > > see if cross-pollination would result in a mutually better solution.\n> >\n> > So, from my POV, the only thing I want is being able to easily tell\n> > which commit got cherry picked where. I really don't care how that's\n> > achieved. Stefan, if there's anything I can do to push it forward,\n> > let me know and if you see anything useful in my patchset, please feel\n> > free to incorporate them in any way.\n> >\n> > Thanks.\n> >\n> > --\n> > tejun\n"},{"id":"365540","messageId":"20181218165101.GY2509588@devbig004.ftw2.facebook.com","threadId":"50012","inReplyTo":"CAPL8ZiuxzdUtkRxALBZ=LjTaKQ6d85BmB9nTQ0JPLnVRo91j8Q@mail.gmail.com","subject":"Re: [PATCHSET] git-reverse-trailer-xrefs: Reverse map cherry-picks and other cross-references","fromName":"Tejun Heo","fromEmail":"tj@kernel.org","sentAt":"2018-12-18T16:51:01Z","receivedAt":"2018-12-18T16:51:06Z","isPatch":false,"sender":{"key":"tj@kernel.org","avatar":null},"body":"Hey, guys.\n\nOn Tue, Dec 18, 2018 at 08:48:35AM -0800, Stefan Xenos wrote:\n> I've just uploaded a new evolve proposal that includes a spec for the\n> \"hiddenmetas\" namespace, where we can store historical cherry-pick\n> information.\n\nTotal noob question - where can I read that? Also, as long as I can\nhave the back references and follow to the existing cherry-picks, I\ndon't have any further requirements, so any working mechanism is great\nby me.\n\nThanks.\n\n-- \ntejun\n"}]}