{"thread":{"id":"58564","subject":"[PATCH 0/9] Trace2 timers and counters and some cleanup","startedAt":"2022-10-04T16:20:16Z","lastAt":"2022-10-25T15:40:23Z","messageCount":73,"participants":["Jeff Hostetler via GitGitGadget","Ævar Arnfjörð Bjarmason","Junio C Hamano","Jeff Hostetler","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"464183","messageId":"pull.1373.git.1664900407.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":null,"subject":"[PATCH 0/9] Trace2 timers and counters and some cleanup","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:19:58Z","receivedAt":"2022-10-04T16:20:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"This patch series add stopwatch timers and global counters to the trace2\nlogging facility. It also does a little housecleaning.\n\nThis is basically a rewrite of the series that I submitted back in December\n2021: [1] and [2]. Hopefully, it addresses all of the concerns raised back\nthen and does it in a way that avoids the issues that stalled that effort.\n\nFirst we start with a few housecleaning commits:\n\n * The first 2 commits are unrelated to this effort, but were required to\n   get the existing code to compile on my Mac with Clang 11.0.0 with\n   DEVELOPER=1. Those can be dropped if there is a better way to do this.\n\n * The 3rd commit is in response a concern about using int rather than\n   size_t for nr and alloc in an ALLOC_GROW() in existing trace2 code.\n\n * The 4th commit cleans up my use of the term \"TLS\" in my thread code.\n\n * The 5th and 6th commits (hopefully) clear up the misunderstandings around\n   the thread_name variable in my thread context structures. My earlier\n   attempts to clean and clarify this led to most of the controversies in\n   the earlier patch series. Hopefully, these 2 commits will improve the\n   clarify matters.\n\n * The 7th commit cleans up a mostly obsolete section in the trace2 API\n   documentation.\n\nFinally, the last 2 commits add the stopwatch timers and the global\ncounters.\n\n[1]\nhttps://lore.kernel.org/git/pull.1099.git.1640012469.gitgitgadget@gmail.com/\n[2]\nhttps://lore.kernel.org/git/pull.1099.v2.git.1640720202.gitgitgadget@gmail.com/\n\nJeff Hostetler (9):\n  builtin/merge-file: fix compiler warning on MacOS with clang 11.0.0\n  builtin/unpack-objects.c: fix compiler warning on MacOS with clang\n    11.0.0\n  trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n  tr2tls: clarify TLS terminology\n  trace2: rename trace2 thread_name argument as name_hint\n  trace2: convert ctx.thread_name to flex array\n  api-trace2.txt: elminate section describing the public trace2 API\n  trace2: add stopwatch timers\n  trace2: add global counter mechanism\n\n Documentation/technical/api-trace2.txt | 190 +++++++++++++++++--------\n Makefile                               |   2 +\n builtin/merge-file.c                   |   4 +-\n builtin/unpack-objects.c               |   2 +-\n t/helper/test-trace2.c                 | 187 ++++++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  95 +++++++++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               | 121 +++++++++++++++-\n trace2.h                               | 101 +++++++++++--\n trace2/tr2_ctr.c                       | 101 +++++++++++++\n trace2/tr2_ctr.h                       | 104 ++++++++++++++\n trace2/tr2_tgt.h                       |  14 ++\n trace2/tr2_tgt_event.c                 |  47 +++++-\n trace2/tr2_tgt_normal.c                |  39 +++++\n trace2/tr2_tgt_perf.c                  |  49 ++++++-\n trace2/tr2_tls.c                       |  43 +++---\n trace2/tr2_tls.h                       |  52 ++++---\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 ++++++++++++++++++\n 19 files changed, 1366 insertions(+), 113 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\n\nbase-commit: 3dcec76d9df911ed8321007b1d197c1a206dc164\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1373%2Fjeffhostetler%2Ftrace2-stopwatch-v4-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1373/jeffhostetler/trace2-stopwatch-v4-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1373\n-- \ngitgitgadget\n"},{"id":"464184","messageId":"870f29166ea5d5c73bd724c862a59d9702a6fe26.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 1/9] builtin/merge-file: fix compiler warning on MacOS with clang 11.0.0","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:19:59Z","receivedAt":"2022-10-04T16:20:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/merge-file.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/merge-file.c b/builtin/merge-file.c\nindex c923bbf2abb..607c3d3f9e1 100644\n--- a/builtin/merge-file.c\n+++ b/builtin/merge-file.c\n@@ -26,9 +26,9 @@ static int label_cb(const struct option *opt, const char *arg, int unset)\n int cmd_merge_file(int argc, const char **argv, const char *prefix)\n {\n \tconst char *names[3] = { 0 };\n-\tmmfile_t mmfs[3] = { 0 };\n+\tmmfile_t mmfs[3] = { { 0 } };\n \tmmbuffer_t result = { 0 };\n-\txmparam_t xmp = { 0 };\n+\txmparam_t xmp = { { 0 } };\n \tint ret = 0, i = 0, to_stdout = 0;\n \tint quiet = 0;\n \tstruct option options[] = {\n-- \ngitgitgadget\n\n"},{"id":"464185","messageId":"43c41f7035d6f5d1f34d847da833ba8f5a6a13be.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 2/9] builtin/unpack-objects.c: fix compiler warning on MacOS with clang 11.0.0","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:00Z","receivedAt":"2022-10-04T16:20:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/unpack-objects.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 43789b8ef29..4b16f1592ba 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -385,7 +385,7 @@ static const void *feed_input_zstream(struct input_stream *in_stream,\n \n static void stream_blob(unsigned long size, unsigned nr)\n {\n-\tgit_zstream zstream = { 0 };\n+\tgit_zstream zstream = { { 0 } };\n \tstruct input_zstream_data data = { 0 };\n \tstruct input_stream in_stream = {\n \t\t.read = feed_input_zstream,\n-- \ngitgitgadget\n\n"},{"id":"464186","messageId":"73704b6f66055fbaf8c11696be9f579c06b471dd.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 3/9] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:01Z","receivedAt":"2022-10-04T16:20:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nUse \"size_t\" rather than \"int\" for the \"alloc\" and \"nr_open_regions\"\nfields in the \"tr2tls_thread_ctx\".  These are used by ALLOC_GROW().\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex b1e327a928e..a90bd639d48 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -11,8 +11,8 @@\n struct tr2tls_thread_ctx {\n \tstruct strbuf thread_name;\n \tuint64_t *array_us_start;\n-\tint alloc;\n-\tint nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n+\tsize_t alloc;\n+\tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n };\n \n-- \ngitgitgadget\n\n"},{"id":"464187","messageId":"7123886a804bfd5bfac1f0fd6deb907e1635d0e5.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 4/9] tr2tls: clarify TLS terminology","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:02Z","receivedAt":"2022-10-04T16:20:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nReduce or eliminate use of the term \"TLS\" in the Trace2 code.\n\nThe term \"TLS\" has two popular meanings: \"thread-local storage\" and\n\"transport layer security\".  In the Trace2 source, the term is associated\nwith the former.  There was concern on the mailing list about it refering\nto the latter.\n\nUpdate the source and documentation to eliminate the use of the \"TLS\" term\nor replace it with the phrase \"thread-local storage\" to reduce ambiguity.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  8 ++++----\n trace2.c                               |  2 +-\n trace2.h                               | 10 +++++-----\n trace2/tr2_tls.c                       |  6 +++---\n trace2/tr2_tls.h                       | 18 +++++++++++-------\n 5 files changed, 24 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 2afa28bb5aa..431d424f9d5 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -685,8 +685,8 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_start\"`::\n \tThis event is generated when a thread is started.  It is\n-\tgenerated from *within* the new thread's thread-proc (for TLS\n-\treasons).\n+\tgenerated from *within* the new thread's thread-proc (because\n+\tit needs to access data in the thread's thread-local storage).\n +\n ------------\n {\n@@ -698,7 +698,7 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_exit\"`::\n \tThis event is generated when a thread exits.  It is generated\n-\tfrom *within* the thread's thread-proc (for TLS reasons).\n+\tfrom *within* the thread's thread-proc.\n +\n ------------\n {\n@@ -1206,7 +1206,7 @@ worked on 508 items at offset 2032.  Thread \"th04\" worked on 508 items\n at offset 508.\n +\n This example also shows that thread names are assigned in a racy manner\n-as each thread starts and allocates TLS storage.\n+as each thread starts.\n \n Config (def param) Events::\n \ndiff --git a/trace2.c b/trace2.c\nindex 0c0a11e07d5..c1244e45ace 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -52,7 +52,7 @@ static struct tr2_tgt *tr2_tgt_builtins[] =\n  * Force (rather than lazily) initialize any of the requested\n  * builtin TRACE2 targets at startup (and before we've seen an\n  * actual TRACE2 event call) so we can see if we need to setup\n- * the TR2 and TLS machinery.\n+ * private data structures and thread-local storage.\n  *\n  * Return the number of builtin targets enabled.\n  */\ndiff --git a/trace2.h b/trace2.h\nindex 88d906ea830..af3c11694cc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -73,8 +73,7 @@ void trace2_initialize_clock(void);\n /*\n  * Initialize TRACE2 tracing facility if any of the builtin TRACE2\n  * targets are enabled in the system config or the environment.\n- * This includes setting up the Trace2 thread local storage (TLS).\n- * Emits a 'version' message containing the version of git\n+ * This emits a 'version' message containing the version of git\n  * and the Trace2 protocol.\n  *\n  * This function should be called from `main()` as early as possible in\n@@ -302,7 +301,8 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n \n /*\n  * Emit a 'thread_start' event.  This must be called from inside the\n- * thread-proc to set up the trace2 TLS data for the thread.\n+ * thread-proc to allow the thread to create its own thread-local\n+ * storage.\n  *\n  * Thread names should be descriptive, like \"preload_index\".\n  * Thread names will be decorated with an instance number automatically.\n@@ -315,8 +315,8 @@ void trace2_thread_start_fl(const char *file, int line,\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\n- * thread-proc to report thread-specific data and cleanup TLS data\n- * for the thread.\n+ * thread-proc so that the thread can access and clean up its\n+ * thread-local storage.\n  */\n void trace2_thread_exit_fl(const char *file, int line);\n \ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 7da94aba522..8d2182fbdbb 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -69,9 +69,9 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void)\n \tctx = pthread_getspecific(tr2tls_key);\n \n \t/*\n-\t * If the thread-proc did not call trace2_thread_start(), we won't\n-\t * have any TLS data associated with the current thread.  Fix it\n-\t * here and silently continue.\n+\t * If the current thread's thread-proc did not call\n+\t * trace2_thread_start(), then the thread will not have any\n+\t * thread-local storage.  Create it now and silently continue.\n \t */\n \tif (!ctx)\n \t\tctx = tr2tls_create_self(\"unknown\", getnanotime() / 1000);\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex a90bd639d48..1297509fd23 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -3,6 +3,12 @@\n \n #include \"strbuf.h\"\n \n+/*\n+ * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n+ * Trace2 source files.  This usage is borrowed from GCC and Windows.\n+ * There is NO relation to \"transport layer security\".\n+ */\n+\n /*\n  * Arbitry limit for thread names for column alignment.\n  */\n@@ -17,9 +23,7 @@ struct tr2tls_thread_ctx {\n };\n \n /*\n- * Create TLS data for the current thread.  This gives us a place to\n- * put per-thread data, such as thread start time, function nesting\n- * and a per-thread label for our messages.\n+ * Create thread-local storage for the current thread.\n  *\n  * We assume the first thread is \"main\".  Other threads are given\n  * non-zero thread-ids to help distinguish messages from concurrent\n@@ -35,7 +39,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n- * Get our TLS data.\n+ * Get the thread-local storage pointer of the current thread.\n  */\n struct tr2tls_thread_ctx *tr2tls_get_self(void);\n \n@@ -45,7 +49,7 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void);\n int tr2tls_is_main_thread(void);\n \n /*\n- * Free our TLS data.\n+ * Free the current thread's thread-local storage.\n  */\n void tr2tls_unset_self(void);\n \n@@ -81,12 +85,12 @@ uint64_t tr2tls_region_elasped_self(uint64_t us);\n uint64_t tr2tls_absolute_elapsed(uint64_t us);\n \n /*\n- * Initialize the tr2 TLS system.\n+ * Initialize thread-local storage for Trace2.\n  */\n void tr2tls_init(void);\n \n /*\n- * Free all tr2 TLS resources.\n+ * Free all Trace2 thread-local storage resources.\n  */\n void tr2tls_release(void);\n \n-- \ngitgitgadget\n\n"},{"id":"464188","messageId":"82f1672e180afcd876505a4354bd9952f70db49e.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 5/9] trace2: rename trace2 thread_name argument as name_hint","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:03Z","receivedAt":"2022-10-04T16:20:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nRename the `thread_name` argument in `tr2tls_create_self()`\nand `trace2_thread_start()` to be `name_hint` to make it clear\nthat the passed argument is a hint that will be used to create\nthe actual `struct tr2tls_thread_ctx.thread_name` variable.\n\nThis should make it clearer in the API that the trace2 layer\ndoes not borrow the caller's string pointer/buffer, but rather\nthat it will use that hint in formatting the actual thread's\nname.  Previous discussion on the mailing list indicated that\nthere was confusion about this point.\n\nThis commit does not change how the `thread_name` field is\nallocated or stored within the `tr2tls_thread_ctx` structure.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  2 +-\n trace2.c                               |  6 +++---\n trace2.h                               | 11 ++++++-----\n trace2/tr2_tls.c                       |  4 ++--\n trace2/tr2_tls.h                       | 17 ++++++++++-------\n 5 files changed, 22 insertions(+), 18 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 431d424f9d5..4fe2d6992ab 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -209,7 +209,7 @@ e.g: `void trace2_child_start(struct child_process *cmd)`.\n \n These messages are concerned with Git thread usage.\n \n-e.g: `void trace2_thread_start(const char *thread_name)`.\n+e.g: `void trace2_thread_start(const char *name_hint)`.\n \n === Region and Data Messages\n \ndiff --git a/trace2.c b/trace2.c\nindex c1244e45ace..c8e5acced2a 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -466,7 +466,7 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code)\n \t\t\t\tfile, line, us_elapsed_absolute, exec_id, code);\n }\n \n-void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n+void trace2_thread_start_fl(const char *file, int line, const char *name_hint)\n {\n \tstruct tr2_tgt *tgt_j;\n \tint j;\n@@ -488,14 +488,14 @@ void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n \t\t */\n \t\ttrace2_region_enter_printf_fl(file, line, NULL, NULL, NULL,\n \t\t\t\t\t      \"thread-proc on main: %s\",\n-\t\t\t\t\t      thread_name);\n+\t\t\t\t\t      name_hint);\n \t\treturn;\n \t}\n \n \tus_now = getnanotime() / 1000;\n \tus_elapsed_absolute = tr2tls_absolute_elapsed(us_now);\n \n-\ttr2tls_create_self(thread_name, us_now);\n+\ttr2tls_create_self(name_hint, us_now);\n \n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_start_fl)\ndiff --git a/trace2.h b/trace2.h\nindex af3c11694cc..fe39dcb5849 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -304,14 +304,15 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n  * thread-proc to allow the thread to create its own thread-local\n  * storage.\n  *\n- * Thread names should be descriptive, like \"preload_index\".\n- * Thread names will be decorated with an instance number automatically.\n+ * The thread name hint should be descriptive, like \"preload_index\" or\n+ * taken from the thread-proc function.  A unique thread name will be\n+ * created from the hint and the thread id automatically.\n  */\n void trace2_thread_start_fl(const char *file, int line,\n-\t\t\t    const char *thread_name);\n+\t\t\t    const char *name_hint);\n \n-#define trace2_thread_start(thread_name) \\\n-\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n+#define trace2_thread_start(name_hint) \\\n+\ttrace2_thread_start_fl(__FILE__, __LINE__, (name_hint))\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 8d2182fbdbb..39b41fd2487 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -31,7 +31,7 @@ void tr2tls_start_process_clock(void)\n \ttr2tls_us_start_process = getnanotime() / 1000;\n }\n \n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n@@ -50,7 +50,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \tstrbuf_init(&ctx->thread_name, 0);\n \tif (ctx->thread_id)\n \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_name);\n+\tstrbuf_addstr(&ctx->thread_name, name_hint);\n \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n \ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 1297509fd23..f1ee58305d6 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -25,17 +25,20 @@ struct tr2tls_thread_ctx {\n /*\n  * Create thread-local storage for the current thread.\n  *\n- * We assume the first thread is \"main\".  Other threads are given\n- * non-zero thread-ids to help distinguish messages from concurrent\n- * threads.\n- *\n- * Truncate the thread name if necessary to help with column alignment\n- * in printf-style messages.\n+ * The first thread in the process will have:\n+ *     { .thread_id=0, .thread_name=\"main\" }\n+ * Subsequent threads are given a non-zero thread_id and a thread_name\n+ * constructed from the id and a \"name hint\" (which is usually based\n+ * upon the name of the thread-proc function).  For example:\n+ *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n+ * This helps to identify and distinguish messages from concurrent threads.\n+ * The ctx.thread_name field is truncated if necessary to help with column\n+ * alignment in printf-style messages.\n  *\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n  */\n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"464189","messageId":"6492b6d2b989e08bb539fff3ffe5bdf50fa0a195.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:04Z","receivedAt":"2022-10-04T16:20:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nConvert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\nto a \"flex array\" at the end of the context structure.\n\nThe `thread_name` field is a constant string that is constructed when\nthe context is created.  Using a (non-const) `strbuf` structure for it\ncaused some confusion in the past because it implied that someone\ncould rename a thread after it was created.  That usage was not\nintended.  Changing it to a \"flex array\" will hopefully make the\nintent more clear.\n\nAlso, move the maximum thread_name truncation to tr2_tgt_perf.c\nbecause it is the only target that needs to worry about output column\nalignment.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tgt_event.c |  2 +-\n trace2/tr2_tgt_perf.c  |  8 ++++++--\n trace2/tr2_tls.c       | 25 +++++++++++++------------\n trace2/tr2_tls.h       |  9 +--------\n 4 files changed, 21 insertions(+), 23 deletions(-)\n\ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 37a3163be12..52f9356c695 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -90,7 +90,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n \n \tjw_object_string(jw, \"event\", event_name);\n \tjw_object_string(jw, \"sid\", tr2_sid_get());\n-\tjw_object_string(jw, \"thread\", ctx->thread_name.buf);\n+\tjw_object_string(jw, \"thread\", ctx->thread_name);\n \n \t/*\n \t * In brief mode, only emit <time> on these 2 event types.\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 8cb792488c8..fdeb3292d3a 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -25,6 +25,7 @@ static int tr2env_perf_be_brief;\n \n #define TR2FMT_PERF_FL_WIDTH (28)\n #define TR2FMT_PERF_MAX_EVENT_NAME (12)\n+#define TR2FMT_PERF_MAX_THREAD_NAME (24)\n #define TR2FMT_PERF_REPO_WIDTH (3)\n #define TR2FMT_PERF_CATEGORY_WIDTH (12)\n \n@@ -107,8 +108,11 @@ static void perf_fmt_prepare(const char *event_name,\n \t}\n \n \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n-\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n-\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n+\tstrbuf_addf(buf, \"%-*.*s | %-*s | \",\n+\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n+\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n+\t\t    ctx->thread_name,\n+\t\t    TR2FMT_PERF_MAX_EVENT_NAME,\n \t\t    event_name);\n \n \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 39b41fd2487..89437e773f6 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -34,7 +34,18 @@ void tr2tls_start_process_clock(void)\n struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n-\tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n+\tstruct tr2tls_thread_ctx *ctx;\n+\tstruct strbuf buf_name = STRBUF_INIT;\n+\tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n+\n+\tif (thread_id)\n+\t\tstrbuf_addf(&buf_name, \"th%02d:\", thread_id);\n+\tstrbuf_addstr(&buf_name, name_hint);\n+\n+\tFLEX_ALLOC_MEM(ctx, thread_name, buf_name.buf, buf_name.len);\n+\tstrbuf_release(&buf_name);\n+\n+\tctx->thread_id = thread_id;\n \n \t/*\n \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n@@ -45,15 +56,6 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n \n-\tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n-\n-\tstrbuf_init(&ctx->thread_name, 0);\n-\tif (ctx->thread_id)\n-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, name_hint);\n-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n-\n \tpthread_setspecific(tr2tls_key, ctx);\n \n \treturn ctx;\n@@ -95,7 +97,6 @@ void tr2tls_unset_self(void)\n \n \tpthread_setspecific(tr2tls_key, NULL);\n \n-\tstrbuf_release(&ctx->thread_name);\n \tfree(ctx->array_us_start);\n \tfree(ctx);\n }\n@@ -113,7 +114,7 @@ void tr2tls_pop_self(void)\n \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n \n \tif (!ctx->nr_open_regions)\n-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n+\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n \n \tctx->nr_open_regions--;\n }\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex f1ee58305d6..be0bc73d08f 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -9,17 +9,12 @@\n  * There is NO relation to \"transport layer security\".\n  */\n \n-/*\n- * Arbitry limit for thread names for column alignment.\n- */\n-#define TR2_MAX_THREAD_NAME (24)\n-\n struct tr2tls_thread_ctx {\n-\tstruct strbuf thread_name;\n \tuint64_t *array_us_start;\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n+\tchar thread_name[FLEX_ARRAY];\n };\n \n /*\n@@ -32,8 +27,6 @@ struct tr2tls_thread_ctx {\n  * upon the name of the thread-proc function).  For example:\n  *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n  * This helps to identify and distinguish messages from concurrent threads.\n- * The ctx.thread_name field is truncated if necessary to help with column\n- * alignment in printf-style messages.\n  *\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n-- \ngitgitgadget\n\n"},{"id":"464190","messageId":"77a4daf9a4ba8c3b064bfb0cd8a813b822d03b3c.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 7/9] api-trace2.txt: elminate section describing the public trace2 API","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:05Z","receivedAt":"2022-10-04T16:20:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nEliminate the mostly obsolete `Public API` sub-section from the\n`Trace2 API` section in the documentation.  Strengthen the referral\nto `trace2.h`.\n\nMost of the technical information in this sub-section was moved to\n`trace2.h` in 6c51cb525d (trace2: move doc to trace2.h, 2019-11-17) to\nbe adjacent to the function prototypes.  The remaining text wasn't\nthat useful by itself.\n\nFurthermore, the text would need a bit of overhaul to add routines\nthat do not immediately generate a message, such as stopwatch timers.\nSo it seemed simpler to just get rid of it.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt | 61 +++-----------------------\n 1 file changed, 7 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 4fe2d6992ab..9d43909d068 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -148,20 +148,18 @@ filename collisions).\n \n == Trace2 API\n \n-All public Trace2 functions and macros are defined in `trace2.h` and\n-`trace2.c`.  All public symbols are prefixed with `trace2_`.\n+The Trace2 public API is defined and documented in `trace2.h`; refer to it for\n+more information.  All public functions and macros are prefixed\n+with `trace2_` and are implemented in `trace2.c`.\n \n There are no public Trace2 data structures.\n \n The Trace2 code also defines a set of private functions and data types\n in the `trace2/` directory.  These symbols are prefixed with `tr2_`\n-and should only be used by functions in `trace2.c`.\n+and should only be used by functions in `trace2.c` (or other private\n+source files in `trace2/`).\n \n-== Conventions for Public Functions and Macros\n-\n-The functions defined by the Trace2 API are declared and documented\n-in `trace2.h`.  It defines the API functions and wrapper macros for\n-Trace2.\n+=== Conventions for Public Functions and Macros\n \n Some functions have a `_fl()` suffix to indicate that they take `file`\n and `line-number` arguments.\n@@ -172,52 +170,7 @@ take a `va_list` argument.\n Some functions have a `_printf_fl()` suffix to indicate that they also\n take a `printf()` style format with a variable number of arguments.\n \n-There are CPP wrapper macros and `#ifdef`s to hide most of these details.\n-See `trace2.h` for more details.  The following discussion will only\n-describe the simplified forms.\n-\n-== Public API\n-\n-All Trace2 API functions send a message to all of the active\n-Trace2 Targets.  This section describes the set of available\n-messages.\n-\n-It helps to divide these functions into groups for discussion\n-purposes.\n-\n-=== Basic Command Messages\n-\n-These are concerned with the lifetime of the overall git process.\n-e.g: `void trace2_initialize_clock()`, `void trace2_initialize()`,\n-`int trace2_is_enabled()`, `void trace2_cmd_start(int argc, const char **argv)`.\n-\n-=== Command Detail Messages\n-\n-These are concerned with describing the specific Git command\n-after the command line, config, and environment are inspected.\n-e.g: `void trace2_cmd_name(const char *name)`,\n-`void trace2_cmd_mode(const char *mode)`.\n-\n-=== Child Process Messages\n-\n-These are concerned with the various spawned child processes,\n-including shell scripts, git commands, editors, pagers, and hooks.\n-\n-e.g: `void trace2_child_start(struct child_process *cmd)`.\n-\n-=== Git Thread Messages\n-\n-These messages are concerned with Git thread usage.\n-\n-e.g: `void trace2_thread_start(const char *name_hint)`.\n-\n-=== Region and Data Messages\n-\n-These are concerned with recording performance data\n-over regions or spans of code. e.g:\n-`void trace2_region_enter(const char *category, const char *label, const struct repository *repo)`.\n-\n-Refer to trace2.h for details about all trace2 functions.\n+CPP wrapper macros are defined to hide most of these details.\n \n == Trace2 Target Formats\n \n-- \ngitgitgadget\n\n"},{"id":"464191","messageId":"19c7bba91ba87a795f53e8b4ce4f285fe184688b.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 8/9] trace2: add stopwatch timers","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:06Z","receivedAt":"2022-10-04T16:20:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd stopwatch timer mechanism to Trace2.\n\nTimers are an alternative to Trace2 Regions.  Regions are useful for\nmeasuring the time spent in various computation phases, such as the\ntime to read the index, time to scan for unstaged files, time to scan\nfor untracked files, and etc.\n\nHowever, regions are not appropriate in all places.  For example,\nduring a checkout, it would be very inefficient to use regions to\nmeasure the total time spent inflating objects from the ODB from\nacross the entire lifetime of the process; a per-unzip() region would\nflood the output and significantly slow the command; and some form of\npost-processing would be requried to compute the time spent in unzip().\n\nTimers can be used to measure a series of timer intervals and emit\na single summary event (at thread and/or process exit).\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  90 ++++++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  98 +++++++++++++\n t/t0211-trace2-perf.sh                 |  49 +++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               |  75 ++++++++++\n trace2.h                               |  43 ++++++\n trace2/tr2_tgt.h                       |   7 +\n trace2/tr2_tgt_event.c                 |  26 ++++\n trace2/tr2_tgt_normal.c                |  23 ++++\n trace2/tr2_tgt_perf.c                  |  24 ++++\n trace2/tr2_tls.c                       |  10 ++\n trace2/tr2_tls.h                       |  10 ++\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 +++++++++++++++++++\n 15 files changed, 784 insertions(+)\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 9d43909d068..75ce6f45603 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -769,6 +769,42 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning in the thread.  This event is generated when a thread\n+\texits for timers that requested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n+`\"timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning aggregated across all threads.  This event is generated\n+\twhen the process exits.\n++\n+------------\n+{\n+\t\"event\":\"timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\n@@ -1200,6 +1236,60 @@ d0 | main                     | data         | r0  |  0.002126 |  0.002126 | fsy\n d0 | main                     | exit         |     |  0.000470 |           |              | code:0\n d0 | main                     | atexit       |     |  0.000477 |           |              | code:0\n ----------------\n+\n+Stopwatch Timer Events::\n+\n+\tMeasure the time spent in a function call or span of code\n+\tthat might be called from many places within the code\n+\tthroughout the life of the process.\n++\n+----------------\n+static void expensive_function(void)\n+{\n+\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t...\n+\tsleep_millisec(1000); // Do something expensive\n+\t...\n+\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+}\n+\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\t...\n+\n+\texpensive_function();\n+\n+\t// Do something else 1...\n+\n+\texpensive_function();\n+\n+\t// Do something else 2...\n+\n+\texpensive_function();\n+\n+\treturn 0;\n+}\n+----------------\n++\n+In this example, we measure the total time spent in\n+`expensive_function()` regardless of when it is called\n+in the overall flow of the program.\n++\n+----------------\n+$ export GIT_TRACE2_PERF_BRIEF=1\n+$ export GIT_TRACE2_PERF=~/log.perf\n+$ t/helper/test-tool trace2 100timer 3 1000\n+...\n+$ cat ~/log.perf\n+d0 | main                     | version      |     |           |           |              | ...\n+d0 | main                     | start        |     |  0.001453 |           |              | t/helper/test-tool trace2 100timer 3 1000\n+d0 | main                     | cmd_name     |     |           |           |              | trace2 (trace2)\n+d0 | main                     | exit         |     |  3.003667 |           |              | code:0\n+d0 | main                     | timer        |     |           |           | test         | name:test1 intervals:3 total:3.001686 min:1.000254 max:1.000929\n+d0 | main                     | atexit       |     |  3.003796 |           |              | code:0\n+----------------\n+\n+\n == Future Work\n \n === Relationship to the Existing Trace Api (api-trace.txt)\ndiff --git a/Makefile b/Makefile\nindex cac3452edb9..820649bf62a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1102,6 +1102,7 @@ LIB_OBJS += trace2/tr2_tgt_event.o\n LIB_OBJS += trace2/tr2_tgt_normal.o\n LIB_OBJS += trace2/tr2_tgt_perf.o\n LIB_OBJS += trace2/tr2_tls.o\n+LIB_OBJS += trace2/tr2_tmr.o\n LIB_OBJS += trailer.o\n LIB_OBJS += transport-helper.o\n LIB_OBJS += transport.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex a714130ece7..f951b9e97d7 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -228,6 +228,101 @@ static int ut_010bug_BUG(int argc, const char **argv)\n \tBUG(\"a %s message\", \"BUG\");\n }\n \n+/*\n+ * Single-threaded timer test.  Create several intervals using the\n+ * TEST1 timer.  The test script can verify that an aggregate Trace2\n+ * \"timer\" event is emitted indicating that we started+stopped the\n+ * timer the requested number of times.\n+ */\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay>\";\n+\n+\tint count = 0;\n+\tint delay = 0;\n+\tint k;\n+\n+\tif (argc != 2)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t\tsleep_millisec(delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct ut_101_data {\n+\tint count;\n+\tint delay;\n+};\n+\n+static void *ut_101timer_thread_proc(void *_ut_101_data)\n+{\n+\tstruct ut_101_data *data = _ut_101_data;\n+\tint k;\n+\n+\ttrace2_thread_start(\"ut_101\");\n+\n+\tfor (k = 0; k < data->count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST2);\n+\t\tsleep_millisec(data->delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST2);\n+\t}\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Multi-threaded timer test.  Create several threads that each create\n+ * several intervals using the TEST2 timer.  The test script can verify\n+ * that an individual Trace2 \"th_timer\" events for each thread and an\n+ * aggregate \"timer\" event are generated.\n+ */\n+static int ut_101timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay> <threads>\";\n+\n+\tstruct ut_101_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_101timer_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -248,6 +343,9 @@ static struct unit_test ut_table[] = {\n \t{ ut_008bug,      \"008bug\",    \"\" },\n \t{ ut_009bug_BUG,  \"009bug_BUG\",\"\" },\n \t{ ut_010bug_BUG,  \"010bug_BUG\",\"\" },\n+\n+\t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n+\t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 22d0845544e..5c28424e657 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -173,4 +173,53 @@ test_expect_success 'using global config, perf stream, return code 0' '\n \ttest_cmp expect actual\n '\n \n+# Exercise the stopwatch timers in a loop and confirm that we have\n+# as many start/stop intervals as expected.  We cannot really test the\n+# actual (total, min, max) timer values, so we have to assume that they\n+# are good, but we can verify the interval count.\n+#\n+# The timer \"test/test1\" should only emit a global summary \"timer\" event.\n+# The timer \"test/test2\" should emit per-thread \"th_timer\" events and a\n+# global summary \"timer\" event.\n+\n+have_timer_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 intervals=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} intervals:${intervals}\" &&\n+\n+\tgrep \"${pattern}\" ${file}\n+}\n+\n+test_expect_success 'stopwatch timer test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test1\" 5 times from \"main\".\n+\ttest-tool trace2 100timer 5 10 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test1\" 5 actual\n+'\n+\n+test_expect_success 'stopwatch timer test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test2\" 5 times each in 3 threads.\n+\ttest-tool trace2 101timer 5 10 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_timer_event \"th01:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th02:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th03:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\n+\t# And we should have 15 total uses.\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n+'\n+\n test_done\ndiff --git a/t/t0211/scrub_perf.perl b/t/t0211/scrub_perf.perl\nindex 299999f0f89..7a50bae6463 100644\n--- a/t/t0211/scrub_perf.perl\n+++ b/t/t0211/scrub_perf.perl\n@@ -64,6 +64,12 @@ while (<>) {\n \t    goto SKIP_LINE;\n \t}\n     }\n+    elsif ($tokens[$col_event] =~ m/timer/) {\n+\t# This also captures \"th_timer\" events\n+\t$tokens[$col_rest] =~ s/ total:\\d+\\.\\d*/ total:_T_TOTAL_/;\n+\t$tokens[$col_rest] =~ s/ min:\\d+\\.\\d*/ min:_T_MIN_/;\n+\t$tokens[$col_rest] =~ s/ max:\\d+\\.\\d*/ max:_T_MAX_/;\n+    }\n \n     # t_abs and t_rel are either blank or a float.  Replace the float\n     # with a constant for matching the HEREDOC in the test script.\ndiff --git a/trace2.c b/trace2.c\nindex c8e5acced2a..c564ff49bb9 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -13,6 +13,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static int trace2_enabled;\n \n@@ -83,6 +84,23 @@ static void tr2_tgt_disable_builtins(void)\n \t\ttgt_j->pfn_term();\n }\n \n+/*\n+ * The signature of this function must match the pfn_timer\n+ * method in the targets.  (Think of this is an apply operation\n+ * across the set of active targets.)\n+ */\n+static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n+\t\t\t\t const struct tr2_timer *timer,\n+\t\t\t\t int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_timer)\n+\t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -110,6 +128,26 @@ static void tr2main_atexit_handler(void)\n \t */\n \ttr2tls_pop_unwind_self();\n \n+\t/*\n+\t * Some timers want per-thread details.  If the main thread\n+\t * used one of those timers, emit the details now (before\n+\t * we emit the aggregate timer values).\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data for the main thread to the final\n+\t * totals.  And then emit the final timer values.\n+\t *\n+\t * Technically, we shouldn't need to hold the lock to update\n+\t * and output the final_timer_block (since all other threads\n+\t * should be dead by now), but it doesn't hurt anything.\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_atexit)\n \t\t\ttgt_j->pfn_atexit(us_elapsed_absolute,\n@@ -541,6 +579,21 @@ void trace2_thread_exit_fl(const char *file, int line)\n \ttr2tls_pop_unwind_self();\n \tus_elapsed_thread = tr2tls_region_elasped_self(us_now);\n \n+\t/*\n+\t * Some timers want per-thread details.  If this thread used\n+\t * one of those timers, emit the details now.\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data from the current (non-main) thread\n+\t * to the final totals.  (We'll accumulate data for the main\n+\t * thread later during \"atexit\".)\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_exit_fl)\n \t\t\ttgt_j->pfn_thread_exit_fl(file, line,\n@@ -795,6 +848,28 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...)\n \tva_end(ap);\n }\n \n+void trace2_timer_start(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_start: invalid timer id: %d\", tid);\n+\n+\ttr2_start_timer(tid);\n+}\n+\n+void trace2_timer_stop(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_stop: invalid timer id: %d\", tid);\n+\n+\ttr2_stop_timer(tid);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex fe39dcb5849..2d146fb32fc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -51,6 +51,7 @@ struct json_writer;\n  * [] trace2_region*    -- emit region nesting messages.\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n+ * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n  */\n \n /*\n@@ -485,6 +486,48 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...);\n \n #define trace2_printf(...) trace2_printf_fl(__FILE__, __LINE__, __VA_ARGS__)\n \n+/*\n+ * Define the set of stopwatch timers.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum must also be added to the\n+ * `tr2_timer_metadata[]` in `trace2/tr2_tmr.c`.\n+ */\n+enum trace2_timer_id {\n+\t/*\n+\t * Define two timers for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_TIMER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_TIMER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional timer definitions before here. */\n+\tTRACE2_NUMBER_OF_TIMERS\n+};\n+\n+/*\n+ * Start/Stop the indicated stopwatch timer in the current thread.\n+ *\n+ * The time spent by the current thread between the _start and _stop\n+ * calls will be added to the thread's partial sum for this timer.\n+ *\n+ * Timer events are emitted at thread and program exit.\n+ *\n+ * Note: Since the stopwatch API routines do not generate individual\n+ * events, they do not take (file, line) arguments.  Similarly, the\n+ * category and timer name values are defined at compile-time in the\n+ * timer definitions array, so they are not needed here in the API.\n+ */\n+void trace2_timer_start(enum trace2_timer_id tid);\n+void trace2_timer_stop(enum trace2_timer_id tid);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 65f94e15748..2a80bef0df5 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -4,6 +4,8 @@\n struct child_process;\n struct repository;\n struct json_writer;\n+struct tr2_timer_metadata;\n+struct tr2_timer;\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -96,6 +98,10 @@ typedef void(tr2_tgt_evt_printf_va_fl_t)(const char *file, int line,\n \t\t\t\t\t uint64_t us_elapsed_absolute,\n \t\t\t\t\t const char *fmt, va_list ap);\n \n+typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n+\t\t\t\t  const struct tr2_timer *timer,\n+\t\t\t\t  int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -132,6 +138,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_fl_t                   *pfn_data_fl;\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n+\ttr2_tgt_evt_timer_t                     *pfn_timer;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 52f9356c695..1196da89ba4 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -9,6 +9,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_event = {\n \t.sysenv_var = TR2_SYSENV_EVENT,\n@@ -617,6 +618,30 @@ static void fn_data_json_fl(const char *file, int line,\n \t}\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"intervals\", timer->interval_count);\n+\tjw_object_double(&jw, \"t_total\", 6, t_total);\n+\tjw_object_double(&jw, \"t_min\", 6, t_min);\n+\tjw_object_double(&jw, \"t_max\", 6, t_max);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -648,4 +673,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 69f80330778..3888c10ef50 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -8,6 +8,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_normal = {\n \t.sysenv_var = TR2_SYSENV_NORMAL,\n@@ -329,6 +330,27 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -360,4 +382,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_fl = NULL,\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex fdeb3292d3a..064aefbbebb 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -10,6 +10,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_perf = {\n \t.sysenv_var = TR2_SYSENV_PERF,\n@@ -559,6 +560,28 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tstrbuf_addf(&buf_payload, (\"name:%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -590,4 +613,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 89437e773f6..1aceb36b010 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -180,3 +180,13 @@ int tr2tls_locked_increment(int *p)\n \n \treturn current_value;\n }\n+\n+void tr2tls_lock(void)\n+{\n+\tpthread_mutex_lock(&tr2tls_mutex);\n+}\n+\n+void tr2tls_unlock(void)\n+{\n+\tpthread_mutex_unlock(&tr2tls_mutex);\n+}\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex be0bc73d08f..4f8e24f1749 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n /*\n  * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n@@ -14,6 +15,9 @@ struct tr2tls_thread_ctx {\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n+\tstruct tr2_timer_block timer_block;\n+\tunsigned int used_any_timer:1;\n+\tunsigned int used_any_per_thread_timer:1;\n \tchar thread_name[FLEX_ARRAY];\n };\n \n@@ -100,4 +104,10 @@ int tr2tls_locked_increment(int *p);\n  */\n void tr2tls_start_process_clock(void);\n \n+/*\n+ * Explicitly lock/unlock our mutex.\n+ */\n+void tr2tls_lock(void);\n+void tr2tls_unlock(void);\n+\n #endif /* TR2_TLS_H */\ndiff --git a/trace2/tr2_tmr.c b/trace2/tr2_tmr.c\nnew file mode 100644\nindex 00000000000..786762dfd26\n--- /dev/null\n+++ b/trace2/tr2_tmr.c\n@@ -0,0 +1,182 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n+\n+#define MY_MAX(a, b) ((a) > (b) ? (a) : (b))\n+#define MY_MIN(a, b) ((a) < (b) ? (a) : (b))\n+\n+/*\n+ * A global timer block to aggregate values from the partial sums from\n+ * each thread.\n+ */\n+static struct tr2_timer_block final_timer_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each stopwatch timer.\n+ *\n+ * This array must match \"enum trace2_timer_id\" and the values\n+ * in \"struct tr2_timer_block.timer[*]\".\n+ */\n+static struct tr2_timer_metadata tr2_timer_metadata[TRACE2_NUMBER_OF_TIMERS] = {\n+\t[TRACE2_TIMER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_TIMER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_start_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\tt->recursion_count++;\n+\tif (t->recursion_count > 1)\n+\t\treturn; /* ignore recursive starts */\n+\n+\tt->start_ns = getnanotime();\n+}\n+\n+void tr2_stop_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\tuint64_t ns_now;\n+\tuint64_t ns_interval;\n+\n+\tassert(t->recursion_count > 0);\n+\n+\tt->recursion_count--;\n+\tif (t->recursion_count)\n+\t\treturn; /* still in recursive call(s) */\n+\n+\tns_now = getnanotime();\n+\tns_interval = ns_now - t->start_ns;\n+\n+\tt->total_ns += ns_interval;\n+\n+\t/*\n+\t * min_ns was initialized to zero (in the xcalloc()) rather\n+\t * than UINT_MAX when the block of timers was allocated,\n+\t * so we should always set both the min_ns and max_ns values\n+\t * the first time that the timer is used.\n+\t */\n+\tif (!t->interval_count) {\n+\t\tt->min_ns = ns_interval;\n+\t\tt->max_ns = ns_interval;\n+\t} else {\n+\t\tt->min_ns = MY_MIN(ns_interval, t->min_ns);\n+\t\tt->max_ns = MY_MAX(ns_interval, t->max_ns);\n+\t}\n+\n+\tt->interval_count++;\n+\n+\tctx->used_any_timer = 1;\n+\tif (tr2_timer_metadata[tid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_timer = 1;\n+}\n+\n+void tr2_update_final_timers(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++) {\n+\t\tstruct tr2_timer *t_final = &final_timer_block.timer[tid];\n+\t\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\t\tif (t->recursion_count) {\n+\t\t\t/*\n+\t\t\t * The current thread is exiting with\n+\t\t\t * timer[tid] still running.\n+\t\t\t *\n+\t\t\t * Technically, this is a bug, but I'm going\n+\t\t\t * to ignore it.\n+\t\t\t *\n+\t\t\t * I don't think it is worth calling die()\n+\t\t\t * for.  I don't think it is worth killing the\n+\t\t\t * process for this bookkeeping error.  We\n+\t\t\t * might want to call warning(), but I'm going\n+\t\t\t * to wait on that.\n+\t\t\t *\n+\t\t\t * The downside here is that total_ns won't\n+\t\t\t * include the current open interval (now -\n+\t\t\t * start_ns).  I can live with that.\n+\t\t\t */\n+\t\t}\n+\n+\t\tif (!t->interval_count)\n+\t\t\tcontinue; /* this timer was not used by this thread */\n+\n+\t\tt_final->total_ns += t->total_ns;\n+\n+\t\t/*\n+\t\t * final_timer_block.timer[tid].min_ns was initialized to\n+\t\t * was initialized to zero rather than UINT_MAX, so we should\n+\t\t * always set both the min_ns and max_ns values the first time\n+\t\t * that we add a partial sum into it.\n+\t\t */\n+\t\tif (!t_final->interval_count) {\n+\t\t\tt_final->min_ns = t->min_ns;\n+\t\t\tt_final->max_ns = t->max_ns;\n+\t\t} else {\n+\t\t\tt_final->min_ns = MY_MIN(t_final->min_ns, t->min_ns);\n+\t\t\tt_final->max_ns = MY_MAX(t_final->max_ns, t->max_ns);\n+\t\t}\n+\n+\t\tt_final->interval_count += t->interval_count;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_per_thread_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * For each timer, if the timer wants per-thread events and\n+\t * this thread used it, emit it.\n+\t */\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (tr2_timer_metadata[tid].want_per_thread_events &&\n+\t\t    ctx->timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &ctx->timer_block.timer[tid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tenum trace2_timer_id tid;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (final_timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &final_timer_block.timer[tid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_tmr.h b/trace2/tr2_tmr.h\nnew file mode 100644\nindex 00000000000..d5753576134\n--- /dev/null\n+++ b/trace2/tr2_tmr.h\n@@ -0,0 +1,140 @@\n+#ifndef TR2_TMR_H\n+#define TR2_TMR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow \"stopwatch\" timers.\n+ *\n+ * Timers can be used to measure \"interesting\" activity that does not\n+ * fit the \"region\" model, such as code called from many different\n+ * regions (like zlib) and/or where data for individual calls are not\n+ * interesting or are too numerous to be efficiently logged.\n+ *\n+ * Timer values are accumulated during program execution and emitted\n+ * to the Trace2 logs at program exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * timers and timer ids using a \"timer block\" array in thread-local\n+ * storage.  This gives us constant time access to each timer within\n+ * each thread, since we want start/stop operations to be as fast as\n+ * possible.  This lets us avoid the complexities of dynamically\n+ * allocating a timer on the first use by a thread and/or possibly\n+ * sharing that timer definition with other concurrent threads.\n+ * However, this does require that we define time the set of timers at\n+ * compile time.\n+ *\n+ * Each thread uses the timer block in its thread-local storage to\n+ * compute partial sums for each timer (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Using this \"timer block\" model costs ~48 bytes per timer per thread\n+ * (we have about six uint64 fields per timer).  This does increase\n+ * the size of the thread-local storage block, but it is allocated (at\n+ * thread create time) and not on the thread stack, so I'm not worried\n+ * about the size.\n+ *\n+ * Partial sums for each timer are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each timer are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"timer metadata\" table contains the \"category\" and \"name\"\n+ * fields for each timer.  This eliminates the need to include those\n+ * args in the various timer APIs.\n+ */\n+\n+/*\n+ * The definition of an individual timer and used by an individual\n+ * thread.\n+ */\n+struct tr2_timer {\n+\t/*\n+\t * Total elapsed time for this timer in this thread in nanoseconds.\n+\t */\n+\tuint64_t total_ns;\n+\n+\t/*\n+\t * The maximum and minimum interval values observed for this\n+\t * timer in this thread.\n+\t */\n+\tuint64_t min_ns;\n+\tuint64_t max_ns;\n+\n+\t/*\n+\t * The value of the clock when this timer was started in this\n+\t * thread.  (Undefined when the timer is not active in this\n+\t * thread.)\n+\t */\n+\tuint64_t start_ns;\n+\n+\t/*\n+\t * Number of times that this timer has been started and stopped\n+\t * in this thread.  (Recursive starts are ignored.)\n+\t */\n+\tuint64_t interval_count;\n+\n+\t/*\n+\t * Number of nested starts on the stack in this thread.  (We\n+\t * ignore recursive starts and use this to track the recursive\n+\t * calls.)\n+\t */\n+\tunsigned int recursion_count;\n+};\n+\n+/*\n+ * Metadata for a timer.\n+ */\n+struct tr2_timer_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this timer\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed-size block of timers to insert into\n+ * thread-local storage.  This wrapper is used to avoid quirks\n+ * of C and the usual need to pass an array size argument.\n+ */\n+struct tr2_timer_block {\n+\tstruct tr2_timer timer[TRACE2_NUMBER_OF_TIMERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to actually start/stop an\n+ * individual timer in the current thread.\n+ */\n+void tr2_start_timer(enum trace2_timer_id tid);\n+void tr2_stop_timer(enum trace2_timer_id tid);\n+\n+/*\n+ * Add the current thread's timer data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_timers(void);\n+\n+/*\n+ * Emit per-thread timer data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+/*\n+ * Emit global total timer values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+#endif /* TR2_TMR_H */\n-- \ngitgitgadget\n\n"},{"id":"464192","messageId":"2bf7cb1f8d01ed3fad43cd62d341a0c9b33b92bd.1664900407.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH 9/9] trace2: add global counter mechanism","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-04T16:20:07Z","receivedAt":"2022-10-04T16:20:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd global counters mechanism to Trace2.\n\nThe Trace2 counters mechanism adds the ability to create a set of\nglobal counter variables and an API to increment them efficiently.\nCounters can optionally report per-thread usage in addition to the sum\nacross all threads.\n\nCounter events are emitted to the Trace2 logs when a thread exits and\nat process exit.\n\nCounters are an alternative to `data` and `data_json` events.\n\nCounters are useful when you want to measure something across the life\nof the process, when you don't want per-measurement events for\nperformance reasons, when the data does not fit conveniently within a\nregion, or when your control flow does not easily let you write the\nfinal total.  For example, you might use this to report the number of\ncalls to unzip() or the number of de-delta steps during a checkout.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  31 ++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  89 +++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  46 +++++++++++\n trace2.c                               |  52 +++++++++++--\n trace2.h                               |  37 +++++++++\n trace2/tr2_ctr.c                       | 101 ++++++++++++++++++++++++\n trace2/tr2_ctr.h                       | 104 +++++++++++++++++++++++++\n trace2/tr2_tgt.h                       |   7 ++\n trace2/tr2_tgt_event.c                 |  19 +++++\n trace2/tr2_tgt_normal.c                |  16 ++++\n trace2/tr2_tgt_perf.c                  |  17 ++++\n trace2/tr2_tls.h                       |   4 +\n 13 files changed, 517 insertions(+), 7 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 75ce6f45603..de5fc250595 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -805,6 +805,37 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_counter\"`::\n+\tThis event logs the value of a counter variable in a thread.\n+\tThis event is generated when a thread exits for counters that\n+\trequested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+`\"counter\"`::\n+\tThis event logs the value of a counter variable across all threads.\n+\tThis event is generated when the process exits.  The total value\n+\treported here is the sum across all threads.\n++\n+------------\n+{\n+\t\"event\":\"counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\ndiff --git a/Makefile b/Makefile\nindex 820649bf62a..29ab417ca3a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += trace.o\n LIB_OBJS += trace2.o\n LIB_OBJS += trace2/tr2_cfg.o\n LIB_OBJS += trace2/tr2_cmd_name.o\n+LIB_OBJS += trace2/tr2_ctr.o\n LIB_OBJS += trace2/tr2_dst.o\n LIB_OBJS += trace2/tr2_sid.o\n LIB_OBJS += trace2/tr2_sysenv.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex f951b9e97d7..1b092c60714 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -323,6 +323,92 @@ static int ut_101timer(int argc, const char **argv)\n \treturn 0;\n }\n \n+/*\n+ * Single-threaded counter test.  Add several values to the TEST1 counter.\n+ * The test script can verify that the final sum is reported in the \"counter\"\n+ * event.\n+ */\n+static int ut_200counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> [<v2> [...]]\";\n+\tint value;\n+\tint k;\n+\n+\tif (argc < 1)\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < argc; k++) {\n+\t\tif (get_i(&value, argv[k]))\n+\t\t\tdie(\"invalid value[%s] -- %s\",\n+\t\t\t    argv[k], usage_error);\n+\t\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST1, value);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Multi-threaded counter test.  Create seveal threads that each increment\n+ * the TEST2 global counter.  The test script can verify that an individual\n+ * \"th_counter\" event is generated with a partial sum for each thread and\n+ * that a final aggregate \"counter\" event is generated.\n+ */\n+\n+struct ut_201_data {\n+\tint v1;\n+\tint v2;\n+};\n+\n+static void *ut_201counter_thread_proc(void *_ut_201_data)\n+{\n+\tstruct ut_201_data *data = _ut_201_data;\n+\n+\ttrace2_thread_start(\"ut_201\");\n+\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v1);\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v2);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int ut_201counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> <v2> <threads>\";\n+\n+\tstruct ut_201_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v1, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v2, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_201counter_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -346,6 +432,9 @@ static struct unit_test ut_table[] = {\n \n \t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n \t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n+\n+\t{ ut_200counter,  \"200counter\", \"<v1> [<v2> [<v3> [...]]]\" },\n+\t{ ut_201counter,  \"201counter\", \"<v1> <v2> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 5c28424e657..0b3436e8cac 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -222,4 +222,50 @@ test_expect_success 'stopwatch timer test/test2' '\n \thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n '\n \n+# Exercise the global counters and confirm that we get the expected values.\n+#\n+# The counter \"test/test1\" should only emit a global summary \"counter\" event.\n+# The counter \"test/test2\" could emit per-thread \"th_counter\" events and a\n+# global summary \"counter\" event.\n+\n+have_counter_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 value=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} value:${value}\" &&\n+\n+\tgrep \"${patern}\" ${file}\n+}\n+\n+test_expect_success 'global counter test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the counter \"test1\" and add n integers.\n+\ttest-tool trace2 200counter 1 2 3 4 5 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test1\" 15 actual\n+'\n+\n+test_expect_success 'global counter test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Add 2 integers to the counter \"test2\" in each of 3 threads.\n+\ttest-tool trace2 201counter 7 13 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_counter_event \"th01:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th02:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th03:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\n+\t# And we should have a single event with the total across all threads.\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test2\" 60 actual\n+'\n+\n test_done\ndiff --git a/trace2.c b/trace2.c\nindex c564ff49bb9..2376500ea05 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -8,6 +8,7 @@\n #include \"version.h\"\n #include \"trace2/tr2_cfg.h\"\n #include \"trace2/tr2_cmd_name.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_dst.h\"\n #include \"trace2/tr2_sid.h\"\n #include \"trace2/tr2_sysenv.h\"\n@@ -101,6 +102,22 @@ static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n \t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n }\n \n+/*\n+ * The signature of this function must match the pfn_counter\n+ * method in the targets.\n+ */\n+static void tr2_tgt_emit_a_counter(const struct tr2_counter_metadata *meta,\n+\t\t\t\t   const struct tr2_counter *counter,\n+\t\t\t\t   int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_counter)\n+\t\t\ttgt_j->pfn_counter(meta, counter, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -132,20 +149,26 @@ static void tr2main_atexit_handler(void)\n \t * Some timers want per-thread details.  If the main thread\n \t * used one of those timers, emit the details now (before\n \t * we emit the aggregate timer values).\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data for the main thread to the final\n-\t * totals.  And then emit the final timer values.\n+\t * Add stopwatch timer and counter data for the main thread to\n+\t * the final totals.  And then emit the final values.\n \t *\n \t * Technically, we shouldn't need to hold the lock to update\n-\t * and output the final_timer_block (since all other threads\n-\t * should be dead by now), but it doesn't hurt anything.\n+\t * and output the final_timer_block and final_counter_block\n+\t * (since all other threads should be dead by now), but it\n+\t * doesn't hurt anything.\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_final_counters(tr2_tgt_emit_a_counter);\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -582,16 +605,20 @@ void trace2_thread_exit_fl(const char *file, int line)\n \t/*\n \t * Some timers want per-thread details.  If this thread used\n \t * one of those timers, emit the details now.\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data from the current (non-main) thread\n-\t * to the final totals.  (We'll accumulate data for the main\n-\t * thread later during \"atexit\".)\n+\t * Add stopwatch timer and counter data from the current\n+\t * (non-main) thread to the final totals.  (We'll accumulate\n+\t * data for the main thread later during \"atexit\".)\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -870,6 +897,17 @@ void trace2_timer_stop(enum trace2_timer_id tid)\n \ttr2_stop_timer(tid);\n }\n \n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (cid < 0 || cid >= TRACE2_NUMBER_OF_COUNTERS)\n+\t\tBUG(\"trace2_counter_add: invalid counter id: %d\", cid);\n+\n+\ttr2_counter_increment(cid, value);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 2d146fb32fc..da670ffd26c 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -52,6 +52,7 @@ struct json_writer;\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n  * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n+ * [] trace2_counter*   -- global counters (messages are deferred).\n  */\n \n /*\n@@ -528,6 +529,42 @@ enum trace2_timer_id {\n void trace2_timer_start(enum trace2_timer_id tid);\n void trace2_timer_stop(enum trace2_timer_id tid);\n \n+/*\n+ * Define the set of global counters.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum be also be added to the\n+ * `tr2_counter_metadata[]` in `trace2/tr2_tr2_ctr.c`.\n+ */\n+enum trace2_counter_id {\n+\t/*\n+\t * Define two counters for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional counter definitions before here. */\n+\tTRACE2_NUMBER_OF_COUNTERS\n+};\n+\n+/*\n+ * Increase the named global counter by value.\n+ *\n+ * Note that this adds `value` to the current thread's partial sum for\n+ * this counter (without locking) and that the complete sum is not\n+ * available until all threads have exited, so it does not return the\n+ * new value of the counter.\n+ */\n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nnew file mode 100644\nindex 00000000000..483ca7c308f\n--- /dev/null\n+++ b/trace2/tr2_ctr.c\n@@ -0,0 +1,101 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_ctr.h\"\n+\n+/*\n+ * A global counter block to aggregrate values from the partial sums\n+ * from each thread.\n+ */\n+static struct tr2_counter_block final_counter_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each global counter.\n+ *\n+ * This array must match the \"enum trace2_counter_id\" and the values\n+ * in \"struct tr2_counter_block.counter[*]\".\n+ */\n+static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTERS] = {\n+\t[TRACE2_COUNTER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_COUNTER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\tc->value += value;\n+\n+\tctx->used_any_counter = 1;\n+\tif (tr2_counter_metadata[cid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_counter = 1;\n+}\n+\n+void tr2_update_final_counters(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++) {\n+\t\tstruct tr2_counter *c_final = &final_counter_block.counter[cid];\n+\t\tconst struct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\t\tc_final->value += c->value;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_per_thread_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * For each counter, if the counter wants per-thread events\n+\t * and this thread used it (the value is non-zero), emit it.\n+\t */\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (tr2_counter_metadata[cid].want_per_thread_events &&\n+\t\t    ctx->counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &ctx->counter_block.counter[cid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tenum trace2_counter_id cid;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (final_counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &final_counter_block.counter[cid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_ctr.h b/trace2/tr2_ctr.h\nnew file mode 100644\nindex 00000000000..a2267ee9901\n--- /dev/null\n+++ b/trace2/tr2_ctr.h\n@@ -0,0 +1,104 @@\n+#ifndef TR2_CTR_H\n+#define TR2_CTR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow global \"counters\".\n+ *\n+ * Counters can be used count interesting activity that does not fit\n+ * the \"region and data\" model, such as code called from many\n+ * different regions and/or where you want to count a number of items,\n+ * but don't have control of when the last item will be processed,\n+ * such as counter the number of calls to `lstat()`.\n+ *\n+ * Counters differ from Trace2 \"data\" events.  Data events are emitted\n+ * immediately and are appropriate for documenting loop counters at\n+ * the end of a region, for example.  Counter values are accumulated\n+ * during the program and final counter values are emitted at program\n+ * exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * counters and counter ids using a fixed size \"counter block\" array\n+ * in thread-local storage.  This gives us constant time, lock-free\n+ * access to each counter within each thread.  This lets us avoid the\n+ * complexities of dynamically allocating a counter and sharing that\n+ * definition with other threads.\n+ *\n+ * Each thread uses the counter block in its thread-local storage to\n+ * increment partial sums for each counter (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Partial sums for each counter are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each counter are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"counter metadata\" table contains the \"category\" and\n+ * \"name\" fields for each counter.  This eliminates the need to\n+ * include those args in the various counter APIs.\n+ */\n+\n+/*\n+ * The definition of an individual counter as used by an individual\n+ * thread (and later in aggregation).\n+ */\n+struct tr2_counter {\n+\tuint64_t value;\n+};\n+\n+/*\n+ * Metadata for a counter.\n+ */\n+struct tr2_counter_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this counter\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed block of counters to insert into thread-local\n+ * storage.  This wrapper is used to avoid quirks of C and the usual\n+ * need to pass an array size argument.\n+ */\n+struct tr2_counter_block {\n+\tstruct tr2_counter counter[TRACE2_NUMBER_OF_COUNTERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to increment a counter for the\n+ * current thread.\n+ */\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value);\n+\n+/*\n+ * Add the current thread's counter data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_counters(void);\n+\n+/*\n+ * Emit per-thread counter data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+/*\n+ * Emit global counter values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+#endif /* TR2_CTR_H */\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 2a80bef0df5..94a334d980a 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -6,6 +6,8 @@ struct repository;\n struct json_writer;\n struct tr2_timer_metadata;\n struct tr2_timer;\n+struct tr2_counter_metadata;\n+struct tr2_counter;\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -102,6 +104,10 @@ typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n \t\t\t\t  const struct tr2_timer *timer,\n \t\t\t\t  int is_final_data);\n \n+typedef void(tr2_tgt_evt_counter_t)(const struct tr2_counter_metadata *meta,\n+\t\t\t\t    const struct tr2_counter *counter,\n+\t\t\t\t    int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -139,6 +145,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n \ttr2_tgt_evt_timer_t                     *pfn_timer;\n+\ttr2_tgt_evt_counter_t                   *pfn_counter;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 1196da89ba4..bb0653e0e6f 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -642,6 +642,24 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tjw_release(&jw);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"count\", counter->value);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -674,4 +692,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 3888c10ef50..b21508e06f7 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -351,6 +351,21 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"%s %s/%s value:%\"PRIu64,\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    counter->value);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -383,4 +398,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 064aefbbebb..cbf8aefd56c 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -582,6 +582,22 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"name:%s value:%\"PRIu64,\n+\t\t    meta->name,\n+\t\t    counter->value);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -614,4 +630,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 4f8e24f1749..e306c9bf3ec 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_tmr.h\"\n \n /*\n@@ -16,8 +17,11 @@ struct tr2tls_thread_ctx {\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n \tstruct tr2_timer_block timer_block;\n+\tstruct tr2_counter_block counter_block;\n \tunsigned int used_any_timer:1;\n \tunsigned int used_any_per_thread_timer:1;\n+\tunsigned int used_any_counter:1;\n+\tunsigned int used_any_per_thread_counter:1;\n \tchar thread_name[FLEX_ARRAY];\n };\n \n-- \ngitgitgadget\n"},{"id":"464226","messageId":"221005.86y1tus9ps.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"6492b6d2b989e08bb539fff3ffe5bdf50fa0a195.1664900407.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-05T11:14:50Z","receivedAt":"2022-10-05T13:04:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 04 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n> to a \"flex array\" at the end of the context structure.\n>\n> The `thread_name` field is a constant string that is constructed when\n> the context is created.  Using a (non-const) `strbuf` structure for it\n> caused some confusion in the past because it implied that someone\n> could rename a thread after it was created.\n\nI think it's been long enough that we could use a reminder about the\n\"some confusion\", i.e. if it was a bug report or something else.\n\n> That usage was not intended.  Changing it to a \"flex array\" will\n> hopefully make the intent more clear.\n\nI see we had some back & forth back in the original submission, although\nhonestly I skimmed this this time around, had forgetten about that, and\nhad this pop out at me, and then found my earlier comments.\n\nI see that exchange didn't end as well as I'd hoped[1], and hopefully we\ncan avoid that here. So having looked at this with fresh eyes maybe\nthese comments/questions help:\n\n * I'm unable to bridge the cap from (paraphrased) \"we must change the\n   type\" to \"mak[ing] the [read-only] intent more clear\".\n\n   I.e. if you go across the codebase and look at various non-const\n   \"char name[FLEX_ARRAY]\" and add a \"const\" to them you'll find cases\n   where we re-write the \"FLEX_ARRAY\" string, e.g. the one in archive.c\n   is one of those (the first grep hit, I stopped looking for others at\n   that point).\n\n   Making it \"const\" will yield:\n   \n      archive.c: In function ‘queue_directory’:\n   archive.c:206:29: error: passing argument 1 of ‘xsnprintf’ discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]\n     206 |         d->len = xsnprintf(d->path, len, \"%.*s%s/\", (int)base->len, base->buf, filename);\n\n   So aside from anything else (and I may be misunderstanding this) why\n   does changing it to a FLEX_ARRAY give us the connotation in the\n   confused API user's mind that it shouldn't be messed with that the\n   \"strbuf\" doesn't give us?\n\n * Now, quoting from the below:\n\n> [...]\n> diff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\n> index 39b41fd2487..89437e773f6 100644\n> --- a/trace2/tr2_tls.c\n> +++ b/trace2/tr2_tls.c\n> @@ -34,7 +34,18 @@ void tr2tls_start_process_clock(void)\n>  struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n>  \t\t\t\t\t     uint64_t us_thread_start)\n>  {\n> -\tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n> +\tstruct tr2tls_thread_ctx *ctx;\n> +\tstruct strbuf buf_name = STRBUF_INIT;\n> +\tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n> +\n> +\tif (thread_id)\n> +\t\tstrbuf_addf(&buf_name, \"th%02d:\", thread_id);\n> +\tstrbuf_addstr(&buf_name, name_hint);\n> +\n> +\tFLEX_ALLOC_MEM(ctx, thread_name, buf_name.buf, buf_name.len);\n> +\tstrbuf_release(&buf_name);\n> +\n> +\tctx->thread_id = thread_id;\n>  \n>  \t/*\n>  \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n> @@ -45,15 +56,6 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n>  \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n>  \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n>  \n> -\tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n> -\n> -\tstrbuf_init(&ctx->thread_name, 0);\n> -\tif (ctx->thread_id)\n> -\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n> -\tstrbuf_addstr(&ctx->thread_name, name_hint);\n> -\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n> -\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n> -\n>  \tpthread_setspecific(tr2tls_key, ctx);\n>  \n>  \treturn ctx;\n\nI found this quote hard to follow because there's functional changes\nthere mixed up with code re-arangement, consider leading with a commit\nlike:\n\t\n\tdiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\n\tindex 39b41fd2487..d7952062007 100644\n\t--- a/trace2/tr2_tls.c\n\t+++ b/trace2/tr2_tls.c\n\t@@ -31,10 +31,24 @@ void tr2tls_start_process_clock(void)\n\t \ttr2tls_us_start_process = getnanotime() / 1000;\n\t }\n\t \n\t+static void fill_thread_name(struct strbuf *buf, const char *name_hint,\n\t+\t\t\t     int thread_id)\n\t+{\n\t+\tif (thread_id)\n\t+\t\tstrbuf_addf(buf, \"th%02d:\", thread_id);\n\t+\tstrbuf_addstr(buf, name_hint);\n\t+\tif (buf->len > TR2_MAX_THREAD_NAME)\n\t+\t\tstrbuf_setlen(buf, TR2_MAX_THREAD_NAME);\n\t+\n\t+}\n\t+\n\t struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n\t \t\t\t\t\t     uint64_t us_thread_start)\n\t {\n\t-\tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n\t+\tstruct tr2tls_thread_ctx *ctx;\n\t+\tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n\t+\n\t+\tctx = xcalloc(1, sizeof(*ctx));\n\t \n\t \t/*\n\t \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n\t@@ -45,14 +59,8 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n\t \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n\t \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n\t \n\t-\tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n\t-\n\t \tstrbuf_init(&ctx->thread_name, 0);\n\t-\tif (ctx->thread_id)\n\t-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n\t-\tstrbuf_addstr(&ctx->thread_name, name_hint);\n\t-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n\t-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n\t+\tfill_thread_name(&ctx->thread_name, name_hint, thread_id);\n\t \n\t \tpthread_setspecific(tr2tls_key, ctx);\n\nI see from [1] that I comment on that before, i.e. that it was\n\"looks-to-be-unrelated\", hopefully the above clarifies that, i.e. that\nit's \"unrelated\" in the sense that we can do it separately with no\nfunctiontal change, making the real change smaller.\n\nIf I then rebase your change on top of that I get the below diff, which\nIMO makes it much clearer what's going on. Commenting on that:\n\t\n\tdiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\n\tindex d7952062007..c540027e75d 100644\n\t--- a/trace2/tr2_tls.c\n\t+++ b/trace2/tr2_tls.c\n\t@@ -37,18 +37,21 @@ static void fill_thread_name(struct strbuf *buf, const char *name_hint,\n\t \tif (thread_id)\n\t \t\tstrbuf_addf(buf, \"th%02d:\", thread_id);\n\t \tstrbuf_addstr(buf, name_hint);\n\t-\tif (buf->len > TR2_MAX_THREAD_NAME)\n\t-\t\tstrbuf_setlen(buf, TR2_MAX_THREAD_NAME);\n\t-\n\t }\n\nOkey, so as explained in the commit message we no longer need to worry\nabout this limit, but I think leading with a change to just change that\nfirst would help. I.e. wouldn't starting with keeping the strbuf and\ndoing this truncation in tr2_tgt_perf.c give you the functiotnal change\nfirst, without the type change?\n\nDoing it this way means we're changing the type, and also removing the\nlimit on thread names for non-perf backends.\n\t \n\t struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n\t \t\t\t\t\t     uint64_t us_thread_start)\n\t {\n\t \tstruct tr2tls_thread_ctx *ctx;\n\t+\tstruct strbuf buf_name = STRBUF_INIT;\n\nOkey, now our scratch buffer is function local, but:\n\n\t \tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n\t \n\t-\tctx = xcalloc(1, sizeof(*ctx));\n\t+\tfill_thread_name(&buf_name, name_hint, thread_id);\n\nWe still need to malloc() that \"struct strbuf\", this is the main thing I\nfound confusing and why I didn't see the point in the original\nseries. I.e. we can normally pull compiler tricks with FLEX_ARRAY to\navoid allocations.\n\nBut here you need to format this string anyway, so we've already\nmalloc'd it, you just....\n\n\t+\n\t+\tFLEX_ALLOC_MEM(ctx, thread_name, buf_name.buf, buf_name.len);\n\n...memcpy() it to the FLEX_ARRAY here, but then...\n\n\t+\tstrbuf_release(&buf_name);\n\n...we have to release this thing we malloc()'d, which was previously the\npointer in the struct. \n\n\t+\n\t+\tctx->thread_id = thread_id;\n\t \n\t \t/*\n\t \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n\nSo, I don't really see the point of this \"flex array for implicit const\"\nper the above, you noted in [1] \"I convert the field to a flex-array to\navoid [...] the allocation\" but what allocation are we really avoiding\nhere?\n\nWe still have to allocate the strbuf as before, we just now allocate the\nstruct as before + the length of that strbuf, then we can free the\nstrbuf.\n\nIs it that the end memory use is lower because while we have a\nallocation for the strbuf we release it right away, and the compiler (on\nsome platforms?) can play tricks with sticking this into padding it was\ngoing to put there anyway, given the length of the string?\n\nI can think of ways this *might* matter, I'm just mainly saying that\nyou're leaving the reader guessing still.\n\nAside: I can imagine that we *could* actually avoid an allocation here\nby being more sneaky.\n\nI.e. you could FLEX_ALLOC_MEM() before populating the strbuf, as we know\nthe format is \"th%02d:\", so the space we need for the string is:\n\n\tstrlen(name_hint) + strlen(\"th\") + strlen(4) /* %02d: + \\0 */;\n\nThen you could memset(&ctx->thread_name, 0, len) and strbuf_attach() the\npointer to the start of that, and voila, you could use strbuf_addf() to\ndo the %02d format part of that.\n\nBut I still don't see how this is an area that justifies that sort of\nmicro-optimization (or worrying about strbuf v.s. flex array),\ni.e. don't we usually just have max ncpu threads anyway (the format\nimplies max 99), so a few strings like \"th01:main\" aren't going to cost\nus much, are they?\n\n<tries it out>\n\nAnyway, if this area was actually performance critical and we *really\ncared* about avoiding allocations wouldn't we want to skip both the\n\"strbuf\" there and the \"FLEX_ARRAY\", and just save away the\n\"thread_hint\" (which the caller hardcodes) and \"thread_nr\", and then\nappend on-the-fly?\n\nI came up with the below to do that, it passes all tests, but contains\nmicro-optimizations that I don't think we need (e.g. I understood you\nwanted to avoid printf, so it does that).\n\nBut I think it's a useful point of discussion. What test(s) do you have\nwhere the \"master\" version, FLEX_ARRAY version, and just not strbuf\nformatting the thing at all differ?\n\t\n\tdiff --git a/json-writer.c b/json-writer.c\n\tindex f1cfd8fa8c6..124ad72d200 100644\n\t--- a/json-writer.c\n\t+++ b/json-writer.c\n\t@@ -161,6 +161,47 @@ void jw_object_string(struct json_writer *jw, const char *key, const char *value\n\t \tappend_quoted_string(&jw->json, value);\n\t }\n\t \n\t+void jw_strbuf_add_thread_name(struct strbuf *out, const char *thread_hint,\n\t+\t\t\t       int thread_id, int max_len)\n\t+{\n\t+\tsize_t oldlen = out->len;\n\t+\n\t+\tif (thread_id) {\n\t+\t\tstrbuf_addf(out, \"th\");\n\t+\t\t/*\n\t+\t\t * We're avoiding printf here when on-the-fly\n\t+\t\t * formatting, but why?\n\t+\t\t */\n\t+\t\tif (thread_id < 10) {\n\t+\t\t\tstrbuf_addch(out, '0');\n\t+\t\t\tstrbuf_addch(out, thread_id % 10 + '0');\n\t+\t\t} else {\n\t+\t\t\tstrbuf_addch(out, thread_id / 10 + '0');\n\t+\t\t\tstrbuf_addch(out, thread_id % 10 + '0');\n\t+\t\t}\n\t+\t\tstrbuf_addch(out, ':');\n\t+\t}\n\t+\tif (max_len) {\n\t+\t\tint added = out->len - oldlen;\n\t+\t\tint limit = max_len - added;\n\t+\n\t+\t\tstrbuf_addf(out, \"%.*s\", limit, thread_hint);\n\t+\t} else {\n\t+\t\tstrbuf_addstr(out, thread_hint);\n\t+\t}\n\t+}\n\t+\n\t+void jw_object_thread(struct json_writer *jw, const char *thread_hint,\n\t+\t\t      int thread_id)\n\t+{\n\t+\tstruct strbuf *out = &jw->json;\n\t+\n\t+\tobject_common(jw, \"thread\");\n\t+\tstrbuf_addch(out, '\"');\n\t+\tjw_strbuf_add_thread_name(out, thread_hint, thread_id, 0);\n\t+\tstrbuf_addch(out, '\"');\n\t+}\n\t+\n\t void jw_object_intmax(struct json_writer *jw, const char *key, intmax_t value)\n\t {\n\t \tobject_common(jw, key);\n\tdiff --git a/json-writer.h b/json-writer.h\n\tindex 209355e0f12..51b78296f8a 100644\n\t--- a/json-writer.h\n\t+++ b/json-writer.h\n\t@@ -77,6 +77,10 @@ void jw_array_begin(struct json_writer *jw, int pretty);\n\t \n\t void jw_object_string(struct json_writer *jw, const char *key,\n\t \t\t      const char *value);\n\t+void jw_strbuf_add_thread_name(struct strbuf *out, const char *thread_hint,\n\t+\t\t\t       int thread_id, int max_len);\n\t+void jw_object_thread(struct json_writer *jw, const char *thread_hint,\n\t+\t\t      const int thread_id);\n\t void jw_object_intmax(struct json_writer *jw, const char *key, intmax_t value);\n\t void jw_object_double(struct json_writer *jw, const char *key, int precision,\n\t \t\t      double value);\n\tdiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\n\tindex bb0653e0e6f..6e480fce34a 100644\n\t--- a/trace2/tr2_tgt_event.c\n\t+++ b/trace2/tr2_tgt_event.c\n\t@@ -91,7 +91,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n\t \n\t \tjw_object_string(jw, \"event\", event_name);\n\t \tjw_object_string(jw, \"sid\", tr2_sid_get());\n\t-\tjw_object_string(jw, \"thread\", ctx->thread_name);\n\t+\tjw_object_thread(jw, ctx->thread_hint, ctx->thread_id);\n\t \n\t \t/*\n\t \t * In brief mode, only emit <time> on these 2 event types.\n\tdiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\n\tindex cbf8aefd56c..9f310756349 100644\n\t--- a/trace2/tr2_tgt_perf.c\n\t+++ b/trace2/tr2_tgt_perf.c\n\t@@ -71,6 +71,8 @@ static void perf_fmt_prepare(const char *event_name,\n\t \t\t\t     const char *category, struct strbuf *buf)\n\t {\n\t \tint len;\n\t+\tint oldlen;\n\t+\tint thread_pad;\n\t \n\t \tstrbuf_setlen(buf, 0);\n\t \n\t@@ -109,11 +111,11 @@ static void perf_fmt_prepare(const char *event_name,\n\t \t}\n\t \n\t \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n\t-\tstrbuf_addf(buf, \"%-*.*s | %-*s | \",\n\t-\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n\t-\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n\t-\t\t    ctx->thread_name,\n\t-\t\t    TR2FMT_PERF_MAX_EVENT_NAME,\n\t+\toldlen = buf->len;\n\t+\tjw_strbuf_add_thread_name(buf, ctx->thread_hint, ctx->thread_id,\n\t+\t\t\t\t  TR2FMT_PERF_MAX_THREAD_NAME);\n\t+\tthread_pad = TR2FMT_PERF_MAX_THREAD_NAME - (buf->len - oldlen);\n\t+\tstrbuf_addf(buf, \"%-*s | %-*s | \", thread_pad, \"\", TR2FMT_PERF_MAX_EVENT_NAME,\n\t \t\t    event_name);\n\t \n\t \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\n\tdiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\n\tindex 02117f808eb..9959ec9b160 100644\n\t--- a/trace2/tr2_tls.c\n\t+++ b/trace2/tr2_tls.c\n\t@@ -31,26 +31,13 @@ void tr2tls_start_process_clock(void)\n\t \ttr2tls_us_start_process = getnanotime() / 1000;\n\t }\n\t \n\t-static void fill_thread_name(struct strbuf *buf, const char *name_hint,\n\t-\t\t\t     int thread_id)\n\t-{\n\t-\tif (thread_id)\n\t-\t\tstrbuf_addf(buf, \"th%02d:\", thread_id);\n\t-\tstrbuf_addstr(buf, name_hint);\n\t-}\n\t-\n\t struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n\t \t\t\t\t\t     uint64_t us_thread_start)\n\t {\n\t-\tstruct tr2tls_thread_ctx *ctx;\n\t-\tstruct strbuf buf_name = STRBUF_INIT;\n\t+\tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n\t \tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n\t \n\t-\tfill_thread_name(&buf_name, name_hint, thread_id);\n\t-\n\t-\tFLEX_ALLOC_MEM(ctx, thread_name, buf_name.buf, buf_name.len);\n\t-\tstrbuf_release(&buf_name);\n\t-\n\t+\tctx->thread_hint = name_hint;\n\t \tctx->thread_id = thread_id;\n\t \n\t \t/*\n\t@@ -120,7 +107,8 @@ void tr2tls_pop_self(void)\n\t \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n\t \n\t \tif (!ctx->nr_open_regions)\n\t-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n\t+\t\tBUG(\"no open regions in thread '%s' '%d'\", ctx->thread_hint,\n\t+\t\t    ctx->thread_id);\n\t \n\t \tctx->nr_open_regions--;\n\t }\n\tdiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\n\tindex e306c9bf3ec..f873615ebef 100644\n\t--- a/trace2/tr2_tls.h\n\t+++ b/trace2/tr2_tls.h\n\t@@ -21,8 +21,8 @@ struct tr2tls_thread_ctx {\n\t \tunsigned int used_any_timer:1;\n\t \tunsigned int used_any_per_thread_timer:1;\n\t \tunsigned int used_any_counter:1;\n\t+\tconst char *thread_hint;\n\t \tunsigned int used_any_per_thread_counter:1;\n\t-\tchar thread_name[FLEX_ARRAY];\n\t };\n\t \n\t /*\n\t\n\n\n\n1. https://lore.kernel.org/git/e3fd64ef-9e26-19da-7327-38ab77ae359a@jeffhostetler.com/\n\n\n\n> @@ -95,7 +97,6 @@ void tr2tls_unset_self(void)\n>  \n>  \tpthread_setspecific(tr2tls_key, NULL);\n>  \n> -\tstrbuf_release(&ctx->thread_name);\n>  \tfree(ctx->array_us_start);\n>  \tfree(ctx);\n>  }\n> @@ -113,7 +114,7 @@ void tr2tls_pop_self(void)\n>  \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n>  \n>  \tif (!ctx->nr_open_regions)\n> -\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n> +\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n>  \n>  \tctx->nr_open_regions--;\n>  }\n> diff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\n> index f1ee58305d6..be0bc73d08f 100644\n> --- a/trace2/tr2_tls.h\n> +++ b/trace2/tr2_tls.h\n> @@ -9,17 +9,12 @@\n>   * There is NO relation to \"transport layer security\".\n>   */\n>  \n> -/*\n> - * Arbitry limit for thread names for column alignment.\n> - */\n> -#define TR2_MAX_THREAD_NAME (24)\n> -\n>  struct tr2tls_thread_ctx {\n> -\tstruct strbuf thread_name;\n>  \tuint64_t *array_us_start;\n>  \tsize_t alloc;\n>  \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n>  \tint thread_id;\n> +\tchar thread_name[FLEX_ARRAY];\n>  };\n>  \n>  /*\n> @@ -32,8 +27,6 @@ struct tr2tls_thread_ctx {\n>   * upon the name of the thread-proc function).  For example:\n>   *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n>   * This helps to identify and distinguish messages from concurrent threads.\n> - * The ctx.thread_name field is truncated if necessary to help with column\n> - * alignment in printf-style messages.\n>   *\n>   * In this and all following functions the term \"self\" refers to the\n>   * current thread.\n\n"},{"id":"464227","messageId":"221005.86tu4is9ib.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] Trace2 timers and counters and some cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-05T13:04:33Z","receivedAt":"2022-10-05T13:09:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Oct 04 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> This patch series add stopwatch timers and global counters to the trace2\n> logging facility. It also does a little housecleaning.\n>\n> This is basically a rewrite of the series that I submitted back in December\n> 2021: [1] and [2]. Hopefully, it addresses all of the concerns raised back\n> then and does it in a way that avoids the issues that stalled that effort.\n>\n> First we start with a few housecleaning commits:\n>\n>  * The first 2 commits are unrelated to this effort, but were required to\n>    get the existing code to compile on my Mac with Clang 11.0.0 with\n>    DEVELOPER=1. Those can be dropped if there is a better way to do this.\n\nThis seems like a good thing to have, but there's no subsequent changes\nto those two files on this topic, so is this just a \"to get it building\non my laptop...\" stashed-on?\n\nI think if so it makes sense to split these up, and as feeback on 1-2/9:\nLet's note what compiler/version & what warning we got, the details\nthere for anyone to dig this up later are missing, i.e. if we ever want\nto remove the workaround syntax.\n\n>  * The 3rd commit is in response a concern about using int rather than\n>    size_t for nr and alloc in an ALLOC_GROW() in existing trace2 code.\n\nThis small bit of cleanup also could perhaps be submitted separately?\nIt's unclear (and I read the concern in the initial thread) if this is\nrequired by anything that follows.\n\n"},{"id":"464248","messageId":"xmqq7d1eqhbf.fsf@gitster.g","threadId":"58564","inReplyTo":"6492b6d2b989e08bb539fff3ffe5bdf50fa0a195.1664900407.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-05T18:03:00Z","receivedAt":"2022-10-05T18:03:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n> to a \"flex array\" at the end of the context structure.\n>\n> The `thread_name` field is a constant string that is constructed when\n> the context is created.  Using a (non-const) `strbuf` structure for it\n> caused some confusion in the past because it implied that someone\n> could rename a thread after it was created.  That usage was not\n> intended.  Changing it to a \"flex array\" will hopefully make the\n> intent more clear.\n\nSurely, \"const struct strbuf name;\" member would be an oxymoron, and\nI agree that this should follow \"use strbuf as an easy-to-work-with\nmechanism to come up with a string, and bake the final value into a\nstruct as a member of type 'const char []'\" pattern.\n\nI recall saying why I thought the flex array was overkill, though.\n\nYou have been storing an up-to-24-byte human readable name by\nembedding a strbuf that has two size_t plus a pointer (i.e. 24-bytes\neven on Windows), and as TR2_MAX_THREAD_NAME is capped at 24 bytes\nanyway, an embedded fixed-size thread_name[TR2_MAX_THREAD_NAME+1]\nmember may be the simplest thing to do, I suspect.\n\nIf we were to allow arbitrarily long thread_name[], which may not be\na bad thing to do (e.g. we do not have to worry about truncation\nmaking two names ambiguous, for example), then the flex array is the\nright direction to go in, though.\n"},{"id":"464298","messageId":"bb4137cb-ff59-cc4c-3d42-23f636758216@jeffhostetler.com","threadId":"58564","inReplyTo":"221005.86tu4is9ib.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 0/9] Trace2 timers and counters and some cleanup","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-06T15:45:04Z","receivedAt":"2022-10-06T15:45:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/5/22 9:04 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Oct 04 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> This patch series add stopwatch timers and global counters to the trace2\n>> logging facility. It also does a little housecleaning.\n>>\n>> This is basically a rewrite of the series that I submitted back in December\n>> 2021: [1] and [2]. Hopefully, it addresses all of the concerns raised back\n>> then and does it in a way that avoids the issues that stalled that effort.\n>>\n>> First we start with a few housecleaning commits:\n>>\n>>   * The first 2 commits are unrelated to this effort, but were required to\n>>     get the existing code to compile on my Mac with Clang 11.0.0 with\n>>     DEVELOPER=1. Those can be dropped if there is a better way to do this.\n> \n> This seems like a good thing to have, but there's no subsequent changes\n> to those two files on this topic, so is this just a \"to get it building\n> on my laptop...\" stashed-on?\n\nRight. I needed them to get \"main\" to build on my laptop before I\nstarted hacking.  I debated sending them in separately, but everyone\nwas busy with the 2.38 release and didn't want to add to the noise for\nsuch a minor thing, since all the CI builds were green...\n\nBut, yeah, I can do that.\n\n> \n> I think if so it makes sense to split these up, and as feeback on 1-2/9:\n> Let's note what compiler/version & what warning we got, the details\n> there for anyone to dig this up later are missing, i.e. if we ever want\n> to remove the workaround syntax.\n> \n>>   * The 3rd commit is in response a concern about using int rather than\n>>     size_t for nr and alloc in an ALLOC_GROW() in existing trace2 code.\n> \n> This small bit of cleanup also could perhaps be submitted separately?\n> It's unclear (and I read the concern in the initial thread) if this is\n> required by anything that follows.\n> \n\nNothing requires this. It was just another \"while I'm here\" fixup.\nHowever, those lines are very close to new/changed lines that I added\nfor the timers and counters, so it would probably cause collisions if\nsent independently.  So I'd like to leave them in this series to\nsimplify things.\n\nThanks,\nJeff\n"},{"id":"464307","messageId":"0a4e7f45-6697-2b75-005e-676e9ba444ab@jeffhostetler.com","threadId":"58564","inReplyTo":"221005.86y1tus9ps.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-06T16:28:02Z","receivedAt":"2022-10-06T16:28:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/5/22 7:14 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Oct 04 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n>> to a \"flex array\" at the end of the context structure.\n>>\n>> The `thread_name` field is a constant string that is constructed when\n>> the context is created.  Using a (non-const) `strbuf` structure for it\n>> caused some confusion in the past because it implied that someone\n>> could rename a thread after it was created.\n> \n> I think it's been long enough that we could use a reminder about the\n> \"some confusion\", i.e. if it was a bug report or something else.\n> \n>> That usage was not intended.  Changing it to a \"flex array\" will\n>> hopefully make the intent more clear.\n> \n> I see we had some back & forth back in the original submission, although\n> honestly I skimmed this this time around, had forgetten about that, and\n> had this pop out at me, and then found my earlier comments.\n> \n> I see that exchange didn't end as well as I'd hoped[1], and hopefully we\n> can avoid that here. So having looked at this with fresh eyes maybe\n> these comments/questions help:\n\nYeah, those conversations went rather poorly.  And yes, I'd like to\navoid all of that.  There's a lot in your note here and it'll take a\nlittle while to digest and respond.  But I did want to ACK, sooner\nrather than later, that we agree on that.\n\nAnd yes, I could split out the truncation into a separate commit.\nAnd then revisit the storage change.\n\nThanks\nJeff\n\n"},{"id":"464315","messageId":"c9534d61-a0ad-2e9a-1504-d5f69eee7e17@github.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] Trace2 timers and counters and some cleanup","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-06T18:12:59Z","receivedAt":"2022-10-06T18:13:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/4/22 12:19 PM, Jeff Hostetler via GitGitGadget wrote:\n> This patch series add stopwatch timers and global counters to the trace2\n> logging facility. It also does a little housecleaning.\n> \n> This is basically a rewrite of the series that I submitted back in December\n> 2021: [1] and [2]. Hopefully, it addresses all of the concerns raised back\n> then and does it in a way that avoids the issues that stalled that effort.\n\nThanks for working on this again. As I mentioned earlier [3], this\nwould be really helpful when doing performance investigations. I\nalso plan to insert some timers and counters as a follow-up when\nthis series stabilizes.\n\n[3] https://lore.kernel.org/git/pull.1365.git.1663938034607.gitgitgadget@gmail.com/\n\nI was unable to find further improvements than the ones you\nalready acknowledged for your v2.\n\nThanks,\n-Stolee\n"},{"id":"464332","messageId":"221006.86ilkwr6wy.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"xmqq7d1eqhbf.fsf@gitster.g","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-06T21:05:12Z","receivedAt":"2022-10-06T21:14:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 05 2022, Junio C Hamano wrote:\n\n> \"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n>> to a \"flex array\" at the end of the context structure.\n>>\n>> The `thread_name` field is a constant string that is constructed when\n>> the context is created.  Using a (non-const) `strbuf` structure for it\n>> caused some confusion in the past because it implied that someone\n>> could rename a thread after it was created.  That usage was not\n>> intended.  Changing it to a \"flex array\" will hopefully make the\n>> intent more clear.\n>\n> Surely, \"const struct strbuf name;\" member would be an oxymoron, and\n> I agree that this should follow \"use strbuf as an easy-to-work-with\n> mechanism to come up with a string, and bake the final value into a\n> struct as a member of type 'const char []'\" pattern.\n>\n> I recall saying why I thought the flex array was overkill, though.\n>\n> You have been storing an up-to-24-byte human readable name by\n> embedding a strbuf that has two size_t plus a pointer (i.e. 24-bytes\n> even on Windows), and as TR2_MAX_THREAD_NAME is capped at 24 bytes\n> anyway, an embedded fixed-size thread_name[TR2_MAX_THREAD_NAME+1]\n> member may be the simplest thing to do, I suspect.\n>\n> If we were to allow arbitrarily long thread_name[], which may not be\n> a bad thing to do (e.g. we do not have to worry about truncation\n> making two names ambiguous, for example), then the flex array is the\n> right direction to go in, though.\n\nWe don't even need that, AFAICT. My reply at [1] is rather long, but the\ntl;dr is that the interface for this API is:\n\t\n\t$ git grep '^\\s+trace2_thread_start'\n\tDocumentation/technical/api-trace2.txt: trace2_thread_start(\"preload_thread\");\n\tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-health\");\n\tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-listen\");\n\tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-worker\");\n\tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-accept\");\n\tcompat/simple-ipc/ipc-win32.c:  trace2_thread_start(\"ipc-server\");\n\tt/helper/test-fsmonitor-client.c:       trace2_thread_start(\"hammer\");\n\tt/helper/test-simple-ipc.c:     trace2_thread_start(\"multiple\");\n\ttrace2.h:       trace2_thread_start_fl((thread_hint), __FILE__, __LINE__)\n\nAnd we are taking e.g. \"preload_thread\" and turning it into strings like\nthese, and saving it into \"struct tr2tls_thread_ctx\".\n\n\t\"preload_thread\", // main thread\n\t\"th01:preload_thread\", // 1st thread\n\t\"th02:preload_thread\" // 2nd thread\n\t[...]\n\nSo, we don't need to strdup() and store that \"preload_thread\" anywhere.\nIt's already a constant string we have hardcoded in the program. We just\nneed to save a pointer to it.\n\nThen we just format the \"%s\" or (if \".thread_id\" == 0) or \"th%02d:%s\"\n(if \".thread_id\" > 0) on-the-fly, the two codepaths that end up using\nthis are already using strbuf_addf(), so just adding to the format there\nis easy.\n\n1. https://lore.kernel.org/git/221005.86y1tus9ps.gmgdl@evledraar.gmail.com/\n"},{"id":"464335","messageId":"xmqqr0zkipva.fsf@gitster.g","threadId":"58564","inReplyTo":"221006.86ilkwr6wy.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-06T21:50:01Z","receivedAt":"2022-10-06T21:50:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n>> If we were to allow arbitrarily long thread_name[], which may not be\n>> a bad thing to do (e.g. we do not have to worry about truncation\n>> making two names ambiguous, for example), then the flex array is the\n>> right direction to go in, though.\n>\n> We don't even need that, AFAICT. ...\n> ...\n> And we are taking e.g. \"preload_thread\" and turning it into strings like\n> these, and saving it into \"struct tr2tls_thread_ctx\".\n>\n> \t\"preload_thread\", // main thread\n> \t\"th01:preload_thread\", // 1st thread\n> \t\"th02:preload_thread\" // 2nd thread\n> \t[...]\n>\n> So, we don't need to strdup() and store that \"preload_thread\" anywhere.\n> It's already a constant string we have hardcoded in the program. We just\n> need to save a pointer to it.\n\nThat sounds even simpler.\n"},{"id":"464341","messageId":"RFC-patch-1.1-8563d017137-20221007T010829Z-avarab@gmail.com","threadId":"58564","inReplyTo":"xmqqr0zkipva.fsf@gitster.g","subject":"[RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-07T01:10:06Z","receivedAt":"2022-10-07T01:10:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Since ee4512ed481 (trace2: create new combined trace facility,\n2019-02-22) the \"thread_name\" member of \"struct tr2tls_thread_ctx\" has\nbeen copied from the caller, but those callers have always passed a\nconstant string:\n\n\t$ git -P grep '^\\s*trace2_thread_start\\('\n\tDocumentation/technical/api-trace2.txt: trace2_thread_start(\"preload_thread\");\n\tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-health\");\n\tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-listen\");\n\tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-worker\");\n\tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-accept\");\n\tcompat/simple-ipc/ipc-win32.c:  trace2_thread_start(\"ipc-server\");\n\tt/helper/test-fsmonitor-client.c:       trace2_thread_start(\"hammer\");\n\tt/helper/test-simple-ipc.c:     trace2_thread_start(\"multiple\");\n\nThis isn't needed for optimization, but apparently[1] there's been\nsome confusion about the non-const-ness of the previous \"struct\nstrbuf\".\n\nUsing the caller's string here makes this more straightforward, as\nit's now clear that we're not dynamically constructing these. It's\nalso what the progress API does with its \"title\" string.\n\nSince we know we're hardcoding these thread names let's BUG() out when\nwe see that the length of the name plus the length of the prefix would\nexceed the maximum length for the \"perf\" format.\n\n1. https://lore.kernel.org/git/82f1672e180afcd876505a4354bd9952f70db49e.1664900407.git.gitgitgadget@gmail.com/\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n\nOn Thu, Oct 06 2022, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n>> So, we don't need to strdup() and store that \"preload_thread\" anywhere.\n>> It's already a constant string we have hardcoded in the program. We just\n>> need to save a pointer to it.\n>\n> That sounds even simpler.\n\nA cleaned up version of the test code I had on top of \"master\", RFC\nbecause I may still be missing some context here. E.g. maybe there's a\nplan to dynamically construct these thread names?\n\n json-writer.c          | 17 +++++++++++++++++\n json-writer.h          |  4 ++++\n trace2/tr2_tgt_event.c |  2 +-\n trace2/tr2_tgt_perf.c  | 10 +++++++---\n trace2/tr2_tls.c       | 14 +++++---------\n trace2/tr2_tls.h       |  9 +++++++--\n 6 files changed, 41 insertions(+), 15 deletions(-)\n\ndiff --git a/json-writer.c b/json-writer.c\nindex f1cfd8fa8c6..569a75bee51 100644\n--- a/json-writer.c\n+++ b/json-writer.c\n@@ -161,6 +161,23 @@ void jw_object_string(struct json_writer *jw, const char *key, const char *value\n \tappend_quoted_string(&jw->json, value);\n }\n \n+void jw_strbuf_add_thread_name(struct strbuf *sb, const char *thread_name,\n+\t\t\t       int thread_id)\n+{\n+\tif (thread_id)\n+\t\tstrbuf_addf(sb, \"th%02d:\", thread_id);\n+\tstrbuf_addstr(sb, thread_name);\n+}\n+\n+void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n+\t\t\t     int thread_id)\n+{\n+\tobject_common(jw, \"thread\");\n+\tstrbuf_addch(&jw->json, '\"');\n+\tjw_strbuf_add_thread_name(&jw->json, thread_name, thread_id);\n+\tstrbuf_addch(&jw->json, '\"');\n+}\n+\n void jw_object_intmax(struct json_writer *jw, const char *key, intmax_t value)\n {\n \tobject_common(jw, key);\ndiff --git a/json-writer.h b/json-writer.h\nindex 209355e0f12..269c203b119 100644\n--- a/json-writer.h\n+++ b/json-writer.h\n@@ -75,6 +75,10 @@ void jw_release(struct json_writer *jw);\n void jw_object_begin(struct json_writer *jw, int pretty);\n void jw_array_begin(struct json_writer *jw, int pretty);\n \n+void jw_strbuf_add_thread_name(struct strbuf *buf, const char *thread_name,\n+\t\t\t       int thread_id);\n+void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n+\t\t\t     int thread_id);\n void jw_object_string(struct json_writer *jw, const char *key,\n \t\t      const char *value);\n void jw_object_intmax(struct json_writer *jw, const char *key, intmax_t value);\ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 37a3163be12..1308cf05df4 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -90,7 +90,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n \n \tjw_object_string(jw, \"event\", event_name);\n \tjw_object_string(jw, \"sid\", tr2_sid_get());\n-\tjw_object_string(jw, \"thread\", ctx->thread_name.buf);\n+\tjw_object_string_thread(jw, ctx->thread_name, ctx->thread_id);\n \n \t/*\n \t * In brief mode, only emit <time> on these 2 event types.\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 8cb792488c8..ab21277eb36 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -69,6 +69,8 @@ static void perf_fmt_prepare(const char *event_name,\n \t\t\t     const char *category, struct strbuf *buf)\n {\n \tint len;\n+\tsize_t oldlen;\n+\tint padlen;\n \n \tstrbuf_setlen(buf, 0);\n \n@@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n \t}\n \n \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n-\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n-\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n-\t\t    event_name);\n+\toldlen = buf->len;\n+\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n+\tpadlen = TR2_MAX_THREAD_NAME - (buf->len - oldlen);;\n+\tstrbuf_addf(buf, \"%-*s | %-*s | \", padlen, \"\",\n+\t\t    TR2FMT_PERF_MAX_EVENT_NAME, event_name);\n \n \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\n \tif (repo)\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 7da94aba522..aa9aeb67fca 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -36,6 +36,9 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n \n+\tif (strlen(thread_name) + TR2_MAX_THREAD_NAME_PREFIX > TR2_MAX_THREAD_NAME)\n+\t\tBUG(\"too long thread name '%s'\", thread_name);\n+\n \t/*\n \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n \t * time in array_us_start[0].  For the main thread this gives us the\n@@ -45,15 +48,9 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n \n+\tctx->thread_name = thread_name;\n \tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n \n-\tstrbuf_init(&ctx->thread_name, 0);\n-\tif (ctx->thread_id)\n-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_name);\n-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n-\n \tpthread_setspecific(tr2tls_key, ctx);\n \n \treturn ctx;\n@@ -95,7 +92,6 @@ void tr2tls_unset_self(void)\n \n \tpthread_setspecific(tr2tls_key, NULL);\n \n-\tstrbuf_release(&ctx->thread_name);\n \tfree(ctx->array_us_start);\n \tfree(ctx);\n }\n@@ -113,7 +109,7 @@ void tr2tls_pop_self(void)\n \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n \n \tif (!ctx->nr_open_regions)\n-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n+\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n \n \tctx->nr_open_regions--;\n }\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex b1e327a928e..f600eb22551 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -4,12 +4,17 @@\n #include \"strbuf.h\"\n \n /*\n- * Arbitry limit for thread names for column alignment.\n+ * Arbitry limit for thread names for column alignment. The overall\n+ * max length is TR2_MAX_THREAD_NAME, and the\n+ * TR2_MAX_THREAD_NAME_PREFIX is the length of the formatted\n+ * '\"th%02d:\", ctx->thread_id' prefix which is added when \"thread_id >\n+ * 0\".\n  */\n+#define TR2_MAX_THREAD_NAME_PREFIX (5)\n #define TR2_MAX_THREAD_NAME (24)\n \n struct tr2tls_thread_ctx {\n-\tstruct strbuf thread_name;\n+\tconst char *thread_name;\n \tuint64_t *array_us_start;\n \tint alloc;\n \tint nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n-- \n2.38.0.971.ge79ff6d20e7\n\n"},{"id":"464342","messageId":"xmqqo7uoh1q0.fsf@gitster.g","threadId":"58564","inReplyTo":"RFC-patch-1.1-8563d017137-20221007T010829Z-avarab@gmail.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-07T01:16:55Z","receivedAt":"2022-10-07T01:17:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> A cleaned up version of the test code I had on top of \"master\", RFC\n> because I may still be missing some context here. E.g. maybe there's a\n> plan to dynamically construct these thread names?\n\nThat's nice to learn, indeed.\n\n> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n> +\t\t\t     int thread_id)\n> +{\n> +\tobject_common(jw, \"thread\");\n> +\tstrbuf_addch(&jw->json, '\"');\n> +\tjw_strbuf_add_thread_name(&jw->json, thread_name, thread_id);\n> +\tstrbuf_addch(&jw->json, '\"');\n> +}\n\n...\n\n> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>  \t}\n>  \n>  \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n> -\t\t    event_name);\n> +\toldlen = buf->len;\n> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n> +\tpadlen = TR2_MAX_THREAD_NAME - (buf->len - oldlen);;\n> +\tstrbuf_addf(buf, \"%-*s | %-*s | \", padlen, \"\",\n> +\t\t    TR2FMT_PERF_MAX_EVENT_NAME, event_name);\n\nHaving to do strbuf_addf() many times may negatively affect perf_*\nstuff, if this code is invoked in the hot path.  I however tend to\ntreat anything that involves an I/O not performance critical, and\nthis certainly falls into that category.\n\n"},{"id":"464367","messageId":"221007.865ygvrjs7.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"xmqqo7uoh1q0.fsf@gitster.g","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-07T10:03:45Z","receivedAt":"2022-10-07T10:49:37Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 06 2022, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n>\n>> A cleaned up version of the test code I had on top of \"master\", RFC\n>> because I may still be missing some context here. E.g. maybe there's a\n>> plan to dynamically construct these thread names?\n>\n> That's nice to learn, indeed.\n>\n>> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n>> +\t\t\t     int thread_id)\n>> +{\n>> +\tobject_common(jw, \"thread\");\n>> +\tstrbuf_addch(&jw->json, '\"');\n>> +\tjw_strbuf_add_thread_name(&jw->json, thread_name, thread_id);\n>> +\tstrbuf_addch(&jw->json, '\"');\n>> +}\n>\n> ...\n>\n>> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>>  \t}\n>>  \n>>  \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n>> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n>> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n>> -\t\t    event_name);\n>> +\toldlen = buf->len;\n>> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n>> +\tpadlen = TR2_MAX_THREAD_NAME - (buf->len - oldlen);;\n>> +\tstrbuf_addf(buf, \"%-*s | %-*s | \", padlen, \"\",\n>> +\t\t    TR2FMT_PERF_MAX_EVENT_NAME, event_name);\n>\n> Having to do strbuf_addf() many times may negatively affect perf_*\n> stuff, if this code is invoked in the hot path.  I however tend to\n> treat anything that involves an I/O not performance critical, and\n> this certainly falls into that category.\n\nYes, and that function already called strbuf_addf() 5-7 times, this adds\none more, but only if \"thread_id\" is > 0.\n\nThe reason I added jw_object_string_thread() was to avoid the malloc() &\nfree() of a temporary \"struct strbuf\", it would have been more\nstraightforward to call jw_object_string() like that.\n\nI don't think anyone cares about the raw performance of the \"perf\"\noutput, but the \"JSON\" one needs to be fast(er).\n\nBut even that output will malloc()/free() for each line it emits, and\noften multiple times within one line (e.g. each time we format a\ndouble).\n\nSo if we do want to optimize this in terms of memory use the lowest\nhanging fruit seems to be to just have a per-thread \"scratch\" buffer\nwe'd write to, we could also observe that we're writing to a file and\njust directly write to it in most cases (although we'd need to be\ncareful to write partial-and-still-invalid JSON lines in that case...).\n"},{"id":"464519","messageId":"8e3524e7-5e92-99d6-294e-4c309a3d44ee@jeffhostetler.com","threadId":"58564","inReplyTo":"221005.86y1tus9ps.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-10T18:31:37Z","receivedAt":"2022-10-10T18:31:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/5/22 7:14 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Oct 04 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n>> to a \"flex array\" at the end of the context structure.\n>>\n>> The `thread_name` field is a constant string that is constructed when\n>> the context is created.  Using a (non-const) `strbuf` structure for it\n>> caused some confusion in the past because it implied that someone\n>> could rename a thread after it was created.\n> \n> I think it's been long enough that we could use a reminder about the\n> \"some confusion\", i.e. if it was a bug report or something else.\n> \n>> That usage was not intended.  Changing it to a \"flex array\" will\n>> hopefully make the intent more clear.\n> \n> I see we had some back & forth back in the original submission, although\n> honestly I skimmed this this time around, had forgetten about that, and\n> had this pop out at me, and then found my earlier comments.\n> \n> I see that exchange didn't end as well as I'd hoped[1], and hopefully we\n> can avoid that here. So having looked at this with fresh eyes maybe\n> these comments/questions help:\n> \n>   * I'm unable to bridge the cap from (paraphrased) \"we must change the\n>     type\" to \"mak[ing] the [read-only] intent more clear\".\n> \n>     I.e. if you go across the codebase and look at various non-const\n>     \"char name[FLEX_ARRAY]\" and add a \"const\" to them you'll find cases\n>     where we re-write the \"FLEX_ARRAY\" string, e.g. the one in archive.c\n>     is one of those (the first grep hit, I stopped looking for others at\n>     that point).\n> \n>     Making it \"const\" will yield:\n>     \n>        archive.c: In function ‘queue_directory’:\n>     archive.c:206:29: error: passing argument 1 of ‘xsnprintf’ discards ‘const’ qualifier from pointer target type [-Werror=discarded-qualifiers]\n>       206 |         d->len = xsnprintf(d->path, len, \"%.*s%s/\", (int)base->len, base->buf, filename);\n> \n>     So aside from anything else (and I may be misunderstanding this) why\n>     does changing it to a FLEX_ARRAY give us the connotation in the\n>     confused API user's mind that it shouldn't be messed with that the\n>     \"strbuf\" doesn't give us?\n[...]\n\nMy change in how we store the thread-name in the thread context was JUST\nto clarify that it should be treated as a constant string and that code\nshould not try to modify it.  There was a comment to that effect last\nyear -- that having it be a strbuf invited one to modify it, when that\nwas not the intent.\n\nThat was all I was trying to do here.  Just make it \"not be a strbuf\".\nPerhaps I lept too far by making it a flex-array.  I probably could\nhave just changed the field to a \"char *\" and detached it from the\n(now local) strbuf.  That would give the same impression, right?\n\n\n[...]\n>>   \t/*\n>>   \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n>> @@ -45,15 +56,6 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n>>   \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n>>   \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n>>   \n>> -\tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n>> -\n>> -\tstrbuf_init(&ctx->thread_name, 0);\n>> -\tif (ctx->thread_id)\n>> -\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n>> -\tstrbuf_addstr(&ctx->thread_name, name_hint);\n>> -\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n>> -\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n>> -\n>>   \tpthread_setspecific(tr2tls_key, ctx);\n>>   \n>>   \treturn ctx;\n> \n> I found this quote hard to follow because there's functional changes\n> there mixed up with code re-arangement, consider leading with a commit\n> like:\n[...]\n\nsorry about that.  yes, there's a bit of churn here because i\nneeded to reorder the thread-name construction to be before we\nallocated the context so that we'd know the buffer size.\n\nand yes, i accidentally mixed in a function change to move the\ntruncation to the perf backend.\n\ni'll redo all of this.\n\n\n[...]\n> <tries it out>\n> \n> Anyway, if this area was actually performance critical and we *really\n> cared* about avoiding allocations wouldn't we want to skip both the\n> \"strbuf\" there and the \"FLEX_ARRAY\", and just save away the\n> \"thread_hint\" (which the caller hardcodes) and \"thread_nr\", and then\n> append on-the-fly?\n> \n> I came up with the below to do that, it passes all tests, but contains\n> micro-optimizations that I don't think we need (e.g. I understood you\n> wanted to avoid printf, so it does that).\n> \n> But I think it's a useful point of discussion. What test(s) do you have\n> where the \"master\" version, FLEX_ARRAY version, and just not strbuf\n> formatting the thing at all differ?\n[...]\n\nnone of this was about micro-optimization.  i was just trying to get\nthe buffer away from a strbuf.  i still want it pre-formatted once\nat thread-start, but that's it.\n\nFWIW, I don't think having it formatted in each event helps anything.\nit would have to go thru sprintf on every message.  it's much better\nto just format it once in the thread-start.\n\n\n[...]\n> \tdiff --git a/json-writer.c b/json-writer.c\n[...] \t\n> \t+void jw_strbuf_add_thread_name(struct strbuf *out, const char *thread_hint,\n> \t+\t\t\t       int thread_id, int max_len)\n> \t+{\n[...]\n> \t+}\n> \t+\n> \t+void jw_object_thread(struct json_writer *jw, const char *thread_hint,\n> \t+\t\t      int thread_id)\n> \t+{\n[...]\n> \t+}\n[...]\n\nWe should not do this.  Just format the name in thread-start and\nlet json-writer print the string as we have been.\n\nAdding thread formatting to json-writer also violates a separation\nof concerns.\n\nI'll re-roll this commit completely.\n\nthanks\nJeff\n"},{"id":"464520","messageId":"3fbc1d79-4b7d-ef5d-1056-0512a3aa4e9a@jeffhostetler.com","threadId":"58564","inReplyTo":"221006.86ilkwr6wy.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 6/9] trace2: convert ctx.thread_name to flex array","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-10T18:39:37Z","receivedAt":"2022-10-10T18:39:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/6/22 5:05 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 05 2022, Junio C Hamano wrote:\n> \n>> \"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>>\n>>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>>\n>>> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n>>> to a \"flex array\" at the end of the context structure.\n>>>\n[...]\n> \n> We don't even need that, AFAICT. My reply at [1] is rather long, but the\n> tl;dr is that the interface for this API is:\n> \t\n> \t$ git grep '^\\s+trace2_thread_start'\n> \tDocumentation/technical/api-trace2.txt: trace2_thread_start(\"preload_thread\");\n> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-health\");\n> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-listen\");\n> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-worker\");\n> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-accept\");\n> \tcompat/simple-ipc/ipc-win32.c:  trace2_thread_start(\"ipc-server\");\n> \tt/helper/test-fsmonitor-client.c:       trace2_thread_start(\"hammer\");\n> \tt/helper/test-simple-ipc.c:     trace2_thread_start(\"multiple\");\n> \ttrace2.h:       trace2_thread_start_fl((thread_hint), __FILE__, __LINE__)\n> \n> And we are taking e.g. \"preload_thread\" and turning it into strings like\n> these, and saving it into \"struct tr2tls_thread_ctx\".\n> \n> \t\"preload_thread\", // main thread\n> \t\"th01:preload_thread\", // 1st thread\n> \t\"th02:preload_thread\" // 2nd thread\n> \t[...]\n> \n> So, we don't need to strdup() and store that \"preload_thread\" anywhere.\n> It's already a constant string we have hardcoded in the program. We just\n> need to save a pointer to it.\n\nCurrent callers tend to pass a string literal.  There's nothing\nto say that they will continue to do so in the future.\n\n\n> Then we just format the \"%s\" or (if \".thread_id\" == 0) or \"th%02d:%s\"\n> (if \".thread_id\" > 0) on-the-fly, the two codepaths that end up using\n> this are already using strbuf_addf(), so just adding to the format there\n> is easy.\n[...]\n\nBut then you'd be formatting this \"th%0d:%s\" on every message\nprinter.  Whereas we can format it once in the thread-start and\nsave the extra work -- at the expense of a string buffer in the\nthread context.\n\nGranted, the event handlers are generating output lines with many\n\"%\" fields, so they are doing non-trivial amounts of work on every\nevent, but by using a pre-formatted thread-name, we don't need to\nincrease that workload.\n\nJeff\n\n\n"},{"id":"464525","messageId":"cb9f8321-d9e6-6f80-a590-a9ad49c7f557@jeffhostetler.com","threadId":"58564","inReplyTo":"RFC-patch-1.1-8563d017137-20221007T010829Z-avarab@gmail.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-10T19:05:42Z","receivedAt":"2022-10-10T19:06:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/6/22 9:10 PM, Ævar Arnfjörð Bjarmason wrote:\n> Since ee4512ed481 (trace2: create new combined trace facility,\n> 2019-02-22) the \"thread_name\" member of \"struct tr2tls_thread_ctx\" has\n> been copied from the caller, but those callers have always passed a\n> constant string:\n> \n> \t$ git -P grep '^\\s*trace2_thread_start\\('\n> \tDocumentation/technical/api-trace2.txt: trace2_thread_start(\"preload_thread\");\n> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-health\");\n> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-listen\");\n> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-worker\");\n> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-accept\");\n> \tcompat/simple-ipc/ipc-win32.c:  trace2_thread_start(\"ipc-server\");\n> \tt/helper/test-fsmonitor-client.c:       trace2_thread_start(\"hammer\");\n> \tt/helper/test-simple-ipc.c:     trace2_thread_start(\"multiple\");\n> \n> This isn't needed for optimization, but apparently[1] there's been\n> some confusion about the non-const-ness of the previous \"struct\n> strbuf\".\n> \n> Using the caller's string here makes this more straightforward, as\n> it's now clear that we're not dynamically constructing these. It's\n> also what the progress API does with its \"title\" string.\n> \n> Since we know we're hardcoding these thread names let's BUG() out when\n> we see that the length of the name plus the length of the prefix would\n> exceed the maximum length for the \"perf\" format.\n> \n> 1. https://lore.kernel.org/git/82f1672e180afcd876505a4354bd9952f70db49e.1664900407.git.gitgitgadget@gmail.com/\n> \n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n\nPLEASE DON'T DO THIS.\n\nIf you don't like my patch, fine.  Let's discuss it.  But DON'T submit\na new one to replace it.  Or worse, try to inject it into the middle\nof an existing series.\n\n\nYes, current callers are passing a string literal and thread-start\ncould take a \"const char*\" to it, but there is no way to guarantee\nthat that is safe if someone decides to dynamically construct their\nthread-name and pass it in (since we don't know the lifetime of that\npointer).  So it is safer to copy it into the thread context so that\nit can be used by later trace messages.\n\n\n[...]\n> +void jw_strbuf_add_thread_name(struct strbuf *buf, const char *thread_name,\n> +\t\t\t       int thread_id);\n> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n> +\t\t\t     int thread_id);\n\nThis violates a separation of concerns.  json-writer is ONLY concerned\nwith formatting valid JSON from basic data types.  It does not know\nabout threads or thread contexts.\n\n`js_strbuf_add_thread_name()` also violates the json-writer conventions\n-- that it takes a \"struct json_writer *\" pointer.  There is nothing\nabout JSON here.\n\nYou might write a helper (inside of tr2_tgt_event.c) that formats a\nthread-name from the id and hint, but that is specific to the Event\ntarget -- not to JSON, nor the JSON writer.\n\nBut then again, why make every trace message from every target format\nthat \"th%0d:%s\" when we could save some time and format it in the\nthread-start and just USE it.\n\n\n[...]\n> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>   \t}\n>   \n>   \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n> -\t\t    event_name);\n> +\toldlen = buf->len;\n> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n\nThis stands out as very wrong.  The _Perf target does not use JSON\nat all, yet here we are calling a jw_ routine.  Again, that code is\nin the wrong place.\n\n\nI'm going to clip the rest of this commit, since the above invalidates\nit.\n\nJeff\n"},{"id":"464527","messageId":"afc73d87-b2d9-72e9-1be5-156f37102747@jeffhostetler.com","threadId":"58564","inReplyTo":"221007.865ygvrjs7.gmgdl@evledraar.gmail.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-10T19:16:45Z","receivedAt":"2022-10-10T19:16:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/7/22 6:03 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Oct 06 2022, Junio C Hamano wrote:\n> \n>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n>>\n>>> A cleaned up version of the test code I had on top of \"master\", RFC\n>>> because I may still be missing some context here. E.g. maybe there's a\n>>> plan to dynamically construct these thread names?\n>>\n>> That's nice to learn, indeed.\n>>\n>>> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n>>> +\t\t\t     int thread_id)\n>>> +{\n>>> +\tobject_common(jw, \"thread\");\n>>> +\tstrbuf_addch(&jw->json, '\"');\n>>> +\tjw_strbuf_add_thread_name(&jw->json, thread_name, thread_id);\n>>> +\tstrbuf_addch(&jw->json, '\"');\n>>> +}\n>>\n>> ...\n>>\n>>> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>>>   \t}\n>>>   \n>>>   \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n>>> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n>>> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n>>> -\t\t    event_name);\n>>> +\toldlen = buf->len;\n>>> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n>>> +\tpadlen = TR2_MAX_THREAD_NAME - (buf->len - oldlen);;\n>>> +\tstrbuf_addf(buf, \"%-*s | %-*s | \", padlen, \"\",\n>>> +\t\t    TR2FMT_PERF_MAX_EVENT_NAME, event_name);\n>>\n>> Having to do strbuf_addf() many times may negatively affect perf_*\n>> stuff, if this code is invoked in the hot path.  I however tend to\n>> treat anything that involves an I/O not performance critical, and\n>> this certainly falls into that category.\n> \n> Yes, and that function already called strbuf_addf() 5-7 times, this adds\n> one more, but only if \"thread_id\" is > 0.\n> \n> The reason I added jw_object_string_thread() was to avoid the malloc() &\n> free() of a temporary \"struct strbuf\", it would have been more\n> straightforward to call jw_object_string() like that.\n> \n> I don't think anyone cares about the raw performance of the \"perf\"\n> output, but the \"JSON\" one needs to be fast(er).\n> \n> But even that output will malloc()/free() for each line it emits, and\n> often multiple times within one line (e.g. each time we format a\n> double).\n> \n> So if we do want to optimize this in terms of memory use the lowest\n> hanging fruit seems to be to just have a per-thread \"scratch\" buffer\n> we'd write to, we could also observe that we're writing to a file and\n> just directly write to it in most cases (although we'd need to be\n> careful to write partial-and-still-invalid JSON lines in that case...).\n> \n\nWRT optimizing memory usage.  We're talking about ~25 byte buffer\nper thread.  Most commands execute in 1 thread -- if they read the\nindex they may have ~10 threads (depending on the size of the index\nand if preload-index is enabled).  So, I don't think we really need\nto optimize this.  Threading is used extensively in fsmonitor-daemon,\nbut it creates a fixed thread-pool at startup, so it may have ~12\nthreads.  Again, not worth optimizing for the thread-name field.\n\nNow, if you want to optimize over all trace2 events (a completely\ndifferent topic), you could create a large scratch strbuf buffer in\neach thread context and use it so that we don't have to malloc/free\nduring each trace message.  That might be worth while.\n\n\nWe must not do partial writes to the trace2 files as we're\nconstructing fields.  The trace2 files are opened with O_APPEND\nso that we get the atomic lseek(2)+write(2) so that lines get\nwritten without overwrites when multiple threads and/or processes\nare tracing.\n\nAlso, when writing to a named pipe, we get \"message\" semantics\non write() boundaries, which makes post-processing easier.\n\nJeff\n"},{"id":"464610","messageId":"221011.86lepmo5dn.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"cb9f8321-d9e6-6f80-a590-a9ad49c7f557@jeffhostetler.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-11T12:52:11Z","receivedAt":"2022-10-11T13:29:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 10 2022, Jeff Hostetler wrote:\n\n> On 10/6/22 9:10 PM, Ævar Arnfjörð Bjarmason wrote:\n>> Since ee4512ed481 (trace2: create new combined trace facility,\n>> 2019-02-22) the \"thread_name\" member of \"struct tr2tls_thread_ctx\" has\n>> been copied from the caller, but those callers have always passed a\n>> constant string:\n>> \t$ git -P grep '^\\s*trace2_thread_start\\('\n>> \tDocumentation/technical/api-trace2.txt: trace2_thread_start(\"preload_thread\");\n>> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-health\");\n>> \tbuiltin/fsmonitor--daemon.c:    trace2_thread_start(\"fsm-listen\");\n>> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-worker\");\n>> \tcompat/simple-ipc/ipc-unix-socket.c:    trace2_thread_start(\"ipc-accept\");\n>> \tcompat/simple-ipc/ipc-win32.c:  trace2_thread_start(\"ipc-server\");\n>> \tt/helper/test-fsmonitor-client.c:       trace2_thread_start(\"hammer\");\n>> \tt/helper/test-simple-ipc.c:     trace2_thread_start(\"multiple\");\n>> This isn't needed for optimization, but apparently[1] there's been\n>> some confusion about the non-const-ness of the previous \"struct\n>> strbuf\".\n>> Using the caller's string here makes this more straightforward, as\n>> it's now clear that we're not dynamically constructing these. It's\n>> also what the progress API does with its \"title\" string.\n>> Since we know we're hardcoding these thread names let's BUG() out\n>> when\n>> we see that the length of the name plus the length of the prefix would\n>> exceed the maximum length for the \"perf\" format.\n>> 1. https://lore.kernel.org/git/82f1672e180afcd876505a4354bd9952f70db49e.1664900407.git.gitgitgadget@gmail.com/\n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>\n> PLEASE DON'T DO THIS.\n>\n> If you don't like my patch, fine.  Let's discuss it.  But DON'T submit\n> a new one to replace it.  Or worse, try to inject it into the middle\n> of an existing series.\n\nI'm not seeking to replace your series, or to tick you off, sorry if it\ncame across like that.\n\nI just thought (and still think) that we were at a point in the\ndiscussion where it seemed clear that I wasn't quite managing to get\nacross to you what I meant, so sending that in the form of working code\nshould clarify things.\n\nPer Junio's \"That's nice to learn, indeed.\" in\n<xmqqo7uoh1q0.fsf@gitster.g> it seems to have had that intended effect\non him. It's marked as an RFC, so not-a-thing-to-pick-up, but just for\ndiscussion.\n\n> Yes, current callers are passing a string literal and thread-start\n> could take a \"const char*\" to it, but there is no way to guarantee\n> that that is safe if someone decides to dynamically construct their\n> thread-name and pass it in (since we don't know the lifetime of that\n> pointer).  So it is safer to copy it into the thread context so that\n> it can be used by later trace messages.\n\nI think that's a defensible opinion, but I also think it's fair to say\nthat:\n\n * This seems to be *the* motivation for why you're doing things the way\n   you're doing them, and at least to this reviewer that wasn't really\n   coming across...\n\n * ...nor the context of why we'd need that sort of guarded API in this\n   case, but not e.g. for another widely-used API like start_progress().\n\n   See 791afae2924 (progress.c tests: make start/stop commands on stdin,\n   2022-02-03) for a case where we're using that where we need to work\n   around its behavior (and no, I didn't make the underlying API that\n   way, it's just a commit of mine where I'm having to work with it).\n\nI think designing our internal APIs to not be quite so guarded is fine,\nand we do that in various other contexts (progress, etc.). We control\nboth the API and its users, so just leaving a \"this must be a constant\"\nshould be enough.\n\nBut even if you want to be paranoid about it there's much easier ways to\ndo that which give you more of the safety you seem to want. E.g. this on\ntop of master (and easily adjusted on top of this RFC patch):\n\t\n\tdiff --git a/trace2.h b/trace2.h\n\tindex 88d906ea830..1c3a98fb30f 100644\n\t--- a/trace2.h\n\t+++ b/trace2.h\n\t@@ -306,12 +306,18 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n\t  *\n\t  * Thread names should be descriptive, like \"preload_index\".\n\t  * Thread names will be decorated with an instance number automatically.\n\t+ * Thread names must point to data that won't change after it's passed\n\t+ * into this function. Once trace2_thread_exit() is called it can be\n\t+ * free'd.\n\t  */\n\t void trace2_thread_start_fl(const char *file, int line,\n\t \t\t\t    const char *thread_name);\n\t \n\t+/*\n\t+ * The \"\" is to assure us that API users pass only constant strings\n\t+ */\n\t #define trace2_thread_start(thread_name) \\\n\t-\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n\t+\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name \"\"))\n\t \n\t /*\n\t  * Emit a 'thread_exit' event.  This must be called from inside the\n\nWill pass, as we only pass it constant strings, but if someone were to\npass a variable it'll blow up, at which point we could provide some\ninline macro/function that would do the required xstrdup().\n\nAll of which I think is *still* being too paranoid, but which I think\n*if* you want the paranoia is much more explicit about what we're trying\nto accomplish with said paranoida, and where the compiler will help you.\n\n> [...]\n>> +void jw_strbuf_add_thread_name(struct strbuf *buf, const char *thread_name,\n>> +\t\t\t       int thread_id);\n>> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n>> +\t\t\t     int thread_id);\n>\n> This violates a separation of concerns.  json-writer is ONLY concerned\n> with formatting valid JSON from basic data types.  It does not know\n> about threads or thread contexts.\n>\n> `js_strbuf_add_thread_name()` also violates the json-writer conventions\n> -- that it takes a \"struct json_writer *\" pointer.  There is nothing\n> about JSON here.\n>\n> You might write a helper (inside of tr2_tgt_event.c) that formats a\n> thread-name from the id and hint, but that is specific to the Event\n> target -- not to JSON, nor the JSON writer.\n\nThat's fair, more on that below.\n\n> But then again, why make every trace message from every target format\n> that \"th%0d:%s\" when we could save some time and format it in the\n> thread-start and just USE it.\n\nIf you actually care about this being fasterer -- and only reason for\nposting this RFC patch is to try to tease out *why* that is -- then this\npart of your concern can be trivially mitigated with having a struct\nmember like:\n\n\tchar thread_id_str[3];\n\nWe'd then just snprintf() into that in tr2tls_create_self(). Then when\nwe print the thread to the JSON or log you'd do so without any\nstrbuf_addf(), just a strbuf_addstr() or strbuf_add().\n\nI think that micro-optimization isn't needed in this case, but it *is*\neasy to do .\n\n> [...]\n>> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>>   \t}\n>>     \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n>> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n>> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n>> -\t\t    event_name);\n>> +\toldlen = buf->len;\n>> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n>\n> This stands out as very wrong.  The _Perf target does not use JSON\n> at all, yet here we are calling a jw_ routine.  Again, that code is\n> in the wrong place.\n>\n> I'm going to clip the rest of this commit, since the above invalidates\n> it.\n\nA helper function being in the wrong place invalidates the whole commit?\n\nI think you're right that this jw_strbuf_add_thread_name() helper should\nlive somewhere else, probably in thread-utils.c.\n\nSo, pretending that it's in whatever place you'd be comfortable with,\nand using whatever naming convention you'd prefer. What do you think\nabout the rest of the commit?\n\nYou snippet it just as you were getting to the meaty part of it, namely:\n\n * With this approach we can BUG() out as soon as we try to construct\n   the main thread if its name is bad, we don't need to wait until\n   runtime when a child thread runs into the limit.\n\n * We no longer need the whole thread-creation-time string duplication,\n   associated storage in the struct etc.\n\n * That struct member is \"const\", addresing your initial concern of\n   (from the upthread commit message):\n\n\tUsing a (non-const) `strbuf` structure for it caused some\n\tconfusion in the past because it implied that someone could\n\trename a thread after it was created.  That usage was not\n\tintended.\n\n   Although I think (and I'm possibly misreading it) that your\n   commentary here is saying that even that's not enough, i.e. we can't\n   just leave it at a \"const\" here, but must assume that an API user\n   will disregard that and modify it after it's passed to us anyway.\n"},{"id":"464611","messageId":"221011.86h70ao4g6.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"afc73d87-b2d9-72e9-1be5-156f37102747@jeffhostetler.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-11T13:31:10Z","receivedAt":"2022-10-11T13:49:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Oct 10 2022, Jeff Hostetler wrote:\n\n> On 10/7/22 6:03 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Thu, Oct 06 2022, Junio C Hamano wrote:\n>> \n>>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n>>>\n>>>> A cleaned up version of the test code I had on top of \"master\", RFC\n>>>> because I may still be missing some context here. E.g. maybe there's a\n>>>> plan to dynamically construct these thread names?\n>>>\n>>> That's nice to learn, indeed.\n>>>\n>>>> +void jw_object_string_thread(struct json_writer *jw, const char *thread_name,\n>>>> +\t\t\t     int thread_id)\n>>>> +{\n>>>> +\tobject_common(jw, \"thread\");\n>>>> +\tstrbuf_addch(&jw->json, '\"');\n>>>> +\tjw_strbuf_add_thread_name(&jw->json, thread_name, thread_id);\n>>>> +\tstrbuf_addch(&jw->json, '\"');\n>>>> +}\n>>>\n>>> ...\n>>>\n>>>> @@ -107,9 +109,11 @@ static void perf_fmt_prepare(const char *event_name,\n>>>>   \t}\n>>>>     \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n>>>> -\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n>>>> -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n>>>> -\t\t    event_name);\n>>>> +\toldlen = buf->len;\n>>>> +\tjw_strbuf_add_thread_name(buf, ctx->thread_name, ctx->thread_id);\n>>>> +\tpadlen = TR2_MAX_THREAD_NAME - (buf->len - oldlen);;\n>>>> +\tstrbuf_addf(buf, \"%-*s | %-*s | \", padlen, \"\",\n>>>> +\t\t    TR2FMT_PERF_MAX_EVENT_NAME, event_name);\n>>>\n>>> Having to do strbuf_addf() many times may negatively affect perf_*\n>>> stuff, if this code is invoked in the hot path.  I however tend to\n>>> treat anything that involves an I/O not performance critical, and\n>>> this certainly falls into that category.\n>> Yes, and that function already called strbuf_addf() 5-7 times, this\n>> adds\n>> one more, but only if \"thread_id\" is > 0.\n>> The reason I added jw_object_string_thread() was to avoid the\n>> malloc() &\n>> free() of a temporary \"struct strbuf\", it would have been more\n>> straightforward to call jw_object_string() like that.\n>> I don't think anyone cares about the raw performance of the \"perf\"\n>> output, but the \"JSON\" one needs to be fast(er).\n>> But even that output will malloc()/free() for each line it emits,\n>> and\n>> often multiple times within one line (e.g. each time we format a\n>> double).\n>> So if we do want to optimize this in terms of memory use the lowest\n>> hanging fruit seems to be to just have a per-thread \"scratch\" buffer\n>> we'd write to, we could also observe that we're writing to a file and\n>> just directly write to it in most cases (although we'd need to be\n>> careful to write partial-and-still-invalid JSON lines in that case...).\n>> \n\nI left more extensive commentary in the side-thread in\nhttps://lore.kernel.org/git/221011.86lepmo5dn.gmgdl@evledraar.gmail.com/,\njust a quick reply here.\n\n> WRT optimizing memory usage.  We're talking about ~25 byte buffer\n> per thread.  Most commands execute in 1 thread -- if they read the\n> index they may have ~10 threads (depending on the size of the index\n> and if preload-index is enabled).  So, I don't think we really need\n> to optimize this.  Threading is used extensively in fsmonitor-daemon,\n> but it creates a fixed thread-pool at startup, so it may have ~12\n> threads.  Again, not worth optimizing for the thread-name field.\n\nYes, I agree it's not worth optimizing.\n\nThe reason for commenting on this part is that it isn't clear to me why\nyour proposed patch then isn't doing the more obvious \"it's not worth\noptimizing\" pattern, per Junio's [1] comment on the initial version.\n\nThe \"flex array\" method is seemingly taking pains to reduce the runtime\nmemory use of these by embedding this string in the space reserved for\nthe struct.\n\nSo it's just meant as a question for you & the proposed patch.\n\n> Now, if you want to optimize over all trace2 events (a completely\n> different topic), you could create a large scratch strbuf buffer in\n> each thread context and use it so that we don't have to malloc/free\n> during each trace message.  That might be worth while.\n\n*nod*\n\n> We must not do partial writes to the trace2 files as we're\n> constructing fields.  The trace2 files are opened with O_APPEND\n> so that we get the atomic lseek(2)+write(2) so that lines get\n> written without overwrites when multiple threads and/or processes\n> are tracing.\n>\n> Also, when writing to a named pipe, we get \"message\" semantics\n> on write() boundaries, which makes post-processing easier.\n\n*nod*\n\n1. https://lore.kernel.org/git/xmqq8rwcjttq.fsf@gitster.g/\n2. https://lore.kernel.org/git/RFC-patch-1.1-8563d017137-20221007T010829Z-avarab@gmail.com/\n"},{"id":"464614","messageId":"xmqqlepmwhhr.fsf@gitster.g","threadId":"58564","inReplyTo":"221011.86lepmo5dn.gmgdl@evledraar.gmail.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-11T14:40:48Z","receivedAt":"2022-10-11T14:41:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> Per Junio's \"That's nice to learn, indeed.\" in\n> <xmqqo7uoh1q0.fsf@gitster.g> it seems to have had that intended effect\n> on him.\n\nI was commenting on the goal, i.e. you \"may still be missing some\ncontext here, maybe there's a plan to ...\", and I meant that the\nplan of the overall effort is something that is nice to learn before\ngoing further.  I was not endorsing the method you are taking to\nachieve that goal, though.\n\nFWIW, I find my code sent in as a comment easier to read than my\nprose alone for any topic, but that is only because it is \"my\" code\nis easy to read for \"me\".  I am sure others would find it\nunnecessary burden to figure out what the alternative/replacement I\nsend out intends to do and why it does so in the way it does, and\nwould rather appreciate if I explained these things in prose that is\neasy to understand and rich in \"why\", which alternative/replacement\ncode would solely lack.  Code snippet helps illustrate points on\n\"how\", but is often a poor replacement for proper explanation\nbecause it is a bad medium to convey \"why\".\n\nSame would apply to your code.  For others, including me, it often\nis a lot of work to figure out what your code is trying to do, and\nmore importantly why it does what it tries to do in the way it does.\n\nAfter all, when you are having hard time communicating why you want\nto do things differently from the patch author in prose, code\nsnippet would probably be the worst primary medium to do so, because\nit is full of \"how exactly\" with little \"why\".\n\n\n"},{"id":"464712","messageId":"e138b178-99d9-e537-2cb1-c240962a35f2@jeffhostetler.com","threadId":"58564","inReplyTo":"221011.86h70ao4g6.gmgdl@evledraar.gmail.com","subject":"Re: [RFC PATCH] trace2 API: don't save a copy of constant \"thread_name\"","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-12T13:31:09Z","receivedAt":"2022-10-12T13:31:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/11/22 9:31 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Oct 10 2022, Jeff Hostetler wrote:\n> \n>> On 10/7/22 6:03 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Thu, Oct 06 2022, Junio C Hamano wrote:\n>>>\n>>>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n>>>>\n>>>>> A cleaned up version of the test code I had on top of \"master\", RFC\n>>>>> because I may still be missing some context here. E.g. maybe there's a\n>>>>> plan to dynamically construct these thread names?\n>>>>\n[...]\n\n> I left more extensive commentary in the side-thread in\n> https://lore.kernel.org/git/221011.86lepmo5dn.gmgdl@evledraar.gmail.com/,\n> just a quick reply here.\n> \n>> WRT optimizing memory usage.  We're talking about ~25 byte buffer\n>> per thread.  Most commands execute in 1 thread -- if they read the\n>> index they may have ~10 threads (depending on the size of the index\n>> and if preload-index is enabled).  So, I don't think we really need\n>> to optimize this.  Threading is used extensively in fsmonitor-daemon,\n>> but it creates a fixed thread-pool at startup, so it may have ~12\n>> threads.  Again, not worth optimizing for the thread-name field.\n> \n> Yes, I agree it's not worth optimizing.\n> \n> The reason for commenting on this part is that it isn't clear to me why\n> your proposed patch then isn't doing the more obvious \"it's not worth\n> optimizing\" pattern, per Junio's [1] comment on the initial version.\n> \n> The \"flex array\" method is seemingly taking pains to reduce the runtime\n> memory use of these by embedding this string in the space reserved for\n> the struct.\n> \n> So it's just meant as a question for you & the proposed patch.\n\nI think we're converging on some common understanding (and I\nthink we've gone around on this topic more than enough).  :-)\n\nI really was just trying to get rid of the strbuf and make it\na fixed string -- I chose a flex-array rather than just detaching\nthe buffer from a local in the thread-start code.  I should have\ndone the latter.  I saw the flex-array as a fixed-size object\nthat can't be replaced or extended (without recreating the\nthread-local storage) -- yes, people could overwrite existing\nbytes in-place in the flex-array, but who does that??\n\n\nI understood what you were asking (illustrated in your RFC).\nThat is, I understood the \"what/how\" you wanted to do to refactor /\nredesign the field, but I couldn't understand the \"why\".  That\nis, why you've taken such interest in this field (and such\na relatively unimportant change).  We've spent nearly a week\ndiscussing it and we both agree that the optimization that I\ndidn't suggest isn't worth doing.  (I'm paraphrasing slightly.) :-)\n\nSo, rather than continuing with the back-n-forth, let me skip\nover the remaining questions in this thread and prepare a re-roll.\nHopefully, I can simplify and more clearly explain the method to\nmy madness and we can move on.\n\n\n>> Now, if you want to optimize over all trace2 events (a completely\n>> different topic), you could create a large scratch strbuf buffer in\n>> each thread context and use it so that we don't have to malloc/free\n>> during each trace message.  That might be worth while.\n> \n> *nod*\n\nI'll make a note to revisit this idea in a future series.\n\nThanks\nJeff\n\n"},{"id":"464735","messageId":"6e7e4f3187e2fbbbb54bb1cf5793bf6e981a5a94.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:24Z","receivedAt":"2022-10-12T18:52:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nUse \"size_t\" rather than \"int\" for the \"alloc\" and \"nr_open_regions\"\nfields in the \"tr2tls_thread_ctx\".  These are used by ALLOC_GROW().\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex b1e327a928e..a90bd639d48 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -11,8 +11,8 @@\n struct tr2tls_thread_ctx {\n \tstruct strbuf thread_name;\n \tuint64_t *array_us_start;\n-\tint alloc;\n-\tint nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n+\tsize_t alloc;\n+\tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n };\n \n-- \ngitgitgadget\n\n"},{"id":"464736","messageId":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.git.1664900407.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Trace2 timers and counters and some cleanup","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:23Z","receivedAt":"2022-10-12T18:52:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 2 of this series to add timers and counters to Trace2.\n\nChanges since V1:\n\n * I dropped the commits concerning compiler errors in Clang 11.0.0 on\n   MacOS. I've sent them to the mailing list in a separate series, since\n   they had nothing to do with the main topic of this series.\n\n * I moved the documentation changes earlier in the series to get it out of\n   the way (and eliminate the need to update it later commits).\n\n * After a long conversation on the mailing list, I redid the two\n   thread-name commits to simplify and hopefully eliminate the remaining\n   misunderstandings and/or short-comings of my previous attempt and\n   explanations. We now use a \"const char *\" for the field in the thread-ctx\n   that we format and detach from a strbuf during thread-start. The goal\n   here is to move away from a modifyable strbuf in the thread-ctx itself\n   (to avoid giving the appearance that a caller could modify the\n   thread-name at some point, when that was not intended).\n\nThe last 2 commits add the stopwatch timers and the global counters and are\nunchanged from the previous version.\n\nJeff Hostetler (7):\n  trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n  tr2tls: clarify TLS terminology\n  api-trace2.txt: elminate section describing the public trace2 API\n  trace2: rename the thread_name argument to trace2_thread_start\n  trace2: convert ctx.thread_name from strbuf to pointer\n  trace2: add stopwatch timers\n  trace2: add global counter mechanism\n\n Documentation/technical/api-trace2.txt | 190 +++++++++++++++++--------\n Makefile                               |   2 +\n t/helper/test-trace2.c                 | 187 ++++++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  95 +++++++++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               | 121 +++++++++++++++-\n trace2.h                               | 101 +++++++++++--\n trace2/tr2_ctr.c                       | 101 +++++++++++++\n trace2/tr2_ctr.h                       | 104 ++++++++++++++\n trace2/tr2_tgt.h                       |  14 ++\n trace2/tr2_tgt_event.c                 |  47 +++++-\n trace2/tr2_tgt_normal.c                |  39 +++++\n trace2/tr2_tgt_perf.c                  |  43 +++++-\n trace2/tr2_tls.c                       |  34 +++--\n trace2/tr2_tls.h                       |  55 ++++---\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 ++++++++++++++++++\n 17 files changed, 1359 insertions(+), 102 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\n\nbase-commit: 3dcec76d9df911ed8321007b1d197c1a206dc164\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1373%2Fjeffhostetler%2Ftrace2-stopwatch-v4-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1373/jeffhostetler/trace2-stopwatch-v4-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1373\n\nRange-diff vs v1:\n\n  1:  870f29166ea <  -:  ----------- builtin/merge-file: fix compiler warning on MacOS with clang 11.0.0\n  2:  43c41f7035d <  -:  ----------- builtin/unpack-objects.c: fix compiler warning on MacOS with clang 11.0.0\n  3:  73704b6f660 =  1:  6e7e4f3187e trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n  4:  7123886a804 =  2:  9dee7a75903 tr2tls: clarify TLS terminology\n  7:  77a4daf9a4b !  3:  804dab9e1a7 api-trace2.txt: elminate section describing the public trace2 API\n     @@ Documentation/technical/api-trace2.txt: take a `va_list` argument.\n      -\n      -These messages are concerned with Git thread usage.\n      -\n     --e.g: `void trace2_thread_start(const char *name_hint)`.\n     +-e.g: `void trace2_thread_start(const char *thread_name)`.\n      -\n      -=== Region and Data Messages\n      -\n  5:  82f1672e180 !  4:  637b422b860 trace2: rename trace2 thread_name argument as name_hint\n     @@ Metadata\n      Author: Jeff Hostetler <jeffhost@microsoft.com>\n      \n       ## Commit message ##\n     -    trace2: rename trace2 thread_name argument as name_hint\n     +    trace2: rename the thread_name argument to trace2_thread_start\n      \n     -    Rename the `thread_name` argument in `tr2tls_create_self()`\n     -    and `trace2_thread_start()` to be `name_hint` to make it clear\n     -    that the passed argument is a hint that will be used to create\n     +    Rename the `thread_name` argument in `tr2tls_create_self()` and\n     +    `trace2_thread_start()` to be `thread_base_name` to make it clearer\n     +    that the passed argument is a component used in the construction of\n          the actual `struct tr2tls_thread_ctx.thread_name` variable.\n      \n     -    This should make it clearer in the API that the trace2 layer\n     -    does not borrow the caller's string pointer/buffer, but rather\n     -    that it will use that hint in formatting the actual thread's\n     -    name.  Previous discussion on the mailing list indicated that\n     -    there was confusion about this point.\n     +    The base name will be used along with the thread id to create a\n     +    unique thread name.\n      \n          This commit does not change how the `thread_name` field is\n          allocated or stored within the `tr2tls_thread_ctx` structure.\n      \n          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n      \n     - ## Documentation/technical/api-trace2.txt ##\n     -@@ Documentation/technical/api-trace2.txt: e.g: `void trace2_child_start(struct child_process *cmd)`.\n     - \n     - These messages are concerned with Git thread usage.\n     - \n     --e.g: `void trace2_thread_start(const char *thread_name)`.\n     -+e.g: `void trace2_thread_start(const char *name_hint)`.\n     - \n     - === Region and Data Messages\n     - \n     -\n       ## trace2.c ##\n      @@ trace2.c: void trace2_exec_result_fl(const char *file, int line, int exec_id, int code)\n       \t\t\t\tfile, line, us_elapsed_absolute, exec_id, code);\n       }\n       \n      -void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n     -+void trace2_thread_start_fl(const char *file, int line, const char *name_hint)\n     ++void trace2_thread_start_fl(const char *file, int line, const char *thread_base_name)\n       {\n       \tstruct tr2_tgt *tgt_j;\n       \tint j;\n     @@ trace2.c: void trace2_thread_start_fl(const char *file, int line, const char *th\n       \t\ttrace2_region_enter_printf_fl(file, line, NULL, NULL, NULL,\n       \t\t\t\t\t      \"thread-proc on main: %s\",\n      -\t\t\t\t\t      thread_name);\n     -+\t\t\t\t\t      name_hint);\n     ++\t\t\t\t\t      thread_base_name);\n       \t\treturn;\n       \t}\n       \n     @@ trace2.c: void trace2_thread_start_fl(const char *file, int line, const char *th\n       \tus_elapsed_absolute = tr2tls_absolute_elapsed(us_now);\n       \n      -\ttr2tls_create_self(thread_name, us_now);\n     -+\ttr2tls_create_self(name_hint, us_now);\n     ++\ttr2tls_create_self(thread_base_name, us_now);\n       \n       \tfor_each_wanted_builtin (j, tgt_j)\n       \t\tif (tgt_j->pfn_thread_start_fl)\n     @@ trace2.h: void trace2_exec_result_fl(const char *file, int line, int exec_id, in\n        *\n      - * Thread names should be descriptive, like \"preload_index\".\n      - * Thread names will be decorated with an instance number automatically.\n     -+ * The thread name hint should be descriptive, like \"preload_index\" or\n     ++ * The thread base name should be descriptive, like \"preload_index\" or\n      + * taken from the thread-proc function.  A unique thread name will be\n     -+ * created from the hint and the thread id automatically.\n     ++ * created from the given base name and the thread id automatically.\n        */\n       void trace2_thread_start_fl(const char *file, int line,\n      -\t\t\t    const char *thread_name);\n     -+\t\t\t    const char *name_hint);\n     ++\t\t\t    const char *thread_base_name);\n       \n      -#define trace2_thread_start(thread_name) \\\n      -\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n     -+#define trace2_thread_start(name_hint) \\\n     -+\ttrace2_thread_start_fl(__FILE__, __LINE__, (name_hint))\n     ++#define trace2_thread_start(thread_base_name) \\\n     ++\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_base_name))\n       \n       /*\n        * Emit a 'thread_exit' event.  This must be called from inside the\n     @@ trace2/tr2_tls.c: void tr2tls_start_process_clock(void)\n       }\n       \n      -struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n     -+struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n     ++struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n       \t\t\t\t\t     uint64_t us_thread_start)\n       {\n       \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n     @@ trace2/tr2_tls.c: struct tr2tls_thread_ctx *tr2tls_create_self(const char *threa\n       \tif (ctx->thread_id)\n       \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n      -\tstrbuf_addstr(&ctx->thread_name, thread_name);\n     -+\tstrbuf_addstr(&ctx->thread_name, name_hint);\n     ++\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n       \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n       \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n       \n     @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n      + * The first thread in the process will have:\n      + *     { .thread_id=0, .thread_name=\"main\" }\n      + * Subsequent threads are given a non-zero thread_id and a thread_name\n     -+ * constructed from the id and a \"name hint\" (which is usually based\n     -+ * upon the name of the thread-proc function).  For example:\n     ++ * constructed from the id and a thread base name (which is usually just\n     ++ * the name of the thread-proc function).  For example:\n      + *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n      + * This helps to identify and distinguish messages from concurrent threads.\n      + * The ctx.thread_name field is truncated if necessary to help with column\n     @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n        * current thread.\n        */\n      -struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n     -+struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n     ++struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n       \t\t\t\t\t     uint64_t us_thread_start);\n       \n       /*\n  6:  6492b6d2b98 !  5:  4bf78e356e2 trace2: convert ctx.thread_name to flex array\n     @@ Metadata\n      Author: Jeff Hostetler <jeffhost@microsoft.com>\n      \n       ## Commit message ##\n     -    trace2: convert ctx.thread_name to flex array\n     +    trace2: convert ctx.thread_name from strbuf to pointer\n      \n          Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n     -    to a \"flex array\" at the end of the context structure.\n     +    to a \"const char*\" pointer.\n      \n          The `thread_name` field is a constant string that is constructed when\n          the context is created.  Using a (non-const) `strbuf` structure for it\n          caused some confusion in the past because it implied that someone\n          could rename a thread after it was created.  That usage was not\n     -    intended.  Changing it to a \"flex array\" will hopefully make the\n     -    intent more clear.\n     -\n     -    Also, move the maximum thread_name truncation to tr2_tgt_perf.c\n     -    because it is the only target that needs to worry about output column\n     -    alignment.\n     +    intended.  Change it to a const pointer to make the intent more clear.\n      \n          Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n      \n     @@ trace2/tr2_tgt_event.c: static void event_fmt_prepare(const char *event_name, co\n       \t * In brief mode, only emit <time> on these 2 event types.\n      \n       ## trace2/tr2_tgt_perf.c ##\n     -@@ trace2/tr2_tgt_perf.c: static int tr2env_perf_be_brief;\n     - \n     - #define TR2FMT_PERF_FL_WIDTH (28)\n     - #define TR2FMT_PERF_MAX_EVENT_NAME (12)\n     -+#define TR2FMT_PERF_MAX_THREAD_NAME (24)\n     - #define TR2FMT_PERF_REPO_WIDTH (3)\n     - #define TR2FMT_PERF_CATEGORY_WIDTH (12)\n     - \n      @@ trace2/tr2_tgt_perf.c: static void perf_fmt_prepare(const char *event_name,\n     - \t}\n       \n       \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n     --\tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n     + \tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n      -\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n     -+\tstrbuf_addf(buf, \"%-*.*s | %-*s | \",\n     -+\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n     -+\t\t    TR2FMT_PERF_MAX_THREAD_NAME,\n     -+\t\t    ctx->thread_name,\n     -+\t\t    TR2FMT_PERF_MAX_EVENT_NAME,\n     ++\t\t    ctx->thread_name, TR2FMT_PERF_MAX_EVENT_NAME,\n       \t\t    event_name);\n       \n       \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\n      \n       ## trace2/tr2_tls.c ##\n     -@@ trace2/tr2_tls.c: void tr2tls_start_process_clock(void)\n     - struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n     +@@ trace2/tr2_tls.c: struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n       \t\t\t\t\t     uint64_t us_thread_start)\n       {\n     --\tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n     -+\tstruct tr2tls_thread_ctx *ctx;\n     -+\tstruct strbuf buf_name = STRBUF_INIT;\n     -+\tint thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n     -+\n     -+\tif (thread_id)\n     -+\t\tstrbuf_addf(&buf_name, \"th%02d:\", thread_id);\n     -+\tstrbuf_addstr(&buf_name, name_hint);\n     -+\n     -+\tFLEX_ALLOC_MEM(ctx, thread_name, buf_name.buf, buf_name.len);\n     -+\tstrbuf_release(&buf_name);\n     -+\n     -+\tctx->thread_id = thread_id;\n     + \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n     ++\tstruct strbuf buf = STRBUF_INIT;\n       \n       \t/*\n       \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n     -@@ trace2/tr2_tls.c: struct tr2tls_thread_ctx *tr2tls_create_self(const char *name_hint,\n     - \tctx->array_us_start = (uint64_t *)xcalloc(ctx->alloc, sizeof(uint64_t));\n     - \tctx->array_us_start[ctx->nr_open_regions++] = us_thread_start;\n     +@@ trace2/tr2_tls.c: struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n     + \n     + \tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n       \n     --\tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n     --\n      -\tstrbuf_init(&ctx->thread_name, 0);\n     --\tif (ctx->thread_id)\n     ++\tstrbuf_init(&buf, 0);\n     + \tif (ctx->thread_id)\n      -\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n     --\tstrbuf_addstr(&ctx->thread_name, name_hint);\n     +-\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n      -\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n      -\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n     --\n     ++\t\tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n     ++\tstrbuf_addstr(&buf, thread_base_name);\n     ++\tif (buf.len > TR2_MAX_THREAD_NAME)\n     ++\t\tstrbuf_setlen(&buf, TR2_MAX_THREAD_NAME);\n     ++\tctx->thread_name = strbuf_detach(&buf, NULL);\n     + \n       \tpthread_setspecific(tr2tls_key, ctx);\n       \n     - \treturn ctx;\n      @@ trace2/tr2_tls.c: void tr2tls_unset_self(void)\n       \n       \tpthread_setspecific(tr2tls_key, NULL);\n       \n      -\tstrbuf_release(&ctx->thread_name);\n     ++\tfree((char *)ctx->thread_name);\n       \tfree(ctx->array_us_start);\n       \tfree(ctx);\n       }\n     @@ trace2/tr2_tls.c: void tr2tls_pop_self(void)\n      \n       ## trace2/tr2_tls.h ##\n      @@\n     -  * There is NO relation to \"transport layer security\".\n     -  */\n     + #define TR2_MAX_THREAD_NAME (24)\n       \n     --/*\n     -- * Arbitry limit for thread names for column alignment.\n     -- */\n     --#define TR2_MAX_THREAD_NAME (24)\n     --\n       struct tr2tls_thread_ctx {\n      -\tstruct strbuf thread_name;\n     ++\tconst char *thread_name;\n       \tuint64_t *array_us_start;\n       \tsize_t alloc;\n       \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n     - \tint thread_id;\n     -+\tchar thread_name[FLEX_ARRAY];\n     - };\n     - \n     - /*\n     -@@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n     -  * upon the name of the thread-proc function).  For example:\n     -  *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n     -  * This helps to identify and distinguish messages from concurrent threads.\n     -- * The ctx.thread_name field is truncated if necessary to help with column\n     -- * alignment in printf-style messages.\n     -  *\n     -  * In this and all following functions the term \"self\" refers to the\n     -  * current thread.\n  8:  19c7bba91ba !  6:  dd6d8e2841b trace2: add stopwatch timers\n     @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n      +\tstruct tr2_timer_block timer_block;\n      +\tunsigned int used_any_timer:1;\n      +\tunsigned int used_any_per_thread_timer:1;\n     - \tchar thread_name[FLEX_ARRAY];\n       };\n       \n     + /*\n      @@ trace2/tr2_tls.h: int tr2tls_locked_increment(int *p);\n        */\n       void tr2tls_start_process_clock(void);\n  9:  2bf7cb1f8d0 !  7:  cf012fcde37 trace2: add global counter mechanism\n     @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n       \tunsigned int used_any_per_thread_timer:1;\n      +\tunsigned int used_any_counter:1;\n      +\tunsigned int used_any_per_thread_counter:1;\n     - \tchar thread_name[FLEX_ARRAY];\n       };\n       \n     + /*\n\n-- \ngitgitgadget\n"},{"id":"464737","messageId":"9dee7a75903936f086d97580441c776978d70b43.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] tr2tls: clarify TLS terminology","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:25Z","receivedAt":"2022-10-12T18:52:42Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nReduce or eliminate use of the term \"TLS\" in the Trace2 code.\n\nThe term \"TLS\" has two popular meanings: \"thread-local storage\" and\n\"transport layer security\".  In the Trace2 source, the term is associated\nwith the former.  There was concern on the mailing list about it refering\nto the latter.\n\nUpdate the source and documentation to eliminate the use of the \"TLS\" term\nor replace it with the phrase \"thread-local storage\" to reduce ambiguity.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  8 ++++----\n trace2.c                               |  2 +-\n trace2.h                               | 10 +++++-----\n trace2/tr2_tls.c                       |  6 +++---\n trace2/tr2_tls.h                       | 18 +++++++++++-------\n 5 files changed, 24 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 2afa28bb5aa..431d424f9d5 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -685,8 +685,8 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_start\"`::\n \tThis event is generated when a thread is started.  It is\n-\tgenerated from *within* the new thread's thread-proc (for TLS\n-\treasons).\n+\tgenerated from *within* the new thread's thread-proc (because\n+\tit needs to access data in the thread's thread-local storage).\n +\n ------------\n {\n@@ -698,7 +698,7 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_exit\"`::\n \tThis event is generated when a thread exits.  It is generated\n-\tfrom *within* the thread's thread-proc (for TLS reasons).\n+\tfrom *within* the thread's thread-proc.\n +\n ------------\n {\n@@ -1206,7 +1206,7 @@ worked on 508 items at offset 2032.  Thread \"th04\" worked on 508 items\n at offset 508.\n +\n This example also shows that thread names are assigned in a racy manner\n-as each thread starts and allocates TLS storage.\n+as each thread starts.\n \n Config (def param) Events::\n \ndiff --git a/trace2.c b/trace2.c\nindex 0c0a11e07d5..c1244e45ace 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -52,7 +52,7 @@ static struct tr2_tgt *tr2_tgt_builtins[] =\n  * Force (rather than lazily) initialize any of the requested\n  * builtin TRACE2 targets at startup (and before we've seen an\n  * actual TRACE2 event call) so we can see if we need to setup\n- * the TR2 and TLS machinery.\n+ * private data structures and thread-local storage.\n  *\n  * Return the number of builtin targets enabled.\n  */\ndiff --git a/trace2.h b/trace2.h\nindex 88d906ea830..af3c11694cc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -73,8 +73,7 @@ void trace2_initialize_clock(void);\n /*\n  * Initialize TRACE2 tracing facility if any of the builtin TRACE2\n  * targets are enabled in the system config or the environment.\n- * This includes setting up the Trace2 thread local storage (TLS).\n- * Emits a 'version' message containing the version of git\n+ * This emits a 'version' message containing the version of git\n  * and the Trace2 protocol.\n  *\n  * This function should be called from `main()` as early as possible in\n@@ -302,7 +301,8 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n \n /*\n  * Emit a 'thread_start' event.  This must be called from inside the\n- * thread-proc to set up the trace2 TLS data for the thread.\n+ * thread-proc to allow the thread to create its own thread-local\n+ * storage.\n  *\n  * Thread names should be descriptive, like \"preload_index\".\n  * Thread names will be decorated with an instance number automatically.\n@@ -315,8 +315,8 @@ void trace2_thread_start_fl(const char *file, int line,\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\n- * thread-proc to report thread-specific data and cleanup TLS data\n- * for the thread.\n+ * thread-proc so that the thread can access and clean up its\n+ * thread-local storage.\n  */\n void trace2_thread_exit_fl(const char *file, int line);\n \ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 7da94aba522..8d2182fbdbb 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -69,9 +69,9 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void)\n \tctx = pthread_getspecific(tr2tls_key);\n \n \t/*\n-\t * If the thread-proc did not call trace2_thread_start(), we won't\n-\t * have any TLS data associated with the current thread.  Fix it\n-\t * here and silently continue.\n+\t * If the current thread's thread-proc did not call\n+\t * trace2_thread_start(), then the thread will not have any\n+\t * thread-local storage.  Create it now and silently continue.\n \t */\n \tif (!ctx)\n \t\tctx = tr2tls_create_self(\"unknown\", getnanotime() / 1000);\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex a90bd639d48..1297509fd23 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -3,6 +3,12 @@\n \n #include \"strbuf.h\"\n \n+/*\n+ * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n+ * Trace2 source files.  This usage is borrowed from GCC and Windows.\n+ * There is NO relation to \"transport layer security\".\n+ */\n+\n /*\n  * Arbitry limit for thread names for column alignment.\n  */\n@@ -17,9 +23,7 @@ struct tr2tls_thread_ctx {\n };\n \n /*\n- * Create TLS data for the current thread.  This gives us a place to\n- * put per-thread data, such as thread start time, function nesting\n- * and a per-thread label for our messages.\n+ * Create thread-local storage for the current thread.\n  *\n  * We assume the first thread is \"main\".  Other threads are given\n  * non-zero thread-ids to help distinguish messages from concurrent\n@@ -35,7 +39,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n- * Get our TLS data.\n+ * Get the thread-local storage pointer of the current thread.\n  */\n struct tr2tls_thread_ctx *tr2tls_get_self(void);\n \n@@ -45,7 +49,7 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void);\n int tr2tls_is_main_thread(void);\n \n /*\n- * Free our TLS data.\n+ * Free the current thread's thread-local storage.\n  */\n void tr2tls_unset_self(void);\n \n@@ -81,12 +85,12 @@ uint64_t tr2tls_region_elasped_self(uint64_t us);\n uint64_t tr2tls_absolute_elapsed(uint64_t us);\n \n /*\n- * Initialize the tr2 TLS system.\n+ * Initialize thread-local storage for Trace2.\n  */\n void tr2tls_init(void);\n \n /*\n- * Free all tr2 TLS resources.\n+ * Free all Trace2 thread-local storage resources.\n  */\n void tr2tls_release(void);\n \n-- \ngitgitgadget\n\n"},{"id":"464738","messageId":"804dab9e1a7fa1cea9355bac92ada16332f1194e.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 3/7] api-trace2.txt: elminate section describing the public trace2 API","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:26Z","receivedAt":"2022-10-12T18:52:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nEliminate the mostly obsolete `Public API` sub-section from the\n`Trace2 API` section in the documentation.  Strengthen the referral\nto `trace2.h`.\n\nMost of the technical information in this sub-section was moved to\n`trace2.h` in 6c51cb525d (trace2: move doc to trace2.h, 2019-11-17) to\nbe adjacent to the function prototypes.  The remaining text wasn't\nthat useful by itself.\n\nFurthermore, the text would need a bit of overhaul to add routines\nthat do not immediately generate a message, such as stopwatch timers.\nSo it seemed simpler to just get rid of it.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt | 61 +++-----------------------\n 1 file changed, 7 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 431d424f9d5..9d43909d068 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -148,20 +148,18 @@ filename collisions).\n \n == Trace2 API\n \n-All public Trace2 functions and macros are defined in `trace2.h` and\n-`trace2.c`.  All public symbols are prefixed with `trace2_`.\n+The Trace2 public API is defined and documented in `trace2.h`; refer to it for\n+more information.  All public functions and macros are prefixed\n+with `trace2_` and are implemented in `trace2.c`.\n \n There are no public Trace2 data structures.\n \n The Trace2 code also defines a set of private functions and data types\n in the `trace2/` directory.  These symbols are prefixed with `tr2_`\n-and should only be used by functions in `trace2.c`.\n+and should only be used by functions in `trace2.c` (or other private\n+source files in `trace2/`).\n \n-== Conventions for Public Functions and Macros\n-\n-The functions defined by the Trace2 API are declared and documented\n-in `trace2.h`.  It defines the API functions and wrapper macros for\n-Trace2.\n+=== Conventions for Public Functions and Macros\n \n Some functions have a `_fl()` suffix to indicate that they take `file`\n and `line-number` arguments.\n@@ -172,52 +170,7 @@ take a `va_list` argument.\n Some functions have a `_printf_fl()` suffix to indicate that they also\n take a `printf()` style format with a variable number of arguments.\n \n-There are CPP wrapper macros and `#ifdef`s to hide most of these details.\n-See `trace2.h` for more details.  The following discussion will only\n-describe the simplified forms.\n-\n-== Public API\n-\n-All Trace2 API functions send a message to all of the active\n-Trace2 Targets.  This section describes the set of available\n-messages.\n-\n-It helps to divide these functions into groups for discussion\n-purposes.\n-\n-=== Basic Command Messages\n-\n-These are concerned with the lifetime of the overall git process.\n-e.g: `void trace2_initialize_clock()`, `void trace2_initialize()`,\n-`int trace2_is_enabled()`, `void trace2_cmd_start(int argc, const char **argv)`.\n-\n-=== Command Detail Messages\n-\n-These are concerned with describing the specific Git command\n-after the command line, config, and environment are inspected.\n-e.g: `void trace2_cmd_name(const char *name)`,\n-`void trace2_cmd_mode(const char *mode)`.\n-\n-=== Child Process Messages\n-\n-These are concerned with the various spawned child processes,\n-including shell scripts, git commands, editors, pagers, and hooks.\n-\n-e.g: `void trace2_child_start(struct child_process *cmd)`.\n-\n-=== Git Thread Messages\n-\n-These messages are concerned with Git thread usage.\n-\n-e.g: `void trace2_thread_start(const char *thread_name)`.\n-\n-=== Region and Data Messages\n-\n-These are concerned with recording performance data\n-over regions or spans of code. e.g:\n-`void trace2_region_enter(const char *category, const char *label, const struct repository *repo)`.\n-\n-Refer to trace2.h for details about all trace2 functions.\n+CPP wrapper macros are defined to hide most of these details.\n \n == Trace2 Target Formats\n \n-- \ngitgitgadget\n\n"},{"id":"464739","messageId":"637b422b8606b3b6d954e6a1959aae450507cdfa.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:27Z","receivedAt":"2022-10-12T18:52:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nRename the `thread_name` argument in `tr2tls_create_self()` and\n`trace2_thread_start()` to be `thread_base_name` to make it clearer\nthat the passed argument is a component used in the construction of\nthe actual `struct tr2tls_thread_ctx.thread_name` variable.\n\nThe base name will be used along with the thread id to create a\nunique thread name.\n\nThis commit does not change how the `thread_name` field is\nallocated or stored within the `tr2tls_thread_ctx` structure.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2.c         |  6 +++---\n trace2.h         | 11 ++++++-----\n trace2/tr2_tls.c |  4 ++--\n trace2/tr2_tls.h | 17 ++++++++++-------\n 4 files changed, 21 insertions(+), 17 deletions(-)\n\ndiff --git a/trace2.c b/trace2.c\nindex c1244e45ace..165264dc79a 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -466,7 +466,7 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code)\n \t\t\t\tfile, line, us_elapsed_absolute, exec_id, code);\n }\n \n-void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n+void trace2_thread_start_fl(const char *file, int line, const char *thread_base_name)\n {\n \tstruct tr2_tgt *tgt_j;\n \tint j;\n@@ -488,14 +488,14 @@ void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n \t\t */\n \t\ttrace2_region_enter_printf_fl(file, line, NULL, NULL, NULL,\n \t\t\t\t\t      \"thread-proc on main: %s\",\n-\t\t\t\t\t      thread_name);\n+\t\t\t\t\t      thread_base_name);\n \t\treturn;\n \t}\n \n \tus_now = getnanotime() / 1000;\n \tus_elapsed_absolute = tr2tls_absolute_elapsed(us_now);\n \n-\ttr2tls_create_self(thread_name, us_now);\n+\ttr2tls_create_self(thread_base_name, us_now);\n \n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_start_fl)\ndiff --git a/trace2.h b/trace2.h\nindex af3c11694cc..74cdb1354f7 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -304,14 +304,15 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n  * thread-proc to allow the thread to create its own thread-local\n  * storage.\n  *\n- * Thread names should be descriptive, like \"preload_index\".\n- * Thread names will be decorated with an instance number automatically.\n+ * The thread base name should be descriptive, like \"preload_index\" or\n+ * taken from the thread-proc function.  A unique thread name will be\n+ * created from the given base name and the thread id automatically.\n  */\n void trace2_thread_start_fl(const char *file, int line,\n-\t\t\t    const char *thread_name);\n+\t\t\t    const char *thread_base_name);\n \n-#define trace2_thread_start(thread_name) \\\n-\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n+#define trace2_thread_start(thread_base_name) \\\n+\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_base_name))\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 8d2182fbdbb..4f7c516ecb6 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -31,7 +31,7 @@ void tr2tls_start_process_clock(void)\n \ttr2tls_us_start_process = getnanotime() / 1000;\n }\n \n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n@@ -50,7 +50,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \tstrbuf_init(&ctx->thread_name, 0);\n \tif (ctx->thread_id)\n \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_name);\n+\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n \ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 1297509fd23..7d1f03a2ea6 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -25,17 +25,20 @@ struct tr2tls_thread_ctx {\n /*\n  * Create thread-local storage for the current thread.\n  *\n- * We assume the first thread is \"main\".  Other threads are given\n- * non-zero thread-ids to help distinguish messages from concurrent\n- * threads.\n- *\n- * Truncate the thread name if necessary to help with column alignment\n- * in printf-style messages.\n+ * The first thread in the process will have:\n+ *     { .thread_id=0, .thread_name=\"main\" }\n+ * Subsequent threads are given a non-zero thread_id and a thread_name\n+ * constructed from the id and a thread base name (which is usually just\n+ * the name of the thread-proc function).  For example:\n+ *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n+ * This helps to identify and distinguish messages from concurrent threads.\n+ * The ctx.thread_name field is truncated if necessary to help with column\n+ * alignment in printf-style messages.\n  *\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n  */\n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"464740","messageId":"4bf78e356e23c947b8328a91ba435a357cd51f43.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] trace2: convert ctx.thread_name from strbuf to pointer","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:28Z","receivedAt":"2022-10-12T18:52:47Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nConvert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\nto a \"const char*\" pointer.\n\nThe `thread_name` field is a constant string that is constructed when\nthe context is created.  Using a (non-const) `strbuf` structure for it\ncaused some confusion in the past because it implied that someone\ncould rename a thread after it was created.  That usage was not\nintended.  Change it to a const pointer to make the intent more clear.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tgt_event.c |  2 +-\n trace2/tr2_tgt_perf.c  |  2 +-\n trace2/tr2_tls.c       | 16 +++++++++-------\n trace2/tr2_tls.h       |  2 +-\n 4 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 37a3163be12..52f9356c695 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -90,7 +90,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n \n \tjw_object_string(jw, \"event\", event_name);\n \tjw_object_string(jw, \"sid\", tr2_sid_get());\n-\tjw_object_string(jw, \"thread\", ctx->thread_name.buf);\n+\tjw_object_string(jw, \"thread\", ctx->thread_name);\n \n \t/*\n \t * In brief mode, only emit <time> on these 2 event types.\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 8cb792488c8..59ca58f862d 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -108,7 +108,7 @@ static void perf_fmt_prepare(const char *event_name,\n \n \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n \tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n-\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n+\t\t    ctx->thread_name, TR2FMT_PERF_MAX_EVENT_NAME,\n \t\t    event_name);\n \n \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 4f7c516ecb6..3a67532aae4 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -35,6 +35,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n+\tstruct strbuf buf = STRBUF_INIT;\n \n \t/*\n \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n@@ -47,12 +48,13 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \n \tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n \n-\tstrbuf_init(&ctx->thread_name, 0);\n+\tstrbuf_init(&buf, 0);\n \tif (ctx->thread_id)\n-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n+\t\tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n+\tstrbuf_addstr(&buf, thread_base_name);\n+\tif (buf.len > TR2_MAX_THREAD_NAME)\n+\t\tstrbuf_setlen(&buf, TR2_MAX_THREAD_NAME);\n+\tctx->thread_name = strbuf_detach(&buf, NULL);\n \n \tpthread_setspecific(tr2tls_key, ctx);\n \n@@ -95,7 +97,7 @@ void tr2tls_unset_self(void)\n \n \tpthread_setspecific(tr2tls_key, NULL);\n \n-\tstrbuf_release(&ctx->thread_name);\n+\tfree((char *)ctx->thread_name);\n \tfree(ctx->array_us_start);\n \tfree(ctx);\n }\n@@ -113,7 +115,7 @@ void tr2tls_pop_self(void)\n \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n \n \tif (!ctx->nr_open_regions)\n-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n+\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n \n \tctx->nr_open_regions--;\n }\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 7d1f03a2ea6..e17cc462f87 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -15,7 +15,7 @@\n #define TR2_MAX_THREAD_NAME (24)\n \n struct tr2tls_thread_ctx {\n-\tstruct strbuf thread_name;\n+\tconst char *thread_name;\n \tuint64_t *array_us_start;\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n-- \ngitgitgadget\n\n"},{"id":"464741","messageId":"cf012fcde37e3f101b46e424331445326f5bf85b.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] trace2: add global counter mechanism","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:30Z","receivedAt":"2022-10-12T18:52:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd global counters mechanism to Trace2.\n\nThe Trace2 counters mechanism adds the ability to create a set of\nglobal counter variables and an API to increment them efficiently.\nCounters can optionally report per-thread usage in addition to the sum\nacross all threads.\n\nCounter events are emitted to the Trace2 logs when a thread exits and\nat process exit.\n\nCounters are an alternative to `data` and `data_json` events.\n\nCounters are useful when you want to measure something across the life\nof the process, when you don't want per-measurement events for\nperformance reasons, when the data does not fit conveniently within a\nregion, or when your control flow does not easily let you write the\nfinal total.  For example, you might use this to report the number of\ncalls to unzip() or the number of de-delta steps during a checkout.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  31 ++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  89 +++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  46 +++++++++++\n trace2.c                               |  52 +++++++++++--\n trace2.h                               |  37 +++++++++\n trace2/tr2_ctr.c                       | 101 ++++++++++++++++++++++++\n trace2/tr2_ctr.h                       | 104 +++++++++++++++++++++++++\n trace2/tr2_tgt.h                       |   7 ++\n trace2/tr2_tgt_event.c                 |  19 +++++\n trace2/tr2_tgt_normal.c                |  16 ++++\n trace2/tr2_tgt_perf.c                  |  17 ++++\n trace2/tr2_tls.h                       |   4 +\n 13 files changed, 517 insertions(+), 7 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 75ce6f45603..de5fc250595 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -805,6 +805,37 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_counter\"`::\n+\tThis event logs the value of a counter variable in a thread.\n+\tThis event is generated when a thread exits for counters that\n+\trequested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+`\"counter\"`::\n+\tThis event logs the value of a counter variable across all threads.\n+\tThis event is generated when the process exits.  The total value\n+\treported here is the sum across all threads.\n++\n+------------\n+{\n+\t\"event\":\"counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\ndiff --git a/Makefile b/Makefile\nindex 820649bf62a..29ab417ca3a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += trace.o\n LIB_OBJS += trace2.o\n LIB_OBJS += trace2/tr2_cfg.o\n LIB_OBJS += trace2/tr2_cmd_name.o\n+LIB_OBJS += trace2/tr2_ctr.o\n LIB_OBJS += trace2/tr2_dst.o\n LIB_OBJS += trace2/tr2_sid.o\n LIB_OBJS += trace2/tr2_sysenv.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex f951b9e97d7..1b092c60714 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -323,6 +323,92 @@ static int ut_101timer(int argc, const char **argv)\n \treturn 0;\n }\n \n+/*\n+ * Single-threaded counter test.  Add several values to the TEST1 counter.\n+ * The test script can verify that the final sum is reported in the \"counter\"\n+ * event.\n+ */\n+static int ut_200counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> [<v2> [...]]\";\n+\tint value;\n+\tint k;\n+\n+\tif (argc < 1)\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < argc; k++) {\n+\t\tif (get_i(&value, argv[k]))\n+\t\t\tdie(\"invalid value[%s] -- %s\",\n+\t\t\t    argv[k], usage_error);\n+\t\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST1, value);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Multi-threaded counter test.  Create seveal threads that each increment\n+ * the TEST2 global counter.  The test script can verify that an individual\n+ * \"th_counter\" event is generated with a partial sum for each thread and\n+ * that a final aggregate \"counter\" event is generated.\n+ */\n+\n+struct ut_201_data {\n+\tint v1;\n+\tint v2;\n+};\n+\n+static void *ut_201counter_thread_proc(void *_ut_201_data)\n+{\n+\tstruct ut_201_data *data = _ut_201_data;\n+\n+\ttrace2_thread_start(\"ut_201\");\n+\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v1);\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v2);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int ut_201counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> <v2> <threads>\";\n+\n+\tstruct ut_201_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v1, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v2, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_201counter_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -346,6 +432,9 @@ static struct unit_test ut_table[] = {\n \n \t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n \t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n+\n+\t{ ut_200counter,  \"200counter\", \"<v1> [<v2> [<v3> [...]]]\" },\n+\t{ ut_201counter,  \"201counter\", \"<v1> <v2> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 5c28424e657..0b3436e8cac 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -222,4 +222,50 @@ test_expect_success 'stopwatch timer test/test2' '\n \thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n '\n \n+# Exercise the global counters and confirm that we get the expected values.\n+#\n+# The counter \"test/test1\" should only emit a global summary \"counter\" event.\n+# The counter \"test/test2\" could emit per-thread \"th_counter\" events and a\n+# global summary \"counter\" event.\n+\n+have_counter_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 value=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} value:${value}\" &&\n+\n+\tgrep \"${patern}\" ${file}\n+}\n+\n+test_expect_success 'global counter test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the counter \"test1\" and add n integers.\n+\ttest-tool trace2 200counter 1 2 3 4 5 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test1\" 15 actual\n+'\n+\n+test_expect_success 'global counter test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Add 2 integers to the counter \"test2\" in each of 3 threads.\n+\ttest-tool trace2 201counter 7 13 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_counter_event \"th01:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th02:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th03:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\n+\t# And we should have a single event with the total across all threads.\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test2\" 60 actual\n+'\n+\n test_done\ndiff --git a/trace2.c b/trace2.c\nindex a93cab7c2b7..279bddf53b4 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -8,6 +8,7 @@\n #include \"version.h\"\n #include \"trace2/tr2_cfg.h\"\n #include \"trace2/tr2_cmd_name.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_dst.h\"\n #include \"trace2/tr2_sid.h\"\n #include \"trace2/tr2_sysenv.h\"\n@@ -101,6 +102,22 @@ static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n \t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n }\n \n+/*\n+ * The signature of this function must match the pfn_counter\n+ * method in the targets.\n+ */\n+static void tr2_tgt_emit_a_counter(const struct tr2_counter_metadata *meta,\n+\t\t\t\t   const struct tr2_counter *counter,\n+\t\t\t\t   int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_counter)\n+\t\t\ttgt_j->pfn_counter(meta, counter, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -132,20 +149,26 @@ static void tr2main_atexit_handler(void)\n \t * Some timers want per-thread details.  If the main thread\n \t * used one of those timers, emit the details now (before\n \t * we emit the aggregate timer values).\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data for the main thread to the final\n-\t * totals.  And then emit the final timer values.\n+\t * Add stopwatch timer and counter data for the main thread to\n+\t * the final totals.  And then emit the final values.\n \t *\n \t * Technically, we shouldn't need to hold the lock to update\n-\t * and output the final_timer_block (since all other threads\n-\t * should be dead by now), but it doesn't hurt anything.\n+\t * and output the final_timer_block and final_counter_block\n+\t * (since all other threads should be dead by now), but it\n+\t * doesn't hurt anything.\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_final_counters(tr2_tgt_emit_a_counter);\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -582,16 +605,20 @@ void trace2_thread_exit_fl(const char *file, int line)\n \t/*\n \t * Some timers want per-thread details.  If this thread used\n \t * one of those timers, emit the details now.\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data from the current (non-main) thread\n-\t * to the final totals.  (We'll accumulate data for the main\n-\t * thread later during \"atexit\".)\n+\t * Add stopwatch timer and counter data from the current\n+\t * (non-main) thread to the final totals.  (We'll accumulate\n+\t * data for the main thread later during \"atexit\".)\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -870,6 +897,17 @@ void trace2_timer_stop(enum trace2_timer_id tid)\n \ttr2_stop_timer(tid);\n }\n \n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (cid < 0 || cid >= TRACE2_NUMBER_OF_COUNTERS)\n+\t\tBUG(\"trace2_counter_add: invalid counter id: %d\", cid);\n+\n+\ttr2_counter_increment(cid, value);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 7a843ac0518..4ced30c0db3 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -52,6 +52,7 @@ struct json_writer;\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n  * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n+ * [] trace2_counter*   -- global counters (messages are deferred).\n  */\n \n /*\n@@ -528,6 +529,42 @@ enum trace2_timer_id {\n void trace2_timer_start(enum trace2_timer_id tid);\n void trace2_timer_stop(enum trace2_timer_id tid);\n \n+/*\n+ * Define the set of global counters.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum be also be added to the\n+ * `tr2_counter_metadata[]` in `trace2/tr2_tr2_ctr.c`.\n+ */\n+enum trace2_counter_id {\n+\t/*\n+\t * Define two counters for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional counter definitions before here. */\n+\tTRACE2_NUMBER_OF_COUNTERS\n+};\n+\n+/*\n+ * Increase the named global counter by value.\n+ *\n+ * Note that this adds `value` to the current thread's partial sum for\n+ * this counter (without locking) and that the complete sum is not\n+ * available until all threads have exited, so it does not return the\n+ * new value of the counter.\n+ */\n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nnew file mode 100644\nindex 00000000000..483ca7c308f\n--- /dev/null\n+++ b/trace2/tr2_ctr.c\n@@ -0,0 +1,101 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_ctr.h\"\n+\n+/*\n+ * A global counter block to aggregrate values from the partial sums\n+ * from each thread.\n+ */\n+static struct tr2_counter_block final_counter_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each global counter.\n+ *\n+ * This array must match the \"enum trace2_counter_id\" and the values\n+ * in \"struct tr2_counter_block.counter[*]\".\n+ */\n+static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTERS] = {\n+\t[TRACE2_COUNTER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_COUNTER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\tc->value += value;\n+\n+\tctx->used_any_counter = 1;\n+\tif (tr2_counter_metadata[cid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_counter = 1;\n+}\n+\n+void tr2_update_final_counters(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++) {\n+\t\tstruct tr2_counter *c_final = &final_counter_block.counter[cid];\n+\t\tconst struct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\t\tc_final->value += c->value;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_per_thread_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * For each counter, if the counter wants per-thread events\n+\t * and this thread used it (the value is non-zero), emit it.\n+\t */\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (tr2_counter_metadata[cid].want_per_thread_events &&\n+\t\t    ctx->counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &ctx->counter_block.counter[cid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tenum trace2_counter_id cid;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (final_counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &final_counter_block.counter[cid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_ctr.h b/trace2/tr2_ctr.h\nnew file mode 100644\nindex 00000000000..a2267ee9901\n--- /dev/null\n+++ b/trace2/tr2_ctr.h\n@@ -0,0 +1,104 @@\n+#ifndef TR2_CTR_H\n+#define TR2_CTR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow global \"counters\".\n+ *\n+ * Counters can be used count interesting activity that does not fit\n+ * the \"region and data\" model, such as code called from many\n+ * different regions and/or where you want to count a number of items,\n+ * but don't have control of when the last item will be processed,\n+ * such as counter the number of calls to `lstat()`.\n+ *\n+ * Counters differ from Trace2 \"data\" events.  Data events are emitted\n+ * immediately and are appropriate for documenting loop counters at\n+ * the end of a region, for example.  Counter values are accumulated\n+ * during the program and final counter values are emitted at program\n+ * exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * counters and counter ids using a fixed size \"counter block\" array\n+ * in thread-local storage.  This gives us constant time, lock-free\n+ * access to each counter within each thread.  This lets us avoid the\n+ * complexities of dynamically allocating a counter and sharing that\n+ * definition with other threads.\n+ *\n+ * Each thread uses the counter block in its thread-local storage to\n+ * increment partial sums for each counter (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Partial sums for each counter are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each counter are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"counter metadata\" table contains the \"category\" and\n+ * \"name\" fields for each counter.  This eliminates the need to\n+ * include those args in the various counter APIs.\n+ */\n+\n+/*\n+ * The definition of an individual counter as used by an individual\n+ * thread (and later in aggregation).\n+ */\n+struct tr2_counter {\n+\tuint64_t value;\n+};\n+\n+/*\n+ * Metadata for a counter.\n+ */\n+struct tr2_counter_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this counter\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed block of counters to insert into thread-local\n+ * storage.  This wrapper is used to avoid quirks of C and the usual\n+ * need to pass an array size argument.\n+ */\n+struct tr2_counter_block {\n+\tstruct tr2_counter counter[TRACE2_NUMBER_OF_COUNTERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to increment a counter for the\n+ * current thread.\n+ */\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value);\n+\n+/*\n+ * Add the current thread's counter data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_counters(void);\n+\n+/*\n+ * Emit per-thread counter data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+/*\n+ * Emit global counter values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+#endif /* TR2_CTR_H */\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 2a80bef0df5..94a334d980a 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -6,6 +6,8 @@ struct repository;\n struct json_writer;\n struct tr2_timer_metadata;\n struct tr2_timer;\n+struct tr2_counter_metadata;\n+struct tr2_counter;\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -102,6 +104,10 @@ typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n \t\t\t\t  const struct tr2_timer *timer,\n \t\t\t\t  int is_final_data);\n \n+typedef void(tr2_tgt_evt_counter_t)(const struct tr2_counter_metadata *meta,\n+\t\t\t\t    const struct tr2_counter *counter,\n+\t\t\t\t    int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -139,6 +145,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n \ttr2_tgt_evt_timer_t                     *pfn_timer;\n+\ttr2_tgt_evt_counter_t                   *pfn_counter;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 1196da89ba4..bb0653e0e6f 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -642,6 +642,24 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tjw_release(&jw);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"count\", counter->value);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -674,4 +692,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 3888c10ef50..b21508e06f7 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -351,6 +351,21 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"%s %s/%s value:%\"PRIu64,\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    counter->value);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -383,4 +398,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 89b30ddc0e4..188068ac5d0 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -578,6 +578,22 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"name:%s value:%\"PRIu64,\n+\t\t    meta->name,\n+\t\t    counter->value);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -610,4 +626,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 2322b0d0ef0..289b62d0721 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_tmr.h\"\n \n /*\n@@ -22,8 +23,11 @@ struct tr2tls_thread_ctx {\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n \tstruct tr2_timer_block timer_block;\n+\tstruct tr2_counter_block counter_block;\n \tunsigned int used_any_timer:1;\n \tunsigned int used_any_per_thread_timer:1;\n+\tunsigned int used_any_counter:1;\n+\tunsigned int used_any_per_thread_counter:1;\n };\n \n /*\n-- \ngitgitgadget\n"},{"id":"464742","messageId":"dd6d8e2841b424ba89672b4d94306f9ec882a868.1665600750.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] trace2: add stopwatch timers","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-12T18:52:29Z","receivedAt":"2022-10-12T18:53:12Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd stopwatch timer mechanism to Trace2.\n\nTimers are an alternative to Trace2 Regions.  Regions are useful for\nmeasuring the time spent in various computation phases, such as the\ntime to read the index, time to scan for unstaged files, time to scan\nfor untracked files, and etc.\n\nHowever, regions are not appropriate in all places.  For example,\nduring a checkout, it would be very inefficient to use regions to\nmeasure the total time spent inflating objects from the ODB from\nacross the entire lifetime of the process; a per-unzip() region would\nflood the output and significantly slow the command; and some form of\npost-processing would be requried to compute the time spent in unzip().\n\nTimers can be used to measure a series of timer intervals and emit\na single summary event (at thread and/or process exit).\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  90 ++++++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  98 +++++++++++++\n t/t0211-trace2-perf.sh                 |  49 +++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               |  75 ++++++++++\n trace2.h                               |  43 ++++++\n trace2/tr2_tgt.h                       |   7 +\n trace2/tr2_tgt_event.c                 |  26 ++++\n trace2/tr2_tgt_normal.c                |  23 ++++\n trace2/tr2_tgt_perf.c                  |  24 ++++\n trace2/tr2_tls.c                       |  10 ++\n trace2/tr2_tls.h                       |  10 ++\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 +++++++++++++++++++\n 15 files changed, 784 insertions(+)\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 9d43909d068..75ce6f45603 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -769,6 +769,42 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning in the thread.  This event is generated when a thread\n+\texits for timers that requested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n+`\"timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning aggregated across all threads.  This event is generated\n+\twhen the process exits.\n++\n+------------\n+{\n+\t\"event\":\"timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\n@@ -1200,6 +1236,60 @@ d0 | main                     | data         | r0  |  0.002126 |  0.002126 | fsy\n d0 | main                     | exit         |     |  0.000470 |           |              | code:0\n d0 | main                     | atexit       |     |  0.000477 |           |              | code:0\n ----------------\n+\n+Stopwatch Timer Events::\n+\n+\tMeasure the time spent in a function call or span of code\n+\tthat might be called from many places within the code\n+\tthroughout the life of the process.\n++\n+----------------\n+static void expensive_function(void)\n+{\n+\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t...\n+\tsleep_millisec(1000); // Do something expensive\n+\t...\n+\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+}\n+\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\t...\n+\n+\texpensive_function();\n+\n+\t// Do something else 1...\n+\n+\texpensive_function();\n+\n+\t// Do something else 2...\n+\n+\texpensive_function();\n+\n+\treturn 0;\n+}\n+----------------\n++\n+In this example, we measure the total time spent in\n+`expensive_function()` regardless of when it is called\n+in the overall flow of the program.\n++\n+----------------\n+$ export GIT_TRACE2_PERF_BRIEF=1\n+$ export GIT_TRACE2_PERF=~/log.perf\n+$ t/helper/test-tool trace2 100timer 3 1000\n+...\n+$ cat ~/log.perf\n+d0 | main                     | version      |     |           |           |              | ...\n+d0 | main                     | start        |     |  0.001453 |           |              | t/helper/test-tool trace2 100timer 3 1000\n+d0 | main                     | cmd_name     |     |           |           |              | trace2 (trace2)\n+d0 | main                     | exit         |     |  3.003667 |           |              | code:0\n+d0 | main                     | timer        |     |           |           | test         | name:test1 intervals:3 total:3.001686 min:1.000254 max:1.000929\n+d0 | main                     | atexit       |     |  3.003796 |           |              | code:0\n+----------------\n+\n+\n == Future Work\n \n === Relationship to the Existing Trace Api (api-trace.txt)\ndiff --git a/Makefile b/Makefile\nindex cac3452edb9..820649bf62a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1102,6 +1102,7 @@ LIB_OBJS += trace2/tr2_tgt_event.o\n LIB_OBJS += trace2/tr2_tgt_normal.o\n LIB_OBJS += trace2/tr2_tgt_perf.o\n LIB_OBJS += trace2/tr2_tls.o\n+LIB_OBJS += trace2/tr2_tmr.o\n LIB_OBJS += trailer.o\n LIB_OBJS += transport-helper.o\n LIB_OBJS += transport.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex a714130ece7..f951b9e97d7 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -228,6 +228,101 @@ static int ut_010bug_BUG(int argc, const char **argv)\n \tBUG(\"a %s message\", \"BUG\");\n }\n \n+/*\n+ * Single-threaded timer test.  Create several intervals using the\n+ * TEST1 timer.  The test script can verify that an aggregate Trace2\n+ * \"timer\" event is emitted indicating that we started+stopped the\n+ * timer the requested number of times.\n+ */\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay>\";\n+\n+\tint count = 0;\n+\tint delay = 0;\n+\tint k;\n+\n+\tif (argc != 2)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t\tsleep_millisec(delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct ut_101_data {\n+\tint count;\n+\tint delay;\n+};\n+\n+static void *ut_101timer_thread_proc(void *_ut_101_data)\n+{\n+\tstruct ut_101_data *data = _ut_101_data;\n+\tint k;\n+\n+\ttrace2_thread_start(\"ut_101\");\n+\n+\tfor (k = 0; k < data->count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST2);\n+\t\tsleep_millisec(data->delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST2);\n+\t}\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Multi-threaded timer test.  Create several threads that each create\n+ * several intervals using the TEST2 timer.  The test script can verify\n+ * that an individual Trace2 \"th_timer\" events for each thread and an\n+ * aggregate \"timer\" event are generated.\n+ */\n+static int ut_101timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay> <threads>\";\n+\n+\tstruct ut_101_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_101timer_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -248,6 +343,9 @@ static struct unit_test ut_table[] = {\n \t{ ut_008bug,      \"008bug\",    \"\" },\n \t{ ut_009bug_BUG,  \"009bug_BUG\",\"\" },\n \t{ ut_010bug_BUG,  \"010bug_BUG\",\"\" },\n+\n+\t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n+\t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 22d0845544e..5c28424e657 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -173,4 +173,53 @@ test_expect_success 'using global config, perf stream, return code 0' '\n \ttest_cmp expect actual\n '\n \n+# Exercise the stopwatch timers in a loop and confirm that we have\n+# as many start/stop intervals as expected.  We cannot really test the\n+# actual (total, min, max) timer values, so we have to assume that they\n+# are good, but we can verify the interval count.\n+#\n+# The timer \"test/test1\" should only emit a global summary \"timer\" event.\n+# The timer \"test/test2\" should emit per-thread \"th_timer\" events and a\n+# global summary \"timer\" event.\n+\n+have_timer_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 intervals=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} intervals:${intervals}\" &&\n+\n+\tgrep \"${pattern}\" ${file}\n+}\n+\n+test_expect_success 'stopwatch timer test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test1\" 5 times from \"main\".\n+\ttest-tool trace2 100timer 5 10 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test1\" 5 actual\n+'\n+\n+test_expect_success 'stopwatch timer test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test2\" 5 times each in 3 threads.\n+\ttest-tool trace2 101timer 5 10 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_timer_event \"th01:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th02:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th03:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\n+\t# And we should have 15 total uses.\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n+'\n+\n test_done\ndiff --git a/t/t0211/scrub_perf.perl b/t/t0211/scrub_perf.perl\nindex 299999f0f89..7a50bae6463 100644\n--- a/t/t0211/scrub_perf.perl\n+++ b/t/t0211/scrub_perf.perl\n@@ -64,6 +64,12 @@ while (<>) {\n \t    goto SKIP_LINE;\n \t}\n     }\n+    elsif ($tokens[$col_event] =~ m/timer/) {\n+\t# This also captures \"th_timer\" events\n+\t$tokens[$col_rest] =~ s/ total:\\d+\\.\\d*/ total:_T_TOTAL_/;\n+\t$tokens[$col_rest] =~ s/ min:\\d+\\.\\d*/ min:_T_MIN_/;\n+\t$tokens[$col_rest] =~ s/ max:\\d+\\.\\d*/ max:_T_MAX_/;\n+    }\n \n     # t_abs and t_rel are either blank or a float.  Replace the float\n     # with a constant for matching the HEREDOC in the test script.\ndiff --git a/trace2.c b/trace2.c\nindex 165264dc79a..a93cab7c2b7 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -13,6 +13,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static int trace2_enabled;\n \n@@ -83,6 +84,23 @@ static void tr2_tgt_disable_builtins(void)\n \t\ttgt_j->pfn_term();\n }\n \n+/*\n+ * The signature of this function must match the pfn_timer\n+ * method in the targets.  (Think of this is an apply operation\n+ * across the set of active targets.)\n+ */\n+static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n+\t\t\t\t const struct tr2_timer *timer,\n+\t\t\t\t int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_timer)\n+\t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -110,6 +128,26 @@ static void tr2main_atexit_handler(void)\n \t */\n \ttr2tls_pop_unwind_self();\n \n+\t/*\n+\t * Some timers want per-thread details.  If the main thread\n+\t * used one of those timers, emit the details now (before\n+\t * we emit the aggregate timer values).\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data for the main thread to the final\n+\t * totals.  And then emit the final timer values.\n+\t *\n+\t * Technically, we shouldn't need to hold the lock to update\n+\t * and output the final_timer_block (since all other threads\n+\t * should be dead by now), but it doesn't hurt anything.\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_atexit)\n \t\t\ttgt_j->pfn_atexit(us_elapsed_absolute,\n@@ -541,6 +579,21 @@ void trace2_thread_exit_fl(const char *file, int line)\n \ttr2tls_pop_unwind_self();\n \tus_elapsed_thread = tr2tls_region_elasped_self(us_now);\n \n+\t/*\n+\t * Some timers want per-thread details.  If this thread used\n+\t * one of those timers, emit the details now.\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data from the current (non-main) thread\n+\t * to the final totals.  (We'll accumulate data for the main\n+\t * thread later during \"atexit\".)\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_exit_fl)\n \t\t\ttgt_j->pfn_thread_exit_fl(file, line,\n@@ -795,6 +848,28 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...)\n \tva_end(ap);\n }\n \n+void trace2_timer_start(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_start: invalid timer id: %d\", tid);\n+\n+\ttr2_start_timer(tid);\n+}\n+\n+void trace2_timer_stop(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_stop: invalid timer id: %d\", tid);\n+\n+\ttr2_stop_timer(tid);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 74cdb1354f7..7a843ac0518 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -51,6 +51,7 @@ struct json_writer;\n  * [] trace2_region*    -- emit region nesting messages.\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n+ * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n  */\n \n /*\n@@ -485,6 +486,48 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...);\n \n #define trace2_printf(...) trace2_printf_fl(__FILE__, __LINE__, __VA_ARGS__)\n \n+/*\n+ * Define the set of stopwatch timers.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum must also be added to the\n+ * `tr2_timer_metadata[]` in `trace2/tr2_tmr.c`.\n+ */\n+enum trace2_timer_id {\n+\t/*\n+\t * Define two timers for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_TIMER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_TIMER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional timer definitions before here. */\n+\tTRACE2_NUMBER_OF_TIMERS\n+};\n+\n+/*\n+ * Start/Stop the indicated stopwatch timer in the current thread.\n+ *\n+ * The time spent by the current thread between the _start and _stop\n+ * calls will be added to the thread's partial sum for this timer.\n+ *\n+ * Timer events are emitted at thread and program exit.\n+ *\n+ * Note: Since the stopwatch API routines do not generate individual\n+ * events, they do not take (file, line) arguments.  Similarly, the\n+ * category and timer name values are defined at compile-time in the\n+ * timer definitions array, so they are not needed here in the API.\n+ */\n+void trace2_timer_start(enum trace2_timer_id tid);\n+void trace2_timer_stop(enum trace2_timer_id tid);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 65f94e15748..2a80bef0df5 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -4,6 +4,8 @@\n struct child_process;\n struct repository;\n struct json_writer;\n+struct tr2_timer_metadata;\n+struct tr2_timer;\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -96,6 +98,10 @@ typedef void(tr2_tgt_evt_printf_va_fl_t)(const char *file, int line,\n \t\t\t\t\t uint64_t us_elapsed_absolute,\n \t\t\t\t\t const char *fmt, va_list ap);\n \n+typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n+\t\t\t\t  const struct tr2_timer *timer,\n+\t\t\t\t  int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -132,6 +138,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_fl_t                   *pfn_data_fl;\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n+\ttr2_tgt_evt_timer_t                     *pfn_timer;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 52f9356c695..1196da89ba4 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -9,6 +9,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_event = {\n \t.sysenv_var = TR2_SYSENV_EVENT,\n@@ -617,6 +618,30 @@ static void fn_data_json_fl(const char *file, int line,\n \t}\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"intervals\", timer->interval_count);\n+\tjw_object_double(&jw, \"t_total\", 6, t_total);\n+\tjw_object_double(&jw, \"t_min\", 6, t_min);\n+\tjw_object_double(&jw, \"t_max\", 6, t_max);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -648,4 +673,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 69f80330778..3888c10ef50 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -8,6 +8,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_normal = {\n \t.sysenv_var = TR2_SYSENV_NORMAL,\n@@ -329,6 +330,27 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -360,4 +382,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_fl = NULL,\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 59ca58f862d..89b30ddc0e4 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -10,6 +10,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_perf = {\n \t.sysenv_var = TR2_SYSENV_PERF,\n@@ -555,6 +556,28 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n+\n+\tstrbuf_addf(&buf_payload, (\"name:%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -586,4 +609,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 3a67532aae4..04900bb4c3a 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -181,3 +181,13 @@ int tr2tls_locked_increment(int *p)\n \n \treturn current_value;\n }\n+\n+void tr2tls_lock(void)\n+{\n+\tpthread_mutex_lock(&tr2tls_mutex);\n+}\n+\n+void tr2tls_unlock(void)\n+{\n+\tpthread_mutex_unlock(&tr2tls_mutex);\n+}\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex e17cc462f87..2322b0d0ef0 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n /*\n  * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n@@ -20,6 +21,9 @@ struct tr2tls_thread_ctx {\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n+\tstruct tr2_timer_block timer_block;\n+\tunsigned int used_any_timer:1;\n+\tunsigned int used_any_per_thread_timer:1;\n };\n \n /*\n@@ -107,4 +111,10 @@ int tr2tls_locked_increment(int *p);\n  */\n void tr2tls_start_process_clock(void);\n \n+/*\n+ * Explicitly lock/unlock our mutex.\n+ */\n+void tr2tls_lock(void);\n+void tr2tls_unlock(void);\n+\n #endif /* TR2_TLS_H */\ndiff --git a/trace2/tr2_tmr.c b/trace2/tr2_tmr.c\nnew file mode 100644\nindex 00000000000..786762dfd26\n--- /dev/null\n+++ b/trace2/tr2_tmr.c\n@@ -0,0 +1,182 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n+\n+#define MY_MAX(a, b) ((a) > (b) ? (a) : (b))\n+#define MY_MIN(a, b) ((a) < (b) ? (a) : (b))\n+\n+/*\n+ * A global timer block to aggregate values from the partial sums from\n+ * each thread.\n+ */\n+static struct tr2_timer_block final_timer_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each stopwatch timer.\n+ *\n+ * This array must match \"enum trace2_timer_id\" and the values\n+ * in \"struct tr2_timer_block.timer[*]\".\n+ */\n+static struct tr2_timer_metadata tr2_timer_metadata[TRACE2_NUMBER_OF_TIMERS] = {\n+\t[TRACE2_TIMER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_TIMER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_start_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\tt->recursion_count++;\n+\tif (t->recursion_count > 1)\n+\t\treturn; /* ignore recursive starts */\n+\n+\tt->start_ns = getnanotime();\n+}\n+\n+void tr2_stop_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\tuint64_t ns_now;\n+\tuint64_t ns_interval;\n+\n+\tassert(t->recursion_count > 0);\n+\n+\tt->recursion_count--;\n+\tif (t->recursion_count)\n+\t\treturn; /* still in recursive call(s) */\n+\n+\tns_now = getnanotime();\n+\tns_interval = ns_now - t->start_ns;\n+\n+\tt->total_ns += ns_interval;\n+\n+\t/*\n+\t * min_ns was initialized to zero (in the xcalloc()) rather\n+\t * than UINT_MAX when the block of timers was allocated,\n+\t * so we should always set both the min_ns and max_ns values\n+\t * the first time that the timer is used.\n+\t */\n+\tif (!t->interval_count) {\n+\t\tt->min_ns = ns_interval;\n+\t\tt->max_ns = ns_interval;\n+\t} else {\n+\t\tt->min_ns = MY_MIN(ns_interval, t->min_ns);\n+\t\tt->max_ns = MY_MAX(ns_interval, t->max_ns);\n+\t}\n+\n+\tt->interval_count++;\n+\n+\tctx->used_any_timer = 1;\n+\tif (tr2_timer_metadata[tid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_timer = 1;\n+}\n+\n+void tr2_update_final_timers(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++) {\n+\t\tstruct tr2_timer *t_final = &final_timer_block.timer[tid];\n+\t\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\t\tif (t->recursion_count) {\n+\t\t\t/*\n+\t\t\t * The current thread is exiting with\n+\t\t\t * timer[tid] still running.\n+\t\t\t *\n+\t\t\t * Technically, this is a bug, but I'm going\n+\t\t\t * to ignore it.\n+\t\t\t *\n+\t\t\t * I don't think it is worth calling die()\n+\t\t\t * for.  I don't think it is worth killing the\n+\t\t\t * process for this bookkeeping error.  We\n+\t\t\t * might want to call warning(), but I'm going\n+\t\t\t * to wait on that.\n+\t\t\t *\n+\t\t\t * The downside here is that total_ns won't\n+\t\t\t * include the current open interval (now -\n+\t\t\t * start_ns).  I can live with that.\n+\t\t\t */\n+\t\t}\n+\n+\t\tif (!t->interval_count)\n+\t\t\tcontinue; /* this timer was not used by this thread */\n+\n+\t\tt_final->total_ns += t->total_ns;\n+\n+\t\t/*\n+\t\t * final_timer_block.timer[tid].min_ns was initialized to\n+\t\t * was initialized to zero rather than UINT_MAX, so we should\n+\t\t * always set both the min_ns and max_ns values the first time\n+\t\t * that we add a partial sum into it.\n+\t\t */\n+\t\tif (!t_final->interval_count) {\n+\t\t\tt_final->min_ns = t->min_ns;\n+\t\t\tt_final->max_ns = t->max_ns;\n+\t\t} else {\n+\t\t\tt_final->min_ns = MY_MIN(t_final->min_ns, t->min_ns);\n+\t\t\tt_final->max_ns = MY_MAX(t_final->max_ns, t->max_ns);\n+\t\t}\n+\n+\t\tt_final->interval_count += t->interval_count;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_per_thread_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * For each timer, if the timer wants per-thread events and\n+\t * this thread used it, emit it.\n+\t */\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (tr2_timer_metadata[tid].want_per_thread_events &&\n+\t\t    ctx->timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &ctx->timer_block.timer[tid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tenum trace2_timer_id tid;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (final_timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &final_timer_block.timer[tid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_tmr.h b/trace2/tr2_tmr.h\nnew file mode 100644\nindex 00000000000..d5753576134\n--- /dev/null\n+++ b/trace2/tr2_tmr.h\n@@ -0,0 +1,140 @@\n+#ifndef TR2_TMR_H\n+#define TR2_TMR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow \"stopwatch\" timers.\n+ *\n+ * Timers can be used to measure \"interesting\" activity that does not\n+ * fit the \"region\" model, such as code called from many different\n+ * regions (like zlib) and/or where data for individual calls are not\n+ * interesting or are too numerous to be efficiently logged.\n+ *\n+ * Timer values are accumulated during program execution and emitted\n+ * to the Trace2 logs at program exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * timers and timer ids using a \"timer block\" array in thread-local\n+ * storage.  This gives us constant time access to each timer within\n+ * each thread, since we want start/stop operations to be as fast as\n+ * possible.  This lets us avoid the complexities of dynamically\n+ * allocating a timer on the first use by a thread and/or possibly\n+ * sharing that timer definition with other concurrent threads.\n+ * However, this does require that we define time the set of timers at\n+ * compile time.\n+ *\n+ * Each thread uses the timer block in its thread-local storage to\n+ * compute partial sums for each timer (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Using this \"timer block\" model costs ~48 bytes per timer per thread\n+ * (we have about six uint64 fields per timer).  This does increase\n+ * the size of the thread-local storage block, but it is allocated (at\n+ * thread create time) and not on the thread stack, so I'm not worried\n+ * about the size.\n+ *\n+ * Partial sums for each timer are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each timer are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"timer metadata\" table contains the \"category\" and \"name\"\n+ * fields for each timer.  This eliminates the need to include those\n+ * args in the various timer APIs.\n+ */\n+\n+/*\n+ * The definition of an individual timer and used by an individual\n+ * thread.\n+ */\n+struct tr2_timer {\n+\t/*\n+\t * Total elapsed time for this timer in this thread in nanoseconds.\n+\t */\n+\tuint64_t total_ns;\n+\n+\t/*\n+\t * The maximum and minimum interval values observed for this\n+\t * timer in this thread.\n+\t */\n+\tuint64_t min_ns;\n+\tuint64_t max_ns;\n+\n+\t/*\n+\t * The value of the clock when this timer was started in this\n+\t * thread.  (Undefined when the timer is not active in this\n+\t * thread.)\n+\t */\n+\tuint64_t start_ns;\n+\n+\t/*\n+\t * Number of times that this timer has been started and stopped\n+\t * in this thread.  (Recursive starts are ignored.)\n+\t */\n+\tuint64_t interval_count;\n+\n+\t/*\n+\t * Number of nested starts on the stack in this thread.  (We\n+\t * ignore recursive starts and use this to track the recursive\n+\t * calls.)\n+\t */\n+\tunsigned int recursion_count;\n+};\n+\n+/*\n+ * Metadata for a timer.\n+ */\n+struct tr2_timer_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this timer\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed-size block of timers to insert into\n+ * thread-local storage.  This wrapper is used to avoid quirks\n+ * of C and the usual need to pass an array size argument.\n+ */\n+struct tr2_timer_block {\n+\tstruct tr2_timer timer[TRACE2_NUMBER_OF_TIMERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to actually start/stop an\n+ * individual timer in the current thread.\n+ */\n+void tr2_start_timer(enum trace2_timer_id tid);\n+void tr2_stop_timer(enum trace2_timer_id tid);\n+\n+/*\n+ * Add the current thread's timer data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_timers(void);\n+\n+/*\n+ * Emit per-thread timer data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+/*\n+ * Emit global total timer values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+#endif /* TR2_TMR_H */\n-- \ngitgitgadget\n\n"},{"id":"464771","messageId":"221012.86edvcn3yc.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"637b422b8606b3b6d954e6a1959aae450507cdfa.1665600750.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-12T21:06:46Z","receivedAt":"2022-10-12T21:10:31Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Oct 12 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Rename the `thread_name` argument in `tr2tls_create_self()` and\n> `trace2_thread_start()` to be `thread_base_name` to make it clearer\n> that the passed argument is a component used in the construction of\n> the actual `struct tr2tls_thread_ctx.thread_name` variable.\n>\n> The base name will be used along with the thread id to create a\n> unique thread name.\n\nMakes sense.\n\n> This commit does not change how the `thread_name` field is\n> allocated or stored within the `tr2tls_thread_ctx` structure.\n\nWhat this commit does change though, which isn't mentioned here, is...\n\n> diff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\n> index 1297509fd23..7d1f03a2ea6 100644\n> --- a/trace2/tr2_tls.h\n> +++ b/trace2/tr2_tls.h\n> @@ -25,17 +25,20 @@ struct tr2tls_thread_ctx {\n>  /*\n>   * Create thread-local storage for the current thread.\n>   *\n> - * We assume the first thread is \"main\".  Other threads are given\n> - * non-zero thread-ids to help distinguish messages from concurrent\n> - * threads.\n> - *\n> - * Truncate the thread name if necessary to help with column alignment\n> - * in printf-style messages.\n> + * The first thread in the process will have:\n> + *     { .thread_id=0, .thread_name=\"main\" }\n> + * Subsequent threads are given a non-zero thread_id and a thread_name\n> + * constructed from the id and a thread base name (which is usually just\n> + * the name of the thread-proc function).  For example:\n> + *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n> + * This helps to identify and distinguish messages from concurrent threads.\n> + * The ctx.thread_name field is truncated if necessary to help with column\n> + * alignment in printf-style messages.\n\n...this documentation, which I'd argue should be a separate change, as\nnothing's changed about the state of the world with this rename of the\nfield, this was all true before this rename.\n"},{"id":"464892","messageId":"xmqq8rlje8cu.fsf@gitster.g","threadId":"58564","inReplyTo":"dd6d8e2841b424ba89672b4d94306f9ec882a868.1665600750.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/7] trace2: add stopwatch timers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-13T21:12:17Z","receivedAt":"2022-10-13T21:12:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Add stopwatch timer mechanism to Trace2.\n>\n>  trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++++\n>  trace2/tr2_tmr.h                       | 140 +++++++++++++++++++\n>  15 files changed, 784 insertions(+)\n>  create mode 100644 trace2/tr2_tmr.c\n>  create mode 100644 trace2/tr2_tmr.h\n\nWhew.  That's a lot of new code and doc to make two calls to\ngetnanotime() and accumulate the differences.\n\nIt was irritating to count zeros in the same constant 1000000000.0\nspelled out 9 times.  Perhaps something like\n\n#define NS_TO_SECONDS(ns) ((double)(ns) / (1000*1000*1000.))\n\nwould have helped?\n\nOther than that, all looked reasonable.\n\nThanks.\n"},{"id":"464893","messageId":"xmqq1qrbe8cm.fsf@gitster.g","threadId":"58564","inReplyTo":"4bf78e356e23c947b8328a91ba435a357cd51f43.1665600750.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 5/7] trace2: convert ctx.thread_name from strbuf to pointer","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-13T21:12:25Z","receivedAt":"2022-10-13T21:12:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Convert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\n> to a \"const char*\" pointer.\n>\n> The `thread_name` field is a constant string that is constructed when\n> the context is created.  Using a (non-const) `strbuf` structure for it\n> caused some confusion in the past because it implied that someone\n> could rename a thread after it was created.  That usage was not\n> intended.  Change it to a const pointer to make the intent more clear.\n>\n> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n> ---\n>  trace2/tr2_tgt_event.c |  2 +-\n>  trace2/tr2_tgt_perf.c  |  2 +-\n>  trace2/tr2_tls.c       | 16 +++++++++-------\n>  trace2/tr2_tls.h       |  2 +-\n>  4 files changed, 12 insertions(+), 10 deletions(-)\n\nLooking good so far.\n"},{"id":"464894","messageId":"xmqqtu47ctrw.fsf@gitster.g","threadId":"58564","inReplyTo":"637b422b8606b3b6d954e6a1959aae450507cdfa.1665600750.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-13T21:12:35Z","receivedAt":"2022-10-13T21:13:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Rename the `thread_name` argument in `tr2tls_create_self()` and\n> `trace2_thread_start()` to be `thread_base_name` to make it clearer\n> that the passed argument is a component used in the construction of\n> the actual `struct tr2tls_thread_ctx.thread_name` variable.\n>\n> The base name will be used along with the thread id to create a\n> unique thread name.\n> ...\n> -struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n> +struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n>  \t\t\t\t\t     uint64_t us_thread_start)\n>  {\n>  \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n> @@ -50,7 +50,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n>  \tstrbuf_init(&ctx->thread_name, 0);\n>  \tif (ctx->thread_id)\n>  \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n> -\tstrbuf_addstr(&ctx->thread_name, thread_name);\n> +\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n>  \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n>  \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n\nThis hunk is very illustrative and highlights the difference between\nthread_base_name parameter and .thread_name member in the context.\n\nGood.\n\n"},{"id":"464895","messageId":"xmqqmt9zctrk.fsf@gitster.g","threadId":"58564","inReplyTo":"9dee7a75903936f086d97580441c776978d70b43.1665600750.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] tr2tls: clarify TLS terminology","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-13T21:12:47Z","receivedAt":"2022-10-13T21:14:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n>  `\"thread_start\"`::\n>  \tThis event is generated when a thread is started.  It is\n> -\tgenerated from *within* the new thread's thread-proc (for TLS\n> -\treasons).\n> +\tgenerated from *within* the new thread's thread-proc (because\n> +\tit needs to access data in the thread's thread-local storage).\n\nThis is a vast improvement, not just \"TLS\" -> \"thread-local strage\",\nbut the original \"for TLS reasons\" would not be understood by anybody\nwho does not already know.\n\n"},{"id":"465293","messageId":"27526718-7961-dc2c-946e-98757b3b36c8@jeffhostetler.com","threadId":"58564","inReplyTo":"221012.86edvcn3yc.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 4/7] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-20T14:40:30Z","receivedAt":"2022-10-20T14:40:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/12/22 5:06 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Oct 12 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Rename the `thread_name` argument in `tr2tls_create_self()` and\n>> `trace2_thread_start()` to be `thread_base_name` to make it clearer\n>> that the passed argument is a component used in the construction of\n>> the actual `struct tr2tls_thread_ctx.thread_name` variable.\n>>\n>> The base name will be used along with the thread id to create a\n>> unique thread name.\n> \n> Makes sense.\n> \n>> This commit does not change how the `thread_name` field is\n>> allocated or stored within the `tr2tls_thread_ctx` structure.\n> \n> What this commit does change though, which isn't mentioned here, is...\n> \n>> diff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\n>> index 1297509fd23..7d1f03a2ea6 100644\n>> --- a/trace2/tr2_tls.h\n>> +++ b/trace2/tr2_tls.h\n>> @@ -25,17 +25,20 @@ struct tr2tls_thread_ctx {\n>>   /*\n>>    * Create thread-local storage for the current thread.\n>>    *\n>> - * We assume the first thread is \"main\".  Other threads are given\n>> - * non-zero thread-ids to help distinguish messages from concurrent\n>> - * threads.\n>> - *\n>> - * Truncate the thread name if necessary to help with column alignment\n>> - * in printf-style messages.\n>> + * The first thread in the process will have:\n>> + *     { .thread_id=0, .thread_name=\"main\" }\n>> + * Subsequent threads are given a non-zero thread_id and a thread_name\n>> + * constructed from the id and a thread base name (which is usually just\n>> + * the name of the thread-proc function).  For example:\n>> + *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n>> + * This helps to identify and distinguish messages from concurrent threads.\n>> + * The ctx.thread_name field is truncated if necessary to help with column\n>> + * alignment in printf-style messages.\n> \n> ...this documentation, which I'd argue should be a separate change, as\n> nothing's changed about the state of the world with this rename of the\n> field, this was all true before this rename.\n> \n\ngood point.  i'll split it and resend.\nthanks\nJeff\n"},{"id":"465294","messageId":"aeb07c4f-f3f2-4965-6b6b-3ba3b10b2103@jeffhostetler.com","threadId":"58564","inReplyTo":"xmqq8rlje8cu.fsf@gitster.g","subject":"Re: [PATCH v2 6/7] trace2: add stopwatch timers","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-20T14:42:12Z","receivedAt":"2022-10-20T14:42:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/13/22 5:12 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Add stopwatch timer mechanism to Trace2.\n[...]\n> It was irritating to count zeros in the same constant 1000000000.0\n> spelled out 9 times.  Perhaps something like\n> \n> #define NS_TO_SECONDS(ns) ((double)(ns) / (1000*1000*1000.))\n> \n> would have helped?\n\ngood point.  i'll resend.\n\nthanks\njeff\n"},{"id":"465314","messageId":"6e7e4f3187e2fbbbb54bb1cf5793bf6e981a5a94.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:02Z","receivedAt":"2022-10-20T18:28:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nUse \"size_t\" rather than \"int\" for the \"alloc\" and \"nr_open_regions\"\nfields in the \"tr2tls_thread_ctx\".  These are used by ALLOC_GROW().\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex b1e327a928e..a90bd639d48 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -11,8 +11,8 @@\n struct tr2tls_thread_ctx {\n \tstruct strbuf thread_name;\n \tuint64_t *array_us_start;\n-\tint alloc;\n-\tint nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n+\tsize_t alloc;\n+\tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n };\n \n-- \ngitgitgadget\n\n"},{"id":"465315","messageId":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v2.git.1665600750.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] Trace2 timers and counters and some cleanup","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:01Z","receivedAt":"2022-10-20T18:28:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 2 of this series to add timers and counters to Trace2.\n\nChanges since V1:\n\n * I dropped the commits concerning compiler errors in Clang 11.0.0 on\n   MacOS. I've sent them to the mailing list in a separate series, since\n   they had nothing to do with the main topic of this series.\n\n * I moved the documentation changes earlier in the series to get it out of\n   the way (and eliminate the need to update it later commits).\n\n * After a long conversation on the mailing list, I redid the two\n   thread-name commits to simplify and hopefully eliminate the remaining\n   misunderstandings and/or short-comings of my previous attempt and\n   explanations. We now use a \"const char *\" for the field in the thread-ctx\n   that we format and detach from a strbuf during thread-start. The goal\n   here is to move away from a modifyable strbuf in the thread-ctx itself\n   (to avoid giving the appearance that a caller could modify the\n   thread-name at some point, when that was not intended).\n\nThe last 2 commits add the stopwatch timers and the global counters and are\nunchanged from the previous version.\n\nJeff Hostetler (8):\n  trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n  tr2tls: clarify TLS terminology\n  api-trace2.txt: elminate section describing the public trace2 API\n  trace2: rename the thread_name argument to trace2_thread_start\n  trace2: improve thread-name documentation in the thread-context\n  trace2: convert ctx.thread_name from strbuf to pointer\n  trace2: add stopwatch timers\n  trace2: add global counter mechanism\n\n Documentation/technical/api-trace2.txt | 190 +++++++++++++++++--------\n Makefile                               |   2 +\n t/helper/test-trace2.c                 | 187 ++++++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  95 +++++++++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               | 121 +++++++++++++++-\n trace2.h                               | 101 +++++++++++--\n trace2/tr2_ctr.c                       | 101 +++++++++++++\n trace2/tr2_ctr.h                       | 104 ++++++++++++++\n trace2/tr2_tgt.h                       |  16 +++\n trace2/tr2_tgt_event.c                 |  47 +++++-\n trace2/tr2_tgt_normal.c                |  39 +++++\n trace2/tr2_tgt_perf.c                  |  43 +++++-\n trace2/tr2_tls.c                       |  34 +++--\n trace2/tr2_tls.h                       |  55 ++++---\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 ++++++++++++++++++\n 17 files changed, 1361 insertions(+), 102 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\n\nbase-commit: 3dcec76d9df911ed8321007b1d197c1a206dc164\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1373%2Fjeffhostetler%2Ftrace2-stopwatch-v4-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1373/jeffhostetler/trace2-stopwatch-v4-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1373\n\nRange-diff vs v2:\n\n 1:  6e7e4f3187e = 1:  6e7e4f3187e trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n 2:  9dee7a75903 = 2:  9dee7a75903 tr2tls: clarify TLS terminology\n 3:  804dab9e1a7 = 3:  804dab9e1a7 api-trace2.txt: elminate section describing the public trace2 API\n 4:  637b422b860 ! 4:  9adf9cee1a9 trace2: rename the thread_name argument to trace2_thread_start\n     @@ trace2/tr2_tls.c: struct tr2tls_thread_ctx *tr2tls_create_self(const char *threa\n      \n       ## trace2/tr2_tls.h ##\n      @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n     - /*\n     -  * Create thread-local storage for the current thread.\n     -  *\n     -- * We assume the first thread is \"main\".  Other threads are given\n     -- * non-zero thread-ids to help distinguish messages from concurrent\n     -- * threads.\n     -- *\n     -- * Truncate the thread name if necessary to help with column alignment\n     -- * in printf-style messages.\n     -+ * The first thread in the process will have:\n     -+ *     { .thread_id=0, .thread_name=\"main\" }\n     -+ * Subsequent threads are given a non-zero thread_id and a thread_name\n     -+ * constructed from the id and a thread base name (which is usually just\n     -+ * the name of the thread-proc function).  For example:\n     -+ *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n     -+ * This helps to identify and distinguish messages from concurrent threads.\n     -+ * The ctx.thread_name field is truncated if necessary to help with column\n     -+ * alignment in printf-style messages.\n     -  *\n        * In this and all following functions the term \"self\" refers to the\n        * current thread.\n        */\n -:  ----------- > 5:  8cb206b7632 trace2: improve thread-name documentation in the thread-context\n 5:  4bf78e356e2 = 6:  8a89e1aa238 trace2: convert ctx.thread_name from strbuf to pointer\n 6:  dd6d8e2841b ! 7:  8e701109976 trace2: add stopwatch timers\n     @@ trace2/tr2_tgt.h\n       struct json_writer;\n      +struct tr2_timer_metadata;\n      +struct tr2_timer;\n     ++\n     ++#define NS_PER_SEC_D ((double)1000*1000*1000)\n       \n       /*\n        * Function prototypes for a TRACE2 \"target\" vtable.\n     @@ trace2/tr2_tgt_event.c: static void fn_data_json_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct json_writer jw = JSON_WRITER_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n     -+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n     -+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n     ++\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     ++\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     ++\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n      +\n      +\tjw_object_begin(&jw, 0);\n      +\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n     @@ trace2/tr2_tgt_normal.c: static void fn_printf_va_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct strbuf buf_payload = STRBUF_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n     -+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n     -+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n     ++\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     ++\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     ++\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n      +\n      +\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n      +\t\t\t\t   \" intervals:%\"PRIu64\n     @@ trace2/tr2_tgt_perf.c: static void fn_printf_va_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct strbuf buf_payload = STRBUF_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / 1000000000.0;\n     -+\tdouble t_min = ((double)timer->min_ns) / 1000000000.0;\n     -+\tdouble t_max = ((double)timer->max_ns) / 1000000000.0;\n     ++\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     ++\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     ++\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n      +\n      +\tstrbuf_addf(&buf_payload, (\"name:%s\"\n      +\t\t\t\t   \" intervals:%\"PRIu64\n 7:  cf012fcde37 ! 8:  5cd8bdde884 trace2: add global counter mechanism\n     @@ trace2/tr2_tgt.h: struct repository;\n      +struct tr2_counter_metadata;\n      +struct tr2_counter;\n       \n     - /*\n     -  * Function prototypes for a TRACE2 \"target\" vtable.\n     + #define NS_PER_SEC_D ((double)1000*1000*1000)\n     + \n      @@ trace2/tr2_tgt.h: typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n       \t\t\t\t  const struct tr2_timer *timer,\n       \t\t\t\t  int is_final_data);\n\n-- \ngitgitgadget\n"},{"id":"465316","messageId":"9dee7a75903936f086d97580441c776978d70b43.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] tr2tls: clarify TLS terminology","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:03Z","receivedAt":"2022-10-20T18:28:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nReduce or eliminate use of the term \"TLS\" in the Trace2 code.\n\nThe term \"TLS\" has two popular meanings: \"thread-local storage\" and\n\"transport layer security\".  In the Trace2 source, the term is associated\nwith the former.  There was concern on the mailing list about it refering\nto the latter.\n\nUpdate the source and documentation to eliminate the use of the \"TLS\" term\nor replace it with the phrase \"thread-local storage\" to reduce ambiguity.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  8 ++++----\n trace2.c                               |  2 +-\n trace2.h                               | 10 +++++-----\n trace2/tr2_tls.c                       |  6 +++---\n trace2/tr2_tls.h                       | 18 +++++++++++-------\n 5 files changed, 24 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 2afa28bb5aa..431d424f9d5 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -685,8 +685,8 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_start\"`::\n \tThis event is generated when a thread is started.  It is\n-\tgenerated from *within* the new thread's thread-proc (for TLS\n-\treasons).\n+\tgenerated from *within* the new thread's thread-proc (because\n+\tit needs to access data in the thread's thread-local storage).\n +\n ------------\n {\n@@ -698,7 +698,7 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_exit\"`::\n \tThis event is generated when a thread exits.  It is generated\n-\tfrom *within* the thread's thread-proc (for TLS reasons).\n+\tfrom *within* the thread's thread-proc.\n +\n ------------\n {\n@@ -1206,7 +1206,7 @@ worked on 508 items at offset 2032.  Thread \"th04\" worked on 508 items\n at offset 508.\n +\n This example also shows that thread names are assigned in a racy manner\n-as each thread starts and allocates TLS storage.\n+as each thread starts.\n \n Config (def param) Events::\n \ndiff --git a/trace2.c b/trace2.c\nindex 0c0a11e07d5..c1244e45ace 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -52,7 +52,7 @@ static struct tr2_tgt *tr2_tgt_builtins[] =\n  * Force (rather than lazily) initialize any of the requested\n  * builtin TRACE2 targets at startup (and before we've seen an\n  * actual TRACE2 event call) so we can see if we need to setup\n- * the TR2 and TLS machinery.\n+ * private data structures and thread-local storage.\n  *\n  * Return the number of builtin targets enabled.\n  */\ndiff --git a/trace2.h b/trace2.h\nindex 88d906ea830..af3c11694cc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -73,8 +73,7 @@ void trace2_initialize_clock(void);\n /*\n  * Initialize TRACE2 tracing facility if any of the builtin TRACE2\n  * targets are enabled in the system config or the environment.\n- * This includes setting up the Trace2 thread local storage (TLS).\n- * Emits a 'version' message containing the version of git\n+ * This emits a 'version' message containing the version of git\n  * and the Trace2 protocol.\n  *\n  * This function should be called from `main()` as early as possible in\n@@ -302,7 +301,8 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n \n /*\n  * Emit a 'thread_start' event.  This must be called from inside the\n- * thread-proc to set up the trace2 TLS data for the thread.\n+ * thread-proc to allow the thread to create its own thread-local\n+ * storage.\n  *\n  * Thread names should be descriptive, like \"preload_index\".\n  * Thread names will be decorated with an instance number automatically.\n@@ -315,8 +315,8 @@ void trace2_thread_start_fl(const char *file, int line,\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\n- * thread-proc to report thread-specific data and cleanup TLS data\n- * for the thread.\n+ * thread-proc so that the thread can access and clean up its\n+ * thread-local storage.\n  */\n void trace2_thread_exit_fl(const char *file, int line);\n \ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 7da94aba522..8d2182fbdbb 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -69,9 +69,9 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void)\n \tctx = pthread_getspecific(tr2tls_key);\n \n \t/*\n-\t * If the thread-proc did not call trace2_thread_start(), we won't\n-\t * have any TLS data associated with the current thread.  Fix it\n-\t * here and silently continue.\n+\t * If the current thread's thread-proc did not call\n+\t * trace2_thread_start(), then the thread will not have any\n+\t * thread-local storage.  Create it now and silently continue.\n \t */\n \tif (!ctx)\n \t\tctx = tr2tls_create_self(\"unknown\", getnanotime() / 1000);\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex a90bd639d48..1297509fd23 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -3,6 +3,12 @@\n \n #include \"strbuf.h\"\n \n+/*\n+ * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n+ * Trace2 source files.  This usage is borrowed from GCC and Windows.\n+ * There is NO relation to \"transport layer security\".\n+ */\n+\n /*\n  * Arbitry limit for thread names for column alignment.\n  */\n@@ -17,9 +23,7 @@ struct tr2tls_thread_ctx {\n };\n \n /*\n- * Create TLS data for the current thread.  This gives us a place to\n- * put per-thread data, such as thread start time, function nesting\n- * and a per-thread label for our messages.\n+ * Create thread-local storage for the current thread.\n  *\n  * We assume the first thread is \"main\".  Other threads are given\n  * non-zero thread-ids to help distinguish messages from concurrent\n@@ -35,7 +39,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n- * Get our TLS data.\n+ * Get the thread-local storage pointer of the current thread.\n  */\n struct tr2tls_thread_ctx *tr2tls_get_self(void);\n \n@@ -45,7 +49,7 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void);\n int tr2tls_is_main_thread(void);\n \n /*\n- * Free our TLS data.\n+ * Free the current thread's thread-local storage.\n  */\n void tr2tls_unset_self(void);\n \n@@ -81,12 +85,12 @@ uint64_t tr2tls_region_elasped_self(uint64_t us);\n uint64_t tr2tls_absolute_elapsed(uint64_t us);\n \n /*\n- * Initialize the tr2 TLS system.\n+ * Initialize thread-local storage for Trace2.\n  */\n void tr2tls_init(void);\n \n /*\n- * Free all tr2 TLS resources.\n+ * Free all Trace2 thread-local storage resources.\n  */\n void tr2tls_release(void);\n \n-- \ngitgitgadget\n\n"},{"id":"465317","messageId":"9adf9cee1a96211cc4c2a305997079c7d6492aea.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:05Z","receivedAt":"2022-10-20T18:28:23Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nRename the `thread_name` argument in `tr2tls_create_self()` and\n`trace2_thread_start()` to be `thread_base_name` to make it clearer\nthat the passed argument is a component used in the construction of\nthe actual `struct tr2tls_thread_ctx.thread_name` variable.\n\nThe base name will be used along with the thread id to create a\nunique thread name.\n\nThis commit does not change how the `thread_name` field is\nallocated or stored within the `tr2tls_thread_ctx` structure.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2.c         |  6 +++---\n trace2.h         | 11 ++++++-----\n trace2/tr2_tls.c |  4 ++--\n trace2/tr2_tls.h |  2 +-\n 4 files changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/trace2.c b/trace2.c\nindex c1244e45ace..165264dc79a 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -466,7 +466,7 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code)\n \t\t\t\tfile, line, us_elapsed_absolute, exec_id, code);\n }\n \n-void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n+void trace2_thread_start_fl(const char *file, int line, const char *thread_base_name)\n {\n \tstruct tr2_tgt *tgt_j;\n \tint j;\n@@ -488,14 +488,14 @@ void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n \t\t */\n \t\ttrace2_region_enter_printf_fl(file, line, NULL, NULL, NULL,\n \t\t\t\t\t      \"thread-proc on main: %s\",\n-\t\t\t\t\t      thread_name);\n+\t\t\t\t\t      thread_base_name);\n \t\treturn;\n \t}\n \n \tus_now = getnanotime() / 1000;\n \tus_elapsed_absolute = tr2tls_absolute_elapsed(us_now);\n \n-\ttr2tls_create_self(thread_name, us_now);\n+\ttr2tls_create_self(thread_base_name, us_now);\n \n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_start_fl)\ndiff --git a/trace2.h b/trace2.h\nindex af3c11694cc..74cdb1354f7 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -304,14 +304,15 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n  * thread-proc to allow the thread to create its own thread-local\n  * storage.\n  *\n- * Thread names should be descriptive, like \"preload_index\".\n- * Thread names will be decorated with an instance number automatically.\n+ * The thread base name should be descriptive, like \"preload_index\" or\n+ * taken from the thread-proc function.  A unique thread name will be\n+ * created from the given base name and the thread id automatically.\n  */\n void trace2_thread_start_fl(const char *file, int line,\n-\t\t\t    const char *thread_name);\n+\t\t\t    const char *thread_base_name);\n \n-#define trace2_thread_start(thread_name) \\\n-\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n+#define trace2_thread_start(thread_base_name) \\\n+\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_base_name))\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 8d2182fbdbb..4f7c516ecb6 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -31,7 +31,7 @@ void tr2tls_start_process_clock(void)\n \ttr2tls_us_start_process = getnanotime() / 1000;\n }\n \n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n@@ -50,7 +50,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \tstrbuf_init(&ctx->thread_name, 0);\n \tif (ctx->thread_id)\n \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_name);\n+\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n \ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 1297509fd23..d4e725f430b 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -35,7 +35,7 @@ struct tr2tls_thread_ctx {\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n  */\n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"465318","messageId":"804dab9e1a7fa1cea9355bac92ada16332f1194e.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] api-trace2.txt: elminate section describing the public trace2 API","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:04Z","receivedAt":"2022-10-20T18:28:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nEliminate the mostly obsolete `Public API` sub-section from the\n`Trace2 API` section in the documentation.  Strengthen the referral\nto `trace2.h`.\n\nMost of the technical information in this sub-section was moved to\n`trace2.h` in 6c51cb525d (trace2: move doc to trace2.h, 2019-11-17) to\nbe adjacent to the function prototypes.  The remaining text wasn't\nthat useful by itself.\n\nFurthermore, the text would need a bit of overhaul to add routines\nthat do not immediately generate a message, such as stopwatch timers.\nSo it seemed simpler to just get rid of it.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt | 61 +++-----------------------\n 1 file changed, 7 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 431d424f9d5..9d43909d068 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -148,20 +148,18 @@ filename collisions).\n \n == Trace2 API\n \n-All public Trace2 functions and macros are defined in `trace2.h` and\n-`trace2.c`.  All public symbols are prefixed with `trace2_`.\n+The Trace2 public API is defined and documented in `trace2.h`; refer to it for\n+more information.  All public functions and macros are prefixed\n+with `trace2_` and are implemented in `trace2.c`.\n \n There are no public Trace2 data structures.\n \n The Trace2 code also defines a set of private functions and data types\n in the `trace2/` directory.  These symbols are prefixed with `tr2_`\n-and should only be used by functions in `trace2.c`.\n+and should only be used by functions in `trace2.c` (or other private\n+source files in `trace2/`).\n \n-== Conventions for Public Functions and Macros\n-\n-The functions defined by the Trace2 API are declared and documented\n-in `trace2.h`.  It defines the API functions and wrapper macros for\n-Trace2.\n+=== Conventions for Public Functions and Macros\n \n Some functions have a `_fl()` suffix to indicate that they take `file`\n and `line-number` arguments.\n@@ -172,52 +170,7 @@ take a `va_list` argument.\n Some functions have a `_printf_fl()` suffix to indicate that they also\n take a `printf()` style format with a variable number of arguments.\n \n-There are CPP wrapper macros and `#ifdef`s to hide most of these details.\n-See `trace2.h` for more details.  The following discussion will only\n-describe the simplified forms.\n-\n-== Public API\n-\n-All Trace2 API functions send a message to all of the active\n-Trace2 Targets.  This section describes the set of available\n-messages.\n-\n-It helps to divide these functions into groups for discussion\n-purposes.\n-\n-=== Basic Command Messages\n-\n-These are concerned with the lifetime of the overall git process.\n-e.g: `void trace2_initialize_clock()`, `void trace2_initialize()`,\n-`int trace2_is_enabled()`, `void trace2_cmd_start(int argc, const char **argv)`.\n-\n-=== Command Detail Messages\n-\n-These are concerned with describing the specific Git command\n-after the command line, config, and environment are inspected.\n-e.g: `void trace2_cmd_name(const char *name)`,\n-`void trace2_cmd_mode(const char *mode)`.\n-\n-=== Child Process Messages\n-\n-These are concerned with the various spawned child processes,\n-including shell scripts, git commands, editors, pagers, and hooks.\n-\n-e.g: `void trace2_child_start(struct child_process *cmd)`.\n-\n-=== Git Thread Messages\n-\n-These messages are concerned with Git thread usage.\n-\n-e.g: `void trace2_thread_start(const char *thread_name)`.\n-\n-=== Region and Data Messages\n-\n-These are concerned with recording performance data\n-over regions or spans of code. e.g:\n-`void trace2_region_enter(const char *category, const char *label, const struct repository *repo)`.\n-\n-Refer to trace2.h for details about all trace2 functions.\n+CPP wrapper macros are defined to hide most of these details.\n \n == Trace2 Target Formats\n \n-- \ngitgitgadget\n\n"},{"id":"465319","messageId":"8cb206b76323e14d8e07f6cfb5aa482a47eb54c5.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] trace2: improve thread-name documentation in the thread-context","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:06Z","receivedAt":"2022-10-20T18:28:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nImprove the documentation of the tr2tls_thread_ctx.thread_name field\nand its relation to the tr2tls_thread_ctx.thread_id field.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 15 +++++++++------\n 1 file changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex d4e725f430b..7d1f03a2ea6 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -25,12 +25,15 @@ struct tr2tls_thread_ctx {\n /*\n  * Create thread-local storage for the current thread.\n  *\n- * We assume the first thread is \"main\".  Other threads are given\n- * non-zero thread-ids to help distinguish messages from concurrent\n- * threads.\n- *\n- * Truncate the thread name if necessary to help with column alignment\n- * in printf-style messages.\n+ * The first thread in the process will have:\n+ *     { .thread_id=0, .thread_name=\"main\" }\n+ * Subsequent threads are given a non-zero thread_id and a thread_name\n+ * constructed from the id and a thread base name (which is usually just\n+ * the name of the thread-proc function).  For example:\n+ *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n+ * This helps to identify and distinguish messages from concurrent threads.\n+ * The ctx.thread_name field is truncated if necessary to help with column\n+ * alignment in printf-style messages.\n  *\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n-- \ngitgitgadget\n\n"},{"id":"465320","messageId":"8a89e1aa238bff874f3fa9a414781e4abc03c526.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] trace2: convert ctx.thread_name from strbuf to pointer","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:07Z","receivedAt":"2022-10-20T18:28:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nConvert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\nto a \"const char*\" pointer.\n\nThe `thread_name` field is a constant string that is constructed when\nthe context is created.  Using a (non-const) `strbuf` structure for it\ncaused some confusion in the past because it implied that someone\ncould rename a thread after it was created.  That usage was not\nintended.  Change it to a const pointer to make the intent more clear.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tgt_event.c |  2 +-\n trace2/tr2_tgt_perf.c  |  2 +-\n trace2/tr2_tls.c       | 16 +++++++++-------\n trace2/tr2_tls.h       |  2 +-\n 4 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 37a3163be12..52f9356c695 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -90,7 +90,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n \n \tjw_object_string(jw, \"event\", event_name);\n \tjw_object_string(jw, \"sid\", tr2_sid_get());\n-\tjw_object_string(jw, \"thread\", ctx->thread_name.buf);\n+\tjw_object_string(jw, \"thread\", ctx->thread_name);\n \n \t/*\n \t * In brief mode, only emit <time> on these 2 event types.\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 8cb792488c8..59ca58f862d 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -108,7 +108,7 @@ static void perf_fmt_prepare(const char *event_name,\n \n \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n \tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n-\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n+\t\t    ctx->thread_name, TR2FMT_PERF_MAX_EVENT_NAME,\n \t\t    event_name);\n \n \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 4f7c516ecb6..3a67532aae4 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -35,6 +35,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n+\tstruct strbuf buf = STRBUF_INIT;\n \n \t/*\n \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n@@ -47,12 +48,13 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \n \tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n \n-\tstrbuf_init(&ctx->thread_name, 0);\n+\tstrbuf_init(&buf, 0);\n \tif (ctx->thread_id)\n-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n+\t\tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n+\tstrbuf_addstr(&buf, thread_base_name);\n+\tif (buf.len > TR2_MAX_THREAD_NAME)\n+\t\tstrbuf_setlen(&buf, TR2_MAX_THREAD_NAME);\n+\tctx->thread_name = strbuf_detach(&buf, NULL);\n \n \tpthread_setspecific(tr2tls_key, ctx);\n \n@@ -95,7 +97,7 @@ void tr2tls_unset_self(void)\n \n \tpthread_setspecific(tr2tls_key, NULL);\n \n-\tstrbuf_release(&ctx->thread_name);\n+\tfree((char *)ctx->thread_name);\n \tfree(ctx->array_us_start);\n \tfree(ctx);\n }\n@@ -113,7 +115,7 @@ void tr2tls_pop_self(void)\n \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n \n \tif (!ctx->nr_open_regions)\n-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n+\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n \n \tctx->nr_open_regions--;\n }\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 7d1f03a2ea6..e17cc462f87 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -15,7 +15,7 @@\n #define TR2_MAX_THREAD_NAME (24)\n \n struct tr2tls_thread_ctx {\n-\tstruct strbuf thread_name;\n+\tconst char *thread_name;\n \tuint64_t *array_us_start;\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n-- \ngitgitgadget\n\n"},{"id":"465321","messageId":"8e701109976777ad8fae1e0cd3908bb11a1fcf93.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] trace2: add stopwatch timers","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:08Z","receivedAt":"2022-10-20T18:28:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd stopwatch timer mechanism to Trace2.\n\nTimers are an alternative to Trace2 Regions.  Regions are useful for\nmeasuring the time spent in various computation phases, such as the\ntime to read the index, time to scan for unstaged files, time to scan\nfor untracked files, and etc.\n\nHowever, regions are not appropriate in all places.  For example,\nduring a checkout, it would be very inefficient to use regions to\nmeasure the total time spent inflating objects from the ODB from\nacross the entire lifetime of the process; a per-unzip() region would\nflood the output and significantly slow the command; and some form of\npost-processing would be requried to compute the time spent in unzip().\n\nTimers can be used to measure a series of timer intervals and emit\na single summary event (at thread and/or process exit).\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  90 ++++++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  98 +++++++++++++\n t/t0211-trace2-perf.sh                 |  49 +++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               |  75 ++++++++++\n trace2.h                               |  43 ++++++\n trace2/tr2_tgt.h                       |   9 ++\n trace2/tr2_tgt_event.c                 |  26 ++++\n trace2/tr2_tgt_normal.c                |  23 ++++\n trace2/tr2_tgt_perf.c                  |  24 ++++\n trace2/tr2_tls.c                       |  10 ++\n trace2/tr2_tls.h                       |  10 ++\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 +++++++++++++++++++\n 15 files changed, 786 insertions(+)\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 9d43909d068..75ce6f45603 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -769,6 +769,42 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning in the thread.  This event is generated when a thread\n+\texits for timers that requested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n+`\"timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning aggregated across all threads.  This event is generated\n+\twhen the process exits.\n++\n+------------\n+{\n+\t\"event\":\"timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\n@@ -1200,6 +1236,60 @@ d0 | main                     | data         | r0  |  0.002126 |  0.002126 | fsy\n d0 | main                     | exit         |     |  0.000470 |           |              | code:0\n d0 | main                     | atexit       |     |  0.000477 |           |              | code:0\n ----------------\n+\n+Stopwatch Timer Events::\n+\n+\tMeasure the time spent in a function call or span of code\n+\tthat might be called from many places within the code\n+\tthroughout the life of the process.\n++\n+----------------\n+static void expensive_function(void)\n+{\n+\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t...\n+\tsleep_millisec(1000); // Do something expensive\n+\t...\n+\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+}\n+\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\t...\n+\n+\texpensive_function();\n+\n+\t// Do something else 1...\n+\n+\texpensive_function();\n+\n+\t// Do something else 2...\n+\n+\texpensive_function();\n+\n+\treturn 0;\n+}\n+----------------\n++\n+In this example, we measure the total time spent in\n+`expensive_function()` regardless of when it is called\n+in the overall flow of the program.\n++\n+----------------\n+$ export GIT_TRACE2_PERF_BRIEF=1\n+$ export GIT_TRACE2_PERF=~/log.perf\n+$ t/helper/test-tool trace2 100timer 3 1000\n+...\n+$ cat ~/log.perf\n+d0 | main                     | version      |     |           |           |              | ...\n+d0 | main                     | start        |     |  0.001453 |           |              | t/helper/test-tool trace2 100timer 3 1000\n+d0 | main                     | cmd_name     |     |           |           |              | trace2 (trace2)\n+d0 | main                     | exit         |     |  3.003667 |           |              | code:0\n+d0 | main                     | timer        |     |           |           | test         | name:test1 intervals:3 total:3.001686 min:1.000254 max:1.000929\n+d0 | main                     | atexit       |     |  3.003796 |           |              | code:0\n+----------------\n+\n+\n == Future Work\n \n === Relationship to the Existing Trace Api (api-trace.txt)\ndiff --git a/Makefile b/Makefile\nindex cac3452edb9..820649bf62a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1102,6 +1102,7 @@ LIB_OBJS += trace2/tr2_tgt_event.o\n LIB_OBJS += trace2/tr2_tgt_normal.o\n LIB_OBJS += trace2/tr2_tgt_perf.o\n LIB_OBJS += trace2/tr2_tls.o\n+LIB_OBJS += trace2/tr2_tmr.o\n LIB_OBJS += trailer.o\n LIB_OBJS += transport-helper.o\n LIB_OBJS += transport.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex a714130ece7..f951b9e97d7 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -228,6 +228,101 @@ static int ut_010bug_BUG(int argc, const char **argv)\n \tBUG(\"a %s message\", \"BUG\");\n }\n \n+/*\n+ * Single-threaded timer test.  Create several intervals using the\n+ * TEST1 timer.  The test script can verify that an aggregate Trace2\n+ * \"timer\" event is emitted indicating that we started+stopped the\n+ * timer the requested number of times.\n+ */\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay>\";\n+\n+\tint count = 0;\n+\tint delay = 0;\n+\tint k;\n+\n+\tif (argc != 2)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t\tsleep_millisec(delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct ut_101_data {\n+\tint count;\n+\tint delay;\n+};\n+\n+static void *ut_101timer_thread_proc(void *_ut_101_data)\n+{\n+\tstruct ut_101_data *data = _ut_101_data;\n+\tint k;\n+\n+\ttrace2_thread_start(\"ut_101\");\n+\n+\tfor (k = 0; k < data->count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST2);\n+\t\tsleep_millisec(data->delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST2);\n+\t}\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Multi-threaded timer test.  Create several threads that each create\n+ * several intervals using the TEST2 timer.  The test script can verify\n+ * that an individual Trace2 \"th_timer\" events for each thread and an\n+ * aggregate \"timer\" event are generated.\n+ */\n+static int ut_101timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay> <threads>\";\n+\n+\tstruct ut_101_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_101timer_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -248,6 +343,9 @@ static struct unit_test ut_table[] = {\n \t{ ut_008bug,      \"008bug\",    \"\" },\n \t{ ut_009bug_BUG,  \"009bug_BUG\",\"\" },\n \t{ ut_010bug_BUG,  \"010bug_BUG\",\"\" },\n+\n+\t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n+\t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 22d0845544e..5c28424e657 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -173,4 +173,53 @@ test_expect_success 'using global config, perf stream, return code 0' '\n \ttest_cmp expect actual\n '\n \n+# Exercise the stopwatch timers in a loop and confirm that we have\n+# as many start/stop intervals as expected.  We cannot really test the\n+# actual (total, min, max) timer values, so we have to assume that they\n+# are good, but we can verify the interval count.\n+#\n+# The timer \"test/test1\" should only emit a global summary \"timer\" event.\n+# The timer \"test/test2\" should emit per-thread \"th_timer\" events and a\n+# global summary \"timer\" event.\n+\n+have_timer_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 intervals=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} intervals:${intervals}\" &&\n+\n+\tgrep \"${pattern}\" ${file}\n+}\n+\n+test_expect_success 'stopwatch timer test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test1\" 5 times from \"main\".\n+\ttest-tool trace2 100timer 5 10 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test1\" 5 actual\n+'\n+\n+test_expect_success 'stopwatch timer test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test2\" 5 times each in 3 threads.\n+\ttest-tool trace2 101timer 5 10 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_timer_event \"th01:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th02:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th03:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\n+\t# And we should have 15 total uses.\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n+'\n+\n test_done\ndiff --git a/t/t0211/scrub_perf.perl b/t/t0211/scrub_perf.perl\nindex 299999f0f89..7a50bae6463 100644\n--- a/t/t0211/scrub_perf.perl\n+++ b/t/t0211/scrub_perf.perl\n@@ -64,6 +64,12 @@ while (<>) {\n \t    goto SKIP_LINE;\n \t}\n     }\n+    elsif ($tokens[$col_event] =~ m/timer/) {\n+\t# This also captures \"th_timer\" events\n+\t$tokens[$col_rest] =~ s/ total:\\d+\\.\\d*/ total:_T_TOTAL_/;\n+\t$tokens[$col_rest] =~ s/ min:\\d+\\.\\d*/ min:_T_MIN_/;\n+\t$tokens[$col_rest] =~ s/ max:\\d+\\.\\d*/ max:_T_MAX_/;\n+    }\n \n     # t_abs and t_rel are either blank or a float.  Replace the float\n     # with a constant for matching the HEREDOC in the test script.\ndiff --git a/trace2.c b/trace2.c\nindex 165264dc79a..a93cab7c2b7 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -13,6 +13,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static int trace2_enabled;\n \n@@ -83,6 +84,23 @@ static void tr2_tgt_disable_builtins(void)\n \t\ttgt_j->pfn_term();\n }\n \n+/*\n+ * The signature of this function must match the pfn_timer\n+ * method in the targets.  (Think of this is an apply operation\n+ * across the set of active targets.)\n+ */\n+static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n+\t\t\t\t const struct tr2_timer *timer,\n+\t\t\t\t int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_timer)\n+\t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -110,6 +128,26 @@ static void tr2main_atexit_handler(void)\n \t */\n \ttr2tls_pop_unwind_self();\n \n+\t/*\n+\t * Some timers want per-thread details.  If the main thread\n+\t * used one of those timers, emit the details now (before\n+\t * we emit the aggregate timer values).\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data for the main thread to the final\n+\t * totals.  And then emit the final timer values.\n+\t *\n+\t * Technically, we shouldn't need to hold the lock to update\n+\t * and output the final_timer_block (since all other threads\n+\t * should be dead by now), but it doesn't hurt anything.\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_atexit)\n \t\t\ttgt_j->pfn_atexit(us_elapsed_absolute,\n@@ -541,6 +579,21 @@ void trace2_thread_exit_fl(const char *file, int line)\n \ttr2tls_pop_unwind_self();\n \tus_elapsed_thread = tr2tls_region_elasped_self(us_now);\n \n+\t/*\n+\t * Some timers want per-thread details.  If this thread used\n+\t * one of those timers, emit the details now.\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data from the current (non-main) thread\n+\t * to the final totals.  (We'll accumulate data for the main\n+\t * thread later during \"atexit\".)\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_exit_fl)\n \t\t\ttgt_j->pfn_thread_exit_fl(file, line,\n@@ -795,6 +848,28 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...)\n \tva_end(ap);\n }\n \n+void trace2_timer_start(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_start: invalid timer id: %d\", tid);\n+\n+\ttr2_start_timer(tid);\n+}\n+\n+void trace2_timer_stop(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_stop: invalid timer id: %d\", tid);\n+\n+\ttr2_stop_timer(tid);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 74cdb1354f7..7a843ac0518 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -51,6 +51,7 @@ struct json_writer;\n  * [] trace2_region*    -- emit region nesting messages.\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n+ * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n  */\n \n /*\n@@ -485,6 +486,48 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...);\n \n #define trace2_printf(...) trace2_printf_fl(__FILE__, __LINE__, __VA_ARGS__)\n \n+/*\n+ * Define the set of stopwatch timers.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum must also be added to the\n+ * `tr2_timer_metadata[]` in `trace2/tr2_tmr.c`.\n+ */\n+enum trace2_timer_id {\n+\t/*\n+\t * Define two timers for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_TIMER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_TIMER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional timer definitions before here. */\n+\tTRACE2_NUMBER_OF_TIMERS\n+};\n+\n+/*\n+ * Start/Stop the indicated stopwatch timer in the current thread.\n+ *\n+ * The time spent by the current thread between the _start and _stop\n+ * calls will be added to the thread's partial sum for this timer.\n+ *\n+ * Timer events are emitted at thread and program exit.\n+ *\n+ * Note: Since the stopwatch API routines do not generate individual\n+ * events, they do not take (file, line) arguments.  Similarly, the\n+ * category and timer name values are defined at compile-time in the\n+ * timer definitions array, so they are not needed here in the API.\n+ */\n+void trace2_timer_start(enum trace2_timer_id tid);\n+void trace2_timer_stop(enum trace2_timer_id tid);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 65f94e15748..094036964d8 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -4,6 +4,10 @@\n struct child_process;\n struct repository;\n struct json_writer;\n+struct tr2_timer_metadata;\n+struct tr2_timer;\n+\n+#define NS_PER_SEC_D ((double)1000*1000*1000)\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -96,6 +100,10 @@ typedef void(tr2_tgt_evt_printf_va_fl_t)(const char *file, int line,\n \t\t\t\t\t uint64_t us_elapsed_absolute,\n \t\t\t\t\t const char *fmt, va_list ap);\n \n+typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n+\t\t\t\t  const struct tr2_timer *timer,\n+\t\t\t\t  int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -132,6 +140,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_fl_t                   *pfn_data_fl;\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n+\ttr2_tgt_evt_timer_t                     *pfn_timer;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 52f9356c695..dbf6625e1b1 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -9,6 +9,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_event = {\n \t.sysenv_var = TR2_SYSENV_EVENT,\n@@ -617,6 +618,30 @@ static void fn_data_json_fl(const char *file, int line,\n \t}\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"intervals\", timer->interval_count);\n+\tjw_object_double(&jw, \"t_total\", 6, t_total);\n+\tjw_object_double(&jw, \"t_min\", 6, t_min);\n+\tjw_object_double(&jw, \"t_max\", 6, t_max);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -648,4 +673,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 69f80330778..f0582a4bf8a 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -8,6 +8,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_normal = {\n \t.sysenv_var = TR2_SYSENV_NORMAL,\n@@ -329,6 +330,27 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\n+\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -360,4 +382,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_fl = NULL,\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 59ca58f862d..399d1fa78e7 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -10,6 +10,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_perf = {\n \t.sysenv_var = TR2_SYSENV_PERF,\n@@ -555,6 +556,28 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\n+\tstrbuf_addf(&buf_payload, (\"name:%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -586,4 +609,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 3a67532aae4..04900bb4c3a 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -181,3 +181,13 @@ int tr2tls_locked_increment(int *p)\n \n \treturn current_value;\n }\n+\n+void tr2tls_lock(void)\n+{\n+\tpthread_mutex_lock(&tr2tls_mutex);\n+}\n+\n+void tr2tls_unlock(void)\n+{\n+\tpthread_mutex_unlock(&tr2tls_mutex);\n+}\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex e17cc462f87..2322b0d0ef0 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n /*\n  * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n@@ -20,6 +21,9 @@ struct tr2tls_thread_ctx {\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n+\tstruct tr2_timer_block timer_block;\n+\tunsigned int used_any_timer:1;\n+\tunsigned int used_any_per_thread_timer:1;\n };\n \n /*\n@@ -107,4 +111,10 @@ int tr2tls_locked_increment(int *p);\n  */\n void tr2tls_start_process_clock(void);\n \n+/*\n+ * Explicitly lock/unlock our mutex.\n+ */\n+void tr2tls_lock(void);\n+void tr2tls_unlock(void);\n+\n #endif /* TR2_TLS_H */\ndiff --git a/trace2/tr2_tmr.c b/trace2/tr2_tmr.c\nnew file mode 100644\nindex 00000000000..786762dfd26\n--- /dev/null\n+++ b/trace2/tr2_tmr.c\n@@ -0,0 +1,182 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n+\n+#define MY_MAX(a, b) ((a) > (b) ? (a) : (b))\n+#define MY_MIN(a, b) ((a) < (b) ? (a) : (b))\n+\n+/*\n+ * A global timer block to aggregate values from the partial sums from\n+ * each thread.\n+ */\n+static struct tr2_timer_block final_timer_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each stopwatch timer.\n+ *\n+ * This array must match \"enum trace2_timer_id\" and the values\n+ * in \"struct tr2_timer_block.timer[*]\".\n+ */\n+static struct tr2_timer_metadata tr2_timer_metadata[TRACE2_NUMBER_OF_TIMERS] = {\n+\t[TRACE2_TIMER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_TIMER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_start_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\tt->recursion_count++;\n+\tif (t->recursion_count > 1)\n+\t\treturn; /* ignore recursive starts */\n+\n+\tt->start_ns = getnanotime();\n+}\n+\n+void tr2_stop_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\tuint64_t ns_now;\n+\tuint64_t ns_interval;\n+\n+\tassert(t->recursion_count > 0);\n+\n+\tt->recursion_count--;\n+\tif (t->recursion_count)\n+\t\treturn; /* still in recursive call(s) */\n+\n+\tns_now = getnanotime();\n+\tns_interval = ns_now - t->start_ns;\n+\n+\tt->total_ns += ns_interval;\n+\n+\t/*\n+\t * min_ns was initialized to zero (in the xcalloc()) rather\n+\t * than UINT_MAX when the block of timers was allocated,\n+\t * so we should always set both the min_ns and max_ns values\n+\t * the first time that the timer is used.\n+\t */\n+\tif (!t->interval_count) {\n+\t\tt->min_ns = ns_interval;\n+\t\tt->max_ns = ns_interval;\n+\t} else {\n+\t\tt->min_ns = MY_MIN(ns_interval, t->min_ns);\n+\t\tt->max_ns = MY_MAX(ns_interval, t->max_ns);\n+\t}\n+\n+\tt->interval_count++;\n+\n+\tctx->used_any_timer = 1;\n+\tif (tr2_timer_metadata[tid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_timer = 1;\n+}\n+\n+void tr2_update_final_timers(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++) {\n+\t\tstruct tr2_timer *t_final = &final_timer_block.timer[tid];\n+\t\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\t\tif (t->recursion_count) {\n+\t\t\t/*\n+\t\t\t * The current thread is exiting with\n+\t\t\t * timer[tid] still running.\n+\t\t\t *\n+\t\t\t * Technically, this is a bug, but I'm going\n+\t\t\t * to ignore it.\n+\t\t\t *\n+\t\t\t * I don't think it is worth calling die()\n+\t\t\t * for.  I don't think it is worth killing the\n+\t\t\t * process for this bookkeeping error.  We\n+\t\t\t * might want to call warning(), but I'm going\n+\t\t\t * to wait on that.\n+\t\t\t *\n+\t\t\t * The downside here is that total_ns won't\n+\t\t\t * include the current open interval (now -\n+\t\t\t * start_ns).  I can live with that.\n+\t\t\t */\n+\t\t}\n+\n+\t\tif (!t->interval_count)\n+\t\t\tcontinue; /* this timer was not used by this thread */\n+\n+\t\tt_final->total_ns += t->total_ns;\n+\n+\t\t/*\n+\t\t * final_timer_block.timer[tid].min_ns was initialized to\n+\t\t * was initialized to zero rather than UINT_MAX, so we should\n+\t\t * always set both the min_ns and max_ns values the first time\n+\t\t * that we add a partial sum into it.\n+\t\t */\n+\t\tif (!t_final->interval_count) {\n+\t\t\tt_final->min_ns = t->min_ns;\n+\t\t\tt_final->max_ns = t->max_ns;\n+\t\t} else {\n+\t\t\tt_final->min_ns = MY_MIN(t_final->min_ns, t->min_ns);\n+\t\t\tt_final->max_ns = MY_MAX(t_final->max_ns, t->max_ns);\n+\t\t}\n+\n+\t\tt_final->interval_count += t->interval_count;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_per_thread_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * For each timer, if the timer wants per-thread events and\n+\t * this thread used it, emit it.\n+\t */\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (tr2_timer_metadata[tid].want_per_thread_events &&\n+\t\t    ctx->timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &ctx->timer_block.timer[tid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tenum trace2_timer_id tid;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (final_timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &final_timer_block.timer[tid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_tmr.h b/trace2/tr2_tmr.h\nnew file mode 100644\nindex 00000000000..d5753576134\n--- /dev/null\n+++ b/trace2/tr2_tmr.h\n@@ -0,0 +1,140 @@\n+#ifndef TR2_TMR_H\n+#define TR2_TMR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow \"stopwatch\" timers.\n+ *\n+ * Timers can be used to measure \"interesting\" activity that does not\n+ * fit the \"region\" model, such as code called from many different\n+ * regions (like zlib) and/or where data for individual calls are not\n+ * interesting or are too numerous to be efficiently logged.\n+ *\n+ * Timer values are accumulated during program execution and emitted\n+ * to the Trace2 logs at program exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * timers and timer ids using a \"timer block\" array in thread-local\n+ * storage.  This gives us constant time access to each timer within\n+ * each thread, since we want start/stop operations to be as fast as\n+ * possible.  This lets us avoid the complexities of dynamically\n+ * allocating a timer on the first use by a thread and/or possibly\n+ * sharing that timer definition with other concurrent threads.\n+ * However, this does require that we define time the set of timers at\n+ * compile time.\n+ *\n+ * Each thread uses the timer block in its thread-local storage to\n+ * compute partial sums for each timer (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Using this \"timer block\" model costs ~48 bytes per timer per thread\n+ * (we have about six uint64 fields per timer).  This does increase\n+ * the size of the thread-local storage block, but it is allocated (at\n+ * thread create time) and not on the thread stack, so I'm not worried\n+ * about the size.\n+ *\n+ * Partial sums for each timer are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each timer are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"timer metadata\" table contains the \"category\" and \"name\"\n+ * fields for each timer.  This eliminates the need to include those\n+ * args in the various timer APIs.\n+ */\n+\n+/*\n+ * The definition of an individual timer and used by an individual\n+ * thread.\n+ */\n+struct tr2_timer {\n+\t/*\n+\t * Total elapsed time for this timer in this thread in nanoseconds.\n+\t */\n+\tuint64_t total_ns;\n+\n+\t/*\n+\t * The maximum and minimum interval values observed for this\n+\t * timer in this thread.\n+\t */\n+\tuint64_t min_ns;\n+\tuint64_t max_ns;\n+\n+\t/*\n+\t * The value of the clock when this timer was started in this\n+\t * thread.  (Undefined when the timer is not active in this\n+\t * thread.)\n+\t */\n+\tuint64_t start_ns;\n+\n+\t/*\n+\t * Number of times that this timer has been started and stopped\n+\t * in this thread.  (Recursive starts are ignored.)\n+\t */\n+\tuint64_t interval_count;\n+\n+\t/*\n+\t * Number of nested starts on the stack in this thread.  (We\n+\t * ignore recursive starts and use this to track the recursive\n+\t * calls.)\n+\t */\n+\tunsigned int recursion_count;\n+};\n+\n+/*\n+ * Metadata for a timer.\n+ */\n+struct tr2_timer_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this timer\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed-size block of timers to insert into\n+ * thread-local storage.  This wrapper is used to avoid quirks\n+ * of C and the usual need to pass an array size argument.\n+ */\n+struct tr2_timer_block {\n+\tstruct tr2_timer timer[TRACE2_NUMBER_OF_TIMERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to actually start/stop an\n+ * individual timer in the current thread.\n+ */\n+void tr2_start_timer(enum trace2_timer_id tid);\n+void tr2_stop_timer(enum trace2_timer_id tid);\n+\n+/*\n+ * Add the current thread's timer data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_timers(void);\n+\n+/*\n+ * Emit per-thread timer data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+/*\n+ * Emit global total timer values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+#endif /* TR2_TMR_H */\n-- \ngitgitgadget\n\n"},{"id":"465322","messageId":"5cd8bdde884e312623da94b61e9fda57c0b3a980.1666290489.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] trace2: add global counter mechanism","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-20T18:28:09Z","receivedAt":"2022-10-20T18:28:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd global counters mechanism to Trace2.\n\nThe Trace2 counters mechanism adds the ability to create a set of\nglobal counter variables and an API to increment them efficiently.\nCounters can optionally report per-thread usage in addition to the sum\nacross all threads.\n\nCounter events are emitted to the Trace2 logs when a thread exits and\nat process exit.\n\nCounters are an alternative to `data` and `data_json` events.\n\nCounters are useful when you want to measure something across the life\nof the process, when you don't want per-measurement events for\nperformance reasons, when the data does not fit conveniently within a\nregion, or when your control flow does not easily let you write the\nfinal total.  For example, you might use this to report the number of\ncalls to unzip() or the number of de-delta steps during a checkout.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  31 ++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  89 +++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  46 +++++++++++\n trace2.c                               |  52 +++++++++++--\n trace2.h                               |  37 +++++++++\n trace2/tr2_ctr.c                       | 101 ++++++++++++++++++++++++\n trace2/tr2_ctr.h                       | 104 +++++++++++++++++++++++++\n trace2/tr2_tgt.h                       |   7 ++\n trace2/tr2_tgt_event.c                 |  19 +++++\n trace2/tr2_tgt_normal.c                |  16 ++++\n trace2/tr2_tgt_perf.c                  |  17 ++++\n trace2/tr2_tls.h                       |   4 +\n 13 files changed, 517 insertions(+), 7 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 75ce6f45603..de5fc250595 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -805,6 +805,37 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_counter\"`::\n+\tThis event logs the value of a counter variable in a thread.\n+\tThis event is generated when a thread exits for counters that\n+\trequested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+`\"counter\"`::\n+\tThis event logs the value of a counter variable across all threads.\n+\tThis event is generated when the process exits.  The total value\n+\treported here is the sum across all threads.\n++\n+------------\n+{\n+\t\"event\":\"counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\ndiff --git a/Makefile b/Makefile\nindex 820649bf62a..29ab417ca3a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += trace.o\n LIB_OBJS += trace2.o\n LIB_OBJS += trace2/tr2_cfg.o\n LIB_OBJS += trace2/tr2_cmd_name.o\n+LIB_OBJS += trace2/tr2_ctr.o\n LIB_OBJS += trace2/tr2_dst.o\n LIB_OBJS += trace2/tr2_sid.o\n LIB_OBJS += trace2/tr2_sysenv.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex f951b9e97d7..1b092c60714 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -323,6 +323,92 @@ static int ut_101timer(int argc, const char **argv)\n \treturn 0;\n }\n \n+/*\n+ * Single-threaded counter test.  Add several values to the TEST1 counter.\n+ * The test script can verify that the final sum is reported in the \"counter\"\n+ * event.\n+ */\n+static int ut_200counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> [<v2> [...]]\";\n+\tint value;\n+\tint k;\n+\n+\tif (argc < 1)\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < argc; k++) {\n+\t\tif (get_i(&value, argv[k]))\n+\t\t\tdie(\"invalid value[%s] -- %s\",\n+\t\t\t    argv[k], usage_error);\n+\t\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST1, value);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Multi-threaded counter test.  Create seveal threads that each increment\n+ * the TEST2 global counter.  The test script can verify that an individual\n+ * \"th_counter\" event is generated with a partial sum for each thread and\n+ * that a final aggregate \"counter\" event is generated.\n+ */\n+\n+struct ut_201_data {\n+\tint v1;\n+\tint v2;\n+};\n+\n+static void *ut_201counter_thread_proc(void *_ut_201_data)\n+{\n+\tstruct ut_201_data *data = _ut_201_data;\n+\n+\ttrace2_thread_start(\"ut_201\");\n+\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v1);\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v2);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int ut_201counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> <v2> <threads>\";\n+\n+\tstruct ut_201_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v1, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v2, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_201counter_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -346,6 +432,9 @@ static struct unit_test ut_table[] = {\n \n \t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n \t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n+\n+\t{ ut_200counter,  \"200counter\", \"<v1> [<v2> [<v3> [...]]]\" },\n+\t{ ut_201counter,  \"201counter\", \"<v1> <v2> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 5c28424e657..0b3436e8cac 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -222,4 +222,50 @@ test_expect_success 'stopwatch timer test/test2' '\n \thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n '\n \n+# Exercise the global counters and confirm that we get the expected values.\n+#\n+# The counter \"test/test1\" should only emit a global summary \"counter\" event.\n+# The counter \"test/test2\" could emit per-thread \"th_counter\" events and a\n+# global summary \"counter\" event.\n+\n+have_counter_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 value=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} value:${value}\" &&\n+\n+\tgrep \"${patern}\" ${file}\n+}\n+\n+test_expect_success 'global counter test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the counter \"test1\" and add n integers.\n+\ttest-tool trace2 200counter 1 2 3 4 5 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test1\" 15 actual\n+'\n+\n+test_expect_success 'global counter test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Add 2 integers to the counter \"test2\" in each of 3 threads.\n+\ttest-tool trace2 201counter 7 13 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_counter_event \"th01:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th02:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th03:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\n+\t# And we should have a single event with the total across all threads.\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test2\" 60 actual\n+'\n+\n test_done\ndiff --git a/trace2.c b/trace2.c\nindex a93cab7c2b7..279bddf53b4 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -8,6 +8,7 @@\n #include \"version.h\"\n #include \"trace2/tr2_cfg.h\"\n #include \"trace2/tr2_cmd_name.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_dst.h\"\n #include \"trace2/tr2_sid.h\"\n #include \"trace2/tr2_sysenv.h\"\n@@ -101,6 +102,22 @@ static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n \t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n }\n \n+/*\n+ * The signature of this function must match the pfn_counter\n+ * method in the targets.\n+ */\n+static void tr2_tgt_emit_a_counter(const struct tr2_counter_metadata *meta,\n+\t\t\t\t   const struct tr2_counter *counter,\n+\t\t\t\t   int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_counter)\n+\t\t\ttgt_j->pfn_counter(meta, counter, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -132,20 +149,26 @@ static void tr2main_atexit_handler(void)\n \t * Some timers want per-thread details.  If the main thread\n \t * used one of those timers, emit the details now (before\n \t * we emit the aggregate timer values).\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data for the main thread to the final\n-\t * totals.  And then emit the final timer values.\n+\t * Add stopwatch timer and counter data for the main thread to\n+\t * the final totals.  And then emit the final values.\n \t *\n \t * Technically, we shouldn't need to hold the lock to update\n-\t * and output the final_timer_block (since all other threads\n-\t * should be dead by now), but it doesn't hurt anything.\n+\t * and output the final_timer_block and final_counter_block\n+\t * (since all other threads should be dead by now), but it\n+\t * doesn't hurt anything.\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_final_counters(tr2_tgt_emit_a_counter);\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -582,16 +605,20 @@ void trace2_thread_exit_fl(const char *file, int line)\n \t/*\n \t * Some timers want per-thread details.  If this thread used\n \t * one of those timers, emit the details now.\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data from the current (non-main) thread\n-\t * to the final totals.  (We'll accumulate data for the main\n-\t * thread later during \"atexit\".)\n+\t * Add stopwatch timer and counter data from the current\n+\t * (non-main) thread to the final totals.  (We'll accumulate\n+\t * data for the main thread later during \"atexit\".)\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -870,6 +897,17 @@ void trace2_timer_stop(enum trace2_timer_id tid)\n \ttr2_stop_timer(tid);\n }\n \n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (cid < 0 || cid >= TRACE2_NUMBER_OF_COUNTERS)\n+\t\tBUG(\"trace2_counter_add: invalid counter id: %d\", cid);\n+\n+\ttr2_counter_increment(cid, value);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 7a843ac0518..4ced30c0db3 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -52,6 +52,7 @@ struct json_writer;\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n  * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n+ * [] trace2_counter*   -- global counters (messages are deferred).\n  */\n \n /*\n@@ -528,6 +529,42 @@ enum trace2_timer_id {\n void trace2_timer_start(enum trace2_timer_id tid);\n void trace2_timer_stop(enum trace2_timer_id tid);\n \n+/*\n+ * Define the set of global counters.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum be also be added to the\n+ * `tr2_counter_metadata[]` in `trace2/tr2_tr2_ctr.c`.\n+ */\n+enum trace2_counter_id {\n+\t/*\n+\t * Define two counters for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional counter definitions before here. */\n+\tTRACE2_NUMBER_OF_COUNTERS\n+};\n+\n+/*\n+ * Increase the named global counter by value.\n+ *\n+ * Note that this adds `value` to the current thread's partial sum for\n+ * this counter (without locking) and that the complete sum is not\n+ * available until all threads have exited, so it does not return the\n+ * new value of the counter.\n+ */\n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nnew file mode 100644\nindex 00000000000..483ca7c308f\n--- /dev/null\n+++ b/trace2/tr2_ctr.c\n@@ -0,0 +1,101 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_ctr.h\"\n+\n+/*\n+ * A global counter block to aggregrate values from the partial sums\n+ * from each thread.\n+ */\n+static struct tr2_counter_block final_counter_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each global counter.\n+ *\n+ * This array must match the \"enum trace2_counter_id\" and the values\n+ * in \"struct tr2_counter_block.counter[*]\".\n+ */\n+static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTERS] = {\n+\t[TRACE2_COUNTER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_COUNTER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\tc->value += value;\n+\n+\tctx->used_any_counter = 1;\n+\tif (tr2_counter_metadata[cid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_counter = 1;\n+}\n+\n+void tr2_update_final_counters(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++) {\n+\t\tstruct tr2_counter *c_final = &final_counter_block.counter[cid];\n+\t\tconst struct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\t\tc_final->value += c->value;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_per_thread_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * For each counter, if the counter wants per-thread events\n+\t * and this thread used it (the value is non-zero), emit it.\n+\t */\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (tr2_counter_metadata[cid].want_per_thread_events &&\n+\t\t    ctx->counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &ctx->counter_block.counter[cid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tenum trace2_counter_id cid;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (final_counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &final_counter_block.counter[cid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_ctr.h b/trace2/tr2_ctr.h\nnew file mode 100644\nindex 00000000000..a2267ee9901\n--- /dev/null\n+++ b/trace2/tr2_ctr.h\n@@ -0,0 +1,104 @@\n+#ifndef TR2_CTR_H\n+#define TR2_CTR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow global \"counters\".\n+ *\n+ * Counters can be used count interesting activity that does not fit\n+ * the \"region and data\" model, such as code called from many\n+ * different regions and/or where you want to count a number of items,\n+ * but don't have control of when the last item will be processed,\n+ * such as counter the number of calls to `lstat()`.\n+ *\n+ * Counters differ from Trace2 \"data\" events.  Data events are emitted\n+ * immediately and are appropriate for documenting loop counters at\n+ * the end of a region, for example.  Counter values are accumulated\n+ * during the program and final counter values are emitted at program\n+ * exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * counters and counter ids using a fixed size \"counter block\" array\n+ * in thread-local storage.  This gives us constant time, lock-free\n+ * access to each counter within each thread.  This lets us avoid the\n+ * complexities of dynamically allocating a counter and sharing that\n+ * definition with other threads.\n+ *\n+ * Each thread uses the counter block in its thread-local storage to\n+ * increment partial sums for each counter (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Partial sums for each counter are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each counter are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"counter metadata\" table contains the \"category\" and\n+ * \"name\" fields for each counter.  This eliminates the need to\n+ * include those args in the various counter APIs.\n+ */\n+\n+/*\n+ * The definition of an individual counter as used by an individual\n+ * thread (and later in aggregation).\n+ */\n+struct tr2_counter {\n+\tuint64_t value;\n+};\n+\n+/*\n+ * Metadata for a counter.\n+ */\n+struct tr2_counter_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this counter\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed block of counters to insert into thread-local\n+ * storage.  This wrapper is used to avoid quirks of C and the usual\n+ * need to pass an array size argument.\n+ */\n+struct tr2_counter_block {\n+\tstruct tr2_counter counter[TRACE2_NUMBER_OF_COUNTERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to increment a counter for the\n+ * current thread.\n+ */\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value);\n+\n+/*\n+ * Add the current thread's counter data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_counters(void);\n+\n+/*\n+ * Emit per-thread counter data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+/*\n+ * Emit global counter values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+#endif /* TR2_CTR_H */\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 094036964d8..95f4c754726 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -6,6 +6,8 @@ struct repository;\n struct json_writer;\n struct tr2_timer_metadata;\n struct tr2_timer;\n+struct tr2_counter_metadata;\n+struct tr2_counter;\n \n #define NS_PER_SEC_D ((double)1000*1000*1000)\n \n@@ -104,6 +106,10 @@ typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n \t\t\t\t  const struct tr2_timer *timer,\n \t\t\t\t  int is_final_data);\n \n+typedef void(tr2_tgt_evt_counter_t)(const struct tr2_counter_metadata *meta,\n+\t\t\t\t    const struct tr2_counter *counter,\n+\t\t\t\t    int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -141,6 +147,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n \ttr2_tgt_evt_timer_t                     *pfn_timer;\n+\ttr2_tgt_evt_counter_t                   *pfn_counter;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex dbf6625e1b1..981863a6602 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -642,6 +642,24 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tjw_release(&jw);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"count\", counter->value);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -674,4 +692,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex f0582a4bf8a..def18674e88 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -351,6 +351,21 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"%s %s/%s value:%\"PRIu64,\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    counter->value);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -383,4 +398,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 399d1fa78e7..db94b2ef47e 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -578,6 +578,22 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"name:%s value:%\"PRIu64,\n+\t\t    meta->name,\n+\t\t    counter->value);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -610,4 +626,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 2322b0d0ef0..289b62d0721 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_tmr.h\"\n \n /*\n@@ -22,8 +23,11 @@ struct tr2tls_thread_ctx {\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n \tstruct tr2_timer_block timer_block;\n+\tstruct tr2_counter_block counter_block;\n \tunsigned int used_any_timer:1;\n \tunsigned int used_any_per_thread_timer:1;\n+\tunsigned int used_any_counter:1;\n+\tunsigned int used_any_per_thread_counter:1;\n };\n \n /*\n-- \ngitgitgadget\n"},{"id":"465327","messageId":"221020.86y1tafhjo.gmgdl@evledraar.gmail.com","threadId":"58564","inReplyTo":"8cb206b76323e14d8e07f6cfb5aa482a47eb54c5.1666290489.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/8] trace2: improve thread-name documentation in the thread-context","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-10-20T18:57:39Z","receivedAt":"2022-10-20T18:58:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 20 2022, Jeff Hostetler via GitGitGadget wrote:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Improve the documentation of the tr2tls_thread_ctx.thread_name field\n> and its relation to the tr2tls_thread_ctx.thread_id field.\n\nGood to see this split off, thanks!\n\n> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n> ---\n>  trace2/tr2_tls.h | 15 +++++++++------\n>  1 file changed, 9 insertions(+), 6 deletions(-)\n>\n> diff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\n> index d4e725f430b..7d1f03a2ea6 100644\n> --- a/trace2/tr2_tls.h\n> +++ b/trace2/tr2_tls.h\n> @@ -25,12 +25,15 @@ struct tr2tls_thread_ctx {\n>  /*\n>   * Create thread-local storage for the current thread.\n>   *\n> - * We assume the first thread is \"main\".  Other threads are given\n> - * non-zero thread-ids to help distinguish messages from concurrent\n> - * threads.\n> - *\n> - * Truncate the thread name if necessary to help with column alignment\n> - * in printf-style messages.\n> + * The first thread in the process will have:\n> + *     { .thread_id=0, .thread_name=\"main\" }\n> + * Subsequent threads are given a non-zero thread_id and a thread_name\n> + * constructed from the id and a thread base name (which is usually just\n> + * the name of the thread-proc function).  For example:\n> + *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n\nI think the example is missing a \":\" after the \"th10\", i.e. it should be\n\"th10:fsm-listen\" per the code in 6/8:\n\n\tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n        [...]\n"},{"id":"465330","messageId":"728716c7-15e2-00f6-e249-e8bb31142d34@jeffhostetler.com","threadId":"58564","inReplyTo":"221020.86y1tafhjo.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v3 5/8] trace2: improve thread-name documentation in the thread-context","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-20T20:15:04Z","receivedAt":"2022-10-20T20:15:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/20/22 2:57 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Oct 20 2022, Jeff Hostetler via GitGitGadget wrote:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Improve the documentation of the tr2tls_thread_ctx.thread_name field\n>> and its relation to the tr2tls_thread_ctx.thread_id field.\n> \n> Good to see this split off, thanks!\n> \n[...]\n>> + * the name of the thread-proc function).  For example:\n>> + *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n> \n> I think the example is missing a \":\" after the \"th10\", i.e. it should be\n> \"th10:fsm-listen\" per the code in 6/8:\n> \n> \tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n>          [...]\n> \n\noops.  :-)\ngood catch.\n\ni'll fix up and resend, but will wait a bit for any other comments.\n\nJeff\n"},{"id":"465340","messageId":"xmqq7d0us0o2.fsf@gitster.g","threadId":"58564","inReplyTo":"8e701109976777ad8fae1e0cd3908bb11a1fcf93.1666290489.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/8] trace2: add stopwatch timers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-20T20:25:01Z","receivedAt":"2022-10-20T20:25:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> +#define NS_PER_SEC_D ((double)1000*1000*1000)\n> ...\n> +\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n> +\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n> +\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n\nHmph, it certainly is an improvement compared to the previous round,\nbut was there a reason why we did not want a more concise\n\n\t#define NS_TO_SECONDS(ns) ((double)(ns) / (1000*1000*1000.))\n\n\tdouble t_total = NS_TO_SECONDS(timer->total_ns);\n\tdouble t_min = NS_TO_SECONDS(timer->min_ns);\n\tdouble t_max = NS_TO_SECONDS(timer->max_ns);\n\nthat does not need to repeat (double) all over?\n\nNot worth a reroll by itself.  Just wanted to know the reasoning\nbehind it, as I suspect I am missing the reason why it is good to\nexplicitly casting with (double) in some places; the above does not\nlook like one, though.\n\nThanks.\n\n"},{"id":"465346","messageId":"b7860e10-b174-2fb2-53eb-568686a961c4@jeffhostetler.com","threadId":"58564","inReplyTo":"xmqq7d0us0o2.fsf@gitster.g","subject":"Re: [PATCH v3 7/8] trace2: add stopwatch timers","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-20T20:52:51Z","receivedAt":"2022-10-20T20:52:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/20/22 4:25 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n> \n>> +#define NS_PER_SEC_D ((double)1000*1000*1000)\n>> ...\n>> +\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n>> +\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n>> +\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n> \n> Hmph, it certainly is an improvement compared to the previous round,\n> but was there a reason why we did not want a more concise\n> \n> \t#define NS_TO_SECONDS(ns) ((double)(ns) / (1000*1000*1000.))\n> \n> \tdouble t_total = NS_TO_SECONDS(timer->total_ns);\n> \tdouble t_min = NS_TO_SECONDS(timer->min_ns);\n> \tdouble t_max = NS_TO_SECONDS(timer->max_ns);\n> \n> that does not need to repeat (double) all over?\n> \n> Not worth a reroll by itself.  Just wanted to know the reasoning\n> behind it, as I suspect I am missing the reason why it is good to\n> explicitly casting with (double) in some places; the above does not\n> look like one, though.\n> \n> Thanks.\n>\n\num, it never occurred to me to make it a macro with an arg.\ni just did a search/replace on the inline constant.\n\nyou're right though. your version is much shorter.\n\ni'll reroll tomorrow with the typo that AEvar found.\n\nThanks\nJeff\n\n\n"},{"id":"465347","messageId":"xmqqy1taqkok.fsf@gitster.g","threadId":"58564","inReplyTo":"b7860e10-b174-2fb2-53eb-568686a961c4@jeffhostetler.com","subject":"Re: [PATCH v3 7/8] trace2: add stopwatch timers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-20T20:55:39Z","receivedAt":"2022-10-20T20:55:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n> um, it never occurred to me to make it a macro with an arg.\n\nHeh, that was what you responded with \"good point\" 6 hours ago ;-)\n\nhttps://lore.kernel.org/git/aeb07c4f-f3f2-4965-6b6b-3ba3b10b2103@jeffhostetler.com/\n\n> i'll reroll tomorrow with the typo that AEvar found.\n\nThanks.  Looking forward to.\n"},{"id":"465516","messageId":"70d683e6-8a91-2752-360d-c13f7ab02604@jeffhostetler.com","threadId":"58564","inReplyTo":"xmqqy1taqkok.fsf@gitster.g","subject":"Re: [PATCH v3 7/8] trace2: add stopwatch timers","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-10-21T21:51:57Z","receivedAt":"2022-10-21T21:52:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/20/22 4:55 PM, Junio C Hamano wrote:\n> Jeff Hostetler <git@jeffhostetler.com> writes:\n> \n>> um, it never occurred to me to make it a macro with an arg.\n> \n> Heh, that was what you responded with \"good point\" 6 hours ago ;-)\n\nd'oh.  6 hours was way too many meetings ago.... :-)\n\nJeff\n"},{"id":"465612","messageId":"6e7e4f3187e2fbbbb54bb1cf5793bf6e981a5a94.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:00Z","receivedAt":"2022-10-24T15:06:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nUse \"size_t\" rather than \"int\" for the \"alloc\" and \"nr_open_regions\"\nfields in the \"tr2tls_thread_ctx\".  These are used by ALLOC_GROW().\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex b1e327a928e..a90bd639d48 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -11,8 +11,8 @@\n struct tr2tls_thread_ctx {\n \tstruct strbuf thread_name;\n \tuint64_t *array_us_start;\n-\tint alloc;\n-\tint nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n+\tsize_t alloc;\n+\tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n };\n \n-- \ngitgitgadget\n\n"},{"id":"465613","messageId":"804dab9e1a7fa1cea9355bac92ada16332f1194e.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] api-trace2.txt: elminate section describing the public trace2 API","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:02Z","receivedAt":"2022-10-24T15:07:01Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nEliminate the mostly obsolete `Public API` sub-section from the\n`Trace2 API` section in the documentation.  Strengthen the referral\nto `trace2.h`.\n\nMost of the technical information in this sub-section was moved to\n`trace2.h` in 6c51cb525d (trace2: move doc to trace2.h, 2019-11-17) to\nbe adjacent to the function prototypes.  The remaining text wasn't\nthat useful by itself.\n\nFurthermore, the text would need a bit of overhaul to add routines\nthat do not immediately generate a message, such as stopwatch timers.\nSo it seemed simpler to just get rid of it.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt | 61 +++-----------------------\n 1 file changed, 7 insertions(+), 54 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 431d424f9d5..9d43909d068 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -148,20 +148,18 @@ filename collisions).\n \n == Trace2 API\n \n-All public Trace2 functions and macros are defined in `trace2.h` and\n-`trace2.c`.  All public symbols are prefixed with `trace2_`.\n+The Trace2 public API is defined and documented in `trace2.h`; refer to it for\n+more information.  All public functions and macros are prefixed\n+with `trace2_` and are implemented in `trace2.c`.\n \n There are no public Trace2 data structures.\n \n The Trace2 code also defines a set of private functions and data types\n in the `trace2/` directory.  These symbols are prefixed with `tr2_`\n-and should only be used by functions in `trace2.c`.\n+and should only be used by functions in `trace2.c` (or other private\n+source files in `trace2/`).\n \n-== Conventions for Public Functions and Macros\n-\n-The functions defined by the Trace2 API are declared and documented\n-in `trace2.h`.  It defines the API functions and wrapper macros for\n-Trace2.\n+=== Conventions for Public Functions and Macros\n \n Some functions have a `_fl()` suffix to indicate that they take `file`\n and `line-number` arguments.\n@@ -172,52 +170,7 @@ take a `va_list` argument.\n Some functions have a `_printf_fl()` suffix to indicate that they also\n take a `printf()` style format with a variable number of arguments.\n \n-There are CPP wrapper macros and `#ifdef`s to hide most of these details.\n-See `trace2.h` for more details.  The following discussion will only\n-describe the simplified forms.\n-\n-== Public API\n-\n-All Trace2 API functions send a message to all of the active\n-Trace2 Targets.  This section describes the set of available\n-messages.\n-\n-It helps to divide these functions into groups for discussion\n-purposes.\n-\n-=== Basic Command Messages\n-\n-These are concerned with the lifetime of the overall git process.\n-e.g: `void trace2_initialize_clock()`, `void trace2_initialize()`,\n-`int trace2_is_enabled()`, `void trace2_cmd_start(int argc, const char **argv)`.\n-\n-=== Command Detail Messages\n-\n-These are concerned with describing the specific Git command\n-after the command line, config, and environment are inspected.\n-e.g: `void trace2_cmd_name(const char *name)`,\n-`void trace2_cmd_mode(const char *mode)`.\n-\n-=== Child Process Messages\n-\n-These are concerned with the various spawned child processes,\n-including shell scripts, git commands, editors, pagers, and hooks.\n-\n-e.g: `void trace2_child_start(struct child_process *cmd)`.\n-\n-=== Git Thread Messages\n-\n-These messages are concerned with Git thread usage.\n-\n-e.g: `void trace2_thread_start(const char *thread_name)`.\n-\n-=== Region and Data Messages\n-\n-These are concerned with recording performance data\n-over regions or spans of code. e.g:\n-`void trace2_region_enter(const char *category, const char *label, const struct repository *repo)`.\n-\n-Refer to trace2.h for details about all trace2 functions.\n+CPP wrapper macros are defined to hide most of these details.\n \n == Trace2 Target Formats\n \n-- \ngitgitgadget\n\n"},{"id":"465614","messageId":"a10c1bd96bba8dec299d6dda740cf2a96fa374a4.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] trace2: add stopwatch timers","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:06Z","receivedAt":"2022-10-24T15:14:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd stopwatch timer mechanism to Trace2.\n\nTimers are an alternative to Trace2 Regions.  Regions are useful for\nmeasuring the time spent in various computation phases, such as the\ntime to read the index, time to scan for unstaged files, time to scan\nfor untracked files, and etc.\n\nHowever, regions are not appropriate in all places.  For example,\nduring a checkout, it would be very inefficient to use regions to\nmeasure the total time spent inflating objects from the ODB from\nacross the entire lifetime of the process; a per-unzip() region would\nflood the output and significantly slow the command; and some form of\npost-processing would be requried to compute the time spent in unzip().\n\nTimers can be used to measure a series of timer intervals and emit\na single summary event (at thread and/or process exit).\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  90 ++++++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  98 +++++++++++++\n t/t0211-trace2-perf.sh                 |  49 +++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               |  75 ++++++++++\n trace2.h                               |  43 ++++++\n trace2/tr2_tgt.h                       |   9 ++\n trace2/tr2_tgt_event.c                 |  26 ++++\n trace2/tr2_tgt_normal.c                |  23 ++++\n trace2/tr2_tgt_perf.c                  |  24 ++++\n trace2/tr2_tls.c                       |  10 ++\n trace2/tr2_tls.h                       |  10 ++\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 +++++++++++++++++++\n 15 files changed, 786 insertions(+)\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 9d43909d068..75ce6f45603 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -769,6 +769,42 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning in the thread.  This event is generated when a thread\n+\texits for timers that requested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n+`\"timer\"`::\n+\tThis event logs the amount of time that a stopwatch timer was\n+\trunning aggregated across all threads.  This event is generated\n+\twhen the process exits.\n++\n+------------\n+{\n+\t\"event\":\"timer\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_timer\",\n+\t\"intervals\":5,         # number of time it was started/stopped\n+\t\"t_total\":0.052741,    # total time in seconds it was running\n+\t\"t_min\":0.010061,      # shortest interval\n+\t\"t_max\":0.011648       # longest interval\n+}\n+------------\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\n@@ -1200,6 +1236,60 @@ d0 | main                     | data         | r0  |  0.002126 |  0.002126 | fsy\n d0 | main                     | exit         |     |  0.000470 |           |              | code:0\n d0 | main                     | atexit       |     |  0.000477 |           |              | code:0\n ----------------\n+\n+Stopwatch Timer Events::\n+\n+\tMeasure the time spent in a function call or span of code\n+\tthat might be called from many places within the code\n+\tthroughout the life of the process.\n++\n+----------------\n+static void expensive_function(void)\n+{\n+\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t...\n+\tsleep_millisec(1000); // Do something expensive\n+\t...\n+\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+}\n+\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\t...\n+\n+\texpensive_function();\n+\n+\t// Do something else 1...\n+\n+\texpensive_function();\n+\n+\t// Do something else 2...\n+\n+\texpensive_function();\n+\n+\treturn 0;\n+}\n+----------------\n++\n+In this example, we measure the total time spent in\n+`expensive_function()` regardless of when it is called\n+in the overall flow of the program.\n++\n+----------------\n+$ export GIT_TRACE2_PERF_BRIEF=1\n+$ export GIT_TRACE2_PERF=~/log.perf\n+$ t/helper/test-tool trace2 100timer 3 1000\n+...\n+$ cat ~/log.perf\n+d0 | main                     | version      |     |           |           |              | ...\n+d0 | main                     | start        |     |  0.001453 |           |              | t/helper/test-tool trace2 100timer 3 1000\n+d0 | main                     | cmd_name     |     |           |           |              | trace2 (trace2)\n+d0 | main                     | exit         |     |  3.003667 |           |              | code:0\n+d0 | main                     | timer        |     |           |           | test         | name:test1 intervals:3 total:3.001686 min:1.000254 max:1.000929\n+d0 | main                     | atexit       |     |  3.003796 |           |              | code:0\n+----------------\n+\n+\n == Future Work\n \n === Relationship to the Existing Trace Api (api-trace.txt)\ndiff --git a/Makefile b/Makefile\nindex cac3452edb9..820649bf62a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1102,6 +1102,7 @@ LIB_OBJS += trace2/tr2_tgt_event.o\n LIB_OBJS += trace2/tr2_tgt_normal.o\n LIB_OBJS += trace2/tr2_tgt_perf.o\n LIB_OBJS += trace2/tr2_tls.o\n+LIB_OBJS += trace2/tr2_tmr.o\n LIB_OBJS += trailer.o\n LIB_OBJS += transport-helper.o\n LIB_OBJS += transport.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex a714130ece7..f951b9e97d7 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -228,6 +228,101 @@ static int ut_010bug_BUG(int argc, const char **argv)\n \tBUG(\"a %s message\", \"BUG\");\n }\n \n+/*\n+ * Single-threaded timer test.  Create several intervals using the\n+ * TEST1 timer.  The test script can verify that an aggregate Trace2\n+ * \"timer\" event is emitted indicating that we started+stopped the\n+ * timer the requested number of times.\n+ */\n+static int ut_100timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay>\";\n+\n+\tint count = 0;\n+\tint delay = 0;\n+\tint k;\n+\n+\tif (argc != 2)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST1);\n+\t\tsleep_millisec(delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST1);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+struct ut_101_data {\n+\tint count;\n+\tint delay;\n+};\n+\n+static void *ut_101timer_thread_proc(void *_ut_101_data)\n+{\n+\tstruct ut_101_data *data = _ut_101_data;\n+\tint k;\n+\n+\ttrace2_thread_start(\"ut_101\");\n+\n+\tfor (k = 0; k < data->count; k++) {\n+\t\ttrace2_timer_start(TRACE2_TIMER_ID_TEST2);\n+\t\tsleep_millisec(data->delay);\n+\t\ttrace2_timer_stop(TRACE2_TIMER_ID_TEST2);\n+\t}\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+/*\n+ * Multi-threaded timer test.  Create several threads that each create\n+ * several intervals using the TEST2 timer.  The test script can verify\n+ * that an individual Trace2 \"th_timer\" events for each thread and an\n+ * aggregate \"timer\" event are generated.\n+ */\n+static int ut_101timer(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <count> <ms_delay> <threads>\";\n+\n+\tstruct ut_101_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.count, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.delay, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_101timer_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -248,6 +343,9 @@ static struct unit_test ut_table[] = {\n \t{ ut_008bug,      \"008bug\",    \"\" },\n \t{ ut_009bug_BUG,  \"009bug_BUG\",\"\" },\n \t{ ut_010bug_BUG,  \"010bug_BUG\",\"\" },\n+\n+\t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n+\t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 22d0845544e..5c28424e657 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -173,4 +173,53 @@ test_expect_success 'using global config, perf stream, return code 0' '\n \ttest_cmp expect actual\n '\n \n+# Exercise the stopwatch timers in a loop and confirm that we have\n+# as many start/stop intervals as expected.  We cannot really test the\n+# actual (total, min, max) timer values, so we have to assume that they\n+# are good, but we can verify the interval count.\n+#\n+# The timer \"test/test1\" should only emit a global summary \"timer\" event.\n+# The timer \"test/test2\" should emit per-thread \"th_timer\" events and a\n+# global summary \"timer\" event.\n+\n+have_timer_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 intervals=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} intervals:${intervals}\" &&\n+\n+\tgrep \"${pattern}\" ${file}\n+}\n+\n+test_expect_success 'stopwatch timer test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test1\" 5 times from \"main\".\n+\ttest-tool trace2 100timer 5 10 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test1\" 5 actual\n+'\n+\n+test_expect_success 'stopwatch timer test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the timer \"test2\" 5 times each in 3 threads.\n+\ttest-tool trace2 101timer 5 10 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_timer_event \"th01:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th02:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\thave_timer_event \"th03:ut_101\" \"th_timer\" \"test\" \"test2\" 5 actual &&\n+\n+\t# And we should have 15 total uses.\n+\thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n+'\n+\n test_done\ndiff --git a/t/t0211/scrub_perf.perl b/t/t0211/scrub_perf.perl\nindex 299999f0f89..7a50bae6463 100644\n--- a/t/t0211/scrub_perf.perl\n+++ b/t/t0211/scrub_perf.perl\n@@ -64,6 +64,12 @@ while (<>) {\n \t    goto SKIP_LINE;\n \t}\n     }\n+    elsif ($tokens[$col_event] =~ m/timer/) {\n+\t# This also captures \"th_timer\" events\n+\t$tokens[$col_rest] =~ s/ total:\\d+\\.\\d*/ total:_T_TOTAL_/;\n+\t$tokens[$col_rest] =~ s/ min:\\d+\\.\\d*/ min:_T_MIN_/;\n+\t$tokens[$col_rest] =~ s/ max:\\d+\\.\\d*/ max:_T_MAX_/;\n+    }\n \n     # t_abs and t_rel are either blank or a float.  Replace the float\n     # with a constant for matching the HEREDOC in the test script.\ndiff --git a/trace2.c b/trace2.c\nindex 165264dc79a..a93cab7c2b7 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -13,6 +13,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static int trace2_enabled;\n \n@@ -83,6 +84,23 @@ static void tr2_tgt_disable_builtins(void)\n \t\ttgt_j->pfn_term();\n }\n \n+/*\n+ * The signature of this function must match the pfn_timer\n+ * method in the targets.  (Think of this is an apply operation\n+ * across the set of active targets.)\n+ */\n+static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n+\t\t\t\t const struct tr2_timer *timer,\n+\t\t\t\t int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_timer)\n+\t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -110,6 +128,26 @@ static void tr2main_atexit_handler(void)\n \t */\n \ttr2tls_pop_unwind_self();\n \n+\t/*\n+\t * Some timers want per-thread details.  If the main thread\n+\t * used one of those timers, emit the details now (before\n+\t * we emit the aggregate timer values).\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data for the main thread to the final\n+\t * totals.  And then emit the final timer values.\n+\t *\n+\t * Technically, we shouldn't need to hold the lock to update\n+\t * and output the final_timer_block (since all other threads\n+\t * should be dead by now), but it doesn't hurt anything.\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_atexit)\n \t\t\ttgt_j->pfn_atexit(us_elapsed_absolute,\n@@ -541,6 +579,21 @@ void trace2_thread_exit_fl(const char *file, int line)\n \ttr2tls_pop_unwind_self();\n \tus_elapsed_thread = tr2tls_region_elasped_self(us_now);\n \n+\t/*\n+\t * Some timers want per-thread details.  If this thread used\n+\t * one of those timers, emit the details now.\n+\t */\n+\ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\n+\t/*\n+\t * Add stopwatch timer data from the current (non-main) thread\n+\t * to the final totals.  (We'll accumulate data for the main\n+\t * thread later during \"atexit\".)\n+\t */\n+\ttr2tls_lock();\n+\ttr2_update_final_timers();\n+\ttr2tls_unlock();\n+\n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_exit_fl)\n \t\t\ttgt_j->pfn_thread_exit_fl(file, line,\n@@ -795,6 +848,28 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...)\n \tva_end(ap);\n }\n \n+void trace2_timer_start(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_start: invalid timer id: %d\", tid);\n+\n+\ttr2_start_timer(tid);\n+}\n+\n+void trace2_timer_stop(enum trace2_timer_id tid)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (tid < 0 || tid >= TRACE2_NUMBER_OF_TIMERS)\n+\t\tBUG(\"trace2_timer_stop: invalid timer id: %d\", tid);\n+\n+\ttr2_stop_timer(tid);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 74cdb1354f7..7a843ac0518 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -51,6 +51,7 @@ struct json_writer;\n  * [] trace2_region*    -- emit region nesting messages.\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n+ * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n  */\n \n /*\n@@ -485,6 +486,48 @@ void trace2_printf_fl(const char *file, int line, const char *fmt, ...);\n \n #define trace2_printf(...) trace2_printf_fl(__FILE__, __LINE__, __VA_ARGS__)\n \n+/*\n+ * Define the set of stopwatch timers.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum must also be added to the\n+ * `tr2_timer_metadata[]` in `trace2/tr2_tmr.c`.\n+ */\n+enum trace2_timer_id {\n+\t/*\n+\t * Define two timers for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_TIMER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_TIMER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional timer definitions before here. */\n+\tTRACE2_NUMBER_OF_TIMERS\n+};\n+\n+/*\n+ * Start/Stop the indicated stopwatch timer in the current thread.\n+ *\n+ * The time spent by the current thread between the _start and _stop\n+ * calls will be added to the thread's partial sum for this timer.\n+ *\n+ * Timer events are emitted at thread and program exit.\n+ *\n+ * Note: Since the stopwatch API routines do not generate individual\n+ * events, they do not take (file, line) arguments.  Similarly, the\n+ * category and timer name values are defined at compile-time in the\n+ * timer definitions array, so they are not needed here in the API.\n+ */\n+void trace2_timer_start(enum trace2_timer_id tid);\n+void trace2_timer_stop(enum trace2_timer_id tid);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 65f94e15748..85c8d2d7f5a 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -4,6 +4,10 @@\n struct child_process;\n struct repository;\n struct json_writer;\n+struct tr2_timer_metadata;\n+struct tr2_timer;\n+\n+#define NS_TO_SEC(ns) ((double)(ns) / 1.0e9)\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\n@@ -96,6 +100,10 @@ typedef void(tr2_tgt_evt_printf_va_fl_t)(const char *file, int line,\n \t\t\t\t\t uint64_t us_elapsed_absolute,\n \t\t\t\t\t const char *fmt, va_list ap);\n \n+typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n+\t\t\t\t  const struct tr2_timer *timer,\n+\t\t\t\t  int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -132,6 +140,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_fl_t                   *pfn_data_fl;\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n+\ttr2_tgt_evt_timer_t                     *pfn_timer;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 52f9356c695..af5a8edb474 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -9,6 +9,7 @@\n #include \"trace2/tr2_sysenv.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_event = {\n \t.sysenv_var = TR2_SYSENV_EVENT,\n@@ -617,6 +618,30 @@ static void fn_data_json_fl(const char *file, int line,\n \t}\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"intervals\", timer->interval_count);\n+\tjw_object_double(&jw, \"t_total\", 6, t_total);\n+\tjw_object_double(&jw, \"t_min\", 6, t_min);\n+\tjw_object_double(&jw, \"t_max\", 6, t_max);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -648,4 +673,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex 69f80330778..b079baf1002 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -8,6 +8,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_normal = {\n \t.sysenv_var = TR2_SYSENV_NORMAL,\n@@ -329,6 +330,27 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n+\n+\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -360,4 +382,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_fl = NULL,\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 59ca58f862d..e69375e9799 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -10,6 +10,7 @@\n #include \"trace2/tr2_tbuf.h\"\n #include \"trace2/tr2_tgt.h\"\n #include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n static struct tr2_dst tr2dst_perf = {\n \t.sysenv_var = TR2_SYSENV_PERF,\n@@ -555,6 +556,28 @@ static void fn_printf_va_fl(const char *file, int line,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_timer(const struct tr2_timer_metadata *meta,\n+\t\t     const struct tr2_timer *timer,\n+\t\t     int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n+\n+\tstrbuf_addf(&buf_payload, (\"name:%s\"\n+\t\t\t\t   \" intervals:%\"PRIu64\n+\t\t\t\t   \" total:%8.6f min:%8.6f max:%8.6f\"),\n+\t\t    meta->name,\n+\t\t    timer->interval_count,\n+\t\t    t_total, t_min, t_max);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -586,4 +609,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_fl = fn_data_fl,\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n+\t.pfn_timer = fn_timer,\n };\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 3a67532aae4..04900bb4c3a 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -181,3 +181,13 @@ int tr2tls_locked_increment(int *p)\n \n \treturn current_value;\n }\n+\n+void tr2tls_lock(void)\n+{\n+\tpthread_mutex_lock(&tr2tls_mutex);\n+}\n+\n+void tr2tls_unlock(void)\n+{\n+\tpthread_mutex_unlock(&tr2tls_mutex);\n+}\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 65836b1399c..a064b66e4cc 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_tmr.h\"\n \n /*\n  * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n@@ -20,6 +21,9 @@ struct tr2tls_thread_ctx {\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n+\tstruct tr2_timer_block timer_block;\n+\tunsigned int used_any_timer:1;\n+\tunsigned int used_any_per_thread_timer:1;\n };\n \n /*\n@@ -107,4 +111,10 @@ int tr2tls_locked_increment(int *p);\n  */\n void tr2tls_start_process_clock(void);\n \n+/*\n+ * Explicitly lock/unlock our mutex.\n+ */\n+void tr2tls_lock(void);\n+void tr2tls_unlock(void);\n+\n #endif /* TR2_TLS_H */\ndiff --git a/trace2/tr2_tmr.c b/trace2/tr2_tmr.c\nnew file mode 100644\nindex 00000000000..786762dfd26\n--- /dev/null\n+++ b/trace2/tr2_tmr.c\n@@ -0,0 +1,182 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_tmr.h\"\n+\n+#define MY_MAX(a, b) ((a) > (b) ? (a) : (b))\n+#define MY_MIN(a, b) ((a) < (b) ? (a) : (b))\n+\n+/*\n+ * A global timer block to aggregate values from the partial sums from\n+ * each thread.\n+ */\n+static struct tr2_timer_block final_timer_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each stopwatch timer.\n+ *\n+ * This array must match \"enum trace2_timer_id\" and the values\n+ * in \"struct tr2_timer_block.timer[*]\".\n+ */\n+static struct tr2_timer_metadata tr2_timer_metadata[TRACE2_NUMBER_OF_TIMERS] = {\n+\t[TRACE2_TIMER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_TIMER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_start_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\tt->recursion_count++;\n+\tif (t->recursion_count > 1)\n+\t\treturn; /* ignore recursive starts */\n+\n+\tt->start_ns = getnanotime();\n+}\n+\n+void tr2_stop_timer(enum trace2_timer_id tid)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\tuint64_t ns_now;\n+\tuint64_t ns_interval;\n+\n+\tassert(t->recursion_count > 0);\n+\n+\tt->recursion_count--;\n+\tif (t->recursion_count)\n+\t\treturn; /* still in recursive call(s) */\n+\n+\tns_now = getnanotime();\n+\tns_interval = ns_now - t->start_ns;\n+\n+\tt->total_ns += ns_interval;\n+\n+\t/*\n+\t * min_ns was initialized to zero (in the xcalloc()) rather\n+\t * than UINT_MAX when the block of timers was allocated,\n+\t * so we should always set both the min_ns and max_ns values\n+\t * the first time that the timer is used.\n+\t */\n+\tif (!t->interval_count) {\n+\t\tt->min_ns = ns_interval;\n+\t\tt->max_ns = ns_interval;\n+\t} else {\n+\t\tt->min_ns = MY_MIN(ns_interval, t->min_ns);\n+\t\tt->max_ns = MY_MAX(ns_interval, t->max_ns);\n+\t}\n+\n+\tt->interval_count++;\n+\n+\tctx->used_any_timer = 1;\n+\tif (tr2_timer_metadata[tid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_timer = 1;\n+}\n+\n+void tr2_update_final_timers(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++) {\n+\t\tstruct tr2_timer *t_final = &final_timer_block.timer[tid];\n+\t\tstruct tr2_timer *t = &ctx->timer_block.timer[tid];\n+\n+\t\tif (t->recursion_count) {\n+\t\t\t/*\n+\t\t\t * The current thread is exiting with\n+\t\t\t * timer[tid] still running.\n+\t\t\t *\n+\t\t\t * Technically, this is a bug, but I'm going\n+\t\t\t * to ignore it.\n+\t\t\t *\n+\t\t\t * I don't think it is worth calling die()\n+\t\t\t * for.  I don't think it is worth killing the\n+\t\t\t * process for this bookkeeping error.  We\n+\t\t\t * might want to call warning(), but I'm going\n+\t\t\t * to wait on that.\n+\t\t\t *\n+\t\t\t * The downside here is that total_ns won't\n+\t\t\t * include the current open interval (now -\n+\t\t\t * start_ns).  I can live with that.\n+\t\t\t */\n+\t\t}\n+\n+\t\tif (!t->interval_count)\n+\t\t\tcontinue; /* this timer was not used by this thread */\n+\n+\t\tt_final->total_ns += t->total_ns;\n+\n+\t\t/*\n+\t\t * final_timer_block.timer[tid].min_ns was initialized to\n+\t\t * was initialized to zero rather than UINT_MAX, so we should\n+\t\t * always set both the min_ns and max_ns values the first time\n+\t\t * that we add a partial sum into it.\n+\t\t */\n+\t\tif (!t_final->interval_count) {\n+\t\t\tt_final->min_ns = t->min_ns;\n+\t\t\tt_final->max_ns = t->max_ns;\n+\t\t} else {\n+\t\t\tt_final->min_ns = MY_MIN(t_final->min_ns, t->min_ns);\n+\t\t\tt_final->max_ns = MY_MAX(t_final->max_ns, t->max_ns);\n+\t\t}\n+\n+\t\tt_final->interval_count += t->interval_count;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_timer_id tid;\n+\n+\tif (!ctx->used_any_per_thread_timer)\n+\t\treturn;\n+\n+\t/*\n+\t * For each timer, if the timer wants per-thread events and\n+\t * this thread used it, emit it.\n+\t */\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (tr2_timer_metadata[tid].want_per_thread_events &&\n+\t\t    ctx->timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &ctx->timer_block.timer[tid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply)\n+{\n+\tenum trace2_timer_id tid;\n+\n+\t/*\n+\t * Accessing `final_timer_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (tid = 0; tid < TRACE2_NUMBER_OF_TIMERS; tid++)\n+\t\tif (final_timer_block.timer[tid].interval_count)\n+\t\t\tfn_apply(&tr2_timer_metadata[tid],\n+\t\t\t\t &final_timer_block.timer[tid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_tmr.h b/trace2/tr2_tmr.h\nnew file mode 100644\nindex 00000000000..d5753576134\n--- /dev/null\n+++ b/trace2/tr2_tmr.h\n@@ -0,0 +1,140 @@\n+#ifndef TR2_TMR_H\n+#define TR2_TMR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow \"stopwatch\" timers.\n+ *\n+ * Timers can be used to measure \"interesting\" activity that does not\n+ * fit the \"region\" model, such as code called from many different\n+ * regions (like zlib) and/or where data for individual calls are not\n+ * interesting or are too numerous to be efficiently logged.\n+ *\n+ * Timer values are accumulated during program execution and emitted\n+ * to the Trace2 logs at program exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * timers and timer ids using a \"timer block\" array in thread-local\n+ * storage.  This gives us constant time access to each timer within\n+ * each thread, since we want start/stop operations to be as fast as\n+ * possible.  This lets us avoid the complexities of dynamically\n+ * allocating a timer on the first use by a thread and/or possibly\n+ * sharing that timer definition with other concurrent threads.\n+ * However, this does require that we define time the set of timers at\n+ * compile time.\n+ *\n+ * Each thread uses the timer block in its thread-local storage to\n+ * compute partial sums for each timer (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Using this \"timer block\" model costs ~48 bytes per timer per thread\n+ * (we have about six uint64 fields per timer).  This does increase\n+ * the size of the thread-local storage block, but it is allocated (at\n+ * thread create time) and not on the thread stack, so I'm not worried\n+ * about the size.\n+ *\n+ * Partial sums for each timer are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each timer are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"timer metadata\" table contains the \"category\" and \"name\"\n+ * fields for each timer.  This eliminates the need to include those\n+ * args in the various timer APIs.\n+ */\n+\n+/*\n+ * The definition of an individual timer and used by an individual\n+ * thread.\n+ */\n+struct tr2_timer {\n+\t/*\n+\t * Total elapsed time for this timer in this thread in nanoseconds.\n+\t */\n+\tuint64_t total_ns;\n+\n+\t/*\n+\t * The maximum and minimum interval values observed for this\n+\t * timer in this thread.\n+\t */\n+\tuint64_t min_ns;\n+\tuint64_t max_ns;\n+\n+\t/*\n+\t * The value of the clock when this timer was started in this\n+\t * thread.  (Undefined when the timer is not active in this\n+\t * thread.)\n+\t */\n+\tuint64_t start_ns;\n+\n+\t/*\n+\t * Number of times that this timer has been started and stopped\n+\t * in this thread.  (Recursive starts are ignored.)\n+\t */\n+\tuint64_t interval_count;\n+\n+\t/*\n+\t * Number of nested starts on the stack in this thread.  (We\n+\t * ignore recursive starts and use this to track the recursive\n+\t * calls.)\n+\t */\n+\tunsigned int recursion_count;\n+};\n+\n+/*\n+ * Metadata for a timer.\n+ */\n+struct tr2_timer_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this timer\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed-size block of timers to insert into\n+ * thread-local storage.  This wrapper is used to avoid quirks\n+ * of C and the usual need to pass an array size argument.\n+ */\n+struct tr2_timer_block {\n+\tstruct tr2_timer timer[TRACE2_NUMBER_OF_TIMERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to actually start/stop an\n+ * individual timer in the current thread.\n+ */\n+void tr2_start_timer(enum trace2_timer_id tid);\n+void tr2_stop_timer(enum trace2_timer_id tid);\n+\n+/*\n+ * Add the current thread's timer data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_timers(void);\n+\n+/*\n+ * Emit per-thread timer data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+/*\n+ * Emit global total timer values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_timers(tr2_tgt_evt_timer_t *fn_apply);\n+\n+#endif /* TR2_TMR_H */\n-- \ngitgitgadget\n\n"},{"id":"465615","messageId":"9dee7a75903936f086d97580441c776978d70b43.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] tr2tls: clarify TLS terminology","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:01Z","receivedAt":"2022-10-24T15:16:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nReduce or eliminate use of the term \"TLS\" in the Trace2 code.\n\nThe term \"TLS\" has two popular meanings: \"thread-local storage\" and\n\"transport layer security\".  In the Trace2 source, the term is associated\nwith the former.  There was concern on the mailing list about it refering\nto the latter.\n\nUpdate the source and documentation to eliminate the use of the \"TLS\" term\nor replace it with the phrase \"thread-local storage\" to reduce ambiguity.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  8 ++++----\n trace2.c                               |  2 +-\n trace2.h                               | 10 +++++-----\n trace2/tr2_tls.c                       |  6 +++---\n trace2/tr2_tls.h                       | 18 +++++++++++-------\n 5 files changed, 24 insertions(+), 20 deletions(-)\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 2afa28bb5aa..431d424f9d5 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -685,8 +685,8 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_start\"`::\n \tThis event is generated when a thread is started.  It is\n-\tgenerated from *within* the new thread's thread-proc (for TLS\n-\treasons).\n+\tgenerated from *within* the new thread's thread-proc (because\n+\tit needs to access data in the thread's thread-local storage).\n +\n ------------\n {\n@@ -698,7 +698,7 @@ The \"exec_id\" field is a command-unique id and is only useful if the\n \n `\"thread_exit\"`::\n \tThis event is generated when a thread exits.  It is generated\n-\tfrom *within* the thread's thread-proc (for TLS reasons).\n+\tfrom *within* the thread's thread-proc.\n +\n ------------\n {\n@@ -1206,7 +1206,7 @@ worked on 508 items at offset 2032.  Thread \"th04\" worked on 508 items\n at offset 508.\n +\n This example also shows that thread names are assigned in a racy manner\n-as each thread starts and allocates TLS storage.\n+as each thread starts.\n \n Config (def param) Events::\n \ndiff --git a/trace2.c b/trace2.c\nindex 0c0a11e07d5..c1244e45ace 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -52,7 +52,7 @@ static struct tr2_tgt *tr2_tgt_builtins[] =\n  * Force (rather than lazily) initialize any of the requested\n  * builtin TRACE2 targets at startup (and before we've seen an\n  * actual TRACE2 event call) so we can see if we need to setup\n- * the TR2 and TLS machinery.\n+ * private data structures and thread-local storage.\n  *\n  * Return the number of builtin targets enabled.\n  */\ndiff --git a/trace2.h b/trace2.h\nindex 88d906ea830..af3c11694cc 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -73,8 +73,7 @@ void trace2_initialize_clock(void);\n /*\n  * Initialize TRACE2 tracing facility if any of the builtin TRACE2\n  * targets are enabled in the system config or the environment.\n- * This includes setting up the Trace2 thread local storage (TLS).\n- * Emits a 'version' message containing the version of git\n+ * This emits a 'version' message containing the version of git\n  * and the Trace2 protocol.\n  *\n  * This function should be called from `main()` as early as possible in\n@@ -302,7 +301,8 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n \n /*\n  * Emit a 'thread_start' event.  This must be called from inside the\n- * thread-proc to set up the trace2 TLS data for the thread.\n+ * thread-proc to allow the thread to create its own thread-local\n+ * storage.\n  *\n  * Thread names should be descriptive, like \"preload_index\".\n  * Thread names will be decorated with an instance number automatically.\n@@ -315,8 +315,8 @@ void trace2_thread_start_fl(const char *file, int line,\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\n- * thread-proc to report thread-specific data and cleanup TLS data\n- * for the thread.\n+ * thread-proc so that the thread can access and clean up its\n+ * thread-local storage.\n  */\n void trace2_thread_exit_fl(const char *file, int line);\n \ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 7da94aba522..8d2182fbdbb 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -69,9 +69,9 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void)\n \tctx = pthread_getspecific(tr2tls_key);\n \n \t/*\n-\t * If the thread-proc did not call trace2_thread_start(), we won't\n-\t * have any TLS data associated with the current thread.  Fix it\n-\t * here and silently continue.\n+\t * If the current thread's thread-proc did not call\n+\t * trace2_thread_start(), then the thread will not have any\n+\t * thread-local storage.  Create it now and silently continue.\n \t */\n \tif (!ctx)\n \t\tctx = tr2tls_create_self(\"unknown\", getnanotime() / 1000);\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex a90bd639d48..1297509fd23 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -3,6 +3,12 @@\n \n #include \"strbuf.h\"\n \n+/*\n+ * Notice: the term \"TLS\" refers to \"thread-local storage\" in the\n+ * Trace2 source files.  This usage is borrowed from GCC and Windows.\n+ * There is NO relation to \"transport layer security\".\n+ */\n+\n /*\n  * Arbitry limit for thread names for column alignment.\n  */\n@@ -17,9 +23,7 @@ struct tr2tls_thread_ctx {\n };\n \n /*\n- * Create TLS data for the current thread.  This gives us a place to\n- * put per-thread data, such as thread start time, function nesting\n- * and a per-thread label for our messages.\n+ * Create thread-local storage for the current thread.\n  *\n  * We assume the first thread is \"main\".  Other threads are given\n  * non-zero thread-ids to help distinguish messages from concurrent\n@@ -35,7 +39,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n- * Get our TLS data.\n+ * Get the thread-local storage pointer of the current thread.\n  */\n struct tr2tls_thread_ctx *tr2tls_get_self(void);\n \n@@ -45,7 +49,7 @@ struct tr2tls_thread_ctx *tr2tls_get_self(void);\n int tr2tls_is_main_thread(void);\n \n /*\n- * Free our TLS data.\n+ * Free the current thread's thread-local storage.\n  */\n void tr2tls_unset_self(void);\n \n@@ -81,12 +85,12 @@ uint64_t tr2tls_region_elasped_self(uint64_t us);\n uint64_t tr2tls_absolute_elapsed(uint64_t us);\n \n /*\n- * Initialize the tr2 TLS system.\n+ * Initialize thread-local storage for Trace2.\n  */\n void tr2tls_init(void);\n \n /*\n- * Free all tr2 TLS resources.\n+ * Free all Trace2 thread-local storage resources.\n  */\n void tr2tls_release(void);\n \n-- \ngitgitgadget\n\n"},{"id":"465616","messageId":"b359a49cec9857879db6f72764451ea886751f0d.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] trace2: add global counter mechanism","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:07Z","receivedAt":"2022-10-24T15:53:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd global counters mechanism to Trace2.\n\nThe Trace2 counters mechanism adds the ability to create a set of\nglobal counter variables and an API to increment them efficiently.\nCounters can optionally report per-thread usage in addition to the sum\nacross all threads.\n\nCounter events are emitted to the Trace2 logs when a thread exits and\nat process exit.\n\nCounters are an alternative to `data` and `data_json` events.\n\nCounters are useful when you want to measure something across the life\nof the process, when you don't want per-measurement events for\nperformance reasons, when the data does not fit conveniently within a\nregion, or when your control flow does not easily let you write the\nfinal total.  For example, you might use this to report the number of\ncalls to unzip() or the number of de-delta steps during a checkout.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/technical/api-trace2.txt |  31 ++++++++\n Makefile                               |   1 +\n t/helper/test-trace2.c                 |  89 +++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  46 +++++++++++\n trace2.c                               |  52 +++++++++++--\n trace2.h                               |  37 +++++++++\n trace2/tr2_ctr.c                       | 101 ++++++++++++++++++++++++\n trace2/tr2_ctr.h                       | 104 +++++++++++++++++++++++++\n trace2/tr2_tgt.h                       |   7 ++\n trace2/tr2_tgt_event.c                 |  19 +++++\n trace2/tr2_tgt_normal.c                |  16 ++++\n trace2/tr2_tgt_perf.c                  |  17 ++++\n trace2/tr2_tls.h                       |   4 +\n 13 files changed, 517 insertions(+), 7 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n\ndiff --git a/Documentation/technical/api-trace2.txt b/Documentation/technical/api-trace2.txt\nindex 75ce6f45603..de5fc250595 100644\n--- a/Documentation/technical/api-trace2.txt\n+++ b/Documentation/technical/api-trace2.txt\n@@ -805,6 +805,37 @@ The \"value\" field may be an integer or a string.\n }\n ------------\n \n+`\"th_counter\"`::\n+\tThis event logs the value of a counter variable in a thread.\n+\tThis event is generated when a thread exits for counters that\n+\trequested per-thread events.\n++\n+------------\n+{\n+\t\"event\":\"th_counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+`\"counter\"`::\n+\tThis event logs the value of a counter variable across all threads.\n+\tThis event is generated when the process exits.  The total value\n+\treported here is the sum across all threads.\n++\n+------------\n+{\n+\t\"event\":\"counter\",\n+\t...\n+\t\"category\":\"my_category\",\n+\t\"name\":\"my_counter\",\n+\t\"count\":23\n+}\n+------------\n+\n+\n == Example Trace2 API Usage\n \n Here is a hypothetical usage of the Trace2 API showing the intended\ndiff --git a/Makefile b/Makefile\nindex 820649bf62a..29ab417ca3a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1094,6 +1094,7 @@ LIB_OBJS += trace.o\n LIB_OBJS += trace2.o\n LIB_OBJS += trace2/tr2_cfg.o\n LIB_OBJS += trace2/tr2_cmd_name.o\n+LIB_OBJS += trace2/tr2_ctr.o\n LIB_OBJS += trace2/tr2_dst.o\n LIB_OBJS += trace2/tr2_sid.o\n LIB_OBJS += trace2/tr2_sysenv.o\ndiff --git a/t/helper/test-trace2.c b/t/helper/test-trace2.c\nindex f951b9e97d7..1b092c60714 100644\n--- a/t/helper/test-trace2.c\n+++ b/t/helper/test-trace2.c\n@@ -323,6 +323,92 @@ static int ut_101timer(int argc, const char **argv)\n \treturn 0;\n }\n \n+/*\n+ * Single-threaded counter test.  Add several values to the TEST1 counter.\n+ * The test script can verify that the final sum is reported in the \"counter\"\n+ * event.\n+ */\n+static int ut_200counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> [<v2> [...]]\";\n+\tint value;\n+\tint k;\n+\n+\tif (argc < 1)\n+\t\tdie(\"%s\", usage_error);\n+\n+\tfor (k = 0; k < argc; k++) {\n+\t\tif (get_i(&value, argv[k]))\n+\t\t\tdie(\"invalid value[%s] -- %s\",\n+\t\t\t    argv[k], usage_error);\n+\t\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST1, value);\n+\t}\n+\n+\treturn 0;\n+}\n+\n+/*\n+ * Multi-threaded counter test.  Create seveal threads that each increment\n+ * the TEST2 global counter.  The test script can verify that an individual\n+ * \"th_counter\" event is generated with a partial sum for each thread and\n+ * that a final aggregate \"counter\" event is generated.\n+ */\n+\n+struct ut_201_data {\n+\tint v1;\n+\tint v2;\n+};\n+\n+static void *ut_201counter_thread_proc(void *_ut_201_data)\n+{\n+\tstruct ut_201_data *data = _ut_201_data;\n+\n+\ttrace2_thread_start(\"ut_201\");\n+\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v1);\n+\ttrace2_counter_add(TRACE2_COUNTER_ID_TEST2, data->v2);\n+\n+\ttrace2_thread_exit();\n+\treturn NULL;\n+}\n+\n+static int ut_201counter(int argc, const char **argv)\n+{\n+\tconst char *usage_error =\n+\t\t\"expect <v1> <v2> <threads>\";\n+\n+\tstruct ut_201_data data = { 0, 0 };\n+\tint nr_threads = 0;\n+\tint k;\n+\tpthread_t *pids = NULL;\n+\n+\tif (argc != 3)\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v1, argv[0]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&data.v2, argv[1]))\n+\t\tdie(\"%s\", usage_error);\n+\tif (get_i(&nr_threads, argv[2]))\n+\t\tdie(\"%s\", usage_error);\n+\n+\tCALLOC_ARRAY(pids, nr_threads);\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_create(&pids[k], NULL, ut_201counter_thread_proc, &data))\n+\t\t\tdie(\"failed to create thread[%d]\", k);\n+\t}\n+\n+\tfor (k = 0; k < nr_threads; k++) {\n+\t\tif (pthread_join(pids[k], NULL))\n+\t\t\tdie(\"failed to join thread[%d]\", k);\n+\t}\n+\n+\tfree(pids);\n+\n+\treturn 0;\n+}\n+\n /*\n  * Usage:\n  *     test-tool trace2 <ut_name_1> <ut_usage_1>\n@@ -346,6 +432,9 @@ static struct unit_test ut_table[] = {\n \n \t{ ut_100timer,    \"100timer\",  \"<count> <ms_delay>\" },\n \t{ ut_101timer,    \"101timer\",  \"<count> <ms_delay> <threads>\" },\n+\n+\t{ ut_200counter,  \"200counter\", \"<v1> [<v2> [<v3> [...]]]\" },\n+\t{ ut_201counter,  \"201counter\", \"<v1> <v2> <threads>\" },\n };\n /* clang-format on */\n \ndiff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh\nindex 5c28424e657..0b3436e8cac 100755\n--- a/t/t0211-trace2-perf.sh\n+++ b/t/t0211-trace2-perf.sh\n@@ -222,4 +222,50 @@ test_expect_success 'stopwatch timer test/test2' '\n \thave_timer_event \"main\" \"timer\" \"test\" \"test2\" 15 actual\n '\n \n+# Exercise the global counters and confirm that we get the expected values.\n+#\n+# The counter \"test/test1\" should only emit a global summary \"counter\" event.\n+# The counter \"test/test2\" could emit per-thread \"th_counter\" events and a\n+# global summary \"counter\" event.\n+\n+have_counter_event () {\n+\tthread=$1 event=$2 category=$3 name=$4 value=$5 file=$6 &&\n+\n+\tpattern=\"d0|${thread}|${event}||||${category}|name:${name} value:${value}\" &&\n+\n+\tgrep \"${patern}\" ${file}\n+}\n+\n+test_expect_success 'global counter test/test1' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Use the counter \"test1\" and add n integers.\n+\ttest-tool trace2 200counter 1 2 3 4 5 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test1\" 15 actual\n+'\n+\n+test_expect_success 'global counter test/test2' '\n+\ttest_when_finished \"rm trace.perf actual\" &&\n+\ttest_config_global trace2.perfBrief 1 &&\n+\ttest_config_global trace2.perfTarget \"$(pwd)/trace.perf\" &&\n+\n+\t# Add 2 integers to the counter \"test2\" in each of 3 threads.\n+\ttest-tool trace2 201counter 7 13 3 &&\n+\n+\tperl \"$TEST_DIRECTORY/t0211/scrub_perf.perl\" <trace.perf >actual &&\n+\n+\t# So we should have 3 per-thread events of 5 each.\n+\thave_counter_event \"th01:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th02:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\thave_counter_event \"th03:ut_201\" \"th_counter\" \"test\" \"test2\" 20 actual &&\n+\n+\t# And we should have a single event with the total across all threads.\n+\thave_counter_event \"main\" \"counter\" \"test\" \"test2\" 60 actual\n+'\n+\n test_done\ndiff --git a/trace2.c b/trace2.c\nindex a93cab7c2b7..279bddf53b4 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -8,6 +8,7 @@\n #include \"version.h\"\n #include \"trace2/tr2_cfg.h\"\n #include \"trace2/tr2_cmd_name.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_dst.h\"\n #include \"trace2/tr2_sid.h\"\n #include \"trace2/tr2_sysenv.h\"\n@@ -101,6 +102,22 @@ static void tr2_tgt_emit_a_timer(const struct tr2_timer_metadata *meta,\n \t\t\ttgt_j->pfn_timer(meta, timer, is_final_data);\n }\n \n+/*\n+ * The signature of this function must match the pfn_counter\n+ * method in the targets.\n+ */\n+static void tr2_tgt_emit_a_counter(const struct tr2_counter_metadata *meta,\n+\t\t\t\t   const struct tr2_counter *counter,\n+\t\t\t\t   int is_final_data)\n+{\n+\tstruct tr2_tgt *tgt_j;\n+\tint j;\n+\n+\tfor_each_wanted_builtin (j, tgt_j)\n+\t\tif (tgt_j->pfn_counter)\n+\t\t\ttgt_j->pfn_counter(meta, counter, is_final_data);\n+}\n+\n static int tr2main_exit_code;\n \n /*\n@@ -132,20 +149,26 @@ static void tr2main_atexit_handler(void)\n \t * Some timers want per-thread details.  If the main thread\n \t * used one of those timers, emit the details now (before\n \t * we emit the aggregate timer values).\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data for the main thread to the final\n-\t * totals.  And then emit the final timer values.\n+\t * Add stopwatch timer and counter data for the main thread to\n+\t * the final totals.  And then emit the final values.\n \t *\n \t * Technically, we shouldn't need to hold the lock to update\n-\t * and output the final_timer_block (since all other threads\n-\t * should be dead by now), but it doesn't hurt anything.\n+\t * and output the final_timer_block and final_counter_block\n+\t * (since all other threads should be dead by now), but it\n+\t * doesn't hurt anything.\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2_emit_final_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_final_counters(tr2_tgt_emit_a_counter);\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -582,16 +605,20 @@ void trace2_thread_exit_fl(const char *file, int line)\n \t/*\n \t * Some timers want per-thread details.  If this thread used\n \t * one of those timers, emit the details now.\n+\t *\n+\t * Likewise for counters.\n \t */\n \ttr2_emit_per_thread_timers(tr2_tgt_emit_a_timer);\n+\ttr2_emit_per_thread_counters(tr2_tgt_emit_a_counter);\n \n \t/*\n-\t * Add stopwatch timer data from the current (non-main) thread\n-\t * to the final totals.  (We'll accumulate data for the main\n-\t * thread later during \"atexit\".)\n+\t * Add stopwatch timer and counter data from the current\n+\t * (non-main) thread to the final totals.  (We'll accumulate\n+\t * data for the main thread later during \"atexit\".)\n \t */\n \ttr2tls_lock();\n \ttr2_update_final_timers();\n+\ttr2_update_final_counters();\n \ttr2tls_unlock();\n \n \tfor_each_wanted_builtin (j, tgt_j)\n@@ -870,6 +897,17 @@ void trace2_timer_stop(enum trace2_timer_id tid)\n \ttr2_stop_timer(tid);\n }\n \n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tif (!trace2_enabled)\n+\t\treturn;\n+\n+\tif (cid < 0 || cid >= TRACE2_NUMBER_OF_COUNTERS)\n+\t\tBUG(\"trace2_counter_add: invalid counter id: %d\", cid);\n+\n+\ttr2_counter_increment(cid, value);\n+}\n+\n const char *trace2_session_id(void)\n {\n \treturn tr2_sid_get();\ndiff --git a/trace2.h b/trace2.h\nindex 7a843ac0518..4ced30c0db3 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -52,6 +52,7 @@ struct json_writer;\n  * [] trace2_data*      -- emit region/thread/repo data messages.\n  * [] trace2_printf*    -- legacy trace[1] messages.\n  * [] trace2_timer*     -- stopwatch timers (messages are deferred).\n+ * [] trace2_counter*   -- global counters (messages are deferred).\n  */\n \n /*\n@@ -528,6 +529,42 @@ enum trace2_timer_id {\n void trace2_timer_start(enum trace2_timer_id tid);\n void trace2_timer_stop(enum trace2_timer_id tid);\n \n+/*\n+ * Define the set of global counters.\n+ *\n+ * We can add more at any time, but they must be defined at compile\n+ * time (to avoid the need to dynamically allocate and synchronize\n+ * them between different threads).\n+ *\n+ * These must start at 0 and be contiguous (because we use them\n+ * elsewhere as array indexes).\n+ *\n+ * Any values added to this enum be also be added to the\n+ * `tr2_counter_metadata[]` in `trace2/tr2_tr2_ctr.c`.\n+ */\n+enum trace2_counter_id {\n+\t/*\n+\t * Define two counters for testing.  See `t/helper/test-trace2.c`.\n+\t * These can be used for ad hoc testing, but should not be used\n+\t * for permanent analysis code.\n+\t */\n+\tTRACE2_COUNTER_ID_TEST1 = 0, /* emits summary event only */\n+\tTRACE2_COUNTER_ID_TEST2,     /* emits summary and thread events */\n+\n+\t/* Add additional counter definitions before here. */\n+\tTRACE2_NUMBER_OF_COUNTERS\n+};\n+\n+/*\n+ * Increase the named global counter by value.\n+ *\n+ * Note that this adds `value` to the current thread's partial sum for\n+ * this counter (without locking) and that the complete sum is not\n+ * available until all threads have exited, so it does not return the\n+ * new value of the counter.\n+ */\n+void trace2_counter_add(enum trace2_counter_id cid, uint64_t value);\n+\n /*\n  * Optional platform-specific code to dump information about the\n  * current and any parent process(es).  This is intended to allow\ndiff --git a/trace2/tr2_ctr.c b/trace2/tr2_ctr.c\nnew file mode 100644\nindex 00000000000..483ca7c308f\n--- /dev/null\n+++ b/trace2/tr2_ctr.c\n@@ -0,0 +1,101 @@\n+#include \"cache.h\"\n+#include \"thread-utils.h\"\n+#include \"trace2/tr2_tgt.h\"\n+#include \"trace2/tr2_tls.h\"\n+#include \"trace2/tr2_ctr.h\"\n+\n+/*\n+ * A global counter block to aggregrate values from the partial sums\n+ * from each thread.\n+ */\n+static struct tr2_counter_block final_counter_block; /* access under tr2tls_mutex */\n+\n+/*\n+ * Define metadata for each global counter.\n+ *\n+ * This array must match the \"enum trace2_counter_id\" and the values\n+ * in \"struct tr2_counter_block.counter[*]\".\n+ */\n+static struct tr2_counter_metadata tr2_counter_metadata[TRACE2_NUMBER_OF_COUNTERS] = {\n+\t[TRACE2_COUNTER_ID_TEST1] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test1\",\n+\t\t.want_per_thread_events = 0,\n+\t},\n+\t[TRACE2_COUNTER_ID_TEST2] = {\n+\t\t.category = \"test\",\n+\t\t.name = \"test2\",\n+\t\t.want_per_thread_events = 1,\n+\t},\n+\n+\t/* Add additional metadata before here. */\n+};\n+\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tstruct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\tc->value += value;\n+\n+\tctx->used_any_counter = 1;\n+\tif (tr2_counter_metadata[cid].want_per_thread_events)\n+\t\tctx->used_any_per_thread_counter = 1;\n+}\n+\n+void tr2_update_final_counters(void)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++) {\n+\t\tstruct tr2_counter *c_final = &final_counter_block.counter[cid];\n+\t\tconst struct tr2_counter *c = &ctx->counter_block.counter[cid];\n+\n+\t\tc_final->value += c->value;\n+\t}\n+}\n+\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n+\tenum trace2_counter_id cid;\n+\n+\tif (!ctx->used_any_per_thread_counter)\n+\t\treturn;\n+\n+\t/*\n+\t * For each counter, if the counter wants per-thread events\n+\t * and this thread used it (the value is non-zero), emit it.\n+\t */\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (tr2_counter_metadata[cid].want_per_thread_events &&\n+\t\t    ctx->counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &ctx->counter_block.counter[cid],\n+\t\t\t\t 0);\n+}\n+\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply)\n+{\n+\tenum trace2_counter_id cid;\n+\n+\t/*\n+\t * Access `final_counter_block` requires holding `tr2tls_mutex`.\n+\t * We assume that our caller is holding the lock.\n+\t */\n+\n+\tfor (cid = 0; cid < TRACE2_NUMBER_OF_COUNTERS; cid++)\n+\t\tif (final_counter_block.counter[cid].value)\n+\t\t\tfn_apply(&tr2_counter_metadata[cid],\n+\t\t\t\t &final_counter_block.counter[cid],\n+\t\t\t\t 1);\n+}\ndiff --git a/trace2/tr2_ctr.h b/trace2/tr2_ctr.h\nnew file mode 100644\nindex 00000000000..a2267ee9901\n--- /dev/null\n+++ b/trace2/tr2_ctr.h\n@@ -0,0 +1,104 @@\n+#ifndef TR2_CTR_H\n+#define TR2_CTR_H\n+\n+#include \"trace2.h\"\n+#include \"trace2/tr2_tgt.h\"\n+\n+/*\n+ * Define a mechanism to allow global \"counters\".\n+ *\n+ * Counters can be used count interesting activity that does not fit\n+ * the \"region and data\" model, such as code called from many\n+ * different regions and/or where you want to count a number of items,\n+ * but don't have control of when the last item will be processed,\n+ * such as counter the number of calls to `lstat()`.\n+ *\n+ * Counters differ from Trace2 \"data\" events.  Data events are emitted\n+ * immediately and are appropriate for documenting loop counters at\n+ * the end of a region, for example.  Counter values are accumulated\n+ * during the program and final counter values are emitted at program\n+ * exit.\n+ *\n+ * To make this model efficient, we define a compile-time fixed set of\n+ * counters and counter ids using a fixed size \"counter block\" array\n+ * in thread-local storage.  This gives us constant time, lock-free\n+ * access to each counter within each thread.  This lets us avoid the\n+ * complexities of dynamically allocating a counter and sharing that\n+ * definition with other threads.\n+ *\n+ * Each thread uses the counter block in its thread-local storage to\n+ * increment partial sums for each counter (without locking).  When a\n+ * thread exits, those partial sums are (under lock) added to the\n+ * global final sum.\n+ *\n+ * Partial sums for each counter are optionally emitted when a thread\n+ * exits.\n+ *\n+ * Final sums for each counter are emitted between the \"exit\" and\n+ * \"atexit\" events.\n+ *\n+ * A parallel \"counter metadata\" table contains the \"category\" and\n+ * \"name\" fields for each counter.  This eliminates the need to\n+ * include those args in the various counter APIs.\n+ */\n+\n+/*\n+ * The definition of an individual counter as used by an individual\n+ * thread (and later in aggregation).\n+ */\n+struct tr2_counter {\n+\tuint64_t value;\n+};\n+\n+/*\n+ * Metadata for a counter.\n+ */\n+struct tr2_counter_metadata {\n+\tconst char *category;\n+\tconst char *name;\n+\n+\t/*\n+\t * True if we should emit per-thread events for this counter\n+\t * when individual threads exit.\n+\t */\n+\tunsigned int want_per_thread_events:1;\n+};\n+\n+/*\n+ * A compile-time fixed block of counters to insert into thread-local\n+ * storage.  This wrapper is used to avoid quirks of C and the usual\n+ * need to pass an array size argument.\n+ */\n+struct tr2_counter_block {\n+\tstruct tr2_counter counter[TRACE2_NUMBER_OF_COUNTERS];\n+};\n+\n+/*\n+ * Private routines used by trace2.c to increment a counter for the\n+ * current thread.\n+ */\n+void tr2_counter_increment(enum trace2_counter_id cid, uint64_t value);\n+\n+/*\n+ * Add the current thread's counter data to the global totals.\n+ * This is called during thread-exit.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_update_final_counters(void);\n+\n+/*\n+ * Emit per-thread counter data for the current thread.\n+ * This is called during thread-exit.\n+ */\n+void tr2_emit_per_thread_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+/*\n+ * Emit global counter values.\n+ * This is called during atexit handling.\n+ *\n+ * Caller must be holding the tr2tls_mutex.\n+ */\n+void tr2_emit_final_counters(tr2_tgt_evt_counter_t *fn_apply);\n+\n+#endif /* TR2_CTR_H */\ndiff --git a/trace2/tr2_tgt.h b/trace2/tr2_tgt.h\nindex 85c8d2d7f5a..bf8745c4f05 100644\n--- a/trace2/tr2_tgt.h\n+++ b/trace2/tr2_tgt.h\n@@ -6,6 +6,8 @@ struct repository;\n struct json_writer;\n struct tr2_timer_metadata;\n struct tr2_timer;\n+struct tr2_counter_metadata;\n+struct tr2_counter;\n \n #define NS_TO_SEC(ns) ((double)(ns) / 1.0e9)\n \n@@ -104,6 +106,10 @@ typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n \t\t\t\t  const struct tr2_timer *timer,\n \t\t\t\t  int is_final_data);\n \n+typedef void(tr2_tgt_evt_counter_t)(const struct tr2_counter_metadata *meta,\n+\t\t\t\t    const struct tr2_counter *counter,\n+\t\t\t\t    int is_final_data);\n+\n /*\n  * \"vtable\" for a TRACE2 target.  Use NULL if a target does not want\n  * to emit that message.\n@@ -141,6 +147,7 @@ struct tr2_tgt {\n \ttr2_tgt_evt_data_json_fl_t              *pfn_data_json_fl;\n \ttr2_tgt_evt_printf_va_fl_t              *pfn_printf_va_fl;\n \ttr2_tgt_evt_timer_t                     *pfn_timer;\n+\ttr2_tgt_evt_counter_t                   *pfn_counter;\n };\n /* clang-format on */\n \ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex af5a8edb474..16f6332755e 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -642,6 +642,24 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tjw_release(&jw);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n+\tjw_object_string(&jw, \"category\", meta->category);\n+\tjw_object_string(&jw, \"name\", meta->name);\n+\tjw_object_intmax(&jw, \"count\", counter->value);\n+\tjw_end(&jw);\n+\n+\ttr2_dst_write_line(&tr2dst_event, &jw.json);\n+\tjw_release(&jw);\n+}\n+\n struct tr2_tgt tr2_tgt_event = {\n \t.pdst = &tr2dst_event,\n \n@@ -674,4 +692,5 @@ struct tr2_tgt tr2_tgt_event = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = NULL,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_normal.c b/trace2/tr2_tgt_normal.c\nindex b079baf1002..fbbef68dfc0 100644\n--- a/trace2/tr2_tgt_normal.c\n+++ b/trace2/tr2_tgt_normal.c\n@@ -351,6 +351,21 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"%s %s/%s value:%\"PRIu64,\n+\t\t    event_name, meta->category, meta->name,\n+\t\t    counter->value);\n+\n+\tnormal_io_write_fl(__FILE__, __LINE__, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_normal = {\n \t.pdst = &tr2dst_normal,\n \n@@ -383,4 +398,5 @@ struct tr2_tgt tr2_tgt_normal = {\n \t.pfn_data_json_fl = NULL,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex e69375e9799..adae8032639 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -578,6 +578,22 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n \tstrbuf_release(&buf_payload);\n }\n \n+static void fn_counter(const struct tr2_counter_metadata *meta,\n+\t\t       const struct tr2_counter *counter,\n+\t\t       int is_final_data)\n+{\n+\tconst char *event_name = is_final_data ? \"counter\" : \"th_counter\";\n+\tstruct strbuf buf_payload = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf_payload, \"name:%s value:%\"PRIu64,\n+\t\t    meta->name,\n+\t\t    counter->value);\n+\n+\tperf_io_write_fl(__FILE__, __LINE__, event_name, NULL, NULL, NULL,\n+\t\t\t meta->category, &buf_payload);\n+\tstrbuf_release(&buf_payload);\n+}\n+\n struct tr2_tgt tr2_tgt_perf = {\n \t.pdst = &tr2dst_perf,\n \n@@ -610,4 +626,5 @@ struct tr2_tgt tr2_tgt_perf = {\n \t.pfn_data_json_fl = fn_data_json_fl,\n \t.pfn_printf_va_fl = fn_printf_va_fl,\n \t.pfn_timer = fn_timer,\n+\t.pfn_counter = fn_counter,\n };\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex a064b66e4cc..f9049805d4d 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -2,6 +2,7 @@\n #define TR2_TLS_H\n \n #include \"strbuf.h\"\n+#include \"trace2/tr2_ctr.h\"\n #include \"trace2/tr2_tmr.h\"\n \n /*\n@@ -22,8 +23,11 @@ struct tr2tls_thread_ctx {\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n \tint thread_id;\n \tstruct tr2_timer_block timer_block;\n+\tstruct tr2_counter_block counter_block;\n \tunsigned int used_any_timer:1;\n \tunsigned int used_any_per_thread_timer:1;\n+\tunsigned int used_any_counter:1;\n+\tunsigned int used_any_per_thread_counter:1;\n };\n \n /*\n-- \ngitgitgadget\n"},{"id":"465618","messageId":"79c6406d492ab629d5d042edaf1507888d5378c0.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 6/8] trace2: convert ctx.thread_name from strbuf to pointer","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:05Z","receivedAt":"2022-10-24T16:10:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nConvert the `tr2tls_thread_ctx.thread_name` field from a `strbuf`\nto a \"const char*\" pointer.\n\nThe `thread_name` field is a constant string that is constructed when\nthe context is created.  Using a (non-const) `strbuf` structure for it\ncaused some confusion in the past because it implied that someone\ncould rename a thread after it was created.  That usage was not\nintended.  Change it to a const pointer to make the intent more clear.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tgt_event.c |  2 +-\n trace2/tr2_tgt_perf.c  |  2 +-\n trace2/tr2_tls.c       | 16 +++++++++-------\n trace2/tr2_tls.h       |  2 +-\n 4 files changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/trace2/tr2_tgt_event.c b/trace2/tr2_tgt_event.c\nindex 37a3163be12..52f9356c695 100644\n--- a/trace2/tr2_tgt_event.c\n+++ b/trace2/tr2_tgt_event.c\n@@ -90,7 +90,7 @@ static void event_fmt_prepare(const char *event_name, const char *file,\n \n \tjw_object_string(jw, \"event\", event_name);\n \tjw_object_string(jw, \"sid\", tr2_sid_get());\n-\tjw_object_string(jw, \"thread\", ctx->thread_name.buf);\n+\tjw_object_string(jw, \"thread\", ctx->thread_name);\n \n \t/*\n \t * In brief mode, only emit <time> on these 2 event types.\ndiff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c\nindex 8cb792488c8..59ca58f862d 100644\n--- a/trace2/tr2_tgt_perf.c\n+++ b/trace2/tr2_tgt_perf.c\n@@ -108,7 +108,7 @@ static void perf_fmt_prepare(const char *event_name,\n \n \tstrbuf_addf(buf, \"d%d | \", tr2_sid_depth());\n \tstrbuf_addf(buf, \"%-*s | %-*s | \", TR2_MAX_THREAD_NAME,\n-\t\t    ctx->thread_name.buf, TR2FMT_PERF_MAX_EVENT_NAME,\n+\t\t    ctx->thread_name, TR2FMT_PERF_MAX_EVENT_NAME,\n \t\t    event_name);\n \n \tlen = buf->len + TR2FMT_PERF_REPO_WIDTH;\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 4f7c516ecb6..3a67532aae4 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -35,6 +35,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n+\tstruct strbuf buf = STRBUF_INIT;\n \n \t/*\n \t * Implicitly \"tr2tls_push_self()\" to capture the thread's start\n@@ -47,12 +48,13 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \n \tctx->thread_id = tr2tls_locked_increment(&tr2_next_thread_id);\n \n-\tstrbuf_init(&ctx->thread_name, 0);\n+\tstrbuf_init(&buf, 0);\n \tif (ctx->thread_id)\n-\t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n-\tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n-\t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n+\t\tstrbuf_addf(&buf, \"th%02d:\", ctx->thread_id);\n+\tstrbuf_addstr(&buf, thread_base_name);\n+\tif (buf.len > TR2_MAX_THREAD_NAME)\n+\t\tstrbuf_setlen(&buf, TR2_MAX_THREAD_NAME);\n+\tctx->thread_name = strbuf_detach(&buf, NULL);\n \n \tpthread_setspecific(tr2tls_key, ctx);\n \n@@ -95,7 +97,7 @@ void tr2tls_unset_self(void)\n \n \tpthread_setspecific(tr2tls_key, NULL);\n \n-\tstrbuf_release(&ctx->thread_name);\n+\tfree((char *)ctx->thread_name);\n \tfree(ctx->array_us_start);\n \tfree(ctx);\n }\n@@ -113,7 +115,7 @@ void tr2tls_pop_self(void)\n \tstruct tr2tls_thread_ctx *ctx = tr2tls_get_self();\n \n \tif (!ctx->nr_open_regions)\n-\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name.buf);\n+\t\tBUG(\"no open regions in thread '%s'\", ctx->thread_name);\n \n \tctx->nr_open_regions--;\n }\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 3ac4380d829..65836b1399c 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -15,7 +15,7 @@\n #define TR2_MAX_THREAD_NAME (24)\n \n struct tr2tls_thread_ctx {\n-\tstruct strbuf thread_name;\n+\tconst char *thread_name;\n \tuint64_t *array_us_start;\n \tsize_t alloc;\n \tsize_t nr_open_regions; /* plays role of \"nr\" in ALLOC_GROW */\n-- \ngitgitgadget\n\n"},{"id":"465620","messageId":"acfae17548c59b3a0145740addc0c1b8f175355a.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] trace2: improve thread-name documentation in the thread-context","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:04Z","receivedAt":"2022-10-24T16:20:11Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nImprove the documentation of the tr2tls_thread_ctx.thread_name field\nand its relation to the tr2tls_thread_ctx.thread_id field.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2/tr2_tls.h | 15 +++++++++------\n 1 file changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex d4e725f430b..3ac4380d829 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -25,12 +25,15 @@ struct tr2tls_thread_ctx {\n /*\n  * Create thread-local storage for the current thread.\n  *\n- * We assume the first thread is \"main\".  Other threads are given\n- * non-zero thread-ids to help distinguish messages from concurrent\n- * threads.\n- *\n- * Truncate the thread name if necessary to help with column alignment\n- * in printf-style messages.\n+ * The first thread in the process will have:\n+ *     { .thread_id=0, .thread_name=\"main\" }\n+ * Subsequent threads are given a non-zero thread_id and a thread_name\n+ * constructed from the id and a thread base name (which is usually just\n+ * the name of the thread-proc function).  For example:\n+ *     { .thread_id=10, .thread_name=\"th10:fsm-listen\" }\n+ * This helps to identify and distinguish messages from concurrent threads.\n+ * The ctx.thread_name field is truncated if necessary to help with column\n+ * alignment in printf-style messages.\n  *\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n-- \ngitgitgadget\n\n"},{"id":"465624","messageId":"9adf9cee1a96211cc4c2a305997079c7d6492aea.1666618868.git.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"[PATCH v4 4/8] trace2: rename the thread_name argument to trace2_thread_start","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:41:03Z","receivedAt":"2022-10-24T18:53:07Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nRename the `thread_name` argument in `tr2tls_create_self()` and\n`trace2_thread_start()` to be `thread_base_name` to make it clearer\nthat the passed argument is a component used in the construction of\nthe actual `struct tr2tls_thread_ctx.thread_name` variable.\n\nThe base name will be used along with the thread id to create a\nunique thread name.\n\nThis commit does not change how the `thread_name` field is\nallocated or stored within the `tr2tls_thread_ctx` structure.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n trace2.c         |  6 +++---\n trace2.h         | 11 ++++++-----\n trace2/tr2_tls.c |  4 ++--\n trace2/tr2_tls.h |  2 +-\n 4 files changed, 12 insertions(+), 11 deletions(-)\n\ndiff --git a/trace2.c b/trace2.c\nindex c1244e45ace..165264dc79a 100644\n--- a/trace2.c\n+++ b/trace2.c\n@@ -466,7 +466,7 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code)\n \t\t\t\tfile, line, us_elapsed_absolute, exec_id, code);\n }\n \n-void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n+void trace2_thread_start_fl(const char *file, int line, const char *thread_base_name)\n {\n \tstruct tr2_tgt *tgt_j;\n \tint j;\n@@ -488,14 +488,14 @@ void trace2_thread_start_fl(const char *file, int line, const char *thread_name)\n \t\t */\n \t\ttrace2_region_enter_printf_fl(file, line, NULL, NULL, NULL,\n \t\t\t\t\t      \"thread-proc on main: %s\",\n-\t\t\t\t\t      thread_name);\n+\t\t\t\t\t      thread_base_name);\n \t\treturn;\n \t}\n \n \tus_now = getnanotime() / 1000;\n \tus_elapsed_absolute = tr2tls_absolute_elapsed(us_now);\n \n-\ttr2tls_create_self(thread_name, us_now);\n+\ttr2tls_create_self(thread_base_name, us_now);\n \n \tfor_each_wanted_builtin (j, tgt_j)\n \t\tif (tgt_j->pfn_thread_start_fl)\ndiff --git a/trace2.h b/trace2.h\nindex af3c11694cc..74cdb1354f7 100644\n--- a/trace2.h\n+++ b/trace2.h\n@@ -304,14 +304,15 @@ void trace2_exec_result_fl(const char *file, int line, int exec_id, int code);\n  * thread-proc to allow the thread to create its own thread-local\n  * storage.\n  *\n- * Thread names should be descriptive, like \"preload_index\".\n- * Thread names will be decorated with an instance number automatically.\n+ * The thread base name should be descriptive, like \"preload_index\" or\n+ * taken from the thread-proc function.  A unique thread name will be\n+ * created from the given base name and the thread id automatically.\n  */\n void trace2_thread_start_fl(const char *file, int line,\n-\t\t\t    const char *thread_name);\n+\t\t\t    const char *thread_base_name);\n \n-#define trace2_thread_start(thread_name) \\\n-\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_name))\n+#define trace2_thread_start(thread_base_name) \\\n+\ttrace2_thread_start_fl(__FILE__, __LINE__, (thread_base_name))\n \n /*\n  * Emit a 'thread_exit' event.  This must be called from inside the\ndiff --git a/trace2/tr2_tls.c b/trace2/tr2_tls.c\nindex 8d2182fbdbb..4f7c516ecb6 100644\n--- a/trace2/tr2_tls.c\n+++ b/trace2/tr2_tls.c\n@@ -31,7 +31,7 @@ void tr2tls_start_process_clock(void)\n \ttr2tls_us_start_process = getnanotime() / 1000;\n }\n \n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start)\n {\n \tstruct tr2tls_thread_ctx *ctx = xcalloc(1, sizeof(*ctx));\n@@ -50,7 +50,7 @@ struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n \tstrbuf_init(&ctx->thread_name, 0);\n \tif (ctx->thread_id)\n \t\tstrbuf_addf(&ctx->thread_name, \"th%02d:\", ctx->thread_id);\n-\tstrbuf_addstr(&ctx->thread_name, thread_name);\n+\tstrbuf_addstr(&ctx->thread_name, thread_base_name);\n \tif (ctx->thread_name.len > TR2_MAX_THREAD_NAME)\n \t\tstrbuf_setlen(&ctx->thread_name, TR2_MAX_THREAD_NAME);\n \ndiff --git a/trace2/tr2_tls.h b/trace2/tr2_tls.h\nindex 1297509fd23..d4e725f430b 100644\n--- a/trace2/tr2_tls.h\n+++ b/trace2/tr2_tls.h\n@@ -35,7 +35,7 @@ struct tr2tls_thread_ctx {\n  * In this and all following functions the term \"self\" refers to the\n  * current thread.\n  */\n-struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_name,\n+struct tr2tls_thread_ctx *tr2tls_create_self(const char *thread_base_name,\n \t\t\t\t\t     uint64_t us_thread_start);\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"465636","messageId":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","threadId":"58564","inReplyTo":"pull.1373.v3.git.1666290489.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Trace2 timers and counters and some cleanup","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-10-24T13:40:59Z","receivedAt":"2022-10-24T20:45:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 4 of this series to add timers and counters to Trace2.\n\nChanges since V3:\n\n * Fixed typo in the new thread-name documentation.\n * Use a simpler NS_TO_SEC() macro for reporting the timer values.\n\nJeff Hostetler (8):\n  trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n  tr2tls: clarify TLS terminology\n  api-trace2.txt: elminate section describing the public trace2 API\n  trace2: rename the thread_name argument to trace2_thread_start\n  trace2: improve thread-name documentation in the thread-context\n  trace2: convert ctx.thread_name from strbuf to pointer\n  trace2: add stopwatch timers\n  trace2: add global counter mechanism\n\n Documentation/technical/api-trace2.txt | 190 +++++++++++++++++--------\n Makefile                               |   2 +\n t/helper/test-trace2.c                 | 187 ++++++++++++++++++++++++\n t/t0211-trace2-perf.sh                 |  95 +++++++++++++\n t/t0211/scrub_perf.perl                |   6 +\n trace2.c                               | 121 +++++++++++++++-\n trace2.h                               | 101 +++++++++++--\n trace2/tr2_ctr.c                       | 101 +++++++++++++\n trace2/tr2_ctr.h                       | 104 ++++++++++++++\n trace2/tr2_tgt.h                       |  16 +++\n trace2/tr2_tgt_event.c                 |  47 +++++-\n trace2/tr2_tgt_normal.c                |  39 +++++\n trace2/tr2_tgt_perf.c                  |  43 +++++-\n trace2/tr2_tls.c                       |  34 +++--\n trace2/tr2_tls.h                       |  55 ++++---\n trace2/tr2_tmr.c                       | 182 +++++++++++++++++++++++\n trace2/tr2_tmr.h                       | 140 ++++++++++++++++++\n 17 files changed, 1361 insertions(+), 102 deletions(-)\n create mode 100644 trace2/tr2_ctr.c\n create mode 100644 trace2/tr2_ctr.h\n create mode 100644 trace2/tr2_tmr.c\n create mode 100644 trace2/tr2_tmr.h\n\n\nbase-commit: 3dcec76d9df911ed8321007b1d197c1a206dc164\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1373%2Fjeffhostetler%2Ftrace2-stopwatch-v4-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1373/jeffhostetler/trace2-stopwatch-v4-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/1373\n\nRange-diff vs v3:\n\n 1:  6e7e4f3187e = 1:  6e7e4f3187e trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n 2:  9dee7a75903 = 2:  9dee7a75903 tr2tls: clarify TLS terminology\n 3:  804dab9e1a7 = 3:  804dab9e1a7 api-trace2.txt: elminate section describing the public trace2 API\n 4:  9adf9cee1a9 = 4:  9adf9cee1a9 trace2: rename the thread_name argument to trace2_thread_start\n 5:  8cb206b7632 ! 5:  acfae17548c trace2: improve thread-name documentation in the thread-context\n     @@ trace2/tr2_tls.h: struct tr2tls_thread_ctx {\n      + * Subsequent threads are given a non-zero thread_id and a thread_name\n      + * constructed from the id and a thread base name (which is usually just\n      + * the name of the thread-proc function).  For example:\n     -+ *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n     ++ *     { .thread_id=10, .thread_name=\"th10:fsm-listen\" }\n      + * This helps to identify and distinguish messages from concurrent threads.\n      + * The ctx.thread_name field is truncated if necessary to help with column\n      + * alignment in printf-style messages.\n 6:  8a89e1aa238 = 6:  79c6406d492 trace2: convert ctx.thread_name from strbuf to pointer\n 7:  8e701109976 ! 7:  a10c1bd96bb trace2: add stopwatch timers\n     @@ trace2/tr2_tgt.h\n      +struct tr2_timer_metadata;\n      +struct tr2_timer;\n      +\n     -+#define NS_PER_SEC_D ((double)1000*1000*1000)\n     ++#define NS_TO_SEC(ns) ((double)(ns) / 1.0e9)\n       \n       /*\n        * Function prototypes for a TRACE2 \"target\" vtable.\n     @@ trace2/tr2_tgt_event.c: static void fn_data_json_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct json_writer jw = JSON_WRITER_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     -+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     -+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n     ++\tdouble t_total = NS_TO_SEC(timer->total_ns);\n     ++\tdouble t_min = NS_TO_SEC(timer->min_ns);\n     ++\tdouble t_max = NS_TO_SEC(timer->max_ns);\n      +\n      +\tjw_object_begin(&jw, 0);\n      +\tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\n     @@ trace2/tr2_tgt_normal.c: static void fn_printf_va_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct strbuf buf_payload = STRBUF_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     -+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     -+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n     ++\tdouble t_total = NS_TO_SEC(timer->total_ns);\n     ++\tdouble t_min = NS_TO_SEC(timer->min_ns);\n     ++\tdouble t_max = NS_TO_SEC(timer->max_ns);\n      +\n      +\tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n      +\t\t\t\t   \" intervals:%\"PRIu64\n     @@ trace2/tr2_tgt_perf.c: static void fn_printf_va_fl(const char *file, int line,\n      +{\n      +\tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n      +\tstruct strbuf buf_payload = STRBUF_INIT;\n     -+\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n     -+\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n     -+\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n     ++\tdouble t_total = NS_TO_SEC(timer->total_ns);\n     ++\tdouble t_min = NS_TO_SEC(timer->min_ns);\n     ++\tdouble t_max = NS_TO_SEC(timer->max_ns);\n      +\n      +\tstrbuf_addf(&buf_payload, (\"name:%s\"\n      +\t\t\t\t   \" intervals:%\"PRIu64\n 8:  5cd8bdde884 ! 8:  b359a49cec9 trace2: add global counter mechanism\n     @@ trace2/tr2_tgt.h: struct repository;\n      +struct tr2_counter_metadata;\n      +struct tr2_counter;\n       \n     - #define NS_PER_SEC_D ((double)1000*1000*1000)\n     + #define NS_TO_SEC(ns) ((double)(ns) / 1.0e9)\n       \n      @@ trace2/tr2_tgt.h: typedef void(tr2_tgt_evt_timer_t)(const struct tr2_timer_metadata *meta,\n       \t\t\t\t  const struct tr2_timer *timer,\n\n-- \ngitgitgadget\n"},{"id":"465660","messageId":"xmqqeduxrmjs.fsf@gitster.g","threadId":"58564","inReplyTo":"6e7e4f3187e2fbbbb54bb1cf5793bf6e981a5a94.1666618868.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/8] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-24T20:31:19Z","receivedAt":"2022-10-24T23:06:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"As I do not see a cover letter for this series, here is the summary\nof the change since the previous round that has been in 'seen'.\n\nI didn't see anything questionable in these.\n\nThanks, will queue.\n\n trace2/tr2_tgt.h        | 2 +-\n trace2/tr2_tgt_event.c  | 6 +++---\n trace2/tr2_tgt_normal.c | 6 +++---\n trace2/tr2_tgt_perf.c   | 6 +++---\n trace2/tr2_tls.h        | 2 +-\n 5 files changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git c/trace2/tr2_tgt.h w/trace2/tr2_tgt.h\nindex 95f4c75472..bf8745c4f0 100644\n--- c/trace2/tr2_tgt.h\n+++ w/trace2/tr2_tgt.h\n@@ -9,7 +9,7 @@ struct tr2_timer;\n struct tr2_counter_metadata;\n struct tr2_counter;\n \n-#define NS_PER_SEC_D ((double)1000*1000*1000)\n+#define NS_TO_SEC(ns) ((double)(ns) / 1.0e9)\n \n /*\n  * Function prototypes for a TRACE2 \"target\" vtable.\ndiff --git c/trace2/tr2_tgt_event.c w/trace2/tr2_tgt_event.c\nindex 981863a660..16f6332755 100644\n--- c/trace2/tr2_tgt_event.c\n+++ w/trace2/tr2_tgt_event.c\n@@ -624,9 +624,9 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n {\n \tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n \tstruct json_writer jw = JSON_WRITER_INIT;\n-\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n-\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n-\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n \n \tjw_object_begin(&jw, 0);\n \tevent_fmt_prepare(event_name, __FILE__, __LINE__, NULL, &jw);\ndiff --git c/trace2/tr2_tgt_normal.c w/trace2/tr2_tgt_normal.c\nindex def18674e8..fbbef68dfc 100644\n--- c/trace2/tr2_tgt_normal.c\n+++ w/trace2/tr2_tgt_normal.c\n@@ -336,9 +336,9 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n {\n \tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n \tstruct strbuf buf_payload = STRBUF_INIT;\n-\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n-\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n-\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n \n \tstrbuf_addf(&buf_payload, (\"%s %s/%s\"\n \t\t\t\t   \" intervals:%\"PRIu64\ndiff --git c/trace2/tr2_tgt_perf.c w/trace2/tr2_tgt_perf.c\nindex db94b2ef47..adae803263 100644\n--- c/trace2/tr2_tgt_perf.c\n+++ w/trace2/tr2_tgt_perf.c\n@@ -562,9 +562,9 @@ static void fn_timer(const struct tr2_timer_metadata *meta,\n {\n \tconst char *event_name = is_final_data ? \"timer\" : \"th_timer\";\n \tstruct strbuf buf_payload = STRBUF_INIT;\n-\tdouble t_total = ((double)timer->total_ns) / NS_PER_SEC_D;\n-\tdouble t_min = ((double)timer->min_ns) / NS_PER_SEC_D;\n-\tdouble t_max = ((double)timer->max_ns) / NS_PER_SEC_D;\n+\tdouble t_total = NS_TO_SEC(timer->total_ns);\n+\tdouble t_min = NS_TO_SEC(timer->min_ns);\n+\tdouble t_max = NS_TO_SEC(timer->max_ns);\n \n \tstrbuf_addf(&buf_payload, (\"name:%s\"\n \t\t\t\t   \" intervals:%\"PRIu64\ndiff --git c/trace2/tr2_tls.h w/trace2/tr2_tls.h\nindex 289b62d072..f9049805d4 100644\n--- c/trace2/tr2_tls.h\n+++ w/trace2/tr2_tls.h\n@@ -38,7 +38,7 @@ struct tr2tls_thread_ctx {\n  * Subsequent threads are given a non-zero thread_id and a thread_name\n  * constructed from the id and a thread base name (which is usually just\n  * the name of the thread-proc function).  For example:\n- *     { .thread_id=10, .thread_name=\"th10fsm-listen\" }\n+ *     { .thread_id=10, .thread_name=\"th10:fsm-listen\" }\n  * This helps to identify and distinguish messages from concurrent threads.\n  * The ctx.thread_name field is truncated if necessary to help with column\n  * alignment in printf-style messages.\n"},{"id":"465699","messageId":"0e58bd35-4f40-ce9e-1088-f7c004527aee@github.com","threadId":"58564","inReplyTo":"pull.1373.v4.git.1666618868.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/8] Trace2 timers and counters and some cleanup","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-25T12:27:20Z","receivedAt":"2022-10-25T12:27:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/24/2022 9:40 AM, Jeff Hostetler via GitGitGadget wrote:\n> Here is version 4 of this series to add timers and counters to Trace2.\n> \n> Changes since V3:\n> \n>  * Fixed typo in the new thread-name documentation.\n>  * Use a simpler NS_TO_SEC() macro for reporting the timer values.\n> \n> Jeff Hostetler (8):\n>   trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n>   tr2tls: clarify TLS terminology\n>   api-trace2.txt: elminate section describing the public trace2 API\n>   trace2: rename the thread_name argument to trace2_thread_start\n>   trace2: improve thread-name documentation in the thread-context\n>   trace2: convert ctx.thread_name from strbuf to pointer\n>   trace2: add stopwatch timers\n>   trace2: add global counter mechanism\n\nI re-read the series as well as looked at the range-diffs for the\nprevious two versions. I continue to think this is a high-quality\nseries and I've used it multiple times in my personal development\nworkflow to investigate certain performance things. I'm looking\nforward to this being merged so we can all use it.\n\nThanks,\n-Stolee\n"},{"id":"465705","messageId":"a7b0b896-7b2b-a263-dd71-8b7b929707b4@github.com","threadId":"58564","inReplyTo":"xmqqeduxrmjs.fsf@gitster.g","subject":"Re: [PATCH v4 1/8] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-10-25T12:35:10Z","receivedAt":"2022-10-25T12:35:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/24/2022 4:31 PM, Junio C Hamano wrote:\n> As I do not see a cover letter for this series, here is the summary\n> of the change since the previous round that has been in 'seen'.\n> \n> I didn't see anything questionable in these.\n> \n> Thanks, will queue.\n\nThe cover letter appears on my end, but I'm on the CC list.\n\nJeff: be sure to CC Junio by adding him to the CC list on your\nPR description for anything you want to have considered for\nqueuing.\n\nThanks,\n-Stolee\n\n"},{"id":"465710","messageId":"xmqq5yg7q5jm.fsf@gitster.g","threadId":"58564","inReplyTo":"0e58bd35-4f40-ce9e-1088-f7c004527aee@github.com","subject":"Re: [PATCH v4 0/8] Trace2 timers and counters and some cleanup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-25T15:36:13Z","receivedAt":"2022-10-25T15:36:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n\n> On 10/24/2022 9:40 AM, Jeff Hostetler via GitGitGadget wrote:\n>> Here is version 4 of this series to add timers and counters to Trace2.\n>> \n>> Changes since V3:\n>> \n>>  * Fixed typo in the new thread-name documentation.\n>>  * Use a simpler NS_TO_SEC() macro for reporting the timer values.\n>> \n>> Jeff Hostetler (8):\n>>   trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx\n>>   tr2tls: clarify TLS terminology\n>>   api-trace2.txt: elminate section describing the public trace2 API\n>>   trace2: rename the thread_name argument to trace2_thread_start\n>>   trace2: improve thread-name documentation in the thread-context\n>>   trace2: convert ctx.thread_name from strbuf to pointer\n>>   trace2: add stopwatch timers\n>>   trace2: add global counter mechanism\n>\n> I re-read the series as well as looked at the range-diffs for the\n> previous two versions. I continue to think this is a high-quality\n> series and I've used it multiple times in my personal development\n> workflow to investigate certain performance things. I'm looking\n> forward to this being merged so we can all use it.\n\nI agree with your assessment.  Let's move it forward.\n\nThanks, all.\n"},{"id":"465711","messageId":"xmqq1qqvq5d2.fsf@gitster.g","threadId":"58564","inReplyTo":"a7b0b896-7b2b-a263-dd71-8b7b929707b4@github.com","subject":"Re: [PATCH v4 1/8] trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctx","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-10-25T15:40:09Z","receivedAt":"2022-10-25T15:40:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n\n> On 10/24/2022 4:31 PM, Junio C Hamano wrote:\n>> As I do not see a cover letter for this series, here is the summary\n>> of the change since the previous round that has been in 'seen'.\n>> \n>> I didn't see anything questionable in these.\n>> \n>> Thanks, will queue.\n>\n> The cover letter appears on my end, but I'm on the CC list.\n\nYeah, it seems vger was a bit constipated yesterday.  Everything\ncame through at the end, and I am happy with the series.\n\n> Jeff: be sure to CC Junio by adding him to the CC list on your\n> PR description for anything you want to have considered for\n> queuing.\n\nEverybody wants their non RFC patches to have considered for\nqueuing, but vger is not that lossy ;-)\n\n"}]}