git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 7/9] trace2: add stopwatch timers

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Dec 22, 2021, 21:57 UTC
Message-ID
<9316427d-56a1-50bc-c88e-fda043b88415@jeffhostetler.com>
In-Reply-To
<5f9c6997-abae-c2c7-0121-eb2d360108b8@gmail.com>
On 12/21/21 9:45 AM, Derrick Stolee wrote:
> On 12/20/2021 10:01 AM, Jeff Hostetler via GitGitGadget wrote:
>> From: Jeff Hostetler <jeffhost@microsoft.com>
>>
>> Add a stopwatch timer mechanism to Git.
[...]
Show 7 quoted lines
> 
>> +# Exercise the stopwatch timer "test" in a loop and confirm that it was
>> +# we have as many start/stop intervals as expected.  We cannot really test
>> +# the (elapsed, min, max) timer values, so we assume they are good.
> 
> We can't check their values, but we could check that their labels are
> emitted.
good point.  i'll add that to the patterns in the grep.
> 
[...]
Show 18 quoted lines
>> +	grep "\"event\":\"timer\".*\"thread\":\"${thread}\".*\"name\":\"${name}\".*\"count\":${count}" $file
> 
> Adding more detail to this pattern.
> 
> This helper could probably benefit from constructing the regex across
> multiple string concatenations, so we can see the different pieces.
> Something like
> 
> 	pattern="\"event\":\"timer\""
> 	pattern="$pattern.*\"thread\":\"${thread}\""
> 	pattern="$pattern.*\"name\":\"${name}\""
> 	pattern="$pattern.*\"count\":\"${count}\""
> 	pattern="$pattern.*\"t_total\":"
> 	pattern="$pattern.*\"t_min\":"
> 	pattern="$pattern.*\"t_max\":"
> 
> 	grep "$pattern" $file
> 
yeah, that helps a lot.  thanks.
[...]
Show 12 quoted lines
>> +/*
>> + * Define the set of stopwatch timers.
>> + *
>> + * We can add more at any time, but they must be defined at compile
>> + * time (to avoid the need to dynamically allocate and synchronize
>> + * them between different threads).
>> + *
>> + * These must start at 0 and be contiguous (because we use them
>> + * elsewhere as array indexes).
> 
> I was worried at first about using an array here, but this is essentially
> one chunk of global memory per process that will not be very large, even
s/process/thread/
> if we add a lot of timer IDs here. If we use this API enough that that
> memory is a problem, then we can refactor the memory to be a hashmap that
> only populates entries for IDs that are used by the process.

we're only talking about 48 bytes per timer being added to the thread context. and it is allocated, not stack based, so i'm not worried about it.

and besides, we get constant time lookups when starting/stopping a timer. And when we get ready to sum across the thread pool, we can do it efficiently.

Show 27 quoted lines
> 
>> + * Any values added to this enum must also be added to the timer definitions
>> + * array.  See `trace2/tr2_tmr.c:tr2tmr_def_block[]`.
>> + */
>> +enum trace2_timer_id {
>> +	/*
>> +	 * Define two timers for testing.  See `t/helper/test-trace2.c`.
>> +	 * These can be used for ad hoc testing, but should not be used
>> +	 * for permanent analysis code.
>> +	 */
>> +	TRACE2_TIMER_ID_TEST1 = 0, /* emits summary event only */
>> +	TRACE2_TIMER_ID_TEST2,     /* emits summary and thread events */
>> +
>> +
>> +	/* Add additional timer definitions before here. */
>> +	TRACE2_NUMBER_OF_TIMERS
>> +};
> 
> ....
> 
>> +static struct tr2tmr_def tr2tmr_def_block[TRACE2_NUMBER_OF_TIMERS] = {
>> +	[TRACE2_TIMER_ID_TEST1] = { "test", "test1", 0 },
>> +	[TRACE2_TIMER_ID_TEST2] = { "test", "test2", 1 },
>> +};
> 
> Although this will always be populated, so maybe my thoughts about how
> to reduce memory load in the hypothetical future are worthless.
yeah, i think this model works well for us.  and it is lock free.
Show 20 quoted lines
> 
>> +void tr2tmr_start(enum trace2_timer_id tid)
>> +{
>> +	struct tr2tls_thread_ctx *ctx = tr2tls_get_self();
>> +	struct tr2tmr_timer *t = &ctx->timers.timer[tid];
>> +
>> +	t->recursion_count++;
>> +	if (t->recursion_count > 1)
>> +		return; /* ignore recursive starts */
>> +
>> +	t->start_us = getnanotime() / 1000;
> 
> Using nanotime gives us the best precision available, and dividing
> by 1000 will lose some precision. This is likely why we saw some
> 0.000000 values for t_min in some of your experiments. That should
> be rare for real uses of this API (such as wrapping lstat() calls).
> 
> But why do we divide by 1000 here at all? 2^63 nanoseconds is
> still 292 years, so we don't risk overflow. You specify uint64_t
> so this isn't different on 32-bit machines.

When I did the original Trace2 parts, I made absolute and relative elapsed times be in microseconds. With the overhead of logging and etc, the lower bits weren't really useful. And then I converted those to "%9.6f" in the trace logs, so that we always have "seconds" in the traces.

I just copied that model when I did timers. But I could see keeping nanoseconds around for these timers (since they don't log on every start/stop, like regions do).

<grin> While drafting this reply I've been fixing up the code in parallel. I converted timers to report integer ns values rather than floats. And in every line of trace output the total/min/max timer values all end in 000 -- because (at least on MacOS) getnanotime() calls gettimeofday() and computes (tv.tv_sec * 1000000000 + tv.tv_usec). </grin>

So maybe I did the original "%9.6" for other reasons.... I'll try it on another OS later and see if it is useful.

Thanks, Jeff

Previous: Derrick StoleeNext: Jeff Hostetler via GitGitGadget
Message 33 of 55 in “Trace2 stopwatch timers and global counters”
  1. 0/9 Trace2 stopwatch timers and global countersJeff Hostetler via GitGitGadget, Dec 20, 2021
  2. 1/9 trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctxJeff Hostetler via GitGitGadget, Dec 20, 2021
  3. 2/9 trace2: convert tr2tls_thread_ctx.thread_name from strbuf to char*Jeff Hostetler via GitGitGadget, Dec 20, 2021
  4. Ævar Arnfjörð BjarmasonDec 20, 2021
  5. Jeff HostetlerDec 20, 2021
  6. Ævar Arnfjörð BjarmasonDec 20, 2021
  7. Jeff HostetlerDec 22, 2021
  8. Junio C HamanoDec 21, 2021
  9. Junio C HamanoDec 21, 2021
  10. Jeff HostetlerDec 22, 2021
  11. Junio C HamanoDec 22, 2021
  12. 3/9 trace2: defer free of TLS CTX until program exit.Jeff Hostetler via GitGitGadget, Dec 20, 2021
  13. Junio C HamanoDec 21, 2021
  14. Jeff HostetlerDec 22, 2021
  15. Junio C HamanoDec 22, 2021
  16. Jeff HostetlerDec 22, 2021
  17. Johannes SixtDec 23, 2021
  18. Junio C HamanoDec 23, 2021
  19. Jeff HostetlerDec 27, 2021
  20. 4/9 trace2: add thread-name override to event targetJeff Hostetler via GitGitGadget, Dec 20, 2021
  21. 5/9 trace2: add thread-name override to perf targetJeff Hostetler via GitGitGadget, Dec 20, 2021
  22. 6/9 trace2: add timer events to perf and event target formatsJeff Hostetler via GitGitGadget, Dec 20, 2021
  23. Ævar Arnfjörð BjarmasonDec 20, 2021
  24. Jeff HostetlerDec 20, 2021
  25. Derrick StoleeDec 21, 2021
  26. 8/9 trace2: add counter events to perf and event target formatsJeff Hostetler via GitGitGadget, Dec 20, 2021
  27. Ævar Arnfjörð BjarmasonDec 20, 2021
  28. Jeff HostetlerDec 22, 2021
  29. 7/9 trace2: add stopwatch timersJeff Hostetler via GitGitGadget, Dec 20, 2021
  30. Ævar Arnfjörð BjarmasonDec 20, 2021
  31. Jeff HostetlerDec 22, 2021
  32. Derrick StoleeDec 21, 2021
  33. Jeff HostetlerDec 22, 2021
  34. 9/9 trace2: add global countersJeff Hostetler via GitGitGadget, Dec 20, 2021
  35. Ævar Arnfjörð BjarmasonDec 20, 2021
  36. Jeff HostetlerDec 22, 2021
  37. Derrick StoleeDec 21, 2021
  38. Matheus TavaresDec 21, 2021
  39. 0/9 Trace2 stopwatch timers and global countersJeff Hostetler via GitGitGadget, Dec 28, 2021
  40. 1/9 trace2: use size_t alloc,nr_open_regions in tr2tls_thread_ctxJeff Hostetler via GitGitGadget, Dec 28, 2021
  41. Ævar Arnfjörð BjarmasonDec 29, 2021
  42. 2/9 trace2: convert tr2tls_thread_ctx.thread_name from strbuf to flex arrayJeff Hostetler via GitGitGadget, Dec 28, 2021
  43. Ævar Arnfjörð BjarmasonDec 29, 2021
  44. Jeff HostetlerDec 29, 2021
  45. 3/9 trace2: defer free of thread local storage until program exit.Jeff Hostetler via GitGitGadget, Dec 28, 2021
  46. 4/9 trace2: add thread-name override to event targetJeff Hostetler via GitGitGadget, Dec 28, 2021
  47. 7/9 trace2: add stopwatch timersJeff Hostetler via GitGitGadget, Dec 28, 2021
  48. 5/9 trace2: add thread-name override to perf targetJeff Hostetler via GitGitGadget, Dec 28, 2021
  49. Ævar Arnfjörð BjarmasonDec 29, 2021
  50. Jeff HostetlerDec 29, 2021
  51. 6/9 trace2: add timer events to perf and event target formatsJeff Hostetler via GitGitGadget, Dec 28, 2021
  52. 8/9 trace2: add counter events to perf and event target formatsJeff Hostetler via GitGitGadget, Dec 28, 2021
  53. 9/9 trace2: add global countersJeff Hostetler via GitGitGadget, Dec 28, 2021
  54. Ævar Arnfjörð BjarmasonDec 29, 2021
  55. Jeff HostetlerDec 30, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.