Re: [RFC PATCH] trace2: don't overload target directories

From: Jeff Hostetler <git@jeffhostetler.com>
Date: Jul 30, 2019, 18:00 UTC
Message-ID: <9ccf15ac-0e53-7044-e7b0-03bb229824d8@jeffhostetler.com>
In-Reply-To: <99e4a0fe409a236d210d95e54cd03fce61daa291.1564438745.git.steadmon@google.com>

On 7/29/2019 6:20 PM, Josh Steadmon wrote:

> trace2 can write files into a target directory. With heavy usage, this
> directory can fill up with files, causing difficulty for
> trace-processing systems.

I'm routing data in my org to a daemon via a Named Pipe or UD Socket, so I'm not seeing the thousands of files problems that you're seeing. However, we were being overwhelmed with lots of "uninteresting" commands and so I added some whitelisting to my post-processing daemon. For example, I want to know about checkout and push times -- I really don't care about rev-parse or config times or other such minor commands.

I went one step further and allow either "(cmd_name)" or the pair "(cmd_name, cmd_mode)". This lets me select all checkouts and limit checkouts to branch-changing ones, for example. I drop any events in my post-processor that does not match any of my whitelist patterns.

Perhaps you could run a quick histogram and see if something would be useful to pre-filter the data. That is, if we had whitelisting within git.exe itself, would you still have too much data and/or would you still need the overload feature that you've proposed in this RFC?

Thanks, Jeff