Re: Contributor Summit Topics and Logistics
- From
- Jeff Hostetler <git@jeffhostetler.com>
- Date
- Jan 30, 2019, 22:26 UTC
- Message-ID
- <e0173444-c5c4-61d0-5a3d-e37cbfd44bad@jeffhostetler.com>
- In-Reply-To
- <87va253lun.fsf@evledraar.gmail.com>
On 1/30/2019 3:57 PM, Ævar Arnfjörð Bjarmason wrote:
> > On Tue, Jan 22 2019, Jeff King wrote: >
...
Show 5 quoted lines
> * "Structured remote logging". We had an RFC spec for turning our trace > format into something more structural with a way to send it to a > remote server. There were both implementation & privacy concernse, > last time at least a couple of users of git reported having in-house > patches for this (not ready for upstream). Where are we on this now?
I won't be attending GitMerge this year, but I can talk about this work here.
My earlier "structured logging" and/or "telemetry" proposals have been replaced by my Trace2 patch series now in "pu".
The Trace2 feature is designed to report trace and performance data from within the git process to a local log file, unix domain socket, or Windows named pipe. Functions in the Trace2 API generate structured data and can write either structured (JSON) or non-structured formats to disk. (It should not be hard to add a binary structured format too, but that is beyond the scope of the current patch series.)
The JSON stream is suitable for post-processing by a local process. This can be a daemon listening to the stream or a cron job processing the trace data after the fact.
I consider it to be the job of the post-processor (after aggregating, filtering or whatever) to decide what to do with the data. This lets the the user and/or sysadmin control how and when data is collected. The post-processor is free to hook into something like syslog or ETW or write to a custom DB.
Post-processing tools are not included in the patch series.
Internally within Microsoft, we have a local Windows Service listening on a named pipe and collecting events from all git processes for our GVFS users in the Windows OS repo. It computes a summary record for each git command, for example combining the argv from the "start" event with the elapsed time from the "exit" event into a single record. The service then sends the aggregate records to a centralized database.
This lets us run various database queries to try to understand pain points that our OS developers are experiencing (and that may not show up on my machine) and help us prioritize future perf and scaling work.
But again, this service is but one possible post-processor and is for internal-use-only.
The Trace2 feature itself does not have any remote capability. It just writes data locally.
Jeff