git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 19/21] usage API: use C99 macros for {usage,usagef,die,error,warning,die}*()

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Dec 28, 2021, 16:32 UTC
Message-ID
<9952005b-9174-7578-7718-e9576b27b4ce@jeffhostetler.com>
In-Reply-To
<211228.861r1xk40d.gmgdl@evledraar.gmail.com>
On 12/27/21 6:01 PM, Ævar Arnfjörð Bjarmason wrote:
Show 55 quoted lines
> 
> On Mon, Dec 27 2021, Jeff Hostetler wrote:
> 
>> On 11/15/21 5:18 PM, Ævar Arnfjörð Bjarmason wrote:
>> [...]
>>> It might be a good change to remove the "fmt" key from the "error"
>>> events as a follow-up change. As these few examples from running the
>>> test suite show it's sometimes redundant (same as the "msg"), rather
>>> useless (just a "%s"), or something we could now mostly aggregate by
>>> file/line instead of the normalized printf format:
>>>         1 file":"builtin/gc.c","line":1391,"msg":"'bogus' is not a
>>> valid task","fmt":"'%s' is not a valid task"}
>>>         1 file":"builtin/for-each-ref.c","line":89,"msg":"format: %(then) atom used more than once","fmt":"%s"}
>>>         1 file":"builtin/fast-import.c","line":411,"msg":"Garbage after mark: N :202 :302x","fmt":"Garbage after mark: %s"}
>>> "Mostly" here assumes that it would be OK if the aggregation changed
>>> between git versions, which may be what all users of trace2 want. The
>>> change that introduced the "fmt" code was ee4512ed481 (trace2: create
>>> new combined trace facility, 2019-02-22), and the documentation change
>>> was e544221d97a (trace2: Documentation/technical/api-trace2.txt,
>>> 2019-02-22).
>>> Both are rather vague on what problem "fmt" solved exactly, aside
>>> from
>>> the obvious one of it being impossible to do meaningful aggregations
>>> due to the "file" and "line" being the same everywhere, which isn't
>>> the case now.
>>> In any case, let's leave "fmt" be for now, the above summary was
>>> given
>>> in case it's interesting to remove it in the future, e.g. to save
>>> space in trace2 payloads.
>>
>> I added the "fmt" field so that we could do aggregations
>> of error messages across multiple users without regard
>> to what branch or filename or percentage or whatever was
>> formatted into the actual "msg" written to stderr.
>>
>> The actual file:line wasn't useful (primarily because it
>> was probably something in usage.c), but even if we fix that
>> it might not be useful if we have users running 10 different
>> versions of Git (because some people don't upgrade immediately).
>>
>> So I'd rather not kill it right now.
> 
> Thanks. I'm not trying to kill it, but just poking at what it was for
> exactly.
> 
> Depending on the answer to that perhaps we didn't need it anymore, but
> the explanation you provide (mostly) makes sense.
> 
> The "mostly" being because I'm assuming that you only need to deal with
> LC_ALL=C users?
> 
> I.e. the documented promise that you can group things by "fmt" doesn't
> hold if you're processing even streams from users who are using a
> translated git, because we'll get the translated format string, not the
> original.

I just did a query on the data we've collected over the last few weeks and there are only English error messages in the database, so yes LC_ALL=C seems to be the norm.

Show 5 quoted lines
> 
> For that we'd need to change the API from/to to:
> 
>      - error(_("some format %s"), ...)
>      + error(N_("some format %s"), ...)

So no, I don't think it is worth the complexity to change this. Besides, wouldn't you need to more machinery under the hood -- to emit the untranslated string to trace2 and the translated string to stderr? (As in, move the translation down a layer??)

My "fmt" field is not worth that effort.

And besides, my goal was only to get the "top 10 or 20 errors" across a large set of users. I guess I'm thinking of it as a sample rather than an exhaustive list, so it is OK if we don't capture the translated strings.

Something else to consider is the GDPR. The "fmt" string is generic (e.g. "path '%s' exists on disk, but not in the index") but doesn't leak an PII or otherwise sensitive data. Whereas the corresponding "msg" field does include the pathname in this example. So if someone is post-processing the data and aggregating, they may want to relay only the "fmt" field and not the "msg" field to their database.

(Granted, there are lots of PII and GDPR problematic fields in the data stream that a post-processor would need to be aware of, but all of that is outside of the scope of the Trace2 logging. I only mention it here because the "fmt" field may be useful for reasons not previously discussed.)

Show 38 quoted lines
> 
> So being able to say "just group on file/line" would be simpler.
> 
> And also "mostly" because the "fmt" case also won't handle these and
> other duplicate formats (but maybe you haven't run into them in
> practice):
> 
>      $ git grep -E '\b(usage|die|error|warning)(_errno)?\("%s\"' -- '*.[ch]' | wc -l
>      90
> 
> So I was somewhat hoping for future work that you'd be OK with the new
> file/line grouping.
> 
> Because keeping "fmt" would eventually need some massive coccinelle
> search/replacement for "_(...)" -> "N_(...)" per the above, even then
> consumers of the stream would get duplicate grouping for the likes of
> "%s".
> 
> Do you think if as a follow-up we had "__func__"[1] along with
> "file/line" that the "file/__func__" combination would be good enough?
> The advantage of that would be that we could punt that "fmt"
> change/complexity and document:
> 
>      If you'd like to group errors the "file/line" pair will be unique
>      enough within a given git version to do so (sans a few codepaths that
>      relay errors from elsewhere).
> 
>      If you'd like a semi-stable grouping across similar git versions the
>      "file/func" pair should be Good Enough for most purposes. Some functions
>      might emit multiple errors, but you'd probably want to group them as
>      similar enough anyway.
> 
> But I realize that those things don't give you exactly the same things
> that "fmt" does, but maybe they're good enough (or even better?), or
> not.
> 
> 1. https://gcc.gnu.org/onlinedocs/gcc/Function-Names.html
> 

I'll have to think about this some and get back to you. Jeff

Previous: Ævar Arnfjörð BjarmasonNext: Elijah Newren
Message 19 of 34 in “C99: show meaningful <file>:<line> in trace2 via macros”
  1. 00/21 C99: show meaningful <file>:<line> in trace2 via macrosÆvar Arnfjörð Bjarmason, Nov 15, 2021
  2. 01/21 git-compat-util.h: clarify GCC v.s. C99-specific in commentÆvar Arnfjörð Bjarmason, Nov 15, 2021
  3. 03/21 usage.c: add a die_message() routineÆvar Arnfjörð Bjarmason, Nov 15, 2021
  4. 02/21 C99 support: hard-depend on C99 variadic macrosÆvar Arnfjörð Bjarmason, Nov 15, 2021
  5. 04/21 usage.c API users: use die_message() where appropriateÆvar Arnfjörð Bjarmason, Nov 15, 2021
  6. 05/21 usage.c + gc: add and use a die_message_errno()Ævar Arnfjörð Bjarmason, Nov 15, 2021
  7. 07/21 common-main.c: call exit(), don't returnÆvar Arnfjörð Bjarmason, Nov 15, 2021
  8. 06/21 config API: don't use vreportf(), make it static in usage.cÆvar Arnfjörð Bjarmason, Nov 15, 2021
  9. 08/21 usage.c: add a non-fatal bug() function to go with BUG()Ævar Arnfjörð Bjarmason, Nov 15, 2021
  10. 10/21 receive-pack: use bug() and BUG_if_bug()Ævar Arnfjörð Bjarmason, Nov 15, 2021
  11. 11/21 cache-tree.c: use bug() and BUG_if_bug()Ævar Arnfjörð Bjarmason, Nov 15, 2021
  12. 12/21 pack-objects: use BUG(...) not die("BUG: ...")Ævar Arnfjörð Bjarmason, Nov 15, 2021
  13. 17/21 usage.c: move usage routines aroundÆvar Arnfjörð Bjarmason, Nov 15, 2021
  14. 21/21 usage API: add "core.usageAddSource" config to add <file>:<line>Ævar Arnfjörð Bjarmason, Nov 15, 2021
  15. 20/21 usage API: make the "{usage,fatal,error,warning,BUG}: " translatableÆvar Arnfjörð Bjarmason, Nov 15, 2021
  16. 19/21 usage API: use C99 macros for {usage,usagef,die,error,warning,die}*()Ævar Arnfjörð Bjarmason, Nov 15, 2021
  17. Jeff HostetlerDec 27, 2021
  18. Ævar Arnfjörð BjarmasonDec 27, 2021
  19. Jeff HostetlerDec 28, 2021
  20. Elijah NewrenDec 28, 2021
  21. Ævar Arnfjörð BjarmasonDec 28, 2021
  22. Elijah NewrenDec 29, 2021
  23. Ævar Arnfjörð BjarmasonDec 28, 2021
  24. Jeff HostetlerDec 29, 2021
  25. 18/21 usage.c: move rename variables in usage routines aroundÆvar Arnfjörð Bjarmason, Nov 15, 2021
  26. 16/21 usage.[ch] API: rename "warn" vars functions to "warning"Ævar Arnfjörð Bjarmason, Nov 15, 2021
  27. 14/21 usage API: create a new usage.h, move API docs thereÆvar Arnfjörð Bjarmason, Nov 15, 2021
  28. 15/21 usage.[ch] API users: use report_fn, not hardcoded prototypeÆvar Arnfjörð Bjarmason, Nov 15, 2021
  29. 13/21 strbuf.h: use BUG(...) not die("BUG: ...")Ævar Arnfjörð Bjarmason, Nov 15, 2021
  30. 09/21 parse-options.[ch] API: use bug() to improve error outputÆvar Arnfjörð Bjarmason, Nov 15, 2021
  31. Taylor BlauNov 16, 2021
  32. Ævar Arnfjörð BjarmasonNov 16, 2021
  33. Taylor BlauNov 16, 2021
  34. Ævar Arnfjörð BjarmasonNov 16, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.