git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [EXTERNAL SENDER] Suspected git grep regression in git 2.40.0

From
Stephane Odul <stephane@clumio.com>
Date
Mar 21, 2023, 20:46 UTC
Message-ID
<51078D7E-C325-4F57-96C1-601B4E102DD9@clumio.com>
In-Reply-To
<b0f4b588-9871-8e59-e5a2-3f8745a7c4cd@grsecurity.net>
Thank you for looking into this so quickly.
I’m unable to reproduce locally reliably but I created a custom pipeline to reproduce more quickly.
Here are the things I found out.
* With the NO_JIT flag and limited to only python files (in our case we only want to grep on py files anyways):
  - git grep -c -P '(*NO_JIT)^[[:alnum:]_]+ = json.load' -- '*.py’
  This is snappy and works, no more error.
* Without the flag and the *.py restriction:
  - git grep -c -P '^[^ #][^#]+sys[.]argv’
    This did not fail but took almost 3m, big performance regression.
  - git grep -c -P '^[[:alnum:]_]+ = json.load’
    Crashed and returned -11. Stderr was empty so I have no idea on what file it failed.
 * With NO_JIT on all the files:
  - git grep -c -P '(*NO_JIT)^[[:alnum:]_]+ = json.load’
   This worked, that pattern is snappy but other patterns are very slow:
  - git grep -c -P '(*NO_JIT)^[^ #][^#]+sys[.]argv’
    Took 8m to complete.
 * Without the flag but only *.py.
    - git grep -c -P '^[[:alnum:]_]+ = json.load' -- '*.py’
     All the patterns run fast (under 1s), and no errors.
Note that I was trying -E and replaced \w with [[:alnum:]_] … I’ll need to revert that, but I don’t thing \w is the issue.

Overall I would say that the issue is likely because the patterns are run against a non ASCII file somewhere in the repo. Our repo is fairly large with files in various formats, including potentially some binaries that would definitely not be proper UTF-8.

For now I have a good workaround which is to only check for *.py files, which we should have done in the first place. The NO_JIT flag slows down things significantly so we will not use it here.
Do you have any recommendation on how to identify which file(s) is causing the crash considering there is nothing in stderr?
Thanks!
Previous: Mathias KrauseNext: Mathias Krause
Message 5 of 11 in “Suspected git grep regression in git 2.40.0”
  1. Stephane OdulMar 21, 2023
  2. Bagas SanjayaMar 21, 2023
  3. Junio C HamanoMar 21, 2023
  4. Mathias KrauseMar 21, 2023
  5. Stephane OdulMar 21, 2023
  6. Mathias KrauseMar 22, 2023
  7. Stephane OdulMar 22, 2023
  8. Mathias KrauseMar 23, 2023
  9. Junio C HamanoMar 23, 2023
  10. Mathias KrauseMar 23, 2023
  11. grep: work around UTF-8 related JIT bug in PCRE2 <= 10.34Mathias Krause, Mar 23, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.