{"thread":{"id":"60282","subject":"git grep performance regression on macOS","startedAt":"2023-09-29T23:57:00Z","lastAt":"2023-10-02T03:08:01Z","messageCount":3,"participants":["Benjamin Hiller","Junio C Hamano","Carlo Marcelo Arenas Bel'on"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482463","messageId":"CAPWWTaDE5559vA1qa0zhBid_ep9ht+PxPSDS5YC7Dk0NN8sp9A@mail.gmail.com","threadId":"60282","inReplyTo":null,"subject":"git grep performance regression on macOS","fromName":"Benjamin Hiller","fromEmail":"benhiller@gmail.com","sentAt":"2023-09-29T23:56:19Z","receivedAt":"2023-09-29T23:57:00Z","isPatch":false,"sender":{"key":"benhiller@gmail.com","avatar":null},"body":"What did you do before the bug happened? (Steps to reproduce your issue)\n\ngit grep seems to have gotten much slower as of git 2.39 on macOS for\ncomplex extended regexes.\nWe noticed this because git secrets --scan was running much more\nslowly for some people on our team, and eventually realized that it\nwas due to them using a newer version of git. git secrets runs a git\ngrep command with an extended regex (this is a somewhat simplified\nversion of the command, but still shows the performance issue):\n\ngit grep -E \"(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}|(\\\"|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)(\\\"|')?\\s*(:|=>|=)\\s*(\\\"|')?[A-Za-z0-9/\\+=]{40}(\\\"|')?|(\\\"|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?(\\\"|')?\\s*(:|=>|=)\\s*(\\\"|')?[0-9]{4}\\-?[0-9]{4}\\-?[0-9]{4}(\\\"|')?\"\n\nWhat did you expect to happen? (Expected behavior)\nWith git 2.38, that command took under half a second to run on a large repo.\nUsing the git (https://github.com/git/git) repo as an example, it took\n0.2s on my laptop.\n\nWhat happened instead? (Actual behavior)\nAfter 2.39, it now takes over 40 seconds on my laptop with the git repo!\n\nWhat's different between what you expected and what actually happened?\nThe command runs much more slowly, though it still does return the\ncorrect result.\n\nAnything else you want to add:\nI confirmed that the performance regression was first introduced in\n2.39. Additionally, I saw that reverting the change to Makefile from\nhttps://github.com/git/git/commit/1819ad327b7a1f19540a819813b70a0e8a7f798f\nfixed the performance regression and the git grep command went back to\ntaking <1 second. That seems to indicate that switching from Git's\nregex library to the native macOS regex library caused this\nperformance regression, but I haven't investigated beyond that to see\nwhy the native macOS regex library is so much slower.\n\nPlease review the rest of the bug report below.\nYou can delete any lines you don't wish to share.\n\n\n[System Info]\ngit version:\ngit version 2.42.0\ncpu: arm64\nno commit associated with this build\nsizeof-long: 8\nsizeof-size_t: 8\nshell-path: /bin/sh\nfeature: fsmonitor--daemon\nuname: Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:41\nPST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T8103 arm64\ncompiler info: clang: 14.0.3 (clang-1403.0.22.14.1)\nlibc info: no libc information available\n$SHELL (typically, interactive shell): /bin/zsh\n\n\n[Enabled Hooks]\npost-checkout\npost-merge\npre-commit\npre-push\n"},{"id":"482464","messageId":"xmqqa5t4gqnd.fsf@gitster.g","threadId":"60282","inReplyTo":"CAPWWTaDE5559vA1qa0zhBid_ep9ht+PxPSDS5YC7Dk0NN8sp9A@mail.gmail.com","subject":"Re: git grep performance regression on macOS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-30T05:45:58Z","receivedAt":"2023-09-30T05:46:18Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Benjamin Hiller <benhiller@gmail.com> writes:\n\n> git grep seems to have gotten much slower as of git 2.39 on macOS for\n> complex extended regexes.\n\n> I confirmed that the performance regression was first introduced in\n> 2.39. Additionally, I saw that reverting the change to Makefile from\n> https://github.com/git/git/commit/1819ad327b7a1f19540a819813b70a0e8a7f798f\n> fixed the performance regression and the git grep command went back to\n> taking <1 second. That seems to indicate that switching from Git's\n> regex library to the native macOS regex library caused this\n> performance regression, but I haven't investigated beyond that to see\n> why the native macOS regex library is so much slower.\n\nYeah, that does sound a plausible explanation.\n\nThe regexp code in compat/ is meant as a fallback implementation for\nplatforms whose regexp library lack certain features we take\nadvantage of, but it has a limitation that it is not unicode aware.\nIn the olden days, regexp library on macOS lacked REG_STARTEND\nfeature, which forced us to use NO_REGEX (hence the fallback\nimplementation we ship that is not unicode aware).  The commit you\ncite makes us use the macOS native regexp library, as somebody on\nthe platform got annoyed enough by the lack of unicode awareness of\nthe fallback implementation, and also noticed that REG_STARTEND is\nsupported by the macOS native regexp library these days.\n\nThe change in 2.39 was unfortunately about correctness.  It would\nhave been nicer if macOS native implementation were faster, but use\nof fallback implementation would be favoring \"performance\" (which\nproduces incorrect results \"faster\" when run with multi-byte\nstrings) over correctness, so it is not likely that a straight\nreverting of the commit is a good idea.\n\n"},{"id":"482517","messageId":"dw26ovefqhtvpdxdrj6rlzmosi5wg2avhsurh452jku3vf2ox3@fjkysuygd4ej","threadId":"60282","inReplyTo":"CAPWWTaDE5559vA1qa0zhBid_ep9ht+PxPSDS5YC7Dk0NN8sp9A@mail.gmail.com","subject":"Re: git grep performance regression on macOS","fromName":"Carlo Marcelo Arenas Bel'on","fromEmail":"carenas@gmail.com","sentAt":"2023-10-02T03:05:59Z","receivedAt":"2023-10-02T03:08:01Z","isPatch":false,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Fri, Sep 29, 2023 at 04:56:19PM -0700, Benjamin Hiller wrote:\n> \n> git grep -E \"(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}|(\\\"|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)(\\\"|')?\\s*(:|=>|=)\\s*(\\\"|')?[A-Za-z0-9/\\+=]{40}(\\\"|')?|(\\\"|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?(\\\"|')?\\s*(:|=>|=)\\s*(\\\"|')?[0-9]{4}\\-?[0-9]{4}\\-?[0-9]{4}(\\\"|')?\"\n\nchanging this code to use `git grep -P` instead will make it at least 7x\nfaster even if you have a pcre2 library without JIT.\n\nCarlo\n"}]}