{"thread":{"id":"56193","subject":"git blame --ignore-rev degenerate performance on large(r) line counts","startedAt":"2021-08-01T12:59:07Z","lastAt":"2021-08-01T12:59:07Z","messageCount":1,"participants":["Stefan Hoffmeister"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"431653","messageId":"CALhB_QNf98O=kOnZ=rmUt90SmqZzxRF2CfyYOKiiJv+Qa2abqg@mail.gmail.com","threadId":"56193","inReplyTo":null,"subject":"git blame --ignore-rev degenerate performance on large(r) line counts","fromName":"Stefan Hoffmeister","fromEmail":"stefan.hoffmeister@econos.de","sentAt":"2021-08-01T12:58:51Z","receivedAt":"2021-08-01T12:59:07Z","isPatch":false,"sender":{"key":"stefan.hoffmeister@econos.de","avatar":null},"body":"git blame has a very useful option --ignore-rev (or --ignore-rev-file)\nwhich allows ignoring of commits for the purpose of attributing blame.\n\nOne use case for this option is a scenario where a repository was\n\"cleaned\" up for line breaks, e.g. via \"git add --renormalize .\"\n\nAlas, git blame with --ignore-rev will fall apart at runtime whenever\nthe number of lines of code affected is high; it will show exponential\nincreases in execution duration on linearly increasing line counts.\n\nThe Python script below demonstrates the undesirable performance\nbehaviour for a single text file and just two commits. In a real world\nscenario, with a repository-wide line-ending fix commit affecting a\nlarge number of _files_ (in addition to some files being large),\nperformance behaviour will be even worse.\n\nThe Python script below will initialize a new repo and do the\nreproduction dance fully isolated and offline; \"matplotlib\" as a\ndependency (pip install matplotlib) is a nice optional add-on to have,\nas it visualizes that on my system (i7-7820HQ, 2.9 GHz) runtime\nduration explodes exponentially at around 35000 lines of \"Hello Hello\"\ntext.\n\n****************\n#!/usr/bin/env bash\n\n# Beware - no error handling\n\nimport matplotlib.pyplot as plt\nimport os\nimport shutil\nimport subprocess\nimport time\nfrom typing import Generator\n\nREPO_LOCATION=\"git-repository-container/git-repo\"\nBLAME_ME_TXT=\"blame-me.txt\"\n\ndef run_iteration(line_count: int):\n    shutil.rmtree(REPO_LOCATION, ignore_errors=True)\n    os.makedirs(REPO_LOCATION, exist_ok=False)\n\n    os.chdir(REPO_LOCATION)\n    subprocess.run([\"git\", \"init\"], capture_output=True)\n\n    subprocess.run([\"git\", \"config\", \"--local\", \"user.name\", \"me\"],\ncapture_output=True)\n    subprocess.run([\"git\", \"config\", \"--local\", \"user.email\",\n\"me@example.com\"], capture_output=True)\n\n    with open(\".gitattributes\", \"w\") as gitattributes:\n        gitattributes.writelines(\"* text=crlf\\n\")\n\n    def produce_some_file_content_sequence(line_count: int) ->\nGenerator[str, None, None]:\n        mytext: str = \"Hello Hello\\r\\n\"\n        num = 0\n        while num < line_count:\n            yield mytext\n            num += 1\n\n    with open(BLAME_ME_TXT, \"w\") as blame_me:\n        for line in produce_some_file_content_sequence(line_count):\n            blame_me.writelines(line)\n\n    subprocess.run([\"git\", \"add\", \".\"], capture_output=True)\n\n    subprocess.run([\"git\", \"commit\", \"-m\", \"Initial commit\"],\ncapture_output=True)\n\n    capture = subprocess.run([\"git\", \"rev-list\", \"HEAD\"], capture_output=True)\n    initial_commit_rev = (capture.stdout).decode().strip()\n\n    with open(\".gitattributes\", \"w\") as gitattributes:\n        gitattributes.writelines(\"* text=auto\\n\")\n\n    subprocess.run([\"git\", \"add\", \"--renormalize\", \".\"], capture_output=True)\n\n    subprocess.run([\"git\", \"commit\", \"-m\", \"Renormalized\"], capture_output=True)\n\n    capture = subprocess.run([\"git\", \"rev-list\", \"--max-count\", \"1\",\n\"HEAD\"], capture_output=True)\n    renormalized_commit_rev = (capture.stdout).decode().strip()\n\n    blame_capture = subprocess.run([\"git\", \"blame\", \"-C ./\", \"--line-porcelain\",\n        \"--ignore-rev\", renormalized_commit_rev,\n        \"HEAD\", BLAME_ME_TXT],\n        capture_output=True)\n\nx=list[int]()\ny=list[float]()\nfor factor in range(40):\n    start = time.time()\n    line_count = 1000 * factor\n    x.append(line_count)\n\n    print(f'Line count: {line_count}')\n    run_iteration(line_count)\n\n    end = time.time()\n    duration = end - start\n    print(duration)\n    y.append(duration)\n\nplt.plot(x, y)\nplt.show()\n"}]}