git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/2] Prevent re-reading 4 GiB files on every status

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
Oct 12, 2023, 16:09 UTC
Message-ID
<20231012160930.330618-1-sandals@crustytoothpaste.net>

Several people have noticed that Git continually re-reads and re-hashes files that are an exact multiple of 4 GiB every time the index is refreshed (for example, when "git status" is called). This is slow and expensive, especially with SHA-1 DC, and it also causes performance problems when these files are used with Git LFS (because the same issue occurs, just with Git LFS hashing the data instead of Git).

Jason Hatton sent a patch previously to fix this, but it lacked tests and didn't get picked up. I've adopted their patch, making some minor changes to the commit message and including some tests, and also including a suitable test helper to make the tests possible. All credit should be directed to Jason, and I'll accept all the responsibility for any problems.

I don't anticipate this being in any way controversial, so I'm not expecting a huge number of rerolls, but of course one or two might be necessary.

Jason Hatton (1):
  Prevent git from rehashing 4GiB files
brian m. carlson (1):
  t: add a test helper to truncate files
 Makefile                 |  1 +
 statinfo.c               | 20 ++++++++++++++++++--
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-truncate.c | 27 +++++++++++++++++++++++++++
 t/t7508-status.sh        | 16 ++++++++++++++++
 6 files changed, 64 insertions(+), 2 deletions(-)
 create mode 100644 t/helper/test-truncate.c
Next: brian m. carlson
Message 1 of 16 in “Prevent re-reading 4 GiB files on every status”
  1. 0/2 Prevent re-reading 4 GiB files on every statusbrian m. carlson, Oct 12, 2023
  2. 1/2 t: add a test helper to truncate filesbrian m. carlson, Oct 12, 2023
  3. Eric SunshineOct 12, 2023
  4. brian m. carlsonOct 13, 2023
  5. Junio C HamanoOct 12, 2023
  6. brian m. carlsonOct 13, 2023
  7. Junio C HamanoOct 13, 2023
  8. Jeff KingOct 16, 2023
  9. 2/2 Prevent git from rehashing 4GiB filesbrian m. carlson, Oct 12, 2023
  10. Junio C HamanoOct 12, 2023
  11. brian m. carlsonOct 12, 2023
  12. Junio C HamanoOct 12, 2023
  13. Jeff KingOct 17, 2023
  14. Jason HattonOct 17, 2023
  15. Junio C HamanoOct 17, 2023
  16. brian m. carlsonOct 18, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.