git/list[1] front-page[2] threads[3] people[4] search[5] about
 

git-blame extremely slow in partial clones due to serial object fetching

From
BLBurke Libbey <burke.libbey@shopify.com>
Date
Nov 19, 2024, 20:16 UTC
Message-ID
<B010051F-B182-4DB8-9469-AA2F53781968@shopify.com>

When running git-blame in a partial clone (--filter=blob:none), it fetches missing blob objects one at a time. This can result in thousands of serial fetch operations, making blame extremely slow, regardless of network latency.

For example, in one large repository, blaming a single large file required fetching about 6500 objects. Each fetch requiring a round-trip means this operation would have taken something on the order of an hour to complete.

The core issue appears to be in fill_origin_blob(), which is called individually for each blob needed during the blame process. While the blame algorithm does need blob contents to make detailed line-matching decisions, it seems like we don't necessarily need the contents just to determine which blobs we'llexamine.

It seems like this could be optimized by batch-fetching the needed objects upfront, rather than fetching them one at a time. This would convert O(n) round-trips into a small number of batch fetches.

Reproduction:
1. Create a partial clone with --filter=blob:none
2. Run git blame on a file with significant history
3. Observe serial fetching of objects in the trace output
Let me know if you need any additional information to investigate this issue.
—burke
Next: Manoraj K
Message 1 of 10 in “git-blame extremely slow in partial clones due to serial object fetching”
  1. Burke LibbeyNov 19, 2024
  2. Manoraj KNov 20, 2024
  3. Jonathan TanNov 20, 2024
  4. Junio C HamanoNov 20, 2024
  5. Han YoungNov 21, 2024
  6. Shubham KanodiaNov 22, 2024
  7. Junio C HamanoNov 22, 2024
  8. Shubham KanodiaNov 22, 2024
  9. Jonathan TanNov 22, 2024
  10. Junio C HamanoNov 25, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.