git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Sensible way to see what objects are being fetched just-in-time in a partial clone?

From
Tao Klerks <tao@klerks.biz>
Date
Aug 26, 2024, 16:38 UTC
Message-ID
<CAPMMpog7=ZnhJhrgZFwzRZibLtK1-LyOhsrp5c4O85ocRFDZxw@mail.gmail.com>
Hi folks,

In working with Partial / Filtered Clone repos, there are situations where objects get fetched just-in-time - eg during a "git blame", if you did a "blob:none" filtered clone, you can easily end up with hundreds of fetches as git iterates backwards through the file history.

I was trying to write a "git blame optimizer" to pre-fetch all the suitable blobs, and it wasn't working right, so the "git blame" was still fetching stuff - but I couldn't see what it was fetching (which made it hard to investigate the bug in my script).

I did end up getting a list of some just-in-time fetched blobs, by dumping a list of *all* the object IDs I had locally, before and after a still-fetching-stuff "git blame" run, and doing a before/after comparison of the resulting list of objects. To get the list of objects found locally I did:

git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered

(ref: a conversation with Peff last year: https://lore.kernel.org/git/20230621064459.GA607974@coredump.intra.peff.net/ )

This was a sucky process though - and I was very surprised that I couldn't see what was being fetched (what the stdin content to the just-in-time fetch calls were) with any of the trace env vars that I was able to find documented: GIT_TRACE, GIT_CURL_VERBOSE, GIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET, GIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW

The only thing I could easily see were the *args* passed to nested git processes.

Is there any way to see what a just-in-time fetch is fetching? Or any way to see the content passed around on stdin in nested git processes?

Thanks, Tao

Next: Junio C Hamano
Message 1 of 4 in “Sensible way to see what objects are being fetched just-in-time in a partial clone?”
  1. Tao KlerksAug 26, 2024
  2. Junio C HamanoAug 26, 2024
  3. Tao KlerksAug 26, 2024
  4. Python-based fetch optimizer script for "blame" in Partial Clones (was: Re: Sensible way to see what objects are being fetched just-in-time in a partial clone?)Tao Klerks, Aug 26, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.