Sensible way to see what objects are being fetched just-in-time in a partial clone?

Hi folks,

In working with Partial / Filtered Clone repos, there are situations where objects get fetched just-in-time - eg during a "git blame", if you did a "blob:none" filtered clone, you can easily end up with hundreds of fetches as git iterates backwards through the file history.

I was trying to write a "git blame optimizer" to pre-fetch all the suitable blobs, and it wasn't working right, so the "git blame" was still fetching stuff - but I couldn't see what it was fetching (which made it hard to investigate the bug in my script).

I did end up getting a list of some just-in-time fetched blobs, by dumping a list of *all* the object IDs I had locally, before and after a still-fetching-stuff "git blame" run, and doing a before/after comparison of the resulting list of objects. To get the list of objects found locally I did:

git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered

(ref: a conversation with Peff last year: https://lore.kernel.org/git/20230621064459.GA607974@coredump.intra.peff.net/ )

This was a sucky process though - and I was very surprised that I couldn't see what was being fetched (what the stdin content to the just-in-time fetch calls were) with any of the trace env vars that I was able to find documented: GIT_TRACE, GIT_CURL_VERBOSE, GIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET, GIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW

The only thing I could easily see were the *args* passed to nested git processes.

Is there any way to see what a just-in-time fetch is fetching? Or any way to see the content passed around on stdin in nested git processes?

Thanks, Tao

Tao Klerks <tao@klerks.biz> writes:

Yeah, lazy fetch codepath seems to be, eh, not quite well polished yet.

I am kind of surprised that there is no trace2() events around promisor_remote_get_direct() or its callers. Perhaps it is a good idea to add one to log how often it is triggered, and how large a batch the callers of the function is making?

Unlike the diff machinery, blame does not have a prefetch machinery. I am glad that somebody is looking at it.

On Mon, Aug 26, 2024 at 7:28 PM Junio C Hamano <gitster@pobox.com> wrote:

I'm not convinced I'm looking at it in a very useful way from your perspective: My C skills being spectacularly lacking for fixing the core code, I am just writing an external python-based wrapper for "my" users.

I will try to "productize" it sufficiently to send here in case it's useful to someone, but all I can really offer the community-at-large is confirmation that in principle, the approach works as you would expect: With some small number of jit-fetches for rename-detection during the revision walk(s), and with one blob-prefetch call afterwards, "git blame" can be made to run cleanly/quickly in a "filter:none" clone even on a file like "git.c", with hundreds of revisions.

On Mon, Aug 26, 2024 at 9:37 PM Tao Klerks <tao@klerks.biz> wrote:

FWIW, here is the script I ended up with, which seems to work reliably for me (through renames etc). Obviously I'd love to see this built into "git blame" itself, but this wrapper might help someone out there in the meantime.