# Sensible way to see what objects are being fetched just-in-time in a partial clone? 4 messages from 2024-08-26 to 2024-08-26. Participants: Tao Klerks, Junio C Hamano. Thread: https://gitlist.dev/t/62006 ## Tao Klerks, 2024-08-26 16:38 Subject: Sensible way to see what objects are being fetched just-in-time in a partial clone? Message-ID: URL: https://gitlist.dev/e/CAPMMpog7%3DZnhJhrgZFwzRZibLtK1-LyOhsrp5c4O85ocRFDZxw%40mail.gmail.com ``` Hi folks, In working with Partial / Filtered Clone repos, there are situations where objects get fetched just-in-time - eg during a "git blame", if you did a "blob:none" filtered clone, you can easily end up with hundreds of fetches as git iterates backwards through the file history. I was trying to write a "git blame optimizer" to pre-fetch all the suitable blobs, and it wasn't working right, so the "git blame" was still fetching stuff - but I couldn't see what it was fetching (which made it hard to investigate the bug in my script). I did end up getting a list of some just-in-time fetched blobs, by dumping a list of *all* the object IDs I had locally, before and after a still-fetching-stuff "git blame" run, and doing a before/after comparison of the resulting list of objects. To get the list of objects found locally I did: git cat-file --batch-check='%(objectname)' --batch-all-objects --unordered (ref: a conversation with Peff last year: https://lore.kernel.org/git/20230621064459.GA607974@coredump.intra.peff.net/ ) This was a sucky process though - and I was very surprised that I couldn't see what was being fetched (what the stdin content to the just-in-time fetch calls were) with any of the trace env vars that I was able to find documented: GIT_TRACE, GIT_CURL_VERBOSE, GIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET, GIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW The only thing I could easily see were the *args* passed to nested git processes. Is there any way to see what a just-in-time fetch is fetching? Or any way to see the content passed around on stdin in nested git processes? Thanks, Tao ``` ## Junio C Hamano, 2024-08-26 17:28 Subject: Re: Sensible way to see what objects are being fetched just-in-time in a partial clone? Message-ID: URL: https://gitlist.dev/e/xmqqv7znjir3.fsf%40gitster.g In-Reply-To: ``` Tao Klerks writes: > This was a sucky process though - and I was very surprised that I > couldn't see what was being fetched (what the stdin content to the > just-in-time fetch calls were) with any of the trace env vars that I > was able to find documented: GIT_TRACE, GIT_CURL_VERBOSE, > GIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET, > GIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW Yeah, lazy fetch codepath seems to be, eh, not quite well polished yet. I am kind of surprised that there is no trace2() events around promisor_remote_get_direct() or its callers. Perhaps it is a good idea to add one to log how often it is triggered, and how large a batch the callers of the function is making? Unlike the diff machinery, blame does not have a prefetch machinery. I am glad that somebody is looking at it. ``` ## Tao Klerks, 2024-08-26 19:37 Subject: Re: Sensible way to see what objects are being fetched just-in-time in a partial clone? Message-ID: URL: https://gitlist.dev/e/CAPMMpoiPeSyLHUd072Q9AozdaCAjpOL03jY-0Y33B2iEQ5EdTQ%40mail.gmail.com In-Reply-To: ``` On Mon, Aug 26, 2024 at 7:28 PM Junio C Hamano wrote: > > > Unlike the diff machinery, blame does not have a prefetch machinery. > I am glad that somebody is looking at it. I'm not convinced I'm looking at it in a very useful way from your perspective: My C skills being spectacularly lacking for fixing the core code, I am just writing an external python-based wrapper for "my" users. I will try to "productize" it sufficiently to send here in case it's useful to someone, but all I can really offer the community-at-large is confirmation that in principle, the approach works as you would expect: With some small number of jit-fetches for rename-detection during the revision walk(s), and with one blob-prefetch call afterwards, "git blame" can be made to run cleanly/quickly in a "filter:none" clone even on a file like "git.c", with hundreds of revisions. ``` ## Tao Klerks, 2024-08-26 20:37 Subject: Python-based fetch optimizer script for "blame" in Partial Clones (was: Re: Sensible way to see what objects are being fetched just-in-time in a partial clone?) Message-ID: URL: https://gitlist.dev/e/CAPMMpojae7t1Z0PQPf%3Dd2WzzbaMwkhvo34b_sxdtUQxvDu4fJw%40mail.gmail.com In-Reply-To: ``` On Mon, Aug 26, 2024 at 9:37 PM Tao Klerks wrote: > > On Mon, Aug 26, 2024 at 7:28 PM Junio C Hamano wrote: > > > > > > Unlike the diff machinery, blame does not have a prefetch machinery. > > I am glad that somebody is looking at it. > > I will try to "productize" it sufficiently to send here in case it's > useful to someone, but all I can really offer the community-at-large > is confirmation that in principle, the approach works as you would > expect: With some small number of jit-fetches for rename-detection > during the revision walk(s), and with one blob-prefetch call > afterwards, "git blame" can be made to run cleanly/quickly in a > "filter:none" clone even on a file like "git.c", with hundreds of > revisions. FWIW, here is the script I ended up with, which seems to work reliably for me (through renames etc). Obviously I'd love to see this built into "git blame" itself, but this wrapper might help someone out there in the meantime. ```