{"thread":{"id":"62006","subject":"Sensible way to see what objects are being fetched just-in-time in a partial clone?","startedAt":"2024-08-26T16:38:43Z","lastAt":"2024-08-26T20:37:51Z","messageCount":4,"participants":["Tao Klerks","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"501675","messageId":"CAPMMpog7=ZnhJhrgZFwzRZibLtK1-LyOhsrp5c4O85ocRFDZxw@mail.gmail.com","threadId":"62006","inReplyTo":null,"subject":"Sensible way to see what objects are being fetched just-in-time in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2024-08-26T16:38:29Z","receivedAt":"2024-08-26T16:38:43Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nIn working with Partial / Filtered Clone repos, there are situations\nwhere objects get fetched just-in-time - eg during a \"git blame\", if\nyou did a \"blob:none\" filtered clone, you can easily end up with\nhundreds of fetches as git iterates backwards through the file\nhistory.\n\nI was trying to write a \"git blame optimizer\" to pre-fetch all the\nsuitable blobs, and it wasn't working right, so the \"git blame\" was\nstill fetching stuff - but I couldn't see what it was fetching (which\nmade it hard to investigate the bug in my script).\n\nI did end up getting a list of some just-in-time fetched blobs, by\ndumping a list of *all* the object IDs I had locally, before and after\na still-fetching-stuff \"git blame\" run, and doing a before/after\ncomparison of the resulting list of objects. To get the list of\nobjects found locally I did:\n\ngit cat-file --batch-check='%(objectname)' --batch-all-objects --unordered\n\n(ref: a conversation with Peff last year:\nhttps://lore.kernel.org/git/20230621064459.GA607974@coredump.intra.peff.net/\n)\n\nThis was a sucky process though - and I was very surprised that I\ncouldn't see what was being fetched (what the stdin content to the\njust-in-time fetch calls were) with any of the trace env vars that I\nwas able to find documented: GIT_TRACE, GIT_CURL_VERBOSE,\nGIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET,\nGIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW\n\nThe only thing I could easily see were the *args* passed to nested git\nprocesses.\n\nIs there any way to see what a just-in-time fetch is fetching? Or any\nway to see the content passed around on stdin in nested git processes?\n\nThanks,\nTao\n"},{"id":"501680","messageId":"xmqqv7znjir3.fsf@gitster.g","threadId":"62006","inReplyTo":"CAPMMpog7=ZnhJhrgZFwzRZibLtK1-LyOhsrp5c4O85ocRFDZxw@mail.gmail.com","subject":"Re: Sensible way to see what objects are being fetched just-in-time in a partial clone?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-26T17:28:48Z","receivedAt":"2024-08-26T17:28:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tao Klerks writes:\n\n> This was a sucky process though - and I was very surprised that I\n> couldn't see what was being fetched (what the stdin content to the\n> just-in-time fetch calls were) with any of the trace env vars that I\n> was able to find documented: GIT_TRACE, GIT_CURL_VERBOSE,\n> GIT_TRACE_PERFORMANCE, GIT_TRACE_PACK_ACCESS, GIT_TRACE_PACKET,\n> GIT_TRACE_PACKFILE, GIT_TRACE_SETUP, GIT_TRACE_SHALLOW\n\nYeah, lazy fetch codepath seems to be, eh, not quite well polished\nyet.\n\nI am kind of surprised that there is no trace2() events around\npromisor_remote_get_direct() or its callers. Perhaps it is a good\nidea to add one to log how often it is triggered, and how large a\nbatch the callers of the function is making?\n\nUnlike the diff machinery, blame does not have a prefetch machinery.\nI am glad that somebody is looking at it.\n"},{"id":"501697","messageId":"CAPMMpoiPeSyLHUd072Q9AozdaCAjpOL03jY-0Y33B2iEQ5EdTQ@mail.gmail.com","threadId":"62006","inReplyTo":"xmqqv7znjir3.fsf@gitster.g","subject":"Re: Sensible way to see what objects are being fetched just-in-time in a partial clone?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2024-08-26T19:37:32Z","receivedAt":"2024-08-26T19:37:45Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Mon, Aug 26, 2024 at 7:28 PM Junio C Hamano wrote:\n>\n>\n> Unlike the diff machinery, blame does not have a prefetch machinery.\n> I am glad that somebody is looking at it.\n\nI'm not convinced I'm looking at it in a very useful way from your\nperspective: My C skills being spectacularly lacking for fixing the\ncore code, I am just writing an external python-based wrapper for \"my\"\nusers.\n\nI will try to \"productize\" it sufficiently to send here in case it's\nuseful to someone, but all I can really offer the community-at-large\nis confirmation that in principle, the approach works as you would\nexpect: With some small number of jit-fetches for rename-detection\nduring the revision walk(s), and with one blob-prefetch call\nafterwards, \"git blame\" can be made to run cleanly/quickly in a\n\"filter:none\" clone even on a file like \"git.c\", with hundreds of\nrevisions.\n"},{"id":"501699","messageId":"CAPMMpojae7t1Z0PQPf=d2WzzbaMwkhvo34b_sxdtUQxvDu4fJw@mail.gmail.com","threadId":"62006","inReplyTo":"CAPMMpoiPeSyLHUd072Q9AozdaCAjpOL03jY-0Y33B2iEQ5EdTQ@mail.gmail.com","subject":"Python-based fetch optimizer script for \"blame\" in Partial Clones (was: Re: Sensible way to see what objects are being fetched just-in-time in a partial clone?)","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2024-08-26T20:37:37Z","receivedAt":"2024-08-26T20:37:51Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Mon, Aug 26, 2024 at 9:37 PM Tao Klerks wrote:\n>\n> On Mon, Aug 26, 2024 at 7:28 PM Junio C Hamano wrote:\n> >\n> >\n> > Unlike the diff machinery, blame does not have a prefetch machinery.\n> > I am glad that somebody is looking at it.\n>\n> I will try to \"productize\" it sufficiently to send here in case it's\n> useful to someone, but all I can really offer the community-at-large\n> is confirmation that in principle, the approach works as you would\n> expect: With some small number of jit-fetches for rename-detection\n> during the revision walk(s), and with one blob-prefetch call\n> afterwards, \"git blame\" can be made to run cleanly/quickly in a\n> \"filter:none\" clone even on a file like \"git.c\", with hundreds of\n> revisions.\n\nFWIW, here is the script I ended up with, which seems to work reliably\nfor me (through renames etc). Obviously I'd love to see this built\ninto \"git blame\" itself, but this wrapper might help someone out there\nin the meantime.\n"}]}