{"thread":{"id":"62009","subject":"Can git log be made to \"follow\" in the same way as git blame? Why / in what way is \"--follow\" broken or limited?","startedAt":"2024-08-26T19:00:39Z","lastAt":"2024-08-26T22:52:33Z","messageCount":3,"participants":["Tao Klerks","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"501694","messageId":"CAPMMpogApZ6VN9sYxgmtHCickkstM6HZq1teeAa+a2t1_BY0sQ@mail.gmail.com","threadId":"62009","inReplyTo":null,"subject":"Can git log be made to \"follow\" in the same way as git blame? Why / in what way is \"--follow\" broken or limited?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2024-08-26T19:00:24Z","receivedAt":"2024-08-26T19:00:39Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI've been working on a \"git blame optimizer for partial clone repos\",\nfollowing up on a thread with Derrick Stolee from 2021 (\nhttps://lore.kernel.org/git/0b57cba9-3ab3-dfdf-5589-a0016eaea634@gmail.com/\n), with the intention to pre-fetch all locally-missing blobs for a\ngiven file in the history of a branch/commit, and it ended up being\nmuch more complex to do that I expected, basically because \"git log\n--follow -- SOMEFILE\" doesn't return all the commits containing\nversions of \"SOMEFILE\" that \"git blame\" will end up visiting.\n\nI thought this strange/interesting, because (as far as I can tell), as\nlong as there are no renames in the history of the file, \"git log --\nSOMEFILE\", without the \"--follow\", *does* seem to return all the\ncommits containing unique versions of the file.\n\nThe only reference to this weirdness that I could find in the doc,\nafter ripping my hair out for a few hours, was in a note on\n\"log.follow\" config, under the \"git log\" doc eg at\nhttps://git-scm.com/docs/git-log : \"This has the same limitations as\n--follow, i.e. it cannot be used to follow multiple files and does not\nwork well on non-linear history.\"\n\nWhat seems weird and interesting to me, is that whatever is going\n\"wrong\" in \"git log --follow\" doesn't happen in \"git blame\". I\ncouldn't find an easy way to demo/prove it, but I found experimentally\nthat the set of blobs examined by \"git blame\" (and fetched\njust-in-time if-needed in a partial clone) is larger than the set of\nblobs in commits output by \"git log --follow -- FILENAME\", but not\nlarger than the set of blobs in commits output by \"git log --\nFILENAME\" (for a file that has not been renamed).\n\nYou can see the strange effect that \"--follow\" has by comparing a run\nwith and without on \"git.c\" in the git project for example - a file\nthat was never renamed:\n\ngit log --pretty='%H' -- git.c | sort | uniq > ~/test1 # 717 commits\ngit log --pretty='%H' --follow -- git.c | sort | uniq > ~/test2 # 537 commits\n\nYou'll find that with \"--follow\", you get a couple extra commits, and\na whole bunch missing. You can try to fill them in with\n\"--full-history\" etc, but while such options are *accepted*, they are\nalso completely and utterly *ignored*.\n\nInsofar as this is a known issue... is there an intelligible reason\nfor it? Should be something we aspire to fix, or should the doc be\nimproved to make it more obvious what this option does and doesn't do?\n\nThanks,\nTao\n"},{"id":"501698","messageId":"xmqqa5gzhxxg.fsf@gitster.g","threadId":"62009","inReplyTo":"CAPMMpogApZ6VN9sYxgmtHCickkstM6HZq1teeAa+a2t1_BY0sQ@mail.gmail.com","subject":"Re: Can git log be made to \"follow\" in the same way as git blame? Why / in what way is \"--follow\" broken or limited?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-26T19:43:55Z","receivedAt":"2024-08-26T19:44:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Tao Klerks <tao@klerks.biz> writes:\n\n> What seems weird and interesting to me, is that whatever is going\n> \"wrong\" in \"git log --follow\" doesn't happen in \"git blame\".\n\nYes, because log.follow was done as a checkbox item but blame was\ndone as a real feature ;-)\n\nIn tree-diff.c:try_to_follow_renames(), you'll notice that it only\nhas a space to remember a single path in .single_follow member in\nthe diff_opts.  That member is the hack.\n\nImagine that the original commit had paths A and B, and over time,\nthe history diverged and in one fork A got renamed to C while\nanother fork B got renamed to C.  Eventually these two forks merge.\nNow you want to \"follow\" C, so .single_follow member will have C.\n\n\n  ----1----3----5(rename A to C)----7---9---10---11\n       \\                               /\n        2----4----6(rename B to C)----8\n\nYou follow the history of one fork and notice that C came from A at\ncommit #5.  Great.  Your .pathspec member will be _switched_ to A\nand you keep following the history of A.\n\nImagine further that your history traversal didn't follow one fork\nfully before following the other fork, but dug commits from newer to\nolder, so your traversal jumps around between two forks.  What\nhappens when your \"git log --follow HEAD -- C\" that has internally\nswitched to follow A already jumps back to follow the other fork at\nthis point?  It does see that A exists (maybe unchanged), and you\nsee A's history, but that is not a releavant history---what ended up\nin the final C from that fork was in B, not A.\n\nUnlike the above checkbox hack, \"git blame\" uses a real data\nstructure to keep track of what came from where.  Instead of a\nglobal \"this single path is what interests us now\", it knows \"in\nthis commit, this is the path we are looking at\", and when it looks\nat the parents of that commit, it checks where that path the child\nwas interested in came from each different parent, and records a\nsimilar \"in this commit (which is parent of the commit we were\nlooking at), this path is what we are interested in\".\n\nTo equip \"git log --follow\" with similar \"correctness\" as \"git\nblame\", you'd need to somehow stop using that single .pathspec thing\nfor the purpose of keeping track of \"which path are we following\nnow?\" and instead use \"this is the path we are following\" that is\nper history traversal path.\n"},{"id":"501704","messageId":"xmqqjzg2gamp.fsf@gitster.g","threadId":"62009","inReplyTo":"xmqqa5gzhxxg.fsf@gitster.g","subject":"Re: Can git log be made to \"follow\" in the same way as git blame? Why / in what way is \"--follow\" broken or limited?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-26T22:52:30Z","receivedAt":"2024-08-26T22:52:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Unlike the above checkbox hack, \"git blame\" uses a real data\n> structure to keep track of what came from where.  Instead of a\n> global \"this single path is what interests us now\", it knows \"in\n> this commit, this is the path we are looking at\", and when it looks\n> at the parents of that commit, it checks where that path the child\n> was interested in came from each different parent, and records a\n> similar \"in this commit (which is parent of the commit we were\n> looking at), this path is what we are interested in\".\n\nFWIW, the above is greatly simplified.  For \"git blame\" to correctly\nhandle a case like \"This commit created file F by taking pieces from\nfiles A, B, C, D, and E\", and annotating the lines in file F, we\nneed to keep track of the set of \"lines n..m of path A\", \"lines l..k\nof path B\", etc., at commit X as the targets of interest, and as we\ndig down the history, figure out where in the parent commits of X\neach of these range of lines come from.  So what \"blame\" uses is\nmuch richer than just a single path per commit being traversed (once\nthe traversal passes through from a commit to all of its parents,\nthis list of \"line ranges per path\" can be released, so that is not\na huge memory burden even for a deep history).\n\nNow \"git log --follow\" does not have to keep track of range of\nlines, but if you start following from file F that was created by\nconcatenating pieces of multiple existing files A, B, ..., and E,\nyou either want to pick one of these 5 and follow it, or you replace\nF with all five of these files and follow them from that point.  In\nany case, you need a richer data structure than the current (ab)use\nof the .pathspec member during the traversal.\n"}]}