{"thread":{"id":"46733","subject":"[idea] File history tracking hints","startedAt":"2017-09-11T07:11:53Z","lastAt":"2017-10-03T00:52:29Z","messageCount":18,"participants":["Pavel Kretov","Stefan Beller","Jeff King","Jacob Keller","Igor Djordjevic","Philip Oakley","Johannes Schindelin","Jeff Hostetler","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"327844","messageId":"CAOZF3=Ouvk8ccME+fXr_T=GL1j4Gx3Hgj3ao_-GQng-noeOubg@mail.gmail.com","threadId":"46733","inReplyTo":null,"subject":"[idea] File history tracking hints","fromName":"Pavel Kretov","fromEmail":"firegurafiku@gmail.com","sentAt":"2017-09-11T07:11:31Z","receivedAt":"2017-09-11T07:11:53Z","isPatch":false,"sender":{"key":"firegurafiku@gmail.com","avatar":null},"body":"Hi all,\n\nExcuse me if the topic I'm going to raise here has been already discussed\non the mailing list, forums, or IRC, but I couldn't find anything related.\n\n\nThe problem:\n\nGit, being \"a stupid content tracker\", doesn't try to keep an eye on\noperations which happens to individual files; things like file renames\naren't recorded during commit, but heuristically detected later.\n\nUnfortunately, the heuristic can only deal with simple file renames with\nno substantial content changes; it's helpless when you:\n\n - rename file and change it's content significantly;\n - split single file into several files;\n - merge several files into another;\n - copy entire file from another commit, and do other things like these.\n\nHowever, if we're able to preserve this information, it's possible\nnot only to do more accurate 'git blame', but also merge revisions with\nfewer conflicts.\n\n\nThe proposal:\n\nThe idea is to let user give hints about what was changed during\nthe commit. For example, if user did a rename which wasn't automatically\ndetected, he would append something like the following to his commit\nmessage:\n\n    Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\ndev-vcs/git/git-2.0.ebuild\n\nor (if full paths of affected files can be unambiguously omitted):\n\n    Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n\nThere may be other hint types:\n\n    Tracking-hint: recreate LICENSE.txt\n    Tracking-hint: split main.c -> main.c cmdline.c\n    Tracking-hint: merge linalg.py <- vector.py matrix.py\n\nor even something like this:\n\n    Tracking-hint: copy json.py <-\nlibs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n\nIf file transformation cannot be described by a single tracking hint, it shall\nbe possible to specify a sequence of hints at once:\n\n    Tracking-hint:\n        split Utils.java -> AppHelpers.java StringHelpers.java\n        recreate Utils.java\n\nNote that in the above example the order of operations really matters, so\nboth lines have to reside in one 'Tracking-hint' block.\n\n* * *\n\nHow do you think, is this idea worth implementing?\nAny other thoughts on this?\n\n-- Pavel Kretov.\n"},{"id":"327856","messageId":"CAGZ79kacL7TodpBFmrPJtasjXYVKTVDb-FmM5QJyAPS-+NYf7w@mail.gmail.com","threadId":"46733","inReplyTo":"CAOZF3=Ouvk8ccME+fXr_T=GL1j4Gx3Hgj3ao_-GQng-noeOubg@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-11T18:11:08Z","receivedAt":"2017-09-11T18:11:14Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Sep 11, 2017 at 12:11 AM, Pavel Kretov <firegurafiku@gmail.com> wrote:\n> Hi all,\n>\n> Excuse me if the topic I'm going to raise here has been already discussed\n> on the mailing list, forums, or IRC, but I couldn't find anything related.\n>\n>\n> The problem:\n>\n> Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n> operations which happens to individual files; things like file renames\n> aren't recorded during commit, but heuristically detected later.\n>\n> Unfortunately, the heuristic can only deal with simple file renames with\n> no substantial content changes; it's helpless when you:\n>\n>  - rename file and change it's content significantly;\n>  - split single file into several files;\n>  - merge several files into another;\n>  - copy entire file from another commit, and do other things like these.\n>\n> However, if we're able to preserve this information, it's possible\n> not only to do more accurate 'git blame', but also merge revisions with\n> fewer conflicts.\n>\n>\n> The proposal:\n>\n> The idea is to let user give hints about what was changed during\n> the commit. For example, if user did a rename which wasn't automatically\n> detected, he would append something like the following to his commit\n> message:\n>\n>     Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n> dev-vcs/git/git-2.0.ebuild\n>\n> or (if full paths of affected files can be unambiguously omitted):\n>\n>     Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n>\n> There may be other hint types:\n>\n>     Tracking-hint: recreate LICENSE.txt\n>     Tracking-hint: split main.c -> main.c cmdline.c\n>     Tracking-hint: merge linalg.py <- vector.py matrix.py\n>\n> or even something like this:\n>\n>     Tracking-hint: copy json.py <-\n> libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n>\n> If file transformation cannot be described by a single tracking hint, it shall\n> be possible to specify a sequence of hints at once:\n>\n>     Tracking-hint:\n>         split Utils.java -> AppHelpers.java StringHelpers.java\n>         recreate Utils.java\n>\n> Note that in the above example the order of operations really matters, so\n> both lines have to reside in one 'Tracking-hint' block.\n>\n> * * *\n>\n> How do you think, is this idea worth implementing?\n> Any other thoughts on this?\n>\n> -- Pavel Kretov.\n\nThis was discussed a couple of times on the mailing list\n(though not recently).\n\nI searched for \"rename tracking files site:public-inbox.org/git\"\nand came up with\nhttps://public-inbox.org/git/Pine.LNX.4.58.0504141102430.7211@ppc970.osdl.org/\n(the nearby emails seem to also be relevant to this discussion)\n\ntl:dr: When encoding these hints, you do it at commit time,\nbut the heuristic can be improved upon later.\nSo you can assume the heuristic is better for the\ncommon case, as someone will fix the heuristic for the\ncommon case. Also Gits model is to track objects.\n"},{"id":"327858","messageId":"20170911184130.4jm7zxinccwcuhgh@sigill.intra.peff.net","threadId":"46733","inReplyTo":"CAOZF3=Ouvk8ccME+fXr_T=GL1j4Gx3Hgj3ao_-GQng-noeOubg@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-09-11T18:41:30Z","receivedAt":"2017-09-11T18:41:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Sep 11, 2017 at 10:11:31AM +0300, Pavel Kretov wrote:\n\n> Unfortunately, the heuristic can only deal with simple file renames with\n> no substantial content changes; it's helpless when you:\n> \n>  - rename file and change it's content significantly;\n>  - split single file into several files;\n>  - merge several files into another;\n>  - copy entire file from another commit, and do other things like these.\n> \n> However, if we're able to preserve this information, it's possible\n> not only to do more accurate 'git blame', but also merge revisions with\n> fewer conflicts.\n\nThis is definitely something that's been discussed before on the list\n(though I'm not sure of the best keywords to dig for; Stefan found one\nthread but I know there have been others).\n\nAnd I don't think it's a totally unreasonable idea, but there are some\ncomplications. The biggest one is that renames are really part of a\n_diff_ between two endpoints. We think of them as attached to a commit\nbecause we tend to talk about commits as a diff from state A to state B.\n\nSo obviously in the diff HEAD^ versus HEAD, we can look at the hints for\nHEAD. But what about \"git diff v1.0 v1.1\", that may cover multiple\ncommits? Right now Git doesn't look at the intermediate commits at all.\nAnd in fact we may not even know what they are, if the command is fed\ntwo trees. Or the two endpoints may not have a sensible history (e.g.,\nconsider diffing between two branches, one of which has been rebased).\n\nBut even if we had a sensible set of commits to pull hints from (e.g.,\nif v1.0 and v1.1 were in a linear relationship), it's not clear to me\nhow you would want to apply them to an end-to-end diff.\n\nSo I don't think that these kind of tracking hints make sense for a lot\nof diffs (including merges, which use diffs between the endpoints and\nthe merge base).\n\nWhich isn't to say that they're useless. I agree that something like\n\"--follow\" could benefit from an annotation that tells us when and how\nto pick up the next step in the traversal. But of course somebody has to\nmake those annotations. If we had a tool to do it automatically, then we\ncould apply the same tool at run-time later.\n\nBut maybe if it were an optional annotation, people would want to use it\nwhen the normal rename logic doesn't kick in. So perhaps a baby step in\nthis direction would be to teach something like \"--follow\" to \"jump\"\nacross a non-rename when it sees a special marking in the commit\nmessage.\n\n-Peff\n"},{"id":"327859","messageId":"CA+P7+xqFrC9MFm+b65q53_SGK7JJUJFYeeLUgxXd8te5rrg5Rw@mail.gmail.com","threadId":"46733","inReplyTo":"CAGZ79kacL7TodpBFmrPJtasjXYVKTVDb-FmM5QJyAPS-+NYf7w@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2017-09-11T18:47:36Z","receivedAt":"2017-09-11T18:48:03Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Mon, Sep 11, 2017 at 11:11 AM, Stefan Beller <sbeller@google.com> wrote:\n> On Mon, Sep 11, 2017 at 12:11 AM, Pavel Kretov <firegurafiku@gmail.com> wrote:\n>> Hi all,\n>>\n>> Excuse me if the topic I'm going to raise here has been already discussed\n>> on the mailing list, forums, or IRC, but I couldn't find anything related.\n>>\n>>\n>> The problem:\n>>\n>> Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n>> operations which happens to individual files; things like file renames\n>> aren't recorded during commit, but heuristically detected later.\n>>\n>> Unfortunately, the heuristic can only deal with simple file renames with\n>> no substantial content changes; it's helpless when you:\n>>\n>>  - rename file and change it's content significantly;\n>>  - split single file into several files;\n>>  - merge several files into another;\n>>  - copy entire file from another commit, and do other things like these.\n>>\n>> However, if we're able to preserve this information, it's possible\n>> not only to do more accurate 'git blame', but also merge revisions with\n>> fewer conflicts.\n>>\n>>\n>> The proposal:\n>>\n>> The idea is to let user give hints about what was changed during\n>> the commit. For example, if user did a rename which wasn't automatically\n>> detected, he would append something like the following to his commit\n>> message:\n>>\n>>     Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n>> dev-vcs/git/git-2.0.ebuild\n>>\n>> or (if full paths of affected files can be unambiguously omitted):\n>>\n>>     Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n>>\n>> There may be other hint types:\n>>\n>>     Tracking-hint: recreate LICENSE.txt\n>>     Tracking-hint: split main.c -> main.c cmdline.c\n>>     Tracking-hint: merge linalg.py <- vector.py matrix.py\n>>\n>> or even something like this:\n>>\n>>     Tracking-hint: copy json.py <-\n>> libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n>>\n>> If file transformation cannot be described by a single tracking hint, it shall\n>> be possible to specify a sequence of hints at once:\n>>\n>>     Tracking-hint:\n>>         split Utils.java -> AppHelpers.java StringHelpers.java\n>>         recreate Utils.java\n>>\n>> Note that in the above example the order of operations really matters, so\n>> both lines have to reside in one 'Tracking-hint' block.\n>>\n>> * * *\n>>\n>> How do you think, is this idea worth implementing?\n>> Any other thoughts on this?\n>>\n>> -- Pavel Kretov.\n>\n> This was discussed a couple of times on the mailing list\n> (though not recently).\n>\n> I searched for \"rename tracking files site:public-inbox.org/git\"\n> and came up with\n> https://public-inbox.org/git/Pine.LNX.4.58.0504141102430.7211@ppc970.osdl.org/\n> (the nearby emails seem to also be relevant to this discussion)\n>\n> tl:dr: When encoding these hints, you do it at commit time,\n> but the heuristic can be improved upon later.\n> So you can assume the heuristic is better for the\n> common case, as someone will fix the heuristic for the\n> common case. Also Gits model is to track objects.\n\nLinus has a pretty long post about this, it's somewhere in that\ndiscussion. Essentially, if you bake in rename detection (or other\nhints) at commit time, then you're stuck with it forever.\n\nAdditionally, there are similar but not *quite* the same operations\nwhich you probably wouldn't bake into at the start, and the types of\nquestions a user wants to ask isn't known at commit time, but rather\nknown at *debug* time in the future when you're digging up history. In\nthis time frame, the user does know what to care about and what kind\nof questions to ask, so it's already natural to ask these questions at\nthat time.\n\nAdditionally, if you have to generate the heuristic every commit,\nyou're increasing time \"wasted\" every commit, where as doing the\nlookup later when a user starts asking questions like during blame or\ndiff would only add time during an operation the user already expects\nto take some time.\n\nThanks,\nJake\n"},{"id":"327866","messageId":"774426e0-24c9-320b-a593-b3d0865cfa68@gmail.com","threadId":"46733","inReplyTo":"CAOZF3=Ouvk8ccME+fXr_T=GL1j4Gx3Hgj3ao_-GQng-noeOubg@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Igor Djordjevic","fromEmail":"igor.d.djordjevic@gmail.com","sentAt":"2017-09-11T20:09:14Z","receivedAt":"2017-09-11T20:09:24Z","isPatch":false,"sender":{"key":"igor.d.djordjevic@gmail.com","avatar":null},"body":"Hi Pavel,\n\nOn 11/09/2017 09:11, Pavel Kretov wrote:\n> Hi all,\n> \n> Excuse me if the topic I'm going to raise here has been already discussed\n> on the mailing list, forums, or IRC, but I couldn't find anything related.\n> \n> \n> The problem:\n> \n> Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n> operations which happens to individual files; things like file renames\n> aren't recorded during commit, but heuristically detected later.\n> \n> Unfortunately, the heuristic can only deal with simple file renames with\n> no substantial content changes; it's helpless when you:\n> \n>  - rename file and change it's content significantly;\n>  - split single file into several files;\n>  - merge several files into another;\n>  - copy entire file from another commit, and do other things like these.\n> \n> However, if we're able to preserve this information, it's possible\n> not only to do more accurate 'git blame', but also merge revisions with\n> fewer conflicts.\n> \n> \n> The proposal:\n> \n> The idea is to let user give hints about what was changed during\n> the commit. For example, if user did a rename which wasn't automatically\n> detected, he would append something like the following to his commit\n> message:\n> \n>     Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n> dev-vcs/git/git-2.0.ebuild\n> \n> or (if full paths of affected files can be unambiguously omitted):\n> \n>     Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n> \n> There may be other hint types:\n> \n>     Tracking-hint: recreate LICENSE.txt\n>     Tracking-hint: split main.c -> main.c cmdline.c\n>     Tracking-hint: merge linalg.py <- vector.py matrix.py\n> \n> or even something like this:\n> \n>     Tracking-hint: copy json.py <-\n> libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n> \n> If file transformation cannot be described by a single tracking hint, it shall\n> be possible to specify a sequence of hints at once:\n> \n>     Tracking-hint:\n>         split Utils.java -> AppHelpers.java StringHelpers.java\n>         recreate Utils.java\n> \n> Note that in the above example the order of operations really matters, so\n> both lines have to reside in one 'Tracking-hint' block.\n> \n> * * *\n> \n> How do you think, is this idea worth implementing?\n> Any other thoughts on this? \n\nHere[1] you can find Linus` reply (from 2005-04-15) to \"rename \ntracking\" discussion, usually quoted to explain the Git philosophy on \nthis point, even referred to as \"one of the most important messages \nin the list archive\"[2] by Junio himself.\n\n[1] https://public-inbox.org/git/Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org/\n[2] https://public-inbox.org/git/xmqqr30qflk9.fsf@gitster.mtv.corp.google.com/\n\nRegards,\nBuga\n"},{"id":"327868","messageId":"E8C827ED458648F78F263F2F2712493B@PhilipOakley","threadId":"46733","inReplyTo":"CAOZF3=Ouvk8ccME+fXr_T=GL1j4Gx3Hgj3ao_-GQng-noeOubg@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-09-11T21:48:46Z","receivedAt":"2017-09-11T21:48:50Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Pavel Kretov\" <firegurafiku@gmail.com>\n> Hi all,\n>\n> Excuse me if the topic I'm going to raise here has been already discussed\n> on the mailing list, forums, or IRC, but I couldn't find anything related.\n>\n>\n> The problem:\n>\n> Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n> operations which happens to individual files; things like file renames\n> aren't recorded during commit, but heuristically detected later.\n>\n> Unfortunately, the heuristic can only deal with simple file renames with\n> no substantial content changes; it's helpless when you:\n>\n> - rename file and change it's content significantly;\n> - split single file into several files;\n> - merge several files into another;\n> - copy entire file from another commit, and do other things like these.\n>\n> However, if we're able to preserve this information, it's possible\n> not only to do more accurate 'git blame', but also merge revisions with\n> fewer conflicts.\n>\n>\n> The proposal:\n>\n> The idea is to let user give hints about what was changed during\n> the commit. For example, if user did a rename which wasn't automatically\n> detected, he would append something like the following to his commit\n> message:\n>\n>    Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n> dev-vcs/git/git-2.0.ebuild\n>\n> or (if full paths of affected files can be unambiguously omitted):\n>\n>    Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n>\n> There may be other hint types:\n>\n>    Tracking-hint: recreate LICENSE.txt\n>    Tracking-hint: split main.c -> main.c cmdline.c\n>    Tracking-hint: merge linalg.py <- vector.py matrix.py\n>\n> or even something like this:\n>\n>    Tracking-hint: copy json.py <-\n> libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n>\n> If file transformation cannot be described by a single tracking hint, it \n> shall\n> be possible to specify a sequence of hints at once:\n>\n>    Tracking-hint:\n>        split Utils.java -> AppHelpers.java StringHelpers.java\n>        recreate Utils.java\n>\n> Note that in the above example the order of operations really matters, so\n> both lines have to reside in one 'Tracking-hint' block.\n>\n> * * *\n>\n> How do you think, is this idea worth implementing?\n> Any other thoughts on this?\n>\n> -- Pavel Kretov.\n\nMaybe use the \"interpret-trailers\" methods for standardising your hints \nlocally (in your team / workplace) to see how it goes and flesh out what \nworks and what doesn't. Trying to decide, a-priori, what are the right hints \nis likely to be the hard part.\n--\nPhilip \n\n"},{"id":"327923","messageId":"alpine.DEB.2.21.1.1709131322470.4132@virtualbox","threadId":"46733","inReplyTo":"E8C827ED458648F78F263F2F2712493B@PhilipOakley","subject":"Re: [idea] File history tracking hints","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-09-13T11:38:07Z","receivedAt":"2017-09-13T11:38:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Philip,\n\nOn Mon, 11 Sep 2017, Philip Oakley wrote:\n\n> From: \"Pavel Kretov\" <firegurafiku@gmail.com>\n> > Hi all,\n> >\n> > Excuse me if the topic I'm going to raise here has been already discussed\n> > on the mailing list, forums, or IRC, but I couldn't find anything related.\n> >\n> >\n> > The problem:\n> >\n> > Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n> > operations which happens to individual files; things like file renames\n> > aren't recorded during commit, but heuristically detected later.\n> >\n> > Unfortunately, the heuristic can only deal with simple file renames with\n> > no substantial content changes; it's helpless when you:\n> >\n> > - rename file and change it's content significantly;\n> > - split single file into several files;\n> > - merge several files into another;\n> > - copy entire file from another commit, and do other things like these.\n> >\n> > However, if we're able to preserve this information, it's possible\n> > not only to do more accurate 'git blame', but also merge revisions with\n> > fewer conflicts.\n> >\n> >\n> > The proposal:\n> >\n> > The idea is to let user give hints about what was changed during\n> > the commit. For example, if user did a rename which wasn't automatically\n> > detected, he would append something like the following to his commit\n> > message:\n> >\n> >    Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n> > dev-vcs/git/git-2.0.ebuild\n> >\n> > or (if full paths of affected files can be unambiguously omitted):\n> >\n> >    Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n> >\n> > There may be other hint types:\n> >\n> >    Tracking-hint: recreate LICENSE.txt\n> >    Tracking-hint: split main.c -> main.c cmdline.c\n> >    Tracking-hint: merge linalg.py <- vector.py matrix.py\n> >\n> > or even something like this:\n> >\n> >    Tracking-hint: copy json.py <-\n> > libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n> >\n> > If file transformation cannot be described by a single tracking hint, it\n> > shall\n> > be possible to specify a sequence of hints at once:\n> >\n> >    Tracking-hint:\n> >        split Utils.java -> AppHelpers.java StringHelpers.java\n> >        recreate Utils.java\n> >\n> > Note that in the above example the order of operations really matters, so\n> > both lines have to reside in one 'Tracking-hint' block.\n> >\n> > * * *\n> >\n> > How do you think, is this idea worth implementing?\n> > Any other thoughts on this?\n> >\n> > -- Pavel Kretov.\n> \n> Maybe use the \"interpret-trailers\" methods for standardising your hints\n> locally (in your team / workplace) to see how it goes and flesh out what works\n> and what doesn't. Trying to decide, a-priori, what are the right hints is\n> likely to be the hard part.\n\nI think this adds a very valuable insight to this discussion: the current\nstate of Git's rename handling is based on the idea that you either record\nthe renames, or you detect them. Like, there is either \"on\" or \"off\". No\nmiddle ground.\n\nHowever, if you understand that there is also the possibility of hints\nthat can help any erroneous rename detection (and *everybody* who\nseriously worked on a massive code base has seen that rename detection\nfail in the most inopportune ways [*1*]), then you are on to something.\n\nSo I totally like the idea of introducing hints, possibly as trailers in\nthe commit message (or as refs/notes/rename/* or whatever) that can be\npicked up by Git versions that know about them, and can be ignored by Git\nversions that insist on the rename detection du jour. With a config option\nto control the behavior, maybe, too.\n\nCiao,\nDscho\n\nFootnote *1*: Just to name a couple of examples from my personal\nexperience, off the top of my head:\n\n- license boiler plates often let Git detect renames/copies where there\n  are none,\n\n- even something as trivial as moving Java classes (and their dependent\n  classes) between packages changes every line referring to said packages,\n  causing Git's rename detection to go for a drink instead of doing its\n  job,\n\n- indentation changes overwhelm Git's rename detection,\n\n- when rename detection would matter most, like, really a lot, to lift the\n  burden of the human beings in front of the computer pouring over\n  hundreds of thousands of files moved from one directory tree to another,\n  that's exactly when Git's rename detection says that there are too many\n  files, here are my union rights, I am going home, good luck to you.\n\nIn light of such experiences, I have to admit that the notion that the\nrename detection can always be improved in hindsight puts quite a bit of\ninsult to injury for those developers who are bitten by it.\n"},{"id":"328084","messageId":"04DDB36236444FFD8C3668AA7B62B154@PhilipOakley","threadId":"46733","inReplyTo":"alpine.DEB.2.21.1.1709131322470.4132@virtualbox","subject":"Re: [idea] File history tracking hints","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-09-14T23:22:02Z","receivedAt":"2017-09-14T23:22:08Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Johannes Schindelin\" <Johannes.Schindelin@gmx.de>\n> Hi Philip,\n>\n> On Mon, 11 Sep 2017, Philip Oakley wrote:\n>\n>> From: \"Pavel Kretov\" <firegurafiku@gmail.com>\n>> > Hi all,\n>> >\n>> > Excuse me if the topic I'm going to raise here has been already \n>> > discussed\n>> > on the mailing list, forums, or IRC, but I couldn't find anything \n>> > related.\n>> >\n>> >\n>> > The problem:\n>> >\n>> > Git, being \"a stupid content tracker\", doesn't try to keep an eye on\n>> > operations which happens to individual files; things like file renames\n>> > aren't recorded during commit, but heuristically detected later.\n>> >\n>> > Unfortunately, the heuristic can only deal with simple file renames \n>> > with\n>> > no substantial content changes; it's helpless when you:\n>> >\n>> > - rename file and change it's content significantly;\n>> > - split single file into several files;\n>> > - merge several files into another;\n>> > - copy entire file from another commit, and do other things like these.\n>> >\n>> > However, if we're able to preserve this information, it's possible\n>> > not only to do more accurate 'git blame', but also merge revisions with\n>> > fewer conflicts.\n>> >\n>> >\n>> > The proposal:\n>> >\n>> > The idea is to let user give hints about what was changed during\n>> > the commit. For example, if user did a rename which wasn't \n>> > automatically\n>> > detected, he would append something like the following to his commit\n>> > message:\n>> >\n>> >    Tracking-hints: rename dev-vcs/git/git-1.0.ebuild ->\n>> > dev-vcs/git/git-2.0.ebuild\n>> >\n>> > or (if full paths of affected files can be unambiguously omitted):\n>> >\n>> >    Tracking-hints: rename git-1.0.ebuild -> git-2.0.ebuild\n>> >\n>> > There may be other hint types:\n>> >\n>> >    Tracking-hint: recreate LICENSE.txt\n>> >    Tracking-hint: split main.c -> main.c cmdline.c\n>> >    Tracking-hint: merge linalg.py <- vector.py matrix.py\n>> >\n>> > or even something like this:\n>> >\n>> >    Tracking-hint: copy json.py <-\n>> > libs/json.py@4db88291251151d8c5c8e4f20430fa4def2cb2ed\n>> >\n>> > If file transformation cannot be described by a single tracking hint, \n>> > it\n>> > shall\n>> > be possible to specify a sequence of hints at once:\n>> >\n>> >    Tracking-hint:\n>> >        split Utils.java -> AppHelpers.java StringHelpers.java\n>> >        recreate Utils.java\n>> >\n>> > Note that in the above example the order of operations really matters, \n>> > so\n>> > both lines have to reside in one 'Tracking-hint' block.\n>> >\n>> > * * *\n>> >\n>> > How do you think, is this idea worth implementing?\n>> > Any other thoughts on this?\n>> >\n>> > -- Pavel Kretov.\n>>\n>> Maybe use the \"interpret-trailers\" methods for standardising your hints\n>> locally (in your team / workplace) to see how it goes and flesh out what \n>> works\n>> and what doesn't. Trying to decide, a-priori, what are the right hints is\n>> likely to be the hard part.\n>\n> I think this adds a very valuable insight to this discussion: the current\n> state of Git's rename handling is based on the idea that you either record\n> the renames, or you detect them. Like, there is either \"on\" or \"off\". No\n> middle ground.\n>\n> However, if you understand that there is also the possibility of hints\n> that can help any erroneous rename detection (and *everybody* who\n> seriously worked on a massive code base has seen that rename detection\n> fail in the most inopportune ways [*1*]), then you are on to something.\n>\n> So I totally like the idea of introducing hints, possibly as trailers in\n> the commit message (or as refs/notes/rename/* or whatever) that can be\n> picked up by Git versions that know about them, and can be ignored by Git\n> versions that insist on the rename detection du jour. With a config option\n> to control the behavior, maybe, too.\n>\n> Ciao,\n> Dscho\n>\n> Footnote *1*: Just to name a couple of examples from my personal\n> experience, off the top of my head:\n>\n> - license boiler plates often let Git detect renames/copies where there\n>  are none,\n>\n> - even something as trivial as moving Java classes (and their dependent\n>  classes) between packages changes every line referring to said packages,\n>  causing Git's rename detection to go for a drink instead of doing its\n>  job,\n>\n> - indentation changes overwhelm Git's rename detection,\n>\n> - when rename detection would matter most, like, really a lot, to lift the\n>  burden of the human beings in front of the computer pouring over\n>  hundreds of thousands of files moved from one directory tree to another,\n>  that's exactly when Git's rename detection says that there are too many\n>  files, here are my union rights, I am going home, good luck to you.\n>\n> In light of such experiences, I have to admit that the notion that the\n> rename detection can always be improved in hindsight puts quite a bit of\n> insult to injury for those developers who are bitten by it.\n\nYour list made me think that the hints should be directed toward what may be \nconsidered existing solutions for those specific awkward cases.\n\nSo the hints could be (by type):\n- template;licence;boiler-plate;standard;reference :: copy\n- word-rename\n- regex for word substitution changes (e.g. which chars are within \n'Word-_0`)\n- regex for white-space changes (i.e. which chars are considered \nwhitespace.)\n- move-dir path/glob spec\n- move-file path/glob spec\n(maybe list each 'group' of moves, so that once found the rest of the rename \ndetection follows the group.)\n\nOnce the particular hint is detected (path qualified) then the clue/hint is \nused to assist in parsing the files to simplify the comparison task and \nlocate common lines, or common word patterns.\n\nThe first example is just a set of alternate terms folk use for the new \nduplicate file file case.\n\nThe second is a hint that there has been a number of fairly global name \nchanges in the files. so not only do a word diff but detect & sumarise those \nglobal changes. (your class move example)\n\nThe third is the more simple global word changes, based on a limited char \nset for a 'word' token list.\nThe fourth is where we are focussed on the white space part (complementing \nthe word token viewpoint)\n\nThe move hints are lists of path specs that each have distinctly moved.\n\nIt may be possible to order the hints as well, so that the detections work \nin the right order, giving the heuristics a better chance!\n\n--\nPhilip\n \n\n"},{"id":"329224","messageId":"alpine.DEB.2.21.1.1709300110350.40514@virtualbox","threadId":"46733","inReplyTo":"04DDB36236444FFD8C3668AA7B62B154@PhilipOakley","subject":"Re: [idea] File history tracking hints","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-09-29T23:12:39Z","receivedAt":"2017-09-29T23:12:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Philip,\n\nOn Fri, 15 Sep 2017, Philip Oakley wrote:\n\n> From: \"Johannes Schindelin\" <Johannes.Schindelin@gmx.de>\n>\n> > In light of such experiences, I have to admit that the notion that the\n> > rename detection can always be improved in hindsight puts quite a bit of\n> > insult to injury for those developers who are bitten by it.\n> \n> Your list made me think that the hints should be directed toward what may be\n> considered existing solutions for those specific awkward cases.\n> \n> So the hints could be (by type):\n> - template;licence;boiler-plate;standard;reference :: copy\n> - word-rename\n> - regex for word substitution changes (e.g. which chars are within 'Word-_0`)\n> - regex for white-space changes (i.e. which chars are considered whitespace.)\n> - move-dir path/glob spec\n> - move-file path/glob spec\n> (maybe list each 'group' of moves, so that once found the rest of the rename\n> detection follows the group.)\n> \n> Once the particular hint is detected (path qualified) then the clue/hint is\n> used to assist in parsing the files to simplify the comparison task and locate\n> common lines, or common word patterns.\n> \n> The first example is just a set of alternate terms folk use for the new\n> duplicate file file case.\n> \n> The second is a hint that there has been a number of fairly global name\n> changes in the files. so not only do a word diff but detect & sumarise those\n> global changes. (your class move example)\n> \n> The third is the more simple global word changes, based on a limited char set\n> for a 'word' token list.\n> The fourth is where we are focussed on the white space part (complementing the\n> word token viewpoint)\n> \n> The move hints are lists of path specs that each have distinctly moved.\n> \n> It may be possible to order the hints as well, so that the detections work in\n> the right order, giving the heuristics a better chance!\n\nI think my point was: no matter how likely we thought any heuristic rename\ndetection can be perfected over time, history proved that suspicion\nincorrect.\n\nTherefore, it would be good to have a way to tell Git about renames\nexplicitly so that it does not even need to use its heuristics.\n\nCiao,\nDscho\n"},{"id":"329242","messageId":"5fb263a8-d83b-64a7-812f-fd8e3748feb6@jeffhostetler.com","threadId":"46733","inReplyTo":"alpine.DEB.2.21.1.1709300110350.40514@virtualbox","subject":"Re: [idea] File history tracking hints","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-09-30T08:02:35Z","receivedAt":"2017-09-30T08:02:43Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 9/29/2017 7:12 PM, Johannes Schindelin wrote:\n> Hi Philip,\n> \n> On Fri, 15 Sep 2017, Philip Oakley wrote:\n> \n>> From: \"Johannes Schindelin\" <Johannes.Schindelin@gmx.de>\n>>\n>>> In light of such experiences, I have to admit that the notion that the\n>>> rename detection can always be improved in hindsight puts quite a bit of\n>>> insult to injury for those developers who are bitten by it.\n>>\n>> Your list made me think that the hints should be directed toward what may be\n>> considered existing solutions for those specific awkward cases.\n>>\n>> So the hints could be (by type):\n>> - template;licence;boiler-plate;standard;reference :: copy\n>> - word-rename\n>> - regex for word substitution changes (e.g. which chars are within 'Word-_0`)\n>> - regex for white-space changes (i.e. which chars are considered whitespace.)\n>> - move-dir path/glob spec\n>> - move-file path/glob spec\n>> (maybe list each 'group' of moves, so that once found the rest of the rename\n>> detection follows the group.)\n>>\n>> Once the particular hint is detected (path qualified) then the clue/hint is\n>> used to assist in parsing the files to simplify the comparison task and locate\n>> common lines, or common word patterns.\n>>\n>> The first example is just a set of alternate terms folk use for the new\n>> duplicate file file case.\n>>\n>> The second is a hint that there has been a number of fairly global name\n>> changes in the files. so not only do a word diff but detect & sumarise those\n>> global changes. (your class move example)\n>>\n>> The third is the more simple global word changes, based on a limited char set\n>> for a 'word' token list.\n>> The fourth is where we are focussed on the white space part (complementing the\n>> word token viewpoint)\n>>\n>> The move hints are lists of path specs that each have distinctly moved.\n>>\n>> It may be possible to order the hints as well, so that the detections work in\n>> the right order, giving the heuristics a better chance!\n> \n> I think my point was: no matter how likely we thought any heuristic rename\n> detection can be perfected over time, history proved that suspicion\n> incorrect.\n> \n> Therefore, it would be good to have a way to tell Git about renames\n> explicitly so that it does not even need to use its heuristics.\n\nAgreed.\n\nIt would be nice if every file (and tree) had a permanent GUID\nassociated with it.  Then the filename/pathname becomes a property\nof the GUIDs.  Then you can exactly know about moves/renames with\nminimal effort (and no guessing).  But I suppose that ship has sailed...\n\nJeff\n\n"},{"id":"329247","messageId":"alpine.DEB.2.21.1.1709301706170.40514@virtualbox","threadId":"46733","inReplyTo":"5fb263a8-d83b-64a7-812f-fd8e3748feb6@jeffhostetler.com","subject":"Re: [idea] File history tracking hints","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-09-30T15:11:53Z","receivedAt":"2017-09-30T15:12:10Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeff,\n\nOn Sat, 30 Sep 2017, Jeff Hostetler wrote:\n\n> On 9/29/2017 7:12 PM, Johannes Schindelin wrote:\n>\n> > Therefore, it would be good to have a way to tell Git about renames\n> > explicitly so that it does not even need to use its heuristics.\n> \n> Agreed.\n> \n> It would be nice if every file (and tree) had a permanent GUID\n> associated with it.  Then the filename/pathname becomes a property\n> of the GUIDs.  Then you can exactly know about moves/renames with\n> minimal effort (and no guessing).  But I suppose that ship has sailed...\n\nYes, that ship has sailed.\n\nBut we still could teach Git to understand certain \"hints\" (that would be\nreally more like \"cluebats\").\n\nSo while we cannot have any GUIDs that are persistent across renames/moves\n(and which users would probably get wrong all the time by using\nthird-party tools that are not Git-rename aware), we have unique\nidentifiers: the object names.\n\nAnd we could easily have a lookup table of pairs of object names, telling\nGit that they were source and target of a rename. When Git would try to\nfigure out whether anything was renamed, it would first look at that\nlookup table and save itself a lot of work (and opportunity to fail) and\nshort-cut the rename detection.\n\nCiao,\nJohannes\n"},{"id":"329270","messageId":"xmqqtvzjv987.fsf@gitster.mtv.corp.google.com","threadId":"46733","inReplyTo":"5fb263a8-d83b-64a7-812f-fd8e3748feb6@jeffhostetler.com","subject":"Re: [idea] File history tracking hints","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-01T03:27:04Z","receivedAt":"2017-10-01T03:27:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n> On 9/29/2017 7:12 PM, Johannes Schindelin wrote:\n>\n>> Therefore, it would be good to have a way to tell Git about renames\n>> explicitly so that it does not even need to use its heuristics.\n>\n> Agreed.\n>\n> It would be nice if every file (and tree) had a permanent GUID\n> associated with it.  Then the filename/pathname becomes a property\n> of the GUIDs.  Then you can exactly know about moves/renames with\n> minimal effort (and no guessing).\n\nI actually like the idea to have a mechanism where the user can give\nhint to influence, or instruction to dictate, how Git determines\n\"this old path moved to this new path\" when comparing two trees.  A\nhuman would not consider a new file (e.g. header file) that begins\nwith a few dozen commonly-seen boilerplate lines (e.g. copyright\nstatement) followed by several lines unique to the new contents to\nbe a rename of a disappearing old file that begins with the same\nboilerplate followed by several lines that are different from what\nis in the new file, but Git's algorithm would give equal weight to\nall of these lines when deciding how similar the new file is to the\nold file, and can misidentify a new file to be a rename of an old\nfile that is unrelated.  Even when Git can and does determine the\npairing correctly, it would be a win if we do not have to recompute\nthe same pairing every time.  So both as hint and as cache, such a\nmechanism would make sense [*1*].\n\nBut \"file ID\" does not have any place to contribute to such a\nmechanism.  Each of two developers working on the same project in a\ndisributed environment can grab the same gist and create a new file\nin his or her tree, perhaps at the same path or at a different\npath.  At the time of such an addition, there is no way for each of\nthem to give these two files the same \"file ID\" (that is how the\nworld works in the distributed environment after all)---which \"file\nID\" should survive when their two histories finally meet and results\nin a single file after a merge?  A file with \"file ID\" may not be\nrenamed but may be copied and evolve separately and differently.\nWhich one should inherit its original \"file ID\" and how does having\n\"file ID\" help us identify the other one is equally related to the\noriginal file?  These two are merely examples that \"file ID\"s would\ncause while solving \"only\" what can be expressed in \"git diff -M\"\noutput (the latter illustrates that it does not even help showing\n\"git diff -C\").\n\nAnd when we stop limiting ourselves to the whole-file renames and\ncopies (which can be expressed in \"git diff\" output) but also want\nto help finer-grained operation like \"git blame\", we'd want to have\nsomething that helps in situations like a single file's contents\nsplit into multiple files and multiple files' contents concatenated\ninto a single new file, both of which happens during code\nrefactoring.  \"file ID\" would not contribute an iota in helping\nthese situations.  \n\nI've said this number of times, and I'll say this again, but one of\nthe most important message in our list archive is gmane:217 aka\n\nhttps://public-inbox.org/git/Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org/\n\nI'd encourge people to read and re-read that message until they can\nrecite it by heart.\n\nLinus mentions \"CVS annotate\"; the message was written long before\nwe had \"git blame\", and it served as a guide when desiging how we\ndig contents movement in various parts of the system.\n\n\n[Footnote]\n\n*1* There are many possible implementations; the most obvious would\n    be to record a pair of blob object names and instruct Git when\n    it seems one side of a pair disappearing and the other side of\n    the pair appearing, take the pair as a rename.  And that would\n    be sufficient for \"git log -M\".  \n\n    Such a cache/hint alone however would not help much in \"git\n    merge\" without further work, as we merge using only the tree\n    state of the three points in the history (i.e. the common\n    ancestor and two tips).  merge-recursive needs to be taught to\n    find the renames at each commit it finds throughout the history\n    from the ancestor and each tip and carry its finding through if\n    it wants to take advantage of such hint/cache.\n"},{"id":"329470","messageId":"CAGZ79kbghnWmvQweup=Z79HnVQQCMM65CKgEO3oqDoRp-Bj=2Q@mail.gmail.com","threadId":"46733","inReplyTo":"xmqqtvzjv987.fsf@gitster.mtv.corp.google.com","subject":"Re: [idea] File history tracking hints","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-02T17:41:41Z","receivedAt":"2017-10-02T17:41:47Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> It would be nice if every file (and tree) had a permanent GUID\n>> associated with it.  Then the filename/pathname becomes a property\n>> of the GUIDs.  Then you can exactly know about moves/renames with\n>> minimal effort (and no guessing).\n>\n...\n\n> https://public-inbox.org/git/Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org/\n>\n> I'd encourge people to read and re-read that message until they can\n> recite it by heart.\n\nI have rethought about the idea of GUIDs as proposed by Jeff and wanted\nto give a reply. After rereading this message, I think my thoughts are\nalready included via:\n\n  - you're doing the work at the wrong point for _another_ reason. You're\n     freezing your (crappy) algorithm at tree creation time, and basically\n     making it pointless to ever create something better later, because even\n     if hardware and software improves, you've codified that \"we have to\n     have crappy information\".\n\n--\nMy design proposal for these \"rename hints\" would be a special trailer,\nroughly:\n\n    Rename: LICENSE -> legal.txt\n    Rename: t/* -> tests/*\n\nor more generally:\n\n    Rename: <pathspec> <delim> <pathspec>\n\nThis however has multiple issues due to potential\nhuman inaccuracies:\n(A) typos in the trailer key or in the pathspec\n   (resulting in different error modes)\n(B) partial hints (We currently have a world of\n   completely missing hints, so I would not expect it to\n   be worse?)\n(C) wrong hints. This ought to be no problem as Git would\n   take some CPU time to conclude the hint was bogus.\n\nFor (A), I would imagine we want a mechanism (e.g. notes)\nto \"correct\" the hints. This is the similar issue as a typo in a\ncommit message, which we currently just ignore if the\ncommit has been merged to e.g. master.\n\nSo maybe we'd just design around that, giving the option\nto give the correct hints via command line.\n\nSo if the commit has the typo'd hint\n\n    Remame:  t/* -> tests/*\n\nthe human would see that (and also conclude that by\nthe commit message), and then invoke\n\ngit log -C -C-hint=\"t/* -> tests/*\" ...\n\nwhich would have the corrected hint and hence deliver\nthe best output.\n\nMaybe the \"-C-hint\" flag is the best starting point when\ngoing in that direction?\n\nThanks,\nStefan\n"},{"id":"329471","messageId":"ea1538e3-2b2e-f7eb-9c0e-e29c15bf2ea9@jeffhostetler.com","threadId":"46733","inReplyTo":"CAGZ79kbghnWmvQweup=Z79HnVQQCMM65CKgEO3oqDoRp-Bj=2Q@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-10-02T18:51:26Z","receivedAt":"2017-10-02T18:51:33Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/2/2017 1:41 PM, Stefan Beller wrote:\n>>> It would be nice if every file (and tree) had a permanent GUID\n>>> associated with it.  Then the filename/pathname becomes a property\n>>> of the GUIDs.  Then you can exactly know about moves/renames with\n>>> minimal effort (and no guessing).\n>>\n> ...\n> \n>> https://public-inbox.org/git/Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org/\n>>\n>> I'd encourge people to read and re-read that message until they can\n>> recite it by heart.\n> \n> I have rethought about the idea of GUIDs as proposed by Jeff and wanted\n> to give a reply. After rereading this message, I think my thoughts are\n> already included via:\n> \n>    - you're doing the work at the wrong point for _another_ reason. You're\n>       freezing your (crappy) algorithm at tree creation time, and basically\n>       making it pointless to ever create something better later, because even\n>       if hardware and software improves, you've codified that \"we have to\n>       have crappy information\".\n> \n> --\n> My design proposal for these \"rename hints\" would be a special trailer,\n> roughly:\n> \n>      Rename: LICENSE -> legal.txt\n>      Rename: t/* -> tests/*\n> \n> or more generally:\n> \n>      Rename: <pathspec> <delim> <pathspec>\n> \n> This however has multiple issues due to potential\n> human inaccuracies:\n> (A) typos in the trailer key or in the pathspec\n>     (resulting in different error modes)\n> (B) partial hints (We currently have a world of\n>     completely missing hints, so I would not expect it to\n>     be worse?)\n> (C) wrong hints. This ought to be no problem as Git would\n>     take some CPU time to conclude the hint was bogus.\n> \n> For (A), I would imagine we want a mechanism (e.g. notes)\n> to \"correct\" the hints. This is the similar issue as a typo in a\n> commit message, which we currently just ignore if the\n> commit has been merged to e.g. master.\n> \n> So maybe we'd just design around that, giving the option\n> to give the correct hints via command line.\n> \n> So if the commit has the typo'd hint\n> \n>      Remame:  t/* -> tests/*\n> \n> the human would see that (and also conclude that by\n> the commit message), and then invoke\n> \n> git log -C -C-hint=\"t/* -> tests/*\" ...\n> \n> which would have the corrected hint and hence deliver\n> the best output.\n> \n> Maybe the \"-C-hint\" flag is the best starting point when\n> going in that direction?\n> \n> Thanks,\n> Stefan\n> \n\nSorry to re-re-...-re-stir up such an old topic.\n\nI wasn't really thinking about commit-to-commit hints.\nI think these have lots of problems.  (If commit A->B does\n\"t/* -> tests/*\" and commit B->C does \"test/*.c -> xyx/*\",\nthen you need a way to compute a transitive closure to see\nthe net-net hints for A->C.  I think that quickly spirals\nout of control.)\n\nNo, I was going in another direction.  For example, if a\ntree-entry contains { file-guid, file-name, file-sha, ... }\nthen when diffing any 2 commits, you can match up files\n(and folders) by their guids.  Renames pop out trivially when\ntheir file-names don't match.  File moves pop out when the\nfile-guids appear in different trees.  Adds and deletes pop\nout when file-guids don't have a peer. (I'm glossing over some\nof the details, but you get the idea.)  To address Junio's\nquestion, independently added files with the same name will\nhave 2 different file-guids.  We amend the merge rules to\nhandle this case and pick one of them (say, the one that\nis sorts less than the other) as the winner and go on.\nAll-in-all the solution is not trivial (as there are a few\nedge cases to deal with), but it better matches the (casual)\nuser's perception of what happened to their tree over time.\nIt also doesn't require expensive code to sniff for renames\non every command (which doesn't scale on really large repos).\n\nBut as I said before, that ship has passed...\nJeff\n"},{"id":"329473","messageId":"CAGZ79kbjfXC3CxMDouUrCUVt-OJXckDtg9U_7=R=FM-eon4ikA@mail.gmail.com","threadId":"46733","inReplyTo":"ea1538e3-2b2e-f7eb-9c0e-e29c15bf2ea9@jeffhostetler.com","subject":"Re: [idea] File history tracking hints","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-02T19:18:59Z","receivedAt":"2017-10-02T19:19:05Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 2, 2017 at 11:51 AM, Jeff Hostetler <git@jeffhostetler.com> wrote:\n\n> Sorry to re-re-...-re-stir up such an old topic.\n>\n> I wasn't really thinking about commit-to-commit hints.\n> I think these have lots of problems.  (If commit A->B does\n> \"t/* -> tests/*\" and commit B->C does \"test/*.c -> xyx/*\",\n> then you need a way to compute a transitive closure to see\n> the net-net hints for A->C.  I think that quickly spirals\n> out of control.)\n\nI agree. Though as a human I can still look at\nA..C giving the hint that t/*.c and xyz/*.c ought to\nbe taken into account for rename detection.\n(which is currently done with -M -C --find-copies-harder\nas a generic \"there are renamed things\", and not the very\nspecific rule, that may be cheaper to examine compared to\nthese generic rules)\n\n> No, I was going in another direction.  For example, if a\n> tree-entry contains { file-guid, file-name, file-sha, ... }\n> then when diffing any 2 commits, you can match up files\n> (and folders) by their guids.  Renames pop out trivially when\n> their file-names don't match.  File moves pop out when the\n> file-guids appear in different trees.  Adds and deletes pop\n> out when file-guids don't have a peer. (I'm glossing over some\n> of the details, but you get the idea.)\n\nHow do you know when a guid needs adaption?\n\n(c.f. origin/jt/packmigrate)\nIf a commit moves a function out of a file into a new file,\nthe ideal version control could notice that the function\nwas moved into a new file and still attribute the original\nauthors by ignoring the move commit.\n\nAnother series in flight could have modified that\nfunction slightly (fixed a bug), such that it's hard to\nreason about these things.\n\nFor guids I imagine the new file gets a new guid, such that\ntracking the function becomes harder?\n\n\n> To address Junio's\n> question, independently added files with the same name will\n> have 2 different file-guids.  We amend the merge rules to\n> handle this case and pick one of them (say, the one that\n> is sorts less than the other) as the winner and go on.\n> All-in-all the solution is not trivial (as there are a few\n> edge cases to deal with), but it better matches the (casual)\n> user's perception of what happened to their tree over time.\n\nThe GUID would be made up at creation time, I assume?\nIs there any input other than the file itself? (I assumed so\ninitially, such that:\n  By having a GUID in the tree, we would divorce from the notion\n  of a \"content addressable file system\" quickly, as we both could\n  create the same tree locally (containing the same blobs) and\n  yet the trees would have different names due to having different\n  GUIDs in them\n), which I'd find undesirable.\n\n> It also doesn't require expensive code to sniff for renames\n> on every command (which doesn't scale on really large repos).\n\nI wonder if the rename detection could be offloaded to a server\n(which scales) that provides a \"hint file\" to clients, such that the\nclients can then cheaply make use of these specific hints.\n"},{"id":"329482","messageId":"f9b722d9-cd37-40f3-7ae4-6f7f3d90de83@jeffhostetler.com","threadId":"46733","inReplyTo":"CAGZ79kbjfXC3CxMDouUrCUVt-OJXckDtg9U_7=R=FM-eon4ikA@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-10-02T20:02:09Z","receivedAt":"2017-10-02T20:02:21Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 10/2/2017 3:18 PM, Stefan Beller wrote:\n> On Mon, Oct 2, 2017 at 11:51 AM, Jeff Hostetler <git@jeffhostetler.com> wrote:\n> \n>> Sorry to re-re-...-re-stir up such an old topic.\n>>\n>> I wasn't really thinking about commit-to-commit hints.\n>> I think these have lots of problems.  (If commit A->B does\n>> \"t/* -> tests/*\" and commit B->C does \"test/*.c -> xyx/*\",\n>> then you need a way to compute a transitive closure to see\n>> the net-net hints for A->C.  I think that quickly spirals\n>> out of control.)\n> \n> I agree. Though as a human I can still look at\n> A..C giving the hint that t/*.c and xyz/*.c ought to\n> be taken into account for rename detection.\n> (which is currently done with -M -C --find-copies-harder\n> as a generic \"there are renamed things\", and not the very\n> specific rule, that may be cheaper to examine compared to\n> these generic rules)\n> \n>> No, I was going in another direction.  For example, if a\n>> tree-entry contains { file-guid, file-name, file-sha, ... }\n>> then when diffing any 2 commits, you can match up files\n>> (and folders) by their guids.  Renames pop out trivially when\n>> their file-names don't match.  File moves pop out when the\n>> file-guids appear in different trees.  Adds and deletes pop\n>> out when file-guids don't have a peer. (I'm glossing over some\n>> of the details, but you get the idea.)\n> \n> How do you know when a guid needs adaption?\n\nI'm not sure I know what you mean by \"adaption\".\n\n> \n> (c.f. origin/jt/packmigrate)\n> If a commit moves a function out of a file into a new file,\n> the ideal version control could notice that the function\n> was moved into a new file and still attribute the original\n> authors by ignoring the move commit.\n\nI think that's an orthogonal problem.  I could move a function\nfrom one file to an existing file or to a new file it doesn't\nmatter.  Attributing those lines back to the original author\n(rather than the mover) is a bit of a pipe dream IMHO.  And I\nhave to wonder if it is always the correct thing to do?  I can\nsee scenarios where you'd want the mover.\n\nI guess there's nothing from stopping the \"ideal VC system\"\ndoing all this line-based analysis, but that shouldn't make\nfile renames expensive to detect (since that is the granularity\nthat people and most tools expect the system to work with).\n\n> \n> Another series in flight could have modified that\n> function slightly (fixed a bug), such that it's hard to\n> reason about these things.\n> \n> For guids I imagine the new file gets a new guid, such that\n> tracking the function becomes harder?\n> \n\nYeah, I'm not thinking about tracking individual functions.\n\n> \n>> To address Junio's\n>> question, independently added files with the same name will\n>> have 2 different file-guids.  We amend the merge rules to\n>> handle this case and pick one of them (say, the one that\n>> is sorts less than the other) as the winner and go on.\n>> All-in-all the solution is not trivial (as there are a few\n>> edge cases to deal with), but it better matches the (casual)\n>> user's perception of what happened to their tree over time.\n> \n> The GUID would be made up at creation time, I assume?\n> Is there any input other than the file itself? (I assumed so\n> initially, such that:\n>    By having a GUID in the tree, we would divorce from the notion\n>    of a \"content addressable file system\" quickly, as we both could\n>    create the same tree locally (containing the same blobs) and\n>    yet the trees would have different names due to having different\n>    GUIDs in them\n> ), which I'd find undesirable.\n\nRight.  A real solution would store the guid data slightly\ndifferently so we could preserve the existing SHA properties.\nMy example was more conceptual.\n\n> \n>> It also doesn't require expensive code to sniff for renames\n>> on every command (which doesn't scale on really large repos).\n> \n> I wonder if the rename detection could be offloaded to a server\n> (which scales) that provides a \"hint file\" to clients, such that the\n> clients can then cheaply make use of these specific hints.\n> \n\nI don't know.  Might be easier to add that computation to the\noccasional client-side housekeeping (somewhat like the commit\ngeneration number computation we keep talking about).\n\nThanks\nJeff\n"},{"id":"329506","messageId":"xmqq3771m54g.fsf@gitster.mtv.corp.google.com","threadId":"46733","inReplyTo":"CAGZ79kbghnWmvQweup=Z79HnVQQCMM65CKgEO3oqDoRp-Bj=2Q@mail.gmail.com","subject":"Re: [idea] File history tracking hints","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-03T00:45:03Z","receivedAt":"2017-10-03T00:45:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n> I have rethought about the idea of GUIDs as proposed by Jeff and wanted\n> to give a reply. After rereading this message, I think my thoughts are\n> already included via:\n>\n>   - you're doing the work at the wrong point for _another_ reason. You're\n>      freezing your (crappy) algorithm at tree creation time, and basically\n>      making it pointless to ever create something better later, because even\n>      if hardware and software improves, you've codified that \"we have to\n>      have crappy information\".\n>\n> --\n> My design proposal for these \"rename hints\" would be a special trailer,\n> roughly:\n>\n>     Rename: LICENSE -> legal.txt\n>     Rename: t/* -> tests/*\n>\n> or more generally:\n>\n>     Rename: <pathspec> <delim> <pathspec>\n\nYes, it is a non starter to have that baked in the log message of a\ncommit object.  The principle Linus lays out in the message does not\nreject such hints stored outside baked-in data structure, which\nallows mistakes to be corrected without affecting the real history,\nthough.\n\nAnother thing that makes what you wrote above of dubious value is\nthat it attaches such hints to \"a commit\" (whether baked inside the\nlog message, or as some form of \"notes\" that can be associated with\na specific commit); it adds hints at a wrong place.\n\nGiven identical pair of trees <X,Y> that are wrapped in two pairs of\ncommits <A> and <B> where A^{tree}=B^{tree} and A^^{tree}=B^^{tree},\nwe do not want to have to give duplicated hints for A and B, to help\n\"git show A\" and \"git show B\" to behave the same.\n\nRather, if we said \"these two blobs A and B are similar and we want\ndiffcore-rename to pair them, no matter where they appear in any two\ntrees\", then \"git diff -M X Y\", where X and Y may not have any\nancestry relationship (they may not even be commits) can be told\nthat the blob A that is in tree X and the blob B that is in tree Y\nare renames or copies, no matter where in these trees the pair of\nblobs appear, and no matter how X and Y are related (or unrelated)\nin the history.\n\nThat is a bigger reason why annotating a commit may be a bad way to\ngo.\n"},{"id":"329508","messageId":"xmqqy3otkq7v.fsf@gitster.mtv.corp.google.com","threadId":"46733","inReplyTo":"f9b722d9-cd37-40f3-7ae4-6f7f3d90de83@jeffhostetler.com","subject":"Re: [idea] File history tracking hints","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-03T00:52:20Z","receivedAt":"2017-10-03T00:52:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n>> How do you know when a guid needs adaption?\n>\n> I'm not sure I know what you mean by \"adaption\".\n\nI think he meant adapting, and I think he is referring to what I\nwrote in the message upthread to explain why \"file ID\" would not\nhelp.\n\nIt seems to me, from reading the remainder of your message, that it\nis also becoming clear to you that \"file ID\" would not help and your\nconceptual thing was merely a hand-waving that was dubious how it\ncould be made into a concrete working design?  Hopefully we can\nconverge on a workable design that does not involve \"file ID\", and\nthat would be a good outcome.\n\n\n"}]}