{"thread":{"id":"14427","subject":"[RFH] Finding all commits that touch the same files as a specific commit","startedAt":"2008-07-12T15:58:38Z","lastAt":"2008-07-14T14:30:52Z","messageCount":7,"participants":["Sverre Rabbelier","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"83090","messageId":"bd6139dc0807120858vc058451lb10933b5225c8521@mail.gmail.com","threadId":"14427","inReplyTo":null,"subject":"[RFH] Finding all commits that touch the same files as a specific commit","fromName":"Sverre Rabbelier","fromEmail":"alturin@gmail.com","sentAt":"2008-07-12T15:58:38Z","receivedAt":"2008-07-12T15:58:38Z","isPatch":false,"sender":{"key":"alturin@gmail.com","avatar":null},"body":"Heya,\n\nCurrently I do the following:\n$git diff-tree --name-status --no-commit-id -r <hash>\nTo get all the files touched by the commit, I do:\n$git rev-list HEAD -- all the returned paths here\nThis works perfectly, except when the subtree merge strategy is used,\nsince in that case I get (example from git.git):\n$ git diff-tree --name-status --no-commit-id -r\n5821988f97b827f6ba81dfeebff932067c88ba6c\nM\tgit-gui.sh\nM\tlib/diff.tcl\n$ git rev-list HEAD -- git-gui.sh lib/diff.tcl\n$\n\nNow it was noticed on #git that git log has a --follow argument which\n-does- catch the rename, but it only works on one file at a time. So,\nmy question is this:\nHow do I find all commits that touch the same files as a specific commit?\nI have described my current approach above, which does not work when\nthe subtree merge strategy is used. I am not stuck to this approach\nthough, if someone comes up with a better way to do this than with\n'git diff-tree' / 'git rev-list' I'm fine by that. I provided with my\ncurrent approach in the hope that someone comes up with a similar\nsolution which means I'll have to edit less ;).\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"83119","messageId":"7viqvavao4.fsf@gitster.siamese.dyndns.org","threadId":"14427","inReplyTo":"bd6139dc0807120858vc058451lb10933b5225c8521@mail.gmail.com","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-13T01:24:11Z","receivedAt":"2008-07-13T01:24:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Sverre Rabbelier\" <alturin@gmail.com> writes:\n\n> Currently I do the following:\n> $git diff-tree --name-status --no-commit-id -r <hash>\n> To get all the files touched by the commit, I do:\n> $git rev-list HEAD -- all the returned paths here\n> This works perfectly, except when the subtree merge strategy is used,\n> since in that case I get (example from git.git):\n> $ git diff-tree --name-status --no-commit-id -r\n> 5821988f97b827f6ba81dfeebff932067c88ba6c\n> M\tgit-gui.sh\n> M\tlib/diff.tcl\n> $ git rev-list HEAD -- git-gui.sh lib/diff.tcl\n> $\n>\n> Now it was noticed on #git that git log has a --follow argument which\n> -does- catch the rename, but it only works on one file at a time. So,\n> my question is this:\n> How do I find all commits that touch the same files as a specific commit?\n> I have described my current approach above, which does not work when\n> the subtree merge strategy is used. I am not stuck to this approach\n> though, if someone comes up with a better way to do this than with\n> 'git diff-tree' / 'git rev-list' I'm fine by that. I provided with my\n> current approach in the hope that someone comes up with a similar\n> solution which means I'll have to edit less ;).\n\nFirst of all, a bad news that everybody should have known since day 1 when\nthe --follow option was introduced.  It merely is a cute hack that works\nmost of the time in trivial histories.  The data structure it uses cannot\nreliably follow renames if you have any nontrivial history.\n\nRevision traversal machinery has a single list of pathspecs to filter the\nresults with, and in the usual traversal, the list never changes.  That is\nwhy you would need to give a list of three pathspecs upfront, like this:\n\n\tgit log -- arch/i386 arch/x86 arch/x86_64\n\nto get the whole picture of how things are consolidated into a single\narch/x86 hierarchy over time from originally two hierarchies.  The\nrevision traversal works by simplifying away commits that do not touch\npath that match any of the given pathspecs, so giving the \"current\" path\n(i.e. arch/x86) is not sufficient.\n\nThe --follow changes the behaviour slightly.  When you have this history:\n\n    ---o---o---o---x---x---x\n\nwhere a file you are interested in (say, arch/i386/kernel/reboot.c)\nexisted in the past in 'o' commits, but was renamed to something else\n(say, arch/x86/kernel/reboot.c) in newer 'x' commits, you would start\nfollowing from the tip of the history like this:\n\n\tgit log --follow arch/x86/kernel/reboot.c\n\nAnd the machinery traverses down the history, showing only the commits\nthat touch the given path.  An interesting thing happens, however, when it\nhits the earliest 'x' commit and realizes that its parent 'o' does not\nhave that path.  It runs the rename detection there, realizes the path it\nis interested in corresponds to a different path in the parent, and\n_updates_ the pathspec to the old name.  I.e. it will from that point on\nbehaves as if you started digging from the tip of this history:\n\n    ---o---o---o\n\nwith a different pathspec:\n\n\tgit log --follow arch/i386/kernel/reboot.c\n\nThis works as long as your history is trivial, but in real life, the world\nis not linear.\n\n          x---x---x---x\n         /       /\n    ----o---o---o\n\nIf commits 'x' have git-gui/git-gui.sh and commits 'o' have git-gui.sh at\nthe root level, you would start digging from the tip with --follow:\n\n\tgit log --follow git-gui/git-gui.sh\n\nWhen it hits the rightmost merge 'x', it realizes the changes to the file\ncame from lower history and switches the pathspec to \"git-gui.sh\" at the\nroot level (the commits that have already been traversed are marked with\nuppercase latters here).\n\n          x---x---X---X\n         /       /\n    ----o---o---O\n\nSwitching the pathspec from \"git-gui/git-gui.sh\" to \"git-gui.sh\" is fine\nfor the purpose of traversing the 'o' history down, but there is a\nproblem.  Remember I said there is a _single_ list of pathspecs the\nrevision traversal machinery keeps track of?  If you switch that single\nlist to \"git-gui.sh\", it means you completely forget that you were\nfollowing \"git-gui/git-gui.sh\".  You cannot follow the upper history\nanymore.\n\nIn order to follow renames reliably in a merge heavy history, you need to\nkeep track of the pathname the file you are interested in appears as _in\neach commit_.  As you traverse down the history, you pass down the\npathname to the parent you visit, so while you are traversing from 'x' to\nearlier 'x', you will keep following \"git-gui/git-gui.sh\", while you\ntraverse down to 'o', you will inspect \"git-gui.sh\".\n\nThe data structure the revision traversal machinery uses does not support\nthis \"path-per-commit\" natively.\n\nThis is the reason \"git-blame\" uses its own traversal engine.  It keeps\ntrack of <commit, path> pairs so that it can mark which line came from\nwhat path in what commit.  When copy/move detection are used, we can even\nnotice that the contents we are interested in came from more than one file\nin the same commits, and the data structure supports it (i.e. it is not\njust a pointer to a single string from \"struct commit\").\n\nFor the purpose of \"git log\" traversal and the \"file renames\" people\nusually talk about, this is overkill; you should however be able to\nbackport the basic idea to revision machinery, if you really cared.\n\nIn a real history, \"file rename\" is a very ill defined concept and is not\nalways useful in practice.  I did a fairly detailed analysis on one\nreal-world history more than two years ago, which is found here:\n\n    http://thread.gmane.org/gmane.comp.version-control.git/13746/focus=13769\n\nIn our own \"git.git\" history, the evolution of what finally landed in\nrevision.c is interesting.  The interesting part of content movement never\ninvolved any file renames --- only bits and pieces migrated over across\nmany files.  That is not something \"file rename tracking\", even with an\nextension to the revision traversal machinery to keep one path per commit\nto record the file you are interested in, can ever give meaningful\nexplanation of the history.  You need a lot more fine grained \"blame\"\ntraversal machinery for that.\n"},{"id":"83153","messageId":"bd6139dc0807130743y178a865amebcf130a68283854@mail.gmail.com","threadId":"14427","inReplyTo":"7viqvavao4.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Sverre Rabbelier","fromEmail":"alturin@gmail.com","sentAt":"2008-07-13T14:43:21Z","receivedAt":"2008-07-13T14:43:21Z","isPatch":false,"sender":{"key":"alturin@gmail.com","avatar":null},"body":"On Sun, Jul 13, 2008 at 3:24 AM, Junio C Hamano <gitster@pobox.com> wrote:\n\n<explanation of the git log traversal machinery snipped>\n\n> In order to follow renames reliably in a merge heavy history, you need to\n> keep track of the pathname the file you are interested in appears as _in\n> each commit_.  As you traverse down the history, you pass down the\n> pathname to the parent you visit, so while you are traversing from 'x' to\n> earlier 'x', you will keep following \"git-gui/git-gui.sh\", while you\n> traverse down to 'o', you will inspect \"git-gui.sh\".\n>\n> The data structure the revision traversal machinery uses does not support\n> this \"path-per-commit\" natively.\n\nWould it be possible to go for a slightly less complicated approach\nand instead of passing replacing the tracked file, append it? We\nalready have a list of files we are tracking, so I assume the data\nstructure does support that. Such would run with the risk of tracking\ntoo much (e.g., you rename a.txt => b.txt, and then later on\ncreate/rename a new a.txt which is then tracked as well).\n\n> This is the reason \"git-blame\" uses its own traversal engine.  It keeps\n> track of <commit, path> pairs so that it can mark which line came from\n> what path in what commit.  When copy/move detection are used, we can even\n> notice that the contents we are interested in came from more than one file\n> in the same commits, and the data structure supports it (i.e. it is not\n> just a pointer to a single string from \"struct commit\").\n\nSo what could be done is use a blame-like mechanism that invokes\nrename detection on each interesting commit and then record that\ninformation? Purely hypothetical though, since I know neither and have\nno time to do so.\n\n> For the purpose of \"git log\" traversal and the \"file renames\" people\n> usually talk about, this is overkill; you should however be able to\n> backport the basic idea to revision machinery, if you really cared.\n\nRight, that'd teach git log how to follow across renames in an\nintelligent manner that works also for non-linear histories at the\ncost of using up more memory and cpu?\n\n> In a real history, \"file rename\" is a very ill defined concept and is not\n> always useful in practice.  I did a fairly detailed analysis on one\n> real-world history more than two years ago, which is found here:\n>\n>    http://thread.gmane.org/gmane.comp.version-control.git/13746/focus=13769\n\nAye, I agree that a 'rename' is hard to define and that a lot of\neffort could be put into supporting 'renames' that are not trivial\n(e.g., more complex than 'git mv foo.txt bar.txt').\n\n> In our own \"git.git\" history, the evolution of what finally landed in\n> revision.c is interesting.  The interesting part of content movement never\n> involved any file renames --- only bits and pieces migrated over across\n> many files.  That is not something \"file rename tracking\", even with an\n> extension to the revision traversal machinery to keep one path per commit\n> to record the file you are interested in, can ever give meaningful\n> explanation of the history.  You need a lot more fine grained \"blame\"\n> traversal machinery for that.\n\nThis makes sense, but it (using blame traversal machinery) is overkill\nfor what I am interested in. What I think would be a good goal in\nsupporting is the subtree merge strategy. It would be nice if 'git log\n--follow-subtree-merge refspec -- filefilter' or such would Just Work\n(TM). Perhaps that the hunk-tracking I am working on with Dscho could\nhelp make 'git log --numstat' more accurate. Those two combined (git\nlog being able to follow across subtree merges and being able to\nrecognise hunks being moved) would be all that I need.\n\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"83159","messageId":"alpine.DEB.1.00.0807131929200.8950@racer","threadId":"14427","inReplyTo":"bd6139dc0807130743y178a865amebcf130a68283854@mail.gmail.com","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-13T18:30:27Z","receivedAt":"2008-07-13T18:30:27Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 13 Jul 2008, Sverre Rabbelier wrote:\n\n> On Sun, Jul 13, 2008 at 3:24 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> <explanation of the git log traversal machinery snipped>\n> \n> > In order to follow renames reliably in a merge heavy history, you need to\n> > keep track of the pathname the file you are interested in appears as _in\n> > each commit_.  As you traverse down the history, you pass down the\n> > pathname to the parent you visit, so while you are traversing from 'x' to\n> > earlier 'x', you will keep following \"git-gui/git-gui.sh\", while you\n> > traverse down to 'o', you will inspect \"git-gui.sh\".\n> >\n> > The data structure the revision traversal machinery uses does not support\n> > this \"path-per-commit\" natively.\n> \n> Would it be possible to go for a slightly less complicated approach\n> and instead of passing replacing the tracked file, append it?\n\nMaybe I miss something, but do you not have to keep track of the file \nnames, in order to keep track of the proper statistics?\n\nIf that is the case, appending does not cut it.\n\nCiao,\nDscho\n"},{"id":"83247","messageId":"bd6139dc0807140417h7c9cd309g7cf911e78e7f78c5@mail.gmail.com","threadId":"14427","inReplyTo":"alpine.DEB.1.00.0807131929200.8950@racer","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Sverre Rabbelier","fromEmail":"alturin@gmail.com","sentAt":"2008-07-14T11:17:09Z","receivedAt":"2008-07-14T11:17:09Z","isPatch":false,"sender":{"key":"alturin@gmail.com","avatar":null},"body":"On Sun, Jul 13, 2008 at 8:30 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> On Sun, 13 Jul 2008, Sverre Rabbelier wrote:\n>> On Sun, Jul 13, 2008 at 3:24 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>> > The data structure the revision traversal machinery uses does not support\n>> > this \"path-per-commit\" natively.\n>>\n>> Would it be possible to go for a slightly less complicated approach\n>> and instead of passing replacing the tracked file, append it?\n>\n> Maybe I miss something, but do you not have to keep track of the file\n> names, in order to keep track of the proper statistics?\n\nHmm, no, this is just to get commits that touched a (set of) file(s).\nI use it to limit the commits I have to check when searching for\nreverts.\n\n> If that is the case, appending does not cut it.\n\nFor the activity metric I think pretending that all files with the\nsame name or renamed versions of those would make sense, which is what\nappending the new name would do. The downside is that all files with\nthe same name get grouped together, I'm not sure which is the lesser\nof two evils. Leaving out information, or (possibly) including too\nmuch.\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"83256","messageId":"alpine.DEB.1.00.0807141311520.8950@racer","threadId":"14427","inReplyTo":"bd6139dc0807140417h7c9cd309g7cf911e78e7f78c5@mail.gmail.com","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-14T12:13:28Z","receivedAt":"2008-07-14T12:13:28Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 14 Jul 2008, Sverre Rabbelier wrote:\n\n> For the activity metric I think pretending that all files with the same \n> name or renamed versions of those would make sense, which is what \n> appending the new name would do. The downside is that all files with the \n> same name get grouped together, I'm not sure which is the lesser of two \n> evils. Leaving out information, or (possibly) including too much.\n\nIMO following the file renames/code moves precisely is really worth the \ntime it takes to calculate.  Otherwise, the statistics will not reflect \nwhat was going on in the project, right?\n\nCiao,\nDscho\n"},{"id":"83266","messageId":"bd6139dc0807140730o2cdc64d0m28a83dfa72d1496b@mail.gmail.com","threadId":"14427","inReplyTo":"alpine.DEB.1.00.0807141311520.8950@racer","subject":"Re: [RFH] Finding all commits that touch the same files as a specific commit","fromName":"Sverre Rabbelier","fromEmail":"alturin@gmail.com","sentAt":"2008-07-14T14:30:52Z","receivedAt":"2008-07-14T14:30:52Z","isPatch":false,"sender":{"key":"alturin@gmail.com","avatar":null},"body":"On Mon, Jul 14, 2008 at 2:13 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> IMO following the file renames/code moves precisely is really worth the\n> time it takes to calculate.  Otherwise, the statistics will not reflect\n> what was going on in the project, right?\n\nAh, I did not mean to imply that it is not worth the time it takes to\ncalculate, more that I do not know how to implement it that way. Of\ncourse if someone has the time/motivation to do so I would very much\nmake use of it, but I do not have time to do so myself (at least not\nuntil after GSoC). I can have GitStats just ignore the subtree merge\ncases (and say that it is not beyond the scope of my project to take\nsuch into account) and have a working program by the end of GSoC. But\nif I spend time on getting this to work instead I might end up with a\nprogram that does follow sub-tree merges but is only half-done, and as\nsuch probably won't receive a \"OK\" grade. So, yes, I would very much\nlike to see this, but no time to look into doing so myself until after\nGSoC.\n\n-- \nCheers,\n\nSverre Rabbelier\n"}]}