{"thread":{"id":"30435","subject":"Does content provenance matter?","startedAt":"2012-05-05T20:49:16Z","lastAt":"2012-05-08T09:23:53Z","messageCount":10,"participants":["Kelly Dean","Thomas Rast","PJ Weisberg","Jakub Narebski","Andrew Ardill","Junio C Hamano","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"190864","messageId":"1336250956.54413.YahooMailClassic@web121505.mail.ne1.yahoo.com","threadId":"30435","inReplyTo":null,"subject":"Does content provenance matter?","fromName":"Kelly Dean","fromEmail":"kellydeanch@yahoo.com","sentAt":"2012-05-05T20:49:16Z","receivedAt":"2012-05-05T20:49:16Z","isPatch":false,"sender":{"key":"kellydeanch@yahoo.com","avatar":null},"body":"Suppose you make dirs B and C, copy file X into B and C, insert \"foo\" somewhere into B/X and the same place into C/X, and commit. Now, you copy \"foo\" from B/X into the same place in the original X, and commit again. Git doesn't record the information about whether \"foo\" was copied from B or C, and this is intentional, on the theory that just content, not provenance, is what matters.\nSuppose instead, you branch master to new branches B and C, insert \"foo\" into B/X, commit, insert \"foo\" into C/X, and commit. Now, you merge B back into master. Git records that master contains \"foo\" because B contained it rather than because C contained it, on the theory that not only content, but also provenance, matters.\nDoes provenance actually matter, or not? The reason git doesn't record it in the first case isn't simply that your editor didn't store that information (and the editor didn't store it because it isn't customary to store it, and there's no standard way to store it); even if the editor were to store the information (e.g. as metadata for X; details not relevant) and a patch to git were submitted for it to record this metadata, the git maintainers would presumably reject this patch, on the basis that it violates git's design specification which says that provenance doesn't matter. For the same reason, git intentionally doesn't distinguish the cases of renaming a file or directory vs. deleting it and creating a new one with the same content, as has already been thoroughly debated.\nThe basic question is, if provenance doesn't matter, then why does a git commit record its parent(s)? Why not omit this information, and figure it out at search time (by looking at all commits with older timestamps), the same as you're supposed to figure out renames at search time and figure out the movement of lines within/among files at search time (by looking at all files in the parent commit(s))? (If speed is an issue, then use an index, but this doesn't require putting such derivative information in the commit record.)\n"},{"id":"190969","messageId":"878vh4flh1.fsf@thomas.inf.ethz.ch","threadId":"30435","inReplyTo":"1336250956.54413.YahooMailClassic@web121505.mail.ne1.yahoo.com","subject":"Re: Does content provenance matter?","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-05-07T08:23:38Z","receivedAt":"2012-05-07T08:23:38Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Kelly Dean writes:\n\n> [copying B/X over to C/X is not recorded as such], on the theory that\n> just content, not provenance, is what matters.\n\n> [merging branches *is* recorded], on the theory that not only content,\n> but also provenance, matters.\n\n> The basic question is, if provenance doesn't matter, then why does a\n> git commit record its parent(s)? Why not omit this information, and\n> figure it out at search time (by looking at all commits with older\n> timestamps), the same as you're supposed to figure out renames at\n> search time and figure out the movement of lines within/among files at\n> search time (by looking at all files in the parent commit(s))?\n\nWhat's the difference between the following series of commits?\n\n Foo\n Bar\n Revert Bar\n\nand\n\n Foo\n\nYou claim that they're the same, because the tree state after each is\nthe same. But I learned that Bar was broken, and recorded it for all to\nsee.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"191044","messageId":"1336427003.53220.YahooMailClassic@web121504.mail.ne1.yahoo.com","threadId":"30435","inReplyTo":"878vh4flh1.fsf@thomas.inf.ethz.ch","subject":"Re: Does content provenance matter?","fromName":"Kelly Dean","fromEmail":"kellydeanch@yahoo.com","sentAt":"2012-05-07T21:43:23Z","receivedAt":"2012-05-07T21:43:23Z","isPatch":false,"sender":{"key":"kellydeanch@yahoo.com","avatar":null},"body":"--- On Mon, 5/7/12, Thomas Rast wrote:\n> What's the difference between the following series of\n> commits?\n>\n> Foo\n> Bar\n> Revert Bar\n>\n> and\n>\n> Foo\n>\n> You claim that they're the same, because the tree state\n> after each is\n> the same. But I learned that Bar was broken, and\n> recorded it for all to see.\nNo, I don't claim they're the same. Different commits have different timestamps (and different commit messages, but that's not useful for automatic searching to find which commits are derived from which others). Consider if \"Revert Bar\" and \"Bar\" didn't point to their parents; could you still deduce from them that Bar was broken? Yes--on the basis of the commit timestamps (which shows their temporal order) and the contents of the trees which the commits point to (which shows that Revert Bar undoes a change made in Bar).\n"},{"id":"191047","messageId":"CAJsNXT=26u-PPNe9_LFoC1tAKpWJRWfBNNKzQB5Ya3X0n4K_eg@mail.gmail.com","threadId":"30435","inReplyTo":"1336427003.53220.YahooMailClassic@web121504.mail.ne1.yahoo.com","subject":"Re: Does content provenance matter?","fromName":"PJ Weisberg","fromEmail":"pj@irregularexpressions.net","sentAt":"2012-05-07T22:14:25Z","receivedAt":"2012-05-07T22:14:25Z","isPatch":false,"sender":{"key":"pj@irregularexpressions.net","avatar":"https://gravatar.com/avatar/aa2c1edcc61b536cc5c9f37fbce084e655446e2309f9818d43f13d47304a602b?d=mp&s=160"},"body":"On Mon, May 7, 2012 at 2:43 PM, Kelly Dean wrote:\n> --- On Mon, 5/7/12, Thomas Rast wrote:\n>> What's the difference between the following series of\n>> commits?\n>>\n>>   Foo\n>>   Bar\n>>   Revert Bar\n>>\n>> and\n>>\n>>   Foo\n>>\n>> You claim that they're the same, because the tree state\n>> after each is\n>> the same.  But I learned that Bar was broken, and\n>> recorded it for all to see.\n> No, I don't claim they're the same. Different commits have different timestamps (and different commit messages, but that's not useful for automatic searching to find which commits are derived from which others). Consider if \"Revert Bar\" and \"Bar\" didn't point to their parents; could you still deduce from them that Bar was broken? Yes--on the basis of the commit timestamps (which shows their temporal order) and the contents of the trees which the commits point to (which shows that Revert Bar undoes a change made in Bar).\n\nBut there could be any number of unrelated commits newer than \"Bar\"\nbut older than \"Revert Bar\" on other branches. Even if you could\ntrust the timestamps to be accurate (you can't), you still can't\ndetermine a commit's parent unambiguously.\n\n-PJ\n\nGehm's Corollary to Clark's Law: Any technology distinguishable from\nmagic is insufficiently advanced.\n"},{"id":"191051","messageId":"m3397ba8lh.fsf@localhost.localdomain","threadId":"30435","inReplyTo":"1336250956.54413.YahooMailClassic@web121505.mail.ne1.yahoo.com","subject":"Re: Does content provenance matter?","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2012-05-07T23:12:59Z","receivedAt":"2012-05-07T23:12:59Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Kelly Dean writes:\n\n> The basic question is, if provenance doesn't matter, then why does a\n> git commit record its parent(s)? Why not omit this information, and\n> figure it out at search time (by looking at all commits with older\n> timestamps),\n\nBecause it is not possible to do in reliable way in face of parallel\nconcurrent branchy development by multiple developers... some of which\ncan have badly set clock.\n\nAlso, multiple merges of the same branch.\n-- \nJakub Narebski\n"},{"id":"191053","messageId":"1336432417.36394.YahooMailClassic@web121504.mail.ne1.yahoo.com","threadId":"30435","inReplyTo":"CAJsNXT=26u-PPNe9_LFoC1tAKpWJRWfBNNKzQB5Ya3X0n4K_eg@mail.gmail.com","subject":"Re: Does content provenance matter?","fromName":"Kelly Dean","fromEmail":"kellydeanch@yahoo.com","sentAt":"2012-05-07T23:13:37Z","receivedAt":"2012-05-07T23:13:37Z","isPatch":false,"sender":{"key":"kellydeanch@yahoo.com","avatar":null},"body":"--- On Mon, 5/7/12, PJ Weisberg wrote:\n> But there could be any number of unrelated commits newer than \"Bar\"\n> but older than \"Revert Bar\" on other branches. Even if you could\n> trust the timestamps to be accurate (you can't), you still can't\n> determine a commit's parent unambiguously.\nTherefore, provenance does matter, and it must be explicitly recorded because it can't necessarily be correctly and fully deduced from content alone. And git does record inter-commit provenance.\nHowever, git doesn't record intra-commit provenance, as I mentioned in my original message. My question is: why this discrepancy? Either provenance matters, or it doesn't; why record it in one case but not the other?\n"},{"id":"191055","messageId":"CAH5451nq=W6qNsMExobipFXhb4tV+WmLu08QwfhzWd2+mJnbYA@mail.gmail.com","threadId":"30435","inReplyTo":"1336432417.36394.YahooMailClassic@web121504.mail.ne1.yahoo.com","subject":"Re: Does content provenance matter?","fromName":"Andrew Ardill","fromEmail":"andrew.ardill@gmail.com","sentAt":"2012-05-08T00:03:28Z","receivedAt":"2012-05-08T00:03:28Z","isPatch":false,"sender":{"key":"andrew.ardill@gmail.com","avatar":"https://gravatar.com/avatar/da14cb7c091dd44dc6c63a4d3361b149acaf25226dc78eb4131a17b93d9b0993?d=mp&s=160"},"body":"On 8 May 2012 09:13, Kelly Dean wrote:\n>\n> --- On Mon, 5/7/12, PJ Weisberg wrote:\n> > But there could be any number of unrelated commits newer than \"Bar\"\n> > but older than \"Revert Bar\" on other branches.  Even if you could\n> > trust the timestamps to be accurate (you can't), you still can't\n> > determine a commit's parent unambiguously.\n> Therefore, provenance does matter, and it must be explicitly recorded\n> because it can't necessarily be correctly and fully deduced from content\n> alone. And git does record inter-commit provenance.\n> However, git doesn't record intra-commit provenance, as I mentioned in my\n> original message. My question is: why this discrepancy? Either provenance\n> matters, or it doesn't; why record it in one case but not the other?\n\nI don't think it is firmly decided that provenance is not important in\nthe intra-commit scope, rather that as you stated such information is\nnot available to us.\n\nMy understanding is that git makes a best guess effort to track the\nflow of content through the repository. If the content is moved, by\ndeleting in one place and adding in another it is easy to see that in\ngit, however if content is merely added, and that same content occurs\nin multiple places in the repository, there is no sane way of knowing\nwhere that content came from.\nEven if the content that was added only occurred in one other place,\nyou would need to check every single file for every single hunk added\nevery single commit in order to be able to determine just where this\ncontent came from. Why stop there though? It's possible we are copying\nthe content from some other branch we don't have checked out at the\nmoment, so every time we commit, let's search the entire repositories\nhistory for an occurrence of each hunk we are adding. This way is\nmadness.\n\nWith regards to file renames, all that has been shown so far is that\nprovenance matters for commit renames. Nothing about the similarities\nbetween the commit parent and rename situations you mention leads me\nto concluded that because provenance is important to one it is\nimportant to the other.\n\nIndeed, one of the arguments against provenance being important in the\nfile rename case is that generally we can determine this information\nfrom the existing information, as opposed to the general commit parent\ncase. There are additional arguments, such as simply recording file\nname changes doesn't capture many situations we would like to know\nabout, for example when a single file is split into two files.\nTracking the content of those files, and hence being able to deduce\nwhere their content came from, solves this and the general rename\nsituation. Trying to guess which file was 'renamed' and which is 'new'\nwhen a file is actually split into two new files would lead to\nmisleading and incomplete information in the end.\n\nSo just because provenance matters in some situations doesn't mean it\nmatters in all (at least in the way we have been applying 'matters'),\nfurthermore there are additional reasons why the existing\ncontent-tracking system is beneficial. Extra layers of rename encoding\nor the 'heritage of data chunks' would be extra work with little added\nbenefit (though there are a few corner cases, from memory, where\nautomatic rename detection fails and so /some/ benefit would be seen).\n\nRegards,\n\nAndrew Ardill\n"},{"id":"191056","messageId":"7vaa1j357a.fsf@alter.siamese.dyndns.org","threadId":"30435","inReplyTo":"878vh4flh1.fsf@thomas.inf.ethz.ch","subject":"Re: Does content provenance matter?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-08T00:08:09Z","receivedAt":"2012-05-08T00:08:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Kelly Dean writes:\n>\n>> [copying B/X over to C/X is not recorded as such], on the theory that\n>> just content, not provenance, is what matters.\n>\n>> [merging branches *is* recorded], on the theory that not only content,\n>> but also provenance, matters.\n>\n>> The basic question is, if provenance doesn't matter, then why does a\n>> git commit record its parent(s)? Why not omit this information, and\n>> figure it out at search time (by looking at all commits with older\n>> timestamps), the same as you're supposed to figure out renames at\n>> search time and figure out the movement of lines within/among files at\n>> search time (by looking at all files in the parent commit(s))?\n>\n> What's the difference between the following series of commits?\n>\n> Foo\n> Bar\n> Revert Bar\n>\n> and\n>\n> Foo\n>\n> You claim that they're the same, because the tree state after each is\n> the same. But I learned that Bar was broken, and recorded it for all to\n> see.\n\nI am not sure if that is what the original poster was claiming.\n\nBut a more illustrative situation to consider is this. What if the change\nwere not just \"copy B/X to C/X\", but was \"concatenate the first half of\nB/X and the second half of C/X to create a new D/X\".\n\nAs it happens, because our commit records the whole tree state and its\nparent commit, the \"content provenance\" of what is in D/X is precisely\ntracked. Look at the tree of the parent commit and look at the result,\nand you will notice the first half of D/X is identical to the first half\nof B/X before the commit and the second half of D/X is identical to the\nsecond half of C/X before the commit.\n\nIn a situation where \"provenance\" is disputed, it does not matter if D/X\nwas created by mechanically running\n\n\thead -n $n B/X >D/X\n\ttail -n $n C/X >C/X\n \nor if you typed the file afresh. You could try to argue \"No, your honour,\nI did not copy from these two files. I typed it myself from scratch and\nthere is no plagiarism involved. They are all my words.\" But in the end,\nby comparing the tree state before your change and after your change, it\nwould be very clear to any sane person that D/X is identical to the first\nhalf of B/X and the second half of C/X.\n\nAlso see http://article.gmane.org/gmane.comp.version-control.git/217 aka\none of the most important messages in the history of the Git mailing list\nfor inspirations.\n"},{"id":"191058","messageId":"7v62c73528.fsf@alter.siamese.dyndns.org","threadId":"30435","inReplyTo":"7vaa1j357a.fsf@alter.siamese.dyndns.org","subject":"Re: Does content provenance matter?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-05-08T00:11:11Z","receivedAt":"2012-05-08T00:11:11Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> But a more illustrative situation to consider is this. What if the change\n> were not just \"copy B/X to C/X\", but was \"concatenate the first half of\n> B/X and the second half of C/X to create a new D/X\".\n\nA crucial question to the original poster was missing here:\n\n If \"a system that tracks provenance better than Git\" wanted to record\n something in such a situation, what does it record and how is the\n recorded information used?\n \n> As it happens, because our commit records the whole tree state and its\n> parent commit, the \"content provenance\" of what is in D/X is precisely\n> tracked. Look at the tree of the parent commit and look at the result,\n> and you will notice the first half of D/X is identical to the first half\n> of B/X before the commit and the second half of D/X is identical to the\n> second half of C/X before the commit.\n>\n> In a situation where \"provenance\" is disputed, it does not matter if D/X\n> was created by mechanically running\n>\n> \thead -n $n B/X >D/X\n> \ttail -n $n C/X >C/X\n\nTypo: \"tail -n $n C/X >>D/x\"\n\n> \n> or if you typed the file afresh. You could try to argue \"No, your honour,\n> I did not copy from these two files. I typed it myself from scratch and\n> there is no plagiarism involved. They are all my words.\" But in the end,\n> by comparing the tree state before your change and after your change, it\n> would be very clear to any sane person that D/X is identical to the first\n> half of B/X and the second half of C/X.\n>\n> Also see http://article.gmane.org/gmane.comp.version-control.git/217 aka\n> one of the most important messages in the history of the Git mailing list\n> for inspirations.\n"},{"id":"191078","messageId":"F5AF82B3BF334F0A88C8755426C725F0@PhilipOakley","threadId":"30435","inReplyTo":"1336432417.36394.YahooMailClassic@web121504.mail.ne1.yahoo.com","subject":"Re: Does content provenance matter?","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2012-05-08T09:23:53Z","receivedAt":"2012-05-08T09:23:53Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Kelly Dean\" Sent: Tuesday, May 08, 2012 12:13\nAM\n> --- On Mon, 5/7/12, PJ Weisberg wrote:\n>> But there could be any number of unrelated commits newer than \"Bar\"\n>> but older than \"Revert Bar\" on other branches. Even if you could\n>> trust the timestamps to be accurate (you can't), you still can't\n>> determine a commit's parent unambiguously.\n> Therefore, provenance does matter, and it must be explicitly recorded\n> because it can't necessarily be correctly and fully deduced from content\n> alone. And git does record inter-commit provenance.\n> However, git doesn't record intra-commit provenance, as I mentioned in my\n> original message. My question is: why this discrepancy?\n\n> Either provenance matters, or it doesn't;\n\nThe logic error is here. There are many other available choices as to\ndeciding the points at which the many provenance quality levels decay. (e.g.\nsee [1]).\n\nPeople eventually give up caring at some level of detail/history, each in a\ndifferent place ;-) It's a choice. e.g. Have you noticed all high\nperformance cars (Porche?) need brightly coloured brake cylinders with\ncarefully specified paint jobs - why? At some point we give up caring how\nsomeone got a few (how few?) characters into a file... It's not right, but\nit's not wrong either.\n\nI've worked with systems (e.g. DOORS) that record every keystroke, and\nrecored every hunk at the undo/redo level, but for little benefit.\n\nGit takes the approach of having lightweight (easy) branching with easy\ncommits, with local history re-writing (rebase), to give users the ability\nto balance between their WIP (work in progress), and their public record.\nBut with strong verification of any given history (e.g. \"My Master\"). It's a\nchoice as to who and where to blame.\n\n> why record it in one case but not the other?\n\nPhilip\n[1] Measuring and Managing Technological Knowledge,\nIEEE Engineering Management Review Winter 1997 p77-88.\nReprinted from Sloan Management Review, Fall 1994\nhttp://sloanreview.mit.edu/the-magazine/1994-fall/3615/measuring-and-managing-technological-knowledge/\n"}]}