{"thread":{"id":"49333","subject":"[Question] Signature calculation ignoring parts of binary files","startedAt":"2018-09-12T19:16:21Z","lastAt":"2018-09-13T17:55:28Z","messageCount":10,"participants":["Randall S. Becker","Johannes Sixt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"357998","messageId":"003901d44acd$12c2bb90$384832b0$@nexbridge.com","threadId":"49333","inReplyTo":null,"subject":"[Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-12T19:16:07Z","receivedAt":"2018-09-12T19:16:21Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"I feel really bad asking this, and I should know the answer, and yet.\n\nI have a binary file that needs to go into a repo intact (unchanged). I also\nhave a program that interprets the contents, like a textconv, that can\noutput the relevant portions of the file in whatever format I like - used\nfor diff typically, dumps in 1K chunks by file section. What I'm looking for\nis to have the SHA1 signature calculated with just the relevant portions of\nthe file so that two actually different files will be considered the same by\ngit during a commit or status. In real terms, I'm trying to ignore the\nCreator metadata of a JPG because it is mutable and irrelevant to my repo\ncontents.\n\nI'm sorry to ask, but I thought this was in .gitattributes but I can't\nconfirm the SHA1 behaviour.\n\nSheepishly,\nRandall\n\n\n-- Brief whoami:\n NonStop developer since approximately 211288444200000000\n UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n\n"},{"id":"358006","messageId":"4102803b-d5de-5812-4cef-569c9869fed1@kdbg.org","threadId":"49333","inReplyTo":"003901d44acd$12c2bb90$384832b0$@nexbridge.com","subject":"Re: [Question] Signature calculation ignoring parts of binary files","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2018-09-12T20:48:04Z","receivedAt":"2018-09-12T20:48:08Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 12.09.18 um 21:16 schrieb Randall S. Becker:\n> I feel really bad asking this, and I should know the answer, and yet.\n> \n> I have a binary file that needs to go into a repo intact (unchanged). I also\n> have a program that interprets the contents, like a textconv, that can\n> output the relevant portions of the file in whatever format I like - used\n> for diff typically, dumps in 1K chunks by file section. What I'm looking for\n> is to have the SHA1 signature calculated with just the relevant portions of\n> the file so that two actually different files will be considered the same by\n> git during a commit or status. In real terms, I'm trying to ignore the\n> Creator metadata of a JPG because it is mutable and irrelevant to my repo\n> contents.\n> \n> I'm sorry to ask, but I thought this was in .gitattributes but I can't\n> confirm the SHA1 behaviour.\n\nYou are looking for a clean filter. See the 'filter' attribute in \ngitattributes(5). Your clean filter program or script should strip the \nunwanted metadata or set it to a constant known-good value.\n\n(You shouldn't need a smudge filter.)\n\n-- Hannes\n"},{"id":"358007","messageId":"004e01d44ada$b4a11ad0$1de35070$@nexbridge.com","threadId":"49333","inReplyTo":"4102803b-d5de-5812-4cef-569c9869fed1@kdbg.org","subject":"RE: [Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-12T20:53:43Z","receivedAt":"2018-09-12T20:53:59Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"> -----Original Message-----\n> From: git-owner@vger.kernel.org <git-owner@vger.kernel.org> On Behalf\n> Of Johannes Sixt\n> Sent: September 12, 2018 4:48 PM\n> To: Randall S. Becker <rsbecker@nexbridge.com>\n> Cc: git@vger.kernel.org\n> Subject: Re: [Question] Signature calculation ignoring parts of binary files\n> \n> Am 12.09.18 um 21:16 schrieb Randall S. Becker:\n> > I feel really bad asking this, and I should know the answer, and yet.\n> >\n> > I have a binary file that needs to go into a repo intact (unchanged).\n> > I also have a program that interprets the contents, like a textconv,\n> > that can output the relevant portions of the file in whatever format I\n> > like - used for diff typically, dumps in 1K chunks by file section.\n> > What I'm looking for is to have the SHA1 signature calculated with\n> > just the relevant portions of the file so that two actually different\n> > files will be considered the same by git during a commit or status. In\n> > real terms, I'm trying to ignore the Creator metadata of a JPG because\n> > it is mutable and irrelevant to my repo contents.\n> >\n> > I'm sorry to ask, but I thought this was in .gitattributes but I can't\n> > confirm the SHA1 behaviour.\n> \n> You are looking for a clean filter. See the 'filter' attribute in gitattributes(5).\n> Your clean filter program or script should strip the unwanted metadata or set\n> it to a constant known-good value.\n> \n> (You shouldn't need a smudge filter.)\n> \n> -- Hannes\n\nThanks Hannes. I thought about the clean filter, but I don't actually want to modify the file when going into git, just for SHA calculation. I need to be able to keep some origin metadata that might change with subsequent copies, so just cleaning the origin is not going to work - actually knowing the original author is important to our process. My objective is to keep the original file 100% exact as supplied and then ignore any changes to the metadata that I don't care about (like Creator) if the remainder of the file is the same.\n\nRegards,\nRandall\n\n\n"},{"id":"358016","messageId":"000001d44ae6$c2a20ac0$47e62040$@nexbridge.com","threadId":"49333","inReplyTo":"004e01d44ada$b4a11ad0$1de35070$@nexbridge.com","subject":"RE: [Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-12T22:20:00Z","receivedAt":"2018-09-12T22:20:13Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 12, 2018 4:54 PM, I wrote:\n> On September 12, 2018 4:48 PM, Johannes Sixt wrote:\n> > Am 12.09.18 um 21:16 schrieb Randall S. Becker:\n> > > I feel really bad asking this, and I should know the answer, and yet.\n> > >\n> > > I have a binary file that needs to go into a repo intact (unchanged).\n> > > I also have a program that interprets the contents, like a textconv,\n> > > that can output the relevant portions of the file in whatever format\n> > > I like - used for diff typically, dumps in 1K chunks by file section.\n> > > What I'm looking for is to have the SHA1 signature calculated with\n> > > just the relevant portions of the file so that two actually\n> > > different files will be considered the same by git during a commit\n> > > or status. In real terms, I'm trying to ignore the Creator metadata\n> > > of a JPG because it is mutable and irrelevant to my repo contents.\n> > >\n> > > I'm sorry to ask, but I thought this was in .gitattributes but I\n> > > can't confirm the SHA1 behaviour.\n> >\n> > You are looking for a clean filter. See the 'filter' attribute in gitattributes(5).\n> > Your clean filter program or script should strip the unwanted metadata\n> > or set it to a constant known-good value.\n> >\n> > (You shouldn't need a smudge filter.)\n> >\n> > -- Hannes\n> \n> Thanks Hannes. I thought about the clean filter, but I don't actually want to\n> modify the file when going into git, just for SHA calculation. I need to be able\n> to keep some origin metadata that might change with subsequent copies, so\n> just cleaning the origin is not going to work - actually knowing the original\n> author is important to our process. My objective is to keep the original file\n> 100% exact as supplied and then ignore any changes to the metadata that I\n> don't care about (like Creator) if the remainder of the file is the same.\n\nI had a thought that might be workable, opinions are welcome on this.\n\nThe commit of my rather weird project is done by a script so I have flexibility in my approach. What I could do is set up a diff textconv configuration so that the text diff of the two JPG files will show no differences if the immutable fields and the image are the same. I can then trigger a git add and git commit for only those files where git diff reports no differences. That way the actual original file is stored in git with 100% fidelity (no cleaning). It's not as elegant as I'd like, but it does solve what I'm trying to do. Does this sound reasonable and/or is there a better way?\n\nCheers,\nRandall\n\n-- Brief whoami:\n NonStop developer since approximately 211288444200000000\n UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n\n"},{"id":"358022","messageId":"xmqq1s9yjp9j.fsf@gitster-ct.c.googlers.com","threadId":"49333","inReplyTo":"000001d44ae6$c2a20ac0$47e62040$@nexbridge.com","subject":"Re: [Question] Signature calculation ignoring parts of binary files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-12T22:59:36Z","receivedAt":"2018-09-12T22:59:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n\n>> author is important to our process. My objective is to keep the original file\n>> 100% exact as supplied and then ignore any changes to the metadata that I\n>> don't care about (like Creator) if the remainder of the file is the same.\n\nThat will *not* work.  If person A gave you a version of original,\nwhich hashes to X after you strip the cruft you do not care about,\nyou would register that original with person A's fingerprint on\nunder the name of X.  What happens when person B gives you another\nversion, which is not byte-for-byte identical to the one you got\nearlier from person A, but does hash to the same X after you strip\nthe cruft?  If you are going to store it in Git, and if by SHA-1 you\nare calling what we perceive as \"object name\" in Git land, you must\nstore that one with person B's fingerprint on it also under the name\nof X.  Now which version will you get from Git when you ask it to\ngive you the object that hashes to X?  \n"},{"id":"358044","messageId":"001101d44b5b$fa337350$ee9a59f0$@nexbridge.com","threadId":"49333","inReplyTo":"xmqq1s9yjp9j.fsf@gitster-ct.c.googlers.com","subject":"RE: [Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-13T12:19:04Z","receivedAt":"2018-09-13T12:19:25Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 12, 2018 7:00 PM, Junio C Hamano wrote:\n> \"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n> \n> >> author is important to our process. My objective is to keep the\n> >> original file 100% exact as supplied and then ignore any changes to\n> >> the metadata that I don't care about (like Creator) if the remainder of\nthe\n> file is the same.\n> \n> That will *not* work.  If person A gave you a version of original, which\n> hashes to X after you strip the cruft you do not care about, you would\n> register that original with person A's fingerprint on under the name of X.\n> What happens when person B gives you another version, which is not byte-\n> for-byte identical to the one you got earlier from person A, but does hash\nto\n> the same X after you strip the cruft?  If you are going to store it in\nGit, and if\n> by SHA-1 you are calling what we perceive as \"object name\" in Git land,\nyou\n> must store that one with person B's fingerprint on it also under the name\nof\n> X.  Now which version will you get from Git when you ask it to give you\nthe\n> object that hashes to X?\n\nThe scenario is slightly different.\n1. Person A gives me a new binary file-1 with fingerprint A1. This goes into\ngit unchanged.\n2. Person B gives me binary file-2 with fingerprint B2. This does not go\ninto git yet.\n3. We attempt a git diff between the committed file-1 and uncommitted file-2\nusing a textconv implementation that strips what we don't need to compare.\n4. If file-1 and file-2 have no difference when textconv is used, file-2 is\nnot added and not committed. It is discarded with impunity, never to be seen\nagain, although we might whine a lot at the user for attempting to put\nfile-2 in - but that's not git's issue.\n5. If file-1 and file-2 have differences when textconv is used, file-2 is\ncommitted with fingerprint B2.\n6. Even if an error is made by the user and they commit file-2 with B2\nregardless of textconv, there will be a human who complains about it, but\ngit has two unambiguous fingerprints that happen to have no diffs after\ntextconv is applied.\n\nMy original hope was that textconv could be used to influence the\nfingerprint, but I do not think that is the case, so I went with an\nalternative. In the application, I am not allowed to strip any cruft off\nfile-1 when it is stored - it must be byte-for-byte the original file. This\napplication is marginally related to a DRM-like situation where we only care\nabout the original image provided by a user, but any copies that are\nprovided by another user with modified metadata will be disallowed from\nrepository.\n\nDoes that make more sense? \n\nCheers,\nRandall\n\n"},{"id":"358052","messageId":"xmqqefdxign2.fsf@gitster-ct.c.googlers.com","threadId":"49333","inReplyTo":"001101d44b5b$fa337350$ee9a59f0$@nexbridge.com","subject":"Re: [Question] Signature calculation ignoring parts of binary files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-13T15:03:29Z","receivedAt":"2018-09-13T15:03:34Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n\n> The scenario is slightly different.\n> 1. Person A gives me a new binary file-1 with fingerprint A1. This goes into\n> git unchanged.\n> 2. Person B gives me binary file-2 with fingerprint B2. This does not go\n> into git yet.\n> 3. We attempt a git diff between the committed file-1 and uncommitted file-2\n> using a textconv implementation that strips what we don't need to compare.\n> 4. If file-1 and file-2 have no difference when textconv is used, file-2 is\n> not added and not committed. It is discarded with impunity, never to be seen\n> again, although we might whine a lot at the user for attempting to put\n> file-2 in - but that's not git's issue.\n\nYou are forgetting that Git is a distributed version control system,\naren't you?  Person A and B can introduce their \"moral equivalent\nbut bytewise different\" copies to their repository under the same\nobject name, and you can pull from them--what happens?\n\nIt is fundamental that one object name given to Git identifies one\nspecific byte sequence contained in an object uniquely.  Once you\nbroke that, you no longer have Git.\n\n"},{"id":"358054","messageId":"003a01d44b77$d0b289a0$72179ce0$@nexbridge.com","threadId":"49333","inReplyTo":"xmqqefdxign2.fsf@gitster-ct.c.googlers.com","subject":"RE: [Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-13T15:38:21Z","receivedAt":"2018-09-13T15:38:38Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 13, 2018 11:03 AM, Junio C Hamano wrote:\n> \"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n> \n> > The scenario is slightly different.\n> > 1. Person A gives me a new binary file-1 with fingerprint A1. This\n> > goes into git unchanged.\n> > 2. Person B gives me binary file-2 with fingerprint B2. This does not\n> > go into git yet.\n> > 3. We attempt a git diff between the committed file-1 and uncommitted\n> > file-2 using a textconv implementation that strips what we don't need to\n> compare.\n> > 4. If file-1 and file-2 have no difference when textconv is used,\n> > file-2 is not added and not committed. It is discarded with impunity,\n> > never to be seen again, although we might whine a lot at the user for\n> > attempting to put\n> > file-2 in - but that's not git's issue.\n> \n> You are forgetting that Git is a distributed version control system,\naren't you?\n> Person A and B can introduce their \"moral equivalent but bytewise\ndifferent\"\n> copies to their repository under the same object name, and you can pull\nfrom\n> them--what happens?\n> \n> It is fundamental that one object name given to Git identifies one\nspecific\n> byte sequence contained in an object uniquely.  Once you broke that, you\nno\n> longer have Git.\n\nAt that point I have a morally questionable situation, agreed. However, both\nare permitted to exist in the underlying tree without conflict in git -\nwhich I do consider a legitimately possible situation that will not break\nthe application at all - although there is a semantic conflict in the\napplication (not in git) that requires human decision to resolve. The fact\nthat both objects can exist in git with different fingerprints is a good\nthing because it provides immutable evidence and ownership of someone\nbypassing the intent of the application.\n\nSo, rather than using textconv, I shall implement this rule in the\napplication rather than trying to configure git to do it. If two conflicting\nobjects enter the commit history, the application will have the\nresponsibility to resolve the semantic/legal conflict.\n\nThanks,\nRandall\n\n\n"},{"id":"358063","messageId":"xmqqva79ffpv.fsf@gitster-ct.c.googlers.com","threadId":"49333","inReplyTo":"xmqqefdxign2.fsf@gitster-ct.c.googlers.com","subject":"Re: [Question] Signature calculation ignoring parts of binary files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-09-13T17:51:40Z","receivedAt":"2018-09-13T17:51:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> \"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n>\n>> The scenario is slightly different.\n>> 1. Person A gives me a new binary file-1 with fingerprint A1. This goes into\n>> git unchanged.\n>> 2. Person B gives me binary file-2 with fingerprint B2. This does not go\n>> into git yet.\n>> 3. We attempt a git diff between the committed file-1 and uncommitted file-2\n>> using a textconv implementation that strips what we don't need to compare.\n>> 4. If file-1 and file-2 have no difference when textconv is used, file-2 is\n>> not added and not committed. It is discarded with impunity, never to be seen\n>> again, although we might whine a lot at the user for attempting to put\n>> file-2 in - but that's not git's issue.\n>\n> You are forgetting that Git is a distributed version control system,\n> aren't you?  Person A and B can introduce their \"moral equivalent\n> but bytewise different\" copies to their repository under the same\n> object name, and you can pull from them--what happens?\n>\n> It is fundamental that one object name given to Git identifies one\n> specific byte sequence contained in an object uniquely.  Once you\n> broke that, you no longer have Git.\n\nHaving said all that, if you want to keep the original with frills\nbut somehow give these bytewise different things that reduce to the\nsame essence (e.g. when passed thru a filter like textconv), I\nsuspect a better approach might be to store both the \"original\" and\nthe result of passing the \"original\" through the filter in the\nobject database.  In the above example, you'll get two \"original\"\nobjects from person A and person B, plus one \"canonical\" object that\nare bytewise different from either of these two originals, but what\nthey reduce to when you use the filter on them.  Then you record the\nfact that to derive the \"essence\" object, you can reduce either\nperson A's or person B's \"original\" through the filter, perhaps by\nusing \"git notes\" attached to the \"essence\" object, recording the\nobject names of these originals (the reason why using notes in this\ndirection is because you can mechanically determine which \"essence\"\nobject any given \"original\" object reduces to---it is just the\nmatter of passing it through the filter.  But there can be more than\none \"original\" that reduces to the same \"essence\").\n\n"},{"id":"358065","messageId":"005e01d44b8a$ef1221e0$cd3665a0$@nexbridge.com","threadId":"49333","inReplyTo":"xmqqva79ffpv.fsf@gitster-ct.c.googlers.com","subject":"RE: [Question] Signature calculation ignoring parts of binary files","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2018-09-13T17:55:13Z","receivedAt":"2018-09-13T17:55:28Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On September 13, 2018 1:52 PM, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > \"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n> >\n> >> The scenario is slightly different.\n> >> 1. Person A gives me a new binary file-1 with fingerprint A1. This\n> >> goes into git unchanged.\n> >> 2. Person B gives me binary file-2 with fingerprint B2. This does not\n> >> go into git yet.\n> >> 3. We attempt a git diff between the committed file-1 and uncommitted\n> >> file-2 using a textconv implementation that strips what we don't need\nto\n> compare.\n> >> 4. If file-1 and file-2 have no difference when textconv is used,\n> >> file-2 is not added and not committed. It is discarded with impunity,\n> >> never to be seen again, although we might whine a lot at the user for\n> >> attempting to put\n> >> file-2 in - but that's not git's issue.\n> >\n> > You are forgetting that Git is a distributed version control system,\n> > aren't you?  Person A and B can introduce their \"moral equivalent but\n> > bytewise different\" copies to their repository under the same object\n> > name, and you can pull from them--what happens?\n> >\n> > It is fundamental that one object name given to Git identifies one\n> > specific byte sequence contained in an object uniquely.  Once you\n> > broke that, you no longer have Git.\n> \n> Having said all that, if you want to keep the original with frills but\nsomehow\n> give these bytewise different things that reduce to the same essence (e.g.\n> when passed thru a filter like textconv), I suspect a better approach\nmight be\n> to store both the \"original\" and the result of passing the \"original\"\nthrough\n> the filter in the object database.  In the above example, you'll get two\n> \"original\"\n> objects from person A and person B, plus one \"canonical\" object that are\n> bytewise different from either of these two originals, but what they\nreduce\n> to when you use the filter on them.  Then you record the fact that to\nderive\n> the \"essence\" object, you can reduce either person A's or person B's\n> \"original\" through the filter, perhaps by using \"git notes\" attached to\nthe\n> \"essence\" object, recording the object names of these originals (the\nreason\n> why using notes in this direction is because you can mechanically\ndetermine\n> which \"essence\"\n> object any given \"original\" object reduces to---it is just the matter of\npassing\n> it through the filter.  But there can be more than one \"original\" that\nreduces\n> to the same \"essence\").\n\nI like that idea. It turns the reduced object into a contract. Thanks.\n\n"}]}