{"thread":{"id":"51453","subject":"[Question] Diff text filters and git add","startedAt":"2019-07-09T21:43:14Z","lastAt":"2019-07-11T17:28:37Z","messageCount":4,"participants":["Randall S. Becker","Jeff King","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"378738","messageId":"009201d5369f$4da5d030$e8f17090$@nexbridge.com","threadId":"51453","inReplyTo":null,"subject":"[Question] Diff text filters and git add","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2019-07-09T21:43:05Z","receivedAt":"2019-07-09T21:43:14Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"Hi all,\n\nI am trying to do something a bit strange and wonder about the best way to\ngo. I have a text filter that presents content of very special binary file\nformats using textconv. What I am wondering is whether using the textconv\nmechanism is sufficient to have git calculate the file signature or whether\nI need to use an external diff engine, so that git add behaves in a stable\nmanner (i.e., does git internally use the textconv mechanism for evaluating\nwhether a file changed or whether the external diff engine is required, or\nwhether this is even possible at all).\n\nThe basic use case is that there is a timestamp embedded in the binary file\nthat I want to forever ignore when committing. I only need this done on one\nspecific machine, which is under Jenkins control, so it's not something\ndevelopers would deal with at all (so the filter config is in one place).\nWhen the binary generator runs, if the two file images are \"similar enough\"\n(as in: the same except for the generated timestamp, and a couple of other\nannoying bits of metadata), I want git to think that they are the same in an\nautomated way, so that when I am constructing commits, I do not want what\nwould be considered a duplicate of what is essentially the same file.\n\nSadly, I cannot modify the generator, so I'm stuck with the files being\nwonky. I also cannot run the generator anywhere downstream, so doing so on\nthe deployment engine is also not an option (don't ask, the generator is\nlimited on where it can be run). Suggestions are welcome, please.\n\nThanks,\nRandall\n\n\n\n"},{"id":"378739","messageId":"20190709215104.GA20936@sigill.intra.peff.net","threadId":"51453","inReplyTo":"009201d5369f$4da5d030$e8f17090$@nexbridge.com","subject":"Re: [Question] Diff text filters and git add","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-07-09T21:51:04Z","receivedAt":"2019-07-09T21:51:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jul 09, 2019 at 05:43:05PM -0400, Randall S. Becker wrote:\n\n> I am trying to do something a bit strange and wonder about the best way to\n> go. I have a text filter that presents content of very special binary file\n> formats using textconv. What I am wondering is whether using the textconv\n> mechanism is sufficient to have git calculate the file signature or whether\n> I need to use an external diff engine, so that git add behaves in a stable\n> manner (i.e., does git internally use the textconv mechanism for evaluating\n> whether a file changed or whether the external diff engine is required, or\n> whether this is even possible at all).\n\nNo, textconv only applies when generating a diff to output, and will\nnever impact what's stored in Git.\n\nIt sounds like you might want a clean filter instead, to sanitize\nthe file contents as they come into Git (and perhaps a matching smudge\nfilter to convert back to the working-tree version if necessary).\n\nYou're talking about \"the diff engine\" here, but note that git-add would\nnever do a diff at all. It cares only about full sha1s (and optimizes\nout re-computing the sha1 on each invocation by using stat data). So\noutside of clean/smudge, there's nothing else going on.\n\n-Peff\n"},{"id":"378762","messageId":"00bb01d5371d$453a8520$cfaf8f60$@nexbridge.com","threadId":"51453","inReplyTo":"20190709215104.GA20936@sigill.intra.peff.net","subject":"RE: [Question] Diff text filters and git add","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2019-07-10T12:44:47Z","receivedAt":"2019-07-10T12:45:03Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On July 9, 2019 5:51 PM, Peff wrote:\n> To: Randall S. Becker <rsbecker@nexbridge.com>\n> Cc: git@vger.kernel.org\n> Subject: Re: [Question] Diff text filters and git add\n> \n> On Tue, Jul 09, 2019 at 05:43:05PM -0400, Randall S. Becker wrote:\n> \n> > I am trying to do something a bit strange and wonder about the best\n> > way to go. I have a text filter that presents content of very special\n> > binary file formats using textconv. What I am wondering is whether\n> > using the textconv mechanism is sufficient to have git calculate the\n> > file signature or whether I need to use an external diff engine, so\n> > that git add behaves in a stable manner (i.e., does git internally use\n> > the textconv mechanism for evaluating whether a file changed or\n> > whether the external diff engine is required, or whether this is even\n> possible at all).\n> \n> No, textconv only applies when generating a diff to output, and will never\n> impact what's stored in Git.\n> \n> It sounds like you might want a clean filter instead, to sanitize the file\n> contents as they come into Git (and perhaps a matching smudge filter to\n> convert back to the working-tree version if necessary).\n> \n> You're talking about \"the diff engine\" here, but note that git-add would never\n> do a diff at all. It cares only about full sha1s (and optimizes out re-computing\n> the sha1 on each invocation by using stat data). So outside of clean/smudge,\n> there's nothing else going on.\n\nThanks. I can script this instead. Will do an external diff then --assume-unchanged when I detect an equivalence.\n\nAppreciate the advice and info,\nRandall\n\n"},{"id":"378838","messageId":"86h87sa3b6.fsf@gmail.com","threadId":"51453","inReplyTo":"00bb01d5371d$453a8520$cfaf8f60$@nexbridge.com","subject":"Re: [Question] Diff text filters and git add","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2019-07-11T17:28:29Z","receivedAt":"2019-07-11T17:28:37Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Randall S. Becker\" <rsbecker@nexbridge.com> writes:\n> On July 9, 2019 5:51 PM, Peff wrote:\n[...]\n>> No, textconv only applies when generating a diff to output, and will never\n>> impact what's stored in Git.\n>> \n>> It sounds like you might want a clean filter instead, to sanitize the file\n>> contents as they come into Git (and perhaps a matching smudge filter to\n>> convert back to the working-tree version if necessary).\n>> \n>> You're talking about \"the diff engine\" here, but note that git-add would never\n>> do a diff at all. It cares only about full sha1s (and optimizes out re-computing\n>> the sha1 on each invocation by using stat data). So outside of clean/smudge,\n>> there's nothing else going on.\n>\n> Thanks. I can script this instead. Will do an external diff then\n> --assume-unchanged when I detect an equivalence.\n\nIf you want to ignore changes, --assume-unchanged (i.e. lying to Git) is\na wrong solution, as it can lead to data loss.  It is meant as\nperformance optimization.\n\nA better solution would be to use --skip-worktree, which though meant\nfor sparse checkout can be used for ignoring changes.  The only problem\nis that it can prevent some safe operations, like git-stash, because git\nthinks that it could lead to data loss.\n\nThough I am not sure if they are needed with clean/smudge filter.\n\nBest,\n-- \nJakub Narębski\n"}]}