{"thread":{"id":"55884","subject":"[RFE] Teach git textconv to support %f","startedAt":"2021-06-08T15:38:42Z","lastAt":"2021-06-09T14:17:09Z","messageCount":4,"participants":["Randall S. Becker","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426783","messageId":"01f901d75c7c$5a8bcb10$0fa36130$@nexbridge.com","threadId":"55884","inReplyTo":null,"subject":"[RFE] Teach git textconv to support %f","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2021-06-08T15:38:33Z","receivedAt":"2021-06-08T15:38:42Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"The filter structure provides a mechanism for providing the working directory's file name path to a filter using a %f argument. This\nrequest is to teach the textconv mechanism to support the same capability.\n\nThe use case comes from a complex content renderer that needs to know what the original file name is, so as to be able to find\nadditional content, by name, that describes the file (base name+different extension).\n\nIf this is considered a good idea, I would be happy to implement this but need a pointer or two of where to look in the code to make\nit happen.\n\nThanks,\nRandall\n\n\n-- Brief whoami:\n NonStop developer since approximately 211288444200000000\n UNIX developer since approximately 421664400\n-- In my real life, I talk too much.\n\n\n"},{"id":"426832","messageId":"xmqq8s3jna0w.fsf@gitster.g","threadId":"55884","inReplyTo":"01f901d75c7c$5a8bcb10$0fa36130$@nexbridge.com","subject":"Re: [RFE] Teach git textconv to support %f","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-06-09T00:42:39Z","receivedAt":"2021-06-09T00:42:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Randall S. Becker\" writes:\n\n> The filter structure provides a mechanism for providing the\n> working directory's file name path to a filter using a %f\n> argument. This request is to teach the textconv mechanism to\n> support the same capability.\n>\n> The use case comes from a complex content renderer that needs to\n> know what the original file name is, so as to be able to find\n> additional content, by name, that describes the file (base\n> name+different extension).\n>\n> If this is considered a good idea, I would be happy to implement\n> this but need a pointer or two of where to look in the code to\n> make it happen.\n\nBoth in diff, grep and cat-file, textconv eventually triggers\ndiff.c::fill_textconv() and calls run_textconv() unless there is a\ncached copy of the resut of running textconv earlier on the same\ncontents. This is because for each textconv driver, the output is\nexpected to be purely a function of the input bytestream, and that\nis why it does not take any other input.\n\nSo, if we have two identical blobs in a tree object under different\npathnames, making the output from textconv different for them\nbecause they sit at different pathnames directly goes against the\nbasic design of the system at the philosophical level.\n\nHaving said that, I _suspect_ (but not verified) that as long as the\ndriver is marked as non-cacheable, it may be acceptable to export a\nnew environment variable, say, GIT_TEXTCONV_PATH, and allow the\ntextconv program to produce different results for the same input. I\nam not offhand sure if it is OK to allow command line substitutions\nlike the filter scripts, though. It would be nice from the point of\nview of consistency if we could do so, but those who use an existing\ntextconv program at a pathname with per-cent in it may get negatively\naffected.\n\nThanks.\n"},{"id":"426840","messageId":"YMA/h9Hpb8z+D3W3@coredump.intra.peff.net","threadId":"55884","inReplyTo":"xmqq8s3jna0w.fsf@gitster.g","subject":"Re: [RFE] Teach git textconv to support %f","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-09T04:11:51Z","receivedAt":"2021-06-09T04:11:55Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 09, 2021 at 09:42:39AM +0900, Junio C Hamano wrote:\n\n> \"Randall S. Becker\" writes:\n> \n> > The filter structure provides a mechanism for providing the\n> > working directory's file name path to a filter using a %f\n> > argument. This request is to teach the textconv mechanism to\n> > support the same capability.\n> >\n> > The use case comes from a complex content renderer that needs to\n> > know what the original file name is, so as to be able to find\n> > additional content, by name, that describes the file (base\n> > name+different extension).\n> >\n> > If this is considered a good idea, I would be happy to implement\n> > this but need a pointer or two of where to look in the code to\n> > make it happen.\n> \n> Both in diff, grep and cat-file, textconv eventually triggers\n> diff.c::fill_textconv() and calls run_textconv() unless there is a\n> cached copy of the resut of running textconv earlier on the same\n> contents. This is because for each textconv driver, the output is\n> expected to be purely a function of the input bytestream, and that\n> is why it does not take any other input.\n> \n> So, if we have two identical blobs in a tree object under different\n> pathnames, making the output from textconv different for them\n> because they sit at different pathnames directly goes against the\n> basic design of the system at the philosophical level.\n> \n> Having said that, I _suspect_ (but not verified) that as long as the\n> driver is marked as non-cacheable, it may be acceptable to export a\n> new environment variable, say, GIT_TEXTCONV_PATH, and allow the\n> textconv program to produce different results for the same input. I\n> am not offhand sure if it is OK to allow command line substitutions\n> like the filter scripts, though. It would be nice from the point of\n> view of consistency if we could do so, but those who use an existing\n> textconv program at a pathname with per-cent in it may get negatively\n> affected.\n\nAs the person who implemented both textconv and its caching, all of that\nsounds quite sensible to me. :)\n\nThe caching mechanism has never been turned on by default, and probably\nwill never be (because one of the original uses for which I wanted\ntextconv is decrypting blobs on the fly, and storing the result would\ndefeat the purpose). So it may be sufficient to say \"if it hurts to turn\non caching with a filter that depends on the path, then don't do it\".\n\nIt may be possible to make it work with a cache, too. The caching\nmechanism also has a validity key, which looks something like this:\n\n [prime the cache with a silly filter]\n $ git -c diff.perl.textconv='tr a-z A-Z <' \\\n -c diff.perl.cachetextconv=true \\\n log -p '*.perl' >/dev/null\n\n [the result is stored via git-notes; the body is the cache-key]\n $ git cat-file commit refs/notes/textconv/perl\n tree a27c3c18d222e47a7943c2844f9ed6c75710d8c4\n author Jeff King 1623211550 -0400\n committer Jeff King 1623211550 -0400\n \n tr a-z A-Z <\n\nI thought at first we could put the pathname into that key, but it's\nreally a key for the _whole_ cache, not an individual entry. So that\nwouldn't work. You'd have to redesign the cache mechanism (which in turn\nmight require assistance from the notes code). So I lean towards \"if it\nhurts, don't do it\". :)\n\nOne final note. Randall said:\n\n> > The use case comes from a complex content renderer that needs to\n> > know what the original file name is, so as to be able to find\n> > additional content, by name, that describes the file (base\n> > name+different extension).\n\nThat needs the filename, but there's also the implication there that\nancillary data would be coming from _other_ files based on that name.\nThat seems a lot more complicated. If I'm a filter and am asked to\nconvert \"foo.one\", I might want to see \"foo.two\". But how do I know from\nwhich commit to get it? It is OK if \"foo.two\" is a globally unambiguous\nname across time, but I'd imagine in most cases you want \"foo.two\" that\naccompanied \"foo.one\" at the time. And now the cache is not even a\nproperty of a blob/filename pair, but rather of the whole tree.\n\n-Peff\n"},{"id":"426889","messageId":"002401d75d3a$19a837a0$4cf8a6e0$@nexbridge.com","threadId":"55884","inReplyTo":"YMA/h9Hpb8z+D3W3@coredump.intra.peff.net","subject":"RE: [RFE] Teach git textconv to support %f","fromName":"Randall S. Becker","fromEmail":"rsbecker@nexbridge.com","sentAt":"2021-06-09T14:16:49Z","receivedAt":"2021-06-09T14:17:09Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On June 9, 2021 12:12 AM, Peff wrote:\n>On Wed, Jun 09, 2021 at 09:42:39AM +0900, Junio C Hamano wrote:\n>\n>> \"Randall S. Becker\" writes:\n>>\n>> > The filter structure provides a mechanism for providing the working\n>> > directory's file name path to a filter using a %f argument. This\n>> > request is to teach the textconv mechanism to support the same\n>> > capability.\n>> >\n>> > The use case comes from a complex content renderer that needs to\n>> > know what the original file name is, so as to be able to find\n>> > additional content, by name, that describes the file (base\n>> > name+different extension).\n>> >\n>> > If this is considered a good idea, I would be happy to implement\n>> > this but need a pointer or two of where to look in the code to make\n>> > it happen.\n>>\n>> Both in diff, grep and cat-file, textconv eventually triggers\n>> diff.c::fill_textconv() and calls run_textconv() unless there is a\n>> cached copy of the resut of running textconv earlier on the same\n>> contents. This is because for each textconv driver, the output is\n>> expected to be purely a function of the input bytestream, and that is\n>> why it does not take any other input.\n>>\n>> So, if we have two identical blobs in a tree object under different\n>> pathnames, making the output from textconv different for them because\n>> they sit at different pathnames directly goes against the basic design\n>> of the system at the philosophical level.\n>>\n>> Having said that, I _suspect_ (but not verified) that as long as the\n>> driver is marked as non-cacheable, it may be acceptable to export a\n>> new environment variable, say, GIT_TEXTCONV_PATH, and allow the\n>> textconv program to produce different results for the same input. I\n>> am not offhand sure if it is OK to allow command line substitutions\n>> like the filter scripts, though. It would be nice from the point of\n>> view of consistency if we could do so, but those who use an existing\n>> textconv program at a pathname with per-cent in it may get negatively\n>> affected.\n>\n>As the person who implemented both textconv and its caching, all of that sounds quite sensible to me. :)\n>\n>The caching mechanism has never been turned on by default, and probably will never be (because one of the original uses for which I\n>wanted textconv is decrypting blobs on the fly, and storing the result would defeat the purpose). So it may be sufficient to say \"if it hurts to\n>turn on caching with a filter that depends on the path, then don't do it\".\n>\n>It may be possible to make it work with a cache, too. The caching mechanism also has a validity key, which looks something like this:\n>\n> [prime the cache with a silly filter]\n> $ git -c diff.perl.textconv='tr a-z A-Z <' \\\n> -c diff.perl.cachetextconv=true \\\n> log -p '*.perl' >/dev/null\n>\n> [the result is stored via git-notes; the body is the cache-key]\n> $ git cat-file commit refs/notes/textconv/perl\n> tree a27c3c18d222e47a7943c2844f9ed6c75710d8c4\n> author Jeff King 1623211550 -0400\n> committer Jeff King 1623211550 -0400\n>\n> tr a-z A-Z <\n>\n>I thought at first we could put the pathname into that key, but it's really a key for the _whole_ cache, not an individual entry. So that\n>wouldn't work. You'd have to redesign the cache mechanism (which in turn might require assistance from the notes code). So I lean\n>towards \"if it hurts, don't do it\". :)\n>\n>One final note. Randall said:\n>\n>> > The use case comes from a complex content renderer that needs to\n>> > know what the original file name is, so as to be able to find\n>> > additional content, by name, that describes the file (base\n>> > name+different extension).\n>\n>That needs the filename, but there's also the implication there that ancillary data would be coming from _other_ files based on that\n>name.\n>That seems a lot more complicated. If I'm a filter and am asked to convert \"foo.one\", I might want to see \"foo.two\". But how do I know\n>from which commit to get it? It is OK if \"foo.two\" is a globally unambiguous name across time, but I'd imagine in most cases you want\n>\"foo.two\" that accompanied \"foo.one\" at the time. And now the cache is not even a property of a blob/filename pair, but rather of the\n>whole tree.\n\nWell, sadly, I see the points you both make. It might be better for my team to come up with a self-describing file format with embedded metadata instead. Thing is, the whole approach of idempotency in git where the commit would immutably contain both the file and its metadata in a separate blob but same commit is appealing - which is the only condition where I would see my original suggestion applying.\n\n-Randall\n\n"}]}