Re: proposal for extending smudge/clean filters with raw file access

From: Junio C Hamano <gitster@pobox.com>
Date: May 12, 2016, 20:14 UTC
Message-ID: <xmqq7fezdoe7.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <20160512182432.GA27427@kitenet.net>

Joey Hess <id@joeyh.name> writes:

> The clean filter has to consume the whole file content on stdin;
> not reading it all will make git think the clean filter failed.
> But, git-annex often doesn't need to read the whole content of a
> work-tree file in order to clean it.

This side, I do not think we even need a new variant. We can just update the code to interact with "clean" so that it the writer to the pipe ignores SIGPIPE, detects EPIPE on write(2), says "ah, the other end does not need the full input to produce its output". The reader from the pipe consumes its output without failing AS LONG AS the "clean" filter exits with zero (we do check its exit status, right?)

And I think we should do the same for any codepath that spawns custom script and feeds it via a pipe from us (I am talking about hooks here).

What may require a new variant is when your clean filter may not even need earlier contents of the file, in which case we are better off not opening the file ourselves and slurping it into core, only to pipe it and earlier part discarded by the filter. "clean-from-fs" filter that gets a path on the working tree and feeds us via pipe would be appropriate to deal with such a requirement.

> The smudge filter has to output the whole file content to stdout.

We cannot do a similar "we can just unconditionally update" like I said on the "clean" side to "smudge", so it would need a new variant. I do not want to call it anything "raw", as there is nothing "raw" about it. "smudge-to-fs" would be a better name.