Re: [PATCH] convert filter: supply path to external driver

From: Jeff King <peff@peff.net>
Date: Dec 20, 2010, 02:24 UTC
Message-ID: <20101220022409.GA4625@sigill.intra.peff.net>
In-Reply-To: <7vzks1e84p.fsf@alter.siamese.dyndns.org>

On Sun, Dec 19, 2010 at 01:59:50PM -0800, Junio C Hamano wrote:

> Modulo one worry.  Don't we have, or don't we at least plant to allow us
> to have, a facility to cache expensive blob conversion result, similar to
> the textconv caching?  How would this change interact with two blobs that
> live in different paths?

Yeah, it has been talked about, but I don't think anyone is working on it (I don't personally use clean/smudge at all, so it is not something I have thought _that_ much about).

This does definitely complicate matters, as the filtering is no longer a pure mapping of sha1->sha1. However, I think in practice we could do just fine by using a multi-level lookup. I.e., mapping a sha1 to be filtered into a tree. Each tree entry would represent the remaining cache parameters. In this case, the only other parameter we have is the path given to the filter (but it could easily be extended to include other parameters, if they existed, in this or other caching cases).

We only get a high-performance lookup for the first part of the multi-level (i.e., the sha1), but that's OK if we assume the number of second-level items is going to be small. Which I think is the case here (a sha1 will tend to be found only under one or a few names).

An alternative would be to combine all parts of the filter under a single lookup key. E.g., calculate and store under sha1(sha1(blob) + filename)). But that means the notes keys are not actual object sha1s, which throws off pruning.

Anyway, that's just my quick thinking on the subject. I don't see any reason to restrict a feature just because we might want to cache it in the future. At the very worst, we could always cache filters which do not use %s, and make only %s users pay the penalty.

-Peff