Re: [RFC] Native access to Git LFS cache

From: Lars Schneider <larsxschneider@gmail.com>
Date: Jun 28, 2016, 13:43 UTC
Message-ID: <5491DA7A-4B1E-4015-879F-F01FF7481143@gmail.com>
In-Reply-To: <alpine.DEB.2.20.1606281513380.12947@virtualbox>

Show 38 quoted lines

> On 28 Jun 2016, at 15:14, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> 
> Hi Duy,
> 
> On Tue, 28 Jun 2016, Duy Nguyen wrote:
> 
>> On Tue, Jun 28, 2016 at 11:40 AM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>>> 
>>> On Mon, 27 Jun 2016, Duy Nguyen wrote:
>>> 
>>>> On Mon, Jun 27, 2016 at 7:38 AM,  <larsxschneider@gmail.com> wrote:
>>>>> ## Proposed solution
>>>>> Git LFS caches its objects under .git/lfs/objects. Most of the time
>>>>> Git LFS objects are already available in the cache (e.g. if you
>>>>> switch branches back and forth). I implemented these "cache hits"
>>>>> natively in Git.  Please note that this implementation is just a
>>>>> quick and dirty proof of concept. If the Git community agrees that
>>>>> this kind of approach would be acceptable then I will start to work
>>>>> on a proper patch series with cross platform support and unit
>>>>> tests.
>>>> 
>>>> Would it be possible to move all this code to a separate daemon?
>>>> Instead of spawning a new process to do the filtering, you send a
>>>> command "convert this" over maybe unix socket and either receive the
>>>> whole result over the socket, or receive a path of the result.
>>> 
>>> Unix sockets are not really portable...
>> 
>> It's the same situation as index-helper. I expect you guys will
>> replace the transport with named pipe or similar.
> 
> Yes, I will have to work on that. But I might need to ask for a change in
> the design if I hit some obstacle there: named pipes are not the same at
> all as Unix sockets.
> 
> Read: it will be painful, and not a general solution. So every new Unix
> socket that you introduce will introduce new problems for me.

Thanks Duy for your suggestion. I considered a daemon, but a daemon makes it always harder for the user as the user needs to ensure the daemon is running! Plus, Dscho's concerns regarding Windows.

I think the core problem is that we invoke the filter for every file: https://github.com/git/git/blob/master/convert.c#L461-L475

Couldn't we start the filter executable at the beginning of the Git process and communicate with it via stdin/stdout whenever we hit the Git filter code? Would that work?

Alternatively, do you see a way to add a "plugin" system to Git? Where Git could be configured to dynamically load a "filter" library?

@Dscho: Do you have a recommendation for interprocess communication that works without trouble on Windows?

Thanks, Lars