Re: [RFC] Native access to Git LFS cache

From: Lars Schneider <larsxschneider@gmail.com>
Date: Jun 28, 2016, 13:22 UTC
Message-ID: <9603B6F9-3BEF-4779-84C3-6DC61D7FBC46@gmail.com>
In-Reply-To: <xmqqmvm6bom5.fsf@gitster.mtv.corp.google.com>

Show 10 quoted lines

> On 27 Jun 2016, at 18:09, Junio C Hamano <gitster@pobox.com> wrote:
> 
> larsxschneider@gmail.com writes:
> 
>> Unfortunately that fix helps only with cloning. Any local Git operation
>> that invokes the clean/smudge filter (e.g. switching branches) is still
>> slow.
> 
> Do you know where the slowness comes from?  Does Joey's new
> clean/smudge interface help GitLFS?

I am pretty sure the startup time of the external clean/smudge process causes the slowness and consequently I don't think Joey's patch would help. The following tests makes me believe that:

I ran the same test as in my original email using the repo with 15,000 LFS files. Instead of the LFS binary I use the fast and simple shell built-in `true` command:

$ git -c filter.lfs.smudge=true -c filter.lfs.clean=true clone https://github.com/larsxschneider/lfstest-manyfiles.git $ cd lfstest-manyfiles/ $ time git -c filter.lfs.smudge=true -c filter.lfs.clean=true checkout removed-files

real 0m47.030s user 0m29.521s sys 0m16.993s

It still takes 47 seconds to switch the branch. Does this test prove my point or do you see a flaw in the test?

Show 13 quoted lines

> You are not likely to get anything that knows that a blob object may
> be named as anything other than SHA-1("blob <len>" + <contents>) to
> Git core.  The remote-object-store idea that was floated by Peff and
> Christian started running with at least maintains that object naming
> property and has a better chance of interacting better with the core,
> but LFS, Annex or anything that would not preserve the object naming
> would not.
> 
> Personally, I view a surrogate blob left by LFS in the tree object
> and filtered via clean/smudge a "smarter" kind of symbolic link that
> points outside what Git controls.  The area outside what Git
> controls is left to be managed by whatever the add-on does; Git
> shouldn't even be aware of how they are structured and/or managed.

I understand and somewhat anticipated your point of view. I will try to find a less intrusive solution.

@Christian/Peff: Is there a place to look for more info about your remote-object-store idea?

Thanks, Lars