Re: [PATCH 00/11] writing out a huge blob to working tree

From: Shawn Pearce <spearce@spearce.org>
Date: May 16, 2011, 00:47 UTC
Message-ID: <BANLkTinCRbDa1ic05SKzSOadicOKfwV04Q@mail.gmail.com>
In-Reply-To: <1305505831-31587-1-git-send-email-gitster@pobox.com>

On Sun, May 15, 2011 at 17:30, Junio C Hamano <gitster@pobox.com> wrote:

> Traditionally, git always read the full contents of an object in memory
> before performing various operations on it, e.g. comparing for diff,
> writing it to the working tree, etc.  A huge blob that you cannot fit
> in memory was very cumbersome to handle.

,,,

Show 5 quoted lines

> Interested parties may want to measure the performance impact of the last
> three patches. The series deliberately ignores core.bigfileThreashold and
> let small and large blobs alike go through the streaming_write_entry()
> codepath, but it _might_ turn out that we would want to use the new code
> only for large-ish blobs.

FWIW in JGit we control this by looking at the object size and comparing to the variable core.streamFileThreshold. For any object below this size we allocate the buffer, unpack into it, and return the buffer to the caller. Only objects above the size use the streaming code paths.

There is a performance difference, at least for us in Java. Most of the overhead seems to be due to running zlib inflate() with a tiny buffer size rather than the full destination buffer. This probably has to do with the cost associated with jumping from the Java bytecode through JNI to the libz library.

-- 
Shawn.