From: Nguyen Thai Ngoc Duy Date: Sat, 19 May 2012 05:31:57 GMT Subject: Re: [PATCH 2/2] index-pack: use streaming interface on large blobs (most of the time) Message-ID: In-Reply-To: On Sat, May 19, 2012 at 5:20 AM, Junio C Hamano wrote: > Nguyễn Thái Ngọc Duy   writes: > >> unpack_raw_entry() will not allocate and return decompressed blobs if >> they are larger than core.bigFileThreshold. >> >> The blob content is needed by sha1_object() in some cases. When we do >> need the blob content, we put it back in core with >> get_data_from_pack(). However we rarely need that in pratice. >> >> The first case is when we find an in-repo blob with the same SHA-1. We >> need to do collision test, byte-on-byte. Normally (e.g. in >> fetch/pull/clone) this does not happen because git avoid to send >> objects that client already has. > > Perhaps the codepath that performs the byte-for-byte comparison can be > taught to stream from the received pack data and whatever was already > in the repository, using the streaming interface?  That way you do not > have to hold all of the both objects at the same time in core, no? Sure. But that does not happen until you are attacked. Maybe later. verify-pack'ing also runs into this, but that's another issue and should be fixed separately (hopefully by verify-pack users). -- Duy