git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] index-pack: use streaming interface on large blobs (most of the time)

From
Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date
May 19, 2012, 05:31 UTC
Message-ID
<CACsJy8CypvcS3sTn1isREYzApN273PcKnMJBpWBrOaO2ZaSLSw@mail.gmail.com>
In-Reply-To
<xmqqzk95168w.fsf@junio.mtv.corp.google.com>
On Sat, May 19, 2012 at 5:20 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 18 quoted lines
> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
>
>> unpack_raw_entry() will not allocate and return decompressed blobs if
>> they are larger than core.bigFileThreshold.
>>
>> The blob content is needed by sha1_object() in some cases. When we do
>> need the blob content, we put it back in core with
>> get_data_from_pack(). However we rarely need that in pratice.
>>
>> The first case is when we find an in-repo blob with the same SHA-1. We
>> need to do collision test, byte-on-byte. Normally (e.g. in
>> fetch/pull/clone) this does not happen because git avoid to send
>> objects that client already has.
>
> Perhaps the codepath that performs the byte-for-byte comparison can be
> taught to stream from the received pack data and whatever was already
> in the repository, using the streaming interface?  That way you do not
> have to hold all of the both objects at the same time in core, no?
Sure. But that does not happen until you are attacked. Maybe later.

verify-pack'ing also runs into this, but that's another issue and should be fixed separately (hopefully by verify-pack users).

-- 
Duy
Previous: Junio C HamanoNext: Junio C Hamano
Message 4 of 16 in “index-pack: hash non-delta objects while reading from stream”
  1. 1/2 index-pack: hash non-delta objects while reading from streamNguyễn Thái Ngọc Duy, May 16, 2012
  2. 2/2 index-pack: use streaming interface on large blobs (most of the time)Nguyễn Thái Ngọc Duy, May 16, 2012
  3. Junio C HamanoMay 18, 2012
  4. Nguyen Thai Ngoc DuyMay 19, 2012
  5. Junio C HamanoMay 18, 2012
  6. 1/6 index-pack: hash non-delta objects while reading from streamNguyễn Thái Ngọc Duy, May 23, 2012
  7. 2/6 index-pack: use streaming interface on large blobs (most of the time)Nguyễn Thái Ngọc Duy, May 23, 2012
  8. 3/6 index-pack: factor out unpack core from get_data_from_packNguyễn Thái Ngọc Duy, May 23, 2012
  9. 4/6 index-pack: use streaming interface for collision test on large blobsNguyễn Thái Ngọc Duy, May 23, 2012
  10. Junio C HamanoMay 23, 2012
  11. 4/6 index-pack: use streaming interface for collision test on large blobsNguyễn Thái Ngọc Duy, May 24, 2012
  12. 5/6 index-pack: avoid collision test when verifying in-repo packNguyễn Thái Ngọc Duy, May 23, 2012
  13. 6/6 sha1_loose_object_info: do not complain out loud on non-existent objectsNguyễn Thái Ngọc Duy, May 23, 2012
  14. Nguyen Thai Ngoc DuyMay 23, 2012
  15. Junio C HamanoMay 23, 2012
  16. Nguyen Thai Ngoc DuyMay 24, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.