git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/2] index-pack: use streaming interface on large blobs (most of the time)

From
Junio C Hamano <gitster@pobox.com>
Date
May 18, 2012, 22:20 UTC
Message-ID
<xmqqzk95168w.fsf@junio.mtv.corp.google.com>
In-Reply-To
<1337172630-23679-2-git-send-email-pclouds@gmail.com>
Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
Show 11 quoted lines
> unpack_raw_entry() will not allocate and return decompressed blobs if
> they are larger than core.bigFileThreshold.
>
> The blob content is needed by sha1_object() in some cases. When we do
> need the blob content, we put it back in core with
> get_data_from_pack(). However we rarely need that in pratice.
>
> The first case is when we find an in-repo blob with the same SHA-1. We
> need to do collision test, byte-on-byte. Normally (e.g. in
> fetch/pull/clone) this does not happen because git avoid to send
> objects that client already has.

Perhaps the codepath that performs the byte-for-byte comparison can be taught to stream from the received pack data and whatever was already in the repository, using the streaming interface? That way you do not have to hold all of the both objects at the same time in core, no?

Previous: Nguyễn Thái Ngọc DuyNext: Nguyen Thai Ngoc Duy
Message 3 of 16 in “index-pack: hash non-delta objects while reading from stream”
  1. 1/2 index-pack: hash non-delta objects while reading from streamNguyễn Thái Ngọc Duy, May 16, 2012
  2. 2/2 index-pack: use streaming interface on large blobs (most of the time)Nguyễn Thái Ngọc Duy, May 16, 2012
  3. Junio C HamanoMay 18, 2012
  4. Nguyen Thai Ngoc DuyMay 19, 2012
  5. Junio C HamanoMay 18, 2012
  6. 1/6 index-pack: hash non-delta objects while reading from streamNguyễn Thái Ngọc Duy, May 23, 2012
  7. 2/6 index-pack: use streaming interface on large blobs (most of the time)Nguyễn Thái Ngọc Duy, May 23, 2012
  8. 3/6 index-pack: factor out unpack core from get_data_from_packNguyễn Thái Ngọc Duy, May 23, 2012
  9. 4/6 index-pack: use streaming interface for collision test on large blobsNguyễn Thái Ngọc Duy, May 23, 2012
  10. Junio C HamanoMay 23, 2012
  11. 4/6 index-pack: use streaming interface for collision test on large blobsNguyễn Thái Ngọc Duy, May 24, 2012
  12. 5/6 index-pack: avoid collision test when verifying in-repo packNguyễn Thái Ngọc Duy, May 23, 2012
  13. 6/6 sha1_loose_object_info: do not complain out loud on non-existent objectsNguyễn Thái Ngọc Duy, May 23, 2012
  14. Nguyen Thai Ngoc DuyMay 23, 2012
  15. Junio C HamanoMay 23, 2012
  16. Nguyen Thai Ngoc DuyMay 24, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.