git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] {fetch,receive}-pack: drop unpack-objects, delay loosing objects until the end

From
Duy Nguyen <pclouds@gmail.com>
Date
Sep 3, 2013, 11:56 UTC
Message-ID
<CACsJy8ButV8Y6Rm=F8dLTw3NvRCJZ1RgJ5f-591ASbOqfDRRTQ@mail.gmail.com>
In-Reply-To
<20130903064938.GB3608@sigill.intra.peff.net>
On Tue, Sep 3, 2013 at 1:49 PM, Jeff King <peff@peff.net> wrote:
Show 14 quoted lines
>>  - by going through index-pack first, then unpack, we pay extra cost
>>    for completing a thin pack into a full one. But compared to fetch's
>>    total time, it should not be noticeable because unpack-objects is
>>    only called when the pack contains a small number of objects.
>
> ...but the cost is paid by total pack size, not number of objects. So if
> I am pushing up a commit with a large uncompressible blob, I've
> effectively doubled my disk I/O. It would make more sense to me for
> index-pack to learn command-line options specifying the limits, and then
> to operate on the pack as it streams in. E.g., to decide after seeing
> the header to unpack rather than index, or to drop large blobs from the
> pack (and put them in their own pack directly) as we are streaming into
> it (we do not know the blob size ahead of time, but we can make a good
> guess if it has a large on-disk size in the pack).

Yeah letting index-pack do the work was my backup plan :) I think if there is a big blob in the pack, then the pack should not be unpacked at all. If you store big blobs in a separate pack you already pay the the lookup cost of one more pack in find_pack_entry(), why go through the process of unpacking? index-pack still has the advantage of streaming though. Will rework.

-- 
Duy
Previous: Jeff KingNext: Jeff King
Message 4 of 12 in “{fetch,receive}-pack: drop unpack-objects, delay loosing objects until the end”
  1. {fetch,receive}-pack: drop unpack-objects, delay loosing objects until the endNguyễn Thái Ngọc Duy, Sep 2, 2013
  2. Eric SunshineSep 2, 2013
  3. Jeff KingSep 3, 2013
  4. Duy NguyenSep 3, 2013
  5. Jeff KingSep 3, 2013
  6. 1/3 index-pack: add --unpack-limit to unpack objectsNguyễn Thái Ngọc Duy, Sep 6, 2013
  7. 2/3 fetch-pack: use index-pack --unpack-limit instead of unpack-objectsNguyễn Thái Ngọc Duy, Sep 6, 2013
  8. Jeff KingSep 8, 2013
  9. 3/3 receive-pack: use index-pack --unpack-limit instead of unpack-objectsNguyễn Thái Ngọc Duy, Sep 6, 2013
  10. Jeff KingSep 8, 2013
  11. Duy NguyenSep 8, 2013
  12. Jeff KingSep 8, 2013

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.