git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v0 3/3] Bigfile: teach "git add" to send a large file straight to a pack

From
Shawn Pearce <spearce@spearce.org>
Date
May 9, 2011, 14:05 UTC
Message-ID
<BANLkTimo_D_s-=zjPAbStvETRh8d_Tpa_w@mail.gmail.com>
In-Reply-To
<1304844455-23570-4-git-send-email-gitster@pobox.com>
On Sun, May 8, 2011 at 01:47, Junio C Hamano <gitster@pobox.com> wrote:
Show 12 quoted lines
> + * NEEDSWORK: This creates one packfile per large blob, because the
> + * caller immediately wants the result sha1, and fast-import can
> + * report the object name via marks mechanism only by closing the
> + * created packfile. We should instead add an internal "stuff objects
> + * into a single pack, all in non-delta representation, keeping track
> + * of only <object-name, offset> tuples in core" API, that keeps one
> + * append-only packfile open at a time.  Have the first call to this
> + * function open a packfile on demand, and make sure the caller calls
> + * another function in the API to close the packfile at end, at which
> + * point the in-core tuples of <object-name, offset> should be written
> + * out as a corresponding pack .idx file and the tentative .pack file
> + * renamed to the final name.

The other problem here is the caller cannot access the written objects until the pack is closed. That is one of the things that has made fast-import difficult for git-svn to use, because git-svn expects the object to be available immediately. I assume that within a single git add or git update-index process we don't need to worry about this, so its probably a non-issue.

-- 
Shawn.
Previous: Junio C HamanoNext: Junio C Hamano
Message 14 of 18 in “git add a-Big-file”
  1. 0/3 git add a-Big-fileJunio C Hamano, May 8, 2011
  2. 1/3 index_fd(): turn write_object and format_check arguments into one flagJunio C Hamano, May 8, 2011
  3. 2/3 index_fd(): split into two helper functionsJunio C Hamano, May 8, 2011
  4. 3/3 Bigfile: teach "git add" to send a large file straight to a packJunio C Hamano, May 8, 2011
  5. Nguyen Thai Ngoc DuyMay 8, 2011
  6. Junio C HamanoMay 8, 2011
  7. 0/4 convert.c clean-upJunio C Hamano, May 9, 2011
  8. 1/4 convert: rename the "eol" global variable to "core_eol"Junio C Hamano, May 9, 2011
  9. 2/4 convert: give saner names to crlf/eol variables, types and functionsJunio C Hamano, May 9, 2011
  10. 3/4 convert: make it safer to add conversion attributesJunio C Hamano, May 9, 2011
  11. 4/4 convert: make it harder to screw up adding a conversion attributeJunio C Hamano, May 9, 2011
  12. Nguyen Thai Ngoc DuyMay 10, 2011
  13. Junio C HamanoMay 10, 2011
  14. Shawn PearceMay 9, 2011
  15. Junio C HamanoMay 9, 2011
  16. Junio C HamanoMay 9, 2011
  17. Ævar Arnfjörð BjarmasonMay 29, 2011
  18. Junio C HamanoMay 29, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.