git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 09/11] streaming: read non-delta incrementally from a pack

From
Shawn Pearce <spearce@spearce.org>
Date
May 16, 2011, 00:58 UTC
Message-ID
<BANLkTin7DALU8sfFxzJo62DGtNE2fsZxXw@mail.gmail.com>
In-Reply-To
<1305505831-31587-10-git-send-email-gitster@pobox.com>
On Sun, May 15, 2011 at 17:30, Junio C Hamano <gitster@pobox.com> wrote:
> +static read_method_decl(pack_non_delta)

I am not a huge fan of these decl macros... but I can see how writing out the same function prototype 3 times is annoying.

> +       switch (in_pack_type) {
> +       default:
> +               return -1; /* we do not do deltas for now */
Haha. Deltas are going to be painful. Very, very painful.

We actually try to stream deltas in JGit. Our implementation isn't useful. It only works if the base object is only accessed in sequential order by the delta instruction sequences.

I had plans to add an external delta base cache to $GIT_DIR/objects/delta-base-cache using a block file format that is random accessible, but has CRC-32 checksums on each block to still ensure there isn't silent data corruption on reads. You can read more about it here http://egit.eclipse.org/r/1724 but the patch is probably stalled and will get abandoned.

I think the better strategy is to avoid delta compression altogether for objects that are so big we cannot materialize them as a contiguous buffer. What a reasonable limit is, I don't know... but its probably got to be around 25-50 MB. The Android project (as an example) has 6+ MB XML documents in their source code repository that are very delta compressible.

-- 
Shawn.
Previous: Junio C HamanoNext: Junio C Hamano
Message 21 of 49 in “writing out a huge blob to working tree”
  1. 00/11 writing out a huge blob to working treeJunio C Hamano, May 16, 2011
  2. 01/11 packed_object_info_detail(): do not return a stringJunio C Hamano, May 16, 2011
  3. Thiago FarinaMay 17, 2011
  4. Junio C HamanoMay 17, 2011
  5. 02/11 sha1_object_info_extended(): expose a bit more infoJunio C Hamano, May 16, 2011
  6. 03/11 sha1_object_info_extended(): hint about objects in delta-base cacheJunio C Hamano, May 16, 2011
  7. Shawn PearceMay 16, 2011
  8. 04/11 unpack_object_header(): make it publicJunio C Hamano, May 16, 2011
  9. 05/11 write_entry(): separate two helper functions outJunio C Hamano, May 16, 2011
  10. 06/11 streaming: a new API to read from the object storeJunio C Hamano, May 16, 2011
  11. Jeff KingMay 18, 2011
  12. Junio C HamanoMay 19, 2011
  13. 07/11 streaming_write_entry(): use streaming API in write_entry()Junio C Hamano, May 16, 2011
  14. 08/11 streaming_write_entry(): support files with holesJunio C Hamano, May 16, 2011
  15. Nguyen Thai Ngoc DuyMay 16, 2011
  16. Junio C HamanoMay 16, 2011
  17. Nguyen Thai Ngoc DuyMay 17, 2011
  18. Junio C HamanoMay 17, 2011
  19. Thiago FarinaMay 16, 2011
  20. 09/11 streaming: read non-delta incrementally from a packJunio C Hamano, May 16, 2011
  21. Shawn PearceMay 16, 2011
  22. Junio C HamanoMay 16, 2011
  23. 10/11 sha1_file.c: expose helpers to read loose objectsJunio C Hamano, May 16, 2011
  24. 11/11 streaming: read loose objects incrementallyJunio C Hamano, May 16, 2011
  25. Shawn PearceMay 16, 2011
  26. Jeff KingMay 18, 2011
  27. Jeff KingMay 18, 2011
  28. Jeff KingMay 18, 2011
  29. Junio C HamanoMay 18, 2011
  30. Jeff KingMay 18, 2011
  31. 00/11 writing out a huge blob to working treeJunio C Hamano, May 19, 2011
  32. 01/11 packed_object_info_detail(): do not return a stringJunio C Hamano, May 19, 2011
  33. 02/11 sha1_object_info_extended(): expose a bit more infoJunio C Hamano, May 19, 2011
  34. 03/11 sha1_object_info_extended(): hint about objects in delta-base cacheJunio C Hamano, May 19, 2011
  35. René ScharfeMay 20, 2011
  36. Junio C HamanoMay 21, 2011
  37. 04/11 unpack_object_header(): make it publicJunio C Hamano, May 19, 2011
  38. 05/11 write_entry(): separate two helper functions outJunio C Hamano, May 19, 2011
  39. 06/11 streaming: a new API to read from the object storeJunio C Hamano, May 19, 2011
  40. René ScharfeMay 20, 2011
  41. Junio C HamanoMay 21, 2011
  42. 07/11 streaming_write_entry(): use streaming API in write_entry()Junio C Hamano, May 19, 2011
  43. Junio C HamanoMay 20, 2011
  44. 08/11 streaming_write_entry(): support files with holesJunio C Hamano, May 19, 2011
  45. 09/11 streaming: read non-delta incrementally from a packJunio C Hamano, May 19, 2011
  46. 10/11 sha1_file.c: expose helpers to read loose objectsJunio C Hamano, May 19, 2011
  47. 11/11 streaming: read loose objects incrementallyJunio C Hamano, May 19, 2011
  48. 02/11 interdiffJunio C Hamano, May 19, 2011
  49. Jeff KingMay 19, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.