git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: GSoC - Some questions on the idea of "Better big-file support".

From
BCBo Chen <chen@chenirvine.org>
Date
Mar 30, 2012, 19:11 UTC
Message-ID
<CA+M5ThS1XiaGJWmSvfwXoqebnH6fK3h6cC7OnQQi=LXzcA0GRw@mail.gmail.com>
In-Reply-To
<CACsJy8APtMsMJ=FrZjOP=DbzuFoemSLJTmkjaiK5Wkq9XtA4rg@mail.gmail.com>
Sorry for replying late.
My questions are inline in the following.
On Wed, Mar 28, 2012 at 2:19 AM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
Show 18 quoted lines
> On Wed, Mar 28, 2012 at 11:38 AM, Bo Chen <chen@chenirvine.org> wrote:
>> Hi, Everyone. This is Bo Chen. I am interested in the idea of "Better
>> big-file support".
>>
>> As it is described in the idea page,
>> "Many large files (like media) do not delta very well. However, some
>> do (like VM disk images). Git could split large objects into smaller
>> chunks, similar to bup, and find deltas between these much more
>> manageable chunks. There are some preliminary patches in this
>> direction, but they are in need of review and expansion."
>>
>> Can anyone elaborate a little bit why many large files do not delta
>> very well?
>
> Large files are usually binary. Depends on the type of binary, they
> may or may not delta well. Those that are compressed/encrypted
> obviously don't delta well because one change can make the final
> result completely different.

Just make clear one of my confusions. Delta operation is to find out the differences between different versions of the same file, right? As I know, delta encoding is to re-encode a file based on the differences between neighboring blocks, thus can help compress a file since after delta encoding, we will have more similar data within the file. Can anyone elaborate a little bit what is the relation between delta operation in git and delta encoding listed above? Thanks.

>
> Another problem with delta-ing large files with git is, current code
> needs to load two files in memory for delta. Consuming 4G for delta 2
> 2GB files does not sound good.

I am wondering why we cannot divide the 2 2GB files into chunks and delta chunks by chunks. Is that any difference, except a little more IOs?

Show 10 quoted lines
>
>> Is it a general problem or a specific problem just for Git?
>> I am really new to Git, can anyone give me some hints on which source
>> codes I should read to learn more about the current code on delta
>> operation? It is said that "there are some preliminary patches in this
>> direction", where can I find these patches?
>
> Read about rsync algorithm [2]. Bup [1] implements the same (I think)
> algorithm, but on top of git. For preliminary patches, have a look at
> jc/split-blob series at commit 4a1242d in git.git.

Make clear my another confusion. The file which has been updated (added, deleted, and modified) is first delta-compressed, and then synchronize to the remote repo by some mechanism (rsync?). I am wondering what is the the relationship between delta operation and rsync.

Show 5 quoted lines
>
> [1] https://github.com/apenwarr/bup
> [2] http://en.wikipedia.org/wiki/Rsync#Algorithm
> --
> Duy
Bo
Previous: Nguyen Thai Ngoc DuyNext: Jeff King
Message 37 of 43 in “GSoC - Some questions on the idea of "Better big-file support".”
  1. Bo ChenMar 28, 2012
  2. Nguyen Thai Ngoc DuyMar 28, 2012
  3. SergioMar 28, 2012
  4. Bo ChenMar 30, 2012
  5. Bo ChenMar 30, 2012
  6. Jeff KingMar 30, 2012
  7. Bo ChenMar 30, 2012
  8. Sergio CallegariMar 31, 2012
  9. Neal KreitzingerMar 31, 2012
  10. Jeff KingApr 2, 2012
  11. Sergio CallegariApr 3, 2012
  12. Neal KreitzingerApr 11, 2012
  13. Jonathan NiederApr 11, 2012
  14. Neal KreitzingerApr 11, 2012
  15. Jeff KingApr 11, 2012
  16. Neal KreitzingerApr 11, 2012
  17. Neal KreitzingerApr 11, 2012
  18. Jonathan NiederApr 11, 2012
  19. Junio C HamanoApr 11, 2012
  20. Jonathan NiederApr 11, 2012
  21. Neal KreitzingerApr 11, 2012
  22. Jeff KingApr 11, 2012
  23. Neal KreitzingerApr 12, 2012
  24. Jeff KingApr 12, 2012
  25. Neal KreitzingerApr 12, 2012
  26. Bo ChenApr 13, 2012
  27. Neal KreitzingerMar 31, 2012
  28. Jeff KingApr 2, 2012
  29. Junio C HamanoApr 2, 2012
  30. Jeff KingApr 3, 2012
  31. Neal KreitzingerMar 31, 2012
  32. Neal KreitzingerMar 31, 2012
  33. Bo ChenMar 31, 2012
  34. Nguyen Thai Ngoc DuyApr 1, 2012
  35. Bo ChenApr 1, 2012
  36. Nguyen Thai Ngoc DuyApr 2, 2012
  37. Bo ChenMar 30, 2012
  38. Jeff KingMar 30, 2012
  39. Jeff KingApr 15, 2012
  40. Neal KreitzingerApr 15, 2012
  41. Jeff KingApr 16, 2012
  42. Neal KreitzingerMay 10, 2012
  43. Jeff KingMay 10, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.