git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: GSoC - Some questions on the idea of

From
NKNeal Kreitzinger <nkreitzinger@gmail.com>
Date
Mar 31, 2012, 16:49 UTC
Message-ID
<4F7735B7.1050707@gmail.com>
In-Reply-To
<20120330203430.GB20376@sigill.intra.peff.net>
On 3/30/2012 3:34 PM, Jeff King wrote:
Show 23 quoted lines
> On Fri, Mar 30, 2012 at 03:51:20PM -0400, Bo Chen wrote:
>
>> The sub-problems of "delta for large file" problem.
>>
>> 1 large file
>>
>> 1.1 text file (always delta well? need to be confirmed)
>
> ...But let's take a step back for a moment. Forget about whether a file
> is binary or not. Imagine you want to store a very large file in
> git.
>
> ...Nowadays, we stream large files directly into their own packfiles,
> and we have to pay the I/O only once (and the memory cost never). As
> a tradeoff, we no longer get delta compression of large objects.
> That's OK for some large objects, like movie files (which don't tend
> to delta well, anyway). But it's not for other objects, like virtual
> machine images, which do tend to delta well.
>
> So can we devise a solution which efficiently stores these
> delta-friendly objects, without losing the performance improvements
> we got with the stream-directly-to-packfile approach?
>

gitconfig or gitattributes could specify big-file handlers for filetypes. It seems a bit ridiculous to expect git to autoconfigure big-file handlers for everything from gif's to vm-images. In the case of vm-images you would need to read the "big-files" man-page and then configure your git for the "vm image handler" for whatever your vm-image wildcards are for those files. For movie files you would also read the big-file man-page and configure "movie file 'x' big file handler' for whatever your movie file wildcards are. Movie files and vm-images are very expectable (version control) but not very normative (source code management) so you need to configure those as needed. More widely-tracked-by-the-public-at-large files like gif, png, etc, could be autoconfigured by git to used the correct big-file handler.

v/r, neal

Previous: Jeff KingNext: Neal Kreitzinger
Message 31 of 43 in “GSoC - Some questions on the idea of "Better big-file support".”
  1. Bo ChenMar 28, 2012
  2. Nguyen Thai Ngoc DuyMar 28, 2012
  3. SergioMar 28, 2012
  4. Bo ChenMar 30, 2012
  5. Bo ChenMar 30, 2012
  6. Jeff KingMar 30, 2012
  7. Bo ChenMar 30, 2012
  8. Sergio CallegariMar 31, 2012
  9. Neal KreitzingerMar 31, 2012
  10. Jeff KingApr 2, 2012
  11. Sergio CallegariApr 3, 2012
  12. Neal KreitzingerApr 11, 2012
  13. Jonathan NiederApr 11, 2012
  14. Neal KreitzingerApr 11, 2012
  15. Jeff KingApr 11, 2012
  16. Neal KreitzingerApr 11, 2012
  17. Neal KreitzingerApr 11, 2012
  18. Jonathan NiederApr 11, 2012
  19. Junio C HamanoApr 11, 2012
  20. Jonathan NiederApr 11, 2012
  21. Neal KreitzingerApr 11, 2012
  22. Jeff KingApr 11, 2012
  23. Neal KreitzingerApr 12, 2012
  24. Jeff KingApr 12, 2012
  25. Neal KreitzingerApr 12, 2012
  26. Bo ChenApr 13, 2012
  27. Neal KreitzingerMar 31, 2012
  28. Jeff KingApr 2, 2012
  29. Junio C HamanoApr 2, 2012
  30. Jeff KingApr 3, 2012
  31. Neal KreitzingerMar 31, 2012
  32. Neal KreitzingerMar 31, 2012
  33. Bo ChenMar 31, 2012
  34. Nguyen Thai Ngoc DuyApr 1, 2012
  35. Bo ChenApr 1, 2012
  36. Nguyen Thai Ngoc DuyApr 2, 2012
  37. Bo ChenMar 30, 2012
  38. Jeff KingMar 30, 2012
  39. Jeff KingApr 15, 2012
  40. Neal KreitzingerApr 15, 2012
  41. Jeff KingApr 16, 2012
  42. Neal KreitzingerMay 10, 2012
  43. Jeff KingMay 10, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.