git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: GSoC - Some questions on the idea of

From
Junio C Hamano <gitster@pobox.com>
Date
Apr 2, 2012, 22:19 UTC
Message-ID
<7vvclhdbew.fsf@alter.siamese.dyndns.org>
In-Reply-To
<20120402214049.GB28926@sigill.intra.peff.net>
Jeff King <peff@peff.net> writes:
Show 7 quoted lines
>   1. You really have 100G of data in the current version that doesn't
>      compress well (e.g., you are storing your music collection). You
>      can't afford to store two copies on your laptop (because you have a
>      fancy SSD, and 100G is expensive again).  You need the working tree
>      version, but it's OK to stream the repo version of a blob from the
>      network when you actually need it (mostly "checkout", assuming you
>      have marked the file as "-diff").

This feels like a good candidate for an independent project that allows you fuse-mount from a remote repository to give you an illusion that you have a checkout of a specific version. Such a remote fuse-server would be an application that is built using Git, but I do not think we are in any business on the client end in such a setup.

So I'll write it off as a "non-Git" issue for now.
The other parts of your message is much more interesting.
Show 22 quoted lines
> Right. This is the same concept, except over the network. So people's
> working repositories are on their own workstations instead of a central
> server. You could even do it today by network-mounting a filesystem and
> pointing your alternates file at it. However, I think it's worth making
> git aware that the objects are on the network for a few reasons:
>
>   1. Git can be more careful about how it handles the objects, including
>      when to fetch, when to stream, and when to cache. For example,
>      you'd want to fetch the manifest of objects and cache it in your
>      local repository, because you want fast lookups of "do I have this
>      object".
>
>   2. Providing remote filesystems on an Internet scale is a management
>      pain (and it's a pain for the user, too). My thought was that this
>      would be implemented on top of http (the connection setup cost is
>      negligible, since these objects would generally be large).
>
>   3. Usually alternate repositories are full repositories that meet the
>      connectivity requirements (so you could run "git fsck" in them).
>      But this is explicitly about taking just a few disconnected large
>      blobs out of the repository and putting them elsewhere. So it needs
>      a new set of tools for managing the upstream repository.

Or you can split out the really large write-only blobs out of SCM control. Every time you introduce a new blob, throw it verbatim in an append-only directory on a networked filesystem under some unique ID as its filename, and maintain a symlink into that networked filesystem under SCM control.

I think git-annex already does something like that...
Previous: Jeff KingNext: Jeff King
Message 29 of 43 in “GSoC - Some questions on the idea of "Better big-file support".”
  1. Bo ChenMar 28, 2012
  2. Nguyen Thai Ngoc DuyMar 28, 2012
  3. SergioMar 28, 2012
  4. Bo ChenMar 30, 2012
  5. Bo ChenMar 30, 2012
  6. Jeff KingMar 30, 2012
  7. Bo ChenMar 30, 2012
  8. Sergio CallegariMar 31, 2012
  9. Neal KreitzingerMar 31, 2012
  10. Jeff KingApr 2, 2012
  11. Sergio CallegariApr 3, 2012
  12. Neal KreitzingerApr 11, 2012
  13. Jonathan NiederApr 11, 2012
  14. Neal KreitzingerApr 11, 2012
  15. Jeff KingApr 11, 2012
  16. Neal KreitzingerApr 11, 2012
  17. Neal KreitzingerApr 11, 2012
  18. Jonathan NiederApr 11, 2012
  19. Junio C HamanoApr 11, 2012
  20. Jonathan NiederApr 11, 2012
  21. Neal KreitzingerApr 11, 2012
  22. Jeff KingApr 11, 2012
  23. Neal KreitzingerApr 12, 2012
  24. Jeff KingApr 12, 2012
  25. Neal KreitzingerApr 12, 2012
  26. Bo ChenApr 13, 2012
  27. Neal KreitzingerMar 31, 2012
  28. Jeff KingApr 2, 2012
  29. Junio C HamanoApr 2, 2012
  30. Jeff KingApr 3, 2012
  31. Neal KreitzingerMar 31, 2012
  32. Neal KreitzingerMar 31, 2012
  33. Bo ChenMar 31, 2012
  34. Nguyen Thai Ngoc DuyApr 1, 2012
  35. Bo ChenApr 1, 2012
  36. Nguyen Thai Ngoc DuyApr 2, 2012
  37. Bo ChenMar 30, 2012
  38. Jeff KingMar 30, 2012
  39. Jeff KingApr 15, 2012
  40. Neal KreitzingerApr 15, 2012
  41. Jeff KingApr 16, 2012
  42. Neal KreitzingerMay 10, 2012
  43. Jeff KingMay 10, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.