git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: GSoC - Some questions on the idea of

From
Jeff King <peff@peff.net>
Date
Apr 3, 2012, 10:07 UTC
Message-ID
<20120403100704.GC14483@sigill.intra.peff.net>
In-Reply-To
<7vvclhdbew.fsf@alter.siamese.dyndns.org>
On Mon, Apr 02, 2012 at 03:19:35PM -0700, Junio C Hamano wrote:
Show 13 quoted lines
> >   1. You really have 100G of data in the current version that doesn't
> >      compress well (e.g., you are storing your music collection). You
> >      can't afford to store two copies on your laptop (because you have a
> >      fancy SSD, and 100G is expensive again).  You need the working tree
> >      version, but it's OK to stream the repo version of a blob from the
> >      network when you actually need it (mostly "checkout", assuming you
> >      have marked the file as "-diff").
> 
> This feels like a good candidate for an independent project that allows
> you fuse-mount from a remote repository to give you an illusion that you
> have a checkout of a specific version.  Such a remote fuse-server would be
> an application that is built using Git, but I do not think we are in any
> business on the client end in such a setup.

I think this is backwards. The primary item you want on the laptop is the working directory, because you will be accessing and manipulating the files. That must always work, whether the network is connected or not. You occasionally will want to perform git operations. Most of these should succeed when disconnected, but it's OK for some operations (like checking out an older version of a large blob) to fail.

But if you are mounting a remote repository and pretending that you have a local checkout, then just accessing the files either requires a network, or you end up caching most of the remote repository.

It would make more sense to me to clone a bare repository of what's upstream, and then fuse-mount the local bare repository to provide a fake working directory. And I believe somebody made such a fuse filesystem in the early days of git. However, I recall that it was read-only. I'm not sure how you would handle writing to the git-mounted directory.

Show 6 quoted lines
> Or you can split out the really large write-only blobs out of SCM control.
> Every time you introduce a new blob, throw it verbatim in an append-only
> directory on a networked filesystem under some unique ID as its filename,
> and maintain a symlink into that networked filesystem under SCM control.
> 
> I think git-annex already does something like that...

Yes, and git-media basically does this, too. But it's awful to use, because the user has to be constantly aware of these special links and managing them. You can't just store a symlink into the networked filesystem. For one thing, the path may be different on each client machine, so a simple symlink doesn't work. For another, symlinks into a blob repository mean that the files must be read-only (since they are basically blob-equivalents). So you don't really get your own copy of the file; you can _replace_ it and update the symlink, but you can't actually modify it.

So what things like git-media end up doing is to try to insert themselves between git and the user, and transparently convert the file into its unique ID on "git add" and tweak the working directory to contain the actual file on checkout. And it kind of works, but there are a lot of rough edges (I don't recall the details, but they came up in past discussions; clean and smudge filters almost get you there, but not quite).

Basically what I'm proposing to do is to just move that logic into git itself, so it can just happen at the blob storage level. I don't think it would even be that much code inside git; you'd want the interface to be pluggable, so all of the heavy lifting would happen inside of a helper (so really, this isn't necessarily even "network alternates" as much as "pluggable alternates").

-Peff
Previous: Junio C HamanoNext: Neal Kreitzinger
Message 30 of 43 in “GSoC - Some questions on the idea of "Better big-file support".”
  1. Bo ChenMar 28, 2012
  2. Nguyen Thai Ngoc DuyMar 28, 2012
  3. SergioMar 28, 2012
  4. Bo ChenMar 30, 2012
  5. Bo ChenMar 30, 2012
  6. Jeff KingMar 30, 2012
  7. Bo ChenMar 30, 2012
  8. Sergio CallegariMar 31, 2012
  9. Neal KreitzingerMar 31, 2012
  10. Jeff KingApr 2, 2012
  11. Sergio CallegariApr 3, 2012
  12. Neal KreitzingerApr 11, 2012
  13. Jonathan NiederApr 11, 2012
  14. Neal KreitzingerApr 11, 2012
  15. Jeff KingApr 11, 2012
  16. Neal KreitzingerApr 11, 2012
  17. Neal KreitzingerApr 11, 2012
  18. Jonathan NiederApr 11, 2012
  19. Junio C HamanoApr 11, 2012
  20. Jonathan NiederApr 11, 2012
  21. Neal KreitzingerApr 11, 2012
  22. Jeff KingApr 11, 2012
  23. Neal KreitzingerApr 12, 2012
  24. Jeff KingApr 12, 2012
  25. Neal KreitzingerApr 12, 2012
  26. Bo ChenApr 13, 2012
  27. Neal KreitzingerMar 31, 2012
  28. Jeff KingApr 2, 2012
  29. Junio C HamanoApr 2, 2012
  30. Jeff KingApr 3, 2012
  31. Neal KreitzingerMar 31, 2012
  32. Neal KreitzingerMar 31, 2012
  33. Bo ChenMar 31, 2012
  34. Nguyen Thai Ngoc DuyApr 1, 2012
  35. Bo ChenApr 1, 2012
  36. Nguyen Thai Ngoc DuyApr 2, 2012
  37. Bo ChenMar 30, 2012
  38. Jeff KingMar 30, 2012
  39. Jeff KingApr 15, 2012
  40. Neal KreitzingerApr 15, 2012
  41. Jeff KingApr 16, 2012
  42. Neal KreitzingerMay 10, 2012
  43. Jeff KingMay 10, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.