{"thread":{"id":"26795","subject":"brtfs COW links and git","startedAt":"2011-03-19T20:15:32Z","lastAt":"2011-03-22T11:43:46Z","messageCount":4,"participants":["Larry D'Anna","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"163775","messageId":"20110319201532.GA6862@cthulhu","threadId":"26795","inReplyTo":null,"subject":"brtfs COW links and git","fromName":"Larry D'Anna","fromEmail":"larry@elder-gods.org","sentAt":"2011-03-19T20:15:32Z","receivedAt":"2011-03-19T20:15:32Z","isPatch":false,"sender":{"key":"larry@elder-gods.org","avatar":"https://avatars.githubusercontent.com/u/3013304?v=4"},"body":"\nI wish git could use COW links.  I wish I could put a large binary into git and\nhave the only underlying filesystem operation be to cp --reflink and to save the\nmetadata.  There are a few complications:\n\nHow does it know which files to reflink?  attributes?  a size limit?\n\nWhat does git gc do with reflinks?\n\nShould diff-delta be reflink-aware?  Perhaps it could query the fs for\nblocklists.\n\nBefore I dive into implementing this, I'd like to get your comments and advice,\nto maximize the chances of success.  \n\nThanks!\n"},{"id":"163899","messageId":"20110321120051.GG16334@sigill.intra.peff.net","threadId":"26795","inReplyTo":"20110319201532.GA6862@cthulhu","subject":"Re: brtfs COW links and git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-21T12:00:51Z","receivedAt":"2011-03-21T12:00:51Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 19, 2011 at 04:15:32PM -0400, Larry D'Anna wrote:\n\n> I wish git could use COW links.  I wish I could put a large binary into git and\n> have the only underlying filesystem operation be to cp --reflink and to save the\n> metadata.  There are a few complications:\n\nI have never used reflink, but my understanding is that the proposed\nsystem call just lets us reflink one entire file. So basically the\nuseful points would be:\n\n  1. on \"git add\", we could reflink the file into the object db\n\n  2. on \"git checkout\", we could reflink the object into the working\n     tree\n\nThe biggest stumbling block is that the object db does not currently\nhold unadorned files. They have an object type and size header at the\nbeginning, and I believe even uncompressed files are stored with a zlib\nheader. So it would require a completely new section of the object db to\nstore these files (as opposed to the current loose objects and\npackfiles).\n\nNote also that during \"git add\" we will need to get the sha1 of the\ndata. So you'll still have to pull all the data from disk, though not\nmaking a copy will save some space and time.\n\nI'm not very knowledgeable on the current state of such things, but\nis there any automatic de-duplication in btrfs? If so, does it depend on\ndata being at the same offsets within files?\n\n> How does it know which files to reflink?  attributes?  a size limit?\n\nProbably supporting both would make sense.\n\n> What does git gc do with reflinks?\n\nIf we had a \"giant literal blobs\" section of the object database, we\nwould not want to pack those objects during a regular gc. It would kill\nyour reflink, but also there's just no point in copying some gigantic\nfile into a pack where it won't actually be delta-compressed.\n\n> Should diff-delta be reflink-aware?  Perhaps it could query the fs for\n> blocklists.\n\nWouldn't it just be sharing underlying data between the working tree and\nthe object database? How would that help us make deltas between objects?\n\n> Before I dive into implementing this, I'd like to get your comments and advice,\n> to maximize the chances of success.\n\nI'm not exactly clear on what you want to implement.\n\n-Peff\n"},{"id":"164010","messageId":"20110322024421.GA15134@cthulhu","threadId":"26795","inReplyTo":"20110321120051.GG16334@sigill.intra.peff.net","subject":"Re: brtfs COW links and git","fromName":"Larry D'Anna","fromEmail":"larry@elder-gods.org","sentAt":"2011-03-22T02:44:21Z","receivedAt":"2011-03-22T02:44:21Z","isPatch":false,"sender":{"key":"larry@elder-gods.org","avatar":"https://avatars.githubusercontent.com/u/3013304?v=4"},"body":"* Jeff King (peff@peff.net) [110321 08:00]:\n> I'm not exactly clear on what you want to implement.\n\nNeither was I, that's why I sent the vague email :-)\n\nI think I have a better understanding now of what would need to be done:\n\n* reading unadorned blobs, as a last resort if the object isn't found elsewhere.\n  use reflink (if available) to copy an unadorned blob into the working directory \n\n* writing unadorned blobs, according to size limit / attribute.\n  this also means computing the sha1 for the blob without reading\n  the entire thing into memory all at once.\n\n* leaving unadorned blobs alone in gc, unless explicitly told not to\n\n* supporting the easy cases of binary diffs for git-upload-pack.  it shouldn't \n  have to send an entire copy of a huge file if only a little bit of the file \n  changed.   This could use fiemap, or maybe bup's rolling checksum, or maybe \n  either, depending on what's available.\n\n* support for applying those binary diffs directly to the on-disk reflink.\n  this could probably just mmap the file and call patch-delta.\n\nI think these features would make some, but not all of the big file use cases\nmuch more usable.  What do you think?\n"},{"id":"164037","messageId":"20110322114346.GC32446@sigill.intra.peff.net","threadId":"26795","inReplyTo":"20110322024421.GA15134@cthulhu","subject":"Re: brtfs COW links and git","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2011-03-22T11:43:46Z","receivedAt":"2011-03-22T11:43:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 21, 2011 at 10:44:21PM -0400, Larry D'Anna wrote:\n\n> * Jeff King (peff@peff.net) [110321 08:00]:\n> > I'm not exactly clear on what you want to implement.\n> \n> Neither was I, that's why I sent the vague email :-)\n\nFair enough. :)\n\n> I think I have a better understanding now of what would need to be done:\n> \n> * reading unadorned blobs, as a last resort if the object isn't found elsewhere.\n>   use reflink (if available) to copy an unadorned blob into the working directory \n> \n> * writing unadorned blobs, according to size limit / attribute.\n>   this also means computing the sha1 for the blob without reading\n>   the entire thing into memory all at once.\n> \n> * leaving unadorned blobs alone in gc, unless explicitly told not to\n\nYeah, all of these are sensible. I do get a little nervous because an\nextra object-db area is a change to such a core part of git. I think the\nperformance improvement would have to be pretty impressive to be worth\nthe trouble (and I think it has the potential to be; I just think you\nwill need numbers to make this palatable for upstream inclusion).\n\nYou could also prototype it with something like git-media that is\nexternal to git. If you haven't read this recent thread, take a look:\n\n  http://thread.gmane.org/gmane.comp.version-control.git/165389/focus=165389\n\n> * supporting the easy cases of binary diffs for git-upload-pack.  it shouldn't \n>   have to send an entire copy of a huge file if only a little bit of the file \n>   changed.   This could use fiemap, or maybe bup's rolling checksum, or maybe \n>   either, depending on what's available.\n\nThe good news is that we already handle binary diffs in upload-pack via\ndeltas. The trick is just efficiently generating them for large files.\nIf you have two large objects which are sharing blocks, one block is a\ndelta candidate for the other (i.e., the remote has told you he has one\nbut needs the other), and you have fiemap support on the sending end,\nthen you should be able to efficiently generate a delta on the fly.\n\nIn practice, I don't know how useful that is. One of the killer\napplications people want for this kind of support is for media files.\nBut when they change, they tend to change a lot. I don't know if deltas\nbetween two versions of an audio or video file will really be useful.\nBut that isn't to say it won't help for some other files. I think part\nof showing good numbers would be defining a plausible workload.\n\n> * support for applying those binary diffs directly to the on-disk reflink.\n>   this could probably just mmap the file and call patch-delta.\n\nThe patch may need to expand a section in the middle of the file. I\ndon't know what kind of syscall support there is for that level of\ntweaking with reflinked files.\n\n> I think these features would make some, but not all of the big file use cases\n> much more usable.  What do you think?\n\nI think they could help, especially for gigantic files where just\ncopying the file is painful, and the extra storage kills you. But there\nare a lot of other issues, too, like git assuming it can pull whole\nblobs into memory during diffs. There is some low-hanging fruit there,\nand that's why I put forward the SoC project idea.\n\n-Peff\n"}]}