{"thread":{"id":"17337","subject":"read-only working copies using links","startedAt":"2009-01-24T09:17:19Z","lastAt":"2009-01-24T19:35:08Z","messageCount":6,"participants":["Chad Dombrova","Sverre Rabbelier","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"101735","messageId":"3EE64C92-CB4C-47BD-9C48-E369AED4B82F@gmail.com","threadId":"17337","inReplyTo":null,"subject":"read-only working copies using links","fromName":"Chad Dombrova","fromEmail":"chadrik@gmail.com","sentAt":"2009-01-24T09:17:19Z","receivedAt":"2009-01-24T09:17:19Z","isPatch":false,"sender":{"key":"chadrik@gmail.com","avatar":null},"body":"hi all,\n\nthere's a major feature for working with large binaries that has not \nyet been addressed by git: the ability to check out a file as a \nsymbolic/hard link to a blob in the repository, instead of duplicating \nthe file into the working copy.\n\nimagine a scenario where one user is putting large binary files into a \ngit repo on a networked server. 100 other users on the server need \nread-only access to this repo. they clone the repo using --shared or \n--local, which saves disk space for the object files, but each of \nthese 100 working copies also creates copies of all the binary files \nat the HEAD revision. it would be 100x as efficient in both disk space \nand checkout speeds if, in place of these files, symbolic or hard \nlinks were made to the blob files in .git/objects.\n\nthe crux of the issue is that the blob objects would have to be stored \nas exact copies of the original files. it would seem there are two \nthings that currently prevent this from happening. 1) blobs are \nstored with compression and 2) they include a small header. \ncompression can be disabled by setting core.loosecompression to 0, so \nthat seems like less of an issue. as for the header, wouldn't it be \npossible to store it separately? in other words, store two files per \nblob directory, a small stub file with the header info and the \nunaltered file data.\n\nwhat are the caveats to a system like this? has anyone looked into \nthis before?\n\n-chad\n\np.s.\ni tried submitting a post through nabble a few days and it said that \nit was still pending, so i thought i'd try submitting directly to the \nmailing list. sorry, if i end up double-posting\n"},{"id":"101737","messageId":"bd6139dc0901240302q6b3c9a13p56fe39a976882133@mail.gmail.com","threadId":"17337","inReplyTo":"3EE64C92-CB4C-47BD-9C48-E369AED4B82F@gmail.com","subject":"Re: read-only working copies using links","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-01-24T11:02:36Z","receivedAt":"2009-01-24T11:02:36Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"Heya,\n\nOn Sat, Jan 24, 2009 at 10:17, Chad Dombrova wrote:\n> the crux of the issue is that the blob objects would have to be stored as\n> exact copies of the original files. it would seem there are two things that\n> currently prevent this from happening. 1) blobs are stored with compression\n> and 2) they include a small header. compression can be disabled by setting\n> core.loosecompression to 0, so that seems like less of an issue. as for the\n> header, wouldn't it be possible to store it separately? in other words,\n> store two files per blob directory, a small stub file with the header info\n> and the unaltered file data.\n\nI think Tim Ansell (cced) was talking about this at the gittogether\n(storing the metadata seperately), as it would benefit sparse/narrow\ncheckout, another advantage supporting his case?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"101760","messageId":"AE9781AD-0B63-4AEE-9B62-20F9720B5DC4@gmail.com","threadId":"17337","inReplyTo":"bd6139dc0901240302q6b3c9a13p56fe39a976882133@mail.gmail.com","subject":"Re: read-only working copies using links","fromName":"Chad Dombrova","fromEmail":"chadrik@gmail.com","sentAt":"2009-01-24T18:39:46Z","receivedAt":"2009-01-24T18:39:46Z","isPatch":false,"sender":{"key":"chadrik@gmail.com","avatar":null},"body":">\n> I think Tim Ansell (cced) was talking about this at the gittogether\n> (storing the metadata seperately), as it would benefit sparse/narrow\n> checkout, another advantage supporting his case?\n>\n\nwhat's the case against it, other than the obvious, that it will take \nmore work?\n\n\n-chad\n"},{"id":"101761","messageId":"bd6139dc0901241043h7087316bgcbcb5d9085d603ef@mail.gmail.com","threadId":"17337","inReplyTo":"AE9781AD-0B63-4AEE-9B62-20F9720B5DC4@gmail.com","subject":"Re: read-only working copies using links","fromName":"Sverre Rabbelier","fromEmail":"srabbelier@gmail.com","sentAt":"2009-01-24T18:43:20Z","receivedAt":"2009-01-24T18:43:20Z","isPatch":false,"sender":{"key":"srabbelier@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3098?v=4"},"body":"On Sat, Jan 24, 2009 at 19:39, Chad Dombrova wrote:\n> what's the case against it, other than the obvious, that it will take more\n> work?\n\nGood question, I think it was mostly that, someone has to implement it\n(possibly as part of packv4). Backwards compatibility is of course\nalways an concern, but I'm not too familiar with the subject, perhaps\nother people on the list (or even those were at the gittogether) can\ncomment?\n\n-- \nCheers,\n\nSverre Rabbelier\n"},{"id":"101770","messageId":"20090124193422.GB18010@coredump.intra.peff.net","threadId":"17337","inReplyTo":"AE9781AD-0B63-4AEE-9B62-20F9720B5DC4@gmail.com","subject":"Re: read-only working copies using links","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-01-24T19:34:22Z","receivedAt":"2009-01-24T19:34:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 24, 2009 at 10:39:46AM -0800, Chad Dombrova wrote:\n\n>> I think Tim Ansell (cced) was talking about this at the gittogether\n>> (storing the metadata seperately), as it would benefit sparse/narrow\n>> checkout, another advantage supporting his case?\n>\n> what's the case against it, other than the obvious, that it will take \n> more work?\n\nI'm not sure this is actually the same as Tim's proposal. Tim wanted to\nstore the commit and tree information separately from the blob\ninformation (since his use case was that blobs are enormous, but the\nrest is reasonable).\n\nAIUI, Chad's proposal is about storing the actual blob data itself\nseparate from the blob object's metadata (i.e., its object type and\nlength headers). Which means that the normal loose object format is not\nacceptable, and you would end up with something like (for example):\n\n .git/objects/pack/pack-full-of-your-regular-stuff.{pack,idx}\n .git/objects/[0-9a-f]{2}/[0-9a-f]{38}/header\n .git/objects/[0-9a-f]{2}/[0-9a-f]{38}/data\n\nor something similar. Then you could hardlink directly to the 'data'\nportion. So you would need:\n\n - to teach everything that ever looks for loose objects how to read\n this new format. In theory, it's all nicely encapsulated in\n sha1_file.c\n\n - to teach checkout routines to hardlink such a case instead of\n copying the file\n\nThe obvious downsides that I can think of are:\n\n - it has the potential to make object reading, which is a core part of\n git (read: very performance- and correctness- sensitive) a lot more\n complex. But maybe the implementation would not be that painful;\n somebody would have to look very closely to see.\n\n - it interacts badly with smudge/clean filters and crlf conversion.\n In those cases you can't hardlink. If you treat this like an\n optimization, though, it's not so bad: we only do the optimization\n when we _can_, and fall back to regular checkout if those other\n options are in effect.\n\n - it's somewhat dangerous to your repository's health. Git's model is\n that object files are immutable (since they are, after all, named\n after their contents). But now you are linking them into your\n working tree, which makes them susceptible to some third party tool\n munging them. So yes, most tools will probably behave, but any tool\n that misbehaves will actually corrupt your repository.\n\n-Peff\n"},{"id":"101771","messageId":"20090124193508.GC18010@coredump.intra.peff.net","threadId":"17337","inReplyTo":"bd6139dc0901241043h7087316bgcbcb5d9085d603ef@mail.gmail.com","subject":"Re: read-only working copies using links","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-01-24T19:35:08Z","receivedAt":"2009-01-24T19:35:08Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 24, 2009 at 07:43:20PM +0100, Sverre Rabbelier wrote:\n\n> On Sat, Jan 24, 2009 at 19:39, Chad Dombrova wrote:\n> > what's the case against it, other than the obvious, that it will take more\n> > work?\n> \n> Good question, I think it was mostly that, someone has to implement it\n> (possibly as part of packv4). Backwards compatibility is of course\n> always an concern, but I'm not too familiar with the subject, perhaps\n> other people on the list (or even those were at the gittogether) can\n> comment?\n\nIf I understand his proposal correctly, such objects must _not_ be part\nof a pack. The whole idea is splitting them _more_, not less.\n\n-Peff\n"}]}