{"thread":{"id":"18421","subject":"Git Large Object Support Proposal","startedAt":"2009-03-19T22:14:52Z","lastAt":"2009-03-20T04:46:09Z","messageCount":11,"participants":["Scott Chacon","Junio C Hamano","david@lang.hm","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"108605","messageId":"d411cc4a0903191514n1e524ebava5895d708a2927c4@mail.gmail.com","threadId":"18421","inReplyTo":null,"subject":"Git Large Object Support Proposal","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-03-19T22:14:52Z","receivedAt":"2009-03-19T22:14:52Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"I have been thinking about this for a while, so I wanted to get some\nfeedback. I've been seeing a number of people interested in using Git\nfor game development and whatnot, or otherwise committing huge files.\nThis will occasionally wreak some havoc on our servers (GitHub)\nbecause of the memory mapping involved.  Thus, we would really like to\nsee a nicer way for Git to handle big files.\n\nThere are two proposals on the GSoC page to deal with this - the\n'remote alternates/lazy clone' idea and the 'sparse/narrow clone'\nidea.  I'm wondering if instead it might be an interesting idea to\nconcentrate on the 'stub objects' for large blobs that Jakub was\ntalking about a few months ago:\n\nhttp://markmail.org/message/my4kvrhsza2yjmlt\n\nBut where Git instead stores a stub object and the large binary object\nis pulled in via a separate mechanism. I was thinking that the client\ncould set a max file size and when a binary object larger than that is\nstaged, Git instead writes a stub blob like:\n\n==\nblob [size]\\0\n[sha of large blob]\n==\n\nThen in the tree, we give the stubbed large file a special mode or type:\n\n==\n100644 blob 3bb0e8592a41ae3185ee32266c860714980dbed7 README\n040000 tree 557b70d2374ae77869711cb583e6d59b8aad5e8b lib\n150000 blob 502feb557e2097d38a643e336f722525bc7ea077 big-ass-file.mpeg\n==\n\nSort of like a symlink, but instead of the blob it points to\ncontaining the link path, it just contains the SHA of the real blob.\nThen we can have a command like 'git media' or something that helps\nmanage those, pull them down from a specified server (specified in a\n.gitmedia file) and transfer new ones up before a push is allowed,\netc.  This makes it sort of a cross between a symlink and a submodule.\n\n== .git/config\n[media]\n    push-url = [aws/scp/sftp/etc server]\n    password = [write password]\n    token = [write token]\n\n== .gitmedia\n[server]\n    pull-url = [aws/scp/sftp/etc read only url]\n\nThis might be nice because all the objects would be local, so most of\nthe changes to tools should be rather small - we can't\nmerge/diff/blame large binary stuff really anyhow, right?  Also, the\nreally large files could be written and served over protocols that are\nbetter for large file transfer (scp, sftp, etc) - the media server\ncould be different than the git server.  Then our servers can stop\nchoking when someone tries to add and push a 2 gig file.\n\nIf two users have different settings, one would simply have the stub\nand the other not, the 'git media update' could check the local db\nfirst before fetching.  If you change the max-file-size at some point,\nthe trees would just either stop using the stubs (if you lowered it)\nfor anything that now fits under the size limit, or start using stubs\nfor files that are now over it.\n\nThe workflow may go something like this:\n\n$ cd git-repo\n$ cp ~/huge-file.mpg .\n$ git media add s3://chacon-media\n# wrote new media server url to .gitmedia\n$ git add .\n# huge-file.mpg is larger than max-file-size (10M) and will be added\nas media (see 'git media')\n$ git status\n# On branch master\n#\n# Changes to be committed:\n#   (use \"git reset HEAD <file>...\" to unstage)\n#\n#\tnew file:   .gitmedia\n#\tnew media:   huge-file.mpg\n#\n$ git push\nUploading new media to s3://chacon-media\nUploading media files 100% (5/5), done.\nNew media uploaded, pushing to Git server\nCounting objects: 14, done.\nCompressing objects: 100% (9/9), done.\nWriting objects: 100% (10/10), 1.04 KiB, done.\nTotal 10 (delta 4), reused 0 (delta 0)\nTo git@github.com:schacon/mediaproject.git\n + dbb5d00...9647674 master -> master\n\n\nOn the client side we would have something like this:\n\n$ git clone git://github.com/schacon/mediaproject.git\nInitialized empty Git repository in /private/tmp/simplegit/.git/\nremote: Counting objects: 270, done.\nremote: Compressing objects: 100% (148/148), done.\nremote: Total 270 (delta 103), reused 198 (delta 77)\nReceiving objects: 100% (270/270), 24.31 KiB, done.\nResolving deltas: 100% (103/103), done.\n# You have unfetched media, run 'git media update' to get large media files\n$ git status\n# On branch master\n#\n# Media files to be fetched:\n#   (use \"git media update <file>...\" to fetch)\n#\n#\tunfetched:   huge-file.mpg\n#\n$ git media update\nFetching media from s3://chacon-media\nFetching media files 100% (1/1), done.\n\n\nAnyhow, you get the picture.  I would be happy to try to get a proof\nof concept of this done, but I wanted to know if there are any serious\nobjections to this approach to large media.\n"},{"id":"108608","messageId":"7veiwt6t6a.fsf@gitster.siamese.dyndns.org","threadId":"18421","inReplyTo":"d411cc4a0903191514n1e524ebava5895d708a2927c4@mail.gmail.com","subject":"Re: Git Large Object Support Proposal","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-19T22:31:41Z","receivedAt":"2009-03-19T22:31:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Scott Chacon <schacon@gmail.com> writes:\n\n> But where Git instead stores a stub object and the large binary object\n> is pulled in via a separate mechanism. I was thinking that the client\n> could set a max file size and when a binary object larger than that is\n> staged, Git instead writes a stub blob like:\n>\n> ==\n> blob [size]\\0\n> [sha of large blob]\n> ==\n\nAn immediate pair of questions are, if you can solve the issue by\ndelegating large media to somebody else (i.e. \"media server\"), and that\nsomebody else can solve the issues you are having, (1) what happens if you\nlower that \"large\" threashold to \"0 byte\"?  Does that somebody else still\nwork fine, and does the git that uses indirection also still work fine?\nIf so why are you using git instead of that somebody else altogether?  and\n(2) what prevents us from stealing the trick that somebody else uses so\nthat git itself can natively handle large blobs without indirection?\n\nWithout thinking the ramifications through myself, this sounds pretty much\nlike a band-aid and will nend up hitting the same \"blob is larger than we\ncan handle\" issue when you follow the indirection eventually, but that is\njust my gut feeling.\n\nThis is an off-topic \"By the way\", but has another topic addressed to you\non git-scm.com/about resolved in any way yet?\n"},{"id":"108616","messageId":"d411cc4a0903191618x503db946n62d3132eece69175@mail.gmail.com","threadId":"18421","inReplyTo":"7veiwt6t6a.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-03-19T23:18:54Z","receivedAt":"2009-03-19T23:18:54Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nOn Thu, Mar 19, 2009 at 3:31 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Scott Chacon <schacon@gmail.com> writes:\n>\n>> But where Git instead stores a stub object and the large binary object\n>> is pulled in via a separate mechanism. I was thinking that the client\n>> could set a max file size and when a binary object larger than that is\n>> staged, Git instead writes a stub blob like:\n>>\n>> ==\n>> blob [size]\\0\n>> [sha of large blob]\n>> ==\n>\n> An immediate pair of questions are, if you can solve the issue by\n> delegating large media to somebody else (i.e. \"media server\"), and that\n> somebody else can solve the issues you are having, (1) what happens if you\n> lower that \"large\" threashold to \"0 byte\"?  Does that somebody else still\n> work fine, and does the git that uses indirection also still work fine?\n> If so why are you using git instead of that somebody else altogether?  and\n\nIn theory it would work fine, where all the commits/trees are\ntransferred over git and all the blobs are basically stored elsewhere,\nbut I would assume it would be much slower for the end user and so\nnobody would do that.  I would imagine users would only use/enable\nthis at all if they have large media files that they don't want to\nhave every version of which cloned every time.  I can't imagine that\nthis would be used at all by more than a small percentage of users,\nbut when large media does need to be in source code, they will not use\nGit (they will use Perforce or SVN), or they will put it in there and\nthen kill their (or our) servers when upload-pack tries to mmap it\n(twice, yes?).  I thought it would be much more efficient for Git to\nhave the ability to simply mark files that don't make sense to pack up\nand be able to keep track of and transfer them via a more appropriate\nprotocol.\n\n> (2) what prevents us from stealing the trick that somebody else uses so\n> that git itself can natively handle large blobs without indirection?\n>\n\nActually, I'm fine with that - phase two of this project, if it made\nsense at all, would be to have another set of git transfer commands\nthat allowed large blobs to be uploaded/downloaded separately,\nimportantly not passing them in the packfile and keeping them loose,\nuncompressed and headerless on disk so they can simply be streamed\nwhen requested.  I am thinking entirely about movies and images that\nare already compressed and there is simply no need to load them\nentirely into memory.  I simply thought that taking advantage of\nservices that already did this (scp, sftp, s3) would be quicker than\nbuilding another set of transfer protocols into Git.\n\n> Without thinking the ramifications through myself, this sounds pretty much\n> like a band-aid and will nend up hitting the same \"blob is larger than we\n> can handle\" issue when you follow the indirection eventually, but that is\n> just my gut feeling.\n\nThe point is that we don't keep this data as 'blob's - we don't try to\ncompress them or add the header to them, they're too big and already\ncompressed, it's a waste of time and often outside the memory\ntolerance of many systems. We keep only the stub in our db and stream\nthe large media content directly to and from disk.  If we do a\n'checkout' or something that would switch it out, we could store the\ndata in '.git/media' or the equivalent until it's uploaded elsewhere.\n\n>\n> This is an off-topic \"By the way\", but has another topic addressed to you\n> on git-scm.com/about resolved in any way yet?\n>\n\nThanks for pointing that out, I missed that thread.  I actually just\npushed out some changes over the last few days - I added the Gnome\nproject since they just announced they're moving to Git, added a link\nto the new OReilly book that just was released and I pulled in some\nvalidation and other misc changes that had been contributed.\n\nCurrently I have to re-gen the Authors data manually, so I do it every\nonce in a while - I just pushed up new data.  Doing it per release is\na good idea, I'll try to get that in the release script.\n"},{"id":"108620","messageId":"alpine.DEB.1.10.0903191625360.16753@asgard.lang.hm","threadId":"18421","inReplyTo":"7veiwt6t6a.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-03-19T23:42:31Z","receivedAt":"2009-03-19T23:42:31Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 19 Mar 2009, Junio C Hamano wrote:\n\n> Scott Chacon <schacon@gmail.com> writes:\n>\n>> But where Git instead stores a stub object and the large binary object\n>> is pulled in via a separate mechanism. I was thinking that the client\n>> could set a max file size and when a binary object larger than that is\n>> staged, Git instead writes a stub blob like:\n>>\n>> ==\n>> blob [size]\\0\n>> [sha of large blob]\n>> ==\n>\n> An immediate pair of questions are, if you can solve the issue by\n> delegating large media to somebody else (i.e. \"media server\"), and that\n> somebody else can solve the issues you are having, (1) what happens if you\n> lower that \"large\" threashold to \"0 byte\"?  Does that somebody else still\n> work fine, and does the git that uses indirection also still work fine?\n> If so why are you using git instead of that somebody else altogether?\n\nideally the difference between useing git with 'large' set to 0 and git \nwith no pack file should be an extra lookup for the indirection.\n\nit may be that some other file manipulation may not be possible for \n'large' files, resulting in some reduced functionality.\n\nin any case, the added efficiancy of using pack files (both for local \nstorage and for network transport) will make handling the 'large' files \nworse than the same size files through git (assuming that they can benifit \nfrom delta compression)\n\n> and\n> (2) what prevents us from stealing the trick that somebody else uses so\n> that git itself can natively handle large blobs without indirection?\n\nthe key thing is that large files do not get mmaped or considered for \ninclusion in pack files (including cloning and pulling pack files)\n\nto make them full first-class citizens you would need to make alternate \ncode paths for everything that currently does mmap, making those paths \neither process the file a different way. in the long run that may be the \nbest thing to do, but that's a lot of change compared to the proposed \nchange.\n\n> Without thinking the ramifications through myself, this sounds pretty much\n> like a band-aid and will nend up hitting the same \"blob is larger than we\n> can handle\" issue when you follow the indirection eventually, but that is\n> just my gut feeling.\n\nit depends on what you are doing with that file when you get to it. if you \nhave to mmap it you may run into the same problem. but if the file is a \nstreaming video, you can transport it around (with rsync, http, etc) \nwithout a problem, and using the file (playing the video) never keeps much \nof the file in memory, so it will be very useful on systems that would \nnever have a chance of accessing the entire file through mmap.\n\nDavid Lang\n"},{"id":"108621","messageId":"7vzlfh5b7y.fsf@gitster.siamese.dyndns.org","threadId":"18421","inReplyTo":"d411cc4a0903191618x503db946n62d3132eece69175@mail.gmail.com","subject":"Re: Git Large Object Support Proposal","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-19T23:44:49Z","receivedAt":"2009-03-19T23:44:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Scott Chacon <schacon@gmail.com> writes:\n\n> The point is that we don't keep this data as 'blob's - we don't try to\n> compress them or add the header to them, they're too big and already\n> compressed, it's a waste of time and often outside the memory\n> tolerance of many systems. We keep only the stub in our db and stream\n> the large media content directly to and from disk.  If we do a\n> 'checkout' or something that would switch it out, we could store the\n> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n\nAha, that sounds like you can just maintain a set of out-of-tree symbolic\nlinks that you keep track of, and let other people (e.g. rsync) deal with\nthe complexity of managing that side of the world.\n\nAnd I think you can start experimenting it without any change to the core\ndatastructures.  In your single-page web site in which its sole html file\nembeds an mpeg movie, you keep track of these two things in git:\n\n\tporn-of-the-day.html\n        porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n\nand any time you want to feed a new movie, you update the symlink to a\ndifferent one that lives outside the source-controlled tree, while\narranging the link target to be updated out-of-band.\n"},{"id":"108622","messageId":"alpine.DEB.1.10.0903191650160.16753@asgard.lang.hm","threadId":"18421","inReplyTo":"7vzlfh5b7y.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-03-19T23:52:19Z","receivedAt":"2009-03-19T23:52:19Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 19 Mar 2009, Junio C Hamano wrote:\n\n> Scott Chacon <schacon@gmail.com> writes:\n>\n>> The point is that we don't keep this data as 'blob's - we don't try to\n>> compress them or add the header to them, they're too big and already\n>> compressed, it's a waste of time and often outside the memory\n>> tolerance of many systems. We keep only the stub in our db and stream\n>> the large media content directly to and from disk.  If we do a\n>> 'checkout' or something that would switch it out, we could store the\n>> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n>\n> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n> links that you keep track of, and let other people (e.g. rsync) deal with\n> the complexity of managing that side of the world.\n>\n> And I think you can start experimenting it without any change to the core\n> datastructures.  In your single-page web site in which its sole html file\n> embeds an mpeg movie, you keep track of these two things in git:\n>\n> \tporn-of-the-day.html\n>        porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n>\n> and any time you want to feed a new movie, you update the symlink to a\n> different one that lives outside the source-controlled tree, while\n> arranging the link target to be updated out-of-band.\n\nthat would work, but the proposed change has some advantages\n\n1. you store the sha1 of the real mpg in the 'large file' blob so you can \ndetect problems\n\n2. since it knows the sha1 of the real file, it can auto-create the real \nfile as needed, without wasting space on too many copies of it.\n\nDavid Lang\n"},{"id":"108625","messageId":"7vtz5p59zp.fsf@gitster.siamese.dyndns.org","threadId":"18421","inReplyTo":"alpine.DEB.1.10.0903191650160.16753@asgard.lang.hm","subject":"Re: Git Large Object Support Proposal","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-20T00:11:22Z","receivedAt":"2009-03-20T00:11:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"david@lang.hm writes:\n\n> On Thu, 19 Mar 2009, Junio C Hamano wrote:\n>\n>> Scott Chacon <schacon@gmail.com> writes:\n>>\n>>> The point is that we don't keep this data as 'blob's - we don't try to\n>>> compress them or add the header to them, they're too big and already\n>>> compressed, it's a waste of time and often outside the memory\n>>> tolerance of many systems. We keep only the stub in our db and stream\n>>> the large media content directly to and from disk.  If we do a\n>>> 'checkout' or something that would switch it out, we could store the\n>>> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n>>\n>> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n>> links that you keep track of, and let other people (e.g. rsync) deal with\n>> the complexity of managing that side of the world.\n>>\n>> And I think you can start experimenting it without any change to the core\n>> datastructures.  In your single-page web site in which its sole html file\n>> embeds an mpeg movie, you keep track of these two things in git:\n>>\n>> \tporn-of-the-day.html\n>>        porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n>>\n>> and any time you want to feed a new movie, you update the symlink to a\n>> different one that lives outside the source-controlled tree, while\n>> arranging the link target to be updated out-of-band.\n>\n> that would work, but the proposed change has some advantages\n>\n> 1. you store the sha1 of the real mpg in the 'large file' blob so you\n> can detect problems\n\nYou store the unique identifier of the real mpg in the symbolic link\ntarget which is a blob payload, so you can detect problems already.  I\ndeliberately said \"unique identifier\"; you seem to think saying SHA-1\nbrings something magical but I do not think it needs to be even blob's\nSHA-1.  Hashing that much data costs.\n\nIn any case, you can have a script (or client-side hook) that does:\n\n    (1) find the out-of-tree symlinks in the index (or in the work tree);\n\n    (2) if it is dangling, and if you have definition of where to get that\n        hierarchy from (e.g ../media), run rsync or wget or whatever\n        external means to grab it.\n\nand call it after \"git pull\" updates from some other place.  The \"git\nmedia\" of Scott's message could be an alias to such a command.\n\nAdding a new type \"external-blob\" would be an unwelcome pain.  Reusing\n\"blob\" so that existing \"blob\" codepath now needs to notice special \"0\"\nthat is not length \"0\" is even bigger pain than that.\n\nAnd that is a pain for unknown benefit, especially when you can start\nexperimenting without any changes to the existing data structure.  In the\nworst case, the experiment may not pan out as well as you hoped and if\nthat is the end of the story, so be it.  It is not a great loss.  If it\nworks well enough and we can have the external large media support without\nany changes to the data structure, that would be really great.  If it\nsort-of works but hits limitation, we can analyze how best to overcome\nthat limitation, and at that time it _might_ turn out to be the best\napproach to introduce a new blob type.\n\nBut I do not think we know that yet.\n\nIn the longer run, as you speculated in your message, I think the native\nblob codepaths need to be updated to tolerate a large, unmappable objects\nbetter.  With that goal in mind, I think it is a huge mistake to\nprematurely introduce an arbitrary distinct \"blob\" and \"large blob\" types,\nif in the end they need to be merged back again; it would force the future\ncode indefinitely to care about the historical \"large blob\" types that was\nonce supported.\n\n> 2. since it knows the sha1 of the real file, it can auto-create the\n> real file as needed, without wasting space on too many copies of it.\n\nHmm, since when SHA-1 is reversible?\n"},{"id":"108627","messageId":"d411cc4a0903191719j1a413d92n9fd8b8f4ee895180@mail.gmail.com","threadId":"18421","inReplyTo":"7vtz5p59zp.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2009-03-20T00:19:15Z","receivedAt":"2009-03-20T00:19:15Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nOn Thu, Mar 19, 2009 at 5:11 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> david@lang.hm writes:\n>\n>> On Thu, 19 Mar 2009, Junio C Hamano wrote:\n>>\n>>> Scott Chacon <schacon@gmail.com> writes:\n>>>\n>>>> The point is that we don't keep this data as 'blob's - we don't try to\n>>>> compress them or add the header to them, they're too big and already\n>>>> compressed, it's a waste of time and often outside the memory\n>>>> tolerance of many systems. We keep only the stub in our db and stream\n>>>> the large media content directly to and from disk.  If we do a\n>>>> 'checkout' or something that would switch it out, we could store the\n>>>> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n>>>\n>>> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n>>> links that you keep track of, and let other people (e.g. rsync) deal with\n>>> the complexity of managing that side of the world.\n>>>\n>>> And I think you can start experimenting it without any change to the core\n>>> datastructures.  In your single-page web site in which its sole html file\n>>> embeds an mpeg movie, you keep track of these two things in git:\n>>>\n>>>      porn-of-the-day.html\n>>>        porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n>>>\n>>> and any time you want to feed a new movie, you update the symlink to a\n>>> different one that lives outside the source-controlled tree, while\n>>> arranging the link target to be updated out-of-band.\n\nIt seems like the main problem here would be that most operations in\nthe working directory would be overwriting not the symlink but the\nfile it points to.  If you do a simple 'cp ~/generated_file.mpg\nporn-of-the-day.mpg' (to upload your newest and bestest porn), it will\noverwrite the '../media/6066f5ae75ec.mpg' file, not the symlink so\nthat we can generate a new symlink.  Then if we haven't uploaded the\n'../media/6066f5ae75ec.mpg' file anywhere yet, it's a goner.  Right?\nWhat you are proposing is almost exactly what I want to do, but I'm\nconcerned with this issue of the symlink reference not working right\nfor normal working directory operations.  If a file is never\noverwritten, however, this is basically identical to what I wanted to\ndo.\n\nScott\n\n\n>>\n>> that would work, but the proposed change has some advantages\n>>\n>> 1. you store the sha1 of the real mpg in the 'large file' blob so you\n>> can detect problems\n>\n> You store the unique identifier of the real mpg in the symbolic link\n> target which is a blob payload, so you can detect problems already.  I\n> deliberately said \"unique identifier\"; you seem to think saying SHA-1\n> brings something magical but I do not think it needs to be even blob's\n> SHA-1.  Hashing that much data costs.\n>\n> In any case, you can have a script (or client-side hook) that does:\n>\n>    (1) find the out-of-tree symlinks in the index (or in the work tree);\n>\n>    (2) if it is dangling, and if you have definition of where to get that\n>        hierarchy from (e.g ../media), run rsync or wget or whatever\n>        external means to grab it.\n>\n> and call it after \"git pull\" updates from some other place.  The \"git\n> media\" of Scott's message could be an alias to such a command.\n>\n> Adding a new type \"external-blob\" would be an unwelcome pain.  Reusing\n> \"blob\" so that existing \"blob\" codepath now needs to notice special \"0\"\n> that is not length \"0\" is even bigger pain than that.\n>\n> And that is a pain for unknown benefit, especially when you can start\n> experimenting without any changes to the existing data structure.  In the\n> worst case, the experiment may not pan out as well as you hoped and if\n> that is the end of the story, so be it.  It is not a great loss.  If it\n> works well enough and we can have the external large media support without\n> any changes to the data structure, that would be really great.  If it\n> sort-of works but hits limitation, we can analyze how best to overcome\n> that limitation, and at that time it _might_ turn out to be the best\n> approach to introduce a new blob type.\n>\n> But I do not think we know that yet.\n>\n> In the longer run, as you speculated in your message, I think the native\n> blob codepaths need to be updated to tolerate a large, unmappable objects\n> better.  With that goal in mind, I think it is a huge mistake to\n> prematurely introduce an arbitrary distinct \"blob\" and \"large blob\" types,\n> if in the end they need to be merged back again; it would force the future\n> code indefinitely to care about the historical \"large blob\" types that was\n> once supported.\n>\n>> 2. since it knows the sha1 of the real file, it can auto-create the\n>> real file as needed, without wasting space on too many copies of it.\n>\n> Hmm, since when SHA-1 is reversible?\n>\n"},{"id":"108629","messageId":"alpine.DEB.1.10.0903191718140.16753@asgard.lang.hm","threadId":"18421","inReplyTo":"7vtz5p59zp.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"","fromEmail":"david@lang.hm","sentAt":"2009-03-20T00:23:49Z","receivedAt":"2009-03-20T00:23:49Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Thu, 19 Mar 2009, Junio C Hamano wrote:\n\n> david@lang.hm writes:\n>\n>> On Thu, 19 Mar 2009, Junio C Hamano wrote:\n>>\n>>> Scott Chacon <schacon@gmail.com> writes:\n>>>\n>>>> The point is that we don't keep this data as 'blob's - we don't try to\n>>>> compress them or add the header to them, they're too big and already\n>>>> compressed, it's a waste of time and often outside the memory\n>>>> tolerance of many systems. We keep only the stub in our db and stream\n>>>> the large media content directly to and from disk.  If we do a\n>>>> 'checkout' or something that would switch it out, we could store the\n>>>> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n>>>\n>>> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n>>> links that you keep track of, and let other people (e.g. rsync) deal with\n>>> the complexity of managing that side of the world.\n>>>\n>>> And I think you can start experimenting it without any change to the core\n>>> datastructures.  In your single-page web site in which its sole html file\n>>> embeds an mpeg movie, you keep track of these two things in git:\n>>>\n>>> \tporn-of-the-day.html\n>>>        porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n>>>\n>>> and any time you want to feed a new movie, you update the symlink to a\n>>> different one that lives outside the source-controlled tree, while\n>>> arranging the link target to be updated out-of-band.\n>>\n>> that would work, but the proposed change has some advantages\n>>\n>> 1. you store the sha1 of the real mpg in the 'large file' blob so you\n>> can detect problems\n>\n> You store the unique identifier of the real mpg in the symbolic link\n> target which is a blob payload, so you can detect problems already.  I\n> deliberately said \"unique identifier\"; you seem to think saying SHA-1\n> brings something magical but I do not think it needs to be even blob's\n> SHA-1.  Hashing that much data costs.\n\nbut hashing the data and using that as the unique identifier gives you \nsome advantages.\n\n1. you can detect file corruption\n\n2. you can trivially detect duplicates (even if the duplicates come from \ndifferent sources)\n\n3. it's repeatable (you will always get the same hash from the same input)\n\n> In any case, you can have a script (or client-side hook) that does:\n>\n>    (1) find the out-of-tree symlinks in the index (or in the work tree);\n>\n>    (2) if it is dangling, and if you have definition of where to get that\n>        hierarchy from (e.g ../media), run rsync or wget or whatever\n>        external means to grab it.\n>\n> and call it after \"git pull\" updates from some other place.  The \"git\n> media\" of Scott's message could be an alias to such a command.\n>\n> Adding a new type \"external-blob\" would be an unwelcome pain.  Reusing\n> \"blob\" so that existing \"blob\" codepath now needs to notice special \"0\"\n> that is not length \"0\" is even bigger pain than that.\n>\n> And that is a pain for unknown benefit, especially when you can start\n> experimenting without any changes to the existing data structure.  In the\n> worst case, the experiment may not pan out as well as you hoped and if\n> that is the end of the story, so be it.  It is not a great loss.  If it\n> works well enough and we can have the external large media support without\n> any changes to the data structure, that would be really great.  If it\n> sort-of works but hits limitation, we can analyze how best to overcome\n> that limitation, and at that time it _might_ turn out to be the best\n> approach to introduce a new blob type.\n>\n> But I do not think we know that yet.\n>\n> In the longer run, as you speculated in your message, I think the native\n> blob codepaths need to be updated to tolerate a large, unmappable objects\n> better.  With that goal in mind, I think it is a huge mistake to\n> prematurely introduce an arbitrary distinct \"blob\" and \"large blob\" types,\n> if in the end they need to be merged back again; it would force the future\n> code indefinitely to care about the historical \"large blob\" types that was\n> once supported.\n\nvalid point.\n\nkeep in mind that what's a \"large, unmappable object\" on one system may be \nno problem on another.\n\n>> 2. since it knows the sha1 of the real file, it can auto-create the\n>> real file as needed, without wasting space on too many copies of it.\n>\n> Hmm, since when SHA-1 is reversible?\n\nwhen it is processing a new, unknown file it can hash it, and look to see \nif a file with that hash exists. if so the work is done, if not it can \ncreate a file with that hash.\n\nby far the best long-term option would be to make all the codepaths handle \nunmappable files, the question is how large a task that would be.\n\nDavid Lang\n"},{"id":"108633","messageId":"7vk56l58l4.fsf@gitster.siamese.dyndns.org","threadId":"18421","inReplyTo":"7vzlfh5b7y.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-03-20T00:41:43Z","receivedAt":"2009-03-20T00:41:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Scott Chacon <schacon@gmail.com> writes:\n>\n>> The point is that we don't keep this data as 'blob's - we don't try to\n>> compress them or add the header to them, they're too big and already\n>> compressed, it's a waste of time and often outside the memory\n>> tolerance of many systems. We keep only the stub in our db and stream\n>> the large media content directly to and from disk.  If we do a\n>> 'checkout' or something that would switch it out, we could store the\n>> data in '.git/media' or the equivalent until it's uploaded elsewhere.\n>\n> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n> links that you keep track of, and let other people (e.g. rsync) deal with\n> the complexity of managing that side of the world.\n>\n> And I think you can start experimenting it without any change to the core\n> datastructures.  In your single-page web site in which its sole html file\n> embeds an mpeg movie, you keep track of these two things in git:\n>\n> \tporn-of-the-day.html\n>       porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n>\n> and any time you want to feed a new movie, you update the symlink to a\n> different one that lives outside the source-controlled tree, while\n> arranging the link target to be updated out-of-band.\n\nI wasn't thinking clearly.\n\nThis is not really a new \"huge blob\" type but is just a slightly different\nflavor of symbolic link.  Its link target name may resemble SHA-1 object\nname, but it does not participate in the reachability computation.  it\nwon't be fetched nor pushed, and if you ever get one via the usual git\ncodepath into your object store, it will be subject to \"git gc\", but you\nare unlikely to place it inside your object store to begin with.  You have\nsomething like:\n\n    100644 2222222222222222222222222222222222222222 porn-of-the-day.html\n    120001 5ed22400803161de2f49331d005be424b7f6d036 porn-of-the-day.mpg\n\nwhere 5ed22400803161de2f49331d005be424b7f6d036 is a blob that stores the\nname of a regular blob object, 6ff87c4664981e4397625791c8ea3bbb5f2279a3,\nin your tree object (and in the index), and:\n\n * When running \"git media\", you have a configuration to tell it where the\n   external media files are kept (e.g. ../media in the previous example),\n   and it rsyncs to ../media/6ff87c4664981e4397625791c8ea3bbb5f2279a3 in\n   some unspecified way from some unspecified place;\n\n * When checking out porn-of-the-day.mpg, it becomes a symbolic link that\n   points at ../media/6ff87c4664981e4397625791c8ea3bbb5f2279a3 (because it\n   follows the same site-specific configuration);\n\n * When comparing the index (that records the 120001 \"slightly different\n   symbolic link\" entry with the shell blob object) and the work tree\n   (that has a symbolic link that points at ../media/6ff87c46649...), you\n   do not look at the contents of the ../media/6ff87c46649... file, but\n   you do look at its name, apply a reverse of the mapping \"checkout\"\n   codepath did to arrive at 6ff87c4664981e4397625791c8ea3bbb5f2279a3\n   SHA-1, compare that with what the shell blob object records.  If you\n   updated the symbolic link in the work tree, \"git add\" would result in\n   creating a new shell object (just like when you change the link target\n   for a normal symbolic link) that records the external blob.\n\nIt still is bothersome that we need to introduce a new tree nodetype\n(rather, a new blob subtype similar to \"regular file blob\", \"symlink\nblob\"), but it is of much less impact than what I originally\nmisunderstood.\n\nHaving said that, if that is what is happening, I do not see the need for\nthe payload to be even a blob SHA-1 name.  Any identifier that is\nconvenient to generate in the application domain could do.\n\nBut that is a minor detail that immediately popped at me; there may be\nother minor details I may find objectionable later.  But overall, I think\nyour proposal makes sense.\n\nI still think a large part of preliminary experiments to see the benefit\nof this approach can and should be done without and before touching the\ncore part (like introduction of the slightly different symlink 1200001\nmode), though.\n"},{"id":"108650","messageId":"20090320044609.GC27160@coredump.intra.peff.net","threadId":"18421","inReplyTo":"7vzlfh5b7y.fsf@gitster.siamese.dyndns.org","subject":"Re: Git Large Object Support Proposal","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-03-20T04:46:09Z","receivedAt":"2009-03-20T04:46:09Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 19, 2009 at 04:44:49PM -0700, Junio C Hamano wrote:\n\n> Aha, that sounds like you can just maintain a set of out-of-tree symbolic\n> links that you keep track of, and let other people (e.g. rsync) deal with\n> the complexity of managing that side of the world.\n> \n> And I think you can start experimenting it without any change to the core\n> datastructures.  In your single-page web site in which its sole html file\n> embeds an mpeg movie, you keep track of these two things in git:\n> \n> \tporn-of-the-day.html\n>         porn-of-the-day.mpg -> ../media/6066f5ae75ec.mpg\n> \n> and any time you want to feed a new movie, you update the symlink to a\n> different one that lives outside the source-controlled tree, while\n> arranging the link target to be updated out-of-band.\n\nI have a repo like this (not porn, but large files :) ) and I use a\nsimilar solution. Instead of large blobs, I have stub files containing a\nURL, and the make process pulls them as necessary. It works pretty well\nin practice. I don't bother with naming the files by sha-1 but instead\ngive them human-readable names, since in my case they are generally\nimmutable (i.e., one a name is assigned, the content doesn't change).\n\n-Peff\n"}]}