{"thread":{"id":"17437","subject":"Appropriateness of git for digital video production versioning","startedAt":"2009-01-29T15:36:55Z","lastAt":"2009-01-30T10:13:54Z","messageCount":4,"participants":["Charles Earl","Shawn O. Pearce","Pau Garcia i Quiles","Christian MICHON"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"102467","messageId":"8c4a72800901290736p4952e53byddca243f300dd8af@mail.gmail.com","threadId":"17437","inReplyTo":null,"subject":"Appropriateness of git for digital video production versioning","fromName":"Charles Earl","fromEmail":"charles.cearl@gmail.com","sentAt":"2009-01-29T15:36:55Z","receivedAt":"2009-01-29T15:36:55Z","isPatch":false,"sender":{"key":"charles.cearl@gmail.com","avatar":null},"body":"Hi,\nAre there past instances of git having been adapted to support version\ncontrol of digital media production workflow?\nI'm evaluating CMS and versioning systems for the backend of a SaaS\nfor digital media production workflow.\nThe bulk of content stored is binary data -- there have been posts on\nthis about integration of various binary diff implemetations with git.\nThe versioning of metadata, scripts, project structure seems to argue\nfor applicability of system such as git -- these fit the paradigm of\ntraditional scm.\nExample content is from media production suites such as Adobe After\nEffects/Premier: video, compositions, etc.\nI'd also like the object storage to be in S3/Amazon BlockStore or\nsimilar remote stores.\nCharles\n"},{"id":"102469","messageId":"20090129154536.GC26880@spearce.org","threadId":"17437","inReplyTo":"8c4a72800901290736p4952e53byddca243f300dd8af@mail.gmail.com","subject":"Re: Appropriateness of git for digital video production versioning","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-01-29T15:45:36Z","receivedAt":"2009-01-29T15:45:36Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Charles Earl wrote:\n> Are there past instances of git having been adapted to support version\n> control of digital media production workflow?\n\nYou are going to run into scaling problems. Git works under the\nassumption that it can malloc() at least 2 complete copies of a\nfile at once, in the same process.\n\nLast time I mucked around with digital media production, the volume\nof data in a video file was *huge*. Its workable on modern systems\nwith terabyte disk arrays and so forth, but modern systems still\ncan't afford the 100 GB of RAM necessary to allow Git to malloc()\nup two blocks of a single 40 GB video file.\n\nAlso, since clients pretty much grab the entire repository when they\nclone it for working access, its going to suck down the entire media\narchive, *all* versions. That could be well into the hundreds of\nTB range and may never complete.\n\n> The bulk of content stored is binary data -- there have been posts on\n> this about integration of various binary diff implemetations with git.\n\nYou mention later about using S3 or BlockStore to hold the binary\ncontent. Maybe the large binary data should be stored in S3, and\nthe Git repository just holds the metadata and scripts, including\nscripts to perform downloads/uploads through S3.\n\n> The versioning of metadata, scripts, project structure seems to argue\n> for applicability of system such as git -- these fit the paradigm of\n> traditional scm.\n\nYea, that's more typical of what Git was designed and built to store.\n\n> Example content is from media production suites such as Adobe After\n> Effects/Premier: video, compositions, etc.\n> I'd also like the object storage to be in S3/Amazon BlockStore or\n> similar remote stores.\n\nMy suggestion?\n\nUse Git for your metadata and scripts. Include a few scripts that\ncan download the large media files from S3 when they are needed,\nand upload new versions when they are modified.\n\nIf you want to store versions over time of the files, sha1sum\nthe media file and use that as the key name in the S3 bucket,\nand store the output of sha1sum into a file within Git. E.g. a\n\".media\" text file just listing out sha1sum and path names:\n\n ceba7222551c722836564535697947e8a9b3e7ce big_file.mpg\n 75c8f5ecb97ec67c1ec949b16c72e6ba1361a528 other_file.mpg\n\nand use a simple script to edit/read that file, accessing S3 as\nnecessary for the operations.\n\n-- \nShawn.\n"},{"id":"102470","messageId":"3af572ac0901290748g5c5b3043j425adbda8f5d7619@mail.gmail.com","threadId":"17437","inReplyTo":"8c4a72800901290736p4952e53byddca243f300dd8af@mail.gmail.com","subject":"Re: Appropriateness of git for digital video production versioning","fromName":"Pau Garcia i Quiles","fromEmail":"pgquiles@elpauer.org","sentAt":"2009-01-29T15:48:35Z","receivedAt":"2009-01-29T15:48:35Z","isPatch":false,"sender":{"key":"pgquiles@elpauer.org","avatar":null},"body":"Hello,\n\nGit is not the appropriate tool. What about using a filesystem with\nversioning support? tux3, btrfs, ZFS and NILFS may work for you.\n\nOn Thu, Jan 29, 2009 at 4:36 PM, Charles Earl wrote:\n> Hi,\n> Are there past instances of git having been adapted to support version\n> control of digital media production workflow?\n> I'm evaluating CMS and versioning systems for the backend of a SaaS\n> for digital media production workflow.\n> The bulk of content stored is binary data -- there have been posts on\n> this about integration of various binary diff implemetations with git.\n> The versioning of metadata, scripts, project structure seems to argue\n> for applicability of system such as git -- these fit the paradigm of\n> traditional scm.\n> Example content is from media production suites such as Adobe After\n> Effects/Premier: video, compositions, etc.\n> I'd also like the object storage to be in S3/Amazon BlockStore or\n> similar remote stores.\n> Charles\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n\n\n\n-- \nPau Garcia i Quiles\nhttp://www.elpauer.org\n(Due to my workload, I may need 10 days to answer)\n"},{"id":"102559","messageId":"46d6db660901300213q21ad099w2b49ceba8690d51a@mail.gmail.com","threadId":"17437","inReplyTo":"8c4a72800901290736p4952e53byddca243f300dd8af@mail.gmail.com","subject":"Re: Appropriateness of git for digital video production versioning","fromName":"Christian MICHON","fromEmail":"christian.michon@gmail.com","sentAt":"2009-01-30T10:13:54Z","receivedAt":"2009-01-30T10:13:54Z","isPatch":false,"sender":{"key":"christian.michon@gmail.com","avatar":"https://gravatar.com/avatar/8a7c327b21187fbcab5c27640a49450eec72e0355dc292501197f27a5a744ec4?d=mp&s=160"},"body":"On Thu, Jan 29, 2009 at 4:36 PM, Charles Earl wrote:\n> Hi,\n> Are there past instances of git having been adapted to support version\n> control of digital media production workflow?\n> I'm evaluating CMS and versioning systems for the backend of a SaaS\n> for digital media production workflow.\n> The bulk of content stored is binary data -- there have been posts on\n> this about integration of various binary diff implemetations with git.\n> The versioning of metadata, scripts, project structure seems to argue\n> for applicability of system such as git -- these fit the paradigm of\n> traditional scm.\n> Example content is from media production suites such as Adobe After\n> Effects/Premier: video, compositions, etc.\n> I'd also like the object storage to be in S3/Amazon BlockStore or\n> similar remote stores.\n> Charles\n> --\n\nfor what it's worth (call me crazy, call me sick), I used msysgit only\non the project files from Windows Movie Maker (not on the media files,\nbut the timeline and my personal cuts, etc...).\n\nit works perfectly, so (whatever system you're using) my advice would\nbe identify the small files and formats saved by your video editing\nsoftware (I see xml files are usually used there), only add these in\nthe git repository and make sure the video inputs and outputs files\nare in .gitignore\n\ndoing so, you have a fast and cheap tracker of your timeline edition,\nsplitting and joining of video files.\n\nHIH\n\n-- \nChristian\n--\nhttp://detaolb.sourceforge.net/, a linux distribution for Qemu with Git inside !\n"}]}