# vcs for hefty video and graphics files 7 messages from 2010-11-22 to 2010-11-25. Participants: Harry Putnam, Philippe Lhoste, Jonathan Nieder, Jakub Narebski, Stefan Monnier. Thread: https://gitlist.dev/t/25810 ## Harry Putnam, 2010-11-22 18:09 Subject: vcs for hefty video and graphics files Message-ID: <877hg55iyd.fsf@newsguy.com> URL: https://gitlist.dev/e/877hg55iyd.fsf%40newsguy.com ``` I hope not to appear to be sneaking in a war of systems, and may rate a bit a high in windbag factor since I've asked a very similar question here but quite a while ago I'd like to briefly describe my projected usage of a versioning system and see what people here think in terms what system best suits to that usage. I'm a light weight semi-professional videographer but I have enough going on to feel a need to keep track of and be able to rollback versions of a project as it is being worked on It would involve on any one projects something like 15 to 60 GB of stuff to keep up with. Large numbers of images and a dozen or 2 dozen video files. All in some stake of compression depending on the codex. I'd like to keep a few version of say an Adobe Premier project with all the attendant files that play a role in it. Ditto for sets of Adobe After effects and attendant files. And many many images and compilation in various states of cutting or whatever editing. And then the whole project including the various Premier and After effects sets and piles of captured video tape. All in DV-avi format. So versions inside versions and then repeated for different projects. Each of which may have more than 1 version. But cutting to the chase... assorted video and graphics files amounting to between 15 - 60 GB in any one project Each project would only run a month or 2 months at the most and then all but the final delivered version would be deleted. That version might be keep for a yr or so. I have some experience with cvs over quite a few yrs but still only a homeboy user and not very skilled or knowledgeable about cvs and even less with mercurial or git, where I do have some tiny bit of experience too. Which of the main contenders: cvs subversion mercurial git bizarre Maybe a few more I don't know about, would be the best candidate for the usage and user described ``` ## Philippe Lhoste, 2010-11-23 10:37 Subject: Re: vcs for hefty video and graphics files Message-ID: URL: https://gitlist.dev/e/icg5ia%245an%241%40dough.gmane.org In-Reply-To: <877hg55iyd.fsf@newsguy.com> ``` On 22/11/2010 19:09, Harry Putnam wrote: > Which of the main contenders: cvs subversion mercurial git bizarre > Maybe a few more I don't know about, would be the best candidate for > the usage and user described bizarre? Never heard of this VCS before... > Each project would only run a month or 2 months at the most and then > all but the final delivered version would be deleted. That version > might be keep for a yr or so. Maybe it is pure heresy, but since all you want is to keep temporarily several very big and nearly incompressible files where diffs (or deltas?) are probably not significant, I would advance that a VCS won't be very useful here. Advantages of VCS are (among others): - make delta of changes to keep as little data as possible - compress this data (?) - keep changes indefinitely to be sure to have them when we need them - share and merge (changes from somebody else, or you elsewhere) Unless I missed something, these advantages doesn't seem to apply there. Some game makers keep track of their (large) binary files, along with the rest of the project (source code). Rarely in isolation. Perforce and PlasticSCM both boast superior support of these files, I won't comment on these allegations (over other VCS), just having no experience here. Somehow, in your case, the good old way of keeping copies renamed to keep the version (or kept in specific directories) might work for you... Perhaps along with a small text file with comments on content of each file. PS.: I don't see why you included Tomcat list... -- Philippe Lhoste -- (near) Paris -- France -- http://Phi.Lho.free.fr -- -- -- -- -- -- -- -- -- -- -- -- -- -- ``` ## Harry Putnam, 2010-11-23 16:19 Subject: Re: vcs for hefty video and graphics files Message-ID: <87oc9g2eti.fsf@newsguy.com> URL: https://gitlist.dev/e/87oc9g2eti.fsf%40newsguy.com In-Reply-To: ``` Philippe Lhoste writes: > Somehow, in your case, the good old way of keeping copies renamed to > keep the version (or kept in specific directories) might work for > you... Perhaps along with a small text file with comments on content > of each file. Thanks for the input... that method has sort of worked so far but somewhat tedious > PS.: I don't see why you included Tomcat list... All I can say about that is that it was NOT my doing. I only included bazaar cvs mercurial, git and subversion. Its a mystery to me too. I hadn't noticed until you mentioned it. I removed it from this reply. A copy of my OP is kept on my home machine. It shows exactly what I sent out, I'm not sure what it goes thru when it gets to gmane server: From: Harry Putnam Subject: vcs for hefty video and graphics files Newsgroups: gmane.comp.version-control.mercurial.general, gmane.comp.version-control.bazaar-ng.general, gmane.comp.version-control.git, gmane.comp.version-control.subversion.user, gmane.comp.version-control.cvs.general Date: Mon, 22 Nov 2010 12:09:14 -0600 Message-ID: <877hg55iyd.fsf@newsguy.com> ``` ## Jonathan Nieder, 2010-11-23 20:41 Subject: Re: vcs for hefty video and graphics files Message-ID: <20101123204149.GA2373@burratino> URL: https://gitlist.dev/e/20101123204149.GA2373%40burratino In-Reply-To: ``` Philippe Lhoste wrote: > Some game makers keep track of their (large) binary files, along > with the rest of the project (source code). Rarely in isolation. > Perforce and PlasticSCM both boast superior support of these files, > I won't comment on these allegations (over other VCS), just having > no experience here. One small thing to add: for this use case, you might like git-annex[1]. It is a shame no single mailing list to follow up to was mentioned. Hopefully the reply will get to Harry, anyway. Regards, Jonathan [1] http://git-annex.branchable.com/ ``` ## Jakub Narebski, 2010-11-24 22:42 Subject: Re: vcs for hefty video and graphics files Message-ID: URL: https://gitlist.dev/e/m3r5ea4a5y.fsf%40localhost.localdomain In-Reply-To: <20101123204149.GA2373@burratino> ``` Jonathan Nieder writes: > Philippe Lhoste wrote: > > > Some game makers keep track of their (large) binary files, along > > with the rest of the project (source code). Rarely in isolation. > > Perforce and PlasticSCM both boast superior support of these files, > > I won't comment on these allegations (over other VCS), just having > > no experience here. > > One small thing to add: for this use case, you might like git-annex[1]. > > It is a shame no single mailing list to follow up to was mentioned. > Hopefully the reply will get to Harry, anyway. > > Regards, > Jonathan > > [1] http://git-annex.branchable.com/ Jonathan, could you add git-annex to "Interfaces, frontends, and tools" page on git wiki: https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools Does git-annex use git-replace? -- Jakub Narebski Poland ShadeHawk on #git ``` ## Jonathan Nieder, 2010-11-25 02:21 Subject: Re: vcs for hefty video and graphics files Message-ID: <20101125022106.GB13274@burratino> URL: https://gitlist.dev/e/20101125022106.GB13274%40burratino In-Reply-To: ``` Jakub Narebski wrote: > Jonathan, could you add git-annex to "Interfaces, frontends, and tools" > page on git wiki: > https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools Ok, done. Please feel free to improve the description. > Does git-annex use git-replace? No, just plain symlinks. See http://git-annex.branchable.com/walkthrough/ to get a feel for it. Regards, Jonathan ``` ## Stefan Monnier, 2010-11-25 17:34 Subject: Re: vcs for hefty video and graphics files Message-ID: URL: https://gitlist.dev/e/jwvr5e91fj2.fsf-monnier%2Bgmane.comp.version-control.bazaar-ng.general%40gnu.org In-Reply-To: <877hg55iyd.fsf@newsguy.com> ``` > It would involve on any one projects something like 15 to 60 GB of > stuff to keep up with. Large numbers of images and a dozen or 2 dozen > video files. All in some stake of compression depending on the codex. FWIW, for a VCS to do a good job on this kind of problem, you'd need to use a representation that lends itself to it. I.e. regardless of what you end up doing, I would recommend you contact the mailing-list of Free Software that can do the kind of video manipulation you want to do, and tell them that you'd need their tool to represent a project in such a way that it has a bunch of big-files that are almost never modified (containing "binary data" such as the source images and audio recordings, say) along with a few other smaller files that mostly contain instructions about how to use the big-files to generate the desired output. Such a representation should lead to very good support from most VCSs. E.g. If these small files use a clean text representation, then a VCS might even be able to do useful merges between different branches of a project (as long as the branches share the same big-files). Stefan ```