Can git support codebases on the order of magnitude of Gigabytes? For example, a game developer might want to use git as their SCM software but their codebases can be in gigabytes in scale. If git cannot support this how would one begin to create this support?
-- _________________________________ Thank You, Brian Herman
Andreas EricssonRe: Git and Very Large Codebases
On 04/20/2010 11:00 AM, Brian Herman wrote:
> Can git support codebases on the order of magnitude of Gigabytes? > For example, a game developer might want to use git as their SCM > software but their codebases can be in gigabytes in scale. > If git cannot support this how would one begin to create this support? >
It depends more on the nature of the files than the size of the total code. With large binary files and such, git will not perform very well. On a code-tree of 1-30KiB source-code files it will perform fantastically (compared to alternatives, that is; Nothing in the entire world will handle multiple gigabytes faster than it can read it from the disk).
I'd suggest splitting the repository in several different ones to make git handle it well. Movies go in one repository, audio go in another, pictures and whatnot in a third, and then you tie it all together with a super-repo that holds the code and the build-system.
In repositories storing mainly binary files you go through the gitattributes(5) man-page and set the necessary attributes for the files in question. In your case, I'd suspect the .gitattributes file would look something like this:
*.avi -diff -delta -crlf -merge *.jpg -diff -delta -crlf -merge *.mp3 -diff -delta -crlf -merge
This will prevent git from attempting delta-compression on the files. It will also not try to create diffs from them, or perform crlf-conversion or merge them. The gitattributes(5) man page has more information regarding this.
Some trial and error with a small test-repository will help you sort this out and understand what it means.
-- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 Considering the successes of the wars on alcohol, poverty, drugs and terror, I think we should give some serious thought to declaring war on peace.
Jakub NarebskiRe: Git and Very Large Codebases
Brian Herman <brianherman@gmail.com> writes:
> Can git support codebases on the order of magnitude of Gigabytes? > For example, a game developer might want to use git as their SCM > software but their codebases can be in gigabytes in scale. > If git cannot support this how would one begin to create this support?
Take a look at git-bigfiles project.
-- Jakub Narebski Poland ShadeHawk on #git