{"thread":{"id":"23533","subject":"Git and Very Large Codebases","startedAt":"2010-04-20T09:00:40Z","lastAt":"2010-04-20T09:46:47Z","messageCount":3,"participants":["Brian Herman","Andreas Ericsson","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"139958","messageId":"h2w4ef15ddc1004200200ocd9df12azf62da7419355ae7c@mail.gmail.com","threadId":"23533","inReplyTo":"s2x4ef15ddc1004200158s2fc6391byfab91784036ac018@mail.gmail.com","subject":"Git and Very Large Codebases","fromName":"Brian Herman","fromEmail":"brianherman@gmail.com","sentAt":"2010-04-20T09:00:40Z","receivedAt":"2010-04-20T09:00:40Z","isPatch":false,"sender":{"key":"brianherman@gmail.com","avatar":null},"body":"Can git support codebases on the order of magnitude of Gigabytes?\nFor example, a game developer might want to use git as their SCM\nsoftware but their codebases can be in gigabytes in scale.\nIf git cannot support this how would one begin to create this support?\n\n--\n_________________________________\nThank You,\nBrian Herman\n"},{"id":"139960","messageId":"4BCD7737.1010709@op5.se","threadId":"23533","inReplyTo":"h2w4ef15ddc1004200200ocd9df12azf62da7419355ae7c@mail.gmail.com","subject":"Re: Git and Very Large Codebases","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2010-04-20T09:43:19Z","receivedAt":"2010-04-20T09:43:19Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"On 04/20/2010 11:00 AM, Brian Herman wrote:\n> Can git support codebases on the order of magnitude of Gigabytes?\n> For example, a game developer might want to use git as their SCM\n> software but their codebases can be in gigabytes in scale.\n> If git cannot support this how would one begin to create this support?\n> \n\nIt depends more on the nature of the files than the size of the total\ncode. With large binary files and such, git will not perform very well.\nOn a code-tree of 1-30KiB source-code files it will perform\nfantastically (compared to alternatives, that is; Nothing in the entire\nworld will handle multiple gigabytes faster than it can read it from\nthe disk).\n\nI'd suggest splitting the repository in several different ones to\nmake git handle it well. Movies go in one repository, audio go in\nanother, pictures and whatnot in a third, and then you tie it all\ntogether with a super-repo that holds the code and the build-system.\n\nIn repositories storing mainly binary files you go through the\ngitattributes(5) man-page and set the necessary attributes for\nthe files in question. In your case, I'd suspect the .gitattributes\nfile would look something like this:\n\n*.avi -diff -delta -crlf -merge\n*.jpg -diff -delta -crlf -merge\n*.mp3 -diff -delta -crlf -merge\n\nThis will prevent git from attempting delta-compression on the\nfiles. It will also not try to create diffs from them, or perform\ncrlf-conversion or merge them. The gitattributes(5) man page has\nmore information regarding this.\n\nSome trial and error with a small test-repository will help you\nsort this out and understand what it means.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"139961","messageId":"m3eiiae9t1.fsf@localhost.localdomain","threadId":"23533","inReplyTo":"h2w4ef15ddc1004200200ocd9df12azf62da7419355ae7c@mail.gmail.com","subject":"Re: Git and Very Large Codebases","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2010-04-20T09:46:47Z","receivedAt":"2010-04-20T09:46:47Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Brian Herman <brianherman@gmail.com> writes:\n\n> Can git support codebases on the order of magnitude of Gigabytes?\n> For example, a game developer might want to use git as their SCM\n> software but their codebases can be in gigabytes in scale.\n> If git cannot support this how would one begin to create this support?\n\nTake a look at git-bigfiles project.\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"}]}