{"thread":{"id":"25495","subject":"GIT as binary repository","startedAt":"2010-10-21T12:52:46Z","lastAt":"2010-10-22T19:20:17Z","messageCount":9,"participants":["Wilson, Kevin Lee (OpenView Engineer)","Tay Ray Chuan","Shawn Pearce","Enrico Weigelt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"153956","messageId":"2EBB46ACFCD6CF48B45E2B1865BD02338037D36B4E@GVW1098EXB.americas.hpqcorp.net","threadId":"25495","inReplyTo":null,"subject":"GIT as binary repository","fromName":"Wilson, Kevin Lee (OpenView Engineer)","fromEmail":"kevin.l.wilson@hp.com","sentAt":"2010-10-21T12:52:46Z","receivedAt":"2010-10-21T12:52:46Z","isPatch":false,"sender":{"key":"kevin.l.wilson@hp.com","avatar":null},"body":"Hello, \n\nWe are investigating the use of GIT as a binary repository solution. Our larger files are near 800MB and the total checked out repo size is about 3 GB the repo size in SVN is more like 20-30GB, if we could prune the history prior to MR, we could get these sizes down considerably. This binary repo is really for our super project build.  From what I have read and learned, this is not a good fit for the GIT tool. Have there been performance improvements lately? Some of the posts I have read have been quite old?\n\nI also have some questions, about how the workflow would be for getting all of the changes merged from several different teams into the one repository would operate. Do we setup a shared system for engineers to perform the merges onto? Our teams are geographically disbursed. \n\nThanks for any light you can shed on this. We are trying to digest a lot of information quickly, so sorry if there are things covered here that are elsewhere.\n\nThanks,\nKevin\n"},{"id":"153965","messageId":"AANLkTim6qsUDf5LS7C9o+C92Aqh7CohLLJ=+13BE3Tze@mail.gmail.com","threadId":"25495","inReplyTo":"2EBB46ACFCD6CF48B45E2B1865BD02338037D36B4E@GVW1098EXB.americas.hpqcorp.net","subject":"Re: GIT as binary repository","fromName":"Tay Ray Chuan","fromEmail":"rctay89@gmail.com","sentAt":"2010-10-21T14:19:51Z","receivedAt":"2010-10-21T14:19:51Z","isPatch":false,"sender":{"key":"rctay89@gmail.com","avatar":"https://avatars.githubusercontent.com/u/61553?v=4"},"body":"Hi,\n\nOn Thu, Oct 21, 2010 at 8:52 PM, Wilson, Kevin Lee (OpenView Engineer)\n wrote:\n> We are investigating the use of GIT as a binary repository solution. Our larger files are near 800MB and the total checked out repo size is about 3 GB the repo size in SVN is more like 20-30GB, if we could prune the history prior to MR, we could get these sizes down considerably. This binary repo is really for our super project build.  From what I have read and learned, this is not a good fit for the GIT tool. Have there been performance improvements lately? Some of the posts I have read have been quite old?\n\ncheck this out:\n\n http://github.com/apenwarr/bup\n\nIt's a modified git system that's purpose-built for large files.\nThat's just about all the sensible information I can share you with\nyou.\n\n-- \nCheers,\nRay Chuan\n"},{"id":"153984","messageId":"AANLkTiniQNTBnqQz-p5E3rjoHq9FnpMkzO=B0yh+Sbxq@mail.gmail.com","threadId":"25495","inReplyTo":"2EBB46ACFCD6CF48B45E2B1865BD02338037D36B4E@GVW1098EXB.americas.hpqcorp.net","subject":"Re: GIT as binary repository","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-21T17:38:50Z","receivedAt":"2010-10-21T17:38:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Thu, Oct 21, 2010 at 5:52 AM, Wilson, Kevin Lee (OpenView Engineer)\n wrote:\n> We are investigating the use of GIT as a binary repository solution. Our larger files are near 800MB and the total checked out repo size is about 3 GB the repo size in SVN is more like 20-30GB, if we could prune the history prior to MR, we could get these sizes down considerably. This binary repo is really for our super project build.  From what I have read and learned, this is not a good fit for the GIT tool. Have there been performance improvements lately? Some of the posts I have read have been quite old?\n\nNot really.\n\nTeams who need to store content like this are taking two approaches:\n\nBite the bullet and use 64 bit systems with a lot of physical memory.\nGit allocates/memmaps two memory blocks equal in size to the file you\nare trying to work with. If you have an 800MB file, you need ~1.6G of\nphysical memory just for the Git executable to touch that file. For\nmost modern desktops and server systems, this is pretty easy to deal\nwith, 4G or 8G of physical memory in a developer workstation is pretty\ninexpensive. If the files aren't delta compressible, you can speed up\ndelta compressing operations that occur during `git gc` by adding a\ngitattribute with the \"-delta\" flag for the relevant path files to\nyour .git/info/attributes file. Unfortunately this may mean that your\nGit repository is large (>20G?), and each developer needs to make a\nfull copy of it when they start to work on the project. That is a lot\nof data to move around, or to store locally. But again when you look\nat the cost of disk on a developer workstation, this may not be an\nissue if your team can adopt a workflow where they don't clone the\nrepository often. (E.g. the Android repository is about 7G,\ndevelopers clone it once and then don't need to again... so it is\ndoable.)\n\nThe other option is to use a different repository for the binary\nfiles. Some teams are using a REST enabled HTTP server like Amazon S3\n(though you probably want something inside your corporate firewall) to\nstore the large binary files. Instead of putting the files into Git\nthe put a small shell script and a pointer to the file into Git. The\nshell script downloads the large binary file when executed, and the\nbuild process (or the developer \"start-up\" instructions) execute the\nscript to get the latest versions bootstrapped on the local\nworkstation.\n\n\n> I also have some questions, about how the workflow would be for getting all of the changes merged from several different teams into the one repository would operate. Do we setup a shared system for engineers to perform the merges onto? Our teams are geographically disbursed.\n\nYes, this is the common approach. Actually what I have started to see\nwith Android is, each distributed office has a shared repository that\nthe engineers in that office interact with on a daily basis. And\nsomeone in each office synchronizes that repository with a single\ncentral repository that exists somewhere else. Because of the nature\nof Git, the central repository can be continuously pulled into the\ndistributed office through a cron script. Engineers in the office can\ntherefore always have a \"fairly latest\" version available, but can\nalso fork off onto a side-branch and defer merging with the other\noffices for a day or two.\n\nAndroid teams are successfully using this approach by running Gerrit\nCode Review[1] as their central server, and using Gerrit's built-in\nreplication feature to push updates to the distributed office servers.\n In effect there is one central server for writes, but a lot of read\noperations are offloaded into the distributed offices local copies.\n\n[1] http://code.google.com/p/gerrit/\n\n-- \nShawn.\n"},{"id":"153995","messageId":"2EBB46ACFCD6CF48B45E2B1865BD02338037D36F2A@GVW1098EXB.americas.hpqcorp.net","threadId":"25495","inReplyTo":"AANLkTiniQNTBnqQz-p5E3rjoHq9FnpMkzO=B0yh+Sbxq@mail.gmail.com","subject":"RE: GIT as binary repository","fromName":"Wilson, Kevin Lee (OpenView Engineer)","fromEmail":"kevin.l.wilson@hp.com","sentAt":"2010-10-21T18:53:18Z","receivedAt":"2010-10-21T18:53:18Z","isPatch":false,"sender":{"key":"kevin.l.wilson@hp.com","avatar":null},"body":"Thanks for the detailed input. \nKevin\n\n-----Original Message-----\nFrom: Shawn Pearce [mailto:spearce@spearce.org] \nSent: Thursday, October 21, 2010 11:39 AM\nTo: Wilson, Kevin Lee (OpenView Engineer)\nCc: git@vger.kernel.org\nSubject: Re: GIT as binary repository\n\nOn Thu, Oct 21, 2010 at 5:52 AM, Wilson, Kevin Lee (OpenView Engineer)\n wrote:\n> We are investigating the use of GIT as a binary repository solution. Our larger files are near 800MB and the total checked out repo size is about 3 GB the repo size in SVN is more like 20-30GB, if we could prune the history prior to MR, we could get these sizes down considerably. This binary repo is really for our super project build.  From what I have read and learned, this is not a good fit for the GIT tool. Have there been performance improvements lately? Some of the posts I have read have been quite old?\n\nNot really.\n\nTeams who need to store content like this are taking two approaches:\n\nBite the bullet and use 64 bit systems with a lot of physical memory.\nGit allocates/memmaps two memory blocks equal in size to the file you\nare trying to work with. If you have an 800MB file, you need ~1.6G of\nphysical memory just for the Git executable to touch that file. For\nmost modern desktops and server systems, this is pretty easy to deal\nwith, 4G or 8G of physical memory in a developer workstation is pretty\ninexpensive. If the files aren't delta compressible, you can speed up\ndelta compressing operations that occur during `git gc` by adding a\ngitattribute with the \"-delta\" flag for the relevant path files to\nyour .git/info/attributes file. Unfortunately this may mean that your\nGit repository is large (>20G?), and each developer needs to make a\nfull copy of it when they start to work on the project. That is a lot\nof data to move around, or to store locally. But again when you look\nat the cost of disk on a developer workstation, this may not be an\nissue if your team can adopt a workflow where they don't clone the\nrepository often. (E.g. the Android repository is about 7G,\ndevelopers clone it once and then don't need to again... so it is\ndoable.)\n\nThe other option is to use a different repository for the binary\nfiles. Some teams are using a REST enabled HTTP server like Amazon S3\n(though you probably want something inside your corporate firewall) to\nstore the large binary files. Instead of putting the files into Git\nthe put a small shell script and a pointer to the file into Git. The\nshell script downloads the large binary file when executed, and the\nbuild process (or the developer \"start-up\" instructions) execute the\nscript to get the latest versions bootstrapped on the local\nworkstation.\n\n\n> I also have some questions, about how the workflow would be for getting all of the changes merged from several different teams into the one repository would operate. Do we setup a shared system for engineers to perform the merges onto? Our teams are geographically disbursed.\n\nYes, this is the common approach. Actually what I have started to see\nwith Android is, each distributed office has a shared repository that\nthe engineers in that office interact with on a daily basis. And\nsomeone in each office synchronizes that repository with a single\ncentral repository that exists somewhere else. Because of the nature\nof Git, the central repository can be continuously pulled into the\ndistributed office through a cron script. Engineers in the office can\ntherefore always have a \"fairly latest\" version available, but can\nalso fork off onto a side-branch and defer merging with the other\noffices for a day or two.\n\nAndroid teams are successfully using this approach by running Gerrit\nCode Review[1] as their central server, and using Gerrit's built-in\nreplication feature to push updates to the distributed office servers.\n In effect there is one central server for writes, but a lot of read\noperations are offloaded into the distributed offices local copies.\n\n[1] http://code.google.com/p/gerrit/\n\n-- \nShawn.\n"},{"id":"154003","messageId":"20101021191925.GE28700@nibiru.local","threadId":"25495","inReplyTo":"2EBB46ACFCD6CF48B45E2B1865BD02338037D36B4E@GVW1098EXB.americas.hpqcorp.net","subject":"Re: GIT as binary repository","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-21T19:19:25Z","receivedAt":"2010-10-21T19:19:25Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Wilson, Kevin Lee (OpenView Engineer) wrote:\n\nHi,\n\n> We are investigating the use of GIT as a binary repository solution.\n> Our larger files are near 800MB and the total checked out repo size\n> is about 3 GB the repo size in SVN is more like 20-30GB, if we could\n> prune the history prior to MR, we could get these sizes down\n> considerably. This binary repo is really for our super project build. \n\nWhat exactly do you need such large binary objects in an git repo ?\n\nIMHO, Git isn't made for large files. I've noticed this when doing\ngit-based mail archives on an old P3 box w/ 256MB physical memory.\nI had to split mbox'es to maildirs.\n\nPerhaps you would like to have a look at some pure object store like\nventi ? (It's not distributed yet, but I'm currently working on an\ndistributed successor, called Nebulon, which will also support\nstrong encryption, on-demand replication, etc).\n\n> I also have some questions, about how the workflow would be for\n> getting all of the changes merged from several different teams\n> into the one repository would operate. \n\nIMHO, there should be some dedicated release manager role, which\nis responsible for merging finished branches into the mainline\n(eg. similar that Linus does for the official Linux tree).\n\nBUT: you perhaps should think carefully, whether you need everything\nin one big repo. Perhaps a bunch of smaller ones (eg. having separate\nmodules in the own repos/trees) would fit better.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"154013","messageId":"20101021195416.GF28700@nibiru.local","threadId":"25495","inReplyTo":"AANLkTim6qsUDf5LS7C9o+C92Aqh7CohLLJ=+13BE3Tze@mail.gmail.com","subject":"Re: GIT as binary repository","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-21T19:54:16Z","receivedAt":"2010-10-21T19:54:16Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Tay Ray Chuan wrote:\n> Hi,\n> \n> On Thu, Oct 21, 2010 at 8:52 PM, Wilson, Kevin Lee (OpenView Engineer)\n> wrote:\n> > We are investigating the use of GIT as a binary repository solution. Our larger files are near 800MB and the total checked out repo size is about 3 GB the repo size in SVN is more like 20-30GB, if we could prune the history prior to MR, we could get these sizes down considerably. This binary repo is really for our super project build.  From what I have read and learned, this is not a good fit for the GIT tool. Have there been performance improvements lately? Some of the posts I have read have been quite old?\n> \n> check this out:\n> \n> http://github.com/apenwarr/bup\n> \n> It's a modified git system that's purpose-built for large files.\n> That's just about all the sensible information I can share you with\n> you.\n\nLooks quite promising, perhaps it can help solving my current\nbackup problems.\n\nMaybe we could implement some of the features, eg. the hashsplit\nformat (maybe it could even combined w/ xdelta ?) or the bupindex\nand some extended metadata directly in git ?\n\n\nBTW: how are the current tree objects structured ? Is there always\none big tree object representing the whole tree or can it be\nsplitted hierachically ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"154031","messageId":"AANLkTikk5FA-3LQNfjj6rq0ouO7CywZ9_FaaZCGRM_YB@mail.gmail.com","threadId":"25495","inReplyTo":"20101021195416.GF28700@nibiru.local","subject":"Re: GIT as binary repository","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-21T21:23:53Z","receivedAt":"2010-10-21T21:23:53Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Thu, Oct 21, 2010 at 12:54 PM, Enrico Weigelt wrote:\n> BTW: how are the current tree objects structured ? Is there always\n> one big tree object representing the whole tree or can it be\n> splitted hierachically ?\n\nTrees in the repository are split hierarchically, but its a flat list\nin the dircache/index file (aka $GIT_DIR/index).\n\n-- \nShawn.\n"},{"id":"154101","messageId":"20101022050257.GB27179@nibiru.local","threadId":"25495","inReplyTo":"AANLkTikk5FA-3LQNfjj6rq0ouO7CywZ9_FaaZCGRM_YB@mail.gmail.com","subject":"Re: GIT as binary repository","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-10-22T05:02:58Z","receivedAt":"2010-10-22T05:02:58Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Shawn Pearce wrote:\n> On Thu, Oct 21, 2010 at 12:54 PM, Enrico Weigelt wrote:\n> > BTW: how are the current tree objects structured ? Is there always\n> > one big tree object representing the whole tree or can it be\n> > splitted hierachically ?\n> \n> Trees in the repository are split hierarchically, but its a flat list\n> in the dircache/index file (aka $GIT_DIR/index).\n\nGood. That should allow optimizations for large trees w/o\nhaving to change object formats.\n\nWhat about splitted blobs ? Is this supported by the object\nformat right now ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"154194","messageId":"AANLkTikwWQgnmOK2o8L+vUazHeHR-AqSrcZ4HK4siOA0@mail.gmail.com","threadId":"25495","inReplyTo":"20101022050257.GB27179@nibiru.local","subject":"Re: GIT as binary repository","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-10-22T19:20:17Z","receivedAt":"2010-10-22T19:20:17Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Thu, Oct 21, 2010 at 10:02 PM, Enrico Weigelt wrote:\n> * Shawn Pearce wrote:\n> What about splitted blobs ? Is this supported by the object\n> format right now ?\n\nNope.\n\n-- \nShawn.\n"}]}