{"thread":{"id":"27007","subject":"blobs (once more)","startedAt":"2011-04-06T08:09:18Z","lastAt":"2011-04-07T06:45:56Z","messageCount":9,"participants":["Pau Garcia i Quiles","Johannes Schindelin","Matthieu Moy","Peter Jönsson P","Michael J Gruber","Martin Langhoff","Magnus Bäck","Miles Bader"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165250","messageId":"BANLkTim3kg1ycGkgWsqaZiqMY9LTKV6DBw@mail.gmail.com","threadId":"27007","inReplyTo":null,"subject":"blobs (once more)","fromName":"Pau Garcia i Quiles","fromEmail":"pgquiles@elpauer.org","sentAt":"2011-04-06T08:09:18Z","receivedAt":"2011-04-06T08:09:18Z","isPatch":false,"sender":{"key":"pgquiles@elpauer.org","avatar":null},"body":"Hello,\n\nBinary large objects. I know it has been discussed once and again but\nI'd like to know if there is something new.\n\nSome corporation hired the company I work for one year ago to develop\na large application. They imposed ClearCase as the VCS. I don't know\nif you have used it but it is a pain in the ass. We have lost weeks of\ndevelopment to site-replication problems, funny merges, etc. We are\ntrying to migrate our project to git, which we have experience with.\n\nOne very important point in this project (which is Windows only) is\nputting binaries in the repository. So far, we have suceeded in not\ndoing that in other projects but we will need to do that in this\nproject.\n\nIn the Windows world, it is not unusual to use third-party libraries\nwhich are only available in binary form. Getting them as source is not\nan option because the companies developing them are not selling the\nsource. Moving from those binary-only dependencies to something else\nis not an option either because what we are using has some unique\nfeatures, be it technical features or support features. In our\nproject, we have about a dozen such binaries, ranging from a few\nhundred kilobytes, to a couple hundred megabytes (proprietary database\nand virtualization engine).\n\nThe usual answer to the \"I need to put binaries in the repository\"\nquestion has been \"no, you do not\". Well, we do. We are in heavy\ndevelopment now, therefore today's version may depend on a certain\nversion of a third-party shared library (DLL) which we only can get in\nbinary form, and tomorrow's version may depend on the next version of\nthat library, and you cannot mix today's source with yesterday's\nthird-party DLL. I. e. to be able to use the code from 7 days ago at\n11.07 AM you need \"git checkout\" to \"return\" our source AND the\nbinaries we were using back then. This is something ClearCase manages\nsatisfactorily.\n\nI have read about:\n- submodules + using different repositories once one \"blob repository\"\ngrows too much. This will be probably rejected because it is quite\ncontrived.\n- git-annex (does not get the files in when cloning, pulling, checking\nout; you need to do it manually)\n- git-media (same as git-annex)\n- boar (no, we do not want to use a VCS for binaries in addition to git)\n- and a few more\n\nSo far the only good solution seems to be git-bigfiles but it's still\nin development.\n\nIs there any good solution for my use case, where version = sources\nversion + binaries version?\n\nThank you.\n\nIf we suceed with git here, the whole corportation (150,000+\nemployees, Fortune 500) may start to move to git in a year. Many\npeople are fed up with CC there.\n\n-- \nPau Garcia i Quiles\nhttp://www.elpauer.org\n(Due to my workload, I may need 10 days to answer)\n"},{"id":"165254","messageId":"alpine.DEB.1.00.1104061121000.2040@bonsai2","threadId":"27007","inReplyTo":"BANLkTim3kg1ycGkgWsqaZiqMY9LTKV6DBw@mail.gmail.com","subject":"Re: blobs (once more)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2011-04-06T09:25:57Z","receivedAt":"2011-04-06T09:25:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 6 Apr 2011, Pau Garcia i Quiles wrote:\n\n> Binary large objects. I know it has been discussed once and again but \n> I'd like to know if there is something new.\n> \n> Some corporation hired the company I work for one year ago to develop a \n> large application. They imposed ClearCase as the VCS. I don't know if \n> you have used it but it is a pain in the ass. We have lost weeks of \n> development to site-replication problems, funny merges, etc. We are \n> trying to migrate our project to git, which we have experience with.\n> \n> One very important point in this project (which is Windows only) is \n> putting binaries in the repository. So far, we have suceeded in not \n> doing that in other projects but we will need to do that in this \n> project.\n> \n> In the Windows world, it is not unusual to use third-party libraries \n> which are only available in binary form. Getting them as source is not \n> an option because the companies developing them are not selling the \n> source. Moving from those binary-only dependencies to something else is \n> not an option either because what we are using has some unique features, \n> be it technical features or support features. In our project, we have \n> about a dozen such binaries, ranging from a few hundred kilobytes, to a \n> couple hundred megabytes (proprietary database and virtualization \n> engine).\n> \n> The usual answer to the \"I need to put binaries in the repository\" \n> question has been \"no, you do not\". Well, we do. We are in heavy \n> development now, therefore today's version may depend on a certain \n> version of a third-party shared library (DLL) which we only can get in \n> binary form, and tomorrow's version may depend on the next version of \n> that library, and you cannot mix today's source with yesterday's \n> third-party DLL. I. e. to be able to use the code from 7 days ago at \n> 11.07 AM you need \"git checkout\" to \"return\" our source AND the binaries \n> we were using back then. This is something ClearCase manages \n> satisfactorily.\n\nI understand. The problem in your case might not be too bad, after all. \nThe problem only arises when you have big files that are compressed. If \nyou check in multiple versions of an uncompressed .dll file, Git will \nusually do a very good job at compressing them.\n\nIf they are compressed, what you probably need is something like a sparse \nclone, which is sort of available in the form of shallow clones, but it is \ntoo limited still.\n\nHaving said that, in another company I work for, they hav 20G repositories \nand they will grow larger. That is something they incurred due to \nhistorical reasons, and they are willing to pay the price in terms of disk \nspace. Due to Git's distributed nature, they had no problems with cloning; \nthey just use a local reference upon initial clone.\n\n> I have read about:\n> - submodules + using different repositories once one \"blob repository\"  \n>   grows too much. This will be probably rejected because it is quite \n>   contrived.\n\nI would also recommend against this, because submodules are a very weak \npart of Git.\n\n> - git-annex (does not get the files in when cloning, pulling, checking \n>   out; you need to do it manually)\n> - git-media (same as git-annex)\n\nYes, this is an option, but a bit klunky.\n\n> - boar (no, we do not want to use a VCS for binaries in addition to git)\n\nI did not know about that.\n\n> - and a few more\n> \n> So far the only good solution seems to be git-bigfiles but it's still\n> in development.\n\nIt has stalled, apparently, but I wanted to have a look at it anyway. Will \nlet you know of my findings!\n\n> Is there any good solution for my use case, where version = sources \n> version + binaries version?\n> \n> Thank you.\n> \n> If we suceed with git here, the whole corportation (150,000+\n> employees, Fortune 500) may start to move to git in a year. Many\n> people are fed up with CC there.\n\nCiao,\nJohannes\n"},{"id":"165261","messageId":"vpqmxk3prki.fsf@bauges.imag.fr","threadId":"27007","inReplyTo":"BANLkTim3kg1ycGkgWsqaZiqMY9LTKV6DBw@mail.gmail.com","subject":"Re: blobs (once more)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-04-06T11:06:21Z","receivedAt":"2011-04-06T11:06:21Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Pau Garcia i Quiles <pgquiles@elpauer.org> writes:\n\n> I have read about:\n> - submodules + using different repositories once one \"blob repository\"\n> grows too much. This will be probably rejected because it is quite\n> contrived.\n> - git-annex (does not get the files in when cloning, pulling, checking\n> out; you need to do it manually)\n> - git-media (same as git-annex)\n> - boar (no, we do not want to use a VCS for binaries in addition to git)\n> - and a few more\n>\n> So far the only good solution seems to be git-bigfiles but it's still\n> in development.\n\nThis seems to be the hot topic of the moment in Git ;-). Loot at the\nmailing-list's archive, like\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/170649\nhttps://git.wiki.kernel.org/index.php/SoC2011Ideas#Better_big-file_support\n\nthere may be a GSoC on the topic. I don't think there's a really good\nsolution for you right now (but I didn't really look closely), but the\nsituation is improving.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"165262","messageId":"41F80411E3CC644A844E6BED6E472FD91196D0CD30@ESESSCMS0363.eemea.ericsson.se","threadId":"27007","inReplyTo":"vpqmxk3prki.fsf@bauges.imag.fr","subject":"RE: blobs (once more)","fromName":"Peter Jönsson P","fromEmail":"peter.p.jonsson@ericsson.com","sentAt":"2011-04-06T11:12:41Z","receivedAt":"2011-04-06T11:12:41Z","isPatch":false,"sender":{"key":"peter.p.jonsson@ericsson.com","avatar":"https://gravatar.com/avatar/7c7bf60738b19f1590a3d39e021adadad713f84adf80a7a405667fe939ef16bb?d=mp&s=160"},"body":"Hi!\n\nHow about using Google's repo-tool instead of submodules? Is that a better solution if one really needs to keep binaries together (in some way) with the source code?\n\nWe are starting to prototype Git and since we need to distribute cross-compilers with the code it would be nice to keep it in a separate repo. Currently we are _heavy_ ClearCase users (oh the horror) with many bad old habits that we are trying to break :)\n\n// Peter \n\n-----Original Message-----\nFrom: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On Behalf Of Matthieu Moy\nSent: den 6 april 2011 13:06\nTo: Pau Garcia i Quiles\nCc: Git Mailing List; Johannes Schindelin\nSubject: Re: blobs (once more)\n\nPau Garcia i Quiles <pgquiles@elpauer.org> writes:\n\n> I have read about:\n> - submodules + using different repositories once one \"blob repository\"\n> grows too much. This will be probably rejected because it is quite \n> contrived.\n> - git-annex (does not get the files in when cloning, pulling, checking \n> out; you need to do it manually)\n> - git-media (same as git-annex)\n> - boar (no, we do not want to use a VCS for binaries in addition to \n> git)\n> - and a few more\n>\n> So far the only good solution seems to be git-bigfiles but it's still \n> in development.\n\nThis seems to be the hot topic of the moment in Git ;-). Loot at the mailing-list's archive, like\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/170649\nhttps://git.wiki.kernel.org/index.php/SoC2011Ideas#Better_big-file_support\n\nthere may be a GSoC on the topic. I don't think there's a really good solution for you right now (but I didn't really look closely), but the situation is improving.\n\n--\nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n--\nTo unsubscribe from this list: send the line \"unsubscribe git\" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"165264","messageId":"4D9C5A79.30207@drmicha.warpmail.net","threadId":"27007","inReplyTo":"alpine.DEB.1.00.1104061121000.2040@bonsai2","subject":"Re: blobs (once more)","fromName":"Michael J Gruber","fromEmail":"git@drmicha.warpmail.net","sentAt":"2011-04-06T12:20:09Z","receivedAt":"2011-04-06T12:20:09Z","isPatch":false,"sender":{"key":"git@grubix.eu","avatar":"https://avatars.githubusercontent.com/u/233215?v=4"},"body":"Johannes Schindelin venit, vidit, dixit 06.04.2011 11:25:\n> Hi,\n> \n> On Wed, 6 Apr 2011, Pau Garcia i Quiles wrote:\n> \n>> Binary large objects. I know it has been discussed once and again but \n>> I'd like to know if there is something new.\n>>\n>> Some corporation hired the company I work for one year ago to develop a \n>> large application. They imposed ClearCase as the VCS. I don't know if \n>> you have used it but it is a pain in the ass. We have lost weeks of \n>> development to site-replication problems, funny merges, etc. We are \n>> trying to migrate our project to git, which we have experience with.\n>>\n>> One very important point in this project (which is Windows only) is \n>> putting binaries in the repository. So far, we have suceeded in not \n>> doing that in other projects but we will need to do that in this \n>> project.\n>>\n>> In the Windows world, it is not unusual to use third-party libraries \n>> which are only available in binary form. Getting them as source is not \n>> an option because the companies developing them are not selling the \n>> source. Moving from those binary-only dependencies to something else is \n>> not an option either because what we are using has some unique features, \n>> be it technical features or support features. In our project, we have \n>> about a dozen such binaries, ranging from a few hundred kilobytes, to a \n>> couple hundred megabytes (proprietary database and virtualization \n>> engine).\n>>\n>> The usual answer to the \"I need to put binaries in the repository\" \n>> question has been \"no, you do not\". Well, we do. We are in heavy \n>> development now, therefore today's version may depend on a certain \n>> version of a third-party shared library (DLL) which we only can get in \n>> binary form, and tomorrow's version may depend on the next version of \n>> that library, and you cannot mix today's source with yesterday's \n>> third-party DLL. I. e. to be able to use the code from 7 days ago at \n>> 11.07 AM you need \"git checkout\" to \"return\" our source AND the binaries \n>> we were using back then. This is something ClearCase manages \n>> satisfactorily.\n> \n> I understand. The problem in your case might not be too bad, after all. \n> The problem only arises when you have big files that are compressed. If \n> you check in multiple versions of an uncompressed .dll file, Git will \n> usually do a very good job at compressing them.\n> \n> If they are compressed, what you probably need is something like a sparse \n> clone, which is sort of available in the form of shallow clones, but it is \n> too limited still.\n> \n> Having said that, in another company I work for, they hav 20G repositories \n> and they will grow larger. That is something they incurred due to \n> historical reasons, and they are willing to pay the price in terms of disk \n> space. Due to Git's distributed nature, they had no problems with cloning; \n> they just use a local reference upon initial clone.\n> \n>> I have read about:\n>> - submodules + using different repositories once one \"blob repository\"  \n>>   grows too much. This will be probably rejected because it is quite \n>>   contrived.\n> \n> I would also recommend against this, because submodules are a very weak \n> part of Git.\n> \n>> - git-annex (does not get the files in when cloning, pulling, checking \n>>   out; you need to do it manually)\n>> - git-media (same as git-annex)\n> \n> Yes, this is an option, but a bit klunky.\n> \n>> - boar (no, we do not want to use a VCS for binaries in addition to git)\n> \n> I did not know about that.\n> \n>> - and a few more\n>>\n>> So far the only good solution seems to be git-bigfiles but it's still\n>> in development.\n> \n> It has stalled, apparently, but I wanted to have a look at it anyway. Will \n> let you know of my findings!\n\nI think in many applications the \"download-on-demand\" approach which\ngit-annex takes is very important. (I don't know how far our\nsparse/shallow supports this.) Also, their remote backends look\ninteresting. And no, I don't want Haskell as yet another language for\nour code base.\n\nFedora handles big files (compressed tar balls) in git with a file\nstore, scripting (fedpkg) and tracking only a text file with hash values\n(\"sources\") in git; somehow a baby version of git-annex.\n\nThe symlink based approach of annex (big file is a symlink to the\n\"object store\" which is indexed by blob content sha1) reminds me very\nmuch of our notes trees and the way textconv-cache uses it. It feels as\nif we already have all the pieces in place. (I don't think we need to\ntrack big files' contents, only their hashes; this is fast for read-only\nmedia, see annex' worm-backend.)\n\nAnother crazy idea would be to \"git replace\" big files by place-holders\n(blob with the big file's sha1 as content) or rather the other way\nround, but I haven't thought this through.\n\nMichael\n"},{"id":"165274","messageId":"BANLkTi=03mrHzfcVhgF9woWhsWb3wenxMQ@mail.gmail.com","threadId":"27007","inReplyTo":"alpine.DEB.1.00.1104061121000.2040@bonsai2","subject":"Re: blobs (once more)","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2011-04-06T14:14:35Z","receivedAt":"2011-04-06T14:14:35Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Apr 6, 2011 at 5:25 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> I understand. The problem in your case might not be too bad, after all.\n> The problem only arises when you have big files that are compressed. If\n> you check in multiple versions of an uncompressed .dll file, Git will\n> usually do a very good job at compressing them.\n\nExcept when they are very large; in that case git tends to OOM. But\njust yesterday Junio posted a proposed patch to honour a max file size\nfor compression (search the archive for 'Git exhausts memory' and\n'core.bigFileThreshold'.\n\nSo Pau might be in luck with current git + Junio's patch + enough RAM\non the workstations. Pau, I definitely suggest you try it out.\n\nIf it still consumes too much memory with the largest filest (ie: the\nVM images you mention), the fedpkg approach (discussed in this thread)\nis good too. It's a Python wrapper around git, which tracks a text\nfile listing the hashes of the large files, and fetches them (or\nuploads them) via SCP. You only need to use it when dealing with the\nlarge files -- most of the time you're just using git.\n\nThe fedpkg code is quite readable, and I've already \"stolen\" some of\nits code for my local needs. Recommended.\n\ncheers,\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- Software Architect - OLPC\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"165288","messageId":"20110406164235.GB17662@jpl.local","threadId":"27007","inReplyTo":"41F80411E3CC644A844E6BED6E472FD91196D0CD30@ESESSCMS0363.eemea.ericsson.se","subject":"Re: blobs (once more)","fromName":"Magnus Bäck","fromEmail":"magnus.back@sonyericsson.com","sentAt":"2011-04-06T16:42:35Z","receivedAt":"2011-04-06T16:42:35Z","isPatch":false,"sender":{"key":"magnus.back@sonyericsson.com","avatar":null},"body":"On Wednesday, April 06, 2011 at 13:12 CEST,\n     Peter Jönsson P <peter.p.jonsson@ericsson.com> wrote:\n\n> How about using Google's repo-tool instead of submodules? Is that\n> a better solution if one really needs to keep binaries together\n> (in some way) with the source code?\n\nI don't know if Repo vs. submodules is the most interesting choice;\nthey're both relying on plain gits for the storage. We stored large\nbinaries in gits for a while too, but for us the major hassle was\nthat it was killing our servers. When too many of the few hundreds of\ndevelopers were fetching multi-GB repos the 8-16 core 24 GB RAM server\n(don't know the exact specs at the time) was really down on its knees.\nNow, we were using JGit (via Gerrit) and I'm sure tuning could've helped\n-- as well as just replicating the data and spreading the load across\nmultiple servers -- but don't forget this factor when choosing the tool.\n\n> We are starting to prototype Git and since we need to distribute\n> cross-compilers with the code it would be nice to keep it in a\n> separate repo. Currently we are _heavy_ ClearCase users (oh the\n> horror) with many bad old habits that we are trying to break :)\n\nWe've been there too.\n\n-- \nMagnus Bäck                   Opinions are my own and do not necessarily\nSW Configuration Manager      represent the ones of my employer, etc.\nSony Ericsson\n"},{"id":"165338","messageId":"buo4o6abpsg.fsf@dhlpc061.dev.necel.com","threadId":"27007","inReplyTo":"BANLkTim3kg1ycGkgWsqaZiqMY9LTKV6DBw@mail.gmail.com","subject":"Re: blobs (once more)","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2011-04-07T05:20:47Z","receivedAt":"2011-04-07T05:20:47Z","isPatch":false,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"Pau Garcia i Quiles <pgquiles@elpauer.org> writes:\n> The usual answer to the \"I need to put binaries in the repository\"\n> question has been \"no, you do not\". Well, we do. We are in heavy\n> development now, therefore today's version may depend on a certain\n> version of a third-party shared library (DLL) which we only can get in\n> binary form, and tomorrow's version may depend on the next version of\n> that library, and you cannot mix today's source with yesterday's\n> third-party DLL. I. e. to be able to use the code from 7 days ago at\n> 11.07 AM you need \"git checkout\" to \"return\" our source AND the\n> binaries we were using back then. This is something ClearCase manages\n> satisfactorily.\n\nIf it were me, I'd just store the huge binaries in some sort of separate\nremote filesystem, and then store the remote-file-system _paths_ to them\nin git (in a simple text file).\n\nThen either use the build system or some sort of git filter to make sure\nthat the actual library was installed before building based on the path\nread from the file in git.\n\n[This would be a pain as a _general_ solution (for git), because it\ninvolves coordination with a the remote file system, etc, but for an\norganization like yours setting up a system for a specific product, it\nshould be fairly easy to set up and maintain -- and particularly so if\nthe main use is to store 3rd party library releases, as they're\ntypically not going to be something that anybody will want to checkin,\nbut rather installed by a small set of people.]\n\n-Miles\n\n-- \nCircus, n. A place where horses, ponies and elephants are permitted to see\nmen, women and children acting the fool.\n"},{"id":"165341","messageId":"alpine.DEB.1.00.1104070836590.2040@bonsai2","threadId":"27007","inReplyTo":"buo4o6abpsg.fsf@dhlpc061.dev.necel.com","subject":"Re: blobs (once more)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2011-04-07T06:45:56Z","receivedAt":"2011-04-07T06:45:56Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 7 Apr 2011, Miles Bader wrote:\n\n> Pau Garcia i Quiles <pgquiles@elpauer.org> writes:\n> > The usual answer to the \"I need to put binaries in the repository\" \n> > question has been \"no, you do not\". Well, we do. We are in heavy \n> > development now, therefore today's version may depend on a certain \n> > version of a third-party shared library (DLL) which we only can get in \n> > binary form, and tomorrow's version may depend on the next version of \n> > that library, and you cannot mix today's source with yesterday's \n> > third-party DLL. I. e. to be able to use the code from 7 days ago at \n> > 11.07 AM you need \"git checkout\" to \"return\" our source AND the \n> > binaries we were using back then. This is something ClearCase manages \n> > satisfactorily.\n> \n> If it were me, I'd just store the huge binaries in some sort of separate \n> remote filesystem, and then store the remote-file-system _paths_ to them \n> in git (in a simple text file).\n\nThat fails for a number of reasons:\n\n- it does not pass the 30,000-feet-high test\n\n- integrity is not guaranteed (anybody can edit the files on the remote \n  file system, and nobody would realize that a \"git checkout HEAD~2000\" \n  ends up being something different from before)\n\n- you would have to reinvent an efficient transfer (e.g. taking into \n  account all the data we have already)\n\n- storage is no longer efficient, especially if you have multiple versions \n  of the same file.\n\n- it is no longer decentralized anymore. Just think about yourself sitting \n  in the middle of antarctica, desperately needing to match a penguin \n  against a database of known penguins. You definitely want to have the \n  database local instead of leeching it down the non-existing wire all the \n  time. Likewise, if you and your group sit, say, on Viti Levu, and \n  develop software with people from New York, Texas, you definitely want\n  a repository-in-the-middle, making it one person's duty to synchronize, \n  say, once per day.\n\nI am sure you can think of more reasons.\n\nCiao,\nJohannes\n"}]}