{"thread":{"id":"27079","subject":"Re: is hosting a read-mostly git repo on a distributed file system practical?","startedAt":"2011-04-13T03:47:15Z","lastAt":"2011-04-13T04:57:10Z","messageCount":2,"participants":["George Spelvin","Jon Seymour"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165724","messageId":"20110413034715.15553.qmail@science.horizon.com","threadId":"27079","inReplyTo":null,"subject":"Re: is hosting a read-mostly git repo on a distributed file system practical?","fromName":"George Spelvin","fromEmail":"linux@horizon.com","sentAt":"2011-04-13T03:47:15Z","receivedAt":"2011-04-13T03:47:15Z","isPatch":false,"sender":{"key":"linux@horizon.com","avatar":null},"body":"> All clients, including the client that occasionally updates the\n> read-mostly repo would be mounting the DFS as a local file system. My\n> environment is one where DFS is easy, but establishing a shared server\n> is more complicated (ie. bureaucratic).\n\n> I guess I am prepared to put up with a slow initial clone (my developer\n> pool will be relatively stable and pulling from a peer via git: or ssh:\n> will usually be acceptable for this occasional need).\n\n> What I am most interested in is the incremental performance. Can my\n> integrator, who occasionally updates the shared repo, avoid automatically\n> repacking it (and hence taking the whole of repo latency hit) and can\n> my developers who are pulling the updates do so reliably without a whole\n> of repo scan?\n\nI think the answers are yes, but I have to make a vouple of things clear:\n* You can *definitely* control repack behaviour.  .keep files are the\n  simplest way to prevent repacking.\n* Are you talking about hosting only a \"bare\" repository, or one with\n  the unpacked source tree as well?  If you try to run git commands on\n  a large network-mounted source tree, things can get more than a bit\n  sluggish; git recursively stats the whole tree fairly frequently.\n  (There are ways to precent that, notably core.ignoreStat, but they\n  make it less friendly.)\n* You can clone from a repository mounted on the file system just as\n  easily as you can from a network server.  So there's no need to set\n  up a server if you find it onconvenient.\n* Normally, the developers will clone from the integrator's repository\n  before doing anything, so the source tree, and any changes they make,\n  will be local.\n* A local clone will try to hard link to the object directory.  I think\n  it will copy them if it fails, or you can force that with \"git clone\n  --no-hardlinks\".  For a more space-saving version, try \"git clone\n  -s\", which will make a sort of soft link to the upstream repository.\n  It's a git concept, so repacking upstream won't do any harm, but you\n  Must Not delete objects from the upstream repository or you'll create\n  dangling references in the downstream.\n* If using the objects on the DFS mount turns out to be slow, you can\n  just do the initial clone with --no-hardlinks.  Then the developers'\n  day-to-day work is all local.\n\nIndeed, you could easily do everything via DFS.  Give everyone a personal\n\"public\" repo to push to, which is read-only to everyone else, and let\nthe integrator pull from those.\n\n> I understand that avoiding repacking for an extended period brings its\n> own problems, so I guess I could live with a local repack followed by\n> an rsync transfer to re-initial the shared remote, if this was\n> warranted.\n\nNormally, you do a generational garbage collection thing.  You repack the\ncurrent work frequently (which is fast to do, and to share, because\nit's small), and the larger, slower, older packs less frequently.\n\nAnyway, I hope this helps!\n"},{"id":"165726","messageId":"BANLkTi=cE4=EC0C4MA5iac6_O-YMsW_SLg@mail.gmail.com","threadId":"27079","inReplyTo":"20110413034715.15553.qmail@science.horizon.com","subject":"Re: is hosting a read-mostly git repo on a distributed file system practical?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2011-04-13T04:57:10Z","receivedAt":"2011-04-13T04:57:10Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"On Wed, Apr 13, 2011 at 1:47 PM, George Spelvin <linux@horizon.com> wrote:\n\n> I think the answers are yes, but I have to make a vouple of things clear:\n> * You can *definitely* control repack behaviour.  .keep files are the\n>  simplest way to prevent repacking.\n\nGood.\n\n> * Are you talking about hosting only a \"bare\" repository, or one with\n>  the unpacked source tree as well?  If you try to run git commands on\n>  a large network-mounted source tree, things can get more than a bit\n>  sluggish; git recursively stats the whole tree fairly frequently.\n>  (There are ways to precent that, notably core.ignoreStat, but they\n>  make it less friendly.)\n\nBare. Developers use local disk for local repos and working tree.\n\n> * You can clone from a repository mounted on the file system just as\n>  easily as you can from a network server.  So there's no need to set\n>  up a server if you find it onconvenient.\n\nAre there advantages to using rsync for the initial clone? Will I get\nbetter restartability in the case that the network is less than 100% reliable?\n\nI do remember trying to use a DFS-file system in the past, before I understood\npack management properly and I seem to recall issues with network reliability.\n\n> Indeed, you could easily do everything via DFS.  Give everyone a personal\n> \"public\" repo to push to, which is read-only to everyone else, and let\n> the integrator pull from those.\n>\n\nI'll probably use ssh-secured peer to peer for publishing purposes.\nThe main thing I want\nthe DFS-hosted repo for is to provide a single, always up, go-to point\nfor the shared tag set.\n\n>\n> Anyway, I hope this helps!\n>\n\nYep, thank you.\n\njon.\n"}]}