{"thread":{"id":"27078","subject":"is hosting a read-mostly git repo on a distributed file system practical?","startedAt":"2011-04-13T01:40:06Z","lastAt":"2011-04-13T02:29:32Z","messageCount":3,"participants":["Jon Seymour","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"165721","messageId":"BANLkTiniGcRAHbt5LP2NVhE+c8grOr5sLA@mail.gmail.com","threadId":"27078","inReplyTo":null,"subject":"is hosting a read-mostly git repo on a distributed file system practical?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2011-04-13T01:40:06Z","receivedAt":"2011-04-13T01:40:06Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"Is it practical to host a read-mostly git repo on a WAN-based\ndistributed file system?\n\nThe idea is that most developers would use the DFS-based repo to track\nthe tip of the development stream, but only the integrator would\npublish updates to the DFS-based repo.\n\nAs such, the need to repack the DFS-based repo will be somewhat, but\nnot completely, reduced.\n\nIs this going to be practical, or are whole of repo operations\neventually going to kill me because of latency and bandwidth issues\nassociated with use of the DFS?\n\nAre there things I can do with the git configuration (such as limiting\nrepacking behaviour) that will help?\n\njon.\n"},{"id":"165722","messageId":"BANLkTimPYchTXiMpnmE47kxiXvJ_c6QZ9Q@mail.gmail.com","threadId":"27078","inReplyTo":"BANLkTiniGcRAHbt5LP2NVhE+c8grOr5sLA@mail.gmail.com","subject":"Re: is hosting a read-mostly git repo on a distributed file system practical?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-04-13T02:06:50Z","receivedAt":"2011-04-13T02:06:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, Apr 12, 2011 at 21:40, Jon Seymour wrote:\n> Is it practical to host a read-mostly git repo on a WAN-based\n> distributed file system?\n\nUsually not. But test it and find out?\n\n> The idea is that most developers would use the DFS-based repo to track\n> the tip of the development stream, but only the integrator would\n> publish updates to the DFS-based repo.\n>\n> As such, the need to repack the DFS-based repo will be somewhat, but\n> not completely, reduced.\n\nServing git clone is basically a repack operation when run over\ngit://, http:// or SSH. If the DFS was mounted as a local filesystem,\ngit clone would turn into a cpio to copy the directory contents. I'm\nnot sure if that is what you are suggesting to do here or not.\n\n> Is this going to be practical, or are whole of repo operations\n> eventually going to kill me because of latency and bandwidth issues\n> associated with use of the DFS?\n\nLatency is a problem. The Git pack file has decent locality, but there\nare some things that could still stand to be improved. It really\ndoesn't work well unless the pack is held completely in the machine's\nmemory.\n\n-- \nShawn.\n"},{"id":"165723","messageId":"BANLkTimvRoj_dop-s=RUdQBENN6Es_TBsA@mail.gmail.com","threadId":"27078","inReplyTo":"BANLkTimPYchTXiMpnmE47kxiXvJ_c6QZ9Q@mail.gmail.com","subject":"Re: is hosting a read-mostly git repo on a distributed file system practical?","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2011-04-13T02:29:32Z","receivedAt":"2011-04-13T02:29:32Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"On Wed, Apr 13, 2011 at 12:06 PM, Shawn Pearce wrote:\n> On Tue, Apr 12, 2011 at 21:40, Jon Seymour wrote:\n>> The idea is that most developers would use the DFS-based repo to track\n>> the tip of the development stream, but only the integrator would\n>> publish updates to the DFS-based repo.\n>>\n>> As such, the need to repack the DFS-based repo will be somewhat, but\n>> not completely, reduced.\n>\n> Serving git clone is basically a repack operation when run over\n> git://, http:// or SSH. If the DFS was mounted as a local filesystem,\n> git clone would turn into a cpio to copy the directory contents. I'm\n> not sure if that is what you are suggesting to do here or not.\n>\n\nAll clients, including the client that occasionally updates the\nread-mostly repo would be mounting the DFS\nas a local file system. My environment is one where DFS is easy, but\nestablishing a shared server is more complicated (ie. bureaucratic).\n\nI guess I am prepared to put up with a slow initial clone (my\ndeveloper pool will be relatively stable and pulling from a\npeer via git: or ssh: will usually be acceptable for this occasional need).\n\nWhat I am most interested in is the incremental performance. Can my\nintegrator, who occasionally\nupdates the shared repo, avoid automatically repacking it (and hence\ntaking the whole of repo latency hit)\nand can my developers who are pulling the updates do so reliably\nwithout a whole of repo scan?\n\n>> Is this going to be practical, or are whole of repo operations\n>> eventually going to kill me because of latency and bandwidth issues\n>> associated with use of the DFS?\n>\n> Latency is a problem. The Git pack file has decent locality, but there\n> are some things that could still stand to be improved. It really\n> doesn't work well unless the pack is held completely in the machine's\n> memory.\n\nI understand that avoiding repacking for an extended period brings its\nown problems, so I guess I could live with a local repack followed by\nan rsync transfer to re-initial the shared remote, if this was\nwarranted.\n\nI agree, there is no substitute for testing this, but experience of\nothers can be helpful in deciding whether it is even worth attempting.\n\n>\n> --\n> Shawn.\n>\n"}]}