{"thread":{"id":"261","subject":"Git for redundant mail servers","startedAt":"2005-04-23T06:42:41Z","lastAt":"2005-04-24T07:45:19Z","messageCount":5,"participants":["David Woodhouse","Jon Seymour","David Lang"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"1368","messageId":"1114238562.3419.29.camel@localhost.localdomain","threadId":"261","inReplyTo":null,"subject":"Git for redundant mail servers","fromName":"David Woodhouse","fromEmail":"dwmw2@infradead.org","sentAt":"2005-04-23T06:42:41Z","receivedAt":"2005-04-23T06:42:41Z","isPatch":false,"sender":{"key":"dwmw2@infradead.org","avatar":"https://gravatar.com/avatar/7afd4f07e0cf7d7e046ae2d23678296b37777c96488e6f3451e78a5514154ebd?d=mp&s=160"},"body":"Random alternative use for git... we could use it to provide a cluster\nof redundant mail delivery/storage servers. \n\nThe principle is simple; you use something like a set of Maildir\nfolders, stored in a git repository. Any action on the mail storage is\ndone as a commit -- that includes delivery of new mail, or user actions\nfrom the IMAP server such as changing flags, deleting or moving mail.\nThese actions are actually fairly efficient when Maildir folders are\nstored in a git repository -- the IMAP model is that mails are\nimmutable, and flag changes are done as renames.\n\nIn the normal case where all the servers are online, each commit is\nimmediately pushed to each remote server. When a server is offline or\nseparated somehow from the rest of the group, it's going to have to do a\nmerge when it reconnects -- we'd implement a Maildir-specific merge\nalgorithm, which really isn't that hard to do.\n\nIn this case we'd probably want to make active use of the feature of git\nwhich allows you to prune history. You don't need to keep any history\nfurther back than the commit which will be the common ancestor when a\ncurrently-absent member of the cluster eventually comes back. In the\ncommon case, that will actually be no history at all, since all members\nwill be present.\n\nYou can then have multiple members of a cluster, each running an SMTP\nserver and allowing for delivery of email, and each running an IMAP\nserver. Clients can connect to any of the machines and receive IMAP\nservice, and email will continue to flow inward, as long as at least one\nmachine in the cluster remains alive. \n\n-- \ndwmw2\n\n"},{"id":"1370","messageId":"2cfc403205042301243841fe0c@mail.gmail.com","threadId":"261","inReplyTo":"1114238562.3419.29.camel@localhost.localdomain","subject":"Re: Git for redundant mail servers","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2005-04-23T08:24:33Z","receivedAt":"2005-04-23T08:24:33Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"On 4/23/05, David Woodhouse <dwmw2@infradead.org> wrote:\n> Random alternative use for git... we could use it to provide a cluster\n> of redundant mail delivery/storage servers.\n> \n> The principle is simple; you use something like a set of Maildir\n> folders, stored in a git repository. Any action on the mail storage is\n> done as a commit -- that includes delivery of new mail, or user actions\n> from the IMAP server such as changing flags, deleting or moving mail.\n> These actions are actually fairly efficient when Maildir folders are\n> stored in a git repository -- the IMAP model is that mails are\n> immutable, and flag changes are done as renames.\n> \n> In the normal case where all the servers are online, each commit is\n> immediately pushed to each remote server. When a server is offline or\n> separated somehow from the rest of the group, it's going to have to do a\n> merge when it reconnects -- we'd implement a Maildir-specific merge\n> algorithm, which really isn't that hard to do.\n> \n\nThis is a cool idea. When the concept is rendered this way, it sounds\na lot like some of the core principles in the architecture of the\nLotus Notes replication engine. I've always thought it would be cool\nto have an open engine that provided similar functionality to the\nLotus Notes replication engine without the naff programming\nenvironment that sits on top. I can see how the git concepts and code\ncould provide the basis of such a solution. Very cool.\n\njon.\n"},{"id":"1494","messageId":"Pine.LNX.4.62.0504232159030.32287@qynat.qvtvafvgr.pbz","threadId":"261","inReplyTo":"2cfc403205042301243841fe0c@mail.gmail.com","subject":"Re: Git for redundant mail servers","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2005-04-24T05:12:21Z","receivedAt":"2005-04-24T05:12:21Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Sat, 23 Apr 2005, Jon Seymour wrote:\n\n> On 4/23/05, David Woodhouse <dwmw2@infradead.org> wrote:\n>> Random alternative use for git... we could use it to provide a cluster\n>> of redundant mail delivery/storage servers.\n>>\n>> The principle is simple; you use something like a set of Maildir\n>> folders, stored in a git repository. Any action on the mail storage is\n>> done as a commit -- that includes delivery of new mail, or user actions\n>> from the IMAP server such as changing flags, deleting or moving mail.\n>> These actions are actually fairly efficient when Maildir folders are\n>> stored in a git repository -- the IMAP model is that mails are\n>> immutable, and flag changes are done as renames.\n>>\n>> In the normal case where all the servers are online, each commit is\n>> immediately pushed to each remote server. When a server is offline or\n>> separated somehow from the rest of the group, it's going to have to do a\n>> merge when it reconnects -- we'd implement a Maildir-specific merge\n>> algorithm, which really isn't that hard to do.\n>>\n>\n> This is a cool idea. When the concept is rendered this way, it sounds\n> a lot like some of the core principles in the architecture of the\n> Lotus Notes replication engine. I've always thought it would be cool\n> to have an open engine that provided similar functionality to the\n> Lotus Notes replication engine without the naff programming\n> environment that sits on top. I can see how the git concepts and code\n> could provide the basis of such a solution. Very cool.\n\nHaving been in several discussions on the cyrus mailing list about \nreplication let me point out a couple basic problems that you have to work \naround.\n\n1. when a new message arrives it gets given a numeric messageid, this \nmessage id is not supposed to change without fairly drastic things \nhappening (the server telling all clients to forget everything they know \nabout the status of the mailbox). this requires syncronization between \nservers if both are receiving messages.\n\n2. git effectivly stores snapshots of things and you deduce the changes by \ncomparing the snapshots. for things like flags changing this is a \nrelativly inefficiant way to replicate changes (although if one server is \noffline for a while it could be a firly efficiant way to do the merge)\n\nand now a couple of starting points\n\nCyrus already implements single-instance store so the concept of the same \nmessage living in multiple places doesn't have to be grafted in. it keeps \nthe message flags seperate from the messages themselves so the messages \ncould be replicated seperatly from the state.\n\npersonally I'm not seeing git being a huge advantage for this, but I do \nsee some advantages and it's very possible I'm missing some others.\n\ngo for it.\n\nDavid Lang\n\n-- \nThere are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.\n  -- C.A.R. Hoare\n"},{"id":"1498","messageId":"1114322071.3419.68.camel@localhost.localdomain","threadId":"261","inReplyTo":"Pine.LNX.4.62.0504232159030.32287@qynat.qvtvafvgr.pbz","subject":"Re: Git for redundant mail servers","fromName":"David Woodhouse","fromEmail":"dwmw2@infradead.org","sentAt":"2005-04-24T05:54:30Z","receivedAt":"2005-04-24T05:54:30Z","isPatch":false,"sender":{"key":"dwmw2@infradead.org","avatar":"https://gravatar.com/avatar/7afd4f07e0cf7d7e046ae2d23678296b37777c96488e6f3451e78a5514154ebd?d=mp&s=160"},"body":"On Sat, 2005-04-23 at 22:12 -0700, David Lang wrote:\n> 1. when a new message arrives it gets given a numeric messageid, this \n> message id is not supposed to change without fairly drastic things \n> happening (the server telling all clients to forget everything they know \n> about the status of the mailbox). this requires syncronization between \n> servers if both are receiving messages.\n\nYeah, that's the most interesting part. One option would be to require\nquorum before a server is allowed to add to a mailbox -- but that would\nrender the thing unsuitable for _intentional_ offline use, where you\nwant to be able to move mails from one folder to another on your laptop\nwhile it's disconnected.\n\nSince it should be relatively rare for 'competing' commits to occur\nduring periods of disconnection, I suspect that the solution doesn't\nhave to be particularly efficient. I'm not sure I'd really want to\nchange UIDVALIDITY if it happened, but perhaps we could simply remove\n_all_ the affected UIDs, and assign new UIDs to the same mails.\n\nIn practice, it's far more important that for us to ensure that an\nexisting UID _never_ refers to a different mail, than it is to make sure\nthat a given mail always keeps the same UID.\n\n> 2. git effectivly stores snapshots of things and you deduce the changes by \n> comparing the snapshots. for things like flags changing this is a \n> relativly inefficiant way to replicate changes (although if one server is \n> offline for a while it could be a firly efficiant way to do the merge)\n\nWe don't have to stick _precisely_ to Maildir -- but flag changes are\njust a rename in Maildir, leaving the mail object entirely intact while\nchanging only the tree. That isn't _so_ bad; but yes, it could probably\nbe done a little better than just \"Maildir in git\".\n\n-- \ndwmw2\n\n"},{"id":"1506","messageId":"Pine.LNX.4.62.0504240028540.32437@qynat.qvtvafvgr.pbz","threadId":"261","inReplyTo":"1114322071.3419.68.camel@localhost.localdomain","subject":"Re: Git for redundant mail servers","fromName":"David Lang","fromEmail":"david.lang@digitalinsight.com","sentAt":"2005-04-24T07:45:19Z","receivedAt":"2005-04-24T07:45:19Z","isPatch":false,"sender":{"key":"david.lang@digitalinsight.com","avatar":null},"body":"On Sun, 24 Apr 2005, David Woodhouse wrote:\n\n> Date: Sun, 24 Apr 2005 15:54:30 +1000\n> From: David Woodhouse <dwmw2@infradead.org>\n> To: David Lang <david.lang@digitalinsight.com>\n> Cc: jon@zeta.org.au, git@vger.kernel.org\n> Subject: Re: Git for redundant mail servers\n> \n> On Sat, 2005-04-23 at 22:12 -0700, David Lang wrote:\n>> 1. when a new message arrives it gets given a numeric messageid, this\n>> message id is not supposed to change without fairly drastic things\n>> happening (the server telling all clients to forget everything they know\n>> about the status of the mailbox). this requires syncronization between\n>> servers if both are receiving messages.\n>\n> Yeah, that's the most interesting part. One option would be to require\n> quorum before a server is allowed to add to a mailbox -- but that would\n> render the thing unsuitable for _intentional_ offline use, where you\n> want to be able to move mails from one folder to another on your laptop\n> while it's disconnected.\n\nIMAP defines an offline mode, I haven't looked at it, but it would have to \ndeal with this in some way.\n\n> Since it should be relatively rare for 'competing' commits to occur\n> during periods of disconnection, I suspect that the solution doesn't\n> have to be particularly efficient. I'm not sure I'd really want to\n> change UIDVALIDITY if it happened, but perhaps we could simply remove\n> _all_ the affected UIDs, and assign new UIDs to the same mails.\n>\n> In practice, it's far more important that for us to ensure that an\n> existing UID _never_ refers to a different mail, than it is to make sure\n> that a given mail always keeps the same UID.\n\ngood point.\n\nthere are two things that will cause competing commits of full mail \nmessages.\n\n1. new mail arriving from the Net (probably via SMTP/LMTP\n\n2. Client actions\n\n2a. direct posting of messages (most common for FCC folders)\n\n2b. copying of messages between folders\n\n2c. flag changes\n\n2d. expunging messages\n\n>> 2. git effectivly stores snapshots of things and you deduce the changes by\n>> comparing the snapshots. for things like flags changing this is a\n>> relativly inefficiant way to replicate changes (although if one server is\n>> offline for a while it could be a firly efficiant way to do the merge)\n>\n> We don't have to stick _precisely_ to Maildir -- but flag changes are\n> just a rename in Maildir, leaving the mail object entirely intact while\n> changing only the tree. That isn't _so_ bad; but yes, it could probably\n> be done a little better than just \"Maildir in git\".\n\nI'm familiar with Cyrus which has a similar concept of mail storage, but \nI'm only vaguely familar with maildir (I don't know all the details of how \nit does things)\n\nthe key question to answer is are you trying to just replicate maildir \nunderneith the normal programs that use it?, or are you trying to have a \nreplicated mailserver and are willing to modify the software as well as \nfiddle with the storage?\n\nif you are after the first then you have to do everything at the \nfilesystem level, if you are after the second it's amuch easier job, but \nyou need to think carefully up front to decide what capabilities you need \nthe software to have and pick the right software to start modifying.\n\nDavid Lang\n\n\n-- \nThere are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.\n  -- C.A.R. Hoare\n"}]}