Re: Git for redundant mail servers

From: David Lang <david.lang@digitalinsight.com>
Date: Apr 24, 2005, 05:12 UTC
Message-ID: <Pine.LNX.4.62.0504232159030.32287@qynat.qvtvafvgr.pbz>
In-Reply-To: <2cfc403205042301243841fe0c@mail.gmail.com>

On Sat, 23 Apr 2005, Jon Seymour wrote:

Show 26 quoted lines

> On 4/23/05, David Woodhouse <dwmw2@infradead.org> wrote:
>> Random alternative use for git... we could use it to provide a cluster
>> of redundant mail delivery/storage servers.
>>
>> The principle is simple; you use something like a set of Maildir
>> folders, stored in a git repository. Any action on the mail storage is
>> done as a commit -- that includes delivery of new mail, or user actions
>> from the IMAP server such as changing flags, deleting or moving mail.
>> These actions are actually fairly efficient when Maildir folders are
>> stored in a git repository -- the IMAP model is that mails are
>> immutable, and flag changes are done as renames.
>>
>> In the normal case where all the servers are online, each commit is
>> immediately pushed to each remote server. When a server is offline or
>> separated somehow from the rest of the group, it's going to have to do a
>> merge when it reconnects -- we'd implement a Maildir-specific merge
>> algorithm, which really isn't that hard to do.
>>
>
> This is a cool idea. When the concept is rendered this way, it sounds
> a lot like some of the core principles in the architecture of the
> Lotus Notes replication engine. I've always thought it would be cool
> to have an open engine that provided similar functionality to the
> Lotus Notes replication engine without the naff programming
> environment that sits on top. I can see how the git concepts and code
> could provide the basis of such a solution. Very cool.

Having been in several discussions on the cyrus mailing list about replication let me point out a couple basic problems that you have to work around.

1. when a new message arrives it gets given a numeric messageid, this 
message id is not supposed to change without fairly drastic things 
happening (the server telling all clients to forget everything they know 
about the status of the mailbox). this requires syncronization between 
servers if both are receiving messages.

2. git effectivly stores snapshots of things and you deduce the changes by 
comparing the snapshots. for things like flags changing this is a 
relativly inefficiant way to replicate changes (although if one server is 
offline for a while it could be a firly efficiant way to do the merge)

and now a couple of starting points

Cyrus already implements single-instance store so the concept of the same message living in multiple places doesn't have to be grafted in. it keeps the message flags seperate from the messages themselves so the messages could be replicated seperatly from the state.

personally I'm not seeing git being a huge advantage for this, but I do see some advantages and it's very possible I'm missing some others.

go for it.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare