Architectural Changes in GFI MailEssentials 14.1

In Q3 2009 we will be releasing GFI MailEssentials 14.1, into which we have put a lot of effort on the architectural front of the product (amongst others). I wanted to take this opportunity to share the architectural changes we have made in GFI MailEssentials 14.1 and discuss their benefits. Inevitably, I will first take you through the historical evolution of the product’s architecture.

The current incarnation of GFI MailEssentials owes its first design to early 2002 (if my memory does not fail me, we released GFI MailEssentials 7 somewhere around June 2002).

The general architecture was quite simple and straightforward:

• We hooked into the Microsoft IIS SMTP server using SMTP Event Sinks.
• Emails were passed on to a scan engine which was plugin based.
• Different anti-spam plug-ins implemented various spam filters.
• These anti-spam plug-ins called a library of actions (which are effectively what gets done with a spam email, for example, saved to a folder on disk, forwarded to a particular mailbox, moved into a particular folder in a mailbox, etc).

This architecture is illustrated in Figure 1 below – it worked transparently (or almost) and equally well both on Windows 2000 IIS SMTP and Exchange 2000 (and eventually Windows 2003 IIS SMTP and Exchange 2003), and it was around through six revisions of GFI MailEssentials – up to GFI MailEssentials 12.

Figure 1: GFI MailEssentials Architecture, v7 – v12.

One thing to note about this architecture is that the anti-spam scanning components all “live” within inetinfo.exe, which had the disadvantage that any instability in our scanning components would bring down IIS – including the SMTP server and also the IIS web server (i.e. any websites on the machine hosted by IIS would also be taken down…)

With the advent of Microsoft Exchange Server 2007, which was in 64-bit, we took out the scan engine into a separate 32-bit process, gfiscans.exe, which appears as the GFI MailEssentials Scan Engine service in the services console. Making use of out-of-proc COM, we started calling the 32-bit scan engine from the 64-bit transport agents written for Exchange 2007. This brought with it two main advantages:

• First of all, since the anti-spam scanning components now “lived” in a separate process, any instability in our scanning components would only bring down the GFI MailEssentials Scan Engine service (i.e. gfiscans.exe).
• Furthermore, any crash in our scanning components could now be recovered from, because the transport agents could detect that the GFI MailEssentials Scan Engine was no longer responding and start it up again – without any impact on email flow.

These changes were at first only rolled out on Microsoft Exchange Server 2007 installations of GFI MailEssentials 12. The architecture of GFI MailEssentials 12 on 32-bit IIS SMTP and Exchange 2000/2003 installations remained as in Figure 1 above.

Figure 2: GFI MailEssentials Architecture, v12 – v14.

When we released GFI MailEssentials 14.0, we introduced support for 64-bit IIS SMTP, and the same architecture as in Figure 2 above was also introduced on 64-bit IIS SMTP installations. The architecture of GFI MailEssentials 12 on 32-bit IIS SMTP and Exchange 2000/2003 installations remained as in Figure 1 above.

With the upcoming release of GFI MailEssentials 14.1, the architecture of GFI MailEssentials on all installations will be as illustrated in Fig 3 below.

• The scan engine is now always hosted in a separate 32-bit process, gfiscans.exe, which appears as the GFI MailEssentials Scan Engine service in the services console.
• Email sources – be they 32-bit IIS SMTP or Exchange 2000/2003 event sinks, 64-bit IIS SMTP event sinks, or 64-bit Exchange 2007/2010 transport agents – all submit emails for processing by the scan engine via out-of-proc COM.
• The various anti-spam filter plugins do their scanning within the gfiscans.exe process.
• Emails for which an action needs to be executed, such as spam emails to be forwarded to some other mailbox or moved to some folder within the recipient’s mailbox, are removed from the IIS SMTP or Exchange queue and queued for processing to a separate process.
• When an email is queued for an action to be executed, the raw email content is spooled to disk, and some metadata is queued to MSMQ.
• Actions are executed within the GFI MailEssentials Managed Attendant Service, the process name of which is contentsecurity.as.attendant.exe.
• An email which has been queued for an action to be executed is picked up by an actions coordinator which is hosted within the Managed Attendant, and passed to a set of plugins which implement the actions.  

Figure 3: GFI MailEssentials Architecture, v12 – v14.1.

The new architecture in GFI MailEssentials 14.1 brings about a number of advantages and improvements:

• No matter what platform GFI MailEssentials is installed on, since the anti-spam scanning components now “live” in a separate process, any instability in our scanning components will only bring down the GFI MailEssentials Scan Engine service (i.e. gfiscans.exe).
• No matter what platform GFI MailEssentials is installed on, any crash in our scanning components could now be recovered from, because the sinks or transport agents can detect that the GFI MailEssentials Scan Engine is no longer responding and start it up again – without any impact on email flow.
• As opposed to the previous architecture, in which actions were executed within the scan engine, increasing the time the scan engine was taking to process spam emails, spam emails are now simply queued to the Managed Attendant, reducing the time in which a response is returned to IIS SMTP or Exchange in the case of spam emails, and thereby improving the performance of the scan engine.
• With the new architecture, when an action being executed on a spam email fails, that email is queued to what is called the recovery queue, where emails in the recovery queue are reprocessed so as to execute any non-executed actions. This makes for a more robust actions framework.

One action for which this makes a particular difference is the “Move to Exchange Folder” action, now known as the “Deliver email to mailbox: in Exchange mailbox sub-folder” option. With the previous architecture, when for some reason or another the Exchange store is not operational, the “Move to Exchange Folder” action would fail, and “Global Actions” would kick in, i.e. the spam email would instead be saved to disk or forwarded to another mailbox as configured. With the new architecture, such emails are instead queued to the recovery queue, which is reprocessed when the Managed Attendant is restarted.

• The new architecture also allowed us to support configuring multiple actions for spam emails so, for example, it is now possible to deliver a copy of a spam email to a sub-folder within the user’s mailbox, as well as forwarding a copy to another mailbox and saving another copy to a folder on disk.
• This new architecture also takes us a number of steps in the direction of supporting Exchange 2010, but that is a post in itself for another time.

As can be seen, although it is a minor release, GFI MailEssentials 14.1 is a substantial update to the product. If you haven’t yet tried out the beta, I recommend you check out the GFI MailEssentials 14.1 beta forum at forums.gfi.com, or visit beta.gfi.com for details on how to join the GFI MailEssentials beta mailing list.