Cloudmark 3048 User Manual
Cloudmark
Cartridge
Installation and
Administration Guide
© 2001-2007 Cloudmark, Inc. All rights reserved. Cloudmark, the Cloudmark logo, and other trademarks,
service marks, and designs are registered or unregistered trademarks of Cloudmark Inc. and its subsidiaries
in the United States and in foreign countries. Other brands and products are trademarks of their respective
holders. All product information is subject to change without notice.
This document may not, in whole or in part, be copied, photocopied, reproduced, translated or reduced to
any electronic medium or machine readable form without prior consent in writing from:
All examples with names, company names or companies that appear in this guide are fictitious and do not
refer to, or portray, in name or substance, any actual names, organizations, entities or institutions. Any
resemblance to any real person, organization, entity or institution is purely coincidental.
While every effort has been made to ensure technical accuracy, information in this document is subject to
change without notice and does not represent a commitment on the part of Cloudmark, Inc. Cloudmark
makes no warranties with respect to this documentation and disclaims any implied warranties of
merchantability and fitness for a particular purpose. Cloudmark shall not be liable for any errors or for
incidental or consequential damages in connection with the furnishing, performance or use of this manual
or examples herein.
Regular expression support is provided by the PCRE library package, which is open source software,
written by Philip Hazel, and copyright by the University of Cambridge, England.
The GIFLIB distribution is Copyright (c) 1997 Eric S. Raymond
Cloudmark, Inc. 128 King Street, 2nd Floor, San Francisco, CA 94107 USA
Cloudmark Europe, Ltd. Carmelite, 50 Victoria Embankment, Blackfriars, London EC4Y ODX UK
Cloudmark Cartridge version 3048
Last modified: March 11, 2008
Jpeglib is copyright (C) 1991-1998, Thomas G. Lane.
ImageMagick is copyright 1999-2007 ImageMagick Studio LLC.
PCRE is a library of functions to support regular expressions whose syntax and semantics are as close as
possible to those of the Perl 5 language.
Release 7 of PCRE is distributed under the terms of the "BSD" licence, as specified below. The
documentation for PCRE, supplied in the "doc" directory, is distributed under the same terms as the
software itself.
Written by: Philip Hazel <ph10@cam.ac.uk>, University of Cambridge Computing Service, Cambridge,
England. Phone: +44 1223 334714. Copyright (c) 1997-2004 University of Cambridge. All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided
that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the
following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the
following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of the University of Cambridge nor the names of its contributors may be used to
endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Contents
CHAPTER 1 Introduction . . . . . . . . . . . . . . . .1
What’s new in Cartridge 3048 . . . . . . . . . . . . . . . . . .1
Cloudmark fingerprinting algorithms . . . . . . . . . . . . . . . .2
Cloudmark Global Threat Network. . . . . . . . . . . . . . . . .2
Micro-updates . . . . . . . . . . . . . . . . . . . . . . . .3
Message scoring . . . . . . . . . . . . . . . . . . . . . . .3
Message categorization . . . . . . . . . . . . . . . . . . . . .4
Cartridge statistics. . . . . . . . . . . . . . . . . . . . . . .4
Whitelisting . . . . . . . . . . . . . . . . . . . . . . . . .5
CHAPTER 2 Cloudmark Cartridge Installation . . . . . . . . .7
The Cartridge installation package . . . . . . . . . . . . . . . . .7
Installing or updating the Cartridge . . . . . . . . . . . . . . . .8
Installation for Cloudmark Authority Engine-based products . . . . . . . . 8
Installation for the Cloudmark Authority Plug-In for SpamAssassin. . . . . . 9
Installation for Cloudmark Immunity . . . . . . . . . . . . . . . . 10
Installation for Openwave Email Mx . . . . . . . . . . . . . . . . 11
Installation for Openwave Edge Gx . . . . . . . . . . . . . . . . 12
CHAPTER 3 Cloudmark Cartridge Configuration . . . . . . 13
CHAPTER 4 Micro-Updates . . . . . . . . . . . . . . 17
Micro-update frequency. . . . . . . . . . . . . . . . . . . . 17
Automatic micro-updates . . . . . . . . . . . . . . . . . . . . 18
Updates at user-specified intervals . . . . . . . . . . . . . . . . 18
iii
Contents
CHAPTER 5 Whitelisting . . . . . . . . . . . . . . . 23
Network interaction. . . . . . . . . . . . . . . . . . . . . . 19
Using HTTP proxies . . . . . . . . . . . . . . . . . . . . . 19
Connection timeout logic . . . . . . . . . . . . . . . . . . . 19
Data files. . . . . . . . . . . . . . . . . . . . . . . . . . 20
Offline files . . . . . . . . . . . . . . . . . . . . . . . . . 20
Data file integrity and security. . . . . . . . . . . . . . . . . . 21
Advanced micro-update configurations . . . . . . . . . . . . . . 21
Using an HTTP proxy . . . . . . . . . . . . . . . . . . . . . 21
Host whitelisting . . . . . . . . . . . . . . . . . . . . . . . 23
Header whitelisting . . . . . . . . . . . . . . . . . . . . . . 24
Body whitelisting. . . . . . . . . . . . . . . . . . . . . . . 25
Envelope whitelisting . . . . . . . . . . . . . . . . . . . . . 25
Sample whitelist configuration file . . . . . . . . . . . . . . . . 27
CHAPTER 6 Cartridge Statistics Reporting . . . . . . . . 29
How statistics are reported to Cloudmark . . . . . . . . . . . . . 29
What statistics are collected . . . . . . . . . . . . . . . . . . 31
Cartridge reporting configuration . . . . . . . . . . . . . . . . 35
APPENDIX A Logging . . . . . . . . . . . . . . . . 37
Common variables . . . . . . . . . . . . . . . . . . . . . . 37
INFO log messages . . . . . . . . . . . . . . . . . . . . . . 38
WARN log messages . . . . . . . . . . . . . . . . . . . . . 41
ERROR log messages . . . . . . . . . . . . . . . . . . . . . 44
CRITICAL log messages . . . . . . . . . . . . . . . . . . . . 48
iv
Contents
Index . . . . . . . . . . . . . . . . . . 49
v
Contents
vi
CHAPTER 1
Introduction
Cloudmark’s gateway solutions use the Cloudmark Cartridge to deliver the latest
Cloudmark anti-abuse technology for your email platform. This guide explains
how to install, configure, and administer the cartridge. You can find out what’s
new in this version of the cartridge in “What’s new in Cartridge 3048” below.
The rest of this chapter introduces the technology behind the Cloudmark
Cartridge:
• “Cloudmark fingerprinting algorithms” on page 2
• “Cloudmark Global Threat Network” on page 2
• “Micro-updates” on page 3
• “Message scoring” on page 3
• “Cartridge statistics” on page 4
• “Whitelisting” on page 5
What’s new in Cartridge 3048
Cartridge 3048 includes these changes:
• The Cartridge now keeps track of its last known viable state and returns to it
upon restarting after a crash.
• A new configuration key specifies an alternate port on which to download
micro-updates. See “micro-update port” on page 16.
• A new configuration key controls how the sending IP address is determined.
See “use envelope for ip information” on page 16.
• The default value for the “use ip information” configuration key is now “yes.”
See “use ip information” on page 16.
• The micro-update file set now includes .fsl and .xrl metadata files. See “Data
files” on page 20.
1
Cloudmark Cartridge Installation and Administration Guide Chapter 1
• A new fingerprinting scheme provides faster processing.
• A new statistics field reports your unique installation ID. See “What statistics
are collected” on page 31.
Cloudmark fingerprinting algorithms
The Cloudmark Cartridge includes Cloudmark’s fingerprinting algorithms,
designed to target the most current spamming techniques. Using these
algorithms, the Cloudmark Cartridge generates a set of fingerprints for each
incoming message.
The Cloudmark Cartridge maintains a cache of all fingerprints that have a known
classification, such as spam, phishing, or virus fingerprints. The fingerprints of
an incoming message are compared to these known fingerprints, and a message
score is generated. This list of known fingerprints is regularly updated with the
latest data from the Cloudmark Global Threat Network, using the micro-updates
mechanism. See “Cloudmark Global Threat Network” below and “Microupdates” on page 3.
Cloudmark Global Threat Network
Cloudmark’s community of millions of end users provides constant, real-time
feedback about which messages are considered spam, phishing, or email-borne
viruses, and which ones are considered legitimate. The Trust Evaluation System
(TES) assigns each user a trust level based on how well the user’s feedback
concurs with that of other trusted users. Less-trusted users have less influence
over network-wide message classification, while the most trusted users have more
influence.
When a sufficient number of trusted users block a certain message as junk, this
message’s fingerprint is flagged. Information about the fingerprint is distributed
throughout the network to automatically block that message (and all its
permutations) for other users.
Micro-updates provide the latest known fingerprints as determined by the
Cloudmark Global Threat Network. By using micro-updates, you protect your
platform against the most current email-borne threats. See “Micro-updates”
below.
2 Cloudmark fingerprinting algorithms
Chapter 1 Introduction
Micro-updates
Cloudmark stores message fingerprints generated though the Global Threat
Network in near-real-time. Micro-updates are the mechanism that allows
Cloudmark customers to download the latest fingerprint data at regular intervals.
Micro-updates enable Cloudmark to
• maintain the highest level of accuracy on spam, virus, and phishing messages
as well as legitimate messages
• handle new varieties of email threats proactively and automatically
• reduce false positives
• eliminate manual message analysis with a fully-automated approach
To maintain the highest possible accuracy, the micro-updates feature must be
correctly configured. For complete information, see Chapter 4, “Micro-Updates”.
Message scoring
When the Cloudmark Cartridge scans a message, it assigns a spam score (as a
percentage) to indicate the likelihood that the message is an abusive message
(such as spam, phishing, or a virus). For example, if the cartridge assigns a
message a score of 99, it means that Cloudmark is 99% certain that message is
bad; a score of 1 means that Cloudmark is almost certain the message is
legitimate.
When used in conjunction with Cloudmark Authority Engine SDK (CMAE
SDK) 2.0 or later, the cartridge may also provide information about each
message’s classification (spam, phishing, virus, and so on). Consult your vendor
to find out whether your implementation of the CMAE SDK supports this
feature.
You can establish your own policies for handling spam, and configure your
application to take action on a message based on its spam score. Such actions
typically include one or more of the following:
• storing spam in a designated folder
• flagging spam messages in the Subject field
• deleting spam
• returning spam to its original sender
Micro-updates 3
Cloudmark Cartridge Installation and Administration Guide Chapter 1
Message categorization
When scoring a message with the Cloudmark Authority Engine SDK’s
CMAE_Score() function, an application can request that the cartridge return a
category and a subcategory for the message. Categories and subcategories are
expressed as integers, which are mapped to categories in the .cats file. See the .cats
file for the list of categories.
For example, using the Authority Engine SDK, the following call produces a
message score and category:
CMAE_Score(CMAE_Envelope Envelope,
const char *RFC822Content, size_t RFC822ContentLength,
unsigned int *ScoreOut,
unsigned int *CategoryOut, unsigned int *SubCategoryOut,
unsigned int *RescanOut, char **AnalysisOut);
If CategoryOut is 7and SubCategoryOut is 0, then the cartridge has categorized
the message as a virus message. The following call provides more information:
CMAE_DescribeCategory(unsigned int Category, unsigned int SubCategory,
char **CategoryDescOut, char **SubCategoryDescOut);
With Category=7 and SubCategory=0, CategoryDescOut would contain an
allocated string “virus”, and SubCategoryDescOut would contain an allocated
string “undefined”.
For detailed information about using this feature in your application, see the
Cloudmark Authority Engine SDK Guide.
Cartridge statistics
By default, the Cloudmark Cartridge sends cartridge configuration information
and message scanning statistics back to Cloudmark. By collecting this
information, Cloudmark can more effectively detect potential accuracy issues
and proactively address them before there is a need for the customer to contact
Cloudmark. If your organization has special privacy concerns, contact
Cloudmark.
For complete information, see Chapter 6, “Cartridge Statistics Reporting”.
4 Message categorization
Chapter 1 Introduction
Whitelisting
A whitelist is a list of trusted senders from whom you always accept email, or
email characteristics which indicate a trusted message. This feature of the
Cloudmark Cartridge minimizes the filtering of legitimate messages and allows
system administrators to conveniently manage the receipt of messages from
known safe senders.
For complete information, see Chapter 5, “Whitelisting”.
Whitelisting 5
Cloudmark Cartridge Installation and Administration Guide Chapter 1
6 Whitelisting
CHAPTER 2
Cloudmark Cartridge Installation
This chapter provides the Cartridge installation instructions:
• “The Cartridge installation package” below.
• “Installing or updating the Cartridge” on page 8
! Be sure to refer to the release notes of each Cartridge version for special
installation instructions.
The Cartridge installation package
The Cartridge installation package is provided in either a TAR or a ZIP file,
depending on your platform. Before installation, verify that the installation
package contains all the required installation files.
Below is a list of the components in a standard Cartridge installation package:
• etc/micro_updates/<dpl_version_number>.dpl
• etc/micro_updates/<rpl_version_number>.rplv1
• etc/micro_updates/<awl_version_number>.awl
• etc/micro_updates/<acf_version_number>.acf
• etc/micro_updates/<csl_version_number>.csl
• etc/micro_updates/<fsl_version_number>.fsl
• etc/micro_updates/<xrl_version_number>.xrl
• etc/micro_updates/<mpl_version_number>.mpl
• etc/micro_updates/<cats_version_number>.cats
• etc/micro_updates/<mfl_version_number>.mfl
7
Cloudmark Cartridge Installation and Administration Guide Chapter 2
• etc/micro_updates/<impl_version_number>.implv1
• etc/micro_updates/states/srl_set.package
• etc/whitelist.cfg.sample
• etc/cartridge.cfg.sample
• lib/cartridge.so
Additional files are downloaded as micro-updates. For more information about
these files, see “Data files” on page 20.
Installing or updating the Cartridge
Follow the installation instructions for the product with which you are using the
Cartridge:
• “Installation for Cloudmark Authority Engine-based products” below
• “Installation for the Cloudmark Authority Plug-In for SpamAssassin” on
page 9
• “Installation for Cloudmark Immunity” on page 10
• “Installation for Openwave Email Mx” on page 11
• “Installation for Openwave Edge Gx” on page 12
These instructions apply to both new installation and updates to existing
Cartridge installations.
Installation for Cloudmark Authority Engine-based products
TO INSTALL THE CARTRIDGE FOR AUTHORITY-BASED PRODUCTS
1 Stop the server/service using the Cloudmark Authority Engine.
2 If you are updating an existing Cartridge installation, remove all of the files in
the etc/micro-updates/ directory, as well as the etc/micro-updates/states/
subdirectory.
The new Cartridge will download the correct files with which to re-populate
this directory.
3 Place the compressed Cartridge file in the product home directory.
4 Decompress it.
• For Linux/Solaris, extract the Cartridge with the following command:
8 Installing or updating the Cartridge
Chapter 2 Cloudmark Cartridge Installation
gzip –d –c < SpamDNA-3048.x.x.x-<platform>.tar.gz | tar xvf –
• For Windows installation, double-click the .zip file, then click Extract.
5 Create the etc/license.cfg file.
This file must contain the two-line license text that you received from
Cloudmark.
6 If you are updating an existing Cartridge installation, update your
cartridge.cfg to the latest defaults listed in the file etc/cartridge.cfg.sample.
7 Restart the server/service using the Cloudmark Authority Engine.
8 Check for the following log message:
INFO:MICROUPDATE: Successfully updated <file> from network
(new serial <serial>)
There should be one such message for every micro-updates file listed in “The
Cartridge installation package” on page 7.
See also the Cloudmark Authority Engine SDK Guide.
Installation for the Cloudmark Authority Plug-In for SpamAssassin
During a new installation of the Cloudmark Authority Plug-In for SpamAssassin,
the Cartridge is installed automatically. To upgrade the Cartridge for an existing
installation of the plug-in, use the instructions below.
T
O UPGRADE THE CARTRIDGE FOR THE SPAMASSASSIN PLUG-IN
1 Become the superuser.
2 Switch to the SpamAssassin CMAE subdirectory:
cd /etc/mail/spamassassin/cmae/cloudmark
This path may vary in your installation. Make sure you are in the cloudmark
subdirectory.
3 Remove all of the files in the etc/micro-updates directory.
The new Cartridge will download the correct files with which to re-populate
this directory.
4 Extract the Cartridge:
gzip –d –c < SpamDNA-3048.x.x.x-<platform>.tar.gz | tar xvf –
Installing or updating the Cartridge 9
Cloudmark Cartridge Installation and Administration Guide Chapter 2
5 If you are upgrading from Cartridge 3046 or earlier, create the etc/license.cfg
file.
This file must contain the two-line license text that you received from
Cloudmark.
6 Update your cartridge.cfg with the latest defaults listed in the file
etc/cartridge.cfg.sample.
7 Restart the CMAE server:
bin/cmaed restart
Installation for Cloudmark Immunity
To install the Cartridge when using Cloudmark Immunity, follow the steps
below:
T
O INSTALL THE CARTRIDGE FOR IMMUNITY
1 Stop Immunity.
2 If you are updating an existing Cartridge installation, remove all of the files in
the etc/micro-updates directory.
The new Cartridge will download the correct files with which to re-populate
this directory.
3 Place the compressed Cartridge update file in the product home directory.
4 Decompress it.
• For Linux/Solaris, extract the Cartridge with the following command:
gzip –d –c < SpamDNA-3048.x.x.x-<platform>.tar.gz | tar xvf –
• For Windows, double-click the .zip file.
5 Create the etc/license.cfg file.
This file must contain the two-line license text that you received from
Cloudmark.
6 If you are updating an existing Cartridge installation, update your
cartridge.cfg with the latest defaults listed in the file etc/cartridge.cfg.sample.
7 Restart Immunity.
If you are installing Cloudmark Immunity for the first time, complete the
following additional steps:
8 Manually copy the file new_cm_egm.db.temp (externally provided) to the
following location before running dbsetup.pl:
• (Immunity 2.0.2) <immunity root>/data/new_cm_egm.db.temp
10 Installing or updating the Cartridge
Chapter 2 Cloudmark Cartridge Installation
• (Immunity 2.0.1) <immunity root>/setup/sql/sqlite/cm_egm.db
(<immunity root> is typically /srv/immunity)
9 The default cartridge.cfg and whitelist.cfg files will not be installed in etc/. To
create the default configuration files, copy etc/cartridge.cfg.sample to
etc/cartridge.cfg and etc/whitelist.cfg.sample to etc/whitelist.cfg.
10Check for the following log message:
Aug 21 12:28:06 : INFO:MICROUPDATE: Successfully updated
<file> from network (new serial <serial>)
There should be one such message for every micro-updates file listed in “The
Cartridge installation package” on page 7.
See also the Cloudmark Immunity Installation and Administration Guide.
Installation for Openwave Email Mx
Before proceeding with the installation instructions below, Cloudmark
recommends backing up the entire $AUTH_HOME directory.
T
O INSTALL THE CARTRIDGE FOR OPENWAVE EMAIL MX
1 Follow these steps:
cd $AUTH_HOME
(AUTH_HOME=value from configuration key /<host>/authority/homeDir).
You should see an “etc” and a “lib” directory here.
cp lib/cartridge.so lib/cartridge.so.<datestamp>
$INTERMAIL/lib/imservctrl stop imextserv
gzip –d –c < SpamDNA-3048.x.x.x-<platform>.tar.gz | tar xvf –
2 Create the etc/license.cfg file.
This file must contain the two-line license text that you received from
Cloudmark.
3 If you are updating an existing Cartridge installation, update cartridge.cfg to
the latest defaults listed in the file etc/cartridge.cfg.sample.
4 Start the extensions service:
$INTERMAIL/lib/imservctrl start imextserv
5 Check for the following log message:
Aug 21 12:28:06 : INFO:MICROUPDATE: Successfully updated
<file> from network (new serial <serial>)
There should be one such message for every micro-updates file listed in “The
Cartridge installation package” on page 7.
Installing or updating the Cartridge 11
Loading...