Multimap module

Multimap module is designed to handle rules that are based on different types of lists that are dynamically updated by Rspamd and called maps. This module is useful for whitelists, blacklists and other lists to be organized via files. It can also load remote lists using HTTP and HTTPS protocols or RESP (REdis Serialization Protocol). This article explains in detail all configuration options and features of this module.

Principles of work

Maps in Rspamd are files or HTTP links that are automatically monitored and reloaded if changed. For example, maps can be defined as following:

map = "http://example.com/file";
map = "file:///etc/rspamd/file.map";
map = "/etc/rspamd/file.map";

Rspamd allows to save traffic for HTTP maps using cached maps and respecting 304 Not modified responce as well as Cache-Control headers and ETag. Maps data is shared between workers and the only worker that is allowed to fetch remote maps is the first controller worker.

The default configuration of this module actively uses compound maps where a map is defined as an array of sources (+ local fallback location). For user defined maps this redundancy is typically unnecessary details. However, that is described in the following FAQ section.

Configuration

The module itself contains a set of rules in form:

MAP_SYMBOL1 { 
  type = "type"; 
  map = "url"; 
  # [optional params...] 
}
MAP_SYMBOL2 { 
  type = "type"; 
  map = "from"; 
  # [optional params...] 
}

...

You should normally define your own rules in the file /etc/rspamd/local.d/multimap.conf.

Map attributes

Mandatory attributes are:

• type - map type
• map - [map data](#map-field-syntax]

Optional map configuration attributes:

• prefilter - defines if the map is used in prefilter mode
• action - for prefilter maps defines action set by map match
• regexp - set to true if your map contain regular expressions
• symbols - array of symbols that this map can insert (for key-value pairs), learn more. Please bear in mind, that if you define this attribute, your map must have entries in form key<spaces>value to match a specific symbol.
• score - score of the symbol (can be redefined in the metric section)
• description - map description
• message - message returned to MTA on prefilter reject action being triggered
• group - group for the symbol (can be redefined in metric)
• require_symbols - expression of symbols that have to match for a specific message: learn more
• filter - match specific part of the input (for example, email domain): here is the complete definition of maps filters

For header maps, you also need to specify the exact header using header option.

Important notice: there is a common confusion between type and filter parameters for multimap module. The main rule of thumb is that type means what information is checked in the map, for example, urls, IPs, headers. filter attribute means how this information is transformed before checking, for example extracting domain.

Selector maps are using selectors framework which defines both extraction and transformation. Hence, this type of maps could be considered as the most basic and flexible map types. All other types of maps could be expressed by some selector map. Furthermore, it is possible to store dependent maps in Redis using selectors framework.

Map field syntax

A combination of files and http where the resulting map is a joint list of its elements:

map = [
  "https://maps.rspamd.com/rspamd/mime_types.inc.zst",
  "${DBDIR}/mime_types.inc.local",
  "fallback+file://${CONFDIR}/mime_types.inc"
]

You cannot combine redis nor cdb maps with generic maps.

Maps flaws

Maps content could be augmented by using of flaws, for example map = regexp;/path/to/file.re. This feature is available from the version 2.0.

  local known_types = {
    {'regexp;', 'regexp'},
    {'re;', 'regexp'},
    {'regexp_multi;', 'regexp_multi'},
    {'re_multi;', 'regexp_multi'},
    {'glob;', 'glob'},
    {'glob_multi;', 'glob_multi'},
    {'radix;', 'radix'},
    {'ipnet;', 'radix'},
    {'set;', 'set'},
    {'hash;', 'hash'},
    {'plain;', 'hash'}
  }

Maps content

Maps can contain keys:

key1
key2

key-value pairs (for multi-symbols maps):

key1 value1
key2 value2
key3 value3:score

any comments:

key1
# Single line comment
key2 # Embedded comment

IP maps can also contain IPs or IP/network in CIDR notation

192.168.1.1
10.0.0.0/8

Map types

Type attribute means what is matched with this map. The following types are supported:

DNS maps are legacy and are not encouraged to use in new projects (use rbl for that).

Maps can also be specified as CDB databases that might somehow be useful for large maps:

SOME_SYMBOL {
    map = "cdb:///path/to/file.cdb";
    type = "from";
}

Regexp maps

All maps with the exception of ip and dnsbl maps support regexp mode. In this mode, all keys in maps are treated as regular expressions, for example:

# Sole key
/example\d+\.com/i
# Key + value (test)
/other\d+\.com/i test
# Comments are still enabled

For performance considerations, use only expressions supported by Hyperscan as this engine provides blazing performance at no additional cost. Currently, there is no way to distinguish what particular regexp was matched in case if multiple regexp were matched.

To enable regexp mode, you should set regexp option to true:

# local.d/multimap.conf
SENDER_FROM_WHITELIST {
  type = "from";
  map = "file:///tmp/from.map";
  regexp = true;
}

Map filters

It is also possible to apply a filtering expression before checking value against some map. This is mainly useful for header rules. Filters are specified with filter option. Rspamd supports the following filters so far:

Content filters

Content maps support the following filters:

Filename filters

Filename maps checks also include detected filename match from version 2.0. For example, if some attachment has .png extension but it has real type detected as image/jpeg then two checks would be performed: for the original attachment and for the detected one. This does not include files in archives as Rspamd does not extract them.

Filename maps support this filters set:

From, rcpt and header filters

These are generic emails and headers filters:

Helo, hostname filters

Mempool filters

• regexp:/re/ - extract data from mempool variable according to some regular expression

Received filters

If no filter is specified real_ip is used by default.

The real_ip and from_ip filters must be used in conjunction with IP maps.

Additionally to these filters, Received maps support the following configuration settings:

• min_pos - Minimum position of Received header to match
• max_pos - Maximum position of Received header to match

Negative values can be specified to match positions relative to the end of Received headers.

• flags - One of more flags which MUST be present to match
• nflags - One or more flags which must NOT be present to match

Currently available flags are ssl (hop used SSL) and authenticated (hop used SMTP authentication).

Selector options filters

• regexp:/re/ - extract data from selector’s results according to some regular expression (usually not needed)

Symbol options filters

• regexp:/re/ - extract data from symbol options according to some regular expression

URL filters

URL maps allows another set of filters (by default, url maps are matched using hostname part):

Pre-filter maps

To enable pre-filter support, you should specify action parameter which can take one of the following values:

• accept - accept the message (no action)
• add header or add_header - add a header to the message
• rewrite subject or rewrite_subject - change the subject
• greylist - greylist the message
• reject - drop the message

No filters will be processed for a message if such a map matches. Multiple symbols or symbol conditions are not supported for prefilter maps by design.

# local.d/multimap.conf
IP_WHITELIST { 
  type = "ip"; 
  map = "/tmp/ip.map"; 
  prefilter = true;
  action = "accept";
}
# Better use RBL module instead
SPAMHAUS_PBL_BLACKLIST { 
  type = "dnsbl"; 
  map = "pbl.spamhaus.org";
  description = "PBL dns block list";
  prefilter = true;
  action = "reject";
}

Multiple symbol maps

From the version 1.3.1, it is possible to define multiple symbols and scores using multimap module. To do that, you should define all possible symbols using symbols option in multimap:

# local.d/multimap.conf
CONTENT_BLACKLISTED {
  type = "content";
  filter = "body"; # can be headers, full, oneline, text, rawtext
  map = "${LOCAL_CONFDIR}/content.map";
  symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
  regexp = true;
}

In this example, you can use 3 symbols:

• CONTENT_BLACKLISTED
• CONTENT_BLACKLISTED1
• CONTENT_BLACKLISTED2

the map:

# Symbol + score
/re1/ CONTENT_BLACKLISTED1:10
# Symbol with default score
/re2/ CONTENT_BLACKLISTED2
# Just a default symbol: CONTENT_BLACKLISTED
/re3/

Symbols that are not defined in the symbols attribute but used in the map are ignored and replaced by the default map symbol. If the value of a key-value pair is missing, then Rspamd just inserts the default symbol with dynamic weight equal to 1.0 (which is multiplied by metric score afterwards).

Get all matches

If you want to match all possible regexps/globs in that list, not a single one, then you need to define multi flag for that map:

# local.d/multimap.conf
CONTENT_BLACKLISTED {
  type = "content";
  filter = "body"; # can be headers, full, oneline, text, rawtext
  map = "${LOCAL_CONFDIR}/content.map";
  symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
  regexp = true;
  multi = true;
}

Conditional maps

From version 1.3.1, it is possible to set up maps that depends on other rules and check map if some certain condition is satisfied. In particular, you can check that a message has a valid SPF policy to perform some whitelisting. However, you don’t want to bother about mailing lists. Then you can write the following map condition:

# local.d/multimap.conf
FROM_WHITELISTED {
  require_symbols = "R_SPF_ALLOW & !MAILLIST";
  type = "from";
  map = "/some/list";
}

You can use any logic expression of other symbols within require_symbols definition. Rspamd automatically inserts dependency for a multimap rule on all symbols that are required by this particular rule. You cannot use symbols added by post-filters here, however, pre-filter and normal filter symbols are allowed.

Redis for maps

From version 1.3.3, it is possible to work with maps which are stored in Redis backend. You can use any external application to put data into Redis database using HSET command (e.g HSET hashkey test@example.org 1). After you can define map as protocol redis:// and specify hash key to read. Redis settings can be defined inside multimap module also.

Combined maps

From version 2.0, you can create maps with multiple values to be checked and joint via expression:

COMBINED_MAP_AND {
  type = "combined";
  rules {
    ip = {
      type = "radix";
      map = "${TESTDIR}/configs/maps/ip.list";
      selector = "ip";
    }
    from {
      map = "${TESTDIR}/configs/maps/domains.list";
      selector = "from:domain";
    }
  }
  expression = "from & ip"
}
COMBINED_MAP_OR {
  type = "combined";
  rules {
    ip = {
      type = "radix";
      map = "${TESTDIR}/configs/maps/ip.list";
      selector = "ip";
    }
    from {
      map = "${TESTDIR}/configs/maps/domains.list";
      selector = "from:domain";
    }
  }
  expression = "from || ip"
}

Combined maps support merely selectors syntax, not general multimap rules.

Dependent maps

Version 2.0 also allows to create dependent maps in Redis where map key depends on some other data extracted from the same message. For example, per user based whitelist.

TODO: write more

Examples

Here are some examples of multimap configurations:

# local.d/multimap.conf
SENDER_FROM_WHITELIST_USER {
  type = "from";
  filter = "email:user";
  map = "file:///tmp/from.map";
  action = "accept"; # Prefilter mode
}

# With Redis backend, also you need specify servers for Redis.
SENDER_FROM_WHITELIST_USER {
  type = "from";
  map = "redis://hashkey";
}

SENDER_FROM_REGEXP {
  type = "header";
  header = "from";
  filter = 'regexp:/.*@/';
  map = "file:///tmp/from_re.map";
}
URL_MAP {
  type = "url";
  filter = "tld";
  map = "file:///tmp/url.map";
}
URL_MAP_RE {
  type = "url";
  filter = 'tld:regexp:/\.[^.]+$/'; # Extracts the last component of URL
  map = "file:///tmp/url.map";
}
FILENAME_BLACKLISTED {
  type = "filename";
  filter = "extension";
  map = "${LOCAL_CONFDIR}/filename.map";
  action = "reject";
  message = "A restricted file type was found";
}
CONTENT_BLACKLISTED {
  type = "content";
  filter = "body"; # can be headers, full, oneline, text, rawtext
  map = "${LOCAL_CONFDIR}/content.map";
  symbols = ["CONTENT_BLACKLISTED1", "CONTENT_BLACKLISTED2"];
  regexp = true;
}
ASN_BLACKLIST {
  type = "asn";
  map = "${LOCAL_CONFDIR}/asnlist.map";
}
LAST_RECEIVED_HEADER_IP_IF_AUTHED {
  type = "received";
  map = "${LOCAL_CONFDIR}/rcvd_ip.map";
  filter = "real_ip";
  min_pos = -1;
  flags = ["authenticated"];
}
SYMBOL_OPTIONS_DBL {
  type = "symbol_options";
  target_symbol = "DBL_ABUSE_REDIR";
  symbols = ["INTERESTING_DOMAIN"];
  map = "${LOCAL_CONFDIR}/dbl_redir_symbols.map";
}
WHITELIST_HELO_RCPT {
  type = "combined";
  prefilter = true;
  action = "accept";
  rules {
    helo {
      map = "${LOCAL_CONFDIR}/helo_smtp.map";
      selector = "helo";
    }
    rcpt = {
      map = "${LOCAL_CONFDIR}/rcpt_internal_subdomains.map";
      selector = "rcpts:domain";
    }
  }
  expression = "helo & rcpt"
}

Example adopted from @kvaps:

• cd /etc/rspamd
• create local.d folder if not exists
• cd local.d
• create multimap.conf in /etc/rspamd/local.d/ folder if it does not exists
• create lists:

touch local_bl_from.map.inc local_bl_ip.map.inc local_bl_rcpt.map.inc \
local_wl_from.map.inc local_wl_ip.map.inc local_wl_rcpt.map.inc

• change permissions:

chmod o+w local_bl_from.map.inc local_bl_ip.map.inc local_bl_rcpt.map.inc \
local_wl_from.map.inc local_wl_ip.map.inc local_wl_rcpt.map.inc

• edit multimap.conf (you should be in /etc/rspamd/local.d/ folder)

# local.d/multimap.conf

# Blacklists
local_bl_ip { type = "ip"; map = "$LOCAL_CONFDIR/local.d/local_bl_ip.map.inc"; symbol = "LOCAL_BL_IP"; description = "Local ip blacklist";score = 3;}
local_bl_from { type = "from"; map = "$LOCAL_CONFDIR/local.d/local_bl_from.map.inc"; symbol = "LOCAL_BL_FROM"; description = "Local from blacklist";score = 3;}
local_bl_rcpt { type = "rcpt"; map = "$LOCAL_CONFDIR/local.d/local_bl_rcpt.map.inc"; symbol = "LOCAL_BL_RCPT"; description = "Local rcpt blacklist";score = 3;}

# Whitelists
local_wl_ip { type = "ip"; map = "$LOCAL_CONFDIR/local.d/local_wl_ip.map.inc"; symbol = "LOCAL_WL_IP"; description = "Local ip whitelist";score = -5;}
local_wl_from { type = "from"; map = "$LOCAL_CONFDIR/local.d/local_wl_from.map.inc"; symbol = "LOCAL_WL_FROM"; description = "Local from whitelist";score = -5;}
local_wl_rcpt { type = "rcpt"; map = "$LOCAL_CONFDIR/local.d/local_wl_rcpt.map.inc"; symbol = "LOCAL_WL_RCPT"; description = "Local rcpt whitelist";score = -5;}