This document describes how to use the Datafeed Rsync Service to block spam using the Spamhaus Project blocking lists (BL). The Datafeed Rsync Service has these advantages over the traditional scheme based on DNS queries over the public DNS infrastructure of the Spamhaus Project:
- it is still based on DNS queries, so changes to do in the mail servers configuration are minimal (in most cases a simple editing of the BL domain);
- all DNS queries are local, so the turnaround time is short and entirely under your control. This means shorter transit times for messages;
- as far as BL checks are concerned, any problem on the outside network or at the Spamhaus servers will not slow down your mail flow. At most, the copy of the BL in use will not be the most current and a bit more spam may be allowed in.
Besides the four BL (SBL, PBL, XBL and DBL), the Datafeed Rsync Service also includes access to two whitelists (WL), SWL and DWL, introduced in September 2010 and currently in the process of being populated.
The Datafeed Rsync Service requires availability of a machine running Linux, FreeBSD, OpenBSD, NetBSD or some other variety of Unix (such as Mac OS/X, Solaris, Tru64, HPUX, AIX, Irix, etc.) The memory and CPU requirements are usually modest, so that old PCs are typically up to the task. For instance, a 400 MHz Pentium with 512 MB of memory is usually sufficient for small organizations. A virtual machine with 512 MB would also be fine.
For customers running Microsoft operating systems, we recommend using the Datafeed Query Service instead. This service is based on DNS queries sent to the Internet rather than local, but answered by an infrastructure entirely separate from the public infrastructure.
You will need to do the following:
- some preparation work (section 1);
- install rsync (section 2);
- install rbldnsd and set it up to run (section 3);
- configure your local DNS resolver (section 4);
- configure your mail servers or antispam appliances (section 5).
Note that this type of setup for using a local copy of blocking lists is quite standard. You may find that most of the setup work described in this document can be used to incorporate other blocking lists in your system.
You need a server with a Linux or Unix-like operating system to be able to run
rsync and
rbldnsd.
rsync is a program to maintain a mirror copy of one or more files, and is nowadays available in the base distribution or as a package on all the major platforms.
rbldnsd is a DNS server specialized to serve blocking list data, and can also run on several Unix-like operating system. It is known to be working on Linux, FreeBSD, NetBSD, OpenBSD, Mac OS X, Solaris, HP-UX, AIX.
Unless your traffic volumes are high (on the scale of millions of messages per day), the hardware requirements for your server are moderate. Normally a server with, say, a 400 MHz CPU and 512MB of RAM is adequate. In particular, the rbldnsd process with all the lists loaded takes usually between 100MB and 150MB of RAM. Disk space usage should remain within 500 MB (also considering the increased storage requirements during the rsync updates).
It is not necessary to assign a dedicated server for this task; most organizations install the software on an existing server, or on a virtual machine.
At the time of this manual version, synchronizing the zones usually implies about 50 MB of incoming data per hour. This quite large amount is mostly caused by the highly dynamic nature of ``zombie'' (trojaned) PCs: hundreds of thousands of IP addresses can enter and leave our XBL database in a single day. For this reason, a bandwidth of at least 512 kbps is recommended to run a Data Feed, and 2 Mbps is desirable. It is possible to confine a Datafeed to use a lesser amount of bandwidth using techniques discussed in the rsync configuration section, but we do not recommend to assign less than 256 kbps. We recommend that users with limited bandwidth use the Rsync Query service instead.
You need to be in control of the DNS server(s) handling general DNS resolutions on your network. If you are currently using DNS resolvers provided by your ISP, prepare to install and use your own (using the DNS server software of your choice). This has nothing to do with the authoritative DNS service for your domain(s), which may well remain hosted by an ISP.
You need to define a directory where Spamhaus files are imported by the
rsync process, and another where they are used by the
rbldnsd process. In the remainder of this document it will be assumed that these two directories are respectively
/usr/local/dnsbl/spamhaus and
/usr/local/dnsbl/rbldnsd, although you can relocate these directories in any place on your system. You will not need to define any subdirectory within those directories.
You will need to create a user named
rbldns and a group also named
rbldns (note that, for historical reasons, there is no "d" at the end of the name). This user will have no shell and no home directory. The most common choice for UID and GID is perhaps 153, but you may choose any unused number.
rbldnsdwill run under this user and group. If you install
rbldnsd as a package prepared for your operating system (if available), this step will likely be done automatically during the installation process, with UID and GID assigned automatically.
Note that rbldnsd does not need write access to the dataset files. For security reasons, all datasets used by rbldnsd must not be owned by the user rbldns.
rsync can run under any existing user in the system.
Make sure that your machine is running with the correct clock, so that your server will synchronize the zone files at the times indicated in the instructions.
It is a good idea to keep the time synchronized with an external source using the ntpd daemon. Please check your system documentation for details.
You need to allow outgoing traffic to port 873/tcp in order to reach the Spamhaus rsync servers. Note that Spamhaus has several servers at different IPs scattered throughout the world, that may change because of maintenance, hardware failures, network malfunctions, IP relocations, DDOS attacks, supplier changes, etc. While the service is stable, IPs of specific servers are not. We therefore discourage people from allowing only rsync traffic going to specific IPs. Since the rsync connections are initiated by you and you will not be running a rsync server (just a client), opening outbound rsync traffic to any destination does not constitute a significant security risk. A large fraction of support tickets opened by our customers is related with useless firewalling of outbound rsync traffic at the customer site.
If your server is running ntpd, traffic from 123/udp and directed to 123/udp should also be allowed in both directions to allow your server to talk with the time servers on the Internet.
Of course, to and from DNS traffic between the rbldnsd server and its clients must also be allowed.rbldnsd only uses 53/udp -- it does not use 53/tcp. Access to the rbldnsd server should be limited to your local DNS resolvers and to any other IP that needs to be able to send queries directly to therbldnsd server. In all cases, do not allow access to the rbldnsd server from unknown clients outside your network.
The
rsync program is one of the best options for data transfer in all cases where a previous version of the file is already available. Since
rsync transmits only file differences across the communication channel, this allows for high efficiency when changes between the current and the previous version constitute a small fraction of the whole file.
This is the typical scenario for blocking lists, which are usually rather large files with a limited number of modifications between one version and the next. Keeping a copy updated using rsync is therefore much more efficient than using ftp or http even when compression schemes are used. While compression can still be used, it brings limited advantages and some important disadvantages (for instance increased CPU load), so we do not use it.
Rather, we have tried to organize data within the XBL zone (which is by far larger than SBL, PBL and DBL) so that file changes between updates tend to cluster in specific areas of the file. Due to the way the rsync algorithm works, such measure decreases the number of bytes transferred in each update and makes the whole synchronization process more efficient.
Nowadays
rsync can usually be found within the regular operating system distributions. Otherwise, it can be
downloaded from the rsync home site and installed ("make install"). Note that you will need only the client component, so you do not need to run the server daemon. Also, there is no need to edit any configuration file.
To verify that your client is working, try the command
rsync na.dr.spamhaus.net::
You should see a 'Welcome' message and a directory list from our server.
Note: If you see a message like
Please contact Spamhaus Technology if you are interested in using this service.
See http://www.spamhaus.org/datafeed/ for further informations.
it means that you are connecting from an IP address not authorized to connect to our Datafeed servers. If the machine has multiple interfaces, try again making sure that the source IP of the connection is the one that you sent to Spamhaus. This can be done using the
--address option of
rsync, like in
rsync --address=1.2.3.4 na.dr.spamhaus.net::
(replace "1.2.3.4" with the IP address on your machine supposed to be authorized to use the Datafeed Rsync service). If this does not solve the problem, contact support.
Note: If you get a "Connection refused" message or can not connect for other reasons, it is likely that some firewall at your side is blocking outbound rsync traffic. Please see the subsection on Firewall settings about the firewalling policies for rsync outbound traffic.
In general, calling
rsync with one argument results in a directory listing, while calling it with two arguments results in a file synchronization. The second argument must be the name of the local file.
The spamhaus-sync.sh script takes care of details, so you do not really need to worry about the command syntax if you are not using rsync for other purposes.
In case you need to access the directory directly, rather than through spamhaus-sync.sh, keep in mind that there are two different but equivalent syntaxes to specify the location of a remote file. One is the traditional rsync syntax
rsync na.dr.spamhaus.net::rbldnsd/remotefile localfile
while the other is the more recent URL syntax
rsync rsync://na.dr.spamhaus.net/rbldnsd/remotefile localfile
Choose the syntax style that you prefer.
You will use a cron job to call a script named
spamhaus-sync.sh every minute. This does not mean that all the zone files will be synchronized every minute. SpamTeq controls the zones update policies through some parameters contained in the zone files, and tuned to optimize the service and avoid congestions when there are major updates. On the basis of these parameters and of the current time, every minute the script will decide what BL and WL files will be updated.
The service now requires version 3.1.2 of spamhaus-sync.sh, distributed since October 27, 2010. If you are a Datafeed customer from before that date, please replace your script with the current one. If you were still running a version 2.x.x script, you also need to change the crontab entry to call the script every minute rather than every 30 minutes.
The line to be defined in /etc/crontab will be like
* * * * * service /usr/local/bin/spamhaus-sync.sh
The shell script
spamhaus-sync.sh can be downloaded using the command
rsync na.dr.spamhaus.net::tools/spamhaus-sync.sh spamhaus-sync.sh
In this example the script will be run as user
service (of course this can be any valid username in your system, but please do not use root!). The reason we execute a script rather than the
rsync program itself is the script allows us to do some additional bookkeeping, get some better diagnostics in case of errors, and to make the system more robust against possible network troubles.
Note: We are not supporting usage of scripts different from spamhaus-sync.sh (which is designed to minimize the risk of congestions). If you have particular needs that spamhaus-sync.sh can not address, please contact us.
Note: In all cases, never use the IP address of a specific Spamhaus rsync server in scripts. IPs can change at any time and without notice in advance. This is essential to achieve continuity of service in presence of adverse network conditions (such as DDoS attacks, etc), or we can simply decide to distribute traffic between different servers, change IPs for server maintenance purposes, add new servers, etc. Note that the time-to-live of DNS records for our rsync servers is very short exactly for these reasons. In practice, a DNS lookup to discover the rsync server IPs should be made before any synchronization.
Note: For the same reasons given above, we do not recommend you block outbound rsync traffic by default, "punching holes" for the Spamhaus servers IPs. These IPs can change at any time, resulting in troubles until you reconfigure the firewall rules. If security is a concern, we recommend you define a rule acting on the source IP, allowing outbound rsync connections to any destination but only if coming from the specific IP of your machine initiating the rsync connections. Remember that we are talking about outbound connections only: you will not have any rsync server running, just a client. Risks are minimal and not larger than the risks associated with running a web browser in your network (which you probably do without having to punch a hole in the firewall for any web site you want to visit).
Note: If your bandwidth availability is limited (of the order of 1 Mbps or less) and the synchronization causes a congestion of your link affecting other activites, you may consider to use the --bwlimit=KBPSoption of rsync, that can decrease the bandwidth at the expense of a longer synchronization time, as discussed in a subsection below.
Before using it, you need to edit
spamhaus-sync.sh to tailor it to your environment. The configuration variables are confined in the initial section of the script, and you are not supposed to need making changes outside of this section.
These notes refer to version 3.1.2 or later of spamhaus-sync.sh, available since October 27, 2010. If you have a version 2.x.x or 3.0.x script from a previous installation, please download the most recent version from our rsync server before proceeding.
We now review the configuration variables. In the most common situations, only TROUBLES, WORKDIR and RSYNCPOOL need to be changed.
- TROUBLES. This variable contains an email address (or a list of addresses separated by commas, without spaces in between) that will receive trouble notifications. If you leave it empty, troubles will not be notified (but some information will still be available in diagnostic files left in the OUTDIR directory).
- WORKDIR. This is the name of the directory where synchronizations are performed, assumed to be /usr/local/dnsbl/spamhaus throughout this manual.
- BLFILES. This variable contains a list of the Spamhaus files to be synchronized. It is usually set to "sbl pbl dbl xbl swl dwl". Edit this variable if you need only some of these datasets.
- SBLPOLICY. Spamhaus makes available a more aggressive "policy" version of the SBL zone. The policy version may contain usually short-lived "escalation" listings that could block some non-spam mail. This option should be chosen only by organizations willing to accept some false positives to the end of getting cooperation from networks with serious spam problems. The "policy" version is the one served by the public DNS infrastructure. To use the policy version set SBLPOLICY to 1, otherwise leave this variable empty.
- RSYNCPOOL. This variable contains the name of the rsync server pool to use. At present we support two pools of servers: na.dr.spamhaus.net referring to servers in North America, andeu.dr.spamhaus.net referring to servers in Europe. Choose the server pool that you believe to be closer (in terms of traveling times of Internet packets) to your geographical location. You mustedit this variable.
- RSYNCHOST_IP_PREFER. If you have reasons to prefer one particular server in our pool (for instance because it is very close to you), you can place its IP in this variable. If we remove that IP from the pool, your preference will become ineffective. At most one IP can be specified. Most users do not specify anything here.
- RSYNCHOST_IP_AVOID. If you have reasons to avoid at all costs one particular server in our pool (for instance because connectivity between that IP and yours is poor), you can place its IP in this variable. At most one IP can be specified. Most users do not specify anything here.
- RSYNCOPTS. rsync options. The current recommended setting is "-L -t --timeout=50". See thersync man page for details. Please handle with care, and do not use the -B/-block-size=SIZEoption and the -z/--compress option.
- VERIFY. This is a flag indicating whether we want cksum verification of data integrity or not. A value of 1 means that verification will be done (recommended). Leave it empty if you do not want cksum verification.
- RSYNC. Name or full pathname of the rsync command. On some platforms rsync sits in a directory which is not part of the default path name for cron jobs, and the full pathname is required. For instance, FreeBSD users may have to prepend /usr/local/bin/.
- CKSUM. Name of the program used for verification, usually cksum but a full pathname can be specified if the command does not sit in a directory included in the default path name for cron jobs (unlikely). It is not used if VERIFY is empty.
- MAILER. This variable contains the name of the program used to mail the trouble notices. The standard Unix program generally called mailx or Mail is the usual choice (on some platform this program is not part of the basic install and must be installed from a package). The variable may contain just the program name or the full pathname. It is not used if TROUBLES is empty.
- HOST and DIG. The script needs to be able to access either one of these two similar programs to perform DNS queries. A full pathname can be specified if the command does not sit in a directory included in the default path name for cron jobs. For instance, Solaris users may have to prepend/usr/sbin/.
- GZIP. Access to the gzip compression/decompression program is needed for updating zones as compressed zone files. A full pathname can be specified if the command does not sit in a directory included in the default path name for cron jobs. If gzip is not available, normal uncompressed zone files will be transferred. This is an experimental feature we are currently investigating: at this time no zone file is compressed yet.
- LOGFILE. Location of a log file, containing one line for each rsync transaction. Lines in the log file contain in the order date, time (UTC), file name, IP address of rsync server, total file size, elapsed time in seconds, rsync exit status.
- OUTDIR. Name of the directory where the output file (and other auxiliary files) generated by this script will be written. This file can be useful when things go wrong, as it contains error messages from all the commands called during the script execution. The default choice is "/tmp" but any choice is fine, as long as the UID running the process can write into it.
- TIMEFILE, SKIPFILE, PSTMPFILE: location of temporary files needed by the script. They may be changed if you wish.
After you have configured all variables, it may be useful to run the script from an interactive shell before creating the new crontab line. If you do that, keep however in mind that the program should be run under the same UID in the crontab (otherwise files can not be overwritten later), and that the path seen by cron tasks is often more restricted than the one seen by commands run interactively (so the script run under cron may fail even if it runs successfully from an interactive shell). Also keep in mind that the script may take a long time to execute the first time you run it, because an amount of the order of 100 MB of data has to be transferred from the net. Subsequent runs are much lighter, because
rsync only transmits differences.
Finally note there should be no output sent to stdout or stderr, as output will go to spamhaus-sync.out in the OUTDIR directory independently on how the script is run. This file is costantly overwritten by different executions, and it is generated only for diagnostic purposes.
The data integrity verification is incorporated in the
spamhaus-sync.sh script. We describe it here briefly, mainly for the benefit of users setting up their own scripts.
If you look at the bottom of a Spamhaus zone file, you will see a line like
# CKSUM: 2767122920 379852
This line, generated by Spamhaus at the end of the zone file construction process, contains the output of the
cksum program (available in nearly all the most common Unix/Linux distributions). The two numbers are in the order a checksum CRC and the total number of bytes in the file, computed by considering the whole zone file except the last line (the one we are talking about). Therefore, this line can be used to verify the integrity of the file at the end of the synchronization procedure. Files that do not pass the verification should not be put into production. Likewise, files that do not contain the CKSUM line at the end are corrupted and should not be put into production. Verification failures are rather uncommon, and if they occur repeatedly they may indicate the presence of serious network or hardware malfunctions.
The Bourne shell code accomplishing the verification is usually the following:
CRC0=`tail -1 xbl | awk '{print $3}'`
CRC1=`sed '$d' xbl | cksum | awk '{print $1}'`
if [ "$CRC0" = "$CRC1" ]
then
(ok)
else
(fail)
fi
where
(ok) and
(fail) should be expanded to the appropriate commands to handle success or failure respectively.
$CRC0 is the checksum as extracted from the file, while
$CRC1 is the checksum as computed from the file.
If it is found that the Datafeed Rsync service is using up too much bandwidth during updates, saturating your line and slowing down other activities, bandwidth usage can limited using the
--bwlimit=KBPS option of
rsync. This option can be added in the
RSYNCOPTS variable in the
spamhaus-sync.sh script. For instance:
RSYNCOPTS="-L -t --timeout=120 --bwlimit=64"
Note: the bandwidth is expressed in kilobytes per second, while line speeds are usually given in bits per second (bps). Therefore, the
KBPS parameter must be multipled by 8K to give the actual bandwidth limit in bps. In the example given above, the limit would be 512 kbps.
Since large updates currently may require transferring about 15 MB (with occasional peaks up to 20 MB or possibly more), the updating time with a limit of KBPS kilobytes per second would result in a transfer time of the order of 250/KBPS minutes: or about 2 minutes for KBPS=128, 4 minutes for KBPS=64, etc. Given that updates should complete within at most a few minutes, going below KBPS=64 (512 kbps) should be regarded as risky in terms of performance. A value as low as KBPS=16 (128 kbps) is almost guaranteed to give problems, and we recommend not to go below 32 ever.
Also note that this mechanism is only effective to decrease the intensity of bandwidth peaks by increasing their duration. It will not decrease the average bandwidth, which is determined by the amount of data to be transferred.
For the same reason, changing the updating interval can not improve the situation much. Updating less often would result in a larger amount of data transferred for each synchronization, and viceversa. In other words, if you can not dedicate enough bandwidth you will simply not be able to keep up with the astounding rate of creation of new spam emitters and spam domains on the Internet. Considering that having fresh copies of XBL and DBL is important to block spam from new emitters and advertising new domains, we do not recommend to increase the updating intervals, as this would mean more spam with limited bandwidth savings. Users with bandwidth problems should consider the Datafeed Query service instead.
rbldnsd is a specialized DNS server, written by Michael J. Tokarev, designed and optimized to answer blocking list queries. Among its advantages:
- rational format for input datasets (in particular, networks can be specified using the CIDR notation);
- ability to merge multiple files in a single blocking list;
- easy way to include exceptions (unlisted addresses located within a listed range);
- extremely small memory footprint;
- extremely limited CPU requirements (very fast);
- high stability and reliability.
For this reason, Spamhaus has chosen
rbldnsd as the DNS server of choice for the Data Feed option, as well as the production software for all the public nameservers.
rbldnsd is capable of sustaining many thousands of DNS queries per second using ordinary PC hardware.
It is important to note that rbldnsd is not a complete nameserver (such as, for instance, Bind) and, in particular, it does not have a caching facility: it can only answer (authoritatively) queries about the blocking list zones it has loaded, and nothing else. Therefore, it will not replace the DNS resolver(s) you are using in your network. The thing to do is to inform the resolver(s) that the local BL/WL queries (and only those!) are answered by the rbldnsd server, and this point is discussed in section 5.
As already indicated,
rbldnsd can run on all the major Unix-like operating systems. Check if a
rbldnsdpackage is available for your platform. If not,
download the source code, compile it (usually this means giving the commands
./configure and
make), and install the resulting executable file in an appropriate directory, such as for instance
/usr/local/sbin. This can be done simply by copying the
rbldnsdexecutable file. You may also wish to install the man page, and check the existence of user and group
rbldns as indicated in a previous section. That is all: there are no configuration files, all the relevant informations are passed via command line when starting the program as described below.
In case of installation troubles please refer to the program documentation or to the rbldnsd mailing list. We suggest that you subscribe to the mailing list anyway (mail volumes are moderate).
Users of the Debian and Ubuntu Linux distributions can install
rbldnsd as a package using
apt-get install rbldnsd
We recommend that you create a local domain -- one which does not exist on the Internet -- to host your local blocking lists. This has several advantages:
- you will be sure that no query is going outside your network due to a configuration mistake;
- you will be more confident that no one from outside will ask BL/WL resolutions to your servers (although restricting access with a firewall is always a good idea!);
- you retain access to the public BL/WL of Spamhaus (for debugging, testing or other needs).
In this document it will be assumed that your BL and WL domain will be dnsbl. Therefore, for instance, you will send queries to sbl.dnsbl rather than to sbl.spamhaus.org. Note that this local domain name is totally arbitrary and can be changed at will.
rbldnsd can combine one or more datasets into a DNS zone, and can simultaneously serve different DNS zones. A dataset is, basically, a file containing IP addresses and ranges; a DNS zone is the domain queried via DNS to obtain informations about an IP address. As far as we are concerned, you will deal with six datasets (sbl, pbl, xbl, dbl, swl and dwl) that will be mapped onto six zones (sbl.dnsbl,pbl.dnsbl, xbl.dnsbl, dbl.dnsbl, swl.dnsbl and dwl.dnsbl). It is also possible to combine the first three datasets (all containing IP addresses) into a single zone zen.dnsbl, with a slight gain in efficiency and a slight loss in flexibility. Furthermore, it is possible to define the individual zones and the combined zone. The kind of mapping to use is entirely up to you. Mappings are defined in the command line arguments used to launch rbldnsd, as discussed in the next section.
rbldnsd reloads itself when it notices that a dataset file changed. So there is no need to give any "reload" command during normal operation.
In this section we discuss the options of the
rbldnsd command which we considered relevant for the operation of your Datafeed Rsync service. A complete list of options can be found on the
rbldnsd man page.
Specifying this option is
required: there is no default. You will give this option multiple times if your
rbldnsd needs to listen to several IP addresses.
If you are going to run rbldnsd on a machine with no regular DNS server (the simplest situation), you will specify the option -b IP.ad.dre.ss, where IP.ad.dre.ss is the primary IP address of your system. This must be an address visible by all the DNS resolver(s) in your network that need to be able to send BL queries -- the resolver(s) used by your mail servers.
If, on the other hand, you are going to run rbldnsd on the same machine hosting your local DNS resolver, or any other DNS server, you need to assign different IPs to the two servers to avoid a conflict on port 53 (if your resolver is Bind 9, it is also possible to bind rbldnsd to a different port, as detailed in section 5.2.1, however this is often problematic and we do not recommend it). If you wish, you can bindrbldnsd to the localhost interface 127.0.0.1 and Bind (or whatever your DNS server is) on the main Ethernet interface. This is valid if rbldnsd needs to be accessed only from the DNS resolver on the same machine. If your DNS server is already occupying port 53 of 127.0.0.1, you can also create a second localhost address (such as 127.0.0.2) and bind rbldnsd to this address (this is often preferable to using a non-standard port number).
If you need to access rbldnsd from other machines in your network, you will have to create an IP alias for the Ethernet interface and bind rbldnsd to that alias.
rbldnsd can be simultaneously bound to several IP addresses on the machine. According to the program documentation, using more than one address implies a small performance penalty. We will assume that you will use just one address; let this address be, for sake of simplicity, 1.2.3.4. Once you have selected the address, make sure that Bind (or whatever your DNS server is) will not bind to this address when starting. By default Bind binds all the available addresses, so to obtain this result you have to editnamed.conf specifying the addresses you want Bind to use. For instance, if your machine has 1.2.3.3 and 1.2.3.4 as IP addresses of the main Ethernet interface,
options {
...
listen-on {
127.0.0.1;
1.2.3.3;
};
...
};
in
named.conf would leave 1.2.3.4 available for
rbldnsd. The
rbldnsd command line option
-b 1.2.3.4 will then instruct
rbldnsd to use that IP.
rbldnsd will do a "chroot" within the directory you indicate with the
-r directory option. This means that this directory will play the role of the root directory
/ as far as the program is concerned, and so the program will be unable to access anything outside it, with an obvious security advantage.
Using the organization discussed in section 2.2, we will use
-r /usr/local/dnsbl
As far as rbldnsd is concerned, the directory that you specify with this option becomes the root directory. Therefore, the program will map an absolute path name like /path/na/me into a filename/usr/local/dnsbl/path/na/me in your filesystem. Moreover, there is no way to refer to files outside the root directory: do not attempt to use symbolic links to escape the root jail.
Moreover, you should use relative pathnames in symbolic links, as in
% cd /usr/local/dnsbl/rbldnsd
% ln -s ../spamhaus/sbl sbl
% ln -s ../spamhaus/pbl pbl
% ln -s ../spamhaus/xbl xbl
% ln -s ../spamhaus/dbl dbl
% ln -s ../spamhaus/swl swl
% ln -s ../spamhaus/dwl dwl
Symbolic links defined using full pathnames would not work correctly when using the chroot option, and would cause
rbldnsd to be unable to find the zone files.
Note: sbl, pbl, xbl, dbl, swl, dwl above are files, not subdirectories.
Within the root directory defined by the -r option, and again referring to the structure discussed in section 2.2, we will use
-w rbldnsd
to specify the directory where
rbldnsd expects to find its files. This will be
/usr/local/dnsbl/rbldnsd in your filesystem. In our case these files are symbolic links.
rbldnsd will be able to follow them, because the files themselves are still within the
rbldnsd root directory.
rbldnsd can continue to process requests even during data reloads using the -f option. This is done by forking a child process to handle requests while the parent reloads the data. The price to pay is extra memory usage since two copies of the data are kept in memory during reloads.
Unless you are very tight on memory, we recommend to select this option. If you do not select it, your mail servers may have to wait a few seconds (or queries may time out and cause some spam to flow through) when rbldnsd reloads.
We recommend, for maximum flexibility, to define independent zones for SBL, PBL, XBL, DBL, SWL, DWL, as well as the combined zone ZEN including SBL, PBL and XBL. This is done by calling
rbldnsd with the following argument list:
sbl.dnsbl:ip4set:sbl \
pbl.dnsbl:ip4trie:pbl \
xbl.dnsbl:ip4tset:xbl \
zen.dnsbl:ip4set:sbl \
zen.dnsbl:ip4trie:pbl \
zen.dnsbl:ip4tset:xbl \
dbl.dnsbl:dnset:dbl \
swl.dnsbl:ip4set:swl \
dwl.dnsbl:dnset:dwl
These arguments indicate how the
XXX.dnsbl zones are composed out of the available datasets. Note how the
zen combined zone is obtained simply by associating three different datasets to the same zone name.
The string after the first colon specifies the method internally used by rbldnsd to map the file into a data structure:
- ip4set is a flexible general method that we recommend for SBL and SWL.
- ip4trie defines a mapping method efficient for datasets like PBL containing networks in the CIDR notation.
- ip4tset is efficient for a dataset like XBL containing only single IPs.
- dnset is a data structure designed to contain domains, rather than IP addresses, and is therefore the structure needed by DBL and DWL.
Note: the \ characters are required to break the command into several lines, and they must be the last character in each line (in other words, make sure that you do not have any space after the \ !). Of course you can also define a single, long command line without any \ character.
Note: the datasets must not be owned by user rbldns. The daemon only needs read access to the datasets.
If you have bound
rbldnsd to an IP address reachable from the external Internet, you may wish to restrict access to make it impossible for unauthorized third parties to use your server for BL lookups. While this can certainly be accomplished by using a firewall, it may be useful to know that
rbldnsd has the possibility to define an ACL (Access Control List).
To this end, create a file named acl into /usr/local/dnsbl/rbldnsd containing
0.0.0.0/1 :ignore
128.0.0.0/1 :ignore
1.2.3.0/24 :pass
The first two lines indicate "block access to the whole Internet" (
rbldnsd does not accept 0.0.0.0/0 as a valid entry), while the last line will contain the network range with allowed access to the service. To specify multiple networks just enter multiple
pass lines. The
ignore keyword means that all DNS queries received by source IPs not included in the
pass lines will simply be absorbed and ignored, without returning any response.
To use the acl file, add :acl:acl to the argument list in the command line (the first acl is a fixed keyword while the second acl is the file name). Do not forget the colon at the beginning; for more details see the rbldnsd man page under acl Dataset.
Note: to implement access restrictions, you must be running rbldnsd release 0.996 or later. Since 0.996 was released in 2006, this condition should now be satisfied unless your O/S is really old.
rbldnsd is a daemon that is typically launched at boot time and left running permanently. You will therefore invoke
rbldnsd at startup time, following the conventions of your operating system and giving all the options and arguments presented above. Remember to launch it in the final stage of the boot process, and particularly after the network interface has been brought up.
If you installed rbldnsd as a package prepared for your operating system, there will be a script to start and stop the program, and a place where options and arguments have to be indicated. If you compiled and installed rbldnsd by yourself, you can use a simple startup script like the following:
#!/bin/sh
# Start the rbldnsd daemon
/usr/local/sbin/rbldnsd -b 1.2.3.4 -f \
-r /usr/local/dnsbl -w rbldnsd \
sbl.dnsbl:ip4set:sbl \
pbl.dnsbl:ip4trie:pbl \
xbl.dnsbl:ip4tset:xbl \
zen.dnsbl:ip4set:sbl \
zen.dnsbl:ip4trie:pbl \
zen.dnsbl:ip4tset:xbl \
dbl.dnsbl:dnset:dbl \
swl.dnsbl:ip4tset:swl \
dwl.dnsbl:dnset:dwl \
:acl:acl
(the last argument is required only if access control is used, as described in section 4.3.6).
The termination script can be as simple as
#!/bin/sh
# Stop the rbldnsd daemon
killall rbldnsd
Note that -- since it binds to a privileged port -- rbldnsd must be started as root. The program drops the root privilege as soon as it binds to the DNS port. From that point on, it runs under the rbldns user.
Verify that the program is started and stopped correctly using these scripts, or the similar scripts supplied by the rbldnsd package for your operating system.
Note: Make sure that no space follows the \ continuation character at the end of each line, otherwise the command will be truncated and some zones will not be loaded!
Linux Debian and Ubuntu users that have installed
rbldnsd as a package (as described in section 4.1.1) should define the following variable in
/etc/default/rbldnsd:
RBLDNSD="dnsbl -b 1.2.3.4 -f -r /usr/local/dnsbl -w rbldnsd \
sbl.dnsbl:ip4set:sbl \
pbl.dnsbl:ip4trie:pbl \
xbl.dnsbl:ip4tset:xbl \
zen.dnsbl:ip4set:sbl \
zen.dnsbl:ip4trie:pbl \
zen.dnsbl:ip4tset:xbl \
dbl.dnsbl:dnset:dbl \
swl.dnsbl:ip4tset:swl \
dwl.dnsbl:dnset:dwl \
:acl:acl"
(the last argument is required only if access control is used, as described in section 4.3.6).
rbldnsd will be launched at boot, and
/etc/init.d/rbldnsd {start|stop|restart|reload}
will start/stop/restart/reload the daemon.
We recommend implementing an access control list (ACL) as described in section 4.3.6. Access to it can also be restricted by using a firewall to limit the transit of 53/UDP packets to and from the client systems.
rbldnsd does not use TCP.
To verify that the daemon is properly running, send A or TXT queries to the zone using the
dig command or the
host command. For instance,
% dig A 2.0.0.127.sbl.dnsbl @1.2.3.4
should show an answer
2.0.0.127.sbl.dnsbl. 300 IN A 127.0.0.2
Equivalently,
host -t A 2.0.0.127.sbl.dnsbl 1.2.3.4
should give
2.0.0.127.sbl.dnsbl has address 127.0.0.2
Note that these commands generate queries that are specifically directed to the rbldnsd server at the address 1.2.3.4. It is crucial to verify that these tests work as expected before proceeding to the next sections.
You now have a
rbldnsd server capable of resolving queries in domains like
sbl.dnsbl. However,
dnsbl is not a top-level domain like
com or
org, and therefore your DNS resolver will not be capable of handling the queries until you inform it that those queries have to be sent to the local
rbldnsd server. This is actually very simple, and this section explains how to do it. In the examples in this section it is assumed, like in the previous sections, that the
rbldnsd server has IP address 1.2.3.4.
There are two methods that can be used to achieve the desired goal: the NS delegation method and the forwarding method.
The NS delegation method can be used regardless of the particular DNS server software that you are using. When using this method, your DNS resolver will learn, through the usual DNS delegation mechanisms, that BL/WL queries have to be sent to the rbldnsd server.
The forwarding method can be used for some common DNS server software such as Bind (however, it can not be used with the Microsoft DNS Server). When using this method, your DNS resolver will forward the query to the rbldnsd server, get the answer and pass it back in a way completely transparent to the client. This method is slightly faster to implement, but it is somewhat less clean and finding errors is more tricky. We recommend to organizations with a complex infrastructure and several mail servers to choose the delegation method if they can.
In the delegation method you create a local DNS zone named
dnsbl, and explicitly define delegation records that indicate that the blocking list subzones are handled by your
rbldnsd server. If you are using the Microsoft DNS Server, use the DNS Manager for this purpose. Otherwise, use the tool you commonly use to create or change information about your local domains.
Create a new primary (master) zone called dnsbl, insert into it an A record pointing to your rbldnsdserver, and the NS record(s) for the list(s), like:
rbldnsd IN A 1.2.3.4
sbl IN NS rbldnsd
pbl IN NS rbldnsd
xbl IN NS rbldnsd
zen IN NS rbldnsd
dbl IN NS rbldnsd
swl IN NS rbldnsd
dwl IN NS rbldnsd
This means that your
rbldnsd server will be known with the name
rbldnsd.dnsbl within your network. DNS queries like
2.0.0.127.sbl.dnsbl directed to the resolver will be diverted to the
rbldnsd server for resolution. In this setup, the
rbldnsd server receives query traffic directly from the machines originating the queries. Restart your DNS resolver after the changes.
Note: sbl, pbl, xbl, zen, dbl, swl, dwl are DNS subdomains, not hosts. Any entry like
zen IN A 1.2.3.4 ; *WRONG*
will not produce any useful result.
This method does not require the creation of a new local zone. It is implemented by editing the DNS server configuration. The details depend on the DNS server you are using.
Reports from the field indicates problems associated with the forwarding method used with DNSSEC. If your DNS infrastructure uses DNSSEC, we recommend using the delegation method instead.
If you are running Bind version 8 or version 9, edit the configuration file
named.conf and insert
zone "dnsbl" {
type forward;
forward only;
forwarders {
1.2.3.4;
};
};
where the IP address of the
rbldnsd server has been put in the
forwarders list (if you have more than one
rbldnsd server, include all of them in the list).
Reload Bind, and it will now forward all queries ending with .dnsbl (that is, all blocking list queries, including non-Spamhaus zones that you might also have) to the rbldnsd server, wait for the answer, and return the answer to the client. In this setup, the rbldnsd server receives all query traffic from the Bind resolver.
Note that Bind 9 also allow to specify a port different from port 53 for each forwarder. This allows you to put rbldnsd on the same IP as Bind, using a different port. This is done using a syntax like
zone "dnsbl" {
type forward;
forward only;
forwarders {
1.2.3.4 port 54;
};
};
In this way you can have Bind and
rbldnsd on the same IP. This is not possible using Bind 8. This method sometimes leads to unexpected problems, and our recommendation is to use the standard port (creating new IP aliases if necessary) unless it is absolutely necessary to run on an alternate port.
If you use the
dnscache program within the
djbdns suite, you should be able to define a forwarder by giving these commands:
cd /service/dnscache
echo 1.2.3.4 > root/servers/dnsbl
chmod 644 root/servers/dnsbl
svc -t .
For further details please
check the djbdns documentation.
You can not use the forwarding method with the Microsoft DNS Server.
By contractual terms you should not redistribute Spamhaus data outside your organization, even if unintentionally. In practice this means that your DNS servers should not answer
dnsbl queries coming from unauthorized parties outside your network. Blocking direct access to
rbldnsd has been already discussed in section 4.3.6, however you should also make sure that
dnsbl queries sent to your DNS resolver from outside are not answered. This is particularly important if you chose the forwarding method.
In fact, as a general rule it is recommended, for security reasons, to ignore all DNS traffic coming from external unauthorized parties -- with the only exception of requests relative to Internet domains for which your server has been designated as an authoritative server.
Under Bind, access can be generally restricted by configuring ACLs in named.conf as follows:
...
acl trusted {
192.168.0.0/24;
1.2.3.0/24;
};
options {
... other options ...
allow-query {
trusted;
};
allow-recursion {
trusted;
};
};
...
In this example, 192.168.0.0/24 is an internal network (RFC1918 addresses) used by your organization, and 1.2.3.0/24 are your external Internet addresses (they may not be present if your Internet gateway is using NAT and you only use internal addresses in your LAN). If your DNS server is also giving authoritative nameservice to specific domains, the general query access restrictions indicated in the
options section can be overridden within the
zone sections for those domains.
For a more detailed discussion of these points, Bind users can consult this document prepared by Team Cymru, while users of the Microsoft DNS server can consult this and this Microsoft technical note.
NOTE: if you chose the forwarding method and your DNS resolver is open to the world, you are likely redistributing Spamhaus data in violation of contractual terms.
After your DNS resolver has been reconfigured as indicated above and reloaded, the tests described in section 4.6 should also work without explicitly directing queries to the
rbldnsd server, but simply using your local resolver. For instance,
% dig A 2.0.0.127.sbl.dnsbl
should produce
2.0.0.127.sbl.dnsbl. 300 IN A 127.0.0.2
We recommend that you perform this test from your mail servers, which will be the main "users". We also recommend to verify that the mail servers do not use any fallback resolver that does not know about the blocking lists. On Unix-like systems the list of resolvers is usually found in the file/etc/resolv.conf. All the nameserver directives in this file should point to DNS resolvers that know how to handle dnsbl resolutions.
Do not proceed to the next section until these tests work as expected.
The last step to do is to configure your mail servers so that they send DNS queries to check if the client IPs are listed on the Spamhaus blocking lists, and reject all mails that satisfy this condition. This can be done using the generic BL lookup facility that you find in virtually all the mail server or antispam appliance products in use today. Please consult your mail server or appliance documentation for details. Moreover, the Spamhaus web site contains some useful instructions and pointers.
In this section we start with some general indications, valid for all mail servers, and describe more in detail what to do for some common servers and appliances.
The whitelists SWL and DWL are not covered in this section yet, firstly because several mail server products are not yet supporting their usage, but also because these lists are currently being constructed slowly and carefully and the number of entries they contain is still rather limited. We expect this situation to improve drastically during 2011. In the meanwhile, we recommend that you start serving them in your local DNS infrastructure as described in the previous sections.
In general, this is what you have to do:
- make sure that the server/appliance resolves DNS names using your DNS resolver(s), configured as described in section 5;
- configure your mail server or antispam appliance to use the external blocking list for IP addresses zen.dnsbl (or sbl.dnsbl, pbl.dnsbl, xbl.dnsbl if you have chosen to go with separate zones). If the configuration includes references to spamhaus.org, replace them with corresponding references to dnsbl.
- configure your mail server or antispam appliance to use the external blocking list for domainsdbl.dnsbl. Again, if the configuration includes references to spamhaus.org, replace them with corresponding references to dnsbl.
Below are instructions on how to accomplish this for some specific cases.
Note: if your setup makes use of a content analyzer separate from the mail server -- such as for instance SpamAssassin -- which also queries the Spamhaus databases to compute the spam scores assigned to messages, make sure to replace all references to spamhaus.org in its configuration with corresponding references to dnsbl.
Note: do not use zen.dnsbl for content analyzers, URL verification, etc.: it contains the PBL database which is not designed for this purpose. Content analysis should be done with sbl-xbl.dnsbl or sbl.dnsbl. The former blocks more spam but may generate occasional false positives.
Locate the
sendmail.mc file and edit it. Insert the following line (as a single line with no linefeeds):
FEATURE(dnsbl, `zen.dnsbl', `"550 Mail from " $&{client_addr} " rejected
using zen.spamhaus.org. Please see http://www.spamhaus.org/query/bl?ip="
$&{client_addr}')dnl
The location of the line of the file is not important. Produce a
sendmail configuration file using the
m4command:
m4 sendmail.mc > sendmail.cf
then place
sendmail.cf into the appropriate directory (usually
/etc or
/etc/mail) and restart
sendmail.
Edit
main.cf (usually located in
/etc/postfix), and add
reject_rbl_client zen.dnsbl
in the list of recipient restrictions, as in
smtpd_recipient_restrictions =
permit_mynetworks,
reject_unauth_destination,
reject_rbl_client zen.dnsbl,
permit
Then create a file named for instance
dnsbl-reply-map containing the line
zen.dnsbl 554 $client blocked using zen.spamhaus.org. Please see $rbl_txt
Create a hash map of it with
postmap hash:dnsbl-reply-map
then insert
rbl_reply_maps = hash:dnsbl-reply-map
in
main.cf. Reload postfix.
First of all, you must have
ucspi-tcp installed to enable you to run
rblsmtpd, the qmail module that does DNS lookups to blocking lists and uses the results to allow or disallow SMTP connections.
Edit the run file (usually located in /var/qmail/supervise/qmail-smtpd) and locate the line invokingtcpserver at the bottom of the file. It will be something like
/usr/local/bin/tcpserver (...options...) \
0 smtp \
qmail-smtpd
(the details can differ in your installation, and you may see
25 in place of
smtp). Insert the invocation of
rblsmtpd just before the
qmail-smtpd argument as follows:
/usr/local/bin/tcpserver (...options...) \
0 smtp \
/usr/local/bin/rblsmtpd -b -r zen.dnsbl \
qmail-smtpd
With this setting, an incoming SMTP connection will launch
rblsmtpd, which will check if the client IP is listed in our databases. If it is listed
rblsmtpd handles the rejection dialogue, while if it is not listed
rblsmtpd launches
qmail-smtpd for normal mail delivery.
Note: Some qmail users have reported that the BL checks are made also for outgoing mail submitted by legitimate users of the mail server using authenticated SMTP. In these cases, the client IP is not a mail server and is likely to be listed in the PBL database. Therefore, users would not be able to send mail. The standard way to solve this problem is to run a separate qmail instance that listens on port 587, accepts only authenticated mail submissions and does not use rblsmtpd. See this example and this discussion.
In this section we describe how to configure SpamAssassin so that it sends query to your local server rather than to the public servers of the Spamhaus Project.
Note that to take advantage of the DBL component, you need SpamAssassin 3.3.1 or later. These instructions assume that your SpamAssassin version is at least 3.3.1. If you are running an earlier release, you should omit all the lines including "DBL".
Go to the SpamAssassin configuration directory, which is usually /etc/mail/spamassassin or/etc/spamassassin. Insert the following definitions (overriding SpamAssassin's own definitions, referring to the Spamhaus Project public service) in the file local.cf:
header __RCVD_IN_ZEN eval:check_rbl('zen','zen.dnsbl.')
header RCVD_IN_XBL eval:check_rbl('zen-lastexternal', 'zen.dnsbl.','127.0.0.[45678]')
header RCVD_IN_PBL eval:check_rbl('zen-lastexternal', 'zen.dnsbl.', '127.0.0.1[01]')
uridnssub URIBL_SBL zen.dnsbl. A 127.0.0.2
urirhssub URIBL_DBL_SPAM dbl.dnsbl. A 127.0.1.2
urirhssub URIBL_DBL_ERROR dbl.dnsbl. A 127.0.1.255
Restart SpamAssassin.
For further information we recommend to consult the SpamAssassin documentation.
This subsection applies only to owners of Barracuda Spam Firewall appliances.
A Barracuda can be configured to query your local servers in place of the public Spamhaus infrastructure. To achieve this goal from the web interface of the appliance:
- select BASIC, then IP Configuration. Look at the IP addresses selected as Primary and Secondary DNS Server. Are these resolvers configured to recognize the dnsbl domain as described in Section 5 ? If yes, proceed to the next point. Otherwise, change these fields so that the Barracuda appliance only uses resolvers that know how to deal with dnsbl requests. If you have just one resolver, leave empty the Secondary DNS Server field. Do not specify nameservers that are not under your control (such as your ISP's nameservers).
- select BLOCK/ACCEPT, then External Blacklists. Set to 'Off' all the spamhaus.org zones that might be configured. Insert zen.dnsbl as a Custom External Blacklist. Hit Save Changes.
This subsection applies only to owners of SonicWALL appliances with Email Security, version 6.x.
A SonicWALL Email Security appliance can be configured to query your local servers in place of the public Spamhaus infrastructure. To achieve this goal from the control interface of the appliance:
- follow System > Host Configuration and look at the DNS server IP addresses (Primary DNS server IP address and, if configured, Fallback DNS server IP address). Are these your DNS resolvers configured to recognize the dnsbl domain as described in Section 5 ? If yes, proceed to the next point. Otherwise, change these fields so that the SonicWALL appliance only uses resolvers that know how to deal with dnsbl requests. If you have a single resolver, indicate just that one. Do not specify nameservers that are not under your control (such as your ISP's nameservers).
- follow Anti-Spam Anti-Phishing > Black List Services (BLS), add a new entry zen.dnsbl. The entry should be automatically enabled after you add it. Disable all the currently configured services referring to spamhaus.org, if any.
This subsection applies only to owners of Symantec Mail Security appliances, 8200 Series.
A Symantec Mail Security appliance can be configured to query your local servers in place of the public Spamhaus infrastructure. To achieve this goal from the web interface of the appliance:
- from the Control Center, follow Settings > Hosts > Edit > DNS and look at the DNS server IP addresses. Are these your DNS resolvers configured to recognize the dnsbl domain as described in Section 5 ? If yes, proceed to the next point. Otherwise, change these fields so that the Symantec appliance only uses resolvers that know how to deal with dnsbl requests. If you have a single resolver, indicate just that one. Do not specify nameservers that are not under your control (such as your ISP's nameservers).
- from the Control Center, follow Policies > Sender Groups > Blocked Senders (Third Party Services). Disable all the currently configured services referring to spamhaus.org, if any, then clickAdd and insert zen.dnsbl.
This subsection applies only to owners of Clearswift MIMEsweeper software for Microsoft Windows. We assume here a version of MIMEsweeper 5.x equipped with the SpamLogic feature.
MIMEsweeper can be configured to query your local servers in place of the public Spamhaus infrastructure. To achieve this goal:
- make sure that the DNS resolver(s) configured on this machine are your DNS resolvers, configured to recognize the dnsbl domain as described in Section 5. If this is the case, proceed to the next point. Otherwise, configure this machine to only use resolvers that know how to deal withdnsbl requests. Do not specify nameservers that are not under your control (such as your ISP's nameservers).
- from the Manager program, follow SMTP Relay > Receiver > Anti-spam > Properties. Replace all the currently configured blocking lists in the spamhaus.org domain with the equivalent ones in thednsbl domain.
