In April, a CERT advisory announced the discovery of two separate buffer-overflow vulnerabilities in Snort, a popular security-monitoring tool used for detecting suspicious network activities. This development was disturbing and ironic: system administrators install and run programs like Snort to improve security, and don’t often consider the possibility that the tools themselves might be attacked and exploited to create entirely new security holes. It’s therefore important to understand precisely what happened here, especially since the same mechanisms used against Snort could threaten other security tools.
In this article, I will review the attacks that have been launched against Snort in the past, as well as the recent (and more serious) buffer overflows. In each case, I’ll discuss the ways Snort developers have responded to the attacks, and the strategies system administrators can take to minimize the risks. Furthermore, I’ll show that Snort’s vulnerabilities extend to other security-monitoring tools, implying that we need to be careful when we use them, as well. Finally, I’ll summarize techniques to do just that: secure monitoring.
Evasion and Stealth Techniques
Snort is a network intrusion detection system, or NIDS. It sniffs packets from the network, and analyzes them using a database of pattern-matching rules. These rules identify a wide range of suspicious or hostile activities, such as misuse of network protocols, attempted buffer overflows, denial-of-service attacks, port scans, and so on. When such activities are detected, Snort can be configured to log the network trace data for more detailed examination, and to send alerts so that system administrators will notice.
Since the main purpose of Snort is detection, a primary goal of attackers is evasion. If attacks can be structured so that they are overlooked by Snort, then system administrators will be left with a false sense of security — arguably a worse situation than if Snort had not been used at all.
The evolution of Snort has been driven largely by the endless arms race between the attackers and the defenders. The open source distribution of Snort allows the bad guys to study the detection mechanisms and devise ways to bypass them. The Snort developers, aided by reports from the user community and examination of often widely available attack tools, then respond by enhancing the algorithms, and the cycle continues. The history of innovation on both sides has been fascinating and instructive.
When new malicious network activities are discovered, pattern-matching rules are added to Snort’s database. These rules are stateless, meaning that only one packet is examined at a time. Snort also supports a variety of preprocessors that track multiple packets, and hence are stateful. The modular design of Snort allows preprocessors to be easily added or modified, and more complicated evasion attempts have been countered by improvements in the preprocessors. The increasing focus on preprocessors for Snort is reminiscent of the expansion of Linux firewall technology from stateless ipchains to stateful iptables.
Attackers frequently exploit IP fragmentation for evasion. For example, the nmap port-scanning tool provides an option to use tiny, fragmented packets that break headers for higher-level protocols into pieces during transmission. Snort’s frag2 preprocessor reassembles the packets, so the headers (and the payload) can be scanned all at once. Similarly, many aspects of TCP can be abused by attackers, for example, by retransmitting packets with slight variations, in the hope that they will be incorrectly handled by Snort. The stream4 preprocessor catches these protocol anomalies and ensures that Snort will interpret the packets as a destination host would. In addition, stream4 reorders packets and reassembles entire TCP sessions, which allows other preprocessors to easily access data as it would have been reliably delivered by TCP. These operations make stream4 one of the most important preprocessors in Snort’s arsenal.
A favorite technique for attackers to defeat simple pattern-matching rules is restructuring network data in many different, but equivalent forms. A variety of Snort preprocessors respond by converting specific protocols to standardized, canonical formats for more reliable matching by other rules and preprocessors. The http_decode preprocessor rewrites the URLs used by HTTP. Similarly, the rpc_decode preprocessor reassembles RPC data that has been split into record fragments in various ways. Attackers have also developed tools like ADMutate to generate polymorphic shell code for use in buffer overflows, and the experimental fnord preprocessor attempts to detect these; this is an area of active research.
Some stealth strategies are based on timing. For example, nmap can do very slow port scans, in the hope that probes separated by long intervals will not be noticed. The portscan preprocessor detects these by performing a statistical analysis of the traffic, and the newer, experimental portscan2 and conversation preprocessors offer potentially improved detection.
There is really only one good defense against these evasion techniques for system administrators: keep your Snort installation up to date. Install new rules frequently: fortunately, this is easy, since the rules database can be downloaded separately from the Snort web site. Upgrade Snort frequently too, in order to obtain the latest preprocessors, and be sure that the desired preprocessors are enabled in the snort.conf configuration file.
Denial-of-Service Attacks
Denial-of-service attacks can launched directly against Snort. These attacks are designed to disrupt the NIDS operation, and are in some sense complementary to evasion tactics, since they are certainly easy to detect. A barrage of traffic will at least contribute to the load on whatever machine is running Snort and will consume system resources, which makes this form of attack potentially more serious. Snort is lightweight and efficient, and can therefore tolerate high traffic rates, but other NIDS programs can be overwhelmed, and some have reportedly even crashed when subjected to these assaults.
Attack tools like stick and snot have been developed to construct random noise packets based on Snort rules. These packets are designed to match the patterns in the rules, and hence produce a flood of bogus alerts. The goal is to conceal some other genuine attack within an avalanche of false alarms. A cunning plan.
Snort retaliates by using the stream4 preprocessor: the random noise packets can be readily identified (and ignored) because they do not belong to a valid TCP session. This is a consequence of the stateless nature of the Snort rules used by the noise generators.
System administrators can best defend against denial-of-service attacks by running Snort on a separate, dedicated monitoring machine. Any extra system resources that are consumed by Snort during a denial-of-service attack will therefore not affect network services running on other machines.
If you are being bothered by stick-style attacks, use Snort’s -z est option to limit alerts to only known, established TCP connections. The stream4 preprocessor must be enabled for this mode to work.
The Latest Threats: Buffer Overflows
The worst kind of attack to fear is one that allows evildoers to hijack the operation of Snort itself. Unfortunately, two specific buffer-overflow vulnerabilities have recently been discovered that show this is possible.
The security holes occurred in separate preprocessors:
-
rpc_decode: An attacker could exploit a flaw in Snort’s computation of the size of packets, and consequently overflow a buffer, causing stack corruption. This problem was found in March 2003, and fixed by Snort 1.9.1. -
stream4: By using carefully crafted TCP sequence numbers, an attacker could cause the overflow of an integer used to check for buffer overruns, leading to heap corruption. This was exposed in April 2003, and repaired in the latest Snort 2.0 release.
Both of these preprocessors are enabled by default, and as we have seen, stream4 is critical for the stateful operation of other preprocessors, as well as for protection against denial-of-service attacks. You really wouldn’t want to run Snort with stream4 disabled.
The impact of these security holes is severe: either can be used to cause Snort to crash, or (even worse) to insert malicious code where it will be executed by Snort.
The good news is that our knowledge of these vulnerabilities and subsequent fixes are the result of security audits, and Snort is an even more valuable tool if it is ironclad. The modular design of preprocessors encourages incorporation of new code, which is an important advantage for Snort, but the latest flaws demonstrate the need for ongoing security audits, especially for any preprocessors that are enabled by default for production use.
The most obvious defensive strategy for system administrators is to keep your Snort installation up to date, and to monitor security advisories so you can apply patches for any newly discovered holes. Most system administrators are already doing this for external network services — the key here is recognize that tools like Snort are also vulnerable, and hence need attention too.
Next, don’t run Snort as root! The CERT advisory says …
“Both vulnerabilities allow remote attackers to execute arbitrary code with the privileges of the user running Snort, typically root.”
… but there is no reason to accept this risk. Although Snort (like all network-sniffing programs) requires superuser access to put interfaces into promiscuous mode (so they will act as wiretaps), Snort can and should run as an alternate user (and group, if appropriate) after initialization is complete. Use Snort’s -u and -g options to specify an unprivileged user and group, chosen to minimize risk by ensuring that only Snort’s own log files are writable.
An even better strategy is to run Snort in a chroot jail. Create a directory with all of the files that Snort needs, remembering shared libraries and so on. (Some experimentation will probably be necessary, but it is worth the effort.) Then specify the directory with Snort’s -t option. A chroot jail works best when Snort runs as an unprivileged user — this isn’t an excuse to continue running Snort as root!
Finally, run Snort on a dedicated, untrusted machine. This will help to contain the damage if Snort is ever compromised, especially if the machine contains little or no state, and can be easily rebuilt.
Vulnerability of Security Monitoring Tools
Buffer-overflow vulnerabilities are, unfortunately, still common, as shown by even a casual examination of recent CERT advisories, and the techniques used against Snort aren’t especially original or innovative. We often speak loosely of network sniffing programs (like Snort) as “listening” to traffic, so it’s easy to conclude that Snort’s recently discovered security holes are just another case of an attack on a network service. We’ve seen it all before, right?
Wrong. In contrast to network servers like sendmail, BIND, and so forth, Snort does not accept connections or receive datagrams from client programs. Instead, Snort passively observes and analyzes network traffic, and from a security standpoint it is actually similar to other monitoring tools like logwatch or web log analyzers. The network traffic plays the same role as a log file.
The easiest way to see this is to separate the packet-capture and traffic-analysis functions that are built into Snort. Although Snort is often run in a configuration that does both, this is not the only way to use the program, nor is it necessarily the most secure mode of operation.
Packet capture can be performed by a variety of network-sniffing programs other than Snort, including tcpdump and Ethereal. This is relatively safe, as long as the network data is not really analyzed, but is simply dumped in “raw” form to trace files, for example, using the command:
tcpdump -w tracefile ...These tools all use the libpcap library (provided with tcpdump), and hence the trace files share a common format, which is very convenient.
Traffic analysis can be done later, possibly using a different program, or on a different machine. For example, the tracefile saved previously could be analyzed by Snort using the command:
snort -r tracefile ...Traffic analysis is considerably more dangerous than simple packet capture, because (as demonstrated by the Snort buffer-overflow vulnerabilities), malformed network data can cause program operations to go awry.
Network-sniffing programs like Snort are often run on machines with unconfigured interfaces, and some system administrators even use modified network cables with severed transmit lines to ensure that only passive observation can occur. While these stealth techniques undoubtedly offer some degree of protection to the monitoring system as a whole, the CERT advisory for Snort warns that:
“… it is not necessary for the attacker[s] to know the IP address of the Snort device they wish to attack; merely sending malicious traffic where it can be observed by an affected Snort sensor is sufficient to exploit these vulnerabilities.”
This is somewhat misleading, however — the danger arises not from the observation but rather from the subsequent analysis of the traffic, and that could happen elsewhere.
The portability of network trace files amplifies the risks. Traffic analysis is often performed on trusted machines, deep within the perimeter protected by external firewalls. Trace files can be sent by email, and public collections are available on web sites such as the Honeynet Project, which offers periodic challenges for study of attacks observed in the wild, or the Internet Traffic Archive, which is more generic, and unfortunately, hasn’t been updated recently.
Although Snort performs more complex traffic analysis than most other network-monitoring tools, and hence offers more opportunities for attack, it’s important to recognize that the other tools are also vulnerable. Ethereal, for example, can perform TCP stream reassembly, and even tcpdump‘s simpler packet printing code is aware of many protocols. Are buffer overflow bugs lurking in this code?
Furthermore, other programs that analyze derived network data indirectly could also be vulnerable. The earlier analogy between network traffic and log files offers a clue about how this could happen. Could an attacker induce a network server like sendmail or BIND to emit a syslog message that would be crafted to compromise logwatch when it later analyzed the system log? Similarly, could a web server be tricked into recording an entry in its log file that would subsequently confuse a web-log analyzer? These kinds of attacks would be difficult because they involve multiple interacting programs, but they are certainly not impossible, and they are conceptually similar to the recently discovered Snort security holes.
Defensive Strategies for Secure Monitoring
When we recognize the threat of attacks on network monitoring tools, and indeed any other tool that analyzes network data (even indirectly), it is clear that we need to pay more attention to writing and using these tools securely, as we do for network servers. Snort’s buffer overflow security holes were discovered during security audits — similar audits for other network-sniffing programs like tcpdump and Ethereal would be welcome news, even (or especially) if they lead to the discovery of other vulnerabilities.
Data from untrusted sources (that is, the network) must be viewed with suspicion by security tools, and subjected to sanity checks before use. Programs need to verify the assumptions (known as invariants) that are critical for correct operation, that is, packet headers don’t lie about sizes, integers won’t overflow, URLs aren’t malformed, printf-style format strings contain no stray “%” characters, etc. These invariants are different for each program, so derived data that is considered sanitized by one program must still be checked by other programs to be safe. Perl provides a wonderful facility for tracking untrusted data: taint mode. Scripts like logwatch, which is written almost entirely in Perl, would benefit from use of Perl’s -T option to track tainted data read from log files. The individual logwatch filter scripts are analogous to Snort’s preprocessor modules, and they deserve a similar defensive posture.
Users of security tools and system administrators can protect themselves first by recognizing vulnerable applications. As we have seen, the most important principle is: follow the data. If it can be traced back to an untrusted source, take steps to mitigate the risk.
The best way to minimize the potential impact of decisions based on maliciously crafted data is to analyze the data in a restricted environment. This can be a dedicated, untrusted machine, or a chroot jail. At the very least, avoid running security tools as root — choose an unprivileged user (and group, if appropriate) that is only allowed minimal access to the data being analyzed, and nothing else. The data often contains sensitive information and requires protection from the general population anyway. Separating the collection of data from subsequent analysis steps can make it easier to set up these safety nets.
Finally, be wary of automated processes that are triggered by decisions based on untrusted network data, especially if the actions require privileged access. For example, Snort provides experimental “flexible response” features (fortunately disabled by default) for killing connections, and a Perl script for dynamically modifying firewall rules, both based on traffic analysis. Imagine the havoc that could ensue if an attacker managed to subvert or gain control of these actions!
Conclusion
Monitoring can be done securely, if proper attention is devoted to the safe use of our available security tools. By recognizing the vulnerabilities and taking steps to run applications in a restricted environment, we can derive the maximum benefit and enhance security, without worrying (too much) that our tools will be hijacked and used against us.
Originally published by O’Reilly’s Linux DevCenter
The Linux Security Cookbook includes real solutions to a wide range of targeted problems, such as sending encrypted email within Emacs, restricting access to network services at particular times of day, firewalling a webserver, preventing IP spoofing, setting up key-based SSH authentication, and much more. With over 150 ready-to-use scripts and configuration files, this unique book helps administrators secure their systems without having to look up specific syntax.
