In Chapter 2, “The Role of Identification and Authentication in Your Environment,” you saw ways to improve the security of I&A by using techniques or tools that defended against threats such as password cracking and network sniffing. It’s clear that access control is needed beyond I&A for a complete security model, but why do you need intrusion detection in addition to I&A?

•  Even improved authentication products, such as Kerberos, have been hacked. Weaknesses in the protocol have been described in Chapter 2 and elsewhere (Bellovin and Merritt, 1991; Dole, Lodin, and Spafford 1997; Mudge 1996).

•  In 1997 the integrated Solaris-DCE login facility also had a serious flaw that rendered I&A untrustworthy. A similar flaw appeared in Silicon Graphics’ IRIX operating system in 1998.

•  Early versions of Security Dynamics ACE server also had problems that you saw in Chapter 2.

•  A flaw in AIX rlogin allowed remote users to gain root access.

There are many other examples of failed I&A subsystems. Because bugs or loose adherence to corporate security guidelines will always exist, I&A will not prevent all hacks. You must at least monitor the activities of users, including simple events such as failed login attempts, in order to detect problems. Preferably, you want to detect attacks in real time and have some automated responses to provide a scalable solution. Deploying an IDS that can detect attack patterns in I&A event data helps you get a handle on your security problems.

To reiterate a theme introduced in the opening chapter of this book:

Good security requires prevention, detection, and responses.

Beyond Access Control

There are similar concerns about access control mechanisms that are responsible for preventing unwanted actions. For nonnetwork resources, such as files, directories, devices, and IPC data structures, access control is designed to limit how subjects and objects interact. To effectively carry out its responsibilities, the reference monitor needs an access control database that is properly configured with the security policy to enforce. This database is the first place things can go wrong. As you saw earlier, either the vendor or your site administrator can improperly configure access control rules (or other aspects of your security policy) that lead to compromises. Remember, properly specifying access control rules for files and directories is an exceedingly complex task as the number of subjects and objects grows.

Next, if there are any bugs in the reference monitor itself, access control will not prevent violations of the policy. Although buffer overflows in privileged programs are not the fault of the reference monitor, these flaws are used to bypass the access control policy defined for the system. Perhaps the greatest latent threat is the large number of home-grown applications or custom programs that contain bugs or configuration errors which can lead to intrusions. As more enterprises connect these legacy back-end applications to front-end Web servers, the risk of penetrations increases.

When your system access control policy is violated, you also want to be able to detect the activity as soon as possible and have a scalable solution for responses. An IDS can add value here. An IDS is designed to detect and respond to attacks that get past your access control systems. The same is true for network access control.

Beyond Network Security

How could it be possible that firewalls and encryption techniques are not enough? A few examples are worth walking through in detail.

In Figure 5.3, a packet-filtering firewall has been configured to allow HTTP traffic to travel in both directions. Two example hacks that can flow through this pipe are test.cgi and phf. Even if these two CGIs in particular are not running at your site, and hopefully they are not, there is always a risk that some internal CGI program has an exploitable weakness.

Figure 5.3  Intrusions unaffected by access control in the firewall.

An increasingly common configuration is shown in Figure 5.4. Here, the perimeter network contains a Web server that must contact a business back-end server to complete the interaction with the customer. In these configurations a proprietary gateway program often communicates through the firewall with the back-end server inside the trusted network. The next few paragraphs describe some problems with this scenario.

Figure 5.4  Gateways are paths for intruders.

Even if customers are using digital certificates to authenticate to the Web server, this same credential is not necessarily meaningful in the security context of the database. The Web server and the database are separated by a security boundary with different subjects, objects, and ACLs. Some customer sites have granted the gateway program, running on the Web server, unlimited access to the database. In other words, when the gateway program connects to the database, it does so with the highest privileges when accessing the database. This programming choice alone should be enough of a reason to run an IDS on the database server.

The gateway program and the back-end server establish a client-server or peer-to-peer relationship. At a minimum, they communicate using a network protocol. Administrators know that certain Internet application protocols are not safe to punch through the firewall. However, these same conscientious employees often will allow proprietary protocols used by the gateway and back-end server to flow through the firewall. People realized weaknesses in some of the Internet application protocols because flaws were discovered by hackers or researchers over a period of years. These protocols are not allowed in the perimeter or through the firewall because they are flawed. The same type of introspection is warranted for private protocols that exist at your site, although private protocols are seldom given the same type of scrutiny. Private application protocols between the perimeter and the trusted network, or those run totally within the trusted network, are also potentially open to attack. Only by monitoring the activities of the participating nodes with an IDS can you be sure that your security policy is not weakened by proprietary application protocols deployed at your site. Although the IDS may not look at the protocol itself, it will detect improper activities on the system that result from weaknesses in private protocols.