Intrusion Detection: Network Security beyond the Firewall
(Publisher: John Wiley & Sons, Inc.)
Author(s): Terry Escamilla
ISBN: 0471290009
Publication Date: 11/01/98

Previous Table of Contents Next


This layered view, though simplistic, is important because at each level you are able to discern security activities not apparent at other levels. Each of these three layers can provide data or instrumentation that describes activities which the other layers cannot detect or understand easily. For example, the operating system knows when someone fails to enter a password correctly. This event is recorded in at least one place, perhaps in more than one place, depending on the amount of logging you have initiated. On the other hand, the network deals primarily with the task of receiving or sending network packets.

When packets are received, the network layer is responsible for routing the packets to the appropriate listening programs. When packets are sent to the network layer by a program, the packets are sent down onto the physical network layer for delivery to the target. The network layer itself is not responsible for controlling whether someone enters a correct password.

Similarly, an application with its own database of user IDs knows whether one of its application-specific users has failed an authentication step. In the ideal case, the application can reuse the repository of users and groups found in the operating system, including the possibility of relying on the operating system’s authentication programs. The trend is for this to occur more frequently on NT systems than UNIX systems, although this is just an opinion. Such tight integration between an application and the OS is rare, and the more common approach is for the application to introduce its own repository for users and groups. When this happens, the OS itself has no knowledge of successful or failed login activities completely under the control of the application.

Warning: Opportunities for Hackers!

The Internet is often used as a replacement for private leased lines. Proprietary client-server protocols that have been operating across private leased lines or LAN connections are sometimes being deployed across the Internet. Unfortunately, the implementers of these solutions don’t always understand the limitations of their protocols.

Why do firewalls block packets? The decision to block a particular type of network traffic is based on the history of that protocol. If the protocol has been hacked in the past, or if it divulges too much information about the private network, the packets are blocked. Examples include source-routing options on IP packets, RPC, most UDP (although this is changing to some degree), finger, Berkeley r-cmds, and NFS. These protocols are not allowed through the firewall because they are deemed to be unsafe. Years of pounding on these protocols by thousands of users have contributed to the decision to avoid running these protocols across untrusted networks. Sometimes, the protocol is sound, but the programs have bugs (sendmail is a good example) in the way in which the data sent in the protocol is processed.

Do you think the same rigorous, robust analysis has been applied to the private protocols that are now being pushed across the Internet? It’s unlikely. Still, the risk seems to be worth the reward. Previously unconnected sites now can share data by transferring data through a socket opened across the Internet. The glamor is too alluring to wait for detailed protocol reviews by security experts. Yet, each of these private protocols presents an opportunity for an attacker to enter your private network, hijack a session, or launch data-driven attacks against the endpoints of the socket.

Many customers do not want to open additional ports through the firewall. Sites may allow HTTP and SMTP through the firewall, but nothing else. Other sites depend on proxy servers for outbound sessions such as FTP, telnet, and Gopher. Some application programmers know about these limitations and are afraid to ask the firewall administrators to open a new port for a proprietary protocol between two enterprise sites. One trick is to write the application so that the endpoints communicate on port 80 (the HTTP port). Ouch! Essentially, the developers are using port 80 as a general-purpose hole through which to punch private protocols.

The problem with this style of thinking is that the fundamental purpose of the packet filter is not understood. The security of the network traffic is not determined by the port number but rather by the protocol that flows through that port on the firewall or by the receiving applications. HTTP can be set up to run on any port, and the security of the protocol is unaffected by this configuration change. To take this reasoning to an extreme, wouldn’t it be just as reasonable to take an unsafe protocol such as rsh and run it through port 80, too? It’s hard to believe someone would answer, “Yes,” to this question.

Commercial IDS Layering

In the commercial IDS marketplace, it is common to find a particular product focusing primarily on network or system monitoring, with modest support for application-level monitoring. Often, this product differentiation strategy is made for competitive reasons. That is, vendors have historically chosen to focus on monitoring system activities or network activities to ensure a positive business case for the company. Like much of the computer security industry, this practice is changing, too, as the need for complete solutions drives requirements. IDSs that process network data, system data, and application data are found in research systems such as EMERALD and NIDES (www.sri.com/csl).

Over time this situation is bound to change. Commercial vendors will combine or cooperate. A strong need exists for a cohesive network and system-level intrusion detection product.

Intrusion detection on a system cannot be solved at a single layer of that system. Here, the layers are simply the network, the operating system above the network, and then the application layer as shown in Figure 6.1. A general rule of thumb is that the component with the most knowledge about the data being attacked is best suited for housing the intrusion detection logic. IP address spoofing, SYN Flood, and Teardrop attacks are detected by the network software layer because these attacks exploit data with meaning at that level of the stack. Buffer overflow attacks executed by users with an active login session are detected by the operating system-level intrusion detection monitor because these events are not generated at the network level. You might argue that if the users are connected over the network, then any keystrokes are visible at the network layer. A network IDS could analyze keystrokes looking for buffer overflow attacks. In research and practice, this approach has not been feasible, and it especially is impossible when the network traffic is encrypted.


Previous Table of Contents Next