I&A for Network Security Entities

Chapter 1 recommended that each time you explore a security topic, you begin by asking simple questions: Who are the subjects? What are the objects? How does access control work? To understand how network security can succeed, you need to list the entities—the subjects and objects—that interact in the model.

How Hackers Exploit Protocols

When two layers on the same node communicate, or when peer layers communicate, they rely on a well-defined protocol and precisely stated message formats. The protocol states who initiates the connection, how the session is terminated, the order of messages, what to do if an error occurs, and other characteristics of the session that are necessary for the exchange to succeed. A protocol also can be thought of as an algorithm because most protocols are defined as state machines.

As with other algorithms, order is important. If you want to bake a cake, getting the right ingredients is just one part of the process. Mixing and cooking the ingredients in the right order are required to achieve an edible result. The same is true for network communications protocols. If one half of the session decides to get creative with the protocol, the results will not be guaranteed.

Getting the order of messages right is important, but you also need to format the messages properly. A baking recipe is useless if the order of the steps is clear, but the steps are not accurately defined. For example, if the recipe is incorrectly copied from a friend and asks for one unit of butter instead of one unit of flour, you will end up with a different dessert. Similarly, in network message exchanges, if message integrity cannot be guaranteed, any dialogue between the peers will not succeed.

Designing a secure distributed protocol is more of an art than a science. When someone at your site invents a new distributed security protocol, alarm bells should sound. Unless this person is knowledgeable and has studied a number of references, a better-than-average chance exists for the protocol to have weaknesses. Here are some examples from real-world situations:

•  A distributed authentication protocol was designed using a challenge-response technique, but the challenge and the response were the same value. A hacker impersonating the recipient could just replay the challenge when asked for the response.

•  A protocol was designed to accept incoming messages of a fixed length. Unfortunately, the program did not check the length of incoming messages (sound familiar?) and, because the system was a public Web server, any anonymous user on the Internet could crash the site.

Both of these examples appeared in commercial products developed by well-known companies with corporate offices on the West Coast. Network security is a complex beast. The merit badge for network security is earned only after years of study and trial-and-error. Designing network protocols is tough. Designing a new network security protocol is for experienced practitioners only.

How Many Network Entities Are There?

How do you think it is possible for peers at layer N on communicating systems to implement appropriate security? Well, you know that the first requirement is to have some form of information that uniquely identifies the peer layer on each node (the subject). In some cases, the identifying information could be the same as that used at other layers in the network. More often, an entity’s identifying information is meaningful only at one particular layer in the model. Unlike the single system that you investigated in Chapters 2 and 3, networks can contain a complex collection of different entities.

You should take a few moments to digest that last paragraph. If you think about the simple three-layer model introduced previously, you can see that up to six different entities can exist just to support the example. There are two network nodes. Each node has three layers. The two nodes communicate using a protocol, and peer layers between nodes communicate using unique protocols. That makes six different entities. Each one must be uniquely identifiable. Each entity also must be authenticated. Compared to a login session for a single user on a system, there are quite a few more places where things can go wrong in network security.

A peer layer does not necessarily rely on the entities of the operating system for its security. For example, an e-mail application runs above the operating system on a computer. On any two nodes in the network that support this mail protocol, some form of I&A exists, which at a minimum identifies the recipient of a mail message. If standard UNIX mail is used, the identities are the same as those found in the password files. If the application is Lotus Notes, the users are entirely different entities from those found in /etc/passwd. Lotus Notes relies on a client-server protocol in which the users are defined on the server and each user has a public key certificate. Therefore, the mail application uses a different security model than the network stack itself. This is a perfect example of how multiple security models can reside on the same system at the same time. Each security model offers opportunities for hackers for the two familiar reasons:

•  Something can be configured improperly.

•  The implementation of one of the models is flawed and is thus susceptible to protocol attacks, buffer overflow attacks, or other hacks.

How do you know when you have an application security model in addition to your operating system security? The distinction between an operating system and an application is often blurry. Some applications are delivered with the operating system and work with operating system security entities such as users and groups. When these applications communicate across a network, other entities become relevant, such as the network address of the computer. For example, the remote terminal application telnet is a standard feature of most UNIX operating systems. Telnet communicates network addresses as part of its protocol. Users and groups in the usual system files are consulted during authentication. The telnet client is also part of NT. (However, inbound telnet is not a standard feature of NT 4.0 systems.)

In general, if you are forced to define new users and groups and enter access control rules using techniques different from the operating system, you are adding a new security model to your system. This new domain of subjects, objects, access control rules, and administrators is one more thing that you must diligently monitor for attacks.