Current intrusion detection systems, which typically consist of a firewall and virus detector at a gateway to the Internet, are relatively ineffective at recognizing and defending against new kinds of network attacks. Their poor performance results from their narrow view of the network, which is limited to the bits of data coming through the gateway. The detection system must guess whether incoming traffic is legitimate or is a worm or virus. Without a broader view of the network, the proportion of incorrect guesses far exceeds the percentage of accurate ones. As a result, current intrusion detection systems are dominated by false alarms.
It is a widely agreed that understanding the state of the overall network can help in determining if any given node (computer) is under attack. However, the classic approach to identifying a network's state is to gather data from each node and send it to a central location for analysis. This process is inefficient and results in flooding the network with data.
An Alternative Approach to Network Security
Intel Research is exploring an alternative, more efficient approach to network security that relies on distributed detection and inference (DDI). Their hypothesis: If sensors and machine learning algorithms were embedded in nodes across the network, each node could assess whether or not it is being attacked by a worm or virus, based on its local view, and communicate this "belief" (the probability that it is under attack) to nearby nodes.
By itself, a local belief is weak, but a collection of similar beliefs can be compelling. Imagine a single PC that is accessed by an unfamiliar remote client for the first time. By itself, this may not indicate a problem. But if every PC in the organization is accessed by the same client at different times of the day, this collective intelligence suggests that a remote computer is probing for weakness.
Similarly, by accumulating beliefs from other nodes and performing local inference on them, individual nodes can begin to infer the overall state of the network with greater accuracy. If the cumulative intelligence suggests the network is under attack, each node could take appropriate preventive action, such as logging off the network.
The objective of this approach is to outrun worms and viruses before they can cause widespread damage to the network, and to do so without flooding the network with data. Researchers believe this approach could prove far more effective in thwarting network attacks than firewalls and antivirus software installed on individual machines. The DDI approach could also enable the development of an infrastructure on which ubiquitous computing applications can be built and deployed on a massive scale.
Leveraging Research in Detecting Outbreaks of Infectious Disease
The DDI project draws on a research effort led by Denver Dash to detect outbreaks of infectious disease within a major metropolitan area. Dash, a key contributor to the DDI project, conducted the research while he was a postdoctoral research fellow at The Center for Biomedical Informatics, prior to joining Intel in the summer of 2003. He and his colleagues developed a model to capture real-time data related to emergency room admissions in the Pittsburgh area, and find correlations between patients' zip codes and other relevant data that might suggest an outbreak of infectious disease in the local population. The goal of the research was to identify an emerging epidemic as early as possible so that healthy members of the population could be alerted and steps taken to inhibit the spread of the infection.
The DDI project is applying the same underlying method (Bayesian network inference) to identify emerging worms and viruses in computer networks as early as possible so that healthy nodes can take proactive steps to avoid infection. The idea is to propagate a belief that the network is under attack as soon as individual nodes start experiencing problems, rather than waiting for the network to be overwhelmed by a worm or virus.
Propagating Beliefs Throughout the Network
Under the DDI approach, each node in the network would apply a machine learning algorithm to infer the state of the network based on local data. If the node is experiencing sluggish performance, it would transmit this intelligence to nearby nodes. Individual nodes would not be capable of diagnosing the source of problems such as sluggish performance. Rather, they would simply communicate the "belief" that there is a problem in the network.
Such beliefs could be propagated rapidly throughout the network, using an algorithm known as an epidemic protocol or a more familiar variation called a gossip protocol. Similar to how gossip is spread, each node would communicate its belief about the state of the network to a few randomly selected nodes nearby. These nodes, in turn, would pass on the belief to several other nodes in similar fashion, adding their own belief. As an individual node begins to accumulate similar feedback from several nodes, the belief becomes more compelling. Eventually, each node will have accumulated enough information to make an intelligent decision about the network's true state, based on this collective knowledge, and can take appropriate action in response.
As beliefs propagate across the network, no individual node will have access to all available intelligence, but this is not necessary. Even knowing what is happening to three or four nearby nodes provides enough data to begin making reasonable inferences about the global properties of the network.
Tuning the Intrusion Detection System
One challenge researchers face is to ensure an appropriate level of sensitivity for the distributed intrusion detection system, to avoid the problem of false positives that plagues current systems.. Any single detector is liable to issue false positives, since no detector can be perfect. So the detectors communicate their "hypotheses" with each other, seeking evidence from other systems that will help to confirm or deny the hypothesis. The approach is backed by a sound mathematical principal which ensures that, if each detector's typical behavior is known, the system can properly account for the likelihood of false positives, and use the pooled set of beliefs of a set of peers to ensure that false positives are dramatically reduced.
Research Progress
Six months into the DDI project, researchers have developed algorithms that would drive a distributed intrusion detection system, and they are beginning to run simulations of worm and virus attacks, using real data to test their approach. They are also working on several refinements to the system. First, they have simulated a system in which evidence of attacks is shared by nodes. This has now evolved into a set of algorithms which share beliefs, as opposed to evidence. Beliefs are a more concise and more useful form of sharing for systems that are attempting to infer properties of their environment. A related research project is also underway at Carnegie Mellon University, where Carlos Guestrin is applying the concept of distributed detection and inference to sensor networks.
While the DDI project is focusing initially on network security, the research could be applied to other problems as well. For example, a distributed detection system could be used by a group of PCs to infer why a given machine is unable to access a Web server. In next-generation data center architectures, it could be used to infer the best way to load balance a group of servers, by learning and inferring the expected workload at each server, and optimally assigning resources to meet the demands of each application.
