Deep configuration assessment

Large, dynamic and complex IP networks are increasingly being used for real-time and critical services. To provide service assurance in such networks, deep configuration assessment is a new approach that is being discussed in the recent publications.^[1][2] Deep configuration assessment is required to detect and help remediate configuration errors in IP Networks that adversely impact security, reliability, Quality of Service and regulatory compliance. Deep Configuration Assessment provides significant cost-savings for organizations that depend on IP Networks, by reducing the labor required for debugging IP Network configurations and reducing the cost of network security and/or downtime incidents.

IP Networks are Large, Dynamic and Complex

IP networks are increasingly assuming major roles throughout the public and private sectors. Every day, they are responsible for transporting real-time and critical voice, video, and data traffic. As a result, it is no longer acceptable for IP networks to deliver “best-effort” service. They are expected to perform at carrier grade level. However, it is enormously challenging to deploy and operate IP networks at consistently high-quality since they are large, complex and dynamic. IP networks consist of devices such as routers, switches and firewalls that are interconnected by network links. These devices are not “plug-and-play”. Rather, they must be provided with specific instructions, also known as scripts or configurations, which indicate exactly how they are to interact with each other to provide the correct end-to-end IP network service. This is why IP device configurations can be considered as the DNA of the network, as they literally control the network’s behavior.

Unfortunately, there is nothing simple or standard about these configurations. Each one must be manually programmed into the network devices, and every vendor uses a different configuration language for its devices. Device configurations change virtually every day in response to new application deployments, organizational or policy changes, new device or technology deployments, device failures, or other reasons. Device configurations have an average of 2000 lines of code for each device. A Fortune 500 enterprise that relies on IP technology can easily have over 50 million lines of configuration code in its network.

However, the numbers of devices and lines of code are only part of the problem. Each device configuration can contain hundreds of parameters for about 20 different IP protocols and technologies that need to work together. Those protocols and technologies must satisfy various and constantly changing service requirements, some of which are inherently contradictory, such as security and connectivity with the Internet. Configuration errors can easily occur due to entry mistakes, feature interaction, poor process, or lack of a network-wide perspective.

Configuration Errors are Common and Have Adverse Impact

The high-frequency and adverse impacts of configuration errors are well documented. BT/Gartner^[3] has estimated that 65% of cyber-attacks exploit systems with vulnerabilities introduced by configuration errors. The Yankee Group^[4] has noted that configuration errors cause 62% of network downtime. A 2009 report^[5] by BT and Huawei discusses how service outages caused by “the human factor” themselves cause more than 30% of network outages, “a major concern for carriers and causes big revenue-loss. Especially for the future All IP network, this problem is becoming increasingly serious”.

Operational experience with IP network assessments of a variety of organizations provides evidence from the field that configuration errors are very common and impactful. As many as five critical errors are discovered per IP network device. This is not surprising since these errors are hard to detect, requiring the simultaneous validation of multiple protocols and device configurations. The errors typically remain latent until they are exploited by cyber-attackers, discovered by auditors, or result in network failures. The errors discovered in various organizations have significant impact on IP network security, availability, Quality of Service and regulatory compliance. A few of these errors and their potential impact are now discussed.

Security

The most obvious configuration errors in this category can be found in firewalls, in the form of “holes” inadvertently left in firewall configurations. These “holes” are rules that permit specific application traffic to pass temporarily through the firewall and are not removed after they are not needed. Cyber-attackers scanning enterprise networks discover these “holes” and craft their attacks on the enterprise infrastructure through these “holes”.

A good example^[6] of how firewalls could be exploited to cause major financial and brand-related damages is illustrated in recent news that U.S. prosecutors have charged a man with stealing data relating to 130,000,000 credit and debit cards. Officials call it the largest case of credit card theft in American history and accuse one man with two un-named co-conspirators of hacking into the payment systems of large commercial retailers. The actual attack method was SQL injection, but the accused could launch it because the firewalls were not configured properly. It is difficult to secure application code against SQL code injection attack, but at least the firewalls should be properly configured to stop attackers from detecting SQL applications and attempting such attacks without detection.

Apart from the obvious firewall holes and mis-configurations, other examples of errors that impact security include:
• Static route on device does not direct application traffic into IPsec tunnel. This results in sensitive traffic remaining unprotected as it transits the network instead of flowing through the secure IPSec tunnel.
• Best-practices for Virtual LAN (VLAN) security, such as disabling dynamic-desire and using root-guard and BPDU-guard on switch access ports, are not followed. Leaving the dynamic-desire VLAN feature enabled in a switch allows an attacker that connects to the switch to monitor all traffic passing through the switch.
• Link left active between devices. If the devices belong to network segments that are not meant to have a direct connection, then a “backdoor” has been introduced that can be exploited by attackers.
• Mismatched IPSec end-points. This results in sensitive traffic remaining unprotected as it transits the network instead of flowing through the secure IPSec tunnel.
• Adequate authentication is not used between devices for exchanging routing protocol information. An attacker can connect to a network device and extract or inject spurious routing information.

Reliability and Availability

Organizations that depend on the IP network to provide a very reliable service have to ensure there are no single points of failure in the network. It is not sufficient to just provide redundant network devices and links at the physical level. It is also critical to ensure that the configurations of the network devices make use of the available redundant physical resources and that the redundancy is ensured across multiple layers.

A good example of the enormous damage that configuration errors can cause is the recent Federal Aviation Administration incident,^[7] where some critical systems went offline for four hours due to IP router mis-configurations, causing hundreds of flight delays and cancellations.

Other examples of mis-configurations that result in single points of failure include: • Mismatched device interface parameters. This mismatch prevents devices from establishing logical connectivity even though physical connectivity exists.
• Standby Routing Protocol (HSRP) inconsistently configured across two routers that are expected to mirror each other. The standby router will not take over when the main router fails.
• Access Control Lists (ACLs) or firewall rules stop specific application traffic on a path. So even if the path provides redundancy in general, the ACLs/rules remain cause for a single point-of-failure to exist for the specific application traffic.
• Use of a single Open Shortest Path First (OSPF) Area Border Router (ABR). The OSPF areas that are connected by the ABR will become isolated if the ABR fails.
• Multiple VPN connections sharing a single physical link or device. Redundancy expected from the multiple VPN connections is not provided due to their dependence on a single physical resource.

In addition to errors that introduce single points of failures described above, other errors in configuration of IP routing protocols such as OSPF, Border Gateway Protocol (BGP), Multi-Protocol Label Switching (MPLS) and Intermediate System-to-Intermediate System (IS-IS) can also impact network reliability. Examples of such errors include:
• Inconsistent routing parameters such as OSPF Hello and Dead interval across multiple routers. OSPF will not function efficiently if such parameters are inconsistent, resulting in ephemeral traffic loops and poor network performance.
• Best practices proposed by vendors and experts for routing protocols, such as use of a full-mesh to connect all internal BGP (IBGP) routers and OSPF route summarizations including IP addresses of all interfaces except the loopback interface of a router, are not followed. This generally results in an unstable network with intermittent connectivity issues that are difficult to debug.
• Use of inappropriate IP addresses, such as addresses assigned to other organizations or private addresses in parts of the network directly exposed to the Internet. Such networks will start advertising routes for IP addresses they do not own, resulting in Internet routing issues.

Quality of Service (QoS)

IP traffic with demanding network latency and packet-loss rate requirements, such as iPhone, Voice over IP (VoIP) and financial services applications, requires appropriate Differentiated Services and other QoS configurations in the network devices. In a large network, it is easy to make errors in the QoS configurations. Examples of errors include:

• Incorrect bandwidth or queue allocation on device interfaces for higher priority traffic. During high-load periods, higher priority traffic will not receive its due bandwidth or queue, resulting in higher latency or packet-loss.
• Inconsistent QoS policy definitions and usage across multiple devices. The same QoS policy may be implemented differently across multiple devices, resulting in application traffic receiving different treatment at the different devices, which can impact latency and packet-loss during periods of high-load.

Regulatory Compliance

The world’s growing reliance on IP and the highly networked nature of government computing environments have also motivated a wave of regulations to improve security, reliability and QoS. In the U.S., the Federal Information Security Management Act of 2002 (FISMA) requires U.S. federal agencies to develop, document and implement security programs. An implementation guideline issued by the Office of Management and Budget (OMB Circular A-130) establishes among other things a minimum set of controls to be included in automated, inter-connected information resources.

The National Institute of Standards and Technology (NIST), a non-regulatory federal agency within the U.S. Department of Commerce, has promulgated security requirements for protecting the confidentiality, integrity and availability of federal information systems and the information handled and transmitted by those systems. NIST’s Guideline on Network Security Testing (SP 800-42^[8]) recommends security testing as a routine part of system and network administration and directs organizations to verify that systems have been configured based on appropriate security mechanisms and policy. Other U.S. federal legislation enacted into law, such as the Sarbanes-Oxley Act and Health Insurance Portability and Accountability Act (HIPAA), fuel the push for network protection.

Outside the U.S., organizations such as the British Standards Institute (BSI), International Organization for Standardization (ISO) and Information Technology Infrastructure Library (ITIL) recognize the complexity of IP networks and the importance of security. The BSI publication Delivering and Managing Real World Network Security explains that networks must be protected against malicious and inadvertent attacks and “meet the business requirements for confidentiality, integrity, availability, non-repudiation, accountability, authenticity and reliability of information and services.”

Examples of network-specific aspects of regulatory compliance are:

FISMA:
• AC-3 Access Enforcement: The information system enforces assigned authorizations for controlling access to the system in accordance with applicable policy.
• CM-6 Configuration Settings: The organization configures the security settings of information technology products to the most restrictive mode consistent with information system operational requirements.
• CP-8 Telecommunications Services: The organization identifies primary and alternate telecommunications services to support the information system and initiates necessary agreements to permit the resumption of system operations for critical mission/business functions within [Assignment: organization-defined time period] when the primary telecommunications capabilities are unavailable.

HIPAA:
• Security Standards: Ensure the confidentiality, integrity and availability of all electronic protected health information the covered entity creates, receives, maintains or transmits.
• Evaluation: Perform a periodic technical and nontechnical evaluation.
• Transmission Security (Integrity Controls): Implement security measures to ensure that electronically transmitted electronic protected health information is not improperly modified without detection until disposed of.

PCI Security Standards Council, LLC Data Security Standard (PCI DSS):
• Requirement 1: Install and maintain a firewall configuration to protect cardholder data.
• Requirement 4: Encrypt transmission of cardholder data across open, public networks.
• Requirement 10: Track and monitor all access to network resources and cardholder data.

Existing Solutions are Ineffective in Detecting the Configuration Errors

Network configuration and change management (NCCM) solutions deployed by many organizations today are ineffective in detecting IP network configuration errors. The high incidence of configuration errors supports this observation. This is because NCCM solutions perform the role of “version control” for network configurations, and do not provide the testing and debugging capability that is needed to detect and remediate errors.

The labor-intensive and constantly changing nature of IP network configurations is analogous to software development. The key difference is that software development has matured to the point where errors are significantly reduced since testing in software development is a well-established process, while there is no similarly rigorous testing process in IP network configurations. The NCCM solutions are analogous to version control solutions used to develop software, which provide the ability to create new configurations and apply them to devices, and perform backups and roll-backs.

The testing and debugging of IP network configurations require new solutions that provide Deep Configuration Assessment, which is beyond the capability of NCCM solutions.

References

1. ^ http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fstamp%2Fstamp.jsp%3Farnumber%3D4808469%26isnumber%3D4808465&authDecision=-203
2. ^ http://www.springer.com/computer/communications/book/978-1-84882-827-8
3. ^ British Telecommunications, “Security and Business Continuity Solutions From BT: Thriving in the Age of the Digital Networked Economy”, 2004.
4. ^ Zeus Kerravala, “As the Value of Enterprise Networks Escalates, So Does the Need for Configuration Management,” The Yankee Group: January 2004.
5. ^ Carl Colwill, Alpha Chen, Himanshu Pant, “Human Factors in Improving Operations Reliability,” n.d., <http://www.ieee-cqr.org/2009/FINAL%20UPLOAD/DAY%203%20-%20THR/HIMANSHU%20PANT%20-%20Human%20Factors%20In%20Improving%20Operation%20Reliability_FinalMay12.pdf> (September 8, 2009)
6. ^ BBC News, “US man ‘stole 130m card numbers’”, 18 August 2009, <http://news.bbc.co.uk/2/hi/business/8206305.stm> (September 8, 2009)
7. ^ http://www.aviationweek.com/aw/generic/story_generic.jsp?channel=none&id=news/ATCOUT111909.xml&headline=New%20Telecom%20Setup%20Blamed%20For%20U.S.%20ATC%20Outage.
8. ^ http://csrc.nist.gov/publications/nistpubs/800-42/NIST-SP800-42.pdf