Page 6 of 11

ADDRESSING DESIGN DECISIONS AT INTEL® ONLINE SERVICES

In this section, we discuss some of the implementation choices made at Intel Online Services that were driven by IP address space concerns. We first talk about the choice of IP address space. We then discuss remote access implementations, and conclude with a description of virtual web site implementations.

Addressing Choices
As we mentioned above, some types of address space are more appropriate than others in different situations. At Intel Online Services (IOS) , we use a hybrid addressing approach for the data center network that uses the most appropriate type of address space depending on the purpose of the host. We use private IP addressing for the data center internal networks and for customer servers that do not need to talk to the Internet. Since our internal networks do not need to talk to the Internet, there is no need to use precious public space. Also, back-end servers that do not need to talk to the Internet gain a measure of security because they are impossible to access directly from the Internet. (They are not immune to attack, however.)

Links to transit ISPs and other ISPs that IOS is peers with uses the address space of those ISPs on the router interfaces of the links. Since the IP address will need to change if the ISPs change and will go away if the ISP is no longer used, use of their space in this situation is not a problem and helps conserve IOS public address space.

For data center hosts and routers that need to communicate directly with the Internet, we have used a variety of address spaces. In North America, we use a class B (/16) that was made available to us. Using available public space allows us to have address space independent of our ISP selection, and it makes multihoming to multiple ISPs much easier. For our data centers in Asia and Europe, we have obtained space from the regional address registries that we own. Using the available class B was not feasible because some ISPs will not accept router advertisements of pieces of traditional class B address space. If we choose to use this space, we would have had to use the same ISPs in Asia, Europe, and in North America in order to have any kind of data center-specific routing policy. The process of obtaining new space took significant effort, but it is well worth it to have address space independent of ISP choice and the ability to multihome.

Size of Address Space Advertised
While IOS data centers are designed to handle thousands of hosts, not all of the hosts need to communicate directly with the Internet. Each data center advertises /19 networks, providing address space for up to 8192 hosts. /19 is the longest prefix that some ISPs will accept. For the data centers in North America using parts of a traditional class B, we also need to advertise the entire /16 network out of two data centers in order to deal with ISPs that do not accept parts of traditional class B networks. These ISPs will route traffic to IOS’s ISPs, which will then route it to the individual data centers.

Autonomous System Number Choices
These choices are similar to our choices of address space. For North American data centers, IOS uses an autonomous system number that it had available. Separate autonomous numbers for each data center were considered wasteful, and in the North American environment, not particularly necessary. For IOS Asian and European data centers, we have obtained AS numbers for APNIC and RIPE, respectively. This allows the data centers independence in ISP selection, and it avoids any possible routing policy conflicts with other data centers on different continents.

Address Allocation Policy
For security and ease of management, IOS has chosen to place each of its customers on separate segments. In doing so, IOS enforces requirements for address utilization that mirror those of the address registries mentioned above. This positions IOS to be able to meet the utilization requirements of the registries when IOS requires more space.

To meet those requirements, IOS uses variable length subnet masks extensively. VLSM allows IOS to assign the appropriate sized subnet to a customer while maintaining utilization requirements. One consequence of this is that for IOS infrastructure hosts that need routing information sent to them, a routing protocol that understands VLSM needs to be used. This eliminates RIP version 1 protocol [3] and basically limits the routing protocol used by hosts to OSPF [4] and RIP version 2 [15].

Remote Access Implementation
IOS data centers offer a variety of remote access options such as Virtual Private Network (VPN), ISDN, and dedicated leased lines (T-1, T-3 etc.) to customers as a way of providing access directly from their networks into their servers hosted at IOS. Some of these options, such as LAN-to-LAN VPNs and leased lines, create network channels into the data center across which customer-end network addressing, that may very likely be in the private IP ranges, becomes visible. In this section, we focus only on these remote access options.

Allowing customer-end IP addressing, whether it is in public or private IP space, inside the data center network makes routing extremely difficult. The data center routing policy needs to account for routing network traffic appropriately to multiple customers’ home networks. Furthermore, across multiple customers, the networks at their ends can be spread all across the public and private IP address ranges, leaving little scope for summarizing networks and as a consequence leading to larger routing tables. In addition, private IP conflicts across customers as well as the data center networks need to be resolved. Considering all the above challenges, we made a decision to hide customer-end internal networks from data center internal networks.

In this light, let us discuss the salient features of IOS’s remote access infrastructure, which is logically represented in Figure 3.

Figure 3: Intel Online Services remote access infrastructure for data center customers

The remote access layer constitutes the devices that provide the various remote access technologies supported at each data center. The firewall layer has devices that enforce security policies, among other things, on remote access traffic. The firewall layer also employs many-to-one as well as static NAT extensively to ensure that customer-end networks are not visible beyond the network aggregation segment. An exception to this rule is when a customer wants to run applications such as the Distributed Component Object Model (DCOM) [16] that does not work across NAT boundaries.

The actual design of the end-to-end network communication, across the above infrastructure, for various customers is dependent on the following factors:

• private IP addressing conflicts with other customers’ networks
• where the network communication is initiated
• whether the applications that need to be run can work across NAT boundaries

Scenario 1: Customer-end network does not conflict with any other customer’s home networks or with data center internal networks. All network communication needs to be initiated from the customer end and inbound into the data center. All required applications work well with NAT.

This is a simple case and can be resolved by translating all of the customer network to one IP address (many-to-one NAT) from the data center’s IP addressing range. Each customer network that fits this scenario is translated to a different IP address. As discussed earlier in the NAT section, many-to-one NAT does not allow for communication to be initiated in both directions. In this design, servers in the data center cannot initiate communication back to customer-end systems.

Scenario 2: This is the same as Scenario 1 but it requires that communication needs to be initiated from the data center to a few customer-end systems.

In such situations, we still create a NAT rule to translate all of the customer’s networks to one single IP address. In addition, we create static one-to-one NAT relationships between specific data center’s IP addresses and specific IP addresses of systems at the customer end. Hence, bi-directional initiation of communication is allowed only for specific customer-end systems.

Scenario 3: Customer-end internal networks conflict with another customer’s home network but do not conflict with data center IP address space. Communication is always initiated from customer end to the data center and all required applications work well in spite of NAT.

To resolve the private IP address conflicts here, we ask customers to perform NAT at their end to hide their internal network from IOS behind either non-conflicting private IP address space or public address space. Furthermore, we translate that IP address space, at the firewall layer in the data center remote access infrastructure, to a single IP address from the data center’s IP address space.

Scenario 4: This is the same as Scenario 3 except that communication needs to be initiated from the data center to some systems at customer end.

For allowing needed end-to-end communication here, we create static one-to-one NAT relationships for the specific systems at the customer end in addition to the solution implemented for Scenario 3. If customers are using many-to-one NAT on their end as well, they will need to set up corresponding one-to-one NAT relationships on their end in order to allow connections to be initiated from the data center to systems at their end.

Scenario 5: Customer-end networks have private IP conflicts with data center’s private IP space.

With the exception of catering to applications that do not work across NAT, we have decided to allocate the data center’s public IP space to the internal networks that such customers would need to reach at the data center. Customers in turn will need to perform NAT at their end to translate their networks to a non-conflicting IP space. This non-conflicting IP space is still hidden from the data center for reasons discussed earlier.

Scenario 6: Customer needs to run applications that do not work across a NAT boundary across one of the remote access channels.

In such cases, we must ensure that the data center’s networks as well as the customer-end systems that need such communication use public IP addressing. In addition, we have to accommodate the routing for customer’s public IP address range in the data center routing policy as it is not possible to hide customer-end networks behind NAT here.

Other Options: End-to-end connectivity design, across the remote access channels discussed in this section, can get quite complex and potentially difficult to troubleshoot. IOS also provides remote access options, such as client-to-LAN VPN and ISDN that provide direct connectivity from an individual desktop system to the data center network and are devoid of design complexities of the remote access options discussed so far. These remote access options hide the customer end IP addressing from the data center network, eliminating the possibility of address conflicts and the need for technologies such as NAT. Instead, these technologies assign an additional IP address from a designated pool of addresses in the data center’s IP address space to the customer-end system while the system is connected. This IP address is used for all communication with systems within the data center. Applications that carry IP address or application port information in the packet payload work without any issue across these remote access channels, since the IP address and port information never changes anywhere between the customer desktop system and the systems in the data center.

This option works well for customers who travel a lot and access their systems from a number of locations or for customers who do not need continuous access to their systems. It does not work well for customers whose connectivity to their systems in the data center must be up all of the time.

Virtual Web Server Implementations
IOS uses the HTTP 1.1 host header technique [18] mentioned above for virtual web site public addresses. This means for m virtual web sites and n web servers, only n+1 public addresses are used. With this implementation, the number of IP addresses required is not sensitive to the number of virtual web sites. Some IOS customers have web usage analysis packages that require that different virtual web sites have different IP addresses. IOS minimizes the impact on address space from such customers by mapping IP virtual addresses to web servers on ports other than the HTTP standard port 80. A different web server virtual IP address might map to port 81 on the web servers, while a different virtual site’s virtual IP address might map to port 82 on those same servers. In this way, for m virtual web sites and n web servers, m + n public addresses are used. This is not as good as the host header implementation, but is much better than the IP address per web server implementation.