Page 4 of 11

ADDRESS SPACE DESIGN ISSUES

When implementing an Internet data center, there are a number of key decisions that need to be made about IP address space. In this section, we discuss four key design points: what address space to use, the size of address space to advertise for a data center, what autonomous system number to use, and the IP address allocation policy. For each of these design issues, we talk about the different choices available and the tradeoffs involved with each choice.

What IP Address Space to Use
The first critical choice that data center implementers have to make is what IP address spaces to use. There are several choices here: 1. Private IP address space 2. Currently owned and used space 3. Space from the data center’s ISPs 4. New space obtained directly from Internet registries 5. Customer space

These choices are not mutually exclusive within a data center, and we will go over the tradeoffs involved with each choice.

Private IP address space has the primary advantage of being plentiful and immediately available. It has the primary disadvantage of not being immediately usable on the Internet without some form of Network Address Translation (to be discussed later) or some kind of proxying technique. This disadvantage is at times not a problem. For applications and services that do not require direct access to the Internet, this is not a concern. Also, for hosts such as database or application servers that do not directly talk to other systems on the Internet, this has some security advantages, as these systems are not vulnerable to direct attack from the Internet. (Do not think that they are invulnerable because of this, however).

If data center implementers already have their own IP address space, another possibility is to use that space. This can be advantageous, as an organization may have plenty of address space to utilize immediately. The prime disadvantages can be with routing. For example, some ISPs do not accept network prefixes longer than a /16 for parts of traditional class B networks. So if you wanted to use part of a /19 part of a traditional class B for a data center, that data center would not be accessible from all ISPs.

Another option is to use space from an ISP. ISPs have address space that they will provide to customers. While this option has the advantage that there is address space to use immediately, this option has a number of powerful disadvantages. The first disadvantage is that using an ISP’s address space will typically limit you to using one ISP. That address space is bound to that ISP, and you typically will not be able to have traffic routed through another ISP. Another disadvantage is that should you choose to discontinue your service with that ISP, you would have to give back all of the space you received, forcing you to renumber all of the hosts directly accessible from the Internet.

Another option is to obtain new address space directly from the Internet registries that distribute space. This option has a number of advantages. A data center using space obtained from the Internet registries can change ISPs without having to worry about renumbering hosts. The registries usually allocate addresses in /19 blocks, making that address space immediately routable. The disadvantage of getting space from registries is that the process takes time, on the order of weeks, and longer if you need to first join the registry. The process also involves rigorous justification of address allocation and why currently owned addresses will not suffice. In addition, once space is allocated, it cannot be used all at once. To use more of an address allocation, another justification process is required, often requiring verification that previously assigned addresses have been used.

A final choice is using the data center’s customers’ address space. When possible, this is good, but it is often not possible, as Internet data center customers usually expect that you will use your own address space. Even if a customer is willing to do this, it may take some time to make changes to the data center’s ISP’s address filtering to make it possible to use that space. Also, customer address space is also vulnerable to the same problems as address space you already own. The customer may use a piece of existing address space, such as part of a class B, that some ISPs may refuse to accept as a route.

Size of Address Space to Advertise from the Data Center
A decision closely related to what address space to use is what size of address space to use and how to advertise it on the Internet. Clearly, you can only advertise the space that you have: that puts an upper limit on the address space advertised for the data center, and thus a lower limit on the prefix length of the network advertised. Many ISPs will not accept route advertisements for networks with prefix lengths longer than /19, which puts an effective upper limit on the prefix length of what you advertise and a lower limit on the size of the address space.

There are a number of factors that affect the size of the address space advertised. First, it depends on how many hosts within the data center need public addresses. You want to advertise enough address space to cover the hosts that need public addresses, both immediately and in the near future. To make routing more manageable and to help reduce the growth in the size of the Internet routing table, it is better to advertise fewer routes. Instead of advertising each network in a data center, if you advertise a single aggregation of those routes, there are fewer routes to manage. As mentioned above, some ISPs will not accept parts of a traditional class B network. One way to deal with such ISPs is to connect data centers to ISPs who will accept parts of a class B. Each data center can advertise a prefix that is short enough to be accepted, while at least one data center can advertise the whole /16 class B network. This way, those ISPs who only accept a whole class B will see a route for the whole class B and send it to the data center’s ISP. Once it is in the data center’s ISP, the ISP will route traffic to the most appropriate data center because each data center is also advertising a route for its section of the class B.

Another factor in deciding what size address space to advertise is the backbone infrastructure connecting data centers. If data centers are connected by a backbone network that has enough capacity to route significant amounts of public Internet traffic coming into one data center that is bound for another (the worst case being that the backbone will handle all of a data center’s traffic), then it is feasible to advertise a single route that aggregates all of the data center’s networks into one. If the backbone connecting the networks doesn’t have the capacity, advertising a single aggregated route can result in performance bottlenecks when end users accessing one data center access that data center through another.

Autonomous Systems Number
The issues and choices regarding Autonomous Systems (AS) numbers are very similar to those regarding IP address space. A data center’s address space can be advertised from the following:

• Private AS
• Currently owned and used AS
• The data center’s ISP’s AS
• A new AS obtained directly from Internet registries
• Customer AS

If a data center’s routes are advertised from an ISP’s main AS number, the data center is locked into using only that ISP and cannot have connections into other ISPs (although it should be mentioned that some ISPs provide a special AS for multihomed customers). Getting a completely new AS number from an Internet registry has the advantage that a data center can change ISPs with much less work, usually just changing entries in routing registries. Multihoming to multiple ISPs is now possible, and end-user access to the data center can be improved by routing traffic based on using the full Internet routing table. The down side to getting a new AS is that it takes time: you usually have to join an Internet registry and apply for an AS number. Also, AS numbers are limited in quantity, as mentioned above, with only 64511 AS numbers available for use directly on the Internet.

Finally, a data center can advertise address space/route from a customer’s AS number. This has the advantage of allowing that address space to be served by multiple ISPs. It has a number of constraints and disadvantages. Only the address space/routes that the customer owns can be advertised as using that AS. As with using a previously owned AS, this option has the disadvantage of being affected by any other policy that organizations and ISPs may implement based on the customer’s AS numbers. The data center effectively becomes an ISP, and the data center’s ISPs often must change AS and network filtering policies to allow that route to be advertised. In addition, routing registry information concerning that network will also have to be changed.

IP Address Allocation Policy
Given that IP addresses are limited in quantity and their use has serious constraints, the allocation policy for IP addresses to data center customers is a serious concern. There are a number of constraints and tradeoffs. The first choice that needs to be made is whether to allocate a separate address space to each customer. From a customer, security, administrative standpoint, it is better to give each customer separate address spaces. Firewall policies are easier to implement on a subnet basis, and any special traffic policies, such as giving certain customers a different path or giving them priority over others, are much easier to implement if customers are on different subnets. Customers may be competitors, and the thought of a competitor on the same subnet may be unpalatable to a data center customer. The cost of separate subnets per customer is loss of usable address space. Each subnet has a subnet number, and typically that is not used as a host name to avoid confusion. Also, each subnet has a broadcast address that cannot be used for hosts. As a result, there are two addresses consumed as overhead for each subnet. The more subnets, the more addresses that are lost from subnetting overhead. Figure 1 shows the fraction of a subnet that is lost to varying degrees of subnet overhead if that space is divided into subnets with prefixes of the specified length. Half of all addresses in a /30 are consumed by subnet overhead even with the lowest possible subnet overhead.

Figure 1: Fraction of overhead per subnet, depending on subnet length and overhead per subnet

The amount of overhead consumed per customer subnet affects usable address space availability. There are router redundancy techniques such as Hot Standby Router Protocol (HSRP) [13] that allow more than one router to handle traffic for a virtual interface. The overhead cost of HSRP is one virtual address and one address per router. Thus for a /29 segment with two routers using HSRP on the subnet interface, 62.5% of the available address space is consumed with overhead, leaving only three usable IP addresses. Figure 1 also shows how usable address space disappears with the increase in per subnet overhead.

From a customer and administration standpoint, it is also advantageous to have as much address space as possible. This makes it easy for a data center customer to expand operations by adding servers. Moving servers to a completely different, larger subnet in order to expand forces a customer to reconfigure all the hosts, typically involving significant downtime. As mentioned above, firewall and other access policies are often configured by subnet, and having as large a subnet as possible dedicated to a customer allows additions and changes to be made to servers without having to change those firewall and access policies.

Of course, since the supply of IP addresses is limited, customers cannot have all the space that might be convenient for them. Data center administrators must consider what happens when address space becomes nearly exhausted. In that case, they need to consider meeting Internet registry requirements to obtain new space. Typical registry requirements for new public IP space are as follows:

• 25% of the new space must be utilized immediately.
• 50% of the space must be used within one year.
• To get more space, the address space must be 80% utilized.

The final consideration that a data center needs to evaluate is economic. An IP address has economic value, and if a customer is willing to pay for space that is unused, the value needs to be weighed against another customer using that space and also generating income.