Cloud Stories: Development and Test Environments

EMC moves development and test environments to the cloud

By Mike Liberty, manager of Lab Services, Unified Storage Division, EMC

 

As a world leader in high-end data-storage solutions, EMC supports IT infrastructure and virtualization technologies and solutions. The development of a new storage product line for virtualized computing environments, the EMC* VNXe* system, provided the perfect opportunity for us to move the development and lab environments into the cloud. The storage industry is extremely competitive, so our goal was to reduce time to market by increasing the speed of the testing process while reducing costs.

We implemented our USD Cloud, a self-service automated and virtualized cloud that enables thousands of developers, quality assurance engineers, and programmers to provision their own work environments via a portal. Prior to USD Cloud, engineers had to produce, install, manage, and update their own IT and server environments. Now they are freed up to focus almost entirely on product development and testing, spending minimal time setting up the work environment.

We started by moving the development system into the cloud, replacing the high-powered engineering desktops with development virtual machines (VMs) based on Linux. We oversubscribed individual servers, because even if more than one developer kicked off a build simultaneously, VMware’s ability to load balance across multiple servers ensured that the high CPU activities were distributed across the system.

Once that transition was complete, we moved the physical test environment into the cloud, through the use of simulators. Now developers can load their code on a simulated product and test it within the cloud rather than having to go to a lab and load the code on a physical piece of machinery. It also enables engineers to monitor all metrics in real time, generate thousands of individual test and development environments, and use a complex simulation facility. We believe the ability to simulate actual USD storage arrays in the USD Cloud is architecturally unique in the industry, and this feature has helped USD to significantly improve the entire product development life cycle.

Setting up the initial infrastructure to handle the needs of the development VMs and the simulator was a major challenge. We use the VCE model for enterprise-class clouds based on Cisco*, EMC, and VMware* technology and solutions: Cisco provides the servers, VMware provides the virtualization layer, and EMC provides the back-end storage. We designed a geographically dispersed system that could handle the load from 500 development systems along with associated simulators running in one cloud—a pretty beefy system—with a large infrastructure at headquarters in Hopkinton, Massachusetts, and smaller pieces of our cloud at our remotes sites around the world.

Our final deployment combines VMware vSphere* 5.0 Enterprise Plus and vCenter* 5.0 with EMC VNX* 5500 storage arrays, as well as Cisco 10 gigabit Fibre Channel over Ethernet (FCoE) networking technology and Cisco UCS* blade servers powered by Intel® Xeon® processors. The result is a highly available cloud capable of supporting up to 1,500 virtual machines.

One of the smartest things we did on our journey to the cloud was to deploy the Cisco UCS blade farm. We tried using rack-mounted servers initially, but we quickly discovered that a blade farm is easier to maintain and takes up much less space. We also started out loading individual servers with a set number of VMs, but quickly added the VMware Distributed Resource Scheduler (DRS) solution for load balancing across servers, which provided us with significantly more capacity to handle more users reliably.

Our cost benefits were immediate. We were able to move 20 high-powered engineering desktops—typically costing $2,000 a box—to a single $5,000 server. If you apply those savings to the current set of 500 users, the savings are pretty large. We also reduced operational costs with the simulator environment. Instead of requiring massive lab space with all the equipment, power, and cooling costs, we could significantly reduce the scope and costs in each of these areas. In summary, we realized a 30 percent reduction in IT hardware costs, greatly reduced shipping costs, 20 percent less time setting up test environments, and a much faster time to market for new products. Efficiency gains will provide more capital in the future to invest in product innovation and quality, and new initiatives to improve time to market.

Mike Liberty

 

"We believe the ability to simulate actual USD storage arrays in the USD Cloud is architecturally unique in the industry, and this feature has helped USD to significantly improve the entire product development life cycle."