Design of Experiments for Performance Tuning

By Kingsum Chow, Ph.D

Software performance analysis of multi-tiered applications is more complex as the number of hardware and software components that can affect performance increases in many dimensions. In Part One of the series¹,J2EE Performance Optimization Part 1, we described a top-down closed loop approach to performance tuning. We then described how to apply the approach to a multi-tiered application in a case study in Part Two²,J2EE Performance Optimization Part 2. While the method described earlier can be applied adequately when you can experiment with one tunable parameter at a time, at times an alternative approach is required, for example, to reduce the time to finish the tuning experiments, or to understand the impact of interaction of factors, which can be done by modifying more than one parameter at a time.

This article describes techniques to use when the number of tunable parameters is too large to examine one variable at a time. We describe how to use the design of experiments (DOE)³ a well-known statistical approach, to manage for performance analysis. We will also introduce multivariate analysis (MVA)⁴ to analyze data that involves multiple variables, and  demonstrate how to apply the technique to optimize system configuration for a multi-tiered enterprise application.

A multi-tiered application typically depends on a large number of tunable parameters for improving the performance of the system. Many of the tunable parameters are useful only for programs with certain workload characteristics; a known set of configuration parameters optimized for one workload may not be the best for another. Re-examining all configuration parameters for different workloads is too time-consuming as the number of configuration parameters is large. The selection of proper optimized parameters for a particular workload is controlled by users via database (DB), operating system (OS), managed runtime (MRT) and application server configurations on all the machines involved. Since each configuration can contain many tunable parameters, the total number of parameters is usually large, and each may specify a large range of values.  We face the following two problems:

• Selection Problem: select the best set of parameters for a workload; solving this problem provides a powerful tool to obtain the best performance for user programs.
  
• Interaction Problem: determine the interactions between tunable parameters; solving this problem helps both the users and the system developers to understand the effects of the tunable parameters.

The search space for the selection problem is very large.  For example, 10 tunable parameters with binary values equals 2¹⁰=1024 experiments to be studied. Most people rely on educated guesses and often sub-optimal selection is used.  Furthermore, the interactions among tunable parameters are often informally described such as “parameters A and B have positive or negative interaction.” 

Our method applies factorial DOE to systematically design a series of experiments that determine the tunable parameters with the best chance of being optimal. It also quantifies the interactions among the switches by suggesting constructive or destructive interactions.  For example, a constructive interaction could be that parameters A and B together will improve the performance by 5% while individually each can only improve the performance by 1%. An example for a destructive interaction could be that parameters A and B together will reduce the performance by 2% despite individually each switch may help the performance by 1%.

Early work ^5,6,7  on applying DOE to performance analysis has been quite successful, as has applying MVA to computer performance ^8,9,10 . This article is complementary to recent performance tuning  work  ^{11,12,13,14,15}  by describing factor analysis on the database system.

We invite you to post a comment (not monitored by customer support) on this page or send a question directly to our support team.