There are five major categories of data that are generally useful for performance tuning. From generic to specific, they are as follows: system performance, execution profile, JVM performance, application-server performance, and the application itself.

• System Performance. The data contains system-level resource utilization statistics, such as CPU utilization (% processor time), disk I/O (% disk time, read/write queue lengths, I/O rates, latencies), network I/O (I/O rates, latencies), memory utilization (amount of virtual memory used, amount of physical memory used, amount of physical memory available), context switches (number of context switches per second), and interrupts (number of system interrupts per second). Many tools are available to collect system performance data. On Windows, the Performance System Monitor from Microsoft's Management Console (known as PERFMON) is easily accessible, and a very useful freeware tool set is available from Sysinternals*. Similarly, on Linux*, many tools, such as sar, top, iostat and vmstat, are available.
  
  
• Execution Profile. The data contains application hotspots and allows us to drill down to the code-level. Hotspots are sequences of code that consume disproportionally large amounts of CPU time (e.g., measured by processor clock cycles). Hot methods or code-blocks deserve attention, as changing such code is likely give you good return on your investment of time. The Intel® VTune™ Performance Analyzer can provide details not only for clock cycles and instructions, but also for many microarchitecture performance statistics for branches, memory references, (e.g. cache misses), and other processor-specific information.
  
  Some JVMs, such as JRockit* also provide hotspot data. The JRockit JVM provides the JRockit Runtime Analyzer (JRA), for this purpose. While the VTune analyzer gives us the whole stack of hotspot performance data, including supporting libraries, OS and drivers, JRA gives Java users a convenient way to get the application-level hotspots through JRockit sampling.
  
  
• JVM Performance. The data contains key statistics that identify performance issues when the workload is run with a JVM. Common statistics are heap memory usage, object life cycles, garbage collections (GC), synchronizations and locks, optimizations, and methods inlining. A JVM typically supports some run-time parameter to collect many such statistics. JRockit provides the added convenience of also collecting such data using the JRA tool for execution-profile data.
  
  
• Application-Server Performance. The data contains application-server performance statistics such as queue depths, utilization of thread pools, and database connection pools. Examples of these tools include BEA WebLogic Console. Enterprise Java Bean* (EJB)-specific statistics should also be collected. The key performance counters are cache utilization, activations, and passivations of EJBs. These statistics are also available from WebLogic through the use of the weblogic.admin command-line tool. As a general performance guideline, passivation and then activation of EJBs is likely to reduce the performance of the workload and should be minimized.
  
  
• Application Performance. The data contains workload-specific performance metrics. For example, TOPS, New Orders 90% Response Time, Manufacturing 90% Response Time, Order Cycle Times, and the gradual change of transactions over steady state are all important statistics for SPECjAppServer2002. Because these are application-specific statistics, creating a simple tool to parse the key statistics for the workload would help automate part of the data analysis and increase the productivity of the performance engineer.

In the case of a J2EE workload, the performance of the back-end database server is also important. The data contains performance metrics including cache hit ratio, disk operation characteristics, SQL response times, and database table activity. Examples of these tools include Oracle’s 9i* Performance Manager and the DB/2* Database System Monitor. For Microsoft SQL Server*, many such statistics are supported conveniently by PERFMON, as well.

Having the right set of tools available is essential for supporting productive performance-tuning activities. In addition to the tools highlighted here, and others are described in the following resources:

• "Enterprise Java Performance: Best Practices"
• "SPECjAppServer2002 Performance Tuning"*

The tools that collect data must have minimal intrusion to the application or else one is not measuring the workload in the usual state. This study experienced very little (<1%) overhead using either the VTune analyzer or JRA for execution-profile data collection. Many other performance-tuning tools introduce significant performance impacts, casting doubt on the usefulness of the data collected.