We used PERFMON to collect overall system-performance statistics, including some network statistics that are described in the previous section. PERFMON adds some overhead to the performance of the systems, probably more so when a large number of objects are collected from several computers. We found that we were able to minimize such impact to less than 1% of our workload by reducing the sampling intervals to 15 seconds, while still collecting the following major statistics from the driver, the application, and the database servers. We collected all counters to a log file (using LOGMAN), as using the GUI adds significant overhead to the system. We processed the PERFMON data at the end of the experiments to analyze system-performance statistics.

The major system performance counters areas follow:

• Network: send/receive (bytes/sec and packets/sec); see previous section.
  
  
• Disk: read/write (bytes/sec and ops/sec); minimizing disk I/O is important, as it reduces latency and response times. We compared some configurations at higher injection rates due to hardware and software improvements with the baseline data and found that increasing the injection rate caused an increase in disk activities. This information alone only shows the symptom of a problem. We combined this with other information to pinpoint the source of the problem, which is related to insufficient cache size for some EJBs. The performance impact due to disk I/O can be from 5-50%, depending on how severe it is.
  
  
• Memory: physical memory (used and available), page faults and pages/sec; paging should be minimized by having sufficient physical memory for the workload. The performance impact can be orders of magnitude if excessive paging is observed. When enough physical memory is installed in the machine, it is important to make sure that it is available for the process you are interested in, and not taken by other residual processes or daemons. In our case, we had to make sure that the memory was available for the Java* heap within the limitations of the OS.
  
  
• System: context switches/sec, processes, processor queue length, system calls/sec, threads; in general, the processor queue length should be within the single-digit range for good performance. Minimizing context switches would be a good idea to allow the processors to spend time on real work. However, SPECjAppServer2002 requires a high level of network activities, and minimizing context switches and getting good response times must be balanced. System calls and number of threads should be monitored. We found that we had quite reasonable performance with about 100 threads running on the application server.
  
  
• Processor: CPU utilizations (user/privileged, individual processors and total) and interrupts/sec; among all the PERFMON counters, total CPU utilizations would be the single most important indicator. If the CPU utilization can only reach a peak, for instance, of 30% while the injection rate is gradually increased, there is likely a bottleneck somewhere outside the scope of the application server. In such a case, further application software, application server and JVM performance improvement may not be realizable, and we may expect low return on such investment. We must find ways to increase the CPU utilization on the application server in order to continue performance analysis. The problems can be related to the network, OS configurations, or interference from some anti-virus checkers that inherently limit the performance of the systems.
  
  
• Process: threads, process queue length, and Process(jrockit)\% Time; the number of threads for the JVM along with the CPU time spent on the JVM process should be monitored to help detect the introduction of performance issues when software or hardware is changed.
  
  
• JRockit: (When the JRockit JVM is used on the application server); we will discuss more about these statistics in the JVM section.
  • JRockit JVM Code Generation(Total)\Methods Generated and Optimized
  • JRockit JVM Memory(Total)\Heap Size, Free Heap, Used Heap Before/After GC, Pause Time, #Young/Old Collections
    
    
• SQL Server (When SQL server is used as the backend):
  • Buffer Manager\Buffer cache hit ratio
  • Databases\Transactions/sec
  • Memory Manager\Total Server Memory (KB)

We have briefly covered key PERFMON statistics and what to look for. There are many counters, and an optimal range of values for many of these counters may change with software and hardware configurations. Thus, it is very important to establish baseline data and continue to monitor if differences occur. Differences in counters due to software or hardware changes, while not necessarily a bad thing, should be considered in the performance-tuning process.