Load Testing Software Using Six Sigma Control Charts

The first question (Six Sigma project) should address stability and answer the basic question:

Is the software performance in control?

Is the performance of the software stable under load?


The question of stability is the first performance question to be asked of a given piece of software and this question should reference a specific workload, what metrics are to be charted (using control charts) for defining stability and a target system configuration.

Statistical Process Control (SPC) defines a process as being stable if the outcome is consistent (repeatable). Whether evaluating a single function call (API) or a complete system the first load testing objective should be a stability test which can be followed by a benchmark, which determines the overall performance capability of the software under test. As a rule of thumb a performance stability test should run for at least 6 hours, subjecting the software under test to loads that approximate what the software will experience when in the production (live) environment.

The question of stability is a benchmarking performance project type, in that the objective is to characterize the stability of the software under test without making changes to the software. This could lead to the second type of performance project (improvement) but it is important to note where one project ends and the other begins, otherwise the single project would be endless. That said it is often the case that benchmarking and process improvement project types are typically iterated as requirements are refined, or change over time.

Whenever a piece of software is created, or changed, a performance stability test should be executed. This testing could be done at the individual component level or, more typically, at the system level.

This type of performance project follows these phases:-

Load Testing Requirements Phase

  • Define the workload requirements for the software under test.
  • Define the system monitoring requirements to be able to determine stability.
  • Define the target system configuration for the software under test.

Load Testing Design Phase

  • Create the load test specification for the load testing scripts that will satisfy the workload requirements.
  • Identify and document the system resource and user response measuring devices.
  • Specify the target system configuration needed to satisfy the system configuration requirements.

Building the Load Testing Scripts and Environment

  • Produce the load testing scripts that will satisfy the load test specification.
  • Enable the monitoring devices that will measure the user response and the system resource usage.
  • Build the system environment as per the target system configuration.

Executing the Load Test

  • Execute the load scripts.
  • Collect the measurements.

Analyzing the Results

  • Produce basic control charts for the measurements taken.

Drawing a Conclusion Regarding Performance Stability

  • From the control charts determine if the software performance is stable with regard to the given workload and the specified measurements.
  • Produce a report that completes this project.
Next steps and recommendations:

If the software under test is stable.....

....with regard to the workload requirements, specified system resources and response measurements then a software performance capability project could follow.

If the software under test is stable...

... with regard to the workload requirements, specified resources and response measurements then another performance stability performance project could begin with different workload, measurements or configuration requirements.

If the software under test is not stable.....

...... with regard to the workload requirements, specified system resources and response measurements then a Counter measures (solutions) for improving software performance project can be initiated. This project type uses the lack of stability as the problem statement and concludes with a counter measure that brings the software to a stable point. This project type follows the typical Six Sigma DMAIC, described in the software performance process improvement section.

If the software under test is not stable ....

......with regard to the workload requirements, specified resources and response measurements then another performance stability project could begin by changing the workload, measurements or configuration requirements. This is similar to recommendation 2 in that another question can be posed regarding stability and that is with new requirements. Note this is a separate project but these types of capacity questions could come under an umbrella project that seeks to understand at what point does this software become unstable. This trial an error approach is iterative but it is important to note that each iteration is a distinct stability project with defined requirements.