Load Testing Software Using Six Sigma Control Charts

The first question (Six Sigma project) should address stability and answer the basic question:

Is the software performance in control?

Is the performance of the software stable under load?


The question of stability is the first performance question to be asked of a given piece of software and this question should reference a specific workload, what metrics are to be charted (using control charts) for defining stability and a target system configuration.

Statistical Process Control (SPC) defines a process as being stable if the outcome is consistent (repeatable). Whether evaluating a single function call (API) or a complete system the first load testing objective should be a stability test which can be followed by a benchmark, which determines the overall performance capability of the software under test. As a rule of thumb a performance stability test should run for at least 6 hours, subjecting the software under test to loads that approximate what the software will experience when in the production (live) environment.

The question of stability is a benchmarking performance project type, in that the objective is to characterize the stability of the software under test without making changes to the software. This could lead to the second type of performance project (improvement) but it is important to note where one project ends and the other begins, otherwise the single project would be endless. That said it is often the case that benchmarking and process improvement project types are typically iterated as requirements are refined, or change over time.

Whenever a piece of software is created, or changed, a performance stability test should be executed. This testing could be done at the individual component level or, more typically, at the system level.

This type of performance project follows these phases:-

Load Testing Requirements Phase

  • Define the workload requirements for the software under test.
  • Define the system monitoring requirements to be able to determine stability.
  • Define the target system configuration for the software under test.

Load Testing Design Phase

  • Create the load test specification for the load testing scripts that will satisfy the workload requirements.
  • Identify and document the system resource and user response measuring devices.
  • Specify the target system configuration needed to satisfy the system configuration requirements.

Building the Load Testing Scripts and Environment

  • Produce the scripts and efficient load testing tools that will satisfy the load test specification.
  • Enable the monitoring devices that will measure the user response and the system resource usage.
  • Build the system environment as per the target system configuration.

Executing the Load Test

  • Execute the load scripts.
  • Collect the measurements.

Analyzing the Results

  • Produce basic control charts for the measurements taken.

Drawing a Conclusion Regarding Performance Stability

  • From the control charts determine if the software performance is stable with regard to the given workload and the specified measurements.
  • Produce a report that completes this project.
Next steps and recommendations:

If the software under test is stable...

...with regard to the workload requirements, specified system resources and response measurements then a software performance capability project could follow.

If the software under test is stable...

... with regard to the workload requirements, specified resources and response measurements then another performance stability performance project could begin with different workload, measurements or configuration requirements.

If the web hosting server is not stable...

...with regard to web hosting and testing, it's important to ensure that your web host is stable, particularly if you're using WordPress. According to Web Hosting Buddy, load testing is an excellent way to ensure that your host can actually handle the amount of traffic that you throw at the server. They did a breakdown on the best WordPress hosting companies this year to shed some light on which ones are able to handle a high visitor load, and which ones aren't, so it's worth a moment to check out their breakdown.

If the software under test is not stable...

... with regard to the workload requirements, specified system resources and response measurements then a Counter measures (solutions) for improving software performance project can be initiated. This project type uses the lack of stability as the problem statement and concludes with a counter measure that brings the software to a stable point. This project type follows the typical Six Sigma DMAIC, described in the software performance process improvement section. It's important to note that there are a variety of free load testing tools you can use in order to perform your tests. An extensive list of this options is available on Dotcom-Tools if you're looking to gather data to help you make a choice.

If the software under test is not stable...

...with regard to the workload requirements, specified resources and response measurements then another performance stability project could begin by changing the workload, measurements or configuration requirements. This is similar to recommendation 2 in that another question can be posed regarding stability and that is with new requirements. It's important to choose examine various different load testing tools in order to find one that works for your specific needs. If you're looking for recommendations on this, take a moment to check out Web Hosting Buddy's list of load testing tools for a detailed breakdown of open source vs. corporate tools.

If your WordPress website isn't stable under load...

...with regard to WordPress websites being stable under load, as mentioned above, thorough load testing can help to mitigate this problem. But the question is—how much load testing do you need to do? Well, this really depends on the amount of visitors that you're going to have. If you're running an e-commerce website and it's Black Friday (or Cyber Monday), you're going to have a lot of ground to cover to ensure that you've got enough server capacity so that your WordPress website doesn't crash. Instructify has a great write-up on choosing the best WordPress hosting company based on load/performance.

If your website bogs down substantially under an unexpected load...

...with regard to capacity planning (because that's what this really is), it's sometimes hard (as mentioned above) to estimate capacity. Most web hosting companies (especially those who host WordPress), have some redundancy built in. WebHostingProf.com has a great breakdown on capacity planning as it relates to WordPress and it's definitely worth checking out (click here to read more).