Here at Nouvola we’re big believers in testing. Our company was based on the premise that businesses need to test their web applications early and often. But how do you run tests? I’ve thought about this a lot recently when I hear functional testing and performance testing described as two distinct processes. It’s almost like people think you have to choose one over the other.
But do you? I’d argue that in the world of web applications these two processes complement each other and really need to live very close to each other to provide a complete view of your application behavior. It doesn’t matter if your website functions as designed, if it functions slowly. Users demand both function and speed and testing needs to reflect this reality.
Most of us spent about six hours a day in front of a computer. A missing feature or two (imperfect function) may not be ideal, but a slow site (poor performance) is probably a bigger problem for the average user. This is why performance testing is so important. Performance is not an issue that relates to specific products or applications. It is a concern for anybody who has a website. Performance testing really needs to be part of the larger umbrella of application testing activities but too often it isn’t. QA teams have become very proficient but their focus so far has been mostly on functional testing.
What is performance testing
Now, to understand what performance testing is, we need to understand what it is not. Performance testing is not simply taking functional tests and seeing how fast they are. To be really effective, performance testing requires different parameters and strategies and even different goals. Some of the factors that need to be addressed include:
- ensuring main user transactions are performing well under specific loads: this is often defined as load testing.
- ensuring the application is robust under heavy loads: this is often defined as stress testing.
- -- ensuring the application performance does not change through time, but remains stable – or better yet, improves -- through infrastructure changes or software changes: this is often called regression testing.
- keeping your overall infrastructure costs low – or better yet, lowering them -- while still achieving high performance metrics: this is often called capacity planning.
- helping prepare you for your next surge in traffic tied to a major holiday or marketing event: A site that crashes on Cyber Monday is a disaster but a slowdown at such a time is almost as bad as a complete crash.
- helping you maintain a strong brand, and increase your conversion rate. Yes, performance can affect your image and customer loyalty in a big way.
Focus on patterns
When testing, it’s important to keep in mind that your customers access your applications from a variety of geographies and devices. One of your main concerns is knowing if the application responds quickly enough across all these devices and locations. So, rather than systematically testing every single function of the app, you might focus on studying the user behavior, identifying the most popular patterns and critical paths, and locating potential bottlenecks.
You might not be able to cover all the possible actions the user might perform, but you can focus on the ones where the user experience of a slow web site or a slow application is likely to have serious consequences, causing the user to leave, causing your site to crash or perform so poorly that your product is not displayed well, or causing the loss of important data.
Identifying the critical patterns and assessing the right parameters for your load is key. For example, the users’ geography might make a difference, because your CDN might not work well, or you might not have one. The volume of users also needs to be calculated, understanding that a stress load can vary greatly. For some websites, 100 users may be a lot; others consider 100,000 a small or routine volume.
Also consider that the people who visit your site are not all using it in the same way. For instance, an e-commerce website will attract virtual browsers as well as serious shoppers. A business application might also have users with different roles and permissions and visibility. You don’t need to define every single user as a separate entity, but you should seek to understand which ones are most representative of your overall load.
Other factors that might impact performance include the browsers, the devices and the operating systems being used to access the site; the duration of the test; the total amount of users, and the traffic patterns that they follow. Again, the goal is not necessarily to have a complete coverage of your application steps, but a good characterization of your load.
Finding the right metrics
Functional testing is pretty straightforward, typically involving a series of steps leading up to a specific desired outcome: push that button, and this page should show up. Performance testing is more complicated: a little bit science and a little bit art. It’s harder to determine a pass/fail criteria in performance testing and the answer is often not as simple as yes or no, but rather a thoughtful consideration of all the information gathered.
As a rule of thumb, the most important metric in performance testing is the page load time. The more information you have about page load time, the better you can assess performance and identify bottlenecks. How much time spent on the browser? How much time spent on the network?
Timing is not quite everything, however. There might also be other criteria to define whether a test passed or failed. Were errors generated because the server was under stress? Was empty data sent because retrieval was slowed down by a poor connection or too many requests? This information can also feed into the assessment.
Bottom line. Performance – as Larry Page has already said -- is becoming Product Feature #1.” But performance is not a “function” of the system. It’s a new paradigm that needs to be introduced in the testing activities. It requires a different strategy and, if done right, it will generate different results than the functional testing so many are more familiar with. QA teams of the future will have to account for this. And create a different approach to testing from what used so far.