Performance testing is not just a box to be checked. It is not something you can do once and forget about. Today there are multiple options for testing web performance, many which fall broadly into the synthetic testing and the RUM, or real user monitoring categories. If you want to thoroughly test for performance you should do both.
RUM and synthetic testing are often seen as two different options, but they ought to be seen as complementary solutions that offer useful information both before the site is launched, and in real time. Each technology provides data that can help control and improve the performance of your site. Each one has strengths and deficiencies. Together, they can provide a pretty complete picture of where you’re operating as designed, and where you’re falling short.
I’ll start with synthetic testing since that’s most likely to come first when testing performance. Synthetic testing is done in a very controlled environment, which brings many benefits. Without all the noise that comes with RUM, synthetic testing can provide more precise measurements. It can detect every factor that might have an impact on performance. If there are improvements, synthetic testing can report what helped; if there is deterioration, it can help identify the culprit.
Because synthetic testing is controlled, it is easier to tailor the test to analyze different characteristics: You can analyze the performance based on the level of traffic, the type of workflows or user scenarios, the presence of think time, variance on payloads, etc. You can set up different traffic patterns and analyze them all. Many of these synthetic testing features are now available in Nouvola DiveCloud.
Synthetic testing can also help you plan for infrastructure improvements, showing how they will impact performance. This becomes useful in capacity planning, where you can test the impact of powerful servers or plug-ins on performance, or analyze how cost-effective a CDN will be.
Synthetic testing can be also used for regression as the load and conditions are very reproducible. Once you create a solid baseline, you can use it against any new tests run before deployment to production. If a new deployment causes performance degradation, this can be immediately spotted before the new release gets into the hands of the real users.
Finally, one of the most powerful aspects of synthetic testing is that it can be used to predict future loads. It can test for a surge in traffic like Cyber Monday retail sales to let you know ahead of time if your site can hold up under the additional weight. It also helps to predict how the site will respond to traffic from different geographies. This is critical as more and more companies operate global businesses.
So, what can’t you learn from synthetic testing?
Since it is, by definition, not real, it may not account for all the noise and real-world traffic situations you encounter when you’re up and running. Some studies show that synthetic tests can show load times to be twice as fast as they really are. More and more, we are understanding that it’s very important to study the end user experience in the most realistic ways. Since RUM is, by definition, real-user monitoring, it can at times provide a more accurate picture. An application can have many different and unexpected scenarios, and it might be difficult to represent them all synthetically. The data that RUM can provide from actual users is valuable, as a complement to synthetic testing. On the other hand, RUM alone can’t help you prepare for peaks of traffic or understand performance degradations.
By using RUM, you can receive alerts about performance issues in real time, and you can take immediate action to resolve the problem quickly.
It’s worth noting also that the information collected in RUM can actually help inform better synthetic testing by creating more realistic tests.
Testing apps is critical to both customer satisfaction and your own peace of mind. To that end, performance testing needs to be thorough. I like this explanation of why synthetic and RUM are like milk and cookies. They are good individually but better together.