Test performance and volume

This document contains Optimizely recommendations for performance and volume testing before Go-Live and after site launch for continual monitoring. There will be variables based on environment differences, tool selection, site architecture, and common user behaviors.

There are a number of testing strategies, appropriate for different scenarios. Some considerations for when to conduct testing are:

Testing prior to being "solution complete" may be useful for finding issues in the code, but it can also waste resources chasing after the wrong issues. Performing tests on the Production environment with a code-complete solution will be the most efficient use of time.

There are multiple tools to use. Complexity and testing strategies may vary based on tool selection.

Synthetic Testing means simulating site. It starts with defining common user behaviors and expected numbers of users, and then increasing the load until the site ceases to deliver acceptable performance levels. Gatling is the tool currently used by Insite.

Common user behaviors are often difficult to define. Google Analytics/Tag Manager has historically been helpful for gathering real data and prioritizing the types of common user behavior.

Typically Synthetic Testing will be done before go-live and during the Alpha/Beta releases to define where the common bottlenecks exist in the site. It may be conducted later if issues are identified to assist in troubleshooting or verification of a fix.

Server-side tool for Capacity and Load Testing (Synthetic Testing). It will only be as valid as your model of a typical user. Google Analytics or the common buyer behavior from a previous site can be helpful for these definitions. Because of different architectures, site customizations and user behaviors this is the most difficult to make a general, prescriptive recommendation. Optimizely is happy to discuss more on a case-by-case basis.

Client Side, Front End Testing tool. This can be used for both synthetic testing and real user monitoring. The focus is on optimizing initial request and limiting render blocking resources (fonts/css/js in head). Our recommendation is to optimize the following areas:

Generally we ignore the following areas of the tool because they have not been pertinent to performance issues:

Real User Monitoring, or Application Performance Monitoring, includes monitoring tools that are always running and tracking environment performance and user traffic, so you choose when to be reactive base on pre-determined performance thresholds. Only someone familiar with the environment will be able to view the tool and diagnose all but the most obvious issues.

There are multiple points at which monitoring should be deployed. Monitoring can be active or passive, firing alerts to proactively show issues, or serve as a tool to diagnose an issue after something has been discovered. Active tools require time to properly tune alerts and ensure site performance is not being affected by the monitoring tool itself.

This is an Application Performance Monitoring tool which measures the full application stack from code down to database for gathering as much information as possible, pinpointing the exact area where an issue exists.

This tool is used for monitoring the Configured CommerceApplication itself. It pin-points code as it is which models are being used and when. It also monitors database queries.

Optimizely operates from the Azure environment. We use the Azure Service Health and your resource overview page for basic server monitoring in the following areas:

Some advanced tools combine monitoring with synthetic testing to give more reliable, repeated benchmarks. Sites which have periods of increased traffic that affect performance may benefit from recurring synthetic testing. It can also be highly useful for comparing speeds pre and post changes to site architecture, code, software patches, and so on.