Below are our testing conditions, targeted at the conservative end of computers and network conditions, so we get exposure to the whole spectrum of experiences. Bolded metrics are ones that we value the most, with SpeedIndex being the top priority.

For our use case we do post-processing of the result data to come up with a metric of "Peak User Sessions Per Hour" that we track. The remaining data we use only for investigating specific cases.

Testing conditions

From Google Analytics of past clients we have created a profile to model what typical traffic is.

Assume 45 seconds per page.

We ramp from 0 new user sessions per second, to 150% of our target, over one hour and stop the test when the 99th percentile response time crosses above 10s. While 10s may seem high, there is less than 2% difference in capacity from using cut-off values like 2 or 3 seconds and 10 seconds. However, 10 seconds is much more tolerant of noise, which leads to more consistent results from one run to the next.

Result Summary: (see below for more info)

The following test is intended to determine what key infrastructure resources are needed to scale the website and its related services.

Assumptions:

This test is not intended to:

The primary resource to plan for is Web Server CPU. The results show that from 2 to 16 cores the application scales linearly, with a stable 4,600-5,000 user sessions per hour per core. The test with a single core shows a efficiency of about 1/2 of dual-core. This is logical as the application code is forced to share a single resource with OS and IO activities. At 36 cores the application becomes unpredictable, from one test to the next there is a large amount of variability in the results. Result of up to 3,800 user sessions per hour per core, a scaling efficiency of 75%, which is expected as hardware inefficiencies arise from multi-socket machines with NUMA. However, I could not get this result again, I got results as low as 970 user sessions per hour per core, for a total of 35,000 user sessions per hour capacity, which is less than the 8 core test.

The testing shows that scaling the services that Configured Commerce depends on closely follows the scaling of the Web tier. SQL has an ideal ratio of 10:1 of the web tier. And again, single-core SQL is derated by 50% because of forced sharing a single resource with OS and IO activities. So a single core SQL server could support a 5 core web server, and a dual core SQL server could support a 20 core web server. Similarly Elasticsearch, has an ideal ratio of 1:20, with a derating of 50% for single core. So a single-core Elasticsearch instance can support a 10 core webserver, and a dual-core can support a 40-core webserver. While scaling SQL and Elasticsearch, you must allow head room for periodic background cleanup tasks. For SQL the sustained CPU should not go above 90% for single-core machines or 80% for multi core machines. For Elasticsearch sustained CPU should no go above 70%.

Ram capacity was not significant in testing. Web server and SQL server ram usage remained around 2GB, which even on the smallest instances left plenty of room for filesystem cache. Elasticsearch used 1gb, which again left room for filesystem cache. For comparison the SQL database is 5.1GB, The elasticsearch index is 1.3GB, and the UserFiles are 60 MB. I expect that the largest driver of RAM usage is the size of the hot and warm datasets, but further testing would need to be done to validate.

As the site is warmed, there is moderate disk access, but then it drops very low. Throughout testing, I never saw more than 2% disk utilization for 2 consecutive data points. Along with RAM I expect hot and warm data set sizes are the largest drivers of this, but further testing would need to be done to validate.

While web CPU is limiting, the application latency climbs until errors occur as timeouts are hit. While this is happen the service remains stable, error rate climbs, but the server successfully responds to as many requests as possible, and the service remains up, but degraded. In contrast when SQL or Elasticsearch is limiting, the failure mode is catastrophic. Resource utilization never quite reaches 100% on SQL or elasticsearch, but queued work climbs, and request latency climbs, until a livelock situation occurs and all work comes to a halt. For SQL it was pagelocking, for Elasticsearch it was garbage collection. In either case, Web CPU climbs sharply, so be aware that it may be SQL or Elasticsearch causing high web CPU. that is100% Web CPU and 80% SQL might be SQL failing, not Web.