Performance and scaling guide

Configured Commerce was designed to operate in a wide variety of IT environments and integration scenarios. Ongoing maintenance activities of the environment, including but not limited to web server log monitoring, database backups, application configuration, and other routine tasks are not covered in this document. However, these activities are part of on-premise hosting scenarios and it is expected that the reader have general familiarity with all administrative aspects of a client-server type application deployed on the Microsoft technology stack.

The performance and sizing guidelines in this document are directional in nature and should not be interpreted as performance guarantees since actual overall application performance depends on a variety of externalities and factors, that is third party web services dependencies, network connectivity, application code, and so on. These factors are not controllable on the deployment and infrastructure level, which this guide is intended to cover.

While it is possible to install and configure all three subsystems on a single machine (server), a typical production deployment requires these subsystems to be installed and configured on distinct machines. Machines in this context refer to physical servers or virtualized resources such as Hypervisor based virtual machines or cloud (AWS/Azure) instances. The actual IT infrastructure landscape will include additional devices such as firewalls, routers, Active Directory, and other elements not depicted here.

Hardware specifications are provided as reference points and for guidance. Specifics depend on the operating system selected and the anticipated amount of processing. Resorting to system requirements as documented by Microsoft given the applied operating system may be a good practice when in doubt.

The following are drivers of computing and memory needs and can be considered in advanced tuning efforts:

Software requirements are summarized for the three different server types typically found in a Configured Commerce enterprise deployment that makes use of back-office integrations to the ERP, OMS, or product information management systems.

SQL Server Standard Edition is appropriate for customers who seek to use Replication or Database mirroring for transferring data from the primary database to the secondary databases for the purpose of database synchronization in redundancy and high availability scenarios. Replication uses the publish-subscribe model. This lets a primary server, referred to as the Publisher, distribute data to one or more secondary servers, or Subscribers. Replication enables real-time availability and scalability across these servers. It supports filtering to provide a subset of data at Subscribers, and allows for partitioned updates. Subscribers are online and available for reporting or other functions, without query recovery. SQL Server offers three types of replication: snapshot, transactional, and merge. Transactional replication provides the lowest latency and is usually used when high availability is needed. Database mirroring is primarily a software solution for increasing database availability. Mirroring is implemented on a per-database basis and works only with databases that use the full recovery model. The simple and bulk-logged recovery models do not support database mirroring.

Always-On Failover Clustering is available in the Standard and Enterprise editions of SQL Server and requires additional shared disk storage capacity in form of SAN or NAS. Failover clustering provides high-availability support for an entire instance of SQL Server. A failover cluster is a combination of one or more nodes, or servers, with two or more shared disks. Applications are each installed into a Microsoft Cluster Service (MSCS) cluster group, known as a resource group. At any time, each resource group is owned by only one node in the cluster. The application service has a virtual name that is independent of the node names, and is referred to as the failover cluster instance name. An application can connect to the failover cluster instance by referencing the failover cluster instance name. The application does not have to know which node hosts the failover cluster instance.

In this section, we introduce three distinct reference architectures that provide representative topologies illustrating how the Configured Commerce subsystems can be deployed to meet client requirements. It should be noted that the proposed architectures are not meant to be prescriptive. Instead, these topologies are intended to provide IT administrators and system engineers some reference points when considering and sizing for their particular audiences and anticipated usage patterns of one or more e-commerce sites hosted within Configured Commerce. Optimizely has however, confirmed that the presented architectures are real-world examples of existing clients.

As indicated earlier in this document, it is highly recommended, for production purposes, that Configured Commerce subsystems be deployed to dedicated machines. Therefore the following topology is considered the minimum production type deployment.

Customers using this type of topology typically have low order volumes and limited number of registered customers. Their online channel is either new or they perform a dedicated pilot with a limited number of their business partners. Also, some customers who are in phase I of their online presence and do not yet enable online purchasing make use of this topology bringing their product catalog in phase I online only running mostly a pure content site without transactions.

The production topology represents a typical production grade deployment configuration most of Optimizely's customers adopt when they launch their e-commerce business on Configured Commerce unless they already have an established e-commerce practice and business that requires additional scale to support millions of transactions per day. Then they will look to the Enterprise topology discussed in the next section.

The production topology accomplishes a few critical improvements over the Minimum topology introduced in the previous section:

Customers using this type of topology typically have substantial order volumes and increasing numbers of registered customers. Their online channel is either established and a mission critical system is required with 98% uptime or better.

Following are typical usage scenarios supported by the production topology:

As the focus shifts from ensuring a redundant and resilient architecture (more up-time), to a scalable architecture that supports higher (or spiking) load patterns, more throughput, and overall larger volumes of transactions (orders), the concept of redundant web servers can be scaled out accordingly. This permits Enterprise customers of to adjust the deployment and subsystems to meet even the most ambitious performance requirements.

In this topology the following areas have been scaled-out to achieve higher capacity:

Customers using this type of topology operate multi-site, multi-geography based large scale e-commerce properties that represent a major revenue channel for the organization.

Following are typical usage scenarios supported by the enterprise topology:

In this section of the document, we discuss additional considerations for sizing and of Configured Commerce. These discussions are intended to provide the interested reader with supplemental information around the .NET based technology stack to entertain related concepts that are typically available within the realm of Microsoft's best practices recommendations pertaining to performance.

Configured Commerce is built on top of the Microsoft .NET technology stack, which is comprised of the following tiered application patterns:

The following discusses three major focus areas that are applicable to Configured Commerce.

Configured Commerce is comprised of a set of subsystems that can be deployed independently. This supports the goal of specialization. The objective here is to break apart the application into smaller pieces to isolate dedicated processes onto dedicated machines and computing power to achieve a more scalable and performant system overall.

Consider the following areas:

Optimization can and should be applied on several levels:

To implement distribution you need to add servers, duplicate the application across them, and implement load balancing. For load balancing, you can use Network Load Balancing (NLB), a service included with all editions of Windows Server. However, hardware load balancing devices are recommended for production environments while NLB may be used in development or QA environments.

Configured Commerce supports both, up and out. The following presents a brief overview of what the two approaches are about and when to use which approach.

With this approach, existing hardware capacity is increased. Existing hardware components, such as a CPU, might be replaced with faster ones, or one might add new hardware components, such as additional memory or solid state drives (SSD). The key hardware components that affect performance and scalability are CPU, memory, disk, and network adapters. An upgrade could also entail replacing existing servers with new servers.

With this approach, more servers are added to the farm to spread application processing load across multiple machines. Doing so increases the overall processing capacity of the system.

Up is a simple option and one that can be cost effective. It does not introduce additional maintenance and support costs. However, any single points of failure remain, which is a risk that should be only accepted in the most minimalistic deployments (refer to Minimum Topology).

Beyond a certain threshold, adding more hardware to the existing servers may not produce the desired results. For an application to scale up effectively, the underlying framework, runtime, and computer architecture must also scale up. Configured Commerce is specifically optimized for multi-threading when it comes to integration processing.

Out enables the addition of more servers in the anticipation of further growth, and provides the flexibility to take a server participating in the farm offline for upgrades with relatively little impact on the cluster. In general, the ability of an application to scale out depends more on its architecture than on underlying infrastructure. Configured Commerce supports out in various areas, such as the web server role, the database server role, and both integration roles, the website side of the integration as well as the back-office side of the integration processor. Each of these roles can be scaled out independently.

Additional performance tuning tips are provided here:

As long as the application usage continues to increase and the transacted business volume expands, and performance tuning is an ongoing effort. A critical step in this process is to measure existing capacity and performance and align the results with anticipated growth assumptions and metrics. Application testing and profiling are techniques used to adequately determining where to spend capacity adjustment efforts. Some of our customers use tools such as Gomez, DynaTrace, or Loadrunner (HP) for these initiatives.

As part of Optimizely's ongoing efforts to ensure that our product can perform and scale according to customer needs and growth dynamics, the product verification team has conducted a variety of load and throughput testing scenarios. These tests were specifically aimed at understanding what hardware configurations would be required to obtain constant throughput behavior of the system as the number of concurrent users increase. That means our core objective was to establish the hardware configurations suited to support the same ratio of orders placed / users on the site as user traffic on the site increased. Our target was around an average of 14 orders per user.

The tests were all conducted within a dedicated AWS environment using a set of distinct configurations to address the following performance and scalability questions:

We attempted to find the impact of a variety of configurations to help in trying to emulate real-life load while keeping a laboratory environment in terms of externalities and testing harness.

JMeter was used as the performance testing tool generating the necessary load on the system. The test cases covered two explicit objectives:

Transactions, for this review, are number of completed page loads. The tests simulate a variety of search and browse paths along with 1 out of 4 users following an ordering path. During the ordering path, products are added to the cart and the checkout process is followed until order placement. The checkout process does not rely on external interfaces for payment, freight or tax calculations in order to eliminate these externalities.

To further provide laboratory conditions, the use of edge caching or other output caching has been disabled and is not used. This means that the test outcomes should not be regarded as final performance characteristics of the product because these can be greatly enhanced when enabling the various caching strategies. However, for the purpose of this testing, the bare system capacity is being tested not the overall experience.

The test results were achieved using different configurations:

The laboratory testing has helped to establish that Configured Commerce can be scaled out (horizontally) by adding web server resources gradually to support increasing volumes of users, transactions, and orders on the site. The test has been executed in an AWS environment using compute optimized cs.xlarge instances with 4 vCPUs and 7.5 GB memory each.

This should provide a good reference point for those clients who are interested in self-hosting but decide to leverage the Amazon infrastructure. For those clients who are hosting on-premise on physical servers or virtualized servers that are managed within their own premises, corresponding translation of the testing results above can be made depending on specific site and infrastructure conditions applicable on a client-by-client basis.