Filter bot and spider traffic from results

Internet bots and spiders are software applications that run repetitive, automated tasks over the Internet. Optimizely uses the IAB/ABC international spiders and bots list to classify non-human traffic and filter it out of your results. The IAB/ABC list is an actively maintained record of user agents used by known spiders and bots. The list is updated on a monthly basis.

Bot filtering does not apply to past bot events, so you should configure it before you start the experiment. Otherwise, if you enable filtering halfway through an experiment, you might have to discard (or export and manually filter) any bot-contaminated event data from the start of the experiment.

To exclude bot traffic in Feature Experimentation, first enable the feature in the UI. Then, when you track user eventstrack user events depending on which SDK you are using and where, you may need to send additional information. If you are using the JavaScript SDK client-side, you are done. Otherwise, you must send the reserved $opt_user_agent attribute containing the user agent to the user context.

Go to Settings > Advanced > Bot Filtering in the Optimizely Feature Experimentation app to toggle bot filtering on or off.

Bot and spider filtering is automatically applied to all traffic as long as the User-Agent header is present in the request.

Bucketing has two meanings, and bot filtering affects only one of them:

Behavior differs in Feature Experimentation. Configure bot filtering in Settings > Advanced > Bot Filtering. The result depends on the SDK and how you send the user context attribute.

For SEO-specific guidance, see AB Testing and Search Engine OptimizationAB Testing and Search Engine Optimization.

Bots and spiders do not behave like real users. They typically run repetitive, automated tasks over your website or app at a much higher rate than real users, with the purpose of crawling your content and analyzing it. As a result, bots and spiders usually inflate the volume of traffic to your site and may skew your conversion metrics.

Removing bot and spider traffic from your results generally decreases the volume of traffic, such as the total number of visitors or sessions, and makes your measurements more accurate.

Optimizely reads the User-Agent header for each request and determines whether it matches with an entry in the IAB/ABC list. If there is a match, Optimizely classifies the request as non-human and automatically excludes it from the results.

Bot and spider filtering relies on the client’s User-Agent header to work. For web traffic, the User-Agent header is normally sent with every request, so Optimizely Web Experimentation and Optimizely Personalization customers automatically benefit from it.

Yes. This capability is available for all Optimizely Feature Experimentation SDKs. You must first enable bot filtering in the Optimizely Feature Experimentation application and then implement the $opt_user_agent attribute unless you are using the JavaScript SDK in the browser. See Filter out botsFilter out bots.

Bot filtering does not change how search engines crawl your site. Bots and spiders continue to access pages as they would otherwise. Bot filtering only removes their activity from your experiment results.

For information on how A/B testing interacts with search engines, see AB Testing and Search Engine OptimizationAB Testing and Search Engine Optimization.