Scheduled index job

5 min

The EPiServer.Find.Cms assembly ships with a scheduled job plug-in, which lets administrators reindex an entire site from the Optimizely Content Management System (CMS) admin mode. The job, Episerver Find Content Indexing Job, can be started manually or with the job scheduling feature.

Note You only need to schedule the indexing job for the specific circumstances described below. CMS instantly indexes any content changes. If you run the indexing job when unnecessary, you are unduly adding to the system load.

Use the scheduled job

There are two scenarios for using the scheduled indexing job to index pages:

After installing the Optimizely Search & Navigation CMS integration for the first time. Otherwise, CMS indexes existing pages only when manually republished.

If automatic indexing fails, you should reindex pages for another reason.

When started manually, the job runs within the IIS server application pool and is interrupted by IIS when the Idle Timeout passes (default is 20 minutes). Indexing a large site may take more than 20 minutes, and it is not advisable to increase the timeout. You should schedule the job, as the CMS scheduled job process is unaffected by the timeout.

Log of indexing activity

After an indexing job, a report prints to the history log displaying execution time and number of pages indexed

Indexing job: Reindexing completed. ExecutionTime: 0minutes 18seconds Number of pages indexed: 180

If an error occurs, the report notifies you where it occurred, gives the error message, and counts two types of errors:

Page errors relating to filtering or loading page.

Batch errors relating to transmitting and indexing files to Optimizely Search & Navigation.

Example

Indexing job: Reindexing completed. ExecutionTime: 0minutes 16seconds Number of pages indexed: 169 Number of page errors: 11 Number of batch errors: 0 An error occured while filtering Page: 113. The 'ShouldIndex' function could not be evaluated for Page: 113. ShouldIndex-exception. (see log for information).

To see the stack trace, activate the logging functionality and trace it in your log file.

Enable continuous indexing job

Starting from Episerver.Find version 16.1.0, there are new features for scheduled index jobs.

The continuous indexing job lets you restart the indexing. If the job crashes, you can restart it, and it will continue the indexing progress from the last indexed content. You must enable this mechanism through the EnableContinuousContentIndexingJob configuration in appsettings.json:

{
    ...
    "EPiServer": {
        "Find": {
            "DefaultIndex": "your_index",
            "ServiceUrl": "https://demo.find.episerver.net/your_key",
            "TrackingSanitizerEnabled": true,
            "TrackingTimeout": 30000,
            "Cms": {
                "EnableContinuousContentIndexingJob": true
            }
        }
    },
    ...
}

Configure the maximum concurrent background tasks used during the indexing job:

{
    ...
    "EPiServer": {
        "Find": {
            ...
            "Cms": {
                "EnableContinuousContentIndexingJob": true,
              	"IndexingJobMaxThread": 4 // to configure the maximum concurrent background tasks
            }
        }
    },
    ...
}

Configure how many queue items are loaded per loop using the continuous indexing job. With this configuration and the maximum concurrent background tasks, you can adjust the job to balance CPU or memory allocations and indexing speed rate as well:

{
    ...
    "EPiServer": {
        "Find": {
            ...
            "Cms": {
                "EnableContinuousContentIndexingJob": true,
              	"IndexingJobMaxThread": 4, // to configure the maximum concurrent background tasks
              	"IndexingJobQueueItemLoadMultiplier": 10 // to configure how many queue items are loaded per loop
            }
        }
    },
    ...
}