Semantic search
📘 Note Semantic search is an experimental feature and may be subject to change.
Optimizely Graph uses AI to match and rank content beyond standard lexical (literal keyword) matching. Pre-trained language models extend keyword matching by capturing the meaning of words and the intent behind a query. This approach is commonly called semantic search, and is also known as neural search, vector search (matching based on numeric representations of meaning called vector embeddings), and dense or sparse neural retrieval. For technical background, see the blog post Do you know what I mean? Introducing semantic search in Optimizely Graph.
Semantic search solves the vocabulary mismatch problem: the keywords a site visitor enters often differ from the words a content marketer uses. For example, a visitor might search for "non-alcoholic cold beverage" and expect content about "cola." A standard search does not return content when the query keywords do not display in the content. Semantic search returns the cola content because it captures the meaning behind the query.
One way to solve the vocabulary mismatch problem is to create synonymssynonyms, which Optimizely Graph supports. Synonyms are effective but time-consuming to maintain. Semantic search automates the work by expanding queries with related meaning and improves the ranking of results.
Semantic search also supports conversational AI, such as chatbots, by supplying relevant results that reduce hallucinations (AI-generated answers that sound plausible but are not grounded in the source content). This technique is called retrieval augmented generation (RAG). See the blog post How to create AI-driven chatbots with Optimizely Graph for how to implement RAG with Optimizely Graph. Use the same vector search technology to cluster content, de-duplicate content, and detect anomalies.
Semantic search works best in combination with traditional keyword search, where the system returns the most relevant content in the order visitors expect. Optimizely Graph treats this as the canonical mode and refers to it as semantic search, although it is a mixed approach between standard keyword search and pure vector search.
How semantic search works in Optimizely Graph
This section explains where semantic search applies in the schema and how to switch a query from keyword ranking to semantic ranking, so you know what to configure before running the examples that follow.
Semantic search applies to searchable string fields and to the full-text search operators contains and match. Set fields with a lot of content, such as the MainBody in Optimizely CMS, as searchable to unlock the full-text search capabilities. Optimizely Graph uses a pre-trained model for semantic search.
To enable semantic search, change the value of _ranking in the orderby to SEMANTIC. By default, Optimizely Graph uses RELEVANCE, which is a standard keyword search ranked with BM25, a scoring formula that weights matches by term frequency and document length. Combine SEMANTIC with other ranking criteria. For example, use the semantic search capabilities and rank the results by field value instead.
Configure the weight of semantic search
Tune how much semantic ranking contributes relative to keyword ranking so that the result order matches your audience's expectations. A higher weight emphasizes meaning over literal keywords; a lower weight favors exact matches.
Optimizely Graph computes the relevance score by adding the semantic-match contribution to the keyword score. Configure the weight of semantic search with the _semanticWeight argument in the orderBy. The default value is 0.2. Adjust the weight by testing it against your use case, queries, and data.
📘 Note The _semanticWeight must be a positive floating point number. When a negative value is provided, semantic search is disabled.
The maximum allowable value is the maximum value of an integer. When the value exceeds that maximum, the system uses the maximum integer value.
The following example has the _semanticWeight set to 0.5:
{
Content(
orderBy: { _ranking: SEMANTIC, _semanticWeight: 0.5 }
where: { _fulltext: { match: "greatest president" } }
) {
total
items {
_score
_fulltext
}
}
}Supported languages
Use this table to confirm that the language of your content has a pre-trained model before you enable semantic ranking. Content in unsupported locales falls back to standard keyword ranking.
The models for semantic search are trained on English and multilingual datasets. Optimizely Graph supports the languages with the following locale values:
Language | 2-letter locale |
|---|---|
Arabic | ar |
Bulgarian | bg |
Catalan | ca |
CJK (special) | cjk |
German | de |
Greek | el |
English | en |
Spanish | es |
Farsi | fa |
Finnish | fi |
French | fr |
Galician | gl |
Hindi | hi |
Hungarian | hu |
Armenian | hy |
Indonesian | id |
Italian | it |
Japanese | ja |
Korean | ko |
Kurdish | ku |
Latvian | lv |
Dutch | nl |
Norwegian | no |
Polish | pl |
Romanian | ro |
Russian | ru |
Swedish | sv |
Thai | th |
Turkish | tr |
Ukrainian | uk |
Chinese | zh |
Examples
The following examples show how semantic ranking changes the result order for queries where literal keyword matches alone would miss the most relevant content.
In the following query, the user wants content about action movie:
{
Content(orderBy: { _ranking: SEMANTIC }, where: { MainBody: { contains: "action movie" } }) {
total
items {
_score
Name
MainBody
_fulltext
}
}
}Optimizely Graph returns the most relevant content to the top.
📘 Note Both "action" and "movie" do not display in the content.
{
"Content": {
"total": 4,
"items": [
{
"_score": 0.013479907,
"Name": "Standard Page 12",
"MainBody": "Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
"_fulltext": [
"Wild Wild West is a 1999 American steampunk Western film co-produced and directed by Barry Sonnenfeld and written by S. S. Wilson and Brent Maddock alongside Jeffrey Price and Peter S. Seaman, from a story penned by brothers Jim and John Thomas. Loosely adapted from The Wild Wild West, a 1960s television series created by Michael Garrison, it is the only production since the television film More Wild Wild West (1980) to feature the characters from the original series. The film stars Will Smith (who previously collaborated with Sonnenfeld on Men in Black two years earlier in 1997) and Kevin Kline as two U.S. Secret Service agents who work together to protect U.S. President Ulysses S. Grant (Kline, in a dual role) and the United States from all manner of dangerous threats during the American Old West.",
"Standard Page 12"
]
},
{
"_score": 0.011364636,
"Name": "Temporary Page Title",
"MainBody": "The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"_fulltext": [
"The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"Temporary Page Title"
]
},
{
"_score": 0.011052586,
"Name": "Wilder Westen",
"MainBody": "Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
"_fulltext": [
"Wilder Westen ist eine – geographisch und historisch grob eingegrenzte – umgangssprachliche Bezeichnung für die ungefähr westlich des Mississippi gelegenen Gebiete der heutigen Vereinigten Staaten. In der auch als „Pionierzeit“ bezeichneten Ära des 19. Jahrhunderts waren sie noch nicht als Bundesstaaten in die Union der Vereinigten Staaten aufgenommen. Im Verlauf der voranschreitenden Landnahme und Urbanisierung nahm die Besiedlung dieser Regionen vor allem durch Angloamerikaner – bzw. aus Europa stammende Immigranten – kontinuierlich zu, bis die Gebiete um 1890 in den organisierten Territorien der Vereinigten Staaten aufgingen. Symbolisch stehen die Öffnung der letzten Indianerterritorien im späteren US-Bundesstaat Oklahoma für die Besiedlung durch Kolonisten 1889–1895 durch eine Serie von Land Runs und das Massaker der United States Army an etwa 200 bis 300 Lakota am Wounded Knee Creek/South Dakota im Dezember 1890 für das Ende der Zeit des Wilden Westens. Mit diesen Ereignissen galten die Indianerkriege ebenso als abgeschlossen wie die Kolonisation der bis dahin von den Vereinigten Staaten beanspruchten Hoheitsgebiete (engl. territories) durch die aus Europa eingewanderten Siedler.",
"Wilder Westen"
]
},
{
"_score": 0.009704749,
"Name": "Arnold Schwarzenegger",
"MainBody": null,
"_fulltext": [
"Arnold Schwarzenegger"
]
}
]
}
}Similarly, when a site visitor queries for california governor with the following query:
{
Content(
locale: en,
orderBy: { _ranking: SEMANTIC }
where: { _fulltext: { match: "california governor" } }
) {
total
items {
_score
Name
_fulltext
}
}
}The system returns the most relevant content at the top.
{
"Content": {
"total": 3,
"items": [
{
"_score": 0.057718195,
"Name": "Arnold Schwarzenegger",
"_fulltext": [
"Arnold Schwarzenegger"
]
},
{
"_score": 0.054158323,
"Name": "Temporary Page Title",
"_fulltext": [
"The American frontier, also known as the Old West, popularly known as the Wild West, encompasses the geography, history, folklore, and culture associated with the forward wave of American expansion in mainland North America that began with European colonial settlements in the early 17th century and ended with the admission of the last few contiguous western territories as states in 1912. This era of massive migration and settlement was particularly encouraged by President Thomas Jefferson following the Louisiana Purchase, giving rise to the expansionist attitude known as \"Manifest Destiny\" and the historians' \"Frontier Thesis\". The legends, historical events and folklore of the American frontier have embedded themselves into United States culture so much so that the Old West, and the Western genre of media specifically, has become one of the defining periods of American national identity.",
"Temporary Page Title"
]
},
{
"_score": 0.05409809,
"Name": "Alan Turing",
"_fulltext": [
"1913-06-22T17:17:56Z",
"Alan Turing",
"true",
"1",
"2",
"3",
"4",
"5"
]
}
]
}
}Combine ranking criteria to mix attribute-based ordering with semantic relevance. For example, rank primarily by date in ascending order and use the semantic relevance score as a tie-breaker, such as in the following example:
{
BiographyPage(
locale: en
orderBy: { Date: ASC, _ranking: SEMANTIC }
where: { _fulltext: { match: "scientist" } }
) {
total
items {
Name
Date
_fulltext
}
}
}