Linguistic processing

A text search server provides dictionary packs to support the linguistic processing of documents and queries that are not in English.

As an alternative to dictionary-based word segmentation, which uses a language-specific dictionary to identify words in the sequence of characters in the document, the search engine uses n-gram segmentation to support the linguistic processing of Chinese, Japanese, and Korean. N-gram segmentation is a method of analysis that considers overlapping sequences of a given number of characters as a single word rather than using blank space to delimit words.

If a text document is in one of the supported languages, linguistic processing is carried out when the input is parsed into tokens (which means when the text is separated into individual words).

For unsupported languages, the document is parsed by using blank space or n-gram segmentation, depending on the script of each character. If the script is used by a language that usually does not use blank space as a word delimiter, such as Han, Hiragana, Hangul, then the document is parsed by n-gram segmentation. Otherwise, the document is parsed by using blank space. Lemmatization, a process that identifies the root form and different grammatical forms of a word, is not performed on unsupported languages.

When you search a text search index, a match is indicated that contains linguistic variations of the query terms. The variations of a word depend on the language of the query.