Dcipher Analytics Help Center
Go to Dcipher Analytics
All Collections
Text segmentation & tokenization

Text segmentation & tokenization

Split your text into manageable pieces through text segmentation and tokenization.

Aslı Duruoğlu avatar Tomas Larsson avatar
3 articles in this collection
Written by Aslı Duruoğlu and Tomas Larsson

Smart segmentation

The Segment Text operation splits text into segments based on semantic similarity as well as references between adjacent sentences.
Aslı Duruoğlu avatar
Written by Aslı Duruoğlu
Updated over a week ago

Pattern-based text splitting

The Split by Pattern operation splits text by pattern, for example into sentences or paragraphs.
Aslı Duruoğlu avatar
Written by Aslı Duruoğlu
Updated over a week ago

Tokenization: splitting text into words and phrases

The Tokenize & Tag operation splits text into meaningful units in the form of words, phrases, and named entities, tagged by part-of-speech.
Tomas Larsson avatar
Written by Tomas Larsson
Updated over a week ago
Dcipher Analytics Help Center
  • About us
  • Our people
  • Privacy policy
  • Blog
  • Use cases