API and code to convert text into indexable tokens.