Uses of Class
com.gengoai.hermes.morphology.Tokenizer.Token
-
Packages that use Tokenizer.Token Package Description com.gengoai.hermes.en com.gengoai.hermes.morphology -
-
Uses of Tokenizer.Token in com.gengoai.hermes.en
Methods in com.gengoai.hermes.en that return types with arguments of type Tokenizer.Token Modifier and Type Method Description Iterable<Tokenizer.Token>ENTokenizer. tokenize(@NonNull Reader reader) -
Uses of Tokenizer.Token in com.gengoai.hermes.morphology
Methods in com.gengoai.hermes.morphology that return Tokenizer.Token Modifier and Type Method Description Tokenizer.TokenStandardTokenizer. next()Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.Methods in com.gengoai.hermes.morphology that return types with arguments of type Tokenizer.Token Modifier and Type Method Description Iterable<Tokenizer.Token>BreakIteratorTokenizer. tokenize(Reader reader)Iterable<Tokenizer.Token>BreakIteratorTokenizer. tokenize(String input)default Iterable<Tokenizer.Token>Tokenizer. tokenize(@NonNull String input)Tokenizes a given string into token.Iterable<Tokenizer.Token>Tokenizer. tokenize(Reader reader)Tokenizes an given reader into tokens.
-