Class BreakIteratorTokenizer

    • Constructor Detail

      • BreakIteratorTokenizer

        public BreakIteratorTokenizer​(Locale locale)
        Instantiates a new Break iterator tokenizer.
        Parameters:
        locale - the locale
    • Method Detail

      • tokenize

        public Iterable<Tokenizer.Token> tokenize​(Reader reader)
        Description copied from interface: Tokenizer
        Tokenizes an given reader into tokens. All IO errors should be rethrown as runtime exceptions.
        Specified by:
        tokenize in interface Tokenizer
        Parameters:
        reader - the reader
        Returns:
        an iterable of tokens.