Class ENTokenizer

    • Constructor Detail

      • ENTokenizer

        public ENTokenizer()
        Instantiates a new English tokenizer.
    • Method Detail

      • tokenize

        public Iterable<Tokenizer.Token> tokenize​(@NonNull
                                                  @NonNull Reader reader)
        Description copied from interface: Tokenizer
        Tokenizes an given reader into tokens. All IO errors should be rethrown as runtime exceptions.
        Specified by:
        tokenize in interface Tokenizer
        Parameters:
        reader - the reader
        Returns:
        an iterable of tokens.