Class StandardTokenizer


  • public class StandardTokenizer
    extends Object
    This class is a scanner generated by JFlex 1.5.0-SNAPSHOT from the specification file /home/ik/prj/gengoai/hermes-pom/core/src/main/jflex/StandardTokenizer.jflex
    • Constructor Detail

      • StandardTokenizer

        public StandardTokenizer​(Reader in)
        Creates a new scanner There is also a java.io.InputStream version of this constructor.
        Parameters:
        in - the java.io.Reader to read input from.
      • StandardTokenizer

        public StandardTokenizer​(InputStream in)
        Creates a new scanner. There is also java.io.Reader version of this constructor.
        Parameters:
        in - the java.io.Inputstream to read input from.
    • Method Detail

      • yyreset

        public final void yyreset​(Reader reader)
        Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL. Internal scan buffer is resized down to its initial length, if it has grown.
        Parameters:
        reader - the new input stream
      • yystate

        public final int yystate()
        Returns the current lexical state.
      • yybegin

        public final void yybegin​(int newState)
        Enters a new lexical state
        Parameters:
        newState - the new lexical state
      • yytext

        public final String yytext()
        Returns the text matched by the current regular expression.
      • yycharat

        public final char yycharat​(int pos)
        Returns the character at position pos from the matched text. It is equivalent to yytext().charAt(pos), but faster
        Parameters:
        pos - the position of the character to fetch. A value from 0 to yylength()-1.
        Returns:
        the character at position pos
      • yylength

        public final int yylength()
        Returns the length of the matched text region.
      • yypushback

        public void yypushback​(int number)
        Pushes the specified amount of characters back into the input stream. They will be read again by then next call of the scanning method
        Parameters:
        number - the number of characters to be read again. This number must not be greater than yylength()!
      • next

        public Tokenizer.Token next()
                             throws IOException
        Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
        Returns:
        the next token
        Throws:
        IOException - if any I/O-Error occurs