public class FastTokenizer
extends Tokenizer
Modifier and Type | Class and Description |
---|---|
private class |
FastTokenizer.DribbleReader
This class is used, when the fast tokenizer encounters a questionable
situation, to dribble out characters to a standard tokenizer that can
do a more complete job.
|
Modifier and Type | Field and Description |
---|---|
private static char[] |
charType |
private FastTokenizer.DribbleReader |
dribbleReader
Used to dribble out tokens to a standard tokenizer; used when
we encounter a case that's hard to figure out.
|
(package private) static char |
fakeChar
We use a special character to mark the end of a
FastTokenizer.DribbleReader . |
(package private) static String |
fakeWord
This is the special word used by DribbleReader
|
private int |
pos
Position within the
source array |
private char[] |
source
Array of characters to read from
|
private Tokenizer |
stdTokenizer
Standard tokenizer, used for hard cases only
|
Constructor and Description |
---|
FastTokenizer(FastStringReader reader)
Create a tokenizer that will tokenize the stream of characters from
the given reader.
|
Modifier and Type | Method and Description |
---|---|
Token |
next()
Retrieve the next token in the stream, or null if there are no more.
|
private static void |
setCharType(char type,
char from,
char to)
Utility method used when setting up the character type table
|
private char[] source
private int pos
source
arraystatic final char fakeChar
FastTokenizer.DribbleReader
.static final String fakeWord
private FastTokenizer.DribbleReader dribbleReader
private Tokenizer stdTokenizer
private static final char[] charType
public FastTokenizer(FastStringReader reader)
reader
- Reader to get data from.private static void setCharType(char type, char from, char to)
public Token next() throws IOException
next
in class TokenStream
IOException