public class TokenizerAdapter extends AbstractTokenizer<java.lang.String>
java.io.StreamTokenizer
and a edu.stanford.nlp.process.Tokenizer.| Modifier and Type | Field and Description |
|---|---|
protected java.lang.String |
eolString |
protected java.io.StreamTokenizer |
st |
nextToken| Constructor and Description |
|---|
TokenizerAdapter(java.io.StreamTokenizer st)
Create a new
TokenizerAdaptor. |
| Modifier and Type | Method and Description |
|---|---|
java.lang.String |
getNext()
Internally fetches the next token.
|
boolean |
isEol(java.lang.String str)
Say whether the
String is the end-of-line token for
this tokenizer. |
void |
setEolString(java.lang.String eolString)
Set the
String returned when the inner tokenizer
returns an end-of-line token. |
hasNext, next, peek, remove, tokenizeprotected final java.io.StreamTokenizer st
protected java.lang.String eolString
public TokenizerAdapter(java.io.StreamTokenizer st)
TokenizerAdaptor. In general, it is
recommended that the passed in StreamTokenizer should
have had resetSyntax() done to it, so that numbers are
returned as entered as tokens of type String, though this
code will cope as best it can.st - The internal java.io.StreamTokenizerpublic java.lang.String getNext()
getNext in class AbstractTokenizer<java.lang.String>public void setEolString(java.lang.String eolString)
String returned when the inner tokenizer
returns an end-of-line token. This will only happen if the
inner tokenizer has been set to eolIsSignificant(true).eolString - The String used to represent eol. It is not allowed
to be null (which would confuse line ends and file end)public boolean isEol(java.lang.String str)
String is the end-of-line token for
this tokenizer.str - The String being tested