public class XtfSpecialTokensFilter
extends TokenFilter
XtfSpecialTokensFilter
class is used by the
XTFTextAnalyzer
class to convert special "bump" count values in
text chunks to actual position increments for words prior to adding them
to a Lucene index. IndexWriter.addDocument()
function is called, Lucene traverses the list of tokens, and calls an
instance of a TokenFilter
derived class to pre-process each
token. The resulting output from the filter is what Lucene actually
adds to the database. XtfSpecialTokensFilter
to pre-process. The
filter recognizes the special token, removes it from the token list,
converts it to a number, and sets it as the position increment for the
first non-special token that follows. The output of the
XtfSpecialTokensFilter
is then a list of actual tokens to be
indexed and their associated position increments. XMLTextProcessor
class, and its member function
insertVirtualWords()
.Modifier and Type | Field and Description |
---|---|
private String |
srcText
A reference to the original contiguous text that the input token list
corresponds.
|
Constructor and Description |
---|
XtfSpecialTokensFilter(TokenStream srcTokens,
String srcText)
Constructor for the
XtfSpecialTokensFilter . |
Modifier and Type | Method and Description |
---|---|
Token |
next()
Return the next output token from this filter.
|
private String srcText
public XtfSpecialTokensFilter(TokenStream srcTokens, String srcText)
XtfSpecialTokensFilter
. srcTokens
- The source token stream to filter.srcText
- The original source text chunk from wich the source
token stream was derived. public Token next() throws IOException
next
in class TokenStream
IOException
- Any exceptions generated by the look-back/look-ahead
character processing performed by this function.
XtfSpecialTokensFilter
class description.