org.dkpro.core.tokit

Class PatternBasedTokenSegmenter

java.lang.Object

org.apache.uima.analysis_component.AnalysisComponent_ImplBase

org.apache.uima.analysis_component.Annotator_ImplBase

org.apache.uima.analysis_component.JCasAnnotator_ImplBase

org.apache.uima.fit.component.JCasAnnotator_ImplBase

org.dkpro.core.tokit.PatternBasedTokenSegmenter

All Implemented Interfaces:

org.apache.uima.analysis_component.AnalysisComponent

public class PatternBasedTokenSegmenter
extends org.apache.uima.fit.component.JCasAnnotator_ImplBase

Split up existing tokens again at particular split-chars. The prefix states whether the split chars should be added as separate Tokens. If the INCLUDE_PREFIX precedes the split pattern, the pattern is included. Consequently, patterns following the EXCLUDE_PREFIX, will not be added as a Token.

Field Summary

Fields

Modifier and Type	Field and Description
static String	EXCLUDE_PREFIX
static String	INCLUDE_PREFIX
static String	PARAM_DELETE_COVER Whether to remove the original token.
static String	PARAM_PATTERNS A list of regular expressions, prefixed with INCLUDE_PREFIX or EXCLUDE_PREFIX.

Constructor Summary

Constructors

Constructor and Description
PatternBasedTokenSegmenter()

Method Summary

Modifier and Type	Method and Description
void	initialize(org.apache.uima.UimaContext aContext)
void	process(org.apache.uima.jcas.JCas aJCas)

Methods inherited from class org.apache.uima.fit.component.JCasAnnotator_ImplBase

getLogger

Methods inherited from class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

getRequiredCasInterface, process

Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase

getCasInstancesRequired, hasNext, next

Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase

batchProcessComplete, collectionProcessComplete, destroy, getContext, getResultSpecification, reconfigure, setResultSpecification

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INCLUDE_PREFIX

public static final String INCLUDE_PREFIX

See Also:

Constant Field Values

EXCLUDE_PREFIX

public static final String EXCLUDE_PREFIX

See Also:

Constant Field Values

PARAM_DELETE_COVER

public static final String PARAM_DELETE_COVER

Whether to remove the original token.

See Also:

Constant Field Values

PARAM_PATTERNS

public static final String PARAM_PATTERNS

A list of regular expressions, prefixed with INCLUDE_PREFIX or EXCLUDE_PREFIX. If neither of the prefixes is used, EXCLUDE_PREFIX is assumed.

See Also:

Constant Field Values

Constructor Detail

PatternBasedTokenSegmenter

public PatternBasedTokenSegmenter()

Method Detail

initialize

public void initialize(org.apache.uima.UimaContext aContext)
                throws org.apache.uima.resource.ResourceInitializationException

Specified by:

initialize in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

initialize in class org.apache.uima.fit.component.JCasAnnotator_ImplBase

Throws:

org.apache.uima.resource.ResourceInitializationException

process

public void process(org.apache.uima.jcas.JCas aJCas)
             throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Specified by:

process in class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException