de.tudarmstadt.ukp.dkpro.core.io.text

Class TokenizedTextWriter

java.lang.Object

org.apache.uima.analysis_component.AnalysisComponent_ImplBase

org.apache.uima.analysis_component.Annotator_ImplBase

org.apache.uima.analysis_component.JCasAnnotator_ImplBase

org.apache.uima.fit.component.JCasConsumer_ImplBase

de.tudarmstadt.ukp.dkpro.core.api.io.JCasFileWriter_ImplBase

de.tudarmstadt.ukp.dkpro.core.io.text.TokenizedTextWriter

All Implemented Interfaces:

org.apache.uima.analysis_component.AnalysisComponent

public class TokenizedTextWriter
extends JCasFileWriter_ImplBase

This class writes a set of pre-processed documents into a large text file containing one sentence per line and tokens split by whitespaces. Optionally, annotations other than tokens (e.g. lemmas) are written as specified by PARAM_FEATURE_PATH.

Nested Class Summary

Nested classes/interfaces inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.JCasFileWriter_ImplBase

JCasFileWriter_ImplBase.NamedOutputStream

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARAM_COVERING_TYPE In the output file, each unit of the covering type is written into a separate line.
static String	PARAM_EXTENSION Set the output file extension.
static String	PARAM_FEATURE_PATH The feature path, e.g.
static String	PARAM_NUMBER_REGEX
static String	PARAM_STOPWORDS_FILE All the tokens listed in this file (one token per line) are replaced by STOP.
static String	PARAM_TARGET_ENCODING Encoding for the target file.

Fields inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.JCasFileWriter_ImplBase

JAR_PREFIX, PARAM_COMPRESSION, PARAM_ESCAPE_DOCUMENT_ID, PARAM_OVERWRITE, PARAM_SINGULAR_TARGET, PARAM_STRIP_EXTENSION, PARAM_TARGET_LOCATION, PARAM_USE_DOCUMENT_ID

Constructor Summary

Constructors

Constructor and Description
TokenizedTextWriter()

Method Summary

Modifier and Type	Method and Description
void	collectionProcessComplete()
void	initialize(org.apache.uima.UimaContext context)
void	process(org.apache.uima.jcas.JCas aJCas)

Methods inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.JCasFileWriter_ImplBase

getCompressionMethod, getOutputStream, getOutputStream, getRelativePath, getTargetLocation, isStripExtension, isUseDocumentId

Methods inherited from class org.apache.uima.fit.component.JCasConsumer_ImplBase

getLogger

Methods inherited from class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

getRequiredCasInterface, process

Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase

getCasInstancesRequired, hasNext, next

Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase

batchProcessComplete, destroy, getContext, getResultSpecification, reconfigure, setResultSpecification

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARAM_TARGET_ENCODING

public static final String PARAM_TARGET_ENCODING

Encoding for the target file. Default is UTF-8.

See Also:

Constant Field Values

PARAM_FEATURE_PATH

public static final String PARAM_FEATURE_PATH

The feature path, e.g. de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token/lemma/value for lemmas. Default: de.tudarmstadt.ukp.dkpro.core.api.segmentation.type.Token (i.e. token texts).

See Also:

Constant Field Values

PARAM_NUMBER_REGEX

public static final String PARAM_NUMBER_REGEX

See Also:

Constant Field Values

PARAM_STOPWORDS_FILE

public static final String PARAM_STOPWORDS_FILE

All the tokens listed in this file (one token per line) are replaced by STOP. Empty lines and lines starting with # are ignored. Casing is ignored.

See Also:

Constant Field Values

PARAM_EXTENSION

public static final String PARAM_EXTENSION

Set the output file extension. Default: .txt.

See Also:

Constant Field Values

PARAM_COVERING_TYPE

public static final String PARAM_COVERING_TYPE

In the output file, each unit of the covering type is written into a separate line. The default (set in DEFAULT_COVERING_TYPE), is sentences so that each sentence is written to a line.

If no linebreaks within a document is desired, set this value to null.

See Also:

Constant Field Values

Constructor Detail

TokenizedTextWriter

public TokenizedTextWriter()

Method Detail

initialize

public void initialize(org.apache.uima.UimaContext context)
                throws org.apache.uima.resource.ResourceInitializationException

Specified by:

initialize in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

initialize in class org.apache.uima.fit.component.JCasConsumer_ImplBase

Throws:

org.apache.uima.resource.ResourceInitializationException

process

public void process(org.apache.uima.jcas.JCas aJCas)
             throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Specified by:

process in class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException

collectionProcessComplete

public void collectionProcessComplete()
                               throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Specified by:

collectionProcessComplete in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

collectionProcessComplete in class JCasFileWriter_ImplBase

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException