de.tudarmstadt.ukp.dkpro.core.io.web1t

Class Web1TWriter

java.lang.Object

org.apache.uima.analysis_component.AnalysisComponent_ImplBase

org.apache.uima.analysis_component.Annotator_ImplBase

org.apache.uima.analysis_component.JCasAnnotator_ImplBase

org.apache.uima.fit.component.JCasAnnotator_ImplBase

de.tudarmstadt.ukp.dkpro.core.io.web1t.Web1TWriter

All Implemented Interfaces:

org.apache.uima.analysis_component.AnalysisComponent

public class Web1TWriter
extends org.apache.uima.fit.component.JCasAnnotator_ImplBase

Web1T n-gram index format writer.

Field Summary

Fields

Modifier and Type	Field and Description
protected String	contextType
static String	PARAM_CONTEXT_TYPE The type being used for segments
static String	PARAM_CREATE_INDEXES Create the indexes that jWeb1T needs to operate.
static String	PARAM_INPUT_TYPES Types to generate n-grams from.
static String	PARAM_LOWERCASE Create a lower case index.
static String	PARAM_MAX_NGRAM_LENGTH Maximum n-gram length.
static String	PARAM_MIN_FREQUENCY Specifies the minimum frequency a NGram must have to be written to the final index.
static String	PARAM_MIN_NGRAM_LENGTH Minimum n-gram length.
static String	PARAM_SPLIT_TRESHOLD The input file(s) is/are split into smaller files for quick access.
static String	PARAM_TARGET_ENCODING Character encoding of the output data.
static String	PARAM_TARGET_LOCATION Location to which the output is written.

Constructor Summary

Constructors

Constructor and Description
Web1TWriter()

Method Summary

Modifier and Type	Method and Description
void	collectionProcessComplete() The input files for each ngram level is read, splitted according to the frequency of the words starting letter in the files and the split files are individually sorted and consolidated.
void	initialize(org.apache.uima.UimaContext context)
void	process(org.apache.uima.jcas.JCas jcas)

Methods inherited from class org.apache.uima.fit.component.JCasAnnotator_ImplBase

getLogger

Methods inherited from class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

getRequiredCasInterface, process

Methods inherited from class org.apache.uima.analysis_component.Annotator_ImplBase

getCasInstancesRequired, hasNext, next

Methods inherited from class org.apache.uima.analysis_component.AnalysisComponent_ImplBase

batchProcessComplete, destroy, getContext, getResultSpecification, reconfigure, setResultSpecification

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARAM_INPUT_TYPES

public static final String PARAM_INPUT_TYPES

Types to generate n-grams from. Example: Token.class.getName() + "/pos/PosValue" for part-of-speech n-grams

See Also:

Constant Field Values

PARAM_TARGET_LOCATION

public static final String PARAM_TARGET_LOCATION

Location to which the output is written.

See Also:

Constant Field Values

PARAM_TARGET_ENCODING

public static final String PARAM_TARGET_ENCODING

Character encoding of the output data.

See Also:

Constant Field Values

PARAM_MIN_NGRAM_LENGTH

public static final String PARAM_MIN_NGRAM_LENGTH

Minimum n-gram length. Default: 1

See Also:

Constant Field Values

PARAM_MAX_NGRAM_LENGTH

public static final String PARAM_MAX_NGRAM_LENGTH

Maximum n-gram length. Default: 3

See Also:

Constant Field Values

PARAM_LOWERCASE

public static final String PARAM_LOWERCASE

Create a lower case index.

See Also:

Constant Field Values

PARAM_CREATE_INDEXES

public static final String PARAM_CREATE_INDEXES

Create the indexes that jWeb1T needs to operate. (default: true)

See Also:

Constant Field Values

PARAM_MIN_FREQUENCY

public static final String PARAM_MIN_FREQUENCY

Specifies the minimum frequency a NGram must have to be written to the final index. The specified value is interpreted as inclusive value, the default is 1. Thus, all NGrams with a frequency of at least 1 or higher will be written.

See Also:

Constant Field Values

PARAM_SPLIT_TRESHOLD

public static final String PARAM_SPLIT_TRESHOLD

The input file(s) is/are split into smaller files for quick access. An own file is created if the first two starting letters (or the starting letter if the word has a length of 1 character) account for at least x% of all starting letters in the input file(s). The default value for splitting a file is 1.0%. Every word that has starting characters which does not suffice the threshold is written with other words that also did not meet the threshold into an own file for miscellaneous words. A high threshold will lead to only a few, but large files and a most likely very large misc. file. A low threshold results in many small files. Use a zero or a negative value to write everything to one file.

See Also:

Constant Field Values

PARAM_CONTEXT_TYPE

public static final String PARAM_CONTEXT_TYPE

The type being used for segments

See Also:

Constant Field Values

contextType

protected String contextType

Constructor Detail

Web1TWriter

public Web1TWriter()

Method Detail

initialize

public void initialize(org.apache.uima.UimaContext context)
                throws org.apache.uima.resource.ResourceInitializationException

Specified by:

initialize in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

initialize in class org.apache.uima.fit.component.JCasAnnotator_ImplBase

Throws:

org.apache.uima.resource.ResourceInitializationException

process

public void process(org.apache.uima.jcas.JCas jcas)
             throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

Specified by:

process in class org.apache.uima.analysis_component.JCasAnnotator_ImplBase

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException

collectionProcessComplete

public void collectionProcessComplete()
                               throws org.apache.uima.analysis_engine.AnalysisEngineProcessException

The input files for each ngram level is read, splitted according to the frequency of the words starting letter in the files and the split files are individually sorted and consolidated.

Specified by:

collectionProcessComplete in interface org.apache.uima.analysis_component.AnalysisComponent

Overrides:

collectionProcessComplete in class org.apache.uima.analysis_component.AnalysisComponent_ImplBase

Throws:

org.apache.uima.analysis_engine.AnalysisEngineProcessException