de.tudarmstadt.ukp.dkpro.core.io.conll

Class Conll2002Reader

java.lang.Object

org.apache.uima.resource.Resource_ImplBase

org.apache.uima.resource.ConfigurableResource_ImplBase

org.apache.uima.collection.CollectionReader_ImplBase

org.apache.uima.fit.component.CasCollectionReader_ImplBase

de.tudarmstadt.ukp.dkpro.core.api.io.ResourceCollectionReaderBase

de.tudarmstadt.ukp.dkpro.core.api.io.JCasResourceCollectionReader_ImplBase

de.tudarmstadt.ukp.dkpro.core.io.conll.Conll2002Reader

All Implemented Interfaces:

org.apache.uima.collection.base_cpm.BaseCollectionReader, org.apache.uima.collection.CollectionReader, org.apache.uima.resource.ConfigurableResource, org.apache.uima.resource.Resource

public class Conll2002Reader
extends JCasResourceCollectionReader_ImplBase

Reads by default the CoNLL 2002 named entity format.

The reader is also compatible with the CoNLL-based GermEval 2014 named entity format, in which the columns are separated by a tab, and there is an extra column for embedded named entities, besides the token number being put in the first column (see below). For that, additional parameters are provided, by which one can determine the column separator, whether there is an additional first column for token numbers, and whether embedded named entities should be read. (Note: Currently, the reader only reads the outer named entities, not the embedded ones.

 The following snippet shows an example of the TSV format 
 # http://de.wikipedia.org/wiki/Manfred_Korfmann [2009-10-17]
 1  Aufgrund          O           O
 2  seiner            O           O
 3  Initiative        O           O
 4  fand              O           O
 5  2001/2002         O           O
 6  in                O           O
 7  Stuttgart         B-LOC       O
 8  ,                 O           O
 9  Braunschweig      B-LOC       O
 10 und               O           O
 11 Bonn              B-LOC       O
 12 eine              O           O
 13 große             O           O
 14 und               O           O
 15 publizistisch     O           O
 16 vielbeachtete     O           O
 17 Troia-Ausstellung B-LOCpart   O
 18 statt             O           O
 19 ,                 O           O
 20 „                 O           O
 21 Troia             B-OTH       B-LOC
 22 -                 I-OTH       O
 23 Traum             I-OTH       O
 24 und               I-OTH       O
 25 Wirklichkeit      I-OTH       O
 26 “                 O           O
 27 .                 O           O
 

1. WORD_NUMBER - token number
2. FORM - token
3. NER1 - outer named entity (BIO encoded)
4. NER2 - embedded named entity (BIO encoded)

The sentence is encoded as one token per line, with information provided in tab-separated columns. The first column contains either a #, which signals the source the sentence is cited from and the date it was retrieved, or the token number within the sentence. The second column contains the token. Name spans are encoded in the BIO-scheme. Outer spans are encoded in the third column, embedded spans in the fourth column.

See Also:

CoNLL 2002 shared task, GermEval 2014 NER task

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Conll2002Reader.ColumnSeparators Column Separators

Nested classes/interfaces inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.ResourceCollectionReaderBase

ResourceCollectionReaderBase.Resource

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARAM_COLUMN_SEPARATOR Column separator parameter.
static String	PARAM_HAS_EMBEDDED_NAMED_ENTITY Has embedded named entity extra column.
static String	PARAM_HAS_HEADER Indicates that there is a header line before the sentence
static String	PARAM_HAS_TOKEN_NUMBER Token number flag.
static String	PARAM_INTERN_TAGS Use the String.intern() method on tags.
static String	PARAM_NAMED_ENTITY_MAPPING_LOCATION Location of the mapping file for named entity tags to UIMA types.
static String	PARAM_READ_NAMED_ENTITY Read named entity information.
static String	PARAM_SOURCE_ENCODING Character encoding of the input data.

Fields inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.ResourceCollectionReaderBase

EXCLUDE_PREFIX, INCLUDE_PREFIX, JAR_PREFIX, KEY_RESOURCE_RESOLVER, PARAM_INCLUDE_HIDDEN, PARAM_LANGUAGE, PARAM_LOG_FREQ, PARAM_PATH, PARAM_PATTERNS, PARAM_SOURCE_LOCATION, PARAM_USE_DEFAULT_EXCLUDES

Fields inherited from interface org.apache.uima.resource.Resource

PARAM_AGGREGATE_SOFA_MAPPINGS, PARAM_CONFIG_MANAGER, PARAM_CONFIG_PARAM_SETTINGS, PARAM_EXTERNAL_OVERRIDE_SETTINGS, PARAM_PERFORMANCE_TUNING_SETTINGS, PARAM_RESOURCE_MANAGER, PARAM_UIMA_CONTEXT

Constructor Summary

Constructors

Constructor and Description
Conll2002Reader()

Method Summary

Modifier and Type	Method and Description
void	getNext(org.apache.uima.jcas.JCas aJCas) Subclasses implement this method rather than JCasResourceCollectionReader_ImplBase.getNext(CAS)
void	initialize(org.apache.uima.UimaContext aContext)

Methods inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.JCasResourceCollectionReader_ImplBase

getNext, initCas, initCas

Methods inherited from class de.tudarmstadt.ukp.dkpro.core.api.io.ResourceCollectionReaderBase

getBase, getBase, getDefaultExcludes, getLanguage, getProgress, getResolver, getResourceIterator, getResources, getSourceLocation, hasNext, initCas, initCas, isSingleLocation, locationToUrl, nextFile, scan

Methods inherited from class org.apache.uima.fit.component.CasCollectionReader_ImplBase

close, getLogger, initialize

Methods inherited from class org.apache.uima.collection.CollectionReader_ImplBase

destroy, getCasInitializer, getProcessingResourceMetaData, initialize, isConsuming, reconfigure, setCasInitializer, typeSystemInit

Methods inherited from class org.apache.uima.resource.ConfigurableResource_ImplBase

getConfigParameterValue, getConfigParameterValue, setConfigParameterValue, setConfigParameterValue

Methods inherited from class org.apache.uima.resource.Resource_ImplBase

getCasManager, getMetaData, getRelativePathResolver, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger, setMetaData

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.uima.resource.ConfigurableResource

getConfigParameterValue, getConfigParameterValue, setConfigParameterValue, setConfigParameterValue

Methods inherited from interface org.apache.uima.resource.Resource

getMetaData, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger

Field Detail

PARAM_COLUMN_SEPARATOR

public static final String PARAM_COLUMN_SEPARATOR

Column separator parameter. Acceptable input values come from Conll2002Reader.ColumnSeparators.
Example usage: if you want to define 'tab' as the column separator the following value should be input for this parameter Conll2002Reader.ColumnSeparators.TAB.getName()

See Also:

Constant Field Values

PARAM_HAS_TOKEN_NUMBER

public static final String PARAM_HAS_TOKEN_NUMBER

Token number flag. When true, the first column contains the token number inside the sentence (as in GermEval 2014 format)

See Also:

Constant Field Values

PARAM_HAS_HEADER

public static final String PARAM_HAS_HEADER

Indicates that there is a header line before the sentence

See Also:

Constant Field Values

PARAM_SOURCE_ENCODING

public static final String PARAM_SOURCE_ENCODING

Character encoding of the input data.

See Also:

Constant Field Values

PARAM_INTERN_TAGS

public static final String PARAM_INTERN_TAGS

Use the String.intern() method on tags. This is usually a good idea to avoid spamming the heap with thousands of strings representing only a few different tags. Default: true

See Also:

Constant Field Values

PARAM_READ_NAMED_ENTITY

public static final String PARAM_READ_NAMED_ENTITY

Read named entity information. Default: true

See Also:

Constant Field Values

PARAM_HAS_EMBEDDED_NAMED_ENTITY

public static final String PARAM_HAS_EMBEDDED_NAMED_ENTITY

Has embedded named entity extra column. Default: false

See Also:

Constant Field Values

PARAM_NAMED_ENTITY_MAPPING_LOCATION

public static final String PARAM_NAMED_ENTITY_MAPPING_LOCATION

Location of the mapping file for named entity tags to UIMA types.

See Also:

Constant Field Values

Constructor Detail

Conll2002Reader

public Conll2002Reader()

Method Detail

initialize

public void initialize(org.apache.uima.UimaContext aContext)
                throws org.apache.uima.resource.ResourceInitializationException

Overrides:

initialize in class ResourceCollectionReaderBase

Throws:

org.apache.uima.resource.ResourceInitializationException

getNext

public void getNext(org.apache.uima.jcas.JCas aJCas)
             throws IOException,
                    org.apache.uima.collection.CollectionException

Description copied from class: JCasResourceCollectionReader_ImplBase

Subclasses implement this method rather than JCasResourceCollectionReader_ImplBase.getNext(CAS)

Specified by:

getNext in class JCasResourceCollectionReader_ImplBase

Parameters:

aJCas - the JCas.

Throws:

IOException - if an i/o error occurs reading the data.

org.apache.uima.collection.CollectionException - if another type of error occurs.