de.tudarmstadt.ukp.dkpro.core.io.jwpl

Class WikipediaTemplateFilteredArticleReader

java.lang.Object

org.apache.uima.resource.Resource_ImplBase

org.apache.uima.resource.ConfigurableResource_ImplBase

org.apache.uima.collection.CollectionReader_ImplBase

org.apache.uima.fit.component.JCasCollectionReader_ImplBase

de.tudarmstadt.ukp.dkpro.core.io.jwpl.WikipediaReaderBase

de.tudarmstadt.ukp.dkpro.core.io.jwpl.WikipediaTemplateFilteredArticleReader

All Implemented Interfaces:

org.apache.uima.collection.base_cpm.BaseCollectionReader, org.apache.uima.collection.CollectionReader, org.apache.uima.resource.ConfigurableResource, org.apache.uima.resource.Resource

public class WikipediaTemplateFilteredArticleReader
extends WikipediaReaderBase

Reads all pages that contain or do not contain the templates specified in the template whitelist and template blacklist.

It is possible to just define a whitelist OR a blacklist. If both whitelist and blacklist are provided, the articles are chosen that DO contain the templates from the whitelist and at the same time DO NOT contain the templates from the blacklist (= the intersection of the "whitelist page set" and the "blacklist page set")

This reader only works if template tables have been generated for the JWPL database using the WikipediaTemplateInfoGenerator.

NOTE: This reader directly extends the WikipediaReaderBase and not the WikipediaStandardReaderBase

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARAM_DOUBLE_CHECK_ASSOCIATED_PAGES If this option is set, discussion pages are rejected that are associated with a blacklisted article.
static String	PARAM_EXACT_TEMPLATE_MATCHING Defines whether to match the templates exactly or whether to match all templates that start with the String given in the respective parameter list.
static String	PARAM_INCLUDE_DISCUSSION_PAGES Whether the reader should read also include talk pages.
static String	PARAM_LIMIT_NUMBER_OF_ARTICLES_TO_READ Optional parameter that allows to define the max number of articles that should be delivered by the reader.
static String	PARAM_ONLY_FIRST_PARAGRAPH If set to true, only the first paragraph instead of the whole article is used.
static String	PARAM_OUTPUT_PLAIN_TEXT Whether the reader outputs plain text or wiki markup.
static String	PARAM_PAGE_BUFFER The page buffer size (#pages) of the page iterator.
static String	PARAM_TEMPLATE_BLACKLIST Defines templates that the articles MUST NOT contain.
static String	PARAM_TEMPLATE_WHITELIST Defines templates that the articles MUST contain.

Fields inherited from class de.tudarmstadt.ukp.dkpro.core.io.jwpl.WikipediaReaderBase

dbconfig, PARAM_CREATE_DATABASE_CONFIG_ANNOTATION, PARAM_DB, PARAM_HOST, PARAM_LANGUAGE, PARAM_PASSWORD, PARAM_USER, wiki

Fields inherited from interface org.apache.uima.resource.Resource

PARAM_AGGREGATE_SOFA_MAPPINGS, PARAM_CONFIG_MANAGER, PARAM_CONFIG_PARAM_SETTINGS, PARAM_EXTERNAL_OVERRIDE_SETTINGS, PARAM_PERFORMANCE_TUNING_SETTINGS, PARAM_RESOURCE_MANAGER, PARAM_UIMA_CONTEXT

Constructor Summary

Constructors

Constructor and Description
WikipediaTemplateFilteredArticleReader()

Method Summary

Modifier and Type	Method and Description
void	getNext(org.apache.uima.jcas.JCas jcas)
org.apache.uima.util.Progress[]	getProgress()
boolean	hasNext()
void	initialize(org.apache.uima.UimaContext context)

Methods inherited from class org.apache.uima.fit.component.JCasCollectionReader_ImplBase

close, getLogger, getNext, initialize

Methods inherited from class org.apache.uima.collection.CollectionReader_ImplBase

destroy, getCasInitializer, getProcessingResourceMetaData, initialize, isConsuming, reconfigure, setCasInitializer, typeSystemInit

Methods inherited from class org.apache.uima.resource.ConfigurableResource_ImplBase

getConfigParameterValue, getConfigParameterValue, setConfigParameterValue, setConfigParameterValue

Methods inherited from class org.apache.uima.resource.Resource_ImplBase

getCasManager, getMetaData, getRelativePathResolver, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger, setMetaData

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.uima.resource.ConfigurableResource

getConfigParameterValue, getConfigParameterValue, setConfigParameterValue, setConfigParameterValue

Methods inherited from interface org.apache.uima.resource.Resource

getMetaData, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger

Field Detail

PARAM_ONLY_FIRST_PARAGRAPH

public static final String PARAM_ONLY_FIRST_PARAGRAPH

If set to true, only the first paragraph instead of the whole article is used.

See Also:

Constant Field Values

PARAM_OUTPUT_PLAIN_TEXT

public static final String PARAM_OUTPUT_PLAIN_TEXT

Whether the reader outputs plain text or wiki markup.

See Also:

Constant Field Values

PARAM_INCLUDE_DISCUSSION_PAGES

public static final String PARAM_INCLUDE_DISCUSSION_PAGES

Whether the reader should read also include talk pages.

See Also:

Constant Field Values

PARAM_DOUBLE_CHECK_ASSOCIATED_PAGES

public static final String PARAM_DOUBLE_CHECK_ASSOCIATED_PAGES

If this option is set, discussion pages are rejected that are associated with a blacklisted article. Analogously, articles are rejected that are associated with a blacklisted discussion page.

This check is rather expensive and could take a long time. This is option is not active if only a whitelist is used.

Default Value: false

See Also:

Constant Field Values

PARAM_LIMIT_NUMBER_OF_ARTICLES_TO_READ

public static final String PARAM_LIMIT_NUMBER_OF_ARTICLES_TO_READ

Optional parameter that allows to define the max number of articles that should be delivered by the reader.

This avoids unnecessary filtering if only a small number of articles is needed.

See Also:

Constant Field Values

PARAM_TEMPLATE_WHITELIST

public static final String PARAM_TEMPLATE_WHITELIST

Defines templates that the articles MUST contain.

If you also define a blacklist, the intersection of both sets is used. (= pages that DO contain templates from the whitelist, but DO NOT contain templates from the blacklist)

See Also:

Constant Field Values

PARAM_TEMPLATE_BLACKLIST

public static final String PARAM_TEMPLATE_BLACKLIST

Defines templates that the articles MUST NOT contain.

If you also define a whitelist, the intersection of both sets is used. (= pages that DO contain templates from the whitelist, but DO NOT contain templates from the blacklist)

See Also:

Constant Field Values

PARAM_EXACT_TEMPLATE_MATCHING

public static final String PARAM_EXACT_TEMPLATE_MATCHING

Defines whether to match the templates exactly or whether to match all templates that start with the String given in the respective parameter list.

Default Value: true

See Also:

Constant Field Values

PARAM_PAGE_BUFFER

public static final String PARAM_PAGE_BUFFER

The page buffer size (#pages) of the page iterator.

See Also:

Constant Field Values

Constructor Detail

WikipediaTemplateFilteredArticleReader

public WikipediaTemplateFilteredArticleReader()

Method Detail

initialize

public void initialize(org.apache.uima.UimaContext context)
                throws org.apache.uima.resource.ResourceInitializationException

Overrides:

initialize in class WikipediaReaderBase

Throws:

org.apache.uima.resource.ResourceInitializationException

hasNext

public boolean hasNext()
                throws IOException,
                       org.apache.uima.collection.CollectionException

Throws:

IOException

org.apache.uima.collection.CollectionException

getNext

public void getNext(org.apache.uima.jcas.JCas jcas)
             throws IOException,
                    org.apache.uima.collection.CollectionException

Overrides:

getNext in class WikipediaReaderBase

Throws:

IOException

org.apache.uima.collection.CollectionException

getProgress

public org.apache.uima.util.Progress[] getProgress()

Specified by:

getProgress in interface org.apache.uima.collection.base_cpm.BaseCollectionReader

Specified by:

getProgress in class WikipediaReaderBase