org.dkpro.core.frequency

Class BerkeleyLmProvider

java.lang.Object

org.dkpro.core.frequency.BerkeleyLmProvider

All Implemented Interfaces:

FrequencyCountProvider

public class BerkeleyLmProvider
extends Object
implements FrequencyCountProvider

Wrapper for the Berkeley LM package. nGram index files are looked up in DKPRO_HOME directory.

Constructor Summary

Constructors

Constructor and Description
BerkeleyLmProvider(String binaryFile, String language)

Method Summary

Modifier and Type	Method and Description
long	getFrequency(String phrase) Get frequency for a phrase
String	getID() Returns an ID that identifies the frequency provider
String	getLanguage() Returns the language for which the frequencies are provided.
double	getLogProbability(String phrase) Get the log probability for a phrase
Iterator<String>	getNgramIterator(int n) Returns an iterator over all ngrams of a given size in the corpus.
long	getNrOfDistinctNgrams(int n) Returns the number of distinct ngrams of a given size in the corpus.
long	getNrOfNgrams(int n) Returns the number of ngrams of a given size in the corpus.
long	getNrOfTokens()
double	getProbability(String phrase) Get the probability (=normalized frequency) for a phrase. The frequency count is normalized by the number of phrases in the corpus.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BerkeleyLmProvider

public BerkeleyLmProvider(String binaryFile,
                          String language)
                   throws Exception

Throws:

Exception

Method Detail

getFrequency

public long getFrequency(String phrase)

Description copied from interface: FrequencyCountProvider

Get frequency for a phrase

Specified by:

getFrequency in interface FrequencyCountProvider

Parameters:

phrase - phrase to search

Returns:

frequency of the phrase. Returns 0 if the phrase does not exist in the corpus.

getProbability

public double getProbability(String phrase)

Description copied from interface: FrequencyCountProvider

Get the probability (=normalized frequency) for a phrase.
The frequency count is normalized by the number of phrases in the corpus.

Specified by:

getProbability in interface FrequencyCountProvider

Parameters:

phrase - phrase to search

Returns:

probability (=normalized frequency) of the phrase. Returns null if the phrase does not exist in the corpus.

getLogProbability

public double getLogProbability(String phrase)

Description copied from interface: FrequencyCountProvider

Get the log probability for a phrase

Specified by:

getLogProbability in interface FrequencyCountProvider

Parameters:

phrase - phrase to search

Returns:

The log probability of the. Returns 0 if the phrase does not exist in the corpus.

getNrOfTokens

public long getNrOfTokens()

Specified by:

getNrOfTokens in interface FrequencyCountProvider

Returns:

The number of tokens in the corpus.

getNrOfNgrams

public long getNrOfNgrams(int n)

Description copied from interface: FrequencyCountProvider

Returns the number of ngrams of a given size in the corpus.

Specified by:

getNrOfNgrams in interface FrequencyCountProvider

Parameters:

n - the ngram size.

Returns:

the number of ngrams.

getNrOfDistinctNgrams

public long getNrOfDistinctNgrams(int n)

Description copied from interface: FrequencyCountProvider

Returns the number of distinct ngrams of a given size in the corpus.

Specified by:

getNrOfDistinctNgrams in interface FrequencyCountProvider

Parameters:

n - the ngram size.

Returns:

the number of ngrams.

getNgramIterator

public Iterator<String> getNgramIterator(int n)

Description copied from interface: FrequencyCountProvider

Returns an iterator over all ngrams of a given size in the corpus.

Specified by:

getNgramIterator in interface FrequencyCountProvider

Parameters:

n - the ngram size.

Returns:

an iterator over the ngrams.

getLanguage

public String getLanguage()

Description copied from interface: FrequencyCountProvider

Returns the language for which the frequencies are provided.

Specified by:

getLanguage in interface FrequencyCountProvider

Returns:

the language.

getID

public String getID()

Description copied from interface: FrequencyCountProvider

Returns an ID that identifies the frequency provider

Specified by:

getID in interface FrequencyCountProvider

Returns:

the ID.