org.dkpro.statistics.agreement.coding

Class KrippendorffAlphaAgreement

java.lang.Object

org.dkpro.statistics.agreement.DisagreementMeasure

org.dkpro.statistics.agreement.coding.WeightedAgreement

org.dkpro.statistics.agreement.coding.KrippendorffAlphaAgreement

All Implemented Interfaces:

ICodingItemSpecificAgreement, IAgreementMeasure, ICategorySpecificAgreement, IChanceCorrectedDisagreement, IItemSpecificAgreement<ICodingAnnotationItem>, IWeightedAgreement

public class KrippendorffAlphaAgreement
extends WeightedAgreement
implements IChanceCorrectedDisagreement, ICategorySpecificAgreement, ICodingItemSpecificAgreement

Implementation of Krippendorff's (1980) alpha-measure for calculating a chance-corrected inter-rater agreement for multiple raters with arbitrary distance/weighting functions. The basic idea is to divide the estimated variance of within the items by the estimated total variance. Before an inter-rater agreement can be calculated, an IDistanceFunction instance needs to be assigned.

References:

• Krippendorff, K.: Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement 30(1):61-70, 1970.
• Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. 2nd edition, Thousand Oaks, CA: Sage Publications, 2004.
• Artstein, R. & Poesio, M.: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4):555-596, 2008.

Author:

Christian M. Meyer

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<Object,Map<Object,Double>>	coincidenceMatrix

Fields inherited from class org.dkpro.statistics.agreement.coding.WeightedAgreement

distanceFunction, study

Constructor Summary

Constructors

Constructor and Description
KrippendorffAlphaAgreement(ICodingAnnotationStudy study, IDistanceFunction distanceFunction) Initializes the instance for the given annotation study.

Method Summary

Modifier and Type	Method and Description
double	calculateCategoryAgreement(Object category) Calculates the inter-rater agreement for the given category.
double	calculateExpectedDisagreement() Calculates the expected inter-rater agreement using the defined distance function to infer the assumed probability distribution.
double	calculateItemAgreement(ICodingAnnotationItem item) Calculates the inter-rater agreement for the given annotation item.
double	calculateObservedDisagreement() Returns the observed disagreement of an annotation study.

Methods inherited from class org.dkpro.statistics.agreement.coding.WeightedAgreement

ensureDistanceFunction, getDistanceFunction, setDistanceFunction

Methods inherited from class org.dkpro.statistics.agreement.DisagreementMeasure

calculateAgreement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dkpro.statistics.agreement.IAgreementMeasure

calculateAgreement

Field Detail

coincidenceMatrix

protected Map<Object,Map<Object,Double>> coincidenceMatrix

Constructor Detail

KrippendorffAlphaAgreement

public KrippendorffAlphaAgreement(ICodingAnnotationStudy study,
                                  IDistanceFunction distanceFunction)

Initializes the instance for the given annotation study. The study should never be null.

Method Detail

calculateObservedDisagreement

public double calculateObservedDisagreement()

Description copied from interface: IChanceCorrectedDisagreement

Returns the observed disagreement of an annotation study. The observed disagreement is basically the proportion of annotation units that the raters disagree on divided by the number of units in the given study.

Specified by:

calculateObservedDisagreement in interface IChanceCorrectedDisagreement

Specified by:

calculateObservedDisagreement in class DisagreementMeasure

calculateExpectedDisagreement

public double calculateExpectedDisagreement()

Calculates the expected inter-rater agreement using the defined distance function to infer the assumed probability distribution.

Specified by:

calculateExpectedDisagreement in interface IChanceCorrectedDisagreement

Overrides:

calculateExpectedDisagreement in class DisagreementMeasure

Throws:

NullPointerException - if the annotation study is null.

ArithmeticException - if there are no items or raters in the annotation study.

calculateItemAgreement

public double calculateItemAgreement(ICodingAnnotationItem item)

Description copied from interface: IItemSpecificAgreement

Calculates the inter-rater agreement for the given annotation item.

Specified by:

calculateItemAgreement in interface IItemSpecificAgreement<ICodingAnnotationItem>

See Also:

IItemSpecificAgreement

calculateCategoryAgreement

public double calculateCategoryAgreement(Object category)

Description copied from interface: ICategorySpecificAgreement

Calculates the inter-rater agreement for the given category.

Specified by:

calculateCategoryAgreement in interface ICategorySpecificAgreement

See Also:

ICategorySpecificAgreement