org.dkpro.statistics.agreement.coding

Class WeightedKappaAgreement

java.lang.Object

org.dkpro.statistics.agreement.DisagreementMeasure

org.dkpro.statistics.agreement.coding.WeightedAgreement

org.dkpro.statistics.agreement.coding.WeightedKappaAgreement

All Implemented Interfaces:

IAgreementMeasure, IChanceCorrectedDisagreement, IWeightedAgreement

public class WeightedKappaAgreement
extends WeightedAgreement
implements IChanceCorrectedDisagreement

Generalization of Cohen's (1960) kappa-measure for calculating a chance-corrected inter-rater agreement for multiple raters with arbitrary distance/weighting functions. Before an inter-rater agreement can be calculated, an IDistanceFunction instance needs to be assigned.

References:

• Cohen, J.: A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1):37-46, Beverly Hills, CA: Sage Publications, 1960.
• Cohen, J.: Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70(4):213-220, Washington, DC: American Psychological Association, 1968.
• Artstein, R. & Poesio, M.: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4):555-596, Cambridge, MA: The MIT Press, 2008.

Author:

Christian M. Meyer

Field Summary

Fields inherited from class org.dkpro.statistics.agreement.coding.WeightedAgreement

distanceFunction, study

Constructor Summary

Constructors

Constructor and Description
WeightedKappaAgreement(ICodingAnnotationStudy study, IDistanceFunction distanceFunction) Initializes the instance for the given annotation study.

Method Summary

Modifier and Type	Method and Description
double	calculateExpectedDisagreement() Calculates the expected inter-rater agreement using the defined distance function to infer the assumed probability distribution.
double	calculateObservedDisagreement() Calculates the observed inter-rater agreement for the annotation study that was passed to the class constructor and the currently assigned distance function.

Methods inherited from class org.dkpro.statistics.agreement.coding.WeightedAgreement

ensureDistanceFunction, getDistanceFunction, setDistanceFunction

Methods inherited from class org.dkpro.statistics.agreement.DisagreementMeasure

calculateAgreement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dkpro.statistics.agreement.IAgreementMeasure

calculateAgreement

Constructor Detail

WeightedKappaAgreement

public WeightedKappaAgreement(ICodingAnnotationStudy study,
                              IDistanceFunction distanceFunction)

Initializes the instance for the given annotation study. The study should never be null.

Method Detail

calculateObservedDisagreement

public double calculateObservedDisagreement()

Calculates the observed inter-rater agreement for the annotation study that was passed to the class constructor and the currently assigned distance function.

Specified by:

calculateObservedDisagreement in interface IChanceCorrectedDisagreement

Specified by:

calculateObservedDisagreement in class DisagreementMeasure

Throws:

NullPointerException - if the study is null.

ArithmeticException - if the study does not contain any item or the number of raters is smaller than 2.

calculateExpectedDisagreement

public double calculateExpectedDisagreement()

Calculates the expected inter-rater agreement using the defined distance function to infer the assumed probability distribution.

Specified by:

calculateExpectedDisagreement in interface IChanceCorrectedDisagreement

Overrides:

calculateExpectedDisagreement in class DisagreementMeasure

Throws:

NullPointerException - if the annotation study is null.

ArithmeticException - if there are no items or raters in the annotation study.