org.dkpro.statistics.agreement.coding

Class FleissKappaAgreement

java.lang.Object

org.dkpro.statistics.agreement.AgreementMeasure

org.dkpro.statistics.agreement.coding.CodingAgreementMeasure

org.dkpro.statistics.agreement.coding.FleissKappaAgreement

All Implemented Interfaces:

ICodingAgreementMeasure, IAgreementMeasure, ICategorySpecificAgreement, IChanceCorrectedAgreement

public class FleissKappaAgreement
extends CodingAgreementMeasure
implements IChanceCorrectedAgreement, ICategorySpecificAgreement

Generalization of Scott's (1955) pi-measure for calculating a chance-corrected inter-rater agreement for multiple raters, which is known as Fleiss' (1971) kappa and Carletta's (1996) K. The basic idea is to average over all pairwise agreements. The measure assumes the same probability distribution for all raters.

References:

• Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76(5):378-381, 1971.
• Carletta, J.: Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics 22(2):249-254, 1996.
• Scott, W.A.: Reliability of content analysis: The case of nominal scale coding. Public Opinion Quaterly 19(3):321-325, 1955.
• Artstein, R. & Poesio, M.: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4):555-596, 2008.

Author:

Christian M. Meyer

Field Summary

Fields inherited from class org.dkpro.statistics.agreement.coding.CodingAgreementMeasure

study

Constructor Summary

Constructors

Constructor and Description
FleissKappaAgreement(ICodingAnnotationStudy study) Initializes the instance for the given annotation study.

Method Summary

Modifier and Type	Method and Description
double	calculateCategoryAgreement(Object category) Calculates the inter-rater agreement for the given category.
double	calculateExpectedAgreement() Calculates the expected inter-rater agreement that assumes the same distribution for all raters and annotations.

Methods inherited from class org.dkpro.statistics.agreement.coding.CodingAgreementMeasure

calculateObservedAgreement, doCalculateItemAgreement, ensureTwoRaters, warnIfMissingValues

Methods inherited from class org.dkpro.statistics.agreement.AgreementMeasure

calculateAgreement

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.dkpro.statistics.agreement.IChanceCorrectedAgreement

calculateObservedAgreement

Methods inherited from interface org.dkpro.statistics.agreement.IAgreementMeasure

calculateAgreement

Constructor Detail

FleissKappaAgreement

public FleissKappaAgreement(ICodingAnnotationStudy study)

Initializes the instance for the given annotation study. The study may never be null.

Method Detail

calculateExpectedAgreement

public double calculateExpectedAgreement()

Calculates the expected inter-rater agreement that assumes the same distribution for all raters and annotations.

Specified by:

calculateExpectedAgreement in interface IChanceCorrectedAgreement

Overrides:

calculateExpectedAgreement in class AgreementMeasure

Throws:

NullPointerException - if the annotation study is null.

ArithmeticException - if there are no items in the annotation study.

calculateCategoryAgreement

public double calculateCategoryAgreement(Object category)

Description copied from interface: ICategorySpecificAgreement

Calculates the inter-rater agreement for the given category.

Specified by:

calculateCategoryAgreement in interface ICategorySpecificAgreement

See Also:

ICategorySpecificAgreement