This study focuses on the problem of assessing inter-observer reliability (IOR) in the case of dichotomous categorical animal-based welfare indicators and the presence of two observers. Based on observations obtained from Animal Welfare Indicators (AWIN) project surveys conducted on nine dairy goat farms, and using udder asymmetry as an indicator, we compared the performance of the most popular agreement indexes available in the literature: Scott’s π, Cohen’s k, kPABAK, Holsti’s H, Krippendorff’s α, Hubert’s Γ, Janson and Vegelius’ J, Bangdiwala’s B, Andrés and Marzo’s ∆, and Gwet’s γ(AC1 ). Confidence intervals were calculated using closed formulas of variance estimates for π, k, kPABAK, H, α, Γ, J, ∆, and γ(AC1 ), while the bootstrap and exact bootstrap methods were used for all the indexes. All the indexes and closed formulas of variance estimates were calculated using Microsoft Excel. The bootstrap method was performed with R software, while the exact bootstrap method was performed with SAS software. k, π, and α exhibited a paradoxical behavior, showing unacceptably low values even in the presence of very high concordance rates. B and γ(AC1 ) showed values very close to the concordance rate, independently of its value. Both bootstrap and exact bootstrap methods turned out to be simpler compared to the implementation of closed variance formulas and provided effective confidence intervals for all the considered indexes. The best approach for measuring IOR in these cases is the use of B or γ(AC1 ), with bootstrap or exact bootstrap methods for confidence interval calculation.

Evaluation of inter-observer reliability of animal welfare indicators: Which is the best index to use? / M. Giammarino, S. Mattiello, M. Battini, P. Quatto, L.M. Battaglini, A.C.L. Vieira, G. Stilwell, M. Renna. - In: ANIMALS. - ISSN 2076-2615. - 11:5(2021 May).

Evaluation of inter-observer reliability of animal welfare indicators: Which is the best index to use?

S. Mattiello
Secondo
;
M. Battini;
2021-05

Abstract

This study focuses on the problem of assessing inter-observer reliability (IOR) in the case of dichotomous categorical animal-based welfare indicators and the presence of two observers. Based on observations obtained from Animal Welfare Indicators (AWIN) project surveys conducted on nine dairy goat farms, and using udder asymmetry as an indicator, we compared the performance of the most popular agreement indexes available in the literature: Scott’s π, Cohen’s k, kPABAK, Holsti’s H, Krippendorff’s α, Hubert’s Γ, Janson and Vegelius’ J, Bangdiwala’s B, Andrés and Marzo’s ∆, and Gwet’s γ(AC1 ). Confidence intervals were calculated using closed formulas of variance estimates for π, k, kPABAK, H, α, Γ, J, ∆, and γ(AC1 ), while the bootstrap and exact bootstrap methods were used for all the indexes. All the indexes and closed formulas of variance estimates were calculated using Microsoft Excel. The bootstrap method was performed with R software, while the exact bootstrap method was performed with SAS software. k, π, and α exhibited a paradoxical behavior, showing unacceptably low values even in the presence of very high concordance rates. B and γ(AC1 ) showed values very close to the concordance rate, independently of its value. Both bootstrap and exact bootstrap methods turned out to be simpler compared to the implementation of closed variance formulas and provided effective confidence intervals for all the considered indexes. The best approach for measuring IOR in these cases is the use of B or γ(AC1 ), with bootstrap or exact bootstrap methods for confidence interval calculation.
Agreement index; Animal-based measure; Dichotomous categorical indicator; Inter-rater reliability
Settore AGR/19 - Zootecnica Speciale
Article (author)
File in questo prodotto:
File Dimensione Formato  
2021_Giammarino et al., Animals.pdf

accesso aperto

Tipologia: Publisher's version/PDF
Dimensione 1.54 MB
Formato Adobe PDF
1.54 MB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

Caricamento pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/2434/847031
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 5
  • ???jsp.display-item.citation.isi??? 5
social impact