Dataset Information

Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?

ABSTRACT: This study focuses on the problem of assessing inter-observer reliability (IOR) in the case of dichotomous categorical animal-based welfare indicators and the presence of two observers. Based on observations obtained from Animal Welfare Indicators (AWIN) project surveys conducted on nine dairy goat farms, and using udder asymmetry as an indicator, we compared the performance of the most popular agreement indexes available in the literature: Scott's π, Cohen's k, kPABAK, Holsti's H, Krippendorff's α, Hubert's Γ, Janson and Vegelius' J, Bangdiwala's B, Andrés and Marzo's ∆, and Gwet's γ(AC1). Confidence intervals were calculated using closed formulas of variance estimates for π, k, kPABAK, H, α, Γ, J, ∆, and γ(AC1), while the bootstrap and exact bootstrap methods were used for all the indexes. All the indexes and closed formulas of variance estimates were calculated using Microsoft Excel. The bootstrap method was performed with R software, while the exact bootstrap method was performed with SAS software. k, π, and α exhibited a paradoxical behavior, showing unacceptably low values even in the presence of very high concordance rates. B and γ(AC1) showed values very close to the concordance rate, independently of its value. Both bootstrap and exact bootstrap methods turned out to be simpler compared to the implementation of closed variance formulas and provided effective confidence intervals for all the considered indexes. The best approach for measuring IOR in these cases is the use of B or γ(AC1), with bootstrap or exact bootstrap methods for confidence interval calculation.

SUBMITTER: Giammarino M

PROVIDER: S-EPMC8157558 | biostudies-literature | 2021 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?

Giammarino Mauro M Mattiello Silvana S Battini Monica M Quatto Piero P Battaglini Luca Maria LM Vieira Ana C L ACL Stilwell George G Renna Manuela M

Animals : an open access journal from MDPI 20210518 5

This study focuses on the problem of assessing inter-observer reliability (IOR) in the case of dichotomous categorical animal-based welfare indicators and the presence of two observers. Based on observations obtained from Animal Welfare Indicators (AWIN) project surveys conducted on nine dairy goat farms, and using udder asymmetry as an indicator, we compared the performance of the most popular agreement indexes available in the literature: Scott's π, Cohen's k, kPABAK, Holsti's H, Krippendorff' ...[more]

PMID: 34069942

Similar Datasets

Project description:BackgroundSeveral radiographic measurements of the humerus can be used to evaluate the treatment outcome of supracondylar fractures in children. Because of the cartilaginous nature of the immature elbow, interpretation of radiographs around this area is sometimes challenging and can be unreliable. This study was conducted to determine the inter-observer and intra-observer reliability of the six commonly used radiographic measurements of the distal humerus.MethodThe Baumann angle, humero-ulna angle, metaphyseal-diaphyseal angle, radial epiphyseal angle, shaft-condylar angle and lateral capitellohumeral angle of the humerus were measured by two observers on the radiographs of uninjured elbows from 58 children. The values between each measurement were compared and correlated using a Pearson coefficient of correlation to determine the inter-observer and intra-observer reliability.ResultsAll of the radiographic parameters showed excellent intra-observer reliability with the correlation coefficient values of the Baumann angle, humero-ulna angle, metaphyseal-diaphyseal angle, radial epiphyseal angle, shaft-condylar angle, lateral capitellohumeral angle as 0.945, 0.95, 0.909, 0.888, 0.961 and 0.975 (p < 0.001), respectively. The inter-observer reliability of the Baumann and humero-ulna angles were also found to be highly correlated at r = 0.843 (p < 0.001) and 0.878 (p < 0.001), respectively. The metaphyseal-diaphyseal angle had poor reliability with r = 0.136 (p = 0.291) while the radial epiphyseal angle, shaft-condylar angle, and lateral capitellohumeral angle demonstrated good reliability with r = 0.675 (p < 0.001), 0.747 (p < 0.001), and 0.686 (p < 0.001), respectively.ConclusionThe Baumann angle and humero-ulna angle measurements of distal humerus showed excellent inter- and intra-observer reliability. Both parameters represent repeatable and reliable methods for determining the outcome of supracondylar humeral fractures in pediatric population.

Project description:Sheep are managed under a variety of different environments (continually outdoors, partially outdoors with seasonal or diurnal variation, continuously indoors) and for different purposes, which makes assessing welfare challenging. This diversity means that resource-based indicators are not particularly useful and, thus, a welfare assessment scheme for sheep, focusing on animal-based indicators, was developed. We focus specifically on ewes, as the most numerous group of sheep present on farm, although many of the indicators may also have relevance to adult male sheep. Using the Welfare Quality® framework of four Principles and 12 Criteria, we considered the validity, reliability, and feasibility of 46 putative animal-based indicators derived from the literature for these criteria. Where animal-based indicators were potentially unreliably or were not considered feasible, we also considered the resource-based indicators of access to water, stocking density, and floor slipperiness. With the exception of the criteria "Absence of prolonged thirst," we suggest at least one animal-based indicator for each welfare criterion. As a minimum, face validity was available for all indicators; however, for many, we found evidence of convergent validity and discriminant validity (e.g., lameness as measured by gait score, body condition score). The reliability of most of the physical and health measures has been tested in the field and found to be appropriate for use in welfare assessment. However, for the majority of the proposed behavioral indicators (lying synchrony, social withdrawal, postures associated with pain, vocalizations, stereotypy, vigilance, response to surprise, and human approach test), this still needs to be tested. In conclusion, the comprehensive assessment of sheep welfare through largely animal-based measures is supported by the literature through the use of indicators focusing on specific aspects of sheep biology. Further work is required for some indicators to ensure that measures are reliable when used in commercial settings.

Project description:BackgroundMusculoskeletal disorders particularly in the back and lower extremities are common among marines. Here, movement-control tests are considered clinically useful for screening and follow-up evaluation. However, few studies have addressed the reliability of clinical tests, and no such published data exists for marines. The present aim was therefore to determine the inter- and intra-observer reliability of clinically convenient tests emphasizing movement control of the back and hip among marines. A secondary aim was to investigate the sensitivity and specificity of these clinical tests for discriminating musculoskeletal pain disorders in this group of military personnel.MethodsThis inter- and intra-observer reliability study used a test-retest approach with six standardized clinical tests focusing on movement control for back and hip. Thirty-three marines (age 28.7 yrs, SD 5.9) on active duty volunteered and were recruited. They followed an in-vivo observation test procedure that covered both low- and high-load (threshold) tasks relevant for marines on operational duty. Two independent observers simultaneously rated performance as "correct" or "incorrect" following a standardized assessment protocol. Re-testing followed 7-10 days thereafter. Reliability was analysed using kappa (κ) coefficients, while discriminative power of the best-fitting tests for back- and lower-extremity pain was assessed using a multiple-variable regression model.ResultsInter-observer reliability for the six tests was moderate to almost perfect with κ-coefficients ranging between 0.56-0.95. Three tests reached almost perfect inter-observer reliability with mean κ-coefficients > 0.81. However, intra-observer reliability was fair-to-moderate with mean κ-coefficients between 0.22-0.58. Three tests achieved moderate intra-observer reliability with κ-coefficients > 0.41. Combinations of one low- and one high-threshold test best discriminated prior back pain, but results were inconsistent for lower-extremity pain.ConclusionsOur results suggest that clinical tests of movement control of back and hip are reliable for use in screening protocols using several observers with marines. However, test-retest reproducibility was less accurate, which should be considered in follow-up evaluations. The results also indicate that combinations of low- and high-threshold tests have discriminative validity for prior back pain, but were inconclusive for lower-extremity pain.

Project description:Three cattle welfare indicators (lameness, dirtiness, and abomasal disorders) were evaluated in 412 slaughter cattle in a cross-sectional study in Austria. The aims of this study were (1) to evaluate the prevalence of lameness, dirtiness of slaughter cattle, and abomasal disorders; and (2) to determine the association between these welfare indicators and animal-related factors (e.g., housing type, carcass weight, transportation and waiting duration of the animals). The lameness prevalence was 0.73%, the abomasal disorders prevalence was 52.43%, and 88.59% of all cattle were contaminated. The latter result indicates that the cattle were kept in a dirty environment. The occurrence of abomasal disorders was associated with cattle housing systems (p ≤ 0.00) and slaughter weight (p = 0.03). The odds for abomasal disorders were 28.0 times higher for cattle housed on slatted flooring compared to cattle kept in a tethered system. The chance for occurrence of abomasal disorders was 3.6 times higher for cattle with a low carcass weight compared to cattle with a high carcass weight. Furthermore, significant associations were found between dirtiness (also referred to as cleanliness or contamination) and husbandry system, sex, and breed. Cattle housed in deep litter boxes had 40.8 times higher odds of being contaminated compared to cattle in a tethered housing system. Cows (odds: 32.9) and heifers (odds: 4.4) had higher odds of being contaminated with feces compared to bulls, whereby female calves (odds: 0.09) and male calves (odds: 0.02) had significantly lower odds of being contaminated. Furthermore, the breeds Brown Swiss (odds: 0.26) and Holstein-Friesian (odds: 0.14) had a significantly lower chance of being contaminated compared to Simmental cattle. Other collected factors, such as production system, transportation duration, life days of the cattle, average daily weight gain, carcass classification, and fat coverage, showed no association with the collected welfare indicators. The study presented here indicates that welfare indicators evaluated for slaughter cattle are suitable to assess cattle welfare, and improvements in husbandry may positively impact both the abomasal physiology and cleanliness of cattle.

Project description:Cubicle characteristics such as cubicle dimensions or management factors such as cow-to-cubicle ratio could affect health and behaviour of dairy cows. The objective of this study was to estimate effects of cubicle characteristics on animal welfare indicators in dairy cattle. A total of 64 loose housing farms in Germany were assessed once during the winter housing period by one experienced assessor. Nearly 15% of the dairy cows had access to pasture during summer months for <6 h/day, whereas 85% were zero-grazing farms. Selected animal welfare indicators (duration of the lying down process, collisions of cows with cubicles, cows lying outside cubicles, cow cleanliness, integument alterations, lameness and subclinical mastitis incidence) of the Welfare Quality® protocol and cubicle characteristics such as cow-to-cubicle ratio and cubicle dimensions were recorded. Data were statistically analysed using a multiple linear regression approach. Pasture access and cubicle type were considered as potential influencing factors. Wider cubicles positively affected the proportion of dairy cows with dirty flanks (-18.5% per 10 cm increase) but increased the number of cows with severe integument alterations (+8.9% per 10 cm increase). Larger lying areas reduced the percentage of cows with dirty udders (-2.9% per 10 cm2 increase). Longer distances from neck rail to curb were associated with higher prevalence of cows with dirty flanks (+3.1% per 10 cm increase) and subclinical mastitis incidence (+1.2% per 10 cm increase). With increasing neck rail height, the duration of the lying down process (-0.1 s per 10% increase), the percentages of cows with dirty legs (-8.4% per 10 cm increase), dirty udders (-7.0% per 10 cm increase) and severe lameness (-3.0% per 10 cm increase) decreased. Compared with rubber mat-equipped cubicles, deep-bedded cubicles showed a reduction in the lying down duration (-0.8 s), percentages of cows with dirty legs (-34.2%), dirty udders (-30.5%) and lesions and swellings (-13.1%). Compared with farms that did not provide any summer grazing, pasture access was associated with an increase of cows with severe lameness (+5.6%). Contrastingly, the number of cows with subclinical mastitis incidence was lower when cows had access to pasture in summer (-5.4%). Findings of the present study indicate several associations between cubicle characteristics and animal welfare in dairy cattle. Bedding type was found as the most influencing factor in terms of health and behaviour. Results of this study are valuable for farmers to identify the optimal cubicle design and improve the animal welfare level.

Project description:The Animal Welfare Center (AWC) in Macedonia was established in 2009. The objectives of the center are animal welfare (AW) education, research, raising public awareness of AW, and increasing cooperation between the stakeholders. One Health (OH) was not the major focus of the AWC work initially, but, rather, a focus that evolved recently. The objective of this study was to evaluate the AWC from the OH perspective as an example case for positioning the AW within the overall OH concept. Three types of evaluation were performed: (1) assessment of OH-ness, by quantitative measurement of the operational and infrastructural aspects of the AWC; (2) impact evaluation, by conducting quantitative surveys on stakeholders and students; and (3) transdisciplinary evaluation, using semi-quantitative evaluation of the links of cooperation between the AWC and the stakeholders in society by the custom designed CACA (Cooperation, Activities, Communication, and Agreement) scoring system. Results for the OH-ness of the AWC showed relatively high scores for OH thinking, planning and working and middle scores for OH learning and sharing dimensions, i.e., dominance of the operational over infrastructural aspects of the AWC. The impact evaluation of the AWC shows that familiarity with the OH concept among stakeholders was low (44% of the respondents). However, there was a commonality among stakeholder's interest about AW and OH. According to the stakeholders' and students' opinions, the influence of AW on Animal, Environmental, and Human Health is relatively high (in the upper third of the 1-10 scale). The transdisciplinary evaluation of the AWC indicated the presence of transdisciplinarity work by the AWC, with a higher focus on the Universities and Research Institutions and some governmental institutions, and less linked with the Non-Governmental Organizations and Professional Associations (Chambers), e.g., the Veterinary Chamber in Macedonia. The evaluations conducted indicated that the AWC's work is closely dedicated to improving animal, environmental, and human health and has a considerable OH role among the stakeholders in the society. This study describes the significant role and importance that AW has in OH.

Dataset Information

Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?

Publications

Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets