FormalPara Key Points

Multiple rounds of consensus and pilot testing led to key domains and criteria for the development of the QA-SIVAS scale as a new instrument in the quality assessment of video analysis studies.

Testing of QA-SIVAS revealed the scale to be a reliable and valid instrument that can easily be adapted into sports research.

Although designed as an assessment tool, QA-SIVAS can also act as a guide for researchers when designing and conducting studies.

1 Introduction

A healthy lifestyle including physical fitness has many benefits, and participation in sports activities is increasing [9]. However, injuries are frequent among athletes of all performance levels [16, 25], and the number one cause of injury-related emergency department visits in young populations is attributed to sports participation [47]. Compared with overuse injuries [24, 35], traumatic sports injuries are less predictable and the current lack of knowledge in preventing these conditions may substantially affect long-term health [17].

Detailed analyses of how injuries happen contribute to a more comprehensive understanding of the underlying factors [3]. A holistic approach is essential and requires the assessment of both the injury situation and biomechanical patterns [3, 13, 21]. Knowledge of the cause of injury has major implications for injury prevention but is also of clinical relevance for early on-field diagnosis.

Video analysis (VA) has been an increasingly used tool to investigate sports injuries over the last 40 years (Fig. 1). It has gained high popularity in recent years, and there is widespread agreement regarding its scientific and clinical value to the sports medicine community [13, 21]. Among others, reports have been published describing situations, mechanisms and biomechanics of injury to the anterior cruciate ligament [13], Achilles tendon [14, 26], adductor longus muscle [43], and hamstring muscles [21]. However, to date, a lack of standardized methods is seen as a major limitation of VA studies [1]. Quality assessment tools, methodological gold standards, and consensus on terminology may increase consistency and allow a better comparison between studies. Most likely, with growing interest in the methodological framework of VA, systematic reviews will become available. A systematic appraisal of the literature provides the highest level of evidence, but can only be as good as the studies included [23]. Consequently, quality and risk of bias assessments of studies become necessary [23]. However, to date, no instruments exist for assessing study quality in video-based sports injury analysis. Thus, the objective was to develop and evaluate a tool that helps to assess the methodological quality of VA studies for the purpose of enhanced sports injury investigation.

Fig. 1
figure 1

Yearly number of studies for the search term “video analysis AND injury” since 1980 (Medline, accessed via PubMed on 21 November 2022)

Full size image

2 Materials and Methods

The present study took place between January and December 2022 in three phases: (a) development of the QA-SIVAS (Quality Appraisal for Sports Injury Video Analysis Studies) scale; (b) evaluation of its reliability and within-item agreement, and (c) assessment of its construct validity and feasibility. Ethical approval was not required as this study did not involve any patient data. All expert panelists provided written consent to participate.

2.1 Scale Development

The QA-SIVAS scale was developed using a modified Delphi approach including expert consensus and pilot testing [47].

At first, experts in the field of sports science and sports medicine (TG, TH, LR, KH, DF, SW, PM, JW, and TK) independently listed criteria they considered important when assessing sports injuries by using VA. One reviewer (TG) summarized the group’s responses according to frequency and topic. The categorized criteria were then sent back to the experts. After comparing their submissions with those of the other reviewers, each reviewer was allowed to comment on other statements and to revise their own submissions. Subsequently, the criteria were again categorized by the same reviewer (TG). A face-to-face panel discussion with all contributing experts involved was then held, and this led to the development of the first version of QA-SIVAS. Five raters (TG, TH, LR, DF, SW) compared the scale against five randomly selected VA studies out of a pool of 21 included VA studies of anterior cruciate ligament (ACL) injuries from a recent systematic review project (PROSPERO registration: CRD42022337340). The preliminary testing was used to further adjust the rating criteria and instructions.

2.2 Reliability

To estimate inter- and intra-rater reliability, three reviewers (TG, LR, TH) independently assessed the risk of bias of all 21 VA studies from the aforementioned recent systematic review project (PROSPERO registration: CRD42022337340) by applying the QA-SIVAS scale. The rating was repeated after 3 weeks. The reliability of the main outcome, the total QA-SIVAS score [%], was estimated by using the Intraclass Correlation Coefficient (ICC3,1). Results were interpreted as ‘poor’ (ICC < 0.4), ‘fair to good’ (0.4–0.75), and ‘excellent’ (> 0.75). Agreement of each QA-SIVAS item was estimated using free-marginal kappa statistics by Brennan and Prediger [39], which minimizes prevalence-related biases. Interpretation of the results was carried out based on Landis and Koch (1977): k < 0 (‘poor’); k = 0.01–0.20 (‘slight’); k = 0.21–0.40 (‘fair’); k = 0.41–0.60 (‘moderate’); k = 0.61–0.80 (‘substantial’); k = 0.81–1.00 (‘almost perfect’).

2.3 Construct Validity

To evaluate construct validity, an approach similar to Jadad et al. [27] and Wilke et al. [47] was used. All 21 studies were assigned to two field-related experts, an orthopedist and sports physician (DD) and an exercise scientist and physiotherapist (IK). They independently rated each study’s quality with 1–4 points (1: poor; 2: poor to moderate; 3: moderate to good; 4: excellent). The mean ratings were correlated to the scores obtained with the QA-SIVAS scale (TG, LR, TH) using Kendall’s tau analysis.

2.4 Rating Time

Rating time was evaluated as the required mean time for the rating of one study. All three reviewers involved in the reliability study (TG, LR, TH) recorded the time (min:sec per study article) needed to evaluate the study quality for each article. Means were calculated for all three reviewers.

Statistical calculations were performed using SPSS version 27.0 (SPSS, Chicago, IL, USA). p-values < 0.05 were considered significant.

3 Results

The final QA-SIVAS scale consists of an 18-item checklist addressing the study design, data source, conduction, reporting, and discussion of VA studies in sports injury assessment. Each item is to be answered with either 0 (no/not stated) or 1 (yes/present) point. The maximum score is 18. The quality rating is expressed as a percentage value (reached score/maximum score [%]). Detailed information about the components and scoring guidelines is displayed in Table 1.

Table 1 Items and scoring criteria of the QA-SIVAS scale
Full size table

3.1 Reliability

The analysis of raters’ agreement with regard to the total score (%) revealed an ICC of 0.98, interpreted as ‘excellent’ (95% confidence interval [CI] 0.96–0.99; p < 0.001). Repeated scoring showed excellent intra-rater reliability for all three raters: ICC rater 1 = 0.99 (95% CI 0.99–0.99; p < 0.001); ICC rater 2 = 0.99 (95% CI 0.98–0.99; p < 0.001), and ICC rater 3 = 0.99 (95% CI 0.98–0.99; p < 0.001).

Free marginal kappa values of the individual items ranged between 0.68 and 1.00 (substantial to almost perfect) with an exact agreement of 84–100% (Table 2).

Table 2 Reliability and percent agreement of individual QA-SIVAS items
Full size table

3.2 Construct Validity

The analysis of construct validity with two experts in the field revealed a strong association of the results obtained by means of the QA-SIVAS scale and the expert ratings (Kendall’s tau B = 0.71, p < 0.001; see Table 3).

Table 3 Ratings for the validity analysis made by two experts in the field (point scale: 1–4) and the corresponding QA-SIVAS scores (point scale: 0–18) for the included 21 studies
Full size table

3.3 Rating Time

The mean rating time among all three reviewers was 10:07 ± 02:27 min:sec per article.

4 Discussion

A state-of-the-art method was applied to develop the QA-SIVAS scale. The tool covers 18 distinct items. Scores of each item can be used cumulatively for an overall judgment of a study’s quality. Inter- and intra-rater reliability, construct validity and user rating time were excellent. Consequently, the QA-SIVAS scale can be used by researchers for judging the quality of studies using video analysis of injuries in sports.

To our knowledge, QA-SIVAS is the first scale that addresses the quality of studies using video-based assessment of sports injuries. As no assessment instrument has been available hitherto, the scale was developed based on the consensus of an interdisciplinary multi-center reviewer team with field-specific knowledge. The final version was implemented in a systematic review covering 21 studies that have investigated ACL injuries in sports (PROSPERO registration: CRD42022337340).

The intra- and inter-rater reliability were excellent, indicating that the QA-SIVAS scale can be reliably used. This finding is comparable to the risk of bias scales from other medical fields. For instance, the inter-rater reliability of the PEDro scale, a commonly used tool in the assessment of clinical trials, ranges between 0.56 and 0.91 [18, 34]. The excellent inter-rater agreement of the QA-SIVAS scale is particularly notable given the multidisciplinary reviewer team of medical doctors, biomechanists, and sports scientists. Potentially, a background in sports injury research may be necessary, as this was a given prerequisite for all reviewers involved in the development of QA-SIVAS. However, the involvement of coaches and players in the research team has clear benefits, and is recommended [42]. Construct validity of the scale can be interpreted as high based on the correlation analysis revealing a Kendall’s tau B-value of 0.71. All assessments were done in a reasonable amount of time (average duration of around 10 min per article). This is comparable to other quality appraisal instruments and supports its feasibility [37, 47]. The rating time may be even less than 10 min if the raters know the studies beforehand (e.g., from performing a systematic literature search).

Our results from the evaluation process give some guidance on how to interpret QA-SIVAS. Most studies included through the systematic search on ACL injuries scored between 60 and 80%. No study achieved maximal scoring. Only a few studies scored < 60%. One study scored > 80%. However, there was only one item (item 3; Characteristics of the study population are included) that was not met by any of the included studies. Although no study has met the criteria of this item (by not reporting the age of the study population in the majority of studies), item 3 was still considered indispensable due to its clinical implication for injury prevention. It may thus be proposed to use the following interpretation when judging study quality: < 60%: low quality; 60–70%: moderate quality; 71–80%: good quality, 81–100%: high quality. Accordingly, five studies were of low quality, eight studies of moderate quality, seven studies of good quality, and one study of high quality.

For studies already completed, the QA-SIVAS scale may help to highlight methodological concerns. The QA-SIVAS scale can thus be used for assessing multiple studies that are included in a systematic review. High-quality systematic reviews are of great importance [41], and can support clinical decision making in sports [2]. However, shortcomings in the quality of included studies are of major concern. Quality assessment of studies is a challenging but indispensable task in order to draw valid conclusions based on the available evidence on a given topic [48]. QA-SIVAS can be used when conducting systematic reviews, and may also help researchers assess the quality of individual studies.

Although designed as an assessment tool, QA-SIVAS can also be a guidance for researchers. To date, only few recommendations on how to standardize VA have been published. These recommendations were specific to sports or injuries such as concussion, rugby or netball [11, 22, 33]. For example, in future VA research, QA-SIVAS may improve study design when used as a checklist in the planning of new studies [48]. By developing the QA-SIVAS scale, it was the firm intention of our group to extend the principles of evidence-based medicine into the field of video-based sports injury assessment. Future studies in the field should adhere to the QA-SIVAS criteria. In the planning of future studies, items 2 and 11 should be given special attention to allow for a homogenous and standardized methodological approach. When reporting study results, particular attention may be given to items 3, 4, and 8 of the QA-SIVAS scale. These data can easily be included when reporting study results. However, only a minority of the studies assessed fulfilled these items.

4.1 Strength and Limitations

To the best of our knowledge, the QA-SIVAS scale is the first specific tool to assess the quality of studies using VA of injuries in sports. It was developed and refined based on multiple rounds of consensus. However, reliability and validity testing was applied to anterior cruciate ligament ruptures without considering other injury mechanisms. Further, the independence of reviewers is not granted as some of the authors have previously worked together on research projects [8]. For this reason, it was a strategic goal to include multidisciplinary reviewers from multiple centers. Nevertheless, the practical implementation of QA-SIVAS is pending. As with other tools, the significance of QA-SIVAS will depend on feedback from the research community and updates may be required in future years.

5 Conclusion

We identified key domains and criteria for the evaluation and development of studies in the field of video-based assessment of sports injuries. By using multiple rounds of consensus and pilot testing, the QA-SIVAS scale was developed as a new instrument in the quality assessment of studies in this field. We have provided evidence that QA-SIVAS is reliable, valid, and can be easily adapted into sports research. Future studies in the field of video-based analysis of sports injuries should adhere to standardized methodological criteria and strict quality guidelines.