Assessing test–retest reliability of patient-reported outcome measures using intraclass correlation coefficients: recommendations for selecting and documenting the analytical formula

Table 1 Two-way mixed-effect analysis of variance (ANOVA) model

Case 3 model of McGraw and Wong [6, p34]:	Two-way mixed model with interaction \({{\varvec{x}}_{{\varvec{i}}{\varvec{j}}}}=\varvec{\mu}+{{\varvec{p}}_{\varvec{i}}}+{{\varvec{t}}_{\varvec{j}}}+{\varvec{p}}{{\varvec{t}}_{{\varvec{i}}{\varvec{j}}}}+{{\varvec{e}}_{{\varvec{i}}{\varvec{j}}}}\), where \(\varvec{\mu}\): grand mean \({{\varvec{p}}_{\varvec{i}}}:\) difference due to patient i (i = 1, …, n), \({{\varvec{p}}_{{\varvec{i}}~}}\sim ~{\mathbf{Normal}}(0,~~\varvec{\sigma}_{{\varvec{p}}}^{2})\) \({{\varvec{t}}_{\varvec{j}}}:\) difference due to time point j (j = 1, …, k), \(\mathop \sum \limits_{{{\varvec{j}}=1}}^{{\varvec{k}}} {{\varvec{t}}_{\varvec{j}}}=0\) \({\varvec{p}}{{\varvec{t}}_{{\varvec{i}}{\varvec{j}}}}:\) interaction between patient i and time point j, \(\mathop \sum \limits_{{{\varvec{j}}=1}}^{{\varvec{k}}} {\varvec{p}}{{\varvec{t}}_{{\varvec{i}}{\varvec{j}}}}=0\) and \({\varvec{p}}{{\varvec{t}}_{{\varvec{i}}{\varvec{j}}}}~\sim ~{\mathbf{Normal}}(0,~~\varvec{\sigma}_{{{\varvec{p}}{\varvec{t}}}}^{2})\) \({{\varvec{e}}_{{\varvec{i}}{\varvec{j}}}}\): random error, \({{\varvec{e}}_{{\varvec{i}}{\varvec{j}}}}~\sim ~{\mathbf{Normal}}(0,~~\varvec{\sigma}_{{\varvec{e}}}^{2})\)

Source of variance	df	MS	Expected components in MS
Between patients	n − 1	MS _P	\(k\sigma _{p}^{2}+\sigma _{e}^{2}\)
Within patients
Between time points	k − 1	MS _T	\(n\mathop \sum \limits_{{j=1}}^{k} t_{j}^{2}/(k - 1)+\frac{k}{{k - 1}}\sigma _{{pt}}^{2}+\sigma _{e}^{2}\)
Error (p × t)	(n − 1)(k − 1)	MS _E	\(\frac{k}{{k - 1}}\sigma _{{pt}}^{2}+\sigma _{e}^{2}\)
ICC (A, 1) of McGraw and Wong [6, p 35] =\(\frac{{\varvec{\sigma}_{{\varvec{p}}}^{2} - \varvec{\sigma}_{{{\varvec{p}}{\varvec{t}}}}^{2}/({\varvec{k}} - 1)}}{{\varvec{\sigma}_{{\varvec{p}}}^{2}+\varvec{\sigma}_{{{\varvec{p}}{\varvec{t}}}}^{2}+\varvec{\sigma}_{{\varvec{e}}}^{2}+\mathop \sum \nolimits_{{{\varvec{j}}=1}}^{{\varvec{k}}} {\varvec{t}}_{{\varvec{j}}}^{2}/({\varvec{k}} - 1)}}~=\frac{{{\varvec{M}}{{\varvec{S}}_{\varvec{P}}} - {\varvec{M}}{{\varvec{S}}_{\varvec{E}}}}}{{{\varvec{M}}{{\varvec{S}}_{\varvec{P}}}+({\varvec{k}} - 1){\varvec{M}}{{\varvec{S}}_{\varvec{E}}}+({\varvec{k}}/{\varvec{n}})({\varvec{M}}{{\varvec{S}}_{\varvec{T}}} - {\varvec{M}}{{\varvec{S}}_{\varvec{E}}})}}\)

A absolute agreement, E, e error, k number of time points, MS mean squares, n number of patients in the test–retest evaluation, P, p patients, T, t time points
In a typical test–retest assessment with two time points, k is 2 in the above ANOVA model and ICC (A, 1) formula. SAS Proc GLM and Proc Mixed can be used to generate the components needed to compute the intraclass correlation coefficient (ICC). Programming information is available upon request to the corresponding author, and a publicly available macro for computing ICCs in the notational system of Shrout and Fleiss can be found at the SAS website http://support.sas.com/kb/25/031.html
The confidence interval formula of ICC (A, 1) for case 3 model of McGraw and Wong [6] can be found on page 41 of the original paper