No suitable Danish questionnaire exists to evaluate patient satisfaction with various patient reported outcome measures. Thus, the aim of this research project was to conduct a study on the translation and cultural adaption of an American patient reported experience measures questionnaire, “Patient Feedback Form”, among Danish patients, and to examine selected psychometric properties within reliability.
Material and methods
In the first phase of the study, the Patient Feedback Form was forward and backward translated following the methodology of existing guidelines. Subsequently, cognitive interviewing was performed with seven cancer patients and seven healthy persons (19–86 years old/6 men and 8 women) to ensure that questions were easy to understand and made sense to Danish interviewees.
In the second phase, phone interviews were carried out with 95 prostate cancer patients after they had responded to the same Patient Feedback Form. Missing data was imputed using the Expectation-Maximization technique. To examine the structure of the questionnaire, an exploratory factor analysis was conducted. Cronbach’s alpha was calculated to investigate internal consistency.
There were only minor disagreements in the translation process, and the reconciliation went smoothly (phase 1). With regard to one item, however, it was difficult to reach a consensus. Through the qualitative validation process, the right solution was found. The results from the psychometric testing (phase 2) showed that four factors had an Eigen value > 1, but only one factor was extracted as the Scree plot had a clear “elbow”, showing a one factor structure that explained 46.1% of the variance. The internal consistency was high as Cronbach’s alpha was 0.89.
The translated, culturally adapted, and validated version of the Patient Feedback Form seems to be suitable for measuring satisfaction with patient reported outcome measures in a Danish setting. While the results should be treated with caution due to the small sample size, psychometric testing indicates that the questionnaire is a valid instrument. However, additional psychometric testing such as hypotheses testing, responsiveness, and test-retest on a larger and more diverse sample size is required to further verify the validity of the instrument.
Several questionnaires to measure patient reported outcome measures (PROMs) exist and have become an increasingly popular source for collecting information on patient conditions, e.g. physical symptoms, toxicities, or psychosocial problems . Some instruments are generic, dealing with issues such as quality of life(QoL), anxiety, depression, and pain, while others are disease-specific . In the past, PROMS have mainly been used in clinical trials to determine safety, efficacy and cost effectiveness of, for example, a new drug [3, 4]. Thus, the data collected in research settings has generally not been available to clinicians . In many cases, the questionnaires have been independent tools that have helped the health care system gain knowledge of, for example, patients´ symptoms and QoL on a general level. Fortunately, PROMs have also moved into the world of routine care – probably eased by electronic data collection – where they are integrated into the patient trajectory with the purpose of influencing treatment and care. In some circumstances, the results are provided to clinicians to improve patient care and focus on patient concerns [4, 5]. Little is known, however, about the value of this integration from a patient perspective or how patients experience filling out the questionnaires. Thus, it is important to explore if the patients found the questionnaire easy to complete, if it improved patient-clinician communication and/or enhanced quality of care. These are relevant issues to examine at a time when focus on patient reported experiences and attention to patient involvement and satisfaction have increased and are mandatory in many health care settings. More research is needed on the effects of PROM interventions in different settings [6,7,8] and to establish what realistic benefits can be gained from using PROMs in routine care . Using a Patient Reported Experience Measures questionnaire (PREM-questionnaire) to evaluate if a given PROM is worthwhile , and/or to identify which PROM(s) to use [4, 10], may be one method to select feasible and patient-acceptable PROMs.
Since no suitable PREM-questionnaire was available in Danish, an American questionnaire entitled “Patient Feedback Form” was chosen [4, 5, 11]. The Patient Feedback Form was selected because it evaluates the usefulness and value of a given PROM from the patient perspective. Thus, the Patient Feedback Form is relevant in situations where the health care system wishes to examine patient satisfaction with PROMs that are integrated into clinical practice. Furthermore, the Patient Feedback Form is short and, due to its generic nature, we expected it to be adaptable to a Danish setting and useful in many different areas within the health care system. To our knowledge, the form has not been translated into other languages. The questionnaire consists of 13 items (Fig. 1). Respondents evaluate their level of agreement/disagreement on a scale with four options to eliminate the neutral response . Two questions have a 3-point option. The Patient Feedback Form has not undergone any traditional psychometric testing in the original language.
Materials and methods
Phase 1 – The translation and cultural adaption process
Preparation and approvals
An expert group was formed to oversee the translation process. The group consisted of a senior oncologist  and a senior nurse who both had experience with translations and cross-cultural adaptions and the project manager.
Permission to translate the Patient Feedback Form was granted from the developer, Ethan Basch , and Claire Snyder [4, 5], who had adapted the questionnaire. According to Danish law, approval from the ethics committee was not required, but the study was registered with the Danish Data Protection Agency.
Forward translation and reconciliation
The Patient Feedback Form was translated into Danish by two independent, experienced translators, who had Danish as their mother tongue, were fluent in English , and had been residents in an English speaking country for more than two years. They did not have a medical background, which was acceptable because the questionnaire does not contain medical language, health care terminology, or require any particular knowledge. Focus was kept on the natural, spoken language with its cultural nuances addressing a common audience .
Comparisons were made between the independent translations regarding ambiguity and discrepancies of words, sentences, or meaning for each item in the questionnaire in order to create a consensus version.
Backward translation and review
The Danish consensus version was back-translated by two independent bilingual translators blind to the original. The translators had English as their mother tongue but had resided in Denmark for several years. As with the forward translations, the translators were asked to take a conceptual approach due to the subjective nature of the construct (patient experience and satisfaction) .
The two translations were then compared to the original to ensure that the translated versions reflected the same item content.
Cognitive interviewing was performed with 7 cancer patients receiving immunotherapy for malignant melanoma and 7 healthy persons (19–86 years/6 men and 8 women). The respondents were selected to ensure an equal distribution across age and gender. A combination of the “think aloud” method and “probing” was applied  to ensure that the items were easy to understand and made sense to a Danish population. Proofreading was performed and a report sent to the developer and adaptor .
Phase 2 – Psychometric testing
There are no general criteria for calculating sample size when assessing internal consistency and factor analysis. The Cosmin guideline, however, contains standards for evaluating the methodological quality of studies on measurement properties . According to the Cosmin checklist, a sample size of minimum 100 respondents or seven respondents times the number of items is recommended . A convenience sample of 102 men with prostate cancer in post-treatment control (54–73 years old) were chosen as respondents because they all had filled out the same PROM-questionnaire concerning satisfaction with treatment and care, and were available as respondents. In total, 95 (93%) accepted the invitation to respond. Not all of the patients had experienced any problems during their post-treatment control and as a consequence, they had not been in contact with a health care professional. Accordingly, they were not able to answer the items in the Patient Feedback Form which deal with this interaction.
In the original version, the Patient Feedback Form was used in connection with cancer patients [4, 5, 11], which explains why we selected this group of patients for psychometric evaluation. The respondents were interviewed over the phone. Phone interviews were chosen to motivate respondents to answer and to facilitate conducting the survey within a short period of time. An expert on questionnaire technique was consulted to make sure that the questionnaire was adapted to the chosen survey format. Consequently, I, me and my were exchanged with you and yours during the interviews. Moreover, a guideline was designed  to make the interaction as smooth as possible. The interviews were carried out by the same interviewer to ensure uniformity.
The structure (i.e. the number of factors) of the Patient Feedback Form was unknown, and it was not possible to make a confirmatory factor analysis because no psychometric testing of the original version had been carried out. Thus, the psychometric evaluation comprised of an exploratory factor analysis (EFA) if the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was > 0.6 and if the Bartlett’s test of sphericity was significant (p < 0.05) . The number of latent factors were decided by evaluating the scree plot and the number of factors with Eigenvalues > 1. The EFA method and rotation of the factors were chosen depending on the number of factors in the initial EFA. If one factor (as expected) was extracted, the maximum likelihood extraction method without rotation was applied . Further, to assess internal consistency, Cronbach’s Alpha (α) was evaluated. The level of α was considered: fair = 0.70–75; moderate = > 0.75–0.80; good = > 0.80–0.85; excellent > 0.85–0.90 . Missing data was assessed by Little’s Missing Completely at Random (MCAR) test . If participants had > 3 missing items (aside from the five items concerning interaction with healthcare professionals), they were excluded from the analysis. In the case of missing data and a non-significant (p > 0.05) MCAR test, the Expectation-Maximization (EM) technique was used to impute data . A significant level of 0.05 was chosen and all analyses were executed using SPSS version 23.
Phase 1 – The translation and cultural adaption process
Overall, consensus was easy to achieve and neither the translators nor the experts felt that they had to compromise. As for the forward translation, minor discrepancies such as the use of synonyms – digital vs. electronic – and different word order were detected. One of the translators, for example, suggested, “Completing the questionnaire improved discussions with my doctor” whereas the other suggested, “Discussions with my doctor were improved because I had completed the questionnaire.” Also, the back-translated versions were close to the original. In the original version, the word “completed” was used for filling out the questionnaire whereas the two backward translators had chosen “answer” and “respond to”. However, it was not possible to reach a consensus on whether or not the English loanword “feedback” should be translated into Danish. The expert group decided to leave it up to the pilot testing, resulting in the word being translated into a Danish word. Also, the respondents found two items (Fig. 1, items 7 and 10) to be almost identical. However, in order to be true to the original, nothing was changed. With regard to item 11, the semantics was changed somewhat. The phrase “Control of”´ did not sit well with the Danish patients, who did not feel it was in their power to be in control – nor did they want to be. “That is the doctor’s job,” as one respondent put it. Instead of control, the Danish respondents suggested the word “involved”, which they found more appropriate. The Danish version was adapted accordingly. Furthermore, the word doctor was changed to healthcare professional to broaden the scope of the questionnaire. All changes were approved by the developer.
Phase 2 - psychometric testing
Of the 95 respondents, 56 respondents (58.9%) were not able to answer all 13 items since they had not been in contact with a healthcare professional; five of the items (Fig. 1, items 6–10) deal with this interaction. Moreover, two respondents had > 3 items missing (when the items about interaction with a healthcare professional where not included) and, therefore, they were excluded (Table 1). The MCAR test showed that data was missing completely at random (p = 0.307). The missing data was replaced by the EM method. The EFA was conducted as the KMO was 0.731 and Bartlett’s test significant (p < 0.001). Four factors had an Eigen value > 1, but only one factor was extracted as the Scree plot had a clear “elbow”, showing one factor explaining 46.1% of the variance. Three items had a factor load < 0.4, (Table 2). The internal consistency was high as Cronbach’s α was 0.89. The inter-item correlations ranged widely between − 0.001-0.773, with items 2 and 5 showing the lowest correlation and items 10 and 11 the highest (Table 3).
Overall, the translated version was equivalent to the original version with only minor changes. However, one item had to be changed due to cultural differences. The results from the psychometric testing supported a one factor-structure and showed a high internal consistency (0.89) in the final Danish version.
In the forward translation, both translators had chosen not to translate the English word “feedback” in the title. The word is a loanword in Danish and the translators believed that the word was so integrated into the Danish language that everyone would understand the meaning. The respondents disagreed on whether or not it was appropriate in the Danish version since there was a risk that older patients in particular would not understand it. Consequently, we decided to choose the Danish word “tilbagemelding” – the best possible translation of feedback – which was also suggested by some of the respondents. Concerning items 7 and 10, which were found to be similar, it might be argued that future respondents may find it annoying that two items are almost identical. However, there are some nuances. The word “discussions” may, for example, be more of an active exchange of opinions between patient and physician whereas “communication” may also be one-sided with the physician setting the agenda. Moreover, the importance of staying true to the original was prioritized. An inter-item correlation of 0.728 supports the argument that, despite the similarity, the items are not redundant. As for the phrase “control of”, which the Danish respondents disapproved of, we decided that cultural adaption was more important than sticking to the original phrase. Due to cultural differences, it may be more natural for American patients to feel in control of treatment and care , whereas the cognitive interviewing suggests that Danish patients prefer to be actively engaged in the process, which is also supported by the patient organization Danish Patients . Accordingly, the wording was changed. Similar cultural adaptions are found in other questionnaire translations .
Far from all patient satisfaction questionnaires have undergone psychometric testing , which is also the case for the original version of this questionnaire. However, initial psychometric testing of the translated version shows satisfactory results. The EFA reveals a one factor latent structure. As less than half of the variance (46.1%) is explained by one factor, the presence of two factors could be discussed. One factor focused on the feasibility of completing the PROM and the other focused on the clinical utility of the questionnaire in the process of health care. Internal consistency is defined as the degree of relation between items , and the high Cronbach’s α (0.89) supports the results of a one factor structure. However, the possibility of an artificially increased Cronbach’s α is present as the test is sensitive to the small number of items within the scale , as well as the imputation of data. Only a slightly higher Cronbach’s α of 0.90 could be reached if items 2 or 3 were deleted, suggesting a high degree of item-interrelatedness.
It is a limitation that data had to be imputed to complete the dataset. In future research, a study sample where the respondents are able to answer all the items, including the ones dealing with contact between patient and health care professional (items 6–10), should be considered. Also, the generalizability of the results may be reduced by the fact that all the respondents were male, prostate cancer patients and limited to those between the ages of 54–73. Furthermore, it has to be taken into consideration that even though the sample size is accurate to test the EFA, a larger sample size is preferable.
Psychometric testing is often left out when a questionnaire is being used, and the fact that some initial testing has been performed is an obvious strength. Furthermore, the questionnaire may be a valuable tool to assess whether or not a given PROM-questionnaire should be implemented in the clinic or to assist clinicians in choosing which questionnaire to use in a given context. There is a need to “capture patient’s experience of treatment and care as a major indicator of health service quality and treatment effectiveness” . Using the Patient Feedback Form may be a possibility. Moreover, future studies including PROMs can be improved by using the present PREM-instrument, which is now available in Danish, allowing researchers and clinicians to measure patient satisfaction parallel to PROMs  and compare results nationally and internationally.
The translated, culturally adapted, and validated Danish version of the Patient Feedback Form seems to be suitable for measuring satisfaction with PROMs in this prostate cancer population. To further verify the validity of the instrument, the next step should be psychometric testing such as hypotheses testing, responsiveness, and test-retest on a larger and more diverse sample size.
Exploratory factor analysis
Missing Completely at Random
Patient-reported Experience Measures
Patient-reported Outcome Measures
Quality of life
Howell D, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26(9):1846–58.
Shirley E, Josephson G, Sanders J. Fundamentals of patient satisfaction measurement. Physician Leadersh J. 2016;3(1):12–7.
McGrail K, Bryan S, Davis J. Let's all go to the PROM: the case for routine patient-reported outcome measurement in Canadian healthcare. Healthc Pap. 2011;11(4):8–18. discussion 55-8
Snyder CF, et al. When using patient-reported outcomes in clinical practice, the measure matters: a randomized controlled trial. J Oncol Pract. 2014;10(5):e299–306.
Snyder CF, et al. Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice. Psychooncology. 2013;22(4):895–901.
Valderas JM, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17(2):179–93.
Kotronoulas G, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32(14):1480–501.
Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract. 2006;12(5):559–68.
Greenhalgh J, Meadows K, Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Pract. 1999;5(4):401–16.
Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18(1):115–23.
Basch E, et al. Patient online self-reporting of toxicity symptoms during chemotherapy. J Clin Oncol. 2005;23(15):3552–61.
De Vet H, Terwee C, Mokkink L, Knol D. Development of a measurement instrument. In Measurement in Medicine: A Practical Guide (Practical Guides to Biostatistics and Epidemiology, pp 30–64). Cambridge: Cambridge University Press; 2011.
Wild Diana M, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health. 2005;8(2):94–104.
Willis, G.B. Cognitive Interviewing - A `how to´ guide. 1999. National Center for Health Statistics. http://www.ucla.edu/search?qa=cognitive%20interviewing. Accessed 22 June 2016.
Baeksted C, et al. Danish translation and linguistic validation of the U.S. National Cancer Institute's patient-reported outcomes version of the common terminology criteria for adverse events (PRO-CTCAE). J Pain Symptom Manag. 2016;52(2):292–7.
Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract. 2011;17(2):268–74.
Mokkink LB, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.
Dillman DA, Smyth JD, Christian LM. Telephone Questionnaires and Implementation. In: Internet, phone, mail, and mixed-mode surveys. Somerset: John Wiley & Sons, Incorporated; 2014. p. 258–300.
Hadi NU, Abdullah N, Sentosa I. An Easy Approach to Exploratory Factor Analysis: Marketing Perspective. Journal of Educational and Social Research. 2016;6(1).
Ponterotto JG, Ruckdeschel DE. An overview of coefficient alpha and a reliability matrix for estimating adequacy of internal consistency coefficients with psychological research measures. Percept Mot Skills. 2007;105(3 Pt 1):997–1014.
Fox-Wasylyshyn SM, El-Masri MM. Handling missing data in self-report measures. Res Nurs Health. 2005;28(6):488–95.
Edgren E. The ethics of resuscitation; differences between Europe and the USA--Europe should not adopt American guidelines without debate. Resuscitation. 1992;23(2):85–9.
Danish Patients https://www.danskepatienter.dk/about-danish-patients. Accessed 4 Aug 2017.
Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003;80(1):99–103.
Howell D, et al. Core domains for a person-focused outcome measurement system in cancer (PROMS-Cancer Core) for routine care: a scoping review and Canadian Delphi consensus. Value Health. 2013;16(1):76–87.
The authors wish to thank the respondents for their participation in the study and the Department of Urology at Odense University Hospital for providing the patients.
The study was supported by Partnership on PRO (The Danish Cancer Society) and the Region of Southern Denmark. None of the organizations had any role in designing the study, collecting, analyzing and interpreting of data or writing the manuscript.
Availability of data and materials
Permissions from the original developer and adaptor, questionnaires, notes and reports are stored at the Department of Oncology, Odense University Hospital. Please contact corresponding author for more information.
Ethics approval and consent to participate
According to Danish law, approval from the ethical committee was not required, but the study was registered with the Danish Data Protection Agency. All respondents have agreed to participate.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Tolstrup, L.K., Pappot, H., Zangger, G. et al. Danish translation, cultural adaption and initial psychometric evaluation of the patient feedback form. Health Qual Life Outcomes 16, 77 (2018). https://doi.org/10.1186/s12955-018-0900-4
- Psychometric testing
- Patients reported experience measures
- Patients reported outcome measures