The PRO Consortium followed a structured, iterative feedback process to reach consensus on its LV methods. The process was similar to a modified Delphi approach [10, 11] in that information was gathered from individuals representing companies experienced in LV as well as researchers with this expertise, and their information was consolidated and returned to the group members for further comment and feedback in two iterative rounds. The process differed in that each individual’s feedback from each round was not shared with group members in successive rounds, but rather adjudicated by the PRO Consortium team (consisting of the Executive Director, two Project Managers, an LV expert, and an external measure development consultant) to prepare revised documents for the following round, and there was no final vote taken to reach consensus. Group members were asked whether they had strong objections to accepting the proposed process in the final feedback round. The LV process was finalized at the point at which no further objections from either the translation company representatives or the pharmaceutical industry representatives were raised.
In 2015, the PRO Consortium initiated the development of an instrument translation process by requesting information from 13 translation companies, members of the Process Subcommittee, and other LV experts to determine whether the ISPOR recommendations were being followed and to identify additional best practices within the translation industry that exceeded the ISPOR recommendations. This information-gathering process involved reviewing documentation (e.g., guidelines, websites) from the 13 translation companies and conducting follow-up discussions via telephone or email to clarify the exact LV methods used by each company. A matrix document was created which listed the steps from the ISPOR recommendations, and then each company’s version of the steps to facilitate comparison of the execution of each step, which resulted in a total of 17 steps. Individual translation company procedures were compared with each other and against the ISPOR recommendations to identify the most rigorous approach for each suggested step in the process based on PRO Consortium team determination. A document was created that listed 17 steps in three columns: the ISPOR steps in column 1, the most rigorous steps identified in our review in column 2, and then the proposed PRO Consortium version of each step in column 3. Rather than looking for the most commonly performed way of executing each step, this approach was intended to identify the best practice approach for each step. The document was then reviewed by an LV expert to ensure that regulatory needs would be met by the proposed PRO Consortium steps and to identify steps that were either company-specific and not required as part of best practice or not necessary for regulatory purposes. This review led to the removal of four steps which were either deemed too burdensome and not required as part of best practice (i.e., risk mitigation assessment and plan, risk audit and summary report) or not the responsibility of the translation company (i.e., establishing a document repository, translatability assessment), and consolidation of two steps into one due to redundancy (forward translation reconciliation and establish reconciled forward translation).
In early 2016, the PRO Consortium used the 12 steps identified during the LV best practices information-gathering stage to draft a proposed Instrument Translation Process in the form of a numbered table that described each recommended step in moderate detail. The proposed process was distributed to the original 13 translation companies along with three others identified in 2016 which perform LV of PRO measures and were known to PRO Consortium member firms. In an effort to reach consensus, the translation companies were asked to review the proposed process document and provide constructive feedback including, if applicable, an indication of agreement with the process described. Recommendations on ways to improve the delineation of the process or identification of areas of concern were also requested. This round of review took place between March and May 2016, with written responses received from 13 of the 16 (81%) companies invited to participate. Concerns were raised regarding most steps and/or descriptions in the process. These concerns were reviewed by the LV expert and then adjudicated by the Process Subcommittee.
Based on feedback received from translation companies during the initial round of review, a number of significant changes were made to the process, additional process documents were prepared in response to needs identified in the feedback round, and these documents were then circulated for review and comment by translation company representatives and Process Subcommittee representatives in July 2016.
A consensus development teleconference was held in August 2016, attended by nine translation company representatives and four Process Subcommittee members, during which further feedback on the revised process documents was discussed by attendees. Additional revisions were made following the teleconference to address the remaining issues and final versions of the process documents were reviewed by the Process Subcommittee and formally approved by the PRO Consortium’s Coordinating Committee. The documents are now available for use by PRO Consortium member firms as well as the broader scientific community.
Changes to the PRO consortium instrument translation process based on input from multiple stakeholders
A number of changes were made to the PRO Consortium’s instrument translation process based on the feedback received from translation companies and member firms. As shown in Fig. 1, the single process document originally distributed was inadequate to delineate the process envisioned by the PRO Consortium, and led to the creation of several separate documents to explain the PRO Consortium Instrument Translation Process: an overview, glossary, flowcharts, and separate process documents to address universal and country-specific approaches (See Table 1. Definitions of Key Terms for definitions). The country-specific process was separated into two documents, one for new translations and another for adaptations of an existing translation.
Other significant changes to the PRO Consortium Translation Process included the following:
Universal vs country-specific approaches
Although most translation companies perform both types of translations, six supported the PRO Consortium’s preference for universal translations, while seven raised questions or expressed concerns with the approach. Change: The ‘Preparation Step’ was revised to state that C-Path, in conjunction with the clinical trial sponsor, will determine if the translations to be performed will follow the universal or country-specific approach.
Use of certified translators
Five translation companies disagreed with this requirement, noting that many of the best translators are not certified and that certification is not available in some countries. Change: Reference to ‘certified’ was replaced with ‘qualified’ to emphasize the need for translators with appropriate qualifications to perform the translations as opposed to having people without proper training translate PRO measures.
Forward translations are handled differently by many translation companies and a minimum number had not been defined. Change: The ‘Forward Translation Step’ was revised to reflect a minimum requirement of two forward translations.
Forward translation reconciliation
The reconciliation process is handled differently by translation companies and the approach initially proposed (independent third party reconciles the two forward translations) was considered too complex by many of them. This finding is consistent with Koller and colleagues , who provide recommendations for decision-making during reconciliation but did not find consensus on the people involved in the process. Change: In the ‘Forward Translation Reconciliation Step,’ multiple options are provided as examples for accomplishing this process, noting that the decision-making must be well-documented.
Several translation companies expressed concerns and some confusion regarding the involvement of in-country consultants while others strongly supported their involvement. Change: Clarified roles on process documents and in Glossary.
There was general confusion regarding what this involved. Change: This step was relabeled the ‘International Harmonization Step’ and requested only a description of the planned methodology and rationale.
Several translation companies questioned whether in-country affiliates should be involved in conjunction with the proofreading step prior to cognitive interviewing, while others felt that their involvement early in the process was very important. Change: This step was made mandatory to provide affiliate representatives with a recognized role in the process and to prevent unsolicited feedback after translations were finalized that could threaten their validity, and a recommendation was added for the clinical trial sponsor to identify affiliates early (during ‘Preparation’), and that, if none are available, alternatives would be provided by the translation company.
Many translation companies questioned how electronic PRO (ePRO) implementation should be integrated into the process. Change: Addressed in ‘Final Review and Documentation (Proofreading) Step’ as recommended best practice.
This was suggested by one translation company and one member firm as a new step. Decision: The sponsor can include this step if desired but it is not required in the PRO Consortium translation process.
Detailed descriptions of the Steps mentioned above are available in Table 3.
Table 2 presents the current set of process documents that describe the PRO Consortium translation process. The Overview was developed to provide both clinical trial sponsors and translation companies with the appropriate context in which to perform LV and awareness of the overall purpose and objectives. We also thought it was necessary to review and clarify terms that are currently being used for LV and created an updated Glossary to be used for consistency across the translation industry.
Given the feedback from translation companies and PRO Consortium member firms, it was decided that the PRO Consortium process needed to allow flexibility regarding whether a given measure or language would follow the universal or country-specific approach [4, 13] and, as a result, detailed descriptions of each process were developed. The Process Subcommittee along with member firms and translation companies worked together to discuss the pros and cons of universal and country-specific translations. The PRO Consortium generally prefers universal language translations for its measures to reduce logistical complexity of managing country-specific versions and minimize variability among translations of the same language but also recognizes that there are some situations where the universal approach is not optimal. For example, health resource utilization varies by country, so translations of these questionnaires must be tailored to the country in question and cannot be translated universally. Flexibility is recommended to accommodate situations where wording of the measure (requiring different terms in different countries) or the language in question (which may vary too much between countries) does not allow universal translations to be utilized. Flowcharts for each of the three processes were developed to provide a visual representation of the sequence of the steps.
Finally, the process steps emphasize the importance of documenting the rationale for decision-making throughout the LV process, which would be compiled in a report on the LV process and results by the translation company. The PRO Consortium created a report template to facilitate standardized reporting across translation companies.
Translation process steps
This section outlines the steps in the PRO Consortium translation process and describes aspects that differentiate this process from the ISPOR recommendations. The process outlined here is intended for use with PRO measures, which make up the majority of measures in development by the PRO Consortium. In cases where a different COA (e.g., clinician-reported outcome, observer-reported outcome, or performance outcome) measure might be developed, the translation process will be reviewed and refined to address the new assessment type.
Table 3 presents the 12 steps in the translation process and provides a brief description of how each is implemented for both the universal and the country-specific approaches. The country-specific adaptation process can occur either within the same study if relevant countries are included initially or subsequent to completion of the initial translation, if an existing translation needs to be adapted for a new country identified later.
It is important to note that semantic and conceptual equivalence  are goals of the PRO Consortium translation process. The evaluation of conceptual equivalence begins during the measure development process with the mandatory translatability assessment conducted to ensure that the concept of interest and the wording of the items are suitable in cultures outside of North America where the initial measure development was conducted. An Item Definition Table describing the concepts being measured along with translation alternatives is also developed as part of the measure development process, as recommended by Herdman and colleagues  for achieving semantic equivalence. Both processes are considered outside the scope of the translation process because they occur during measure development but provide the necessary foundation to improve the semantic and conceptual equivalence of the translations. Both types of equivalence are further evaluated during the translation process as described in Table 3 (see Steps 5 and 6).
The PRO Consortium translation process expands upon the ISPOR recommendations  in several ways. One major difference is that the PRO Consortium process takes into account both the universal and country-specific approach and delineates each approach including two variations of the country-specific approach in a very detailed, step-by-step fashion. A later ISPOR Task Force report  explored the issue of same language in different countries and discussed the advantages and disadvantages of universal and country-specific approaches at a high level, but it did not provide sufficient detail to be operationalized by the PRO Consortium.
A second way that the PRO Consortium process differs from the ISPOR recommendations is the incorporation of a required affiliate review. Member firms strongly endorsed this step as an important way for their local affiliates to provide input into the translations during the process rather than after the translation has been finalized when changes could contradict the evidence generated to support the translation throughout the LV process. In response to feedback from translation companies that this step could cause delays or result in unnecessary changes, it was refined to include an explanation with specific directions about the nature of the review that would be provided to affiliates so that only critical issues would be raised. In addition, for those firms without affiliates willing to participate in the process, a provision for the translation company to provide a suitable alternative to fill this role was included. For this reason, the process mentions identification of affiliate reviewers during the preparation step very early on to ensure that a determination can be made if alternatives need to be engaged and that all reviewers are aware of the timelines and can respond in a timely way to the request for review.
A third differentiator in the PRO Consortium process is the inclusion of considerations for electronic implementation of the measures and the awareness of its potential impact on the translation process. First, there is a potential need for supplementary text to be translated to accompany the measure when implemented electronically, including skip alerts or other error messages that are not necessary on paper. It is best to identify such text early in the process so that a decision can be made whether to include it in the above-described process or to have it translated in a separate effort along with other navigational text, which does not require the rigorous translation and cognitive interviewing process described here. Second, a critical part of the electronic implementation of the measures is the proofreading of final screenshots to ensure accurate transfer of the translated content to the electronic platform. Ideally, this screenshot proofreading process would be conducted by the same translation company responsible for the translation of the content to prevent unauthorized changes, and therefore this work needs to be included in the translation company’s scope of work. It is also critical for both the translation company and the electronic COA provider to coordinate the screenshot proofreading process to meet study start-up timelines.
Another unique aspect of the PRO Consortium translation process is the decision to encourage use of the same translation company selected for initial measure translations to produce all subsequent translations. The rationale for this recommendation is that using the same company for all translations of a specific measure will ensure that a consistent and appropriate methodology is used across all languages, and that the firm would maintain institutional memory regarding the measure’s nuances and previous decisions leading to increased and improved harmonization of translations developed over time. The goal is to enhance translation company commitment to the quality and accuracy of translations of the measure because of its ongoing involvement in the LV process. This approach will also likely expedite the process for subsequent translations.
There are some limitations to the methods used to develop consensus for the PRO Consortium translation process. The stakeholders involved in the process included the constituents of the PRO Consortium but did not include input from researchers who conduct studies in multiple languages outside of a pharmaceutical sponsor context. Input from such researchers could have provided a useful viewpoint on the proposed process and considerations for improvement. Another limitation is that the PRO Consortium did not conduct any comparison studies to test different methods to provide empirical support for the decisions made about best practices. Studies comparing different translation methodologies have been conducted [15,16,17] and found that the different translations had similar measurement properties and resulted in reliable measures. To date, better methods to conduct LV comparison or feasibility studies still need to be developed. The PRO Consortium recommends that sponsors review the results of translations used in their clinical trials by language to further evaluate the comparability of measurement properties as an important next step in the evaluation of the translation process. As the translation of several PRO Consortium measures is currently underway, further insights into the feasibility of the process will be evaluated and the process revised if needed.