November 1998 - Volume 3 - Issue 4 A Publication for Members of Medical Outcomes Trust

Scientific Advisory Committee Overview & Update

At the Trust’s Annual State-of-the-Art Health Outcomes Conference held in Boston on November 2, 1998, Alvin Tarlov, M.D., President of the Medical Outcomes Trust, provided an overview and update of the activities of the Trust’s Scientific Advisory Committee. This article is an edited summary of Dr. Tarlov’s presentation.

Background: The mission of the Medical Outcomes Trust is to promote the universal adoption of health outcome assessment in health care. Integral to the Trust’s work to effect the widespread use of health outcomes assessment is the: 1) acceptance of rigorous standards for outcome measures, 2) development of a library of instruments that meet those standards, and 3) distribution, royalty free, of those instruments with instructions for their use.

Early in 1994, the Board of Trustees (BOT) of the Medical Outcomes Trust (MOT) appointed a seven member Scientific Advisory Committee (SAC) to establish measurement standards for health outcomes measures and to serve a “certifying” function for outcomes measuring instruments submitted to the Trust for consideration. SAC functions independently in an “arms length” relationship to the Board. SAC membership is designed to reflect both the technical psychometric skills required to evaluate survey instruments, as well as experience in the practical use of the instruments in research, clinical practice, quality improvement, or health plan management. Both North American and European members are represented. An Asian member is being sought. Edward Perrin of the University of Washington was the founding Chairman and succeeded as Chairperson in November 1998 by Kathleen Lohr of Research Triangle Institute in North Carolina. Dr. Perrin continues to serve as a SAC member. Other members of SAC are listed on the back page of this publication.

Instrument Review Process: Following inquiry to the Trust, instrument developers receive an application packet containing an application form and descriptions of the detailed information required by SAC for evaluation of the measures psychometric and performance standards. The completed application materials, after being sent to the Trust, are copied and distributed to each SAC member. Two SAC members, a primary reviewer and a secondary reviewer, are appointed to prepare written critiques of a submitted instrument although all SAC members review every submitted instrument. The primary and secondary reviewers’ written reports are distributed to all SAC members. At the next meeting of the SAC, which generally meets quarterly, the committee evaluates the instrument according to the Instrument Review Criteria (see below) and formulates either a request for additional data and information from the measure developer or a recommendation to the BOT to accept the instrument. Generally, there is a unanimous vote of the SAC when acceptance is recommended to the Board.

Once the instrument is accepted, a blue and gold acceptance medallion is appended to the instrument signifying that it has been reviewed and accepted (“certified”) by the Trust. Instrument packets are then printed for distribution to anyone requesting the instrument’s use. The packet includes the instrument, the scoring manual, references and bibliography, and other materials based on the recommendation of the measure’s developer. Currently, the Trust has approved and distributes 25 instruments. The Trust distributes instruments without royalty or profit but at a cost that recovers the expenses of bringing the instrument to the public.

Instrument Review Criteria: SAC established eight criteria to evaluate instruments that are submitted to the Trust for inclusion in the library. The relative importance of criteria for instrument review may differ depending on the intended use(s) and application(s) of the instrument. Instruments may be intended to (1) distinguish between two or more groups, (2) assess change over time and/or (3) predict future status. Thus, evaluation of the properties of an instrument is context specific. An instrument that works well for one purpose or in one setting or population may not do so when applied for another purpose or in another setting or population. Ideally, one would want to have evidence of the measurement properties of an instrument for each of its intended applications. The following eight instrument attributes serve as the principal foci of the review.

  • The Conceptual and Measurement Model
  • Reliability
  • Validity
  • Responsiveness or sensitivity to change
  • Interpretability
  • Burden
  • Alternative Forms of Administration
  • Cultural And Language Adaptations

Printed copies of the full-length Instrument Review Criteria are available from the Trust.

Translation Criteria: As the integration of patient-based outcome measures into all sectors of health care expands, the need increases for instruments capable of capturing such data across cultures within a nation and across nations. In some cases, “translation” involves minor modifications to an original version such as the Canadian or Australian/New Zealand versions of the SF-36 which were developed in English in the United States. In most cases, however, the process is much more comprehensive and involves the complete translation of the instrument from one language to another.

The rise in demand for translated instruments stems in part from the need for aggregated data in clinical trials conducted in several countries to test the efficacy and effectiveness of procedures and medications, and in part from the shift to an increasingly “global” orientation of health care. Two reasons for the increase in the need for global information are: 1) the growing demand for evidence of effectiveness and efficacy as soon as possible before procedures or pharmaceuticals can be approved for provision and/or payments, and 2) the fact that for some conditions, only international studies provide enough cases for users to be confident of the results generated by the studies, e.g., with certain rare disorders. Because different cultures uphold different values and norms, however, oftentimes the mere linguistic translation of an instrument is not sufficient—an instrument must also be appropriately adapted for the culture in question. This process can be lengthy.

In April 1997, SAC developed a subsidiary set of criteria to address the needs for translated instruments. The criteria are related to: 1) the process of translation and adaptation, and 2) the documentation necessary for translated versions submitted to the Trust for approval and distribution. SAC applies these criteria to evaluate instruments submitted to the Trust for approval of the translation only. For SAC to conduct a review of the translation, the original (source) instrument must have been previously approved by the Trust. The Trust expects that this process will further enhance the Trust’s portfolio of instruments and increase the availability and ease of access to translated instruments.

SAC also believes that the Trust should inform members and potential users of the existence of such translations and adaptations even though the measurement properties of the translated instrument (as opposed to the original instrument) might not as yet be fully evaluated. Such a step is predicated on two provisions: that such instruments meet, at minimum, accepted standards of the translation process and that the original instrument had been previously approved by the Trust. The following are the Minimal Translation Criteria developed by SAC and approved by the BOT.

  • The source instrument and its measurement properties have been reviewed by the SAC and approved by the Trust BOT
  • Evidence has been provided of a minimum of two independent forward translations of the instrument
  • Evidence has been provided of a minimum of one back translation of the instrument
  • A detailed description has been provided of the translation process, including a description of the population involved in the evaluation of the appropriateness of the instrument and, whenever possible, a description of the qualifications of the translators used to conduct the translations
  • The final forward and backward translation document and documentation of modifications made have been provided.

Some Advantages of Instrument Approval by the Trust: Several benefits come to measure developers when instruments are accepted into the Trust’s library, i.e.:

  • Awarding peer recognition of the quality of the instrument by virtue of SAC’s approval offers a formal identification of the instrument with published standards developed by an international committee of experts
  • Distribution of the instrument and related materials world-wide
  • Managing routine inquiries about the instrument
  • Tracking requests of the instrument’s use through the Trust’s Project Registry which collects the following data: a contact person’s name, the study objective, the population being studied (e.g., age, gender), the clinical condition of the population, intervention, other instruments used, and whether a publication or report will be generated
  • Publicizing the instrument’s availability
  • Providing ongoing editorial coverage of the instrument.

Possibilities for the Future: We believe that SAC provides a unique service to the outcomes assessment movement by helping to establish high measurement standards and the possibility of international comparison and aggregation of outcomes data. The Trust is only four years old and is a small organization. Yet it adds quality and standardization dimensions to an important field that has entered an accelerated growth phase.

To update its services to the evolving needs of the field, the Trust has under consideration expansion of the instrument categories for attention including:

  • all instruments submitted to the Trust, regardless of whether the developer wants the Trust to distribute the instrument catagories for attention:
  • outcomes instruments to include measures of satisfaction/consumer assess- ment of health plans and physicians
  • screening instruments for depression, alcoholism, child psychiatric disturbances, family violence, health behaviors (tobacco use, exercise, nutrition and so forth)
  • instruments to evaluate family functioning.

The Trust is also considering approaches to evaluate a potentially new generation of outcomes instruments that use computer adaptive techniques and item response theory. Finally, we are grateful for and receptive to all suggestions to improve the Trust’s service to the outcomes and quality fields. Please call (617) 426-4046, fax (617) 426-4131 or e-mail ( your input.