A quarterly publication for members of the Medical Outcomes Trust
July 1997

In This Issue . . .

Evaluating Outcomes of Cardiac Surgery

Summaries of efforts to evaluate outcomes of cardiac surgery

Measures for Evaluating Outcomes of Cardiac Surgery

Research Summary: 1995-1996

Developing Risk-Adjustment Models
The Role of Social Survey and Clinical Variables

A New Approach to Outcomes Management for Patients with Diabetes

Health of Seniors Outcome Measure

The Health Of The Public:
Moving Toward Consumer Driven Health Improvement

The Trust is grateful to
Richard Goss, MD, MPH
John Spertus, MD, MPH
William Nugent, MD
for their input into this issue.

Evaluating Outcomes of Cardiac Surgery

At the Trust's third State-of-the-Art Health Outcomes Conference in May, Dr. Al Tarlov, President of the Trust, spoke of the status of the health outcomes field in terms of its acceptance on a conceptual level versus actual implementation. He described a graphical representation of the "Diffusion of Innovation," a subfield of communication science advanced substantially by the work of Everett Rogers and detailed in his 1995 fifth edition Diffusion of Innovation. In this case, the innovation is the application of health outcomes measurement. Typically, once the adoption of an innovation surpasses 50 percent, its further adoption as a concept is irreversible.

Dr. Tarlov suggested two points to illustrate where the field is today. One point represents the adoption of the concept of using health outcomes and the other point represents actual implementation of the innovation.

Dr. Tarlov further portrayed a field in which a substantial majority of people and organizations in the field, perhaps as many 80 percent or more, have adopted the concept. In contrast, however, Dr. Tarlov suggested that implementation of outcomes measurement lags far behind its adoption as a concept. He believes that organizations actually collecting standardized patient-based health and well-being information, and measuring outcomes over time, are a rare find to date, and perhaps represent as few as one to three percent. In other examples of social, political, agricultural, recreational, informational or economic innovations, the adoption of an innovation becomes irreversible once implementation approximates 16 percent. In short, we are at a very formative stage in realizing the innovation of health outcomes measurement: an interesting, challenging, and pivotal point where a majority of people recognize the value of health outcomes, but few actually use them.

Dr. Tarlov's status report of the field was consistent with the Trust's experience of developing this issue of the Monitor. This Monitor highlights state-of-the-art work related to the use of health outcomes for tracking patients who have undergone cardiac surgery, primarily, though not exclusively, cardiac artery bypass graft (CABG) surgery (the topic of cardiac surgery was identified as a priority by Trust members in the Trust's Member Survey conducted in 1996). This is the first issue of the Monitor to feature a surgical intervention or procedure as opposed to a chronic clinical condition.

As we researched the topic of patient-based outcomes in cardiac surgery we found that the project directors or investigators with whom we spoke thought it was premature to feature their projects in the Monitor. Thus, in this issue we highlight a wide range of projects in the Initiatives section. The aim of this issue of the Monitor remains the same: to present the cutting edge outcomes measurement applications in a particular clinical area.

Cardiac Surgery Outcomes: The Evolution of a Field

Coronary heart disease (CHD) is the leading cause of death in the United States. It afflicts over six million individuals in this country and is responsible for causing over one million heart attacks annually. The costs of treating CHD in terms of medical care, lost earnings, and lost productivity are well over 50 billion dollars a year.

In the past three decades, great strides have been made to improve the efficacy and effectiveness of therapies, interventions and procedures, and have yielded substantial reductions in CHD-related mortality. Throughout the early to mid-1970's there were several large clinical trials designed to examine the efficacy of medical treatment and technology that demonstrated the benefits of CABG over medical therapies for certain conditions.

In the 1980's, the focus of research in CHD broadened to include effectiveness as well as efficacy. Adding effectiveness research to the national agenda was a pragmatic approach based on the conduct of studies in the context of existing systems that organize, deliver, finance and regulate care. This research included a variety of initiatives using large databases such as the Health Care Financing Administration's (HCFA) publication of annual mortality rates by hospital between 1987 and 1993, and the New York State and Pennsylvania efforts to identify and compare heart surgery programs and specific clinicians on the basis of risk-adjusted post-operative mortality rates. While these projects were significant advances, some of their findings are controversial given the limitations of methods to risk-adjust the data and populations used as the basis of the estimates.

The common threads that tie the work of the past three decades together are the primarily dependent variables which were studied, IE, mortality, and a variety of cost and utilization measures. These variables relate to the severity of patients' illness, what happens in the operating room or soon thereafter, and are readily accessible from existing data sets. Mortality is the most common clinical dependent variable studied in cardiac surgery research for two reasons: first, it is the primary concern of CABG patients and their families, e.g.: How many people survive the surgery? Do more people survive the surgery with one doctor or at one hospital versus another? Secondly, mortality remains the only clinical variable for which we have standardized, reliable, and valid data on a widespread and consistent basis.

A Case Summary of Mortality as Outcome

One ongoing, landmark initiative, the Northern New England Cardiovascular Disease Study Group (NNECVDSG) has produced a great deal of information about processes of care that reduce the mortality associated with CABG. The NNECVDSG is a regional collaboration which formed in 1987 in response to the increasing demand for greater accountability for health care quality by third-party payers, internal and external regulatory agencies, and consumers. The NNECVDSG has demonstrated significant differences in both mortality and utilization across multiple hospital sites across northern New England. Surgeons practicing within the Group are informed of the outcomes of their surgical interventions, how the results change over time, and how they compare with other physicians in the study group.

Based on a sample of 6,488 consecutive patients who had CABG surgery, hospital mortality was reduced by 24%. The NNECVDSG used a threefold approach to achieve this result: 1) distributing individual and aggregate, risk-adjusted mortality data to clinicians three times a year; 2) conducting training sessions in continuous quality improvement techniques annually, attended by clinicians, scientists, and hospital administrators; and 3) conducting site visits by teams consisting of an industrial engineer and representatives of the medical, nursing, and perfusion staff.

Next Steps

Great progress has been made to improve the outcomes of care for people with CHD: for people undergoing CABG surgery, the vast majority survive the procedure. However, once patients survive surgery they ask: What will my life be like now? Will I be able to work again? Will I be able to walk a mile when I was previously only able to walk several blocks? When comparing surgeons and hospitals, is there a difference in the rate of people who get better, stay the same, or get worse after CABG surgery?

There are an array of methodolgical and logistic challenges that must be addressed as the field unfolds, including: 1) identifying and implementing standardized surveys, and/or learning how to use them in clinical practice; 2) addressing practical and logistical challenges related to administering surveys at appropriate points of care, e.g., before the surgery, at discharge, and/or at strategic points post-operatively; 3) a need to consolidate multiple efforts to gather information from patients, and thus, reduce the burden on patients to fill out forms or answer questions; 4) overcoming methodological challenges to improve quality data and our ability to interpret information in a useful and meaningful way; and 5) bringing information systems on-line and making them readily accessible to clinicians so they can access patient-based information as needed.

Back to Top


Changes in nutritional status, age, and post-operative health outcomes in older persons undergoing elective coronary artery bypass surgery

The objective of this prospective longitudinal study is to describe the extent to which post-operative (post-op) health outcomes vary as a function of change in nutritional status (NS) and age in older persons undergoing elective coronary artery bypass grafting (surgery). Recent studies report the incidence of poor NS in the hospitalized elderly at 22-43%, and it is imperative to examine the effects of poor NS on health outcomes in elderly surgical patients.

Data collection for this project started in January of 1997, and a total of 100 subjects will be recruited from the practice of eight surgeons who perform 70% of cardiac surgeries at one institution. Participants in this study are community dwelling persons, 65 and older, without dementia or a history of cancer. Data are collected at three time points: pre-admission (T0), post-op (T1), and post-discharge (T2). At T0 the SF-36 will be used to measure health status, and demographic data will be collected, in addition to information regarding mental status and nutritional health. The SF-36 will be administered again at T2.

Multiple regression analysis will be used to examine the extent to which post-op health outcomes vary as a function of change in NS and age.

Contact: Rose Ann DiMaria, RN, MSN, Principal Investigator. 1800 Roundhill Rd., #1006, Charleston, WV 25314. E-Mail: RDGKG@aol.com.

The coronary stent outcomes study

This study is a collaborative, multidimensional data project to explore outcomes and processes related to placement of coronary stents, as the efficacy and appropriateness of stents have not been carefully studied. Clinical, patient satisfaction, and functional status data are being collected and analyzed on 800 stent cases.

In order to ascertain the efficacy of stent surgery, an 11-member Select Network Cardiology Steering Committee designed a pilot study to examine the use of coronary stents relative to (1) process of care issues; (2) patient characteristics affecting outcomes; (3) provider characteristics; and (4) incidence of complications and reoperations. The Study population comprises 800 patients in whom coronary stents were placed during November 1995- August 1996. To ascertain satisfaction and

functional status outcomes, patients were surveyed by mail. The combination of patient-level clinical and survey data will allow multidimensional analyses of coronary stent outcomes. Study results will be available by the end of 1997.

Contact Information: Nancy Hardie, MPH, MS, Project Specialist, Health Care Outcomes and Evaluation, Blue Cross, Blue Shield and Blue Plus of Minnesota, 3535 Blue Cross Road, P.O. Box 64179, Route R3-11, St. Paul, MN 55164-0179.

Grant/Riverside Methodist Hospitals, OhioHealth: Functional health outcomes assessment following cardiac surgery

In 1996, over 1600 patients underwent cardiac surgery at the Riverside campus of OhioHealth. A multidisciplinary cardiac surgery process improvement team has been in place since 1994 to address numerous patient population issues, including: 1) improvement of quality of care; 2) responsible resource utilization; and 3) monitoring outcomes. For nearly two years, RNs from the intermediate step-down unit have followed the patients of two cardiothoracic surgeons via phone calls at seven and 30 days post-discharge for assessments of activity level, pain or discomfort, non-routine visits to a physician or ER, and readmissions. The questionnaires were developed by the bedside care givers with the assistance of the multidisciplinary team. The data are analyzed by an epidemiologist familiar with the cardiac surgery process, and then taken back to the process improvement team for review.

In November of 1996, a functional health outcome pilot was initiated which incorporated the SF-12 into the existing process for following cardiac surgery patients post-discharge. A total of 136 patients were enrolled through February of 1997 and will be followed until the spring of 1998. The Surgical Clinical Case Manager supervised the baseline SF-12 assessment and follow-up phone calls by the RNs. An SF-12 assessment was added to the 30-day phone call. In addition, the same information is collected at six and twelve months through mailings. Initial evaluation of the data included comparisons of baseline and 30-day post-surgery SF-12 summary measures. Data from the existing cardiac surgery database will be utilized for comprehensive analyses of this patient population.

A proposal to expand functional health outcomes measurement to all cardiac surgery, as well as angioplasty patients, over the next year is under review.

Contact: Holly Holl, MS, Epidemiologist, Quality and Resource Management Department, or Patti J. Crego, MS, RN, CCRN, Surgical Clinical Case Manager- Heart Services, Grant/Riverside Methodist Hospitals, 3535 Olentangy River Road, Columbus, OH 43214.

Late outcomes in elderly patients with open heart surgery

This study examined the long-term outcomes, including survival and quality of life in patients over 70 years of age who have undergone open heart surgery, including coronary artery bypass graft (CABG) and CABG/valve replacement.

A cohort of 401 patients who underwent surgery in 1987 or 1988 was retrospectively identified using a clinical database of a cardiac surgical group. Quality of life was measured with the SF-36, which was administered over the telephone, at five to eight year follow-up (the follow-up point was dependent on the year the patient had surgery, and the point in time the patient was located for follow-up). Out of the cohort of 401, 98% were accounted for, and a total of 176 patients were able to complete the SF-36.

The mean survival time for patients with no co-morbidities was 6.0 years, while those with one co-morbidity had a survival time of 5.3 years. Gender, type of surgery, or emergent status did not affect long-term survival, and as for quality of life, almost all mean SF-36 scores for the CABG group at follow-up were equal to or higher than

US means for individuals 75 years of age or older (the one exception was the mental health score, which was lower than the US norm). Co-morbidities which significantly affected quality of life included chronic obstructive pulmonary disease (COPD) and diabetes. Furthermore, quality of life decreased as the number of co-morbidities at time of surgery increased.

Contact: Sandra Magnetti, DrPH, West Virginia University School of Medicine. Department of Community Medicine, IOEH PO Box 9190, Morgantown, WV 26506-9190.

Measuring quality of life outcomes in patients treated for coronary artery disease in Alberta: The APPROACH Project

The purpose of Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) is to compare and contrast the outcomes of treatment of a large prospective cohort of Coronary Artery Disease (CAD) patients (symptomatic) treated with medical therapy, percutaneous transluminal coronary angioplasty (PTCA), and coronary-artery bypass grafting (CABG).

Patients who enroll in this eight-year, observational study are followed up yearly, on the anniversary of their initial catheritization. Quality of life data are collected one year following entry into the database with the Seattle Angina Questionnaire (SAQ).

This study began in January of 1995, and in its initial year, 6,065 patients met the eligibility requirements. It is anticipated that this project will recruit a total of 18,000 patients and continue until the year 2002. Findings thus far indicate that SAQ scores demonstrate differences which can be attributed to treatment group, as well as gender, and in some instances, age.

Contact: Colleen Norris, MN, Project Manager, Approach Project. 1070 EDC, 8440-112 Street, Edmonton, Canada, T6G 2B7.

Older adults' recovery outcomes following coronary artery bypass surgery and the effect of formal cardiac rehabilitation

The purpose of this study is to examine older adults' long-term recovery outcomes following coronary artery bypass surgery (CABS) and to examine differences in recovery between those who attend a formal cardiac rehabilitation program (FCRP) and those who do not. It is important for health care providers to understand the factors that contribute to long-term recovery so that interventions can be developed to enhance the recovery of older adults' following CABS.

The specific aims of this study are to answer the following research questions: (1) What are the factors that predict older adults' functional status, well-being, and exercise behavior six months after CABS?; (2) Do older adults who participate in FCRP have better functional status, well-being, and exercise behavior than those who do not attend?; and (3) Will these differences remain when controlling for the effects of social support, gender, self-efficacy, co-morbidity, depression, and cardiac functional status? The SF-36 will be used to collect necessary patient-based information.

A sample of 136 adults 70 years of age and older recovering from CABS will be recruited from a hospital in Northeastern Ohio. Data collection will start in July of 1997, and will occur for 15 months. Data will be collected before discharge, six weeks, and six months after discharge from the hospital.

Contact: Mary A. Dolansky, RN, PhD Candidate. Case Western Reserve University, Frances Payne Bolton School of Nursing. 10900 Euclid Avenue, Cleveland, Ohio 44106-4904. E-Mail: MAD15@po.cwru.edu.

Predicting outcomes in patients undergoing coronary artery bypass surgery and open heart valve surgery

This study of 370 coronary artery bypass surgery and open health valve surgery patients combines biological and social determinants of health in predicting outcomes post cardiac surgery. Patients were recruited prospectively from a teaching hospital, which serves approximately 500,000 people, in the Province of Saskatchewan, Canada. Interviews were conducted before surgery, three months after surgery, and one year after surgery. Information regarding perioperative status, demographics, depression, coping reactions and health status (measured using the SF-36) was gathered. One-third of the one year interviews have yet to be conducted.

A second study of the same population has been designed to determine whether psychosocial interventions improve quality of life outcomes after open heart surgery. We propose to demonstrate that relaxation therapy with positive reinforcement will improve patients' coping beyond provision of pre- and postoperative information alone. This randomized study will begin in fall of 1997.

Contact: David Johnson, MD, CM or Tracey Carr, Department of Medicine, Royal University Hospital, University of Saskatchewan. 103 Hospital Drive, Saskatoon, Canada, S7N 0W8.

Quality of life outcomes after CABG surgery

The Quality of Life Outcomes After CABG surgery study is being conducted by a team of cardiothoracic surgeons in Washington state and by researchers at the University of Washington. The objective of the study is to examine patient-reported outcomes after bypass surgery using the SF-36 and Seattle Angina Questionnaire (SAQ) and to identify pre-operative factors associated with improvement in score. Over 1000 patients have been enrolled from multiple sites in Washington. Patients who consented to the program were asked to fill out baseline socio-demographic information along with the SF-36 and SAQ. Clinical and hospital course data were also collected. The SF-36, SAQ and several questions about perception of improvement were administered via a postal survey six months and one year after surgery.

Approximately 80% of patients returned surveys at each time point. Analysis is now being conducted to describe the patterns of change in score for global health status (SF-36) and the cardiac specific health status (SAQ). Using multi-variate analysis, factors that are independent predictors of improvement in health status will be identified. Additional analyses will examine the relationship between prospectively measured change in health status score and patient perception of improvement at each time point. It is hoped that these findings will contribute to our understanding of the impact of CABG surgery on quality of life and promote the use of health status as an outcome for quality improvement activities.

Contact: J. Richard Goss, MD, MPH, Clinical Assistant Professor of Medicine, University of Washington School of Medicine/FHCQ, 83 South King St., Suite 215, Seattle, WA 98104; John Spertus, MD, MPH, Section of Cardiology, 2301 Holmes St., Kansas City, MO 64108.

Work in progress with the SF-36

What began as a faculty collaborative to introduce process improvement concepts to health professionals has resulted in a unique relationship between Psychiatry and Cardiothoracic Surgery at Dartmouth-Hitchcock Medical Center.

The initial goal of the collaborative was to improve the process of timely psychiatric consults for patients. Complete SF-36 data collected preoperatively and six-months post-operatively on 515 CABG patients is currently being analyzed. As a member of the Northern New England Cardiovascular Disease Study Group, information on severity of illness, comorbid conditions, and process and outcome variables is also collected. After merging the two datasets, one relationship that was chosen to be focused upon was that of preoperative mental health scores and post-operative length of stay. It was found that those who scored in the bottom 50% of the mental health category tended to stay in the hospital longer, even after controlling for comorbid disease, severity of illness, and age.

Physicians working in psychiatry and behavioral medicine have chosen further screening tools (CES-D, Beck Anxiety Inventory, BASIS-32, and TWEAK) to identify those patients that may benefit from a brief preoperative intervention. Intervention will be geared to need, and could involve nursing, social services, psychiatry, and behavioral medicine.

Contact: Virginia Beggs, MS, RNC, ARNP, Section of Cardiothoracic Surgery, Dartmouth-Hitchcock Medical Center, One Medical Center Drive, Lebanon, NH 03756.

Video-assisted saphenous vein harvest in support of coronary artery bypass surgery

The Center for Health Services and Outcomes Research at Camcare Health Education and Research Institute will conduct two outcomes studies on video-assisted saphenous vein harvest in support of coronary artery bypass surgery. These studies are a continuation of retrospective outcomes research conducted in 1996. The first study is a prospective longitudinal randomized controlled clinical trial comparing clinical, financial, and patient-related outcomes (using the SF-36 and the Quality of Well-being scale) between the "open" and endoscopic procedures. The second study will focus on building a cost-effectiveness model for determining economic benefits between the two procedures. It is anticipated that these research efforts will include partnering with the manufacturer of the harvesting kits, and will evolve into multi-site comparative studies.

The proposed outcomes research initiatives have two primary objectives: first, to begin the development of a rational research platform which supports stakeholder decision-making for adopting and/or rejecting endoscopic saphenous vein harvest as the standard of care; second, to publish the model as a research platform in peer-reviewed journals to educate stakeholders as well as expose the outcomes research to the academic community.

Contact: James H. Forsythe, PhD, Director, Center for Health Services and Outcomes Research, Camcare Health Education and Research Institute, 3211 MacCorkle Avenue, Charleston, WV 25304.


Condition-Specific Functional Status/Quality of Life Measures

Seattle Angina Questionnaire (SAQ)

A 19-item, self-administered questionnaire designed to measure functional status of coronary artery patients. The questionnaire is composed of five scales to assess clinically important dimensions of coronary artery disease: Physical Limitation, Anginal Stability, Anginal Frequency, Treatment Satisfaction, and Disease Perception.

Spertus JA, Winder JA, Dewhurst TA, Deyo RA, Fihn SD. Monitoring the quality of life in patients with coronary artery disease. Am J Cardiol 1994; 74:1240-44.

Spertus JA, Winder JA, Dewhurst TA, Deyo RA, Prodzinski J, McDonnell M, Fihn SD. Development and evaluation of the Seattle Angina Questionnaire: a new functional status measure for coronary artery disease. JACC 1995; 25(2):333-41.

Generic Functional Status/Quality of Life Measures

The following instruments, most of which are distributed by the Trust, have been used to measure functional status and quality of life in evaluations of individuals undergoing cardiac surgery.

Psychological Adjustment to Illness Scale (PAIS)

Folks DG, Blake DJ, Fleece L, Sokol RS, Freeman AM. Quality of life six months after coronary artery bypass surgery; a preliminary report. South Med J 1986; 79:397.

Langeluddecke P, Baird D, Hughes C, Tennant C, Fulcher G. A perspective evaluation of the psychosocial effects of coronary effects of bypass surgery. J Psychosom Res 1989; 33(1):37-45.

Quality of Well-being Scale

Ganiats TG, Palinkas LA, Kaplan RM. Comparison of Quality of Well-being scale and functional status index in patients with atrial fibrillation. Medical Care 1992; 30:958-64.

Sickness Impact Profile

Artinian NT, Duggan CH. Sex differences in patient recovery patterns after coronary artery bypass surgery. Heart Lung 1995; 24(6):483-94.

Page SA, Verhoef MJ, Emes CG. Quality of life, bypass surgery, and the elderly. Can J Cardiol 1995; 1(9):777-82.

Redeker NS, Mason DJ, Wykpisz E, Glica B, Miner C. First postoperative week activity patterns and recovery in women after coronary artery bypass surgery. Nursing Research 1994; 43:168-73.

Redeker NS, Mason DJ, Wykpisz E, Glica B. Women's patterns of activity over 6 months after coronary artery bypass surgery. Heart Lung 1995; 24(6):502-11.

SF-36 Health Survey

Allen LW, Frigon L. Outcome assessments in a changing hospital environment: elective CABG patients (abstract). 1995 AHSR & FHSR Annual Meeting Abstract Book. Chicago: 1995;51.

Cleary PD, Epstein AM, Oster G, Mossissey GS, Stason WB, Debussey S, Plachetka J, Zimmerman M. Health-related quality of life among patients undergoing percutaneous transluminal coronary angioplasty. Medical Care 1991; 29(10):939-50.

McCarthy MJ, Shroyer AL, Sethi GK, Moritz TE, Henderson WG, Grover FL, et al. Self-report measures for assessing treatment outcomes in cardiac surgery patients. Medical Care 1995; 33(10 Suppl):OS76-OS85.

Nugent WC, Schults WC, Plume SK, Batalden PB, Nelson EC. Designing an instrument panel to monitor and improve coronary artery bypass grafting. Journal of Clinical Outcomes Management 1994; 1(2):57-64.

Phillips RC, Lansky DJ. Outcomes management in heart valve replacement surgery: early experience. Journal of Heart Valve Disease 1992; 1:42-50.

Rector TS, Ormaza SM, Kubo SM. Health status of heart transplant recipients versus patients waiting heart transplantation: a preliminary evaluation of the SF-36 questionnaire. J Heart and Lung Transplan 1993; 12(6 Pt 1):983-6.

Reifler DR, Feinglass J, Slavensky R, Martin GJ, Manheim L, McCarthy WJ. Functional outcomes for patients with intermittent claudication: bypass surgery versus angioplasty versus noninvasive management. Journal of Vascular Medicine and Biology 1994; 5(5-6):203-11.

Rubin HR, Curtis L, Hatch N. Patient-reported health status: is it an independent predictor of health outcomes following bypass surgery? [abstract] In: 1995 AHSR & FHSR Annual Meeting Abstract Book. Proceedings of the 12th Annual AHSR & FHSR Annual Meeting: 115 June 4-6. Chicago. 1995:137.

Research Summary: 1995-1996

The following articles were identified through a medline search on cardiac surgery or CABG surgery and quality of life. The abstracts have been abbreviated and arranged in reverse chronological order.

Improvement in quality of life and exercise capacity after coronary bypass surgery.

Sjland H, Wiklund I, Caidahl K, Haglid M, Westberg S, Herlitz J. Arch Intern Med. 1996 Feb; 156(3):265-71.

Outcome after coronary artery bypass grafting is usually evaluated by exercise stress testing. Increased exercise capacity and reduced angina pectoris have been equated with improved quality of life, but this represents a limited view. This study sought to prospectively evaluate the effects of coronary artery bypass grafting on quality of life and exercise capacity and their interrelationship. In a consecutive series of patients (N = 2365) who underwent coronary artery bypass grafting, a questionnaire to assess quality of life before and two years after surgery was administered. The greatest improvement in quality of life after coronary artery bypass grafting appeared in those patients with the most impaired exercise capacity, those with the most severe angina pectoris, and women. Improvement in exercise capacity was greatest in patients with the poorest preoperative exercise capacity and in men. These findings indicate that exercise testing is of limited value as a measure of quality of life and that assessment by a questionnaire has a complementary place.

Quality of life following coronary bypass surgery.

Speziale G, Ruvolo G, Marino B. J Cardiovasc Surg (Torino). 1996 Feb; 37(1):75-8.

Quality of Life (QL) following coronary bypass surgery (CABG) can be modified by medical and non-medical factors (IE, functional class, medical therapy, psychological changes, economic status, educational level, return to work, country origin). The aim of this study was to evaluate QL in 203 patients that underwent CABG (Group A). QL perception was assessed by 5 questionnaires self-rated by the patient. Data obtained by surgical series were compared with 107 patients with coronary artery disease and treated medically (Group B), and a population of 102 normal subjects (Group C). Our findings demonstrated that general well-being, functional status and social participation improved in Group A. Sexual activity decreased in Group A. CABG did not modify job satisfaction. Chest pain was the main variable negatively influencing return to work; age, work before CABG, low educational level, and country area were negatively related non-medical variables.

Predictors of enhanced well-being after coronary artery bypass surgery.

Steine S, Laerum E, Eritsland J, Arnesen H. J Intern Med. 1996 Jan; 239(1):69-73.

The objectives of this study were to assess patients' perception of the therapeutic outcome after coronary artery bypass surgery and to find predictors for increased well-being. Self-administered questionnaires (Family APGAR and GHQ-30) were completed on admission and at the follow-up after 12 months, together with functional classification according to the NYHA index. The majority of the patients reported significant improvement in their physical and psychosocial functioning one year after coronary artery bypass surgery. Mental distress and male sex were significant predictors of enhanced well-being. In conclusion, questionnaires on psychosocial well-being such as the GHQ-30 may, in addition to health status measurements, offer additional useful information when coronary artery bypass surgery is considered.

Prospective study of quality of life before and after open heart operations.

Chocron S, Etievent JP, Viel JF, Dussaucy A, Clement F, Alwan K, Neidhardt M, Schipman N. Ann Thorac Surg. 1996 Jan; 61(1):153-7.

The aim of this prospective study, with completion of questionnaires before and three months after open heart operations, was to evaluate the improvement of quality of life (using the Nottingham Health Profile), brought about by these operations and the predictors of this improvement. From January to July 1994, 215 consecutive patients underwent elective open heart operations. The comparison between mean preoperative and postoperative scores showed an improvement in all sections of quality of life. An average of 80% of patients were improved by their operations. The improvement was similar for patients who underwent coronary artery bypass grafting versus valve replacement, and for patients with no postoperative events versus those with nonlethal postoperative complications. The strongest predictive factors for quality of life are in this case, age and New York Heart Association functional class.

A regional collaborative effort for CQI in cardiovascular disease- Northern New England Cardiovascular Disease Study Group.

Malenka DJ, O'Connor GT. Jt Comm J Qual Improv. 1995 Nov; 21(11):627-33.

The Northern New England Cardiovascular Disease Study Group has met at least three times a year since 1987 to improve the care of patients with cardiovascular disease. The group's collaborative nature has allowed members to have explicit discussions about medical decision making and practice and to benchmark with one another. By collaborating, members have been able to accumulate a large enough experience to examine the fine structure of adverse events, learn from them, and institute meaningful changes. Focusing on coronary artery bypass grafting (CABG) procedures, the group has used three improvement strategies: (1) Outcomes are monitored across institutions. All members receive reports with information on their experience, their organization's experience, and the regional experience. (2) All members receive training in quality improvement tools and techniques. (3) Members conduct comparative process analysis and benchmarking efforts to learn best practices for CABG surgery. In terms of results, the average in-hospital mortality associated with CABG surgery in the region has decreased. The group determined that low output failure is the most common cause of post-CABG death across all hospitals, while other causes of death (for example, stroke, bleeding, arrhythmias) have more uneven distributions across hospitals.

Quality of life in octogenarians after open heart surgery.

Kumar P, Zehr KJ, Chang A, Cameron DE, Baumgartner WA. Chest. 1995 Oct; 108(4):919-26.

The purpose of this study was to determine the quality of life in octogenarians after open heart surgery. Despite an increasing number of cardiac operations on octogenarians, the outcome as measured by functional status, independence of living, and psychological parameters of quality of life remain unproved. Two groups of octogenarians were reviewed retrospectively to determine operative mortality and functional results. Group 1 (n = 15, mean age of 83.2 years) and group 2 (n = 53, mean age 83.0 years) were studied. Operations included isolated coronary artery bypass grafting (group 1, 10; group 2, 29) and valve replacements +/- coronary artery bypass grafting +/- other procedures (group 1: 5; group 2, 24). Group 1 had 9% hospital mortality and 53% actuarial survival after a mean follow-up of 6.3 years. Group 2 had 17% hospital mortality and 72% actuarial survival after a mean follow-up of 1.5 years. At follow-up, significant improvements were observed in New York Heart Association (NYHA) angina class, congestive cardiac failure class, number of cardiovascular symptoms, and indices for satisfaction with overall life and general affect in both groups. Further, group 2 also showed significant improvements in independence of living, ease of life, and Karnofsky dependency category, but these improvements were less evident in group 1 after a longer period of follow-up.

Relationship between quality of life and exercise test findings after coronary artery bypass surgery.

Sjland H, Wiklund I, Caidahl K, Albertsson P, Herlitz J. Int J Cardiol. 1995 Oct; 51(3):221-32.

This study examined the correlation between quality of life and exercise testing in 554 patients two years after coronary artery bypass surgery. Traditionally, evaluations after coronary bypass surgery have focused on physical performance, medication, and anginal symptoms, which cannot be said to represent quality of life. The Physical Activity Score, the Nottingham Health Profile and the Psychological General Well-being Index were used for evaluation of quality of life. Significant correlations, although of small or moderate magnitude, were found between exercise capacity, chest pain and most subscales of quality of life, with the highest correlation coefficients for dimensions reflecting physical abilities and pain. While quality of life correlates significantly with exercise capacity and chest pain during exercise two years after coronary bypass surgery, only dimensions of pain and physical performance are reasonably well correlated with exercise test results.

Quality of life following coronary surgery and balloon angioplasty; more chest pain and social inhibition following angioplasty.

Van Berkel TF, Erdman RA, Breeman A, Boersma H, Van den Brand MJ. Ned Tijdschr Geneeskd. 1995 Aug; 139(34):1733-7.

The purpose of this comparative, prospective study was to determine the differences in quality of life between patients who had a coronary artery bypass graft (CABG) and patients who had a percutaneous transluminal coronary angioplasty (PTCA). Quality of life just before the intervention was measured using psychological questionnaires. The quality of life aspects were not shown to differ between the two groups, except for 'social inhibition' (p < 0.05): the PTCA group experienced more problems and had less energy (p < 0.01). No significant differences between the PTCA and CABG groups were found regarding to the change in quality of life caused by the intervention.

Functional outcomes of cardiac surgery for the elderly.

Davids D, Verderber A. J Cardiovasc Nurs. 1995 Jul; 9(4):96-101.

Despite their greater risk of perioperative complications and/or death after coronary bypass surgery, older patients represent an increasing proportion of such patients. This article reviews and summarizes the results of studies addressing the functional status and quality of life of elderly patients after cardiac surgery. Study findings indicate that many elderly patients benefit as much as younger patients from this surgery. Nurses in a position to serve as advocates for patients considering this surgery or requiring care after the procedure must be skilled at assessing functional status in these patients and monitoring for signs of postoperative functional deficit.

Developing Risk-Adjustment Models:
The Role of Social Survey and Clinical Variables

In the economics of health insurance, health risk is defined as "the expected value per person of per capita costs of efficiently-provided, comprehensive (preventive, diagnostic, and therapeutic) health care services delivered to a defined population for a specific time period," e.g., one enrollment month or year (Hornbrook and Goodman 1991). A risk-adjustment (RA) model is intended to serve as a national, standardized system for predicting a population's future health care costs. RA models break down variation in health insurance premiums into two categories: risk (IE, enrollees' permanent health status/use propensity) and inefficiency. That is, these models determine which portion of health insurance costs relates to true financial risk and which portion relates to health system inefficiencies. Health risk is a probability distribution, where predictability is achieved through pooling individuals into large groups. While no one can really estimate whether a given individual might have an automobile accident or get cancer in the following year, we can estimate the number of automobile accidents or cancer cases in a population in that year.

Editor's Note:

This article, written by Dr. Mark C. Hornbrook and Trust staff members, summarizes his presentation given at the Trust's third State-of-the-Art Health Outcomes Conference.

Dr. Hornbrook is Senior Investigator and Director of the Research Program in Health Services, Social and Economic Studies, Kaiser Permanente, Center for Health Research.

Editorial Advisors for Research and Policy

Anne M. Damiano, ScD
Covance Health Economics and Outcomes Services

Robert B. Keller, MD
Maine Medical Assessment Foundation

Marianne Sullivan, PhD
University of Goteborg

Health risk is also a classification system that groups individuals into homogenous clusters based on their underlying permanent, IE, persisting, health status. Plans enrolling sicker members have higher costs to produce the same levels of life-years saved and improvements in health-related quality of life (HRQoL) compared to plans enrolling healthier members. To compare per member per month (PMPM) costs and prices across plans producing different results, an adjustment model is required. With the aging of the U.S. population and the increased policy concern about patient-related outcomes measurement, functional health status (FHS) and overall well-being of patients are becoming the focus of clinical, quality assurance, performance measurement, and risk adjustment efforts.

Forecasting the components of future health care expense is a complex process, including a myriad of factors that are seldom constant:

Because some of these factors are obvious and causal, care needs from one year to the next follow directly. Others relate to community preferences, expectations, habits, and willingness to allocate resources to health care. In contrast, many expense-driving events are completely random.

Providers and health plans know the populations that require the most care- low-birthweight newborns, pregnant women, the frail elderly, those with massive organ failure, those with progressive or severe chronic conditions- and little can be done to change that within the time horizon of most health plans. By accounting for underlying exogenous (or "inherited") health status, however, plans, payers, and consumer groups can predict which patient groups are likely to be costly. Without effective RA models that rely on patient-based information, selection bias and market failure are inevitable. Some health plans will gain major advantage over their competitors simply because they are more successful in exploiting the weaknesses of existing payment models, e.g., Medicare's AAPCC. This means that development and refinement of risk models is a continuous process on both sides of the market and that policymakers and payers need to be just as expert as health plans in forecasting risk.

Rationale for Risk Adjustment

As competing health plans contract with multiple risk-based payers such as Medicare, Medicaid, and employer groups, adversely-selected plans- those with sicker, and therefore more costly, enrollees- are likely to be driven out of the market, unless payers risk-adjust their enrolled populations using pre-enrollment member baseline characteristics. Risk adjustment methods account for health status differences in enrollment populations and thereby "equalize" them for a variety of purposes. Such purposes include developing payment models and comparing health plan or provider group performance. Risk adjustment is required because individuals have voluntary choice among health plans and medical treatments, which can lead to systematic differences in health status and costs. Risk adjustment is essential to the design of rational, clinically meaningful, and equitable systems of care. It is intended to motivate health plans to: 1) increase access to health care for all populations, 2) seek fair and accurate payment, and 3) continuously strive to improve quality and outcomes of care.

To date, the economic incentives of most risk-based payment systems penalize plans for caring for sicker and/or more functionally dependent patients and reward them for caring for the healthiest individuals. Some plans find it easier to reduce their risks by one percent than their PMPM costs. This situation is complicated by the fact that consumers often conceal poor health status from their health plans at enrollment because they fear they will be excluded from the plan. Because of perverse incentives built into the systems, health plans may be tempted to enroll only healthy consumers- people who need and will use health care the least- and avoid selling to the neediest (see Lee and Rogal (1997) for a more detailed summary of the rationale and methods of risk-adjusted payments).

RA's policy goal is to promote consistency and equity of treatment of health plans and providers in payment, quality assurance, and performance measurement systems. If these are achieved, clinicians can concentrate less on patient insurance plans and more on health status and health problems. By developing and using RA models to budget and pay for care, and to assess performance, health plans that enroll high-risk persons will be rewarded, transforming the popular notion that the healthiest persons generate the highest profits. RA seeks to equalize rates of return from the healthiest to the sickest patients. RA will also create appropriate and fair compensation for plans that enroll low-risk persons.

Conceptual Framework

The conceptual model to predict medical care utilization developed by Andersen and Newman (1973) provides three categories of independent variables- predisposing, enabling, and need factors. This model serves as the starting place to develop a conceptual framework for RA. "Need" refers to health status, which is represented by such factors as diagnosis and functional health status; need motivates patients to seek care. "Predisposing" variables are social and behavioral factors that capture patient propensity to use services, e.g., health concern, belief in the efficacy of medicine. "Enabling" characteristics include a range of socioeconomic variables and individual access, or lack thereof, to health insurance.

While need and predisposing variables are central to RA methodologies, enabling characteristics are not to be included in RA models. Enabling characteristics include income, health insurance, and social privilege. We do not seek to reinforce existing access barriers and inequalities that are unrelated to health status. Complicating this situation are a variety of factors that are simultaneously need and enabling characteristics, e.g., race/ethnicity, social class, education, and neighborhood, because they predict poor (or high) health status. To include such factors in RA models would be to perpetuate inequities in the current health care system by further limiting access to care among groups that have been systematically excluded from health care in the past.

The conceptual framework proposed for RA stresses the central role of functional health status (FHS) because it serves as the unifying metric: a common denominator that has meaning to patients, clinicians, health policy analysts, and health plan administrators. It characterizes the status of individuals who have many different disease or health conditions in common terms, improving our ability to compare and to make more equitable and meaningful decisions. It also focuses on the dimensions of health and illness- ability to perform usual, desired activities- that give most satisfaction to patients. With increasing rates of frailty and chronic illness among their memberships, health plans are becoming FHS managers rather than disease eradicators.

Estimation of Risk Weights

Unit of Analysis

One of the first decisions to make in developing a risk model is the unit of analysis. Health plans and their actuaries traditionally set premiums for subscriber units (SU). However, the individual, not SU, is the metric for delivering clinical care and, thus, provides for the most accurate information as well as the most variation in values. Individual-level data are also comparable across plans, an essential feature. SU data are an arbitrary insurance convention and, therefore, lack the clinical meaning of the individual. Because multi-person SUs are aggregations of individuals (commonly families), considerable averaging of within-SU expense variation occurs, which serves to improve apparent prediction performance of risk models. This gain in prediction accuracy is achieved at the cost of loss of information on intra-SU variation in health status and use propensity. The individual approach allows each person, both parents and children, to contribute to the risk model their own equally-weighted health status and use propensity characteristics. In an SU model, a family of eight persons is counted the same as a single-person subscriber unit. The clinical shortcomings of an SU model are obvious.

Health Risk Indicators

The "health risk indicators menu" for creating risk models includes: cost-weighted disease-specific mortality weights, demographics, prior use, morbidities, physiological measures (e.g., body mass index and blood pressure) and laboratory results (e.g., hematocrit, urinalysis), and functional and/or perceived health status. The criteria governing selection of variables for a risk model include: demonstrated reliability and validity as health status measures, availability and cost of the measures, the degree to which the measures are standardized, the degree to which they are understandable to users in different settings and different parts of the country, the ease with which they can be audited, and their utility in multiple applications (including clinical care, health plan administration, and to define and operationalize health policy, such as identifying high-priority target populations).

Using mortality weights carries the appearance of paying health plans for higher death rates, which most policymakers view with alarm. Paying for higher prior use appears to be compensating health plans for waste, which also raises concerns for policymakers. Physiologic measures and laboratory tests, while directly related to medical decision-making, are still too complex to include in risk models. Therefore, we focus here on three categories of health status risk indicators that have been explored as options for inclusion in risk-adjustment models: demographics, morbidities, and functional health status (e.g., Hornbrook and Goodman 1995, 1996, Gruenberg et al. 1996, Hornbrook et al. 1996).


Demographic variables are useful because they are universally understood, readily available, and offer many important ways to describe enrollees. In addition to the traditional variables of age and gender, other useful variables include source of payment because it captures the nature of the enrollee when direct health status measures are not available. Medicare status usually identifies individuals as aged, disabled, and/or retired. Medicaid status is a socioeconomic correlate of health. These discriminate between employer group and self-pay enrollees. Institutionalization is another useful variable that indicates nursing home use among frail aged and disabled populations. "Working aged" status distinguishes between older individuals who work and those who don't, while "early retiree" status tends to pick up persons who retire for reasons of poor health. Other demographic variables include enrollee status, e.g., subscriber, spouse, dependent, and subscriber unit size.

Demographic risk variables have several important strengths- they are: objective, understandable, inexpensive to collect, universal and well-defined, valid, easily obtained from enrollment data, and easily audited. Age and gender capture variation in the prevalence and severity of diagnoses across the lifespan. The weaknesses of demographic variables are: they are not direct measures of health status; demographic groups exhibit considerable within-group heterogeneity; they capture very little (approximately one percent) variation in annual per capita expense; and they can be easily skewed through creative enrollment and disenrollment practices.


Like demographic information, diagnoses are relatively easy to access from automated inpatient discharge abstract systems and insurance claims files. Diagnostic data are available in many forms, e.g., from self-reported medical histories, principal and secondary inpatient diagnoses, and primary and secondary ambulatory encounter diagnoses. Treated morbidities have several strengths for risk modeling: chronic diseases often persist until death and are strong predictors of future medical care use and expense; serious acute diseases may need continuing care; some diagnostic codes capture severity of illness and are strong predictors of use and costs. Furthermore, diagnostic codes are widely available from automated clinical and claims systems. Disadvantages of using diagnostic information to adjust risk include: treated diagnoses erroneously assume adequate access to care for all; current information systems usually do not indicate whether a diagnosis is the primary presenting problem, an adverse reaction to treatment, a "rule-out" diagnosis, or a final diagnosis; different providers use different words for documenting the same disease and different coders use a variety of codes for the same disease; a diagnosis that is coded is not necessarily being treated and treated conditions are not always documented; and there is no structure to assess "episodes of care."

Functional Health Status

Demographic and disease-based risk factors are integral to the development of risk models, but FHS is a critical factor. Demographics, morbidities, and FHS are not competing approaches in risk adjustment. Rather, the three should be used in combination to build the most accurate risk profile of a population. Persons with diabetes, for example, will report themselves to be at all ranges of perceived and functional health. This information is extremely useful in discriminating among diabetics who are severely limited by their disease from those whose positive mental health enables them to function at higher levels than expected for their physiologic condition. Ideally, an HMO-based all-payer health status prospective RA model will use automated demographic and diagnostic data together with a comprehensive survey strategy to collect FHS information.

Because FHS portrays the effects of disease on individuals, it is a good indicator of the salience of an illness to individuals, the meaningfulness of illness to their lives, and, hence, their propensity to seek medical care. FHS is directly related to individual state of health and well-being. It is a useful indicator of "prodromal risk"- problems that many not have crossed a clinical threshold and been diagnosed yet.

A variety of well known and well tested FHS measures are available, including the SF-36, the SF-12, the SF-36V, the Sickness Impact Profile (SIP), the Quality of Well-Being (QWB), Basic Activities of Daily Living (BADL), and Instrumental Activities of Daily Living (IADL). These measures are reliable, valid, and consistent across populations. They have demonstrated prediction power. Because FHS is collected directly from patients and consumers, it is less subject to direct manipulation by health plans than risk indicators derived from health plan data systems.

Much like demographics and diagnoses, FHS data have disadvantages. Conducting surveys incurs high costs and administrative burdens, particularly if the survey is done solely for risk-adjustment purposes (as opposed to the multiple purposes for which FHS is well-suited). Another problem with FHS is that there are likely to be low response rates to the surveys and individuals most at-risk are likely to be most difficult to find (in nursing homes, foster homes, assisted living, relative's homes). The instruments may demonstrate low predictive accuracy if demographics, symptoms, and diseases are omitted from the risk model (because they were not included in the survey and there are no alternative data sources). Standardized instruments such as the SF-36 are often perceived as biased against certain populations, including severely disabled and frail, less-educated individuals, respondents who do not speak English, and respondents who do not trust the organization administering/sponsoring the survey.

Measuring FHS presents a paradox to risk modeling. In order for someone to participate in a self-report survey, s/he must meet a number of conditions. They must have a certain level of cognitive function, the ability to read at a sixth grade level, the ability to hear the questions (if they are asked by another person), the ability to speak to answer the questions (in response to a surveyor), or a certain degree of physical dexterity (to manipulate a writing utensil or computer mouse/keyboard if they are answering a questionnaire). Thus, completing a self-report survey requires a certain level of function that makes it inherently biased against people who may be most in need. Methods for identifying and selecting surrogate respondents are needed for impaired persons. This is already standard practice in long-term care settings. Health plans need to realize that considerable hidden functional impairment may exist in members' homes, but family members or other caregivers usually have sufficient knowledge of the situation to provide accurate FHS responses for impaired persons. Paradoxically, if the barriers of a self-report survey can be overcome and FHS data are collected, they have enormous advantages for a variety of purposes: clinical assessment and care planning, risk-adjustment of health plan payments, risk-adjustment of provider payments/budgets, consumer scorecards, outcomes assessment, and patient satisfaction.

Single Versus Multipurpose Surveys

Conducting FHS surveys for the single purpose of risk adjustment comes with a number of disadvantages. First, this approach is the most vulnerable to manipulation by health plans and providers because response manipulation will not generate any self-inhibiting pressures. Second, FHS surveys are easy targets for legal challenges, particularly in the area of respondent confidentiality and potential harm to consumers from misuse of their information. Third, like all surveys, FHS surveys often yield poor response rates without appropriate follow-up with nonrespondents (e.g., reminders, remails, and telephone calls). Finally, special purpose RA FHS surveys may be less valid than other approaches because responses are not directly linked to other consumer benefit; IE, risk adjustment is a complex concept that is not easy to communicate and understand; moreover, its impact on individuals is unobservable.

Conversely, many advantages accrue to conducting multipurpose FHS surveys. First, the survey may be easily incorporated into routine clinical care, e.g., when patients are sitting in the waiting room before a visit, and FHS should govern the goals of medical care treatments. Second, response rates will be higher when the answers carry immediate physician reinforcement during the ensuing visit. Third, the data collected with a multipurpose survey are less vulnerable to manipulation by health plans and providers.

The Ultimate Goal

The ideal FHS survey design will attempt to reach the entire population of covered lives "just-in-time" to enable prediction of salient declines in health status. Those persons with severe chronic illness should be surveyed at every visit to define the trajectory of their functioning and overall health-related quality of life. Persons with a moderate or slight chronic illness should be surveyed approximately two to four times a year to detect declines in FHS and HRQoL. Non-users should be assessed annually as a safety net to detect unmet health care needs, which may arise because of barriers to acess, consumer attitudes, or ignorance about health and medicine. The safety net is very important because morbidity models only detect needs that have been validated by physicians, which means access had to occur.

While many health plans are proactive in collecting FHS data, much more data are still needed in order to improve: generalizability across geographic regions, urban/rural location, and type of health plan; stability of risk weights for rare diseases; and identification of populations with special needs. The more that a standard instrument such as the SF-36 is used, the more efficient our efforts to compare information across plans and over time. In addition, by sharing experiences and information, plans are more likely to improve outreach efforts to underserved populations and to identify and experiment with innovations related to data collection and automation.

Any model, including simple demographic models, can predict the mean of a large population very accurately, but it takes a complex, multifaceted model to produce accurate and stable predictions for biased subgroups of populations: the very healthy versus the very ill.


1. Hornbrook MC, Goodman MJ (1991). Health plan case mix: definition, measurement, and use. In Hornbrook MC (Ed.), Advances in Health Economics and Health Services Research, Vol. 12. Greenwich, Connecticut: JAI Press Inc., pp. 109-146.

2. Andersen R, Newman JF (1973). "Societal and Individual Determinants of Medical Care Utilization in the United States." The Milbank Memorial Fund Quarterly: Health and Society 51(1) : 95-124.

3. Gruenberg L, Kaganova J, Hornbrook MC (1996). Improving the AAPCC with health status measures from the MCBS. Health Care Financing Review 17:59-76.

4. Hornbrook MC, Goodman MJ (1995). Assessing relative health plan risk with the RAND-36 health survey. Inquiry 32:56-74.

5. Hornbrook MC, Goodman MJ (1996). Chronic disease, functional health status, and demographics: a multi-dimensional approach to risk adjustment. Health Services Research 31:283-307.

6. Hornbrook MC, Goodman MJ, Fishman PA, Bachman DJ, Rosetti MO, Nelson AF (1996). Global risk-assessment models: two-part, three-segment Approach. Risk Adjusters: Where Are We? Proceedings of the Society of Actuaries Health Care Symposium. Schaumburg, IL: Society of Actuaries, pp. 117-140.

7. Lee C, Rogal D (1997). Risk Adjustment: A Key to Changing Incentives in the Health Insurance Market. Washington, D.C.: Alpha Center, The Robert Wood Johnson Foundation.

A New Approach to Outcomes Management
for Patients with Diabetes

The current challenge for the health outcomes field is to facilitate the shift from simply collecting functional health status information at different points in time to the implementation of quality improvement interventions which measure and improve patient outcomes. Treatment guidelines for many conditions are now available with the potential to improve outcomes. However, the experience and data of Ofstead & Associates Inc. indicate that many providers are not adhering to the central tenets of the guidelines.

Written by:

Cori Ofstead
Principal Consultant, Ofstead & Associates, Inc., St. Paul, MN

Diane Gobran, MBA
Principal Consultant, Ofstead & Associates, Inc., St. Paul, MN

Michele Bonneville, RN, MSN, CDE,
Clinical Nurse Specialist and Coordinator for the Diabetes Self-Management Program at Burns Clinic Medical Center, Petoskey, MI

Editorial Advisors for Clinical Practice Applications

Laurence Benz, MPT
Kentucky Orthopedic Rehab Team

Sarah Purdy, MD

Marcia Stevic, PhD, RN
Health Services Advisory Group

Maximizing patient outcomes of care requires a willingness to: 1) change processes of care by using treatment guidelines as a reference point; 2) measure patient health status with clinical indicators, e.g., lab results and symptoms, and information about patient-based function and well-being, and 3) measure and track those data over time, using them to calculate and evaluate patient outcomes.

About Diabetes

Diabetes is a serious disease which has enormous costs for both patients and society. It is estimated that over 14 million people in the United States currently have diabetes, and the total cost of diabetes is estimated to exceed $100 billion annually. It is commonly understood that the risk of diabetic complications is associated with high blood sugar, high blood pressure, and high cholesterol levels. The importance of educating people with diabetes about self-management of their condition is supported by the fact that The Diabetes Control and Complications Trial (DCCT)1 found that tighter control of blood sugar correlated with fewer complications.

However, tight control of blood sugar may be difficult to establish and maintain while the patient is outside the physician's office. Physicians must rely on the patient to manage his or her own condition and this requires attending to a number of different disciplines. Unfortunately, patients tend to contact their physicians when they experience symptoms, and patients experiencing elevated blood sugars, high blood pressure, and/or abnormal lipids are often asymptomatic. Frequently, by the time these patients do complain of symptoms, it may be too late to prevent serious damage or adverse events.

Treatment of a serious disease without visible symptoms can only be accomplished through intensive patient education and training, and on-going clinical follow-up. According to the American Diabetes Association (ADA)2 and the DCCT, achieving near-normal or normal blood glucose levels requires comprehensive training in self-management and, for most individuals, intensive treatment programs. The ADA states that an effective care program should include routine monitoring of the patient's clinical status to enable early identification, intervention and management of problems. This type of monitoring is accomplished in part through routine blood tests and examinations of patients' feet and eyes.

Until the DCCT was published in 1993, many physicians relied on dietitians to educate patients with diabetes about their own care. The DCCT conclusively demonstrated that comprehensive education is critical to managing and/or stabilizing diabetes. While many providers today recognize the need for and value of patient education, still many cite an array of logistical and financial barriers to assuring that their patients receive comprehensive education. These barriers include:

Health Status Measurement for Patients with Diabetes

Collecting patient-derived health status data has become a familiar facet of outcomes programs. Many providers recognize the value of such information in patient management. David Lansky3 has said that health status measures are useful as prognostic or predictive tools to assist in identifying patients: 1) at greater risk of adverse physiological or psychological outcomes; 2) in need of focused or supplemental therapies; and 3) more likely to use health care services.

Therefore, patients with diabetes should be routinely asked: How are you functioning in your daily life physically? Emotionally? Socially?

Treatment Guidelines

Treatment guidelines are designed to provide "best practice" information and a consistent framework for the management of patients with a particular condition. As a result of the recent migration toward managed care, and quality improvement and accountability, providers and organizations are re-evaluating their current practices. Many providers find themselves practicing "downstream medicine," that is, saving drowning victims as opposed to moving upstream and teaching potential victims how to swim. To alleviate this situation systematic processes must be implemented with the goal of preventing complications as opposed to waiting for their manifestation.

The DCCT and other studies have demonstrated that proactive, preventive care programs for people with diabetes are necessary, and that good outcomes are indeed possible. Furthermore, good care for diabetic patients goes beyond the clinical treatment of diabetes. Many patients with chronic conditions such as diabetes consider their specialist to be their primary care provider, and this inevitably increases the responsibilities of this provider, as s/he must identify and treat concurrent conditions.

Depression is a concurrent condition that frequently goes unrecognized and untreated in patients with diabetes.4 In fact, research has shown that 20-40% of patients with chronic medical conditions are at risk for depression.5 In 1993, The Agency for Health Care Policy and Research (AHCPR) guideline for the diagnosis and treatment of major depression in primary care settings substantiated the need for primary care and other physicians to screen their medical patients for the risk of depression.6 The Prime-MD 1000 Study demonstrated that patients identified as "at risk" for depression appear to have significantly diminished functional status and quality of life compared with those without depression.7 This is especially unfortunate, since Major Depressive Disorder is a treatable condition- with appropriate care, 80% of depressed patients will fully recover within four months.

The Burns Clinic Medical Center Outcomes Management Program

The Burns Clinic Medical Center, with 123 physicians serving 26 counties, is the major provider of outpatient health care in northern Michigan. In 1994, Burns Clinic partnered with PhyCor Inc., headquartered in Nashville, TN. PhyCor operates 47 clinics with about 3,280 physicians in 28 states. In 1996, the Burns Clinic became involved in the PhyCor Outcomes Program, choosing to focus on diabetes. Currently, 11 internal medicine providers at the main clinic in Petoskey, MI are enrolling patients.

Early in PhyCor's data collection efforts for its Outcomes Program, clinicians observed that people with diabetes seemed to be experiencing high rates of depression. In light of this finding, the Burns Clinic began developing a comprehensive diabetes management program that incorporates processes to identify people with depression and introduce treatment as early as possible, and to measure outcomes over time. The following is a case study of the quality improvement program at Burns Clinic.

The Diabetes Self-Management Program

The Diabetes Self-Management Program (DSMP) is a comprehensive, outcomes-based educational system available to patients with Type I, Type II, and gestational diabetes. The DSMP was implemented at the Burns Clinic and certified by the Michigan Department of Community Health in 1996. Medicare/Medicaid and most insurers pay for the program.

At Burns, the critical first step towards improving the clinical management of people with diabetes is to ensure that patients receive the routine laboratory tests and physical exams recommended in the ADA Guideline. The availability of aggregate lab results highlighted the need for a program to improve patient self-management. To further increase the percentage of completed lab tests, the outcomes program staff are currently designing a new feedback mechanism for providers.

The DSMP educational program involves 6-8 hours of class time, and includes instruction by a Certified Diabetes Educator and a Registered Dietitian. Patients learn day-to-day management skills that allow them to successfully reach their own goals. An outline of the Burns Clinic Diabetes Patient Education Program includes: an introduction to diabetes, carbohydrate counting, meter instruction and data management, medications and adjustment protocols, beginning or advanced insulin therapy, complications and ADA goals for control, self-management skills, weight loss and follow-up, and lipid management.

Burns currently collects generic patient-reported functional status and quality of life data as well as condition-specific, diabetes-related data. A physician survey is employed to collect clinical indicators and laboratory values.

A DSMP program coordinator tracks program participants' clinical outcomes three to six months after they complete the program to determine its effectiveness over time. The results are promising. There has been significant improvement in overall blood sugar control for patients who have completed the program. At enrollment, the average total glycosylated hemoglobin of program participants was 11.4%. Results of this test three to six months after program completion showed an average total glycosylated hemoglobin of 7.8%, a result that is within normal limits. On average, non-participants did not have a similar drop in overall blood sugar level at their six months follow-up. Currently, the program coordinator is working with providers and clinic staff to improve referrals to the program.

Depression Guideline

PhyCor's analysis of their aggregate outcomes data pool also demonstrated a high prevalence of depression risk among people with diabetes. Similarly, the outcomes staff at Burns observed a significant prevalence of patients "at risk" for depression in their own data and that these patients had lower functional health status scores than other patients.

In response to these findings, Burns is collaborating with Ofstead & Associates, Inc. and the Centers for Outcomes Research & Effectiveness at the University of Arkansas for Medical Sciences (UAMS) to customize the AHCPR guideline for the diagnosis and treatment of depression in primary care. In June of 1997, physicians who enroll patients in the outcomes program at Burns Clinic participated in an educational program geared toward early identification and treatment of depression in people with diabetes. As part of the new depression management program, Burns will be participating in a research study led by Brenda Booth, Ph.D. and Theresa Kramer, Ph.D. at UAMS. The study seeks to determine the prevalence of depression in the Burns' diabetic population, and assess the effect of early identification of depression and appropriate treatment on patients' functional and clinical outcomes.


Integrating "best practice" principles with outcomes data can provide organizations with a new and more effective approach for managing patient care. As a benchmark of quality care, guidelines lead organizations towards practicing upstream medicine. The DCCT conclusively states that those patients who successfully self-manage will be the strongest swimmers.

As the Burns case study clearly indicates, patient blood sugar control can be significantly improved by incorporating treatment guidelines and patient education into a diabetes management program. The DSMP, combined with the use of health status and clinical data, allows Burns to monitor patients' clinical status, recognize the risk of depression and take appropriate remedial actions, and improve care. With their new programs, Burns Clinic is actively changing processes of care to maximize individual patient outcomes for people with diabetes.


1. DCCT Research Group. The Effect of Intensive Treatment of Diabetes on the Development and Progression of Long-Term Complications in Insulin Dependent Diabetes Mellitus. NEJM 1993: 329; 977.

2. American Diabetes Association. Position Statement: Standards of Medical Care for Patients with Diabetes Mellitus. Diabetes Care 1997: 20

(Supplement 1); S5.

3. Lansky D, Butler JBV, and Waller F. Using Health Status Measures in the Hospital Setting: From Acute Care to 'Outcomes Management.'

Medical Care 1992: 30(5); MS57.

4. Hirschfeld R, Keller M, et al. The National Depressive and Manic-Depressive Association Consensus Statement on the Undertreatment of Depression.

JAMA 1997: 277(4); 333.

5. Depression Outcomes Initiatives. Medical Outcomes Trust Monitor 1997: 2(1); 6.

6. Depression Guideline Panel. Depression in Primary Care: Volume 2. Treatment of Major Depression. Clinical Practice Guideline Number 5, Rockville, MD: Agency for Health Care Policy and Research, Public Health Service, U.S. Department of Health and Human Services; April 1993. AHCPR Publication # 93-0551.

7. Spitzer RL, Kroenke K, et al. Health-Related Quality of Life in Primary Care Patients with Mental Disorders. JAMA 1995: 274(19); 1511.

Health of Seniors Outcome Measure

As of January, 1997, the Healthcare Financing Administration (HCFA) requires managed care organizations (and eventually will require fee-for-service plans) serving the Medicare population to survey patients to determine changes in functional health status or health-related quality of life. This is HCFA's first outcome measure for Medicare risk plan members and it will fundamentally change the way providers who serve this population are evaluated. To date, most health care quality measures have focused on processes of care. This initiative is a milestone in the field of health outcomes, and will surely act as a catalyst to improve the quality and value of healthcare.

Editors Note:

The Trust is grateful to HCFA, NCQA and The Health Institute for background information for this article.


Editorial Advisors for Health System Improvement

Don Nielsen, MD

Dana Gelb Safran, ScD
The Health Institute, New England Medical Center

MaryAnn Stump, MA
BlueCross BlueShield BluePlus, Minnesota

The Health of Seniors Outcome Measure is part of the most recent version of the National Committee for Quality Assurance (NCQA) Health Plan Employer Data and Information Set (HEDIS). HEDIS 3.0, the third version of the measure set recently released by NCQA, is a set of standardized performance measures designed to provide purchasers and consumers with information to assist them in making reliable and meaningful comparisons of the performance of managed health care plans.

Development of the Health of Seniors Measure

HEDIS 3.0 was developed by the Committee of Performance Measurement (CPM), a broad-based committee whose members reflect diverse constituencies, e.g., providers, purchasers, consumers, health plans, and academics. The CPM was organized and staffed by NCQA and funded by public and private sources.

HEDIS 3.0 includes seven Medicare measures, six process measures (flu shots for older adults, breast cancer screening, beta blocker treatment after a heart attack, eye exams for people with diabetes, follow-up after a hospitalization for mental illness and advising smokers to quit) and the Health of Seniors Outcome Measure.

The CPM achieved consensus that the SF-36 was the best health outcomes measure to use in the Health of Seniors Outcome Measure because it has been extensively researched. Moreover, both its reliability and validity have been repeatedly demonstrated. The use of the SF-36 was approved by the CPM in the summer of 1996. A Technical Expert Panel composed of ten experts continues to work on and refine the measure.

Health Systems Improvement

The implementation of the Health of Seniors Outcome Measure holds tremendous potential to change the nation's health systems in ways that are fundamental and far-reaching. This is due, in part, to the singular utility and relevance of the measure. However, it is also true because of the coordinated, collaborative, and effective partnership that produced the measure including some of the most important macrolevel organizations in health care in the United States today, most notably NCQA and HCFA.

HCFA is undergoing a fundamental change in its approach to buying health care services. It is leaving behind the role of "bill payer" and moving forward in the direction of becoming a value-based beneficiary-centered purchaser. It is HCFA's goal to evaluate and improve the value of care providers offer and to assist its beneficiaries in the selection of providers based on the quality of care they offer. Determining the value of care requires measuring cost and quality of care. Providers' rates of change on the Health of Seniors Measure at two year intervals will be used to quantify quality. The HEDIS 3.0 measures for the Medicare Managed Care population, particularly the Health of Seniors Measure, is one of several new HCFA initiatives that use information collected from Medicare beneficiaries to evaluate health care.

By virtue of the Health of Seniors measure in HEDIS 3.0, NCQA, HCFA, and other users of the measure will be making the following important change: for the first time, health plans will be held accountable for patients' health and well-being over time. The measure reflects the belief that high quality health care can have a significant effect on beneficiaries' status, IE, that health care can either improve, or at least slow the rate of decline in, senior plan members' ability to lead active and independent lives.

A major advantage of the Health of Seniors measure is its relevance and utility for the Medicare population. Individuals 65 years of age and older are likely to have multiple chronic diseases or conditions. The standardized SF-36 measure provides a common metric with which beneficiaries can be assessed, and health plans can be evaluated on the basis of performance. Health-related quality of life is a common denominator that permits a standardized assessment of the burden of disease and health conditions among populations, regardless of the number or type of conditions people have.

Description of The Health of Seniors Measure

The Health of Seniors is a global outcome measure based on the Medical Outcomes Study (MOS) SF-36 Health Survey. The SF-36 has been used in well over 200 published articles and demonstrated over time to be a reliable and valid instrument. In addition to the SF-36, the Health of Seniors Measure collects information for purposes of standardizing plan-to-plan risk adjustment. The additional data items include a standardized checklist of co-morbid conditions and sociodemographic variables, e.g., household size, social support, education, and gender.

The Health of Seniors measure will track patient outcomes by using summary scores computed separately for physical and mental health outcomes over a two-year period. For each outcome, change scores will be calculated for analysis and will be reported to indicate the percentage of Medicare plan members in three categories: better, same, or worse, at the two-year follow-up point (see Figure 1). It is expected that the differences between good, poor, and average plans will be detectable, and that the tracking and publishing of results will affect both quality and the kind of care provided.

Individual members will be categorized as 'worse' if the change in their SF-36 scores are negative and larger than expected. They will be classified as 'same' if the change in scores is within the expected range, and 'better' if the change in functional scores is positive and larger than expected. The resulting percentages in each category will be adjusted for the additional co-morbid conditions and socioeconomic factors collected in the survey. Because outcome is defined as a change in score, each respondent serves as his or her own "control."

Instrumentation and Data Collection

Beginning this year, a cohort of 1,000 Medicare beneficiaries will be selected from the eligible enrollment from over 200 Medicare managed care plans. To be eligible, a person must have been enrolled in the health plan continuously for at least six months. The Health of Seniors Measure will be mailed at the outset of selection, and again two years later. The change in responses in the two year period will be compared to an expected change based on national norms.

Experience has demonstrated that older Americans tend to respond to surveys, particularly those dealing with health status, at a rate in excess of 70%. This response rate will yield a statistically valid sample. For the mental component of the survey the denominator will consist of all persons who complete both surveys. For the physical component, the denominator will also include persons who die or move into long-term care facilities and therefore, are not able to return the questionnaire. In these cases, the members will be counted as having worsened.

To ensure maximum objectivity and reliability of the data, several vendors will be trained and certified to administer surveys to Medicare beneficiaries.

Availability of Health of Seniors Outcome Measure Data

Change scores for the Health of Seniors Measure will first be available in 1999. Plans will receive aggregated physical and mental scores each year, as well as the scores of the scales that make up the aggregate scores. Data sent to the health plans will not include patient identifiers.

While information about these health outcomes will not be available for two years, there is a great deal of interest among health plans, providers, and purchasers in the first generation of baseline information. For the first time, aggregate data of the Health of Seniors Measure will provide patient-based functional health status profiles of health plans' Medicare populations. To date, this information which will enable health plans to be risk-adjusted and compared in a meaningful way, is unavailable.

The Health Of The Public:
Moving Toward Consumer Driven Health Improvement

The focus of this presentation was on the role of health outcomes assessment in planning for population health services. Dr. Coye discussed a unique and forward thinking perspective on the health outcomes field, IE, a consumer-oriented point of view in which consumers will not only participate in defining what we value and measure in health care, but will drive this process. Accountability will no longer be measured solely by providers, insurers, and employers. Consumers, using networked health information systems, will be able to hold providers accountable for the quality of their care. The development of networked consumer health information systems such as internet-enabled technologies for accessing information; tracking and managing health; and communicating directly with providers and insurers will create a second revolution in the accountability and evaluation of health care. This revolution is just as profound as the first caused by managed care.

Editors Note:

This article is based on, and summarizes, the main points of Dr. Molly Coye's keynote presentation given at the 1997 State-of-the-Art Health Outcomes Conference in May of 1997.

Dr. Coye is the Executive Vice President of Strategic Development at HealthDesk Corporation.

Editorial Advisors for Introduction to Health Outcomes

Jame E. Dewey, PhD
HCIA/Response Technologies

Mark Kosinski, MA
Health Institute, New England Medical Center

Susan Laird
Sewickley Valley Hospital


All health care purchasers, public and private, spend money on medical care with little evidence of its effectiveness. The government spends additional billions of dollars to ensure that the goals of public heath are achieved. Unfortunately, there is no direct relationship between the amount of dollars spent on health care and the health status of a population. Further, we do not collect the information we need to understand the relationship between health care spending and the health of populations, or to hold the providers and systems that deliver care accountable. Without on-going, standardized measurement of individuals' health status, providers and communities cannot improve the public's health or allocate resources in a responsible way. This should, and increasingly does, cause great dissatisfaction amongst policymakers and even some informed taxpayers. The challenge is therefore to articulate the quality tenants of accountability, as opposed to its potential to chastise, to the appropriate decision-makers.


There are two significant movements underway that contribute to the emerging revolution as described above. One trend is the development of user-friendly computer technology platforms that will enhance the communication between consumers and health care providers in ways we are not currently able to envision. The other movement is the rapid growth of consumer demand for information, choice, and active participation in health care decision-making. The rapidity of these developments suggest that major changes in consumer health expectations and roles will take place over the coming decades.

Managed Care and Patient-Based Outcomes

There are several stages in the evolution of managed care systems. Many markets have already passed through the first stage of simply discounted payment for services without actual management of care. Much of the country is now in the early stage of "fragmented" managed care. This is the point at which physician payments are capitated but the rest of the delivery and management systems are still fragmented with contradictory incentives.

In the third stage of managed care, care is truly managed and the various elements of the system are knit together into a "virtually integrated system." This third stage includes standardized health status assessment and the evaluation of patient health outcomes over time. Developing and implementing such a system is an honorable and important task, yet it is also a formidable undertaking which is likely to take us the next 20 years, at least, to fully achieve.

The most critical factor in the realization of ideally effective managed health care systems is the financial driver. The financing and reimbursing of services play a central role in determining how services are actually delivered and managed. To date, we are only just beginning to tie the measurement of patient health status, quality of life, and outcomes into the financing and payment of services. Until the financial incentives are in place, the implementation of patient health status measurement over time will not become the norm.

Purchasers aiming to maximize the value of the health care dollars they spend are beginning to move toward measurement of health status of communities at large, beyond select groups that enter the health care system under specific insurance arrangements. Moreover, purchasers will begin to focus on the functional status and productivity of whole communities, in addition to clinical status and satisfaction with services. This means understanding the extent to which children can go to school and learn, workers can go to work and perform their job, elders can retain their independence in the community, and ultimately, understanding the productivity of the community, both in economic terms and in social values.

To accomplish this, the traditional public health model used to assess the heath status of populations must be supplemented by new approaches. This model has focused primarily on measuring the morbidity and mortality of populations. However, mortality and the relatively crude measures of morbidity that have been used to date are not very useful tools for helping individuals and providers work together to improve health. There are great opportunities for collaboration between health services researchers and those in the field of public health towards the goal of designing interventions to improve the health of populations and the use of patient-based health outcomes and functional assessments to evaluate the effectiveness of interventions.

Intersect of Patients, Providers and Technology

At the individual patient level, a growing number of entities, including providers, MCOs, pharmacies, home care agencies, and so forth, are trying to educate and provide guidance to individuals with chronic disease. But during actual encounters, providers and patients may not have the time together necessary to have needs met. Not surprisingly, patients find it difficult to manage their own care and have their needs met under these circumstances, however, such conditions have motivated patients with information-seeking behaviors to self-manage their conditions.

Increasingly, health care consumers have a range of credible sources from which they can seek information which enables informed decision-making. Proactive consumer behavior is integral to the continued evolution of managed care and it is important that it is positively rewarded, as such behaviors are conducive to outcomes assessment. However, mere assessment is not enough- methodologies must match the learning processes of the individual in order to keep him or her engaged. This is likely to occur through the diversification of new technology platforms, but as these platforms evolve and diffuse, health care providers are themselves behooved to explore new technologies capable of both collecting and disseminating information.

New technological platforms that support improved communication between patients and health care providers have the potential to enhance this situation by: 1) defining disease management and health maintenance objectives with much more specificity; 2) communicating those objectives and the associated treatment plans to patients more clearly; and 3) creating and facilitating teamwork between patients and providers to better achieve their mutual objectives. In addition to these objectives related to clinical practice, advanced computer technology will simultaneously improve how systems examine and track population characteristics that will provide information that is unprecedented at subpopulation levels.

Using New Technologies

To achieve our goals in health care, at least three things must take place. First, user-friendly and affordable computers, software applications and other communications technologies must become more readily accessible to providers. Second, physicians must "buy" into these new ways of practicing medicine. This is likely to come naturally with new cohorts of physicians who understand computer technology and its vast potential, IE the ability to access information from patients more effectively than allowed in one, or perhaps even a series, of office visit(s).

Finally, and most importantly, is the need for physicians to incorporate functional status assessment into their work with patients. In addition to the clinical information such as signs, symptoms, lab values and other diagnostic data, the health care team must focus on patients' day-to-day lives: Can they do their work at their jobs and can they function at home, in their social lives? Can children play and go to school?

A View of the Future

The bringing together of well informed, proactive consumers with providers armed with both clinical data and practical, functionally-oriented information and advanced technology, suggests a positive scenario for the future- increasing health care quality as measured by multidimensional health outcomes information.

The creation of, for example, networked health information systems will increase consumers' knowledge and expectations. This will accelerate the transformation of health care institutions as well as require health professionals to seek new skills and competencies to reflect the most updated practice parameters and provide patients the best possible quality of care. These fundamental changes in structure and process will help us achieve efficacy for patients. If patients believe that they can successfully manage their own health, they will be likely to attempt to do so. While advancing technologies will help accomplish such goals, in addition we will need: 1) providers able and willing to use these tools; 2) guidelines and protocols to enable behavioral changes regarding the use of technologies just as there are protocols and guidelines now for treatment of disease; 3) guidelines targeted to subpopulations, IE males with diabetes, 50 years of age and older, living in rural communities; and 4) widespread dissemination of these protocols.

If the above efficacy is achieved, the potential exists for the facilitation of outcomes measurement, which could in turn lead to new, necessary applications: we could finally hold governments accountable for public health expenditures; managed care would be supported as it enters its most consumer-oriented phase. Sadly, these opportunities will be missed if providers on the whole are not able to supersede widespread ambivalence towards new technologies.

1997 Medical Outcomes TRUST
The Monitor is a quarterly publication for members of the Medical Outcomes TRUST.

Jennifer DeLee

The Publication Group

On-Line Design:
Daniel W. Krueger