Fig. 1. The responses from 15 experts to the question, "How many people per 100,000 residents will have an acoustic tumor diagnosed this year in your geographical area?" are plotted showing the number of experts on the vertical axis. For comparison, the estimated incidence in Denmark is 0.8 per 100,000 persons.'

Methods

Subjects

The subjects were the 90 members of the House Ear Institute Alumni Fellowship Group. The membership includes physicians affiliated with the House Ear Clinic and physicians who have completed a 1-year clinical fellowship with the House Ear Clinic.

Questionnaire Using Delphi Technique

The Delphi method has three characteristics: (1) Anonymous response. Opinions of members were obtained by means of formal questionnaires. The list of experts and their answers were revealed to no one. (2) Iteration and controlled feedback. The median responses from the first round of questionnaires were sent to the members. The experts were asked if they wished to change any of their answers on the first questionnaire after they had reviewed the median answers for the group. (3) Statistical group response. The group opinion was defined as the median answer to each question on the second round of questions. Dalkey (ref. 5) designed these features of the Delphi method to minimize the biasing effects of dominant individuals, of irrelevant communications, and of group pressure toward conformity.

The initial questionnaire of 20 questions was sent to the 90 members of the Alumni Fellowship Group.

The questionnaire asked for low, middle, and high estimates of the true answer. Of the 90 members, 56 did not return the questionnaire, 19 returned an incomplete questionnaire, and 15 returned a completed questionnaire. The expert group comprised 14 neurotologists actively involved in the care of patients with acoustic tumors and one otologist active in diagnosis but not management of the tumors. Four of the experts practiced in an academic setting, 11 were in a private practice setting, and no one was in a managed care setting.

The 17% response rate led to concern that the sample may not be representative of the group as a whole. To address this concern, I analyzed the age distribution of the sample using 5-year bin widths and center points at ages divisible by five, such as 35, 40, and 45. The responders were from all age groups, from 35 years to 65 years, with a trend toward more responders in the younger age group, similar to the age distribution of the Alumni Fellowship Group. I did not find any evidence of a preponderance of responders from the more experienced or from the recently trained.

Fig. 2. Experts' opinions about tumor probability for various parameters. Parameters are sorted ascending probability from top to bottom. The horizontal axis shows probability on a logarithmic scale. The vertical lines labeled ABR and MRI show the experts' median threshold criterion for ordering ABR tests and MRI, a risk of 1/1,000 and 1/100, respectively. PTA 1010, a patient with a pure-tone average (PTA) of 10 dB hearing level in each ear; Incidence, annual incidence in the expert's geographic area; PTA 1020, PTA of 10 dB in one ear and 20 dB in the other; Word 60~2, word score of 60% in one ear and 52% In the other; Low SNHL, low frequency sensorineural hearing loss in one ear; Tinnitus, tinnitus in one ear; PTA 10~0, PTA of 10 dB in one ear and 30 dB in the other; Word 60 44, word score of 60% in one ear and 44% in the other; Word 60~6, word score of 60% in one ear and 36% in the other; PTA 10~0, PTA of 10 dB in one ear and 50 dB in the other; Word 60~, word score of 60% in one ear and 8% in the other; Telephone, acquired hearing loss in one ear that caused the patient to use the other ear for listening on the telephone; Sudden HL, sudden sensorineural hearing loss in one ear, regardless of responsiveness to steroid therapy; Progressive HL, documented progressive unilateral hearing loss; PTA 1090, PTA of 10 dB in one ear and 90 dB in the other. NF2, patient has a parent with neurofibromatosis type 2.

To further assess representativeness, I sampled the 56 non-responders. A second questionnaire that was both shortened (five questions) and simplified was sent out with a check for ten dollars. Thirty-one of the 56 responded. Twenty-two of these 31 initial non-responders returned the check either with or without the completed simplified questionnaire. Answers from the group of 31 did not differ significantly from the corresponding answers from the initial responders (ANOVA, F statistic 0.946, p = 0.331 or greater for each answer). Likely causes for failure to return the first questionnaire were too many questions (20) and questions asking for opinions in quantitative terms, a task to which many likely were unaccustomed. Data were analyzed with a personal computer, database, and spreadsheet and graphics software.

Results

Incidence of Acoustic Tumor

The experts were asked, "How many people per 100,000 residents will have an acoustic tumor diagnosed this year in your geographical area?" The median answer was 6.5 tumors per 100,000 persons, with a range of 1 to 100 (Fig. 1). This range of answers was large, differing by a factor of 100. To get an idea of the confidence of the answers, I asked the experts to give low and high estimates as first and third quartiles. The range for the low estimate was 0.5 to 75 with a median of 2.0. The range for the high estimate was 1.5 to 125 with a median of 12.5. Figure 2 summarizes the experts' estimates of the probability of the existence of an acoustic tumor given a variety of conditions. Each is addressed as follows.

Sudden Sensorineural Hearing Loss

The experts were asked, "What is the chance that a person with a sudden sensorineural hearing loss will have an acoustic tumor in the involved ear?" The median answer was 1%, which is consistent with published figure of 0.8%. The experts answered that the risk for acoustic tumor was still 1% if the hearing loss responded to systemic steroids.

Neurofibromatosis Type 2

The experts were asked, "What is the chance that a patient who has a parent with neurofibromatosis 2 will have bilateral acoustic tumors during the patient's lifetime?" The true answer is 0.50.4 The median expert answer was 0.50.

Asymmetric Sensory Hearing Loss

The experts were asked, "What is the chance that a patient seen for the first time with no previous audiograms and no difference in hearing between ears will have an acoustic tumor?" The median score was 0.002%. What if the patient had a 10 dB difference in hearing between ears? The median response was 0.075%. Apparently, the experts gave considerable importance to a 10 dB difference between ears. They increased the risk for acoustic tumor about 40-fold compared with patients with no difference between ears. What if the patient had a 20 dB difference in hearing between ears? The experts answered that a 20 dB difference between ears further increased the risk of acoustic tumor to a median of 0.75%. What if the patient had a 40 dB difference in hearing between ears? This additional hearing loss increased the risk for tumor to a median of 1.0%. What if the patient had an 80 dB difference in hearing between ears? This additional hearing loss increased the risk for tumor to a median response of 3.7%.

Asymmetric Word Recognition Scores

The experts were asked, "What is the chance that a patient seen for the first time with no previous audiograms, symmetrical pure-tone thresholds and word recognition scores for NU-6 words delivered in 25 word taped lists who achieved a maximum score of 60% in the better ear and a maximum score of 52% in the poorer ear will have an acoustic tumor?" The 8% difference between 60% and 52% was not statistically significant for an individual patient (p = 0.39, Fisher's exact test). The median expert opinion was that the risk was 0.15% for this patient. We designed this question to crosscheck with the case of a symmetric audiogram, no difference between ears (expert answer was 0.002%).

What if the maximum word recognition score in the poorer ear is 44%? The 16% difference between ears in this question was not statistically significant (p = 0.20, Fisher's exact test). The experts answered that this patient had a 0.8% likelihood of having an acoustic tumor.

What if the maximum word recognition score in the poorer ear is 36%? This 24% difference between the better ear and the poorer ear approached statistical significance (p = 0.078, Fisher's exact test, and 95% critical differences of 36 and 84 (ref. 6). The experts answered that the risk for acoustic tumor was 0.8%, similar to that for patients with a 16% difference between ears.

What if the maximum word recognition score in the poorer ear is 8%? The 52% difference between ears was significant for an individual subject (p = 0.00010, Fisher's exact test). The experts gave this patient a 1.0% probability of having an acoustic tumor.

Threshold for Recommending Auditory Brain Stem Response Tests and Magnetic Resonance Imaging

The experts were asked, "How much risk of acoustic tumor should the patient have before an ABR or a MRI should be recommended?" The mean answer for ABR testing was 0.1 % and for MRI was 1.0%.

The Cost of Diagnosing Each Acoustic Tumor

The experts were asked, "How much should society be willing to spend for every acoustic tumor diagnosed (the cost of diagnosing one tumor plus the cost of testing all the false positive cases needed to diagnose one tumor)?" The mean answer was $25,000, with a range from $2,000 to $800,000.

Unilateral Tinnitus

The experts were asked, "What is the chance that a patient with unilateral tinnitus will have an acoustic tumor in the involved ear?" The median response was 0.5%.

Unilateral Distortion

The experts were asked, "What is the chance that a patient with unilateral distortion on the telephone, but normal audiogram, will have an acoustic tumor in the involved ear?" The median response was 1.3%.

Progressive Hearing Loss

The experts were asked, "What is the chance that a patient will have an acoustic tumor if they have a documented progressive unilateral hearing loss over the course of two years?" The median response was 3%.

Low-Frequency Hearing Loss

The experts were asked, "What is the chance that a patient will have an acoustic tumor if they have fluctuating low frequency sensorineural hearing loss?" The purpose of this question was to address an earlier finding that showed that hearing at 250 and 500 Hz was much less sensitive to the effects of acoustic tumors than higher frequencies. (ref. 3) The median response was 0.5%.

Discussion

Figure 2 summarizes the results. Clinicians may use these data as a guideline for ordering diagnostic tests when a patient has the signs or symptoms listed in the figure. This is the only currently available source of risk data for many of the listed parameters. However, systematic bias affects any expert opinion and causes the opinion to vary from the true answer. (ref. 7) Known types of bias include calibration errors, value-induced bias, and anchoring and adjustment errors.

Calibration

Calibration is the ability to assign a probability that a patient has a certain characteristic that agrees with the observed frequency of the characteristic. (ref. 3) Calibration bias accounts for a general tendency to overestimate when true likelihood is low and to underestimate when true likelihood is high. For example, the reported incidence of tumor is low, 0.8 per 100,000 persons. However, the median estimate given by the experts in the study, 6.5 per 100,000 persons, was an overestimate of the incidence by an order of magnitude. In the experts' defense, the quoted incidence was reported in 1984, before the widespread use of high-resolution MRI. The ability of MRI to depict tumors as small as 2 mm may increase the reported incidence. However, I am unaware of any published data that would support an incidence as high as 6.5 per 100,000 persons.

Value-Induced Bias

Value-induced bias occurs when a clinician erroneously links the probability of an event to the perceived clinical importance of the event. (ref. 9) In situations in which the clinical evidence suggests that two outcomes are equally likely, clinicians may view the more serious outcome as more likely than the less serious outcome. For example, the experts in the study chose a strategy of ABR screening of patients at 0.1% risk for having an acoustic tumor. The consequence of setting a low threshold is that it would cost at least $370,000 for every tumor diagnosed ($250,000 in charges assumed for ABR tests on 1000 patients at $250 per ABR test and $120,000 in charges for MRI for the 100 patients with positive ABR test results at $1200 per imaging session, given 10% false-positive ABR results). The cost would be higher than $370,000 if the ABR charge exceeded $250, the MRI charge exceeded $1200, or the false-positive rate for ABR testing exceeded 10%. Even if MRI charges were $500, low in the range of charges, the cost would still be $300,000 per tumor diagnosed. The experts underestimated the high cost of their ABR strategy by an order of magnitude, proposing that expenditures should be limited to $25,000 per acoustic tumor diagnosed. Value-induced bias may have swayed the experts, leading them to choose a low threshold because the clinical importance of not missing the diagnosis of acoustic tumor is high. Further, calibration bias may have influenced the experts to underestimate the societal cost per diagnosis because the monetary cost is remarkably high.

Anchoring and Adjustment

Clinicians may use anchoring and adjustment for probability estimation. In this method, a clinician chooses an anchored value for the probability of an event and adjusts this value according to the individuating features of the specific patient. (ref. 10) In clinical settings, clinicians may derive the anchored value from their subjective impressions of their clinical experience or from population-based data about the prevalence of a specific disease or the likelihood of a particular outcome. Clinicians then derive patient-specific adjustments to this anchored value from information about an individual patient's clinical features. Errors in estimating the anchor point or errors in making patient-specific adjustments, or both, may cause predictive inaccuracy. Errors in anchor point estimation may arise from choosing an inappropriate population from which to estimate disease base rates, from inexperience, or from neglecting base-rate information altogether.

Errors of anchoring and adjustment probably occurred in the present study. Of the low estimate (25th percentile) results, only two of 15 of the experts estimated the incidence of acoustic tumor less than 1 per 100,000 persons. This implies that the experts stated confidence intervals that were too narrow and reflected more certainty than was justified by their knowledge of the incidence of acoustic tumors. This bias is common to both naive and sophisticated judges and is attributable to anchoring. (ref. 7) One possible explanation is that clinicians perceive the incidence of acoustic tumor as being relatively high because the consequences of having an acoustic tumor are great. Another explanation is that in the process of anchoring, one has the tendency to overestimate small probabilities.

Clinical Application

A limitation of probability-based management protocols is that bias can cause physicians to mis-assign probabilities of disease to patients in such a way that they choose a course of action that is neither cost saving nor effective. If we were to recommend ABR testing and MRI when the risk for tumor exceeds a certain threshold, such a protocol would result in overuse of ABR testing and MRI if physicians overestimated risk for tumor. This is a legitimate concern because of the effects of systematic bias that limited the ability of the experts in the study to assess risk for acoustic tumor. The findings of this study illustrate that practice parameters should not require clinicians, not even expert ones, to judge the risks for acoustic tumor. Instead, parameters should provide guidelines by example to inform clinicians about findings that have enough risk to warrant ordering a diagnostic test.

Examples of findings supported by objective data that merit ordering a diagnostic test include the following: family history of neurofibromatosis type 2, sudden sensorineural hearing loss, PTA difference of 10 dB or greater, and word recognition score differences between ears that exceed the Thorton and Raffin 95% critical differences. (ref. 2-4, 6) Examples of findings that warrant a diagnostic test based on the experts' opinions but not documented objectively include the following (Fig. 2): unilateral low frequency hearing loss, unilateral tinnitus, unilateral distortion while using the telephone, sudden sensorineural hearing loss whether or not steroid responsive, and progressive hearing loss documented over 2 years.

The foregoing are relative indications for ordering diagnostic tests. The ultimate decision should be tempered by the clinician's judgment given the unique set of circumstances presented by each patient. Given rapidly changing medical technology, the clinician should also be given a free hand in choosing the appropriate diagnostic tests.

The expert group was representative of physicians trained at the House Ear Clinic. The group may not be representative of any other group, not even the population of all neurotologists. With the small sample size, I was unable to address the effect of variation within the group. Variables not studied include the following: the number of years in practice, whether the expert was a neurotologist or otologist, and whether the practice setting was academic, private, or managed care. These variables influence individual clinicians, but as yet the effect is not known.

Summary

Expert opinion showed that many parameters typically associated with acoustic tumor carry enough perceived risk for tumor to warrant further diagnostic tests, specifically ABR testing and MRI. The expert opinion displayed systematic bias that clinicians must consider if they apply the opinion to clinical decision making. Clinicians may legitimately use the opinions as guidelines, but the opinions are inappropriate as rules to be used by insurance companies or lawyers to decide between the right and wrong way to practice medicine.

I wish to thank Karen I. Berliner, PhD, for her support as a statistician and colleague. I also wish to thank Loren J. Bartels, MD, Derald E. Brackmann, MD, Robert A. Dobie, MD, Robert A. Goldenberg, MD, Jack S. Pulec, MD, Clough Shelton, MD, and David F. Wilson, MD, for their critical reviews of a draft of this paper before it was submitted for publication.

References

 1. Tos M, Thomsen J. Epidemiology of acoustic neurinomas. J Laryngol Otol 1984;98:68592.

2. Shaia FT, Sheehy JL. Sudden sensorineural hearing impairment a report of 1220 cases. Laryngoscope 1976;86:38998.

3. Mangham CA. Hearing threshold difference between ears and risk of acoustic tumor. Otolaryngol Head Neck Surg 1991; 105:8147.

4. Martuza RL, Eldridge R. Neurofibromatosis 2. N Engl J Med 1988;318:6848.

5. Dalkey NC. The Delphi method: an experimental study of group opinion. Memorandum RM5888PR. Santa Monica, Calif.: The Rand Corp; June 1969. p. 179.

6. Thornton AR, Raffin MJM. Speech-discrimination scores modeled as a binomial variable. J Speech Hear Res 1978;21:507 18.

7. Kahneman D. Slovic P, Tversky A, editors. Judgment under uncertainty: heuristics and biases. Cambridge: Cambridge University Press, 1982.

8. Sehulman KA, Esearee JJ, Eisenberg JM, Hershey JC, Young MJ, McCarthy DM, et al. Assessing physician's estimates of the probability of coronary artery disease: the influence of patient characteristics. Med Decis Making 1992;12:10914.

9. Wallsten TS. Physician and medical student bias in evaluating diagnostic information. Med Decis Making 1981;2:14564.

10. Poses RM, Cebul RD, Collins M, Fager SS. The accuracy of experienced physicians probability estimates for patients with sore throats. JAMA 1985;254:92592.

From the Seattle Ear Clinic.

Supported in part by a grant from the Alumni Fellowship Group of the House Ear Institute.

Reprint requests: Charles A. Mangham, MD, MS, Seattle Ear Clinic, 600 Broadway, Suite 340, Seattle, WA 981223571.

Copyright (c) 1997 by the American Academy of Otolaryngology Head and Neck Surgery Foundation, Inc.

01945998/97/$5.00 + 0 23/1/76275

• Acoustic tumor facts
• Diagnosis
• Treatment
  - Microsurgery
  - Observation
  - Radiation
• Complications
• NIH Consensus Statement on treatment

• Abstracts of acoustic tumor papers (leads to 3 full papers)
• ANA USA patient support
• Washington patient support
• Internet search
• Pamphlets for patients

• Contact the clinic
• Address, phone and fax
• Dr. Charles Mangham, Jr.
• Lodging and transportation
• Travel directions
• Weather
• About this website