| Japanese Journal of Clinical Oncology | Pages |
The Efficacy and Limitations of Repeated Slide Conferences for Improving Interobserver Agreement When Judging Nuclear Atypia of Breast Cancer
Introduction
Materials And Methods
Slide Conferences
Statistical Analysis
Results
Interobserver Agreement for Nuclear Atypia
Intraobserver Reproducibility of Nuclear Atypia Judgment
Discussion
Appendix
Acknowledgments
References
The Efficacy and Limitations of Repeated Slide Conferences for Improving Interobserver Agreement When Judging Nuclear Atypia of Breast Cancer
Methods: In order to standardize the nuclear atypia criteria, five slide conferences were held. A total of 57 observers assigned nuclear atypia scores to 119 breast carcinomas that were presented using a slide projector or a TV monitor and discussed their histological findings. The percentage interobserver agreements per tumor and per conference and [kappa] value per conference were estimated and compared among the conferences. The percentage intraobserver reproducibility per tumor between the last two conferences was compared with the percentage interobserver agreement for 20 tumors.The [kappa] value was also calculated for each of 27 observers to evaluate scoring reproducibility.
Results: The percentage interobserver agreement per conference was constant (75-78%) throughout the five meetings and the rate of tumors with >80% agreement per tumor became higher in later conferences. The [kappa] value was 0.42, 0.25, 0.42, 0.51 and 0.50 for the first, second, third, fourth and fifth conferences, respectively. The tumors with a lower percentage interobserver agreement also had a lower percentage intraobserver reproducibility and such scoring variations were attributed to the intermediate nature of the degree of tumor atypia. In 26 of 27 observers, intraobserver agreement for 20 tumors was estimated from the [kappa] value to range from moderate to almost perfect.
Conclusion: We concluded that the repeated slide conferences conducted by the pathology section were an effective means of standardizing the subjective histopathological criteria used to assess tumors. However, the achievement of a good scoring agreement would be difficult for tumors with an intermediate degree of atypia.
INTRODUCTION
In 1995, a large-scale project to conduct a multi-institutional protocol study named the National Surgical Adjuvant Study of Breast Cancer (NSAS-BC) was started in Japan with the aim of comparing the efficacies of two therapeutic regimens for breast cancer patients without axillary lymph node metastases but who were at high risk of recurrence (1). Before entry into this randomized study, patients with node-negative breast cancer had to be classified into high- and low-risk groups, according to histopathological criteria, at each participating hospital. We organized the NSAS-BC pathology section, which comprised pathologists who were going to participate in this protocol study, in order to establish histological criteria, standardize the central criteria among these pathologists and monitor the quality of the study. We established histological criteria for identifying patients with higher risk node-negative breast carcinomas in Japan by examining the grades of nuclear atypia and mitotic counts (1,2). The next problem was to minimize interobserver and intraobserver variations in the assessment of the subjective criteria.
The histological parameters of the primary cancer, including the histological grade of atypia, nuclear atypia and number of mitotic figures, are powerful prognostic indicators for patients with and without axillary lymph node metastasis (3-7). However, it has been argued repeatedly that interobserver variation and poor intraobserver reproducibility are major weaknesses when evaluating these parameters (8-12). Although there are many pros and cons to this argument (8-15), recent reports accept the value of these histological parameters and now practical procedures to reduce inter- and intraobserver variation, e.g. how to evaluate nuclear atypia and mitotic counts and how to modify the mitotic counts according to the microscope's properties, are being implemented (13-15). One of the best ways of reducing subjective variations in criteria would be repetitive co-examination of tumor tissues in identical microscopic fields by observers using the same microscope and to reach agreement about the scores after discussion of each case. However, it is difficult for a large number of observers to examine whole tumor tissue specimens simultaneously. Therefore, instead, we held repeated slide conferences where several microscopic fields of each tumor tissue were shown to observers using a slide projector or a TV monitor.
In this study, we assessed whether such slide conferences improved interobserver agreement for the criteria of nuclear atypia by percentage agreement and kappa ([kappa]) statistics and tried to reveal the efficacy and limitations of such repeated slide conferences for standardizing nuclear atypia criteria.
MATERIALS AND METHODS
Slide Conferences
Five slide conferences were held in 1996-97 for the purpose of criteria standardization, maintenance of agreement level and data acquisition (Table 1). A total of 51 collaborating pathologists or surgeons, who are or had been involved in routine diagnostic procedures for breast tumors at the collaborating hospitals, attended the conferences (see Appendix). The criteria for scoring nuclear atypia that were established and approved by the pathology section are shown in Table 2 (2). At the slide conferences, color photomicrograph slides of breast cancers showing various degrees of nuclear atypia were presented using a slide projector or a TV monitor. Overall, 119 invasive ductal carcinomas were examined by 57 observers, 13 of whom participated in all five meetings, two in four, 12 in three, 17 in two and 13 in one (mean 2.7 times).
Table 1.
| Slide conference | No. of observers | No. of tumors | Materials |
| First (March 1996) | 23 | 12 | Color slides |
| Second (June 1996) | 23 | 14 | Color slides |
| Third (September 1996) | 32 | 39 | TV monitor |
| Fourth (December 1996) | 39 | 34 | Color slides |
| Fifth (September 1997) | 39 | 20 | Color slides |
Table 2.
| 1 point | Nuclei of uniform size and shape. The nuclei are not hyperchromatic or may be hyperchromatic with evenly dispersed chromatin or with finely granular chromatin without clumping |
| 2 points | Between scores 1 and 3 |
| 3 points | Pleomorphic nuclei of various sizes showing hyperchromatism with coarse and irregular distribution often associated with large nucleoli |
At the first and second conferences, photomicrographs of tissue specimens of node-negative breast carcinomas resected at the National Cancer Center Hospital, Tokyo, were presented. At the third, fourth and fifth meetings, cases were presented using color photomicrograph slides and glass slides of the tissue section specimens of node-negative primary breast cancer which had been resected at the collaborating hospitals.
At the first, second, fourth and fifth conferences, two to four photomicrograph slides per tumor, usually taken through ×20 and ×40 objective lenses, were presented using a slide projector. At the third conference, the microscopic fields of several representative parts of each tumor, usually at original magnifications of ×100, ×200 and ×400, were shown on a TV monitor.
At the conferences, all the observers assigned nuclear atypia scores to the tumors and the attending members discussed the judgments made by one of them. The data acquired were analyzed to evaluate the interobserver variation and the result were shown to observers at the next conference, when the photomicrograph slides were presented again. At the fifth conference, 27 observers who had participated in both the fourth and fifth meetings were also enrolled in the intraobserver reproducibility study.
Statistical Analysis
The modal nuclear atypia score for each tumor panel in each conference was acquired and `percentage interobserver agreement per tumor' was calculated by
| [(number of observations that gave the modal score)/(number of all observations)] × 100 |
The `percentage interobserver agreement per conference' was defined as the average of the percentage interobserver agreements of all the tumors that were examined at a conference. The level of the percentage agreements per tumor and per conference was compared among the conferences.
The degree of interobserver agreement for nuclear atypia was also tested using the generalized [kappa] test for more than two observers (16-18). In accordance with the criteria of Landis and Koch (19), the [kappa] statistics were divided into several scales to determine the strength of agreement.
The `percentage intraobserver reproducibility per tumor' was given by
| [(number of observers who assigned an identical score to the tumor at both conferences)/(number of all observers)] × 100 |
The correlation between percentage interobserver agreement and percentage intraobserver reproducibility was compared for 20 tumors using the correlation coefficient test.
To estimate the scoring reproducibility of individual observers, the concordance between the initial and secondary scores for the 20 tumors assigned by each observer was also analyzed using the pairwise [kappa] test.
RESULTS
Interobserver Agreement for Nuclear Atypia
The distribution of modal nuclear atypia scores in tumors examined at each conference is presented in Table 3. The percentage interobserver agreement per conference was 76, 75, 75, 78 and 77% in the first, second, third, fourth and fifth meetings, respectively. An interobserver agreement per tumor of >80% was acquired in 33, 36, 36, 53 and 55% of tumors examined at the first, second, third, fourth and fifth conferences, respectively (4).
Table 3
| Slide conference | No. of tumors | No. of observers | % Agreement per conference |
[kappa] value | |||
| Total | Modal score | ||||||
| NA1 | NA2 | NA3 | |||||
| First | 12 | 2 | 8 | 2 | 23 | 76 | 0.42 |
| Second | 14 | 0 | 6 | 8 | 23 | 75 | 0.25 |
| Third | 39 | 4 | 23 | 13 | 32 | 75 | 0.42 |
| Fourth | 34 | 5 | 20 | 11 | 39 | 78 | 0.51 |
| Fifth | 20 | 5 | 11 | 4 | 39 | 77 | 0.50 |
Table 4.
| % Interobserver agreement per tumor | No. of tumors (%) | ||||
| First | Second | Third | Fourth | Fifth | |
| (n = 23) | (n = 23) | (n = 32) | (n = 39) | (n = 39) | |
| 91-100 | 3 4 | 1 5 | 8 14 | 9 18 | 6 11 |
| 81-90 | 1 (33%) | 4 (36%) | 6 (36%) | 9 (53%) | 5 (55%) |
| 71-80 | 3 | 4 | 11 | 4 | 4 |
| 61-70 | 3 | 5 | 7 | 6 | 4 |
| 51-60 | 2 | 0 | 4 | 4 | 1 |
| 41-50 | 0 | 0 | 3 | 2 | 0 |
| Total | 12 | 14 | 39 | 34 | 20 |
n = Number of observers.
The generalized [kappa] value was 0.42, 0.25, 0.42, 0.51 and 0.50 at each meeting, respectively. The degree of interobserver agreement in tumors as a whole was adjudged moderate at the first, third, fourth and fifth conferences and adjudged fair at the second conference. Complete agreement was reached for no, no, two, four and two tumors at each conference. The distribution of the nuclear atypia score assigned to 34 tumors by 39 observers at the fourth conference is shown in Table 5. Fig. Figure 1. Photomicrographs of invasive ductal carcinoma cases for which nuclear atypia scores were consistent among observers at the slide conference. (A) Score 1 tumor (tumor No. 2 in Table 5, National Osaka Hospital). Cancer cells show nuclei uniform in size and shape and their nuclei have an inconspicuous chromatin pattern. (B) Score 2 tumor (tumor No. 14 in Table 5, National Cancer Center). Cancer cells show nuclei relatively irregular in shape. A fine granular chromatin pattern and an uneven chromatin distribution are seen. (C) Score 3 tumor (tumor No. 30 in Table 5, Saitama Cancer Center). Nuclear polymorphism is conspicuous. Chromatin granule condensation shows a coarse and irregular distribution, with a vesicular appearance of the nuclei. Large nucleoli are also evident. H & E stain, ×40 objective lens. Table 5 
Tumor
No. of observations
Modal NA
% Interobserver
agreement per tumor
NA1
NA2
NA3
1/IDC
36
3
0
1
92
2/IDC
35
4
0
1
90
3/IDC
33
5
1
1
85
4/IDC
31
8
0
1
79
5/IDC
18
21
0
1
54
6/IDC
18
18
3
1/2
46
7/IDC
13
26
0
2
67
8/IDC
12
27
0
2
69
9/IDC
11
27
1
2
69
10/IDC
9
28
2
2
72
11/IDC
10
25
4
2
64
12/IDC
4
34
1
2
87
13/IDC
3
35
1
2
90
14/IDC
0
38
1
2
97
15/IDC
0
37
2
2
95
16/IDC
0
36
3
2
92
17/IDC
1
34
4
2
87
18/IDC
3
29
7
2
74
19/IDC
2
31
6
2
79
20/IDC
1
32
6
2
82
21/IDC
2
26
11
2
67
22/IDC
1
27
11
2
69
23/IDC
1
23
15
2
59
24/IDC
1
19
19
2/3
49
25/IDC
1
18
20
3
51
26/IDC
0
18
21
3
54
27/IDC
0
7
32
3
82
28/IDC
0
6
33
3
85
29/IDC
0
6
33
3
85
30/IDC
0
3
36
3
92
31/IDC
0
0
39
3
100
32/IDC
0
0
39
3
100
33/IDC
0
0
39
3
100
34/IDC
0
0
39
3
100
Intraobserver Reproducibility of Nuclear Atypia Judgment
The intraobserver reproducibility per tumor varied from 48 to 100% (Table 6). When the percentage interobserver agreement per tumor among 39 pathologists at the fourth meeting was compared with the percentage intraobserver reproducibility per tumor for the 20 tumors, there was a nearly significant linear correlation (r = 0.702, t = 4.182, p = 0.05). Thus, tumors with a lower percentage interobserver agreement also had a lower percentage intraobserver reproducibility.
Table 6
| Tumor | % Interobserver per tumor | % Intraobserver reproducibility per tumor |
| 1 | 92 (36/39)* | 89 (24/27)[dagger] |
| 2 | 90 (35/39) | 96 (26/27) |
| 3 | 85 (33/39) | 81 (22/27) |
| 4 | 79 (31/39) | 48 (13/27) |
| 5 | 54 (21/39) | 63 (17/27) |
| 6 | 46 (18/39) | 70 (19/27) |
| 7 | 67 (26/39) | 78 (21/27) |
| 8 | 69 (27/39) | 70 (19/27) |
| 15 | 95 (37/39) | 96 (26/27) |
| 16 | 92 (36/39) | 81 (22/27) |
| 19 | 79 (31/39) | 85 (23/27) |
| 20 | 82 (32/39) | 74 (20/27) |
| 22 | 69 (27/39) | 59 (16/27) |
| 23 | 59 (23/39) | 56 (15/27) |
| 24 | 49 (19/39) | 74 (20/27) |
| 26 | 54 (21/39) | 70 (19/27) |
| 28 | 85 (33/39) | 70 (19/27) |
| 31 | 100 (39/39) | 100 (27/27) |
| 32 | 100 (39/39) | 100 (27/27) |
| 33 | 100 (39/39) | 100 (27/27) |
[dagger]Number of observers who assigned an identical score to the tumor at both conferences/total number of observers.
The [kappa] values for the reproducibility of 26 of the 27 observers varied from 0.40 to 0.92 and the reproducibility was adjudged moderate, substantial or almost perfect (Table 7). The [kappa] value for the remaining observer was 0.22 and the reproducibility was adjudged fair. The average [kappa] value for the 27 observers was 0.62.
Table 7.
| [kappa] value | Degree of agreement | No. of observers |
| 0.901-1 | Almost perfect | 1 |
| 0.801-0.900 | Almost perfect | 2 |
| 0.701-0.800 | Substantial | 6 |
| 0.601-0.700 | Substantial | 7 |
| 0.501-0.600 | Moderate | 9 |
| 0.401-0.500 | Moderate | 1 |
| 0.301-0.400 | Fair | 0 |
| 0.201-0.300 | Fair | 1 |
| 0-0.200 | Slight | 0 |
DISCUSSION
The NSAS-BC study protocol stipulates that the nuclear grade must be judged by pathologists who are involved in routine diagnosis at each participating hospital. Therefore, the establishment of central criteria and standardization of the subjective criteria among the pathologists appeared to be important for maintaining the reliability of this study. Recent studies have suggested strongly that interobserver agreement in breast cancer grading can be achieved when a grading scheme with specific guidelines is used (13-15).
As a consequence of repeated slide conferences, we were able to judge that the level of agreement for the nuclear atypia scores among observers was maintained at a level of >75% and at a moderate degree from analysis of percentage agreement and the generalized [kappa], and that the level of agreement tended to increase as the number of repeated slide conferences increased. For almost all the observers, the level of intraobserver reproducibility was also kept at a moderate, substantial or almost perfect level at the last two conferences. These results indicate that the trial conducted to standardize the subjective criteria among the observers achieved satisfactory results. Further beneficial effects of such an exercise may be achieved by showing images of typical and non-typical tumors with a color atlas which has been edited by this pathology section and have been provided to every collaborating pathologist. Such trials may also be effective for standardizing assessments of mitotic counts, structural atypia and other histological and cytological parameters.
It was shown that excellent agreement of histological grade tended to be more likely for extremely low-grade and extremely high-grade tumors (13). Therefore, it is a novel finding that almost complete interobserver agreement was reached readily not only for tumors with typical nuclear atypia scores of 1 and 3 but also for those with a typical nuclear atypia score of 2, when standardization of the subjective criteria among observers was achieved. However, there were many tumors (e.g. Nos 5, 6 and 23-25 in Table 5) for which observations giving scores of 1 and 2 or 2 and 3 were almost equal in number. These tumors were considered to be of an intermediate nature with scores between 1 and 2 or between 2 and 3 and resulted in a lower level of both interobserver agreement and intraobserver reproducibility.
Even if standardization is complete among observers for typical tumors with nuclear atypia scores of 1, 2 and 3, scoring variations for these intermediate tumors may occur owing to the categorization system itself that reduces the numerous degrees of nuclear atypia to only three. In routine histological diagnosis, pathologists frequently state that the final nuclear atypia score for a tumor is `2.5' or that for another tumor it is 2 but close to 3. If a tumor is of an intermediate nature (e.g. a score of `2.5'), half of the observers will assign a score of 2 and half a score of 3. Such a case would be adjudged as having a concordance of only 50% or less in terms of percentage agreement or [kappa]. However, this result does not indicate severe interobserver disagreement but a mean natural normal distribution of scoring for such an intermediate case. Without specific guidelines, it would not be easy to reach better agreement for scoring such intermediate tumors.
The three-score system for nuclear atypia could be modified in two ways to overcome its inherent limitations. One way would be to increase the numbers of score categories to five or nine. For example, five score categories comprising 1, 1-2, 2, 2-3 and 3 and comprising 1, 1 > 2, 1 = 2, 1 < 2, 2, 2 > 3, 2 = 3, 2 < 3 and 3. However, such detailed categorization does not seem very practical unless the criteria are well established and much better standardization of the subjective criteria than that for three-score systems is achieved. The other practical method is to assign an equivocal tumor with a score between 1 and 2 or 2 and 3 a score of 2 automatically. In this case, we would need to establish schematic guidelines for differentiating typical tumors with scores 1, 2 or 3 from equivocal tumors in order to achieve good agreement.
In conclusion, repeated well conducted slide conferences by an organized pathology section are an effective means of standardizing the subjective histopathological criteria used to assess cancers. Our attempt to standardize subjective criteria indicated that this method should increase the accuracy of large-scale protocol studies based on histopathological criteria. However, diversity in the degree of nuclear atypia among tumors and the limitation of the grading system itself would prevent perfect interobserver agreement and intraobserver reproducibility.
Appendix
NSAS-BC pathology section: Katsushige Yamashiro (National Sapporo Hospital, Sapporo), Noriko Kimura (Tohoku University Hospital, Sendai), Masao Hori and Masayuki Itabashi (Ibaraki Prefectural Central Hospital, Tomobe, Ibaraki), Kazuei Hoshi and Seiji Igarashi (Tochigi Cancer Center, Utsunomiya), Shiro Sugihara and Akira Ogawa (Gunma Cancer Center, Ota, Gunma), Tetsunari Oyama and Kenji Kashiwabara (Gunma University, Maebashi), Takahiro Hasebe (National Cancer Center East, Kashiwa, Chiba), Noboru Onoda (National Tokyo Medical Center, Tokyo), Fumihiko Tanaka (Teikyo University Hospital, Tokyo), Hitoshi Niino and Kiyoshi Saito (International Medical Center of Japan, Tokyo), Kaori Kameyama and Hitoshi Sugiura (Keio University Hospital, Tokyo), Hiroshi Imamura and Motohiko Aiba (Tokyo Women's Medical College Second Hospital, Tokyo), Hiroshi Kaise (Tokyo Medical College Hospital, Tokyo), Masafumi Suzuki and Makio Kawakami (Tokyo Jikei Medical College Hospital, Tokyo), Mikihito Ito, Kimihiko Masuda and Toshimasa Uekusa (St. Luke's International Hospital, Tokyo), Shinobu Umemura and Yutaka Tsutsumi (Tokai University Hospital, Isehara, Kanagawa), Yoichi Kameda (Kanagawa Cancer Center, Yokohama), Keiichi Honma (Niigata Cancer Center, Niigata), Takachika Ozawa (Hamamatsu Medical Center, Hamamatsu), Kazumori Arai and Hiroyuki Muro (Shizuoka General Hospital, Shizuoka), Hiroshi Kobayashi (Seirei Hamamatsu General Hospital, Hamamatsu), Shigeo Nakamura (Aichi Cancer Center Hospital, Nagoya), Shuh Ichihara (National Nagoya Hospital, Nagoya), Hiroshi Sugiura (Kasugai City Hosital, Kasugai, Aichi), Yoshiharu Nara (Yokkaichi City Hospital, Yokkaichi, Mie), Tsutomu Kasugai and Masayuki Mano (Osaka Medical Center for Cancer and Cardiovascular Diseases, Osaka), Masashi Takeda and Kunimitsu Kawahara (National Osaka Hospital, Osaka), Masahiko Tsujimoto (Osaka Police Hospital, Osaka), Hideo Morino (Labor Welfare Cooporation, Kansai Rosai Hospital, Amagasaki, Hyogo), Toshiya Hojo and Hiroki Inui (Kinki University Hospital, Osaka-Sayama), Terumasa Sashikata (Hyogo Adult Disease Center, Akashi, Hyogo), Takuya Moriya (Kawasaki Medical College Kawasaki Hospital, Okayama), Tetsumi Yamane (National Kure Hospital, Kure, Hiroshima), Shozo Ohsumi and Koichi Mandai (Shikoku Cancer Center, Matsuyama), Nobuya Sano and Kansei Komaki (Tokushima University Hospital, Tokushima) and Hideharu Fujii (National Nagasaki Central Hospital, Ohmura, Nagasaki).
Acknowledgments
This study was supported in part by a grant for post-marketing studies of anticancer drugs from the Human Science Foundation entrusted from the Ministry of Health and Welfare, Japan and by a sponsorship grant from Taiho Pharmaceutical Co., Tokyo, Japan. We thank Ms Momoko Kitahara and Ms Teruko Takarabe for excellent assistance with this work.
References
This article has been cited by other articles:
This page is run by Oxford University Press, Great Clarendon Street, Oxford OX2 6DP, as part of the OUP Journals
Comments and feedback: www-admin{at}oup.co.uk
Last modification: 4 Mar 1999
Copyright© 1999 Foundation for Promotion of Cancer Research, 1999.
CiteULike 
Connotea 
Del.icio.us What's this?
T. Watanabe, M. Sano, S. Takashima, T. Kitaya, Y. Tokuda, M. Yoshimoto, N. Kohno, K. Nakagami, H. Iwata, K. Shimozuma, et al.
Oral Uracil and Tegafur Compared With Classic Cyclophosphamide, Methotrexate, Fluorouracil As Postoperative Chemotherapy in Patients With Node-Negative, High-Risk Breast Cancer: National Surgical Adjuvant Study for Breast Cancer 01 Trial
J. Clin. Oncol.,
March 20, 2009;
27(9):
1368 - 1374.
[Abstract]
[Full Text]
[PDF]
A Paradiso, I O Ellis, F A Zito, E Marubini, S Pizzamiglio, and P Verderio
Short- and long-term effects of a training session on pathologists' performance: the INQAT experience for histological grading in breast cancer
J. Clin. Pathol.,
March 1, 2009;
62(3):
279 - 281.
[Abstract]
[Full Text]
[PDF]
S. Kyoda, S. Kinoshita, H. Takeyama, K. Uchida, and T. Morikawa
HER-2 Protein Overexpression in Metastatic Breast Carcinoma Found at Autopsy
Jpn. J. Clin. Oncol.,
November 1, 2008;
38(11):
743 - 747.
[Abstract]
[Full Text]
[PDF]
E. Iwamoto, T. Fukutomi, and S. Akashi-Tanaka
Validation and Problems of St-Gallen Recommendations of Adjuvant Therapy for Node-negative Invasive Breast Cancer in Japanese Patients
Jpn. J. Clin. Oncol.,
June 1, 2001;
31(6):
259 - 262.
[Abstract]
[Full Text]
[PDF]
This Article 
Abstract
Alert me when this article is cited
Alert me if a correction is posted
Services
Email this article to a friend
Similar articles in this journal
Similar articles in ISI Web of Science
Similar articles in PubMed
Alert me to new issues of the journal
Add to My Personal Archive
Download to citation manager
Search for citing articles in:
ISI Web of Science (8)
Request Permissions
Google Scholar
Articles by Tsuda, H
Articles by Watanabe, T
Search for Related Content
PubMed
PubMed Citation
Articles by Tsuda, H
Articles by Watanabe, T
Social Bookmarking
What's this?