Visual Memory Effects on Intraoperator Study Design
Visual Memory Effects on Intraoperator Study Design
Objectives The objective of this research was to determine test intervals between intraoperator case reviews to minimize the impact of recall.
Methods Three pathologists were presented with a group of 120 slides and subsequently challenged with a study set of 120 slides after 2-week and 4-week intervals. The challenge set consisted of 60 slides seen during the initial review and 60 slides previously unseen within the study. Pathologists rendered a diagnosis for each slide and indicated whether they recalled seeing the slide previously (yes/no).
Results Two weeks after having been shown 60 cases from a challenge set of 120 cases, the pathologists correctly remembered 26, 22, and 24 cases or 40% overall. After 4 weeks, the pathologists correctly recalled 31% of cases previously seen.
Conclusions Pathologists were capable of recalling from memory cases seen previously at 2 and 4 weeks. Recall rates may be sufficiently high to affect intraobserver study design.
The US Food and Drug Administration (FDA) has classified digital whole-slide imaging (WSI) as a class III medical device, thereby requiring stringent validation requirements prior to use as a primary diagnostic modality. The College of American Pathologists (CAP) Pathology and Laboratory Quality Center recently issued a series of 12 guidelines for WSI validation based on a review of literature and the opinion of a panel of pathologists whose expertise included WSI. The panel recommended the use of intraoperator case reviews for the validation process in which the same reviewer examines cases using both WSI and light microscopy. The period between case examinations (ie, washout period) was recommended to be a minimum of 2 weeks but not to exceed 1 year. The washout period minimum limit was determined to be necessary to reduce recall bias, and the 1-year maximum period was recommended to ensure that clinical practice and diagnostic criteria remain similar between case reviews. The washout period was established based on empirical data of previous WSI validation study designs and not on published studies regarding intraobserver recall bias, a fact noted by the panel. The purpose of this study was to investigate the relationship between pathologist visual case recall and the length of time between reviews of the same case by the same operator. Data from this research can be used as a guide to perform valid intraobserver (ie, crossover) pathology image evaluation studies.
CAP reviewers could find few data in the academic literature regarding pathologist ability to remember particular cases, and while the radiology community has a longer history of validating digital diagnostic imaging tools, the effect of recall bias is not well understood by the field. The FDA offers general guidance to reduce recall bias and recommends that sufficient time be allowed between intraoperator reviews of the same image to reduce recall bias, but the definition of sufficient is not given. Despite the paucity of information in the medical imaging literature, the neuroscience literature contains substantial information concerning memory and visual recall that may be relevant in formulating guidelines for pathology WSI studies. A full analysis of memory models is beyond the scope of this study, but studies on the fidelity of long-term memory and retrieval in comparison with short-term memory are germane to address the hypothesis that pathologists would be capable of remembering specific slide cases for more than 2 weeks.
Research performed by Standing demonstrated that humans are capable of retaining and accurately remembering large amounts of visual data. More recent research indicates that while visual recall declines over time, the level of fidelity (ie, accuracy and details) of long-term and visual working memory has similar limits. Visual recall mechanisms operate similarly for simple objects (eg, a cup) and for complex images or scenes (eg, a golf course), and individuals are able to distinguish between previously viewed images from images of the same category that have not been previously viewed (eg, different golf course scenes). Fidelity of long-term memory recall increases when the reviewer has preexisting knowledge of the visual content and is also increased when time is allowed for image encoding (ie, the process of categorizing and committing to memory). These studies demonstrated that if a new image can be accommodated within a framework of similar concepts, the long-term retention in memory is increased.
Abstract and Introduction
Abstract
Objectives The objective of this research was to determine test intervals between intraoperator case reviews to minimize the impact of recall.
Methods Three pathologists were presented with a group of 120 slides and subsequently challenged with a study set of 120 slides after 2-week and 4-week intervals. The challenge set consisted of 60 slides seen during the initial review and 60 slides previously unseen within the study. Pathologists rendered a diagnosis for each slide and indicated whether they recalled seeing the slide previously (yes/no).
Results Two weeks after having been shown 60 cases from a challenge set of 120 cases, the pathologists correctly remembered 26, 22, and 24 cases or 40% overall. After 4 weeks, the pathologists correctly recalled 31% of cases previously seen.
Conclusions Pathologists were capable of recalling from memory cases seen previously at 2 and 4 weeks. Recall rates may be sufficiently high to affect intraobserver study design.
Introduction
The US Food and Drug Administration (FDA) has classified digital whole-slide imaging (WSI) as a class III medical device, thereby requiring stringent validation requirements prior to use as a primary diagnostic modality. The College of American Pathologists (CAP) Pathology and Laboratory Quality Center recently issued a series of 12 guidelines for WSI validation based on a review of literature and the opinion of a panel of pathologists whose expertise included WSI. The panel recommended the use of intraoperator case reviews for the validation process in which the same reviewer examines cases using both WSI and light microscopy. The period between case examinations (ie, washout period) was recommended to be a minimum of 2 weeks but not to exceed 1 year. The washout period minimum limit was determined to be necessary to reduce recall bias, and the 1-year maximum period was recommended to ensure that clinical practice and diagnostic criteria remain similar between case reviews. The washout period was established based on empirical data of previous WSI validation study designs and not on published studies regarding intraobserver recall bias, a fact noted by the panel. The purpose of this study was to investigate the relationship between pathologist visual case recall and the length of time between reviews of the same case by the same operator. Data from this research can be used as a guide to perform valid intraobserver (ie, crossover) pathology image evaluation studies.
CAP reviewers could find few data in the academic literature regarding pathologist ability to remember particular cases, and while the radiology community has a longer history of validating digital diagnostic imaging tools, the effect of recall bias is not well understood by the field. The FDA offers general guidance to reduce recall bias and recommends that sufficient time be allowed between intraoperator reviews of the same image to reduce recall bias, but the definition of sufficient is not given. Despite the paucity of information in the medical imaging literature, the neuroscience literature contains substantial information concerning memory and visual recall that may be relevant in formulating guidelines for pathology WSI studies. A full analysis of memory models is beyond the scope of this study, but studies on the fidelity of long-term memory and retrieval in comparison with short-term memory are germane to address the hypothesis that pathologists would be capable of remembering specific slide cases for more than 2 weeks.
Research performed by Standing demonstrated that humans are capable of retaining and accurately remembering large amounts of visual data. More recent research indicates that while visual recall declines over time, the level of fidelity (ie, accuracy and details) of long-term and visual working memory has similar limits. Visual recall mechanisms operate similarly for simple objects (eg, a cup) and for complex images or scenes (eg, a golf course), and individuals are able to distinguish between previously viewed images from images of the same category that have not been previously viewed (eg, different golf course scenes). Fidelity of long-term memory recall increases when the reviewer has preexisting knowledge of the visual content and is also increased when time is allowed for image encoding (ie, the process of categorizing and committing to memory). These studies demonstrated that if a new image can be accommodated within a framework of similar concepts, the long-term retention in memory is increased.
Source...