(Radiographics. 2001;21:535-547.)
© RSNA, 2001
Image Content Extraction: Application to MR Images of the Brain1
Usha Sinha, PhD,
Anthony Ton, MS,
Amy Yaghmai, MD,
Ricky K. Taira, PhD and
Hooshang Kangarloo, MD
1 From the Department of Radiological Sciences, UCLA School of Medicine, 924 Westwood Blvd, Suite 420, Los Angeles, CA 90024 (U.S., A.Y., R.K.T., H.K.); the School of Medicine, New York University, New York, NY (A.T.); and the Department of Radiological Sciences, Children’s Hospital and Medical Center, Seattle, Wash (R.K.T.). Presented as an infoRAD exhibit at the 1999 RSNA scientific assembly. Received April 6, 2000; revision requested May 31 and received August 16; accepted August 29. Address correspondence to U.S. (e-mail: usinha@itmedicine.net).
 |
Abstract
|
|---|
A system for automatically extracting image content features was developed that combines registration to a labeled atlas with natural language processing of free-text radiology reports. The system was then tested with T1-weighted, spoiled gradient-echo magnetic resonance (MR) imaging studies of the brain performed in nine patients. The locations of 599 structures were visually assessed by an experienced radiologist and compared with the locations indicated by automated output. The in-plane accuracy of the contours was subjectively evaluated as either good, moderate, or poor. The criterion for classifying a structure as correctly located was that 90% or more of all the images containing the structure had to be correctly identified. For 98% of the structures, the images identified by the automated algorithm agreed with those identified by the radiologist, and in 83% of cases, image contours showed a good in-plane overlap. The results of this validation study demonstrate that this combination of registration and natural language processing is accurate in identifying relevant images from brain MR imaging studies. However, the range of applicability of this technique has yet to be determined by applying the technique to a large number of studies.
Index Terms: Computers • Computers, multimedia • Radiology and radiologists, design of radiological facilities • Radiology reporting systems
|
Introduction
|
|---|
Radiologic studies often produce a large number of images, only a small subset of which contain relevant information. For example, a typical brain magnetic resonance (MR) imaging study may comprise 150–250 images, only 15% of which are associated with the findings dictated in the radiology report. The remaining images often are not needed by referring physicians or are not useful for inclusion within teaching files. At radiology workstations, users must often spend time navigating through irrelevant images with slow or clumsy interfaces. The quality of a digital library retrieval system would most likely improve if only relevant images from a large study were retrieved.
Current retrieval schemes from most picture archiving and communication systems (PACS) are based on structured patient demographic and procedural information (1),(2). These data are automatically extracted from the Digital Imaging and Communication in Medicine (DICOM) headers and included in indexes that facilitate retrieval of the relevant images. These retrieval systems support queries such as, "Retrieve the brain MR image obtained on 2/11/2000 in patient John Doe." Recent developments in image prefetching that incorporate medical or radiologic knowledge and image retrieval heuristics facilitate intelligent retrieval but are still not based on content (3). We are interested in developing a retrieval system that can support queries containing image content constraints such as, "Retrieve selected images from a brain MR imaging study that contain the sylvian fissure." Relevant structure identification would be a simple process if an automated segmentation program were available that worked accurately on a wide variety of images (4). However, this is not as yet a clinical reality.
The lack of a universally applicable image analysis module has led several groups to suggest a knowledge-based approach that can customize the image analysis task with object-centered hierarchic planning (5),(6). This modular, object-oriented image analysis approach has been used for the indexing, storage, and retrieval of medical images by content (7),(8). One such system, known as I2C, has also been successfully incorporated into a World Wide Web–based architecture and provides services that can potentially create widely accessible digital medical image libraries (9). Several groups have proposed indexes for content-based image retrieval based on features such as shape, color, texture, and sketches (10)–(13).
Indexes composed of specific imaging features provide a way to limit the search space, but significant semantic information may get lost in the process. The Photobook project is an effort to represent the perceptually salient aspects of stored images with semantics-preserving image compression, which reduces images to a small set of perceptually significant coefficients (14). Another approach to content-based retrieval involves combining the information from photo captions with image analysis output to search for images of human faces (15). Recently, a method was proposed that combines semantic indexing with the Unified Medical Language System (UMLS) metathesaurus and knowledge-based image analysis (16). An extension of the N-word combination method for searching free-text documents has been applied to image indexing. With this system, a global signature is computed that represents the content of each image in an abstract sense (17).
We have developed a novel method of content-based image retrieval with specific application to brain MR imaging studies. Our approach combines relevant structured information derived from free-text radiology reports by a natural language processor (NLP) with an automated registration algorithm that maps patient images to a labeled brain atlas. Registration to the atlas results in automatic identification of images containing a given structure of interest as well as anatomic labeling of the entire study data set. In this article, we discuss and illustrate this innovative, content-based image retrieval system.
|
Materials and Methods
|
|---|
In this section, we first describe a general image content extraction architecture that can be adapted for different organs, imaging modalities, and report input types (structured or free text). The architectural issues include the types of processing modules and knowledge to be used and the types of intermediate data to be generated for a system whose primary goal is to identify relevant structures on images from radiologic studies. The architecture clarifies the following procedures: (a) automated image volume registration to a labeled digital atlas, (b) identification and refinement of organ-specific contours, (c) computation of quantitative imaging features such as texture, heterogeneity, shape, and size, and (d) automatic extraction and structuring of dictated findings from the corresponding radiology report. In the latter part of this section, we describe a specific implementation of this architecture for brain MR imaging studies with the goal of locating the images that contain the structures described in the corresponding free-text radiology report.
General Architecture
The overall processing architecture is shown in Figure 1. The particulars of registration, segmentation, and feature extraction algorithms are not specified because the general architecture does not depend on these details. The choice of a specific algorithm depends on various factors including imaging modality, anatomy, and patient demographics. Algorithms (and associated parameters) can either be selected by the user or set by the system (based on predefined expert rules). It is also possible to incorporate machine-learning techniques for automated selection of different image-processing algorithms.
Study Identifier.
The Study Identifier extracts information from the DICOM headers in the studies, including patient demographics, imaging modality, anatomy, reason for study, image geometry, and acquisition parameters. This information is used to select or customize the atlas and image segmentation algorithms. In a multiseries study, this module also identifies the most appropriate image series for registration using the criteria of maximum resolution and anatomy coverage.
Atlas Selector.
The Atlas Selector selects the most closely matched anatomic atlas based on patient age, disease condition, and imaging modality. For example, the optimum brain atlas for a 70-year-old patient with Alzheimer disease is a dementia brain atlas. The table that allows mapping of relevant parameters in a given patient to a particular atlas is created by experts and stored within the knowledge base, where the metadata used to describe the atlas include the relevant anatomic organ or organ part description, age and disease condition applicability, and the imaging modality used to construct the atlas. Currently, a growing number of multimodality, multiorgan atlases for different patient conditions are available (eg, labeled atlases of the brain and, more recently, of the brainstem) and include disease- and age-specific data sets (18),(19). Registration of multimodality images has been reported for a number of non–brain structures as well (20)–(22). The knowledge base should also include a table that allows mapping of common anatomic terms to controlled vocabulary codes (eg, Systematized Nomenclature of Human and Veterinary Medicine[SNOMED], UMLS) to support mapping to anatomic descriptions found in the corresponding radiology reports.
Registration Selector.
It is important that the registration algorithm be completely automated, accurate, and robust. A number of efficient two-dimensional and three-dimensional (3D) registration algorithms have been reported that require little or no manual intervention (23). The knowledge base should include expert-defined rules (or more sophisticated statistical classifiers) related to the choice of registration algorithm and optimal processing parameters for the current imaging study. These rules should be based on published validation studies of registration algorithms or determined by performing registration experiments on typical clinical studies.
Contour Generator.
The Contour Generator performs registration using the algorithm (and parameters) selected by the Registration Selector. The output of the registration algorithm is a matrix that defines the spatial transformation between the patient image data set and the atlas space. This matrix is used to estimate the locations of structures on the patient images from the contours defined for the atlas. Because the image acquisition geometry is known for each series, the transformation matrix can also be used to identify the relevant structures on all the images in a given study. In addition, the Contour Generator identifies the images associated with the location descriptions of findings extracted from the radiology report.
Natural Language Processor.
The input for the NLP is the free-text radiology report. The NLP then outputs a set of structured frames. Each frame contains the important attribute descriptions reported for a particular finding. Attributes typically found in radiology reports include size, location, extent, severity, certainty, and interpretation. The attributes of a finding are expressed in terms of a set of logical relations. A logical relation includes a predicate (eg, "has-size") and three arguments: head, relation, and value. Thus, a logical relation might read: has-size ("mass" [head], EQUALS [relation], "5 cm" [value]). The attribute required for initial localization of the findings for image content extraction is termed the location attribute. The NLP extracts location information from the text and expresses this information as a set of logical relations. For example, the phrase "mass in right anterior lobe" creates the following logical relations: (a) has-location (mass, in, lobe), (b) has-direction (lobe, EQUALS, right), and (c) has-direction (lobe, EQUALS, anterior).
Term Mapper.
The input to the Term Mapper is either the set of location-specific logical relations from the NLP or anatomic terms entered from a structured data entry module. The Term Mapper accesses a dictionary that defines a list of synonyms for each atlas term. This synonym dictionary represents an attempt to standardize the anatomic descriptions found in free-text reports. Alternatively, all structures listed in the atlas could be coded to a standardized terminology such as SNOMED or UMLS (24),(25). The structured findings would then be mapped directly to the standardized terminology. An important aspect of including this standardized terminology coding step is that it permits relaxation from smaller to larger parent structures from the hierarchy of anatomic listings. This feature is useful if the atlas does not contain a structure mentioned in the report because the general location in the image data set can still be determined if a "parent structure" can be identified in the atlas.
Image Segmenter.
Accurate registration between the atlas and patient image data sets should result in correctly defined structures. However, this cannot be physically realized due to the biologic variability in brain morphology. The models generated by the Contour Generator could potentially be regarded as "rough estimate" contours. These contours could then serve as the initial seed contours for segmentation algorithms. For instance, elastically deformable models have recently been applied successfully in medical image segmentation problems (26). Furthermore, non–anatomy-type attributes (eg, shape, size, extent, intensity [density]) related to a finding described in the report and extracted by the NLP could provide an additional context for knowledge-based segmentation methods.
Feature Extractor.
The Feature Extractor calculates quantitative features for each finding segmented by the Image Segmenter. These features include descriptors of shape, size, border characteristics, and homogeneity, which can then serve as more precise image content indexes complementing the traditional qualitative descriptions found in radiology reports.
Specific Implementation: Brain MR Imaging Studies
The Study Identifier is implemented with a JAVA program that reads the DICOM headers associated with the imaging study. This provides structured information related to each series in the study including imaging plane (axial, coronal, sagittal, oblique), sequence type (two-dimensional or 3D), section thickness, section spacing, number of sections, and echo time and repetition time. The Study Identifier has been programmed to prefer a 3D patient image data set for atlas registration owing to the higher resolution of this sequence.
A single brain atlas was used in our study. The atlas was derived from an averaged high-contrast template based on 3D volume data obtained in nine patients with a T1-weighted, spoiled gradient-echo MR imaging sequence with a 256 x 256 matrix (18). Sixty-eight structures have been defined as 3D contours in the stereotactic coordinates of the template. The coordinates were projected on a section-by-section basis to facilitate ready identification and display of the atlas images containing different anatomic structures. The brain atlas was processed (scaled) so that the gray-scale-intensity range of the atlas images was comparable to that of the acquired patient images. Atlas images were resectioned along imaging planes (eg, the brain atlas images were resectioned along the axial planes to register a set of axially acquired images).
In addition, we used a single registration algorithm: the Automated Image Registration (AIR) program, version 3.0, developed by Woods et al (27),(28). This algorithm was chosen because of our requirement of minimal user intervention. It is available at http://bishopw.loni.ucla.edu/AIR3 and is based on the matching of voxel intensities. The algorithm has been tested extensively for accuracy using both inter- and intrasubject registration (27),(28). The spatial transformation model used in our study is the 3D affine linear model. This model has 12 parameters and allows rotation, translation, and independent rescaling along any arbitrary set of axes. The 12-parameter affine model has been validated in previous intersubject MR imaging studies as optimal in terms of registration accuracy and computation time (28). The algorithm uses an index denoted as the cost function to quantitatively measure how well the images are registered. For the registrations reported in this article (ie, from patient image data set to atlas) a cost function called the scaled least-squared difference was used, wherein the difference between the resampled image and the reference image is computed at each voxel and the square of this difference is averaged across voxels to generate the least-squared cost function. The scaled least-squared difference image allows global intensity rescaling of the images relative to each other and was chosen to account for possible intensity scale differences between the atlas and patient image data sets.
The input to the Contour Generator is the spatial transformation matrix outputted by the registration algorithm, and the Contour Generator uses this matrix to transform coordinates of structures defined in the atlas space into coordinates within the patient image data set. The coordinates are then projected on a section-by-section basis (x and y coordinates collated at the same z value) to facilitate ready identification of the patient images containing the anatomic structure of interest. The Contour Generator then creates a file containing a list of coordinates for structures of interest associated with that imaging study.
The NLP we used is that described in a study by Taira et al (29). The module was installed on a UNIX server and processed all radiology reports offline. The architecture of the statistical NLP module includes (a) a structural analyzer that isolates sections as well as individual sentences within the radiology report, (b) a lexical analyzer that looks up semantic and syntactic features of words in a lexicon, (c) a parser that determines word dependencies, (d) a semantic interpreter that interprets the word links created by the parser and outputs a set of logical relations, and (e) a discourse processor that determines whether a finding is new or is a referent to a finding from a previous term. The parser and semantic interpreter use statistical methods, and the structural analyzer uses a maximum entropy classifier. Each structured finding has a head (keyword) and modifiers and is represented as a frame, which is then stored in an individual file. For example, in the location term "right anterior lobe," "lobe" is the keyword and the modifiers are "right" and "anterior."
The Term Mapper includes a synonym glossary that lists structures identified in the brain atlas. The synonym glossary was created manually with input from several sources including radiologists, textbooks, and electronic brain glossaries. As outlined in the previous section, the output of the NLP for an anatomic location is a term that includes a keyword and one or more modifiers. The atlas structure corresponding to an NLP term is determined with a string search of the synonym glossary. The query for this string search is constructed from the NLP term, which normally consists of two to four words. All the words are included in the first query. Failure to locate this combination of words results in a search with a new query. New queries are formed by successively dropping modifiers from the origi-nal term. This process continues until a specific combination results in a successful retrieval. In forming these queries, the keyword is never dropped because doing so may result in retrievals that are not related to the primary concept.
The Image Segmenter currently defines only a bounded area for each structure identified by the Contour Generator. The boundary box is definedso that it encloses all the contour points and has a margin whose width is determined by the size of the structure. This bounded area serves to localize the structure with a high degree of certainty. The Feature Extractor determines the size (in voxels) and the center of mass of each structure.
The imaging studies used in our investigation were chosen to represent typical clinical MR im-aging examinations of the brain and included seven 3D T1-weighted and two 3D T2-weighted data sets. In this preliminary study, only axially acquired images were chosen for evaluation. No postprocessing of the images was performed prior to registration. Figures 2 and 3 show the user interface for selected MR imaging studies.
View larger version (103K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 2. User interface for image registration. Top panel shows the atlas image (left), original patient image (center), and resectioned patient image (right). The original patient image data were reconstructed along atlas views to allow visualization of the results of the registration algorithm. Good alignment of the resectioned patient image with the atlas image is readily seen. The section number in the original image data set matches the anatomy within the atlas. Thus, the atlas and patient images were clearly not aligned prior to registration. The bottom panel displays the free-text report (left) and the list of available atlas structures with defined contours (right). The user can display contour models of the structures on all three image data sets.
|
|
View larger version (100K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 3. User interface for image registration. Top panel shows the contours of the structures contained in the report as seen on the atlas image (left), the original patient image (center), and the resectioned patient image (right). Below each image is a text field showing the sections in that data set containing the structure of interest. The table (lower left panel) shows the findings and attributes outputted by the NLP and identifies one structure from the free-text report in Figure 2 as "ventricle" modified by "lateral." The synonym dictionary (Term Mapper) helps map this anatomic description to the corresponding term or terms in the atlas. Because the radiology report did not specify any details beyond "lateral ventricle," the Term Mapper found seven structures in the atlas that included this term in the description. The lower right panel displays some (but not all) of the structures identified by the NLP because the atlas presently contains only a limited number of defined structures, resulting in incomplete labeling of the patient images. In contrast, all terms in the atlas that matched the location attribute "lateral ventricles" identified by the NLP are listed.
|
|
The locations of 599 structures were visually assessed by an experienced radiologist (A.Y.) and compared with the locations indicated by auto-mated output. The in-plane accuracy of the con-tours was subjectively evaluated on a 3-point scale as either good (Fig 4), moderate (Fig 5), or poor (Fig 6). The criterion for classifying a structure as correctly located was that 90% or more of all the images containing the structure had to be correctly identified.
View larger version (159K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 5a. Contours rated as "moderate" by the radiologist. MR images show the automated contour for the olfactory sulcus (arrows in a) and for part of the lateral ventricles (arrows in b).
|
|
View larger version (165K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 5b. Contours rated as "moderate" by the radiologist. MR images show the automated contour for the olfactory sulcus (arrows in a) and for part of the lateral ventricles (arrows in b).
|
|
View larger version (148K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 6a. Contours rated as "poor" by the radiologist. Brain MR images show structures that have been incorrectly identified by the automated algorithm, including the corpus callosum (a) and part of the lateral ventricles (b). Arrow indicates the contours generated by the automated algorithm for these two structures.
|
|
View larger version (158K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 6b. Contours rated as "poor" by the radiologist. Brain MR images show structures that have been incorrectly identified by the automated algorithm, including the corpus callosum (a) and part of the lateral ventricles (b). Arrow indicates the contours generated by the automated algorithm for these two structures.
|
|
|
Results
|
|---|
The Table qualitatively summarizes the accuracy of automated registration as evaluated with visual assessment. For 98% of the structures, the images identified by the automated algorithm agreed with those identified by the radiologist, and in 83% of cases, image contours showed a good in-plane overlap. Intraobserver variation in contour classification varied by less than 1% (ie, less than four of 599 contours were rated differently in independent trials). In an independent study, the accuracy of the NLP module was found to have recall and precision values of 90% and 89%, respectively (29). The effectiveness of the system in correctly identifying brain structures as small as1–2 mm is shown in Figure 7. As shown in Figure 7d, even a relatively small structure such as the temporal horn of the lateral ventricle (2–3 mm) is correctly located by the automated algorithm. Automated in-plane localization of this structure was rated by the radiologist as "good" in 80% of cases and as "moderate" in 20%.
View larger version (177K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 7a. Structures identified accurately by the automated algorithm. MR images show the atrium and occipital horn of the lateral ventricles (medial surface) (arrow in a), the hippocampal formation (inferior surface) (arrow in b), the olfactory sulcus (arrow in c), and the temporal horn of the lateral ventricles (arrow in d). The smallest structure that was contoured accurately in most patients was the temporal horn of the lateral ventricle (2-3 mm).
|
|
View larger version (154K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 7b. Structures identified accurately by the automated algorithm. MR images show the atrium and occipital horn of the lateral ventricles (medial surface) (arrow in a), the hippocampal formation (inferior surface) (arrow in b), the olfactory sulcus (arrow in c), and the temporal horn of the lateral ventricles (arrow in d). The smallest structure that was contoured accurately in most patients was the temporal horn of the lateral ventricle (2-3 mm).
|
|
View larger version (185K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 7c. Structures identified accurately by the automated algorithm. MR images show the atrium and occipital horn of the lateral ventricles (medial surface) (arrow in a), the hippocampal formation (inferior surface) (arrow in b), the olfactory sulcus (arrow in c), and the temporal horn of the lateral ventricles (arrow in d). The smallest structure that was contoured accurately in most patients was the temporal horn of the lateral ventricle (2-3 mm).
|
|
View larger version (161K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 7d. Structures identified accurately by the automated algorithm. MR images show the atrium and occipital horn of the lateral ventricles (medial surface) (arrow in a), the hippocampal formation (inferior surface) (arrow in b), the olfactory sulcus (arrow in c), and the temporal horn of the lateral ventricles (arrow in d). The smallest structure that was contoured accurately in most patients was the temporal horn of the lateral ventricle (2-3 mm).
|
|
|
Discussion
|
|---|
An important requirement of an image content extraction system used to process a large number of studies is that it be entirely automated. The AIR registration algorithm typically works best with brain images from which non–brain struc-tures have been removed. Several automated image-processing algorithms have been developed for removing the skull component from brain images. However, given the variability of clinical data, often there is still a need for considerable manual intervention to process all images correctly. The nine image data sets processed in this pilot study show that for the purpose of image identification, the AIR registration algorithm worked effectively without the need for removing the skull components from the image. Our initial experience with sagittal and coronal images indicates that they may require some preprocessing to eliminate non–brain structures for accurate registration. This is due to the greater amount of non–brain structures visible in these two orientations compared with the axial view. The registration algorithm takes 15–20 minutes to complete; however, this will not impact the user because the algorithm runs in the background, triggered by the arrival of images to our PACS database (Fig 1).
The accuracy of the AIR registration algorithm that has been reported in MR imaging validation studies ranges from 75 to 150 µm for intrasubject registration and from 0.6 to 1.9 mm for intersubject registration. Intersubject or subject-to-atlas registrations have a lower accuracy due to the variability in brain morphology across the population. Intrasubject registrations are typically performed to monitor changes in serial studies (eg, tumor growth on a set of images acquired at different times). In these studies, registration accuracy must be high to record small changes in tumor size. In our study, however, the primary focus of the registration was to locate the images that contained the relevant structures identified by the NLP. Thus, the requirements for the accuracy of the registration algorithm are relaxed because registration accuracy has to be only within a section thickness.
As noted earlier, we found excellent correlation between identification of structures with the automatic algorithm and that by visual assessment. Although our sample size was limited, the studies were chosen at random from brain MR imaging examinations and represent typical clinical studies. Thus, we strongly believe that a more comprehensive evaluation will show our method to be a robust technique for localizing brain structures on patient images. Furthermore, the accurate in-plane localization of more than 80% of the structures indicates that these represent a good "rough estimate" for contour-refining segmentation algorithms. A more appropriate evaluation would be to compare the accuracy of identification of relevant images and structures by the image content extractor versus identification by a radiologist. This evaluation was not performed due to the availability of only a limited number of structures in the brain atlas.
An analysis of the test results shows that several factors can lead to incorrect identification of structures on patient images. For images of brains that do not show gross morphologic distortions from the brain atlas of healthy volunteers, the major sources of error are biologic variability and inconsistent data acquisition. The latter includes studies acquired under low-resolution conditions (low in-plane resolution, thick or noncontiguous sections) or with limited coverage of the brain. However, inaccuracies in the identification of structural contours due to these factors are small compared with those in brains that have been morphologically altered by an underlying disease process. Fairly substantial changes in brain structure including distortion and displacement can be caused by tumor (>1 cm) or edema. In these cases, contours of structures adjacent to the tumor cannot be accurately determined with registration to a "normal" brain atlas (Fig 8). We are currently investigating techniques for modeling the effects of a tumor (given its size, shape, and location) on adjacent anatomy. A disease-specific brain atlas will have to be used for registering data obtained in patients with diseases such as Alzheimer disease that cause gross morphologic changes (18). The availability of such atlases could extend the scope of this technique to include patients with these conditions.
View larger version (176K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 8a. Effect of a large tumor with edema on structure identification. (a) MR image shows a tumor distorting the normal anatomy and causing incorrect location of the contours of the lateral ventricles (arrow). (b) MR image demonstrates normal findings with accurate identification of structures (arrow).
|
|
View larger version (175K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 8b. Effect of a large tumor with edema on structure identification. (a) MR image shows a tumor distorting the normal anatomy and causing incorrect location of the contours of the lateral ventricles (arrow). (b) MR image demonstrates normal findings with accurate identification of structures (arrow).
|
|
As stated earlier, the accuracy of the image indexing algorithm proposed in this article depends on the accuracy of the NLP algorithm. Preliminary experiments evaluating various aspects of the algorithm have been performed on chest radiology reports with the following results: parser recall, 89%; precision performance, 90%; semantic interpreter recall, 79%; and precision, 87% (29). When the system is fine-tuned for neuroradiology reports, similar recall and precision values are expected. Thus, we anticipate that most of the findings in a radiology report will be correctly identified by the NLP. We have also developed an editing interface that incorporates speech recognition and NLP features to enable radiologists to view and edit the structured findings as part of their workflow. This system could potentially be integrated with the image content extraction module, thereby eliminating structuring errors from the NLP.
The term mapping process also introduces a source of error in that an incorrect synonym match will result in the wrong structure being identified. At present, the interface allows the user to manually add synonyms to the atlas nomenclature. As more cases are processed and verified, the synonym dictionary will expand to include a fairly comprehensive list. A simple string search is used in the current system to map the structured findings to those contained in the atlas. Our future plans include incorporating more sophisticated term mapping programs. The incorporation of a standardized nomenclature such as SNOMED RT (24) to classify atlas structures will allow navigation in a hierarchic arrangement of brain structures.
The implementation and validation studies described in this article were limited to MR imaging studies of the brain but can readily be extended to include brain images from other modalities such as computed tomography (CT) and positron emission tomography (PET). This extensibility depends on one main factor: accuracy of registration of CT and PET brain data to an MR imaging atlas. Multimodality registration combining MR imaging with CT or PET has been validated, and accuracy usually approximates the resolution of the lower-resolution image for intermodality registrations (28). Accuracy will be slightly lower for multimodality registration of patient image data to the atlas but should be sufficient for localization of structures. Extension to non–brain structures may not be quite as straightforward as extension to other modalities but is still feasible depending on the availability of accurate registration algorithms developed for specific body parts.
|
Conclusions
|
|---|
Our limited validation study helps confirm that image content can be characterized by combining registration to a labeled atlas and natural language processing of free-text reports. The range of applicability of this technique must still be tested by applying the technique to a large number of studies (200–300 studies). However, limitations can be inferred even from preliminary validation studies: The technique will fail whenever patient anatomy is altered significantly from the atlas anatomy (eg, by an underlying disease process). The use of disease-specific atlases may alleviate the problem to some extent but may not be extensible to all patient conditions (eg, neoplastic involvement). Alternate strategies such as querying by a reference image or by image attributes (10–14) may have to be used for these studies.
|
Acknowledgments
|
|---|
The authors thank Dr Paul Thompson for the brain atlas used in this study and for his participation in extensive discussions.
|
Footnotes
|
|---|
Abbreviations: AIR = Automated Image Registration, DICOM = Digital Imaging and Communication in Medicine, NLP = natural language processor, PACS = picture archiving and communication system, PET = positron emission tomography, SNOMED = Systematized Nomenclature of Human and Veterinary Medicine, 3D = three-dimensional, UMLS = Unified Medical Language System
|
References
|
|---|
-
Ladder H, Poppel BM, Bakker AR. Prefetching: PACS image management optimization using HIS/RIS information. Proc SPIE Med Imaging: PACS Design and Evaluation 1991; :227-233.
-
Wilson DL, Smith D, Rice B. Intelligent prefetch strategies for historical images in a large PACS. Proc SPIE Medical Imaging: PACS Design and Evaluation 1994; :112-123.
-
Hu PJ, Wei CP, Sheng ORL. A knowledge-based patient image prefetching system: design, evaluation and management. Top Health Inform Manage 1999; 20:42-57.
-
Pavlidis T. Why progress in machine vision is so slow. Patt Recogn Lett 1992; 13:221-225.
-
Kulikowski CA, Gong L, Mezrich RS. Knowledge-based medical image analysis and representation for integrating content definition with the radiological report. Methods Inf Med 1995; 34:96-103.[Medline]
-
Chameroy V, Aubry F, Di Paola R. A knowledge-based approach to medical image processing monitoring. Proc SPIE Med Imaging 1995; 2434:705-717.
-
Orphanoudakis SC, Chronaki C, Kostomanolakis S. I2C: a system for the indexing, storage, and retrieval of medical images by content. Med Inform 1994; 19:109-122.
-
Aubry F, Chameroy V, Di Paola R. A medical image object-oriented database with image processing and automatic reorganization capabilities. Comput Med Imaging Graph 1996; 20:315-331.[Medline]
-
Chronaki CE, Zabulis X, Orphanoudakis SC. I2Cnet medical image annotation service. Med Inform 1997; 22:337-347.
-
Niblack W, Barder R, Equitz W, et al. The QBIC project: querying images by content using color, texture and shape. Proc SPIE Medical Imaging: Storage and Retrieval for Image and Video Databases 1993; 1908:173-181.
-
Gupta A, Weymouth T, Jain R. Semantic queries with pictures: the VIMSYS model. Proceedings of the 17th International Conference on Very Large Databases 1991; :69-79.
-
Robinson GP, Tagare HD, Duncan JS, Jaffe CC. Medical image collection indexing: shape-based retrieval using KD-trees. Comput Med Imaging Graph 1996; 20:209-217.[Medline]
-
Tagare HD, Jaffe CC, Duncan J. Medical image databases: a content-based retrieval approach. J Am Med Inform Assoc 1997; 4:184-198.[Abstract/Free Full Text]
-
Pentland A, Picard RW, Sclaroff S. Photobook: tools for content-based manipulation of image databases. Proc SPIE: Image and Information Systems 1994; 2185:34-47.
-
Srihari RK. Automatic indexing and content-based retrieval of captioned images. IEEE Computer 1995; 28:49-56.
-
Lowe HJ, Antipov I, Hersh W, Smith CA, Mailhot M. Automated semantic indexing of imaging reports to support retrieval of medical images in the multimedia electronic medical record. Methods Inf Med 1999; 38:303-307.[Medline]
-
Kelly P, Cannon M. Experience with CANDID: comparison algorithm for navigating digital image databases. Proc SPIE: Image and Information Systems 1994; 2368:64-74.
-
Thompson PM, Woods RP, Mega MS, Toga AW. Mathematical/computational challenges in creating deformable and probabilistic atlases of the human brain. Hum Brain Mapp 2000; 9:81-92.[Medline]
-
Niemann K, van den Boom R, Haeselbarth K, Afshar F. A brainstem stereotactic atlas in a three-dimensional magnetic resonance imaging navigation system: first experiences with atlas-to-patient registration. J Neurosurg 1999; 90:891-901.[Medline]
-
Hristov DH, Fallone BG. A gray-level image alignment algorithm for registration of portal images and digitally reconstructed radiographs. Med Phys 1996; 23:75-84.[Medline]
-
Wong TZ, Connolly LP, Treves ST. Registration of three-dimensional magnetic resonance and radionuclide skeletal images. Clin Nucl Med 1999; 24:859-863.[Medline]
-
Rueckert D, Sonoda LI, Hayes C, Hill DL, Leach MO, Hawkes DJ. Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging 1999; 18:712-721.[Medline]
-
Hawkes DJ. Algorithms for radiological image registration and their clinical application. J Anat 1998; 193:347-361.
-
Spackman KA, Campbell KE, Cote RA. SNOMED RT: a reference terminology for health care. AMIA Fall Symp Proc 1997; :640-644.
-
Humphreys BL, Lindberg DAB, Schoolman HM, Barnett GO. The Unified Medical Language Medical System: an informatics research collaboration. J Am Med Inform Assoc 1998; 5:1-11.[Abstract/Free Full Text]
-
Tagare HD. Deformable 2D template matching using orthogonal curves. IEEE Trans Med Imaging 1997; 16:108-117.[Medline]
-
Woods RP, Grafton ST, Holmes CJ, Cherry SR, Mazziotta JC. Automated image registration. I. General methods and intrasubject, intramodality validation. J Comput Assist Tomogr 1998; 22:139-152.[Medline]
-
Woods RP, Grafton ST, Watson JD, Sicotte NL, Mazziotta JC. Automated image registration. II. Intersubject validation of linear and nonlinear models. J Comput Assist Tomogr 1998; 22:153-165.[Medline]
-
Taira RK, Soderland SG. A statistical natural language processor for medical reports. AMIA Fall Symp Proc 1999; :970-974.
Top
Abstract
Introduction
