(Radiographics. 1999;19:745-764.)
© RSNA, 1999
Three-dimensional Volume Rendering of Spiral CT Data: Theory and Method1
Paul S. Calhoun, BFA,
Brian S. Kuszyk, MD,
David G. Heath, PhD,
Jennifer C. Carley, BS and
Elliot K. Fishman, MD
1 From the Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Hospital, 600 N Wolfe St, Baltimore, MD 21287 (P.S.C., B.S.K., J.C.C., E.K.F., D.G.H.), and HipGraphics, Baltimore, Md (D.G.H.). Presented as an infoRAD exhibit at the 1997 RSNA scientific assembly. Received June 1, 1998; revision requested August 3 and final revision received October 29; accepted October 30. Address reprint requests to E.K.F.
 |
Abstract
|
|---|
Three-dimensional (3D) medical images of computed tomographic (CT) data sets can be generated with a variety of computer algorithms. The three most commonly used techniques are shaded surface display, maximum intensity projection, and, more recently, 3D volume rendering. Implementation of 3D volume rendering involves volume data management, which relates to operations including acquisition, resampling, and editing of the data set; rendering parameters including window width and level, opacity, brightness, and percentage classification; and image display, which comprises techniques such as "fly-through" and "fly-around," multiple-view display, obscured structure and shading depth cues, and kinetic and stereo depth cues. An understanding of both the theory and method of 3D volume rendering is essential for accurate evaluation of the resulting images. Three-dimensional volume rendering is useful in a wide variety of applications but is just now being incorporated into commercially available software packages for medical imaging. Although further research is needed to determine the efficacy of 3D volume rendering in clinical applications, with wider availability and improved cost-to-performance ratios in computing, 3D volume rendering is likely to enjoy widespread acceptance in the medical community.
Index Terms: Computed tomography (CT), image processing, **.12115, **.121172 • Computed tomography (CT), technology, **.12115, **.12117 • Computed tomography (CT), three-dimensional, **.12115, **.12117 • Computed tomography (CT), volume rendering, **.12115, **.12117
|
INTRODUCTION
|
|---|
The rapid development of spiral (helical) computed tomography (CT) has resulted in exciting new applications for CT. One of these applications, three-dimensional (3D) CT with volume rendering, is now a major area of clinical and academic interest. Three-dimensional volume rendering is proving to be far more than just a solution in search of a problem. One of the greatest advantages of spiral CT with 3D volume rendering is that it provides all the necessary information in a single radiologic study (and therefore at the lowest possible price) in cases that previously required two or more studies. Three-dimensional volume rendering generates clinically accurate and immediately available images from the full CT data set without extensive editing. It allows the radiologist and clinician to address specific questions concerning patient care by interactively exploring different aspects of the data set. In contrast to the growing problem of information overload presented by the large acquisition rates of modern scanners, 3D volume rendering has the potential to simplify the standard radiologic study.
Three-dimensional images integrate a series of axial CT sections into a form that is often easier to interpret than the sections themselves and can be made to appear similar to other, more familiar images such as catheter angiograms. The most widely used 3D imaging techniques to date have been shaded surface display (SSD) and maximum intensity projection (MIP) (1–8). Volume rendering has existed since the mid-1980s but has not been widely available commercially until recently. All 3D rendering techniques represent a 3D volume of data in one or more two-dimensional (2D) planes, conveying the spatial relationships inherent in the data with use of visual depth cues. To understand how these techniques work, it may be helpful to think of the volume of data as a cube floating within a computer monitor. The data are organized into a 3D matrix of volume elements (voxels). The screen of the computer monitor is a 2D surface composed of discrete picture elements (pixels). Presenting what is stored in memory (ie, floating within the monitor) on a 2D screen is a challenge, but it is the very problem that 3D reconstruction software has creatively solved. Each 3D rendering technique relies on mathematic formulas to determine for each pixel what portion of the data in memory should be displayed on the screen and how that portion should be weighted to best represent spatial relationships. Voxel selection is usually accomplished by projecting lines (rays) through the data set that correspond to the pixel matrix of the desired 2D image. Differences in the images produced with various 3D rendering techniques are the result of variations in how voxels are selected and weighted.
It will soon be economically infeasible (and perhaps even impossible) to sit at a light box reviewing the hundreds of axial images obtained in a single patient. The development of computer workstations for medical imaging now enables the radiologist and clinician to view CT data sets in a variety of display formats including standard axial images, reconstructed images in any plane, and high-quality 3D images.
In this article, we compare 3D volume rendering of spiral CT data with other rendering techniques (shaded surface display, maximum intensity projection). We present a brief history of 3D volume rendering and discuss the implementation of this promising technology in terms of strategies for volume data management, selection of rendering parameters, and features of data display. In addition, we present recent research on the accuracy of the technique in specific medical applications and discuss a number of current clinical applications of 3D volume rendering of spiral CT data.
|
THREE-DIMENSIONAL RENDERING TECHNIQUES
|
|---|
Three-dimensional rendering could not have been developed without advances in computer hardware, software, and display technology. Progress has been incremental and often limited by the state of the art in any one of these technologies on which development depends. Despite these constraints, SSD and MIP have remained functional by making use of only about 10% of the available CT data and implementing very simple rendering schemes (9), although this compromise limits the accuracy of rendered images. Volume rendering incorporates the entire data set into a 3D image (10–13). Initially, image processing and display was very time consuming: Several hours were required to render an animation loop for viewing. However, recent advances in computer hardware have made volume rendering a practical, interactive technique that allows processing and display to occur in real time (minimum, 5–10 frames/sec) at relatively inexpensive workstations.
The fundamental differences in the way SSD, MIP, and 3D volume rendering process and display medical data have important implications. Although static surface-rendered images of skeletal disease may appear more "3D" than those created with volume rendering, their clinical utility is compromised by an inability to show subcortical detail. MIP images have a tendency to misrepresent anatomic spatial relationships because the projected data do not take spatial location into account. MIP often requires extensive editing to eliminate unwanted data and thus create useful images. Recent studies have reported mixed results with these early rendering techniques. There have been many useful clinical applications for MIP and SSD techniques. However, several investigators have found standard axial or multiplanar images to be more accurate (3,6,8).
We have shown volume rendering to be superior to SSD in demonstrating surgically created femur fractures (14). Moreover, in volume rendering the dynamic range of the image is preserved, enabling display of vessels, soft tissue, and viscera. Relationships between vasculature and viscera can be demonstrated as well as vascular patency, thrombus, or calcification. In 3D volume rendering, various parameters such as surface shading and opacity can help reveal both surface and internal detail and allow the radiologist or clinician to clearly distinguish most disease. We have noted potentially significant interobserver variability in applications in which quantification is critical; however, we have also observed that careful selection of rendering parameters results in high levels of accuracy (15,16). The flexibility and superior image fidelity of volume rendering make it a very useful technique in an increasing number of applications.
Shaded Surface Display
SSD, also known as surface rendering, was the first 3D rendering technique applied to medical data sets. Its early development in the 1970s was a logical extension of new computer graphics and image-processing techniques and innovations in data segmentation (ie, division of a volume into multiple areas or objects [primitives]) and display (17–22). Since that time, the medical and computing communities have worked together to develop new applications of and refinements to existing 3D imaging technologies.
SSD is a process in which apparent surfaces are determined within the volume of data and an image representing the derived surfaces is displayed (Fig 1). Much of the research in this area has focused on how surfaces are determined. Results of this research have also been used effectively in volume rendering for automated segmentation. Simple thresholding is a commonly used technique designed to segment structures of interest for surface rendering. In this technique, each voxel intensity within the data set is determined to be within some user-specified range of attenuation values (eg, bone attenuation). The fidelity of the resulting images to actual anatomy depends in part on the value range selected. More advanced surface generation techniques such as "marching cubes" (23) use simple thresholding to select voxels but also use voxel values to generate surfaces that are placed and oriented more accurately.
View larger version (40K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 1. Diagram illustrates SSD rendering technique. Numbers represent the voxel values for a sample 2D data set. An algorithm is applied to locate a "surface" within the data set at the margin of a region of voxels with intensities ranging from 6 to 9. Standard computer graphics techniques are then used to generate a surface that represents the defined region of voxel values.
|
|
Surface contours are typically modeled as a number of overlapping polygons derived from the boundary of the selected region of interest. A virtual light source is computed for each polygon, and the object is displayed with the resulting surface shading. Multiple overlapping surfaces can be displayed on a single image with the additional implementation of partial opacities. Surface rendering is widely available in commercial CT image processing packages and is used clinically.
Advantages of surface rendering include superior speed and flexibility in image rendering. Many of today's graphics computers are optimized for the display of surface models. Applications in surgical planning (eg, for maxillofacial reconstruction) take advantage of this capability that allows surface models to be interactively repositioned and manipulated. In addition, for purposes of measurement a derived surface can be displayed with subvoxel precision with the "marching cubes" technique mentioned earlier. This capability has positive implications for the visualization of small structures such as cranial sutures. In general, surface-rendered images have the clearest volume depth cues of all 3D images. A common criticism of surface rendering is that the surface is derived from only a small percentage (less than 10% by some estimates) of the available data. In addition, surface rendering is not adequate for the visualization of structures that do not have naturally well-differentiated surfaces.
Maximum Intensity Projection
MIP is a 3D rendering technique that evaluates each voxel along a line from the viewer's eye through the volume of data and selects the maximum voxel value, which is then used as the displayed value (Fig 2). MIP is also widely available in commercial 3D software packages. The clinical utility of MIP has been evaluated extensively, and MIP has proved to be particularly useful in its original application: creating angiographic images from CT and magnetic resonance (MR) imaging data. However, Schreiner et al (24) have shown that different versions of the MIP algorithm can produce very different images. This suggests that 3D rendering algorithms such as MIP and volume rendering should be thought of as families of related image processing techniques rather than as single entities.
View larger version (16K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 2. Diagram illustrates MIP rendering technique. Each voxel along a line from the viewer's eye through the volume of data is evaluated, and the maximum voxel value is selected on the basis of maximum intensity. The resulting displayed value (8) corresponds to this maximum voxel value.
|
|
MIP has a number of related artifacts and shortcomings that must be taken into account to interpret the rendered images properly (Fig 3). The displayed pixel intensity will represent only the material with the highest intensity along the projected ray. A high-intensity material such as calcification will obscure information from intravascular contrast material. This limitation can be partially overcome with use of nonlinear transfer functions or, more practically, through volume editing. Volume editing can take the form of a preprocessing step (eg, section-by-section ["slab"] editing) or an interactive process (eg, "sliding-slab" MIP). Use of the highest intensity value also increases the mean background intensity of the image, in effect selecting the "noisiest" voxels and thereby decreasing the visibility of vessels in enhancing structures such as the kidney and liver. MIP images are typically not displayed with surface shading or other depth cues, which can make assessment of 3D relationships difficult. Also, volume averaging (the effect of finite volume resolution) coupled with the MIP algorithm commonly leads to MIP artifacts. A normal small vessel passing obliquely through a volume may have a "string of beads" appearance because it is only partially represented by voxels along its length. The severity of these artifacts depends on the resampling filters used. Despite its limitations, MIP usually has superior accuracy compared with surface rendering in CT angiography and produces relatively reproducible results with different operators.
View larger version (100K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 3a. Use of volume rendering versus MIP rendering for pulmonary artery mapping. (a) Volume-rendered image clearly depicts pulmonary arteries in the midplane including central vessels. (b) Corresponding MIP image appears to depict more vessels but is actually displaying the entire vascular map of the data set. This is best appreciated by noting the highly opacified superior vena cava and right innominate vein. One limitation of MIP is the inability to clearly define individual structures when they overlap. Interactive volume rendering does not have this limitation.
|
|
View larger version (103K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 3b. Use of volume rendering versus MIP rendering for pulmonary artery mapping. (a) Volume-rendered image clearly depicts pulmonary arteries in the midplane including central vessels. (b) Corresponding MIP image appears to depict more vessels but is actually displaying the entire vascular map of the data set. This is best appreciated by noting the highly opacified superior vena cava and right innominate vein. One limitation of MIP is the inability to clearly define individual structures when they overlap. Interactive volume rendering does not have this limitation.
|
|
Three-dimensional Volume Rendering
As commonly implemented, 3D volume rendering takes the entire volume of data, sums the contributions of each voxel along a line from the viewer's eye through the data set, and displays the resulting composite for each pixel of the display (Fig 4). Incorporation of information from the entire volume can lead to greater fidelity to the data; however, much more powerful computers are required to perform volume rendering at a reasonable speed. Differences in implementation of various volume rendering techniques result in varying quality and utility among applications. Nevertheless, volume rendering is useful in a wide variety of applications ranging from seismic data display to wind tunnel testing and is now being incorporated into commercially available software packages for medical imaging. With wider availability and improved cost-to-performance ratios in computing, volume rendering is likely to enjoy widespread use in the medical community.
View larger version (41K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 4. Diagram illustrates volume rendering technique. The upper portion of the diagram represents a histogram of the voxel values in the data "ray" depicted beneath it. An opacity of 0% and 100% are assigned for values of 5 or lower and 9 or higher, respectively. The resulting intermediate opacities for values 6, 7, and 8 are 25%, 50%, and 75%, respectively. The lower portion of the diagram shows the equation and progressive computational results used to determine weighted summation along the "ray" through the volume. The resulting displayed value (6) is affected by both opacity (as determined in the graph at the top) and the value of underlying voxels.
|
|
|
HISTORY OF 3D VOLUME RENDERING
|
|---|
The first implementation of a volume rendering technique grew out of research done at the Mayo Clinic in the 1970s. In the mid-1980s, advances in image processing hardware and integration of new data manipulation techniques led to parallel developments in volume rendering at the University of North Carolina (marked by the publication of Levoy's work [25]) and at Pixar (San Rafael, Calif) (10). The work at Pixar led to a collaboration with the CT research group at our institution (26), from which our work originated.
The work at Pixar actually derived from the work of computer scientists at LucasFilms in San Rafael. The computer graphics group at LucasFilms was brought together by Ed Catamull, PhD, and Alvey Ray Smith, PhD, who were recruited by George Lucas to expand the world of computer graphics and thereby create more realistic computer-generated images for movies such as Star Wars. Innovations in both hardware and software led to the development of a parallel processing system, the Pixar Image Computer. Volume rendering on this system was developed by development team members Robert Drebin, Pat Hanrahan, and Loren Carpenter. Recent work in the area of 3D volume rendering has made use of significant improvements in the rendering speed of computers built by Silicon Graphics (Mountain View, Calif). Software developed by Brian Cabral at Silicon Graphics (27) has refined the company's proprietary texture mapping technique for medical applications. Specific features of volume rendering described in this article are those of a modified development version of this software, which has been in use at our institution for the past 4 years.
|
IMPLEMENTATION OF 3D VOLUME RENDERING
|
|---|
Volume rendering has been implemented both on specialized hardware and in software for more readily available systems. There are advantages and disadvantages to each of these approaches. Software implementations benefit from the potential of being loosely tied to the computer platform for which they were written. This allows a certain level of platform portability that is not possible with hardware implementations. The downside of software implementation is limited rendering speed. A machine-independent approach puts more demands on the central processing unit and cannot take advantage of specialized chip sets that may be available. All predictions point to steady advances in the clock speed of the central processing unit, and there is good reason to believe that volume rendering will soon be possible on standard desktop computers. As of this writing, however, many of the interactive features of volume rendering can be effectively implemented only on high-end computer workstations with specialized graphics hardware.
Both generations of our volume rendering programs have taken advantage of specialized computer graphics hardware. A new generation of parallel processing computers is now commercially available that provides even more advantages for volume rendering in hardware. Parallel processing dramatically improves rendering speed by computing all of the pixel values in a 3D image in parallel (ie, simultaneously) rather than serially (ie, one at a time) as is typical with software implementations. Three-dimensional images can be rendered in real time at rates exceeding 5–10 frames/sec. The importance of interactivity with the clinical data set that real-time rendering makes possible should not be underestimated.
Our laboratory currently uses our own modified version of the "Volren" real-time volume rendering program developed by a team led by Brian Cabral at Silicon Graphics (27). This implementation underwent extensive optimization and many extensions and was the forerunner of 3DVirtuoso, a commercial package manufactured by Siemens Medical Systems (Iselin, NJ). As more potential users have discovered the benefits of interactive volume rendering, computer manufacturers and researchers have responded with support for volume rendering through hardware and software libraries. For example, Silicon Graphics' proprietary extensions to the OpenGL graphics application programming interface are becoming an industry standard, which should lead to more hardware support for volume rendering. Independent research in parallel architectures for volume rendering has led to the development of a personal computer card (Mitsubishi Electric America, Sunnyvale, Calif) that supports fast, high-quality volume rendering.
Our discussion of the implementation of 3D volume rendering considers matters relating to volume data management, rendering parameters, and image display. This threefold division roughly corresponds to the process by which volume data move from the scanner to the workstation screen.
Volume Data Management
Data management relates to operations including acquisition, resampling, and editing of the data set. Data movement and formatting have direct implications for the utility of 3D rendering. Many early innovations in 3D rendering were essentially creative methods for preprocessing data to increase computing speed or facilitate visualization of anatomy and disease. As computer systems have become faster, some of this processing (eg, editing with clip planes) has become interactive. The overriding goal of data management is to maintain the fidelity and usefulness of patient data. The computing power necessary to work with the full volume data set can be prohibitively expensive. Advances in computer image processing will continue to increase the speed and reduce the cost of rendering. At present, data management for volume rendering can both reduce computing overhead and increase real-time interactivity by conforming the volume data set to the capabilities of a particular computer platform.
Acquisition.—The quality of data generated from modern medical scanners continues to improve. The evolution from conventional to spiral CT has advantages for 3D volume rendering. Tube heating limited the earliest spiral CT scanners to acquisition of thin sections through a small area of the body or thick sections through a larger area. Newer scanners allow longer acquisition times, resulting in larger volumes of very high resolution data. Advances in spiral CT scanner technology allow high-quality vascular imaging as well as improvements in a wide range of traditional imaging applications including higher sensitivity and specificity in detection of lung nodules and liver lesions. These high-resolution data are ideal for 3D volume rendering. It is now possible to achieve optimal contrast enhancement, which allows new imaging techniques such as 3D CT angiography. With spiral CT, axial images are reconstructed at arbitrary intervals, which also improves the quality of multiplanar and 3D reconstruction.
Resampling.—Resampling is accomplished with an algorithm that maps a volume data set into a new, often smaller volume. This is usually done to reduce image processing time and may be applied independently for dynamic or static display. User settings for image quality determine the amount of resampling the computer will perform on the volume data set prior to rendering. Resampling is often broken down into two separate operations. Axial sections are scaled in the x and y axes to reduce the size of the data set (eg, from 512 x 512 to 256 x 256). Resampling in the z axis may be arbitrarily set to any suitable value. Spiral CT scans are also reconstructed in this dimension, with image quality being a factor of pitch and collimation. With some volume rendering techniques, the z axis must be calibrated in units equal to those of the x and y axes to simplify computations; on other workstations, this is not necessary. Regardless, however, performance may be optimized by scaling data to available memory. This is particularly true for implementations on specialized hardware (27).
Editing.—Manual editing is typically performed to "eliminate" an object such as an organ from surrounding structures. For each original axial image, an outline of the region of interest must be drawn that segments the image into data to be displayed or suppressed. The opacity value of voxels to be suppressed is set to 0 (see "Opacity"). This may reduce the effective size of the data set and further accelerate rendering. Manual editing can be very time-consuming: It often takes 30–60 minutes to edit a single image set. Simple methods have been developed to accelerate the process; for example, contiguous sections can be grouped together and edited as a "slab" (28). For some applications (eg, MIP), manual editing may be necessary to eliminate opaque structures. However, manual editing is generally unnecessary in real-time volume rendering with clip planes as discussed later.
The process of segmentation with manual editing relies on the higher functions of human cognition to recognize discrete entities on an image (eg, individual organs, tumor). This task—a complex analysis of size, shape, intensity, location, texture, and proximity to surrounding structures—is extremely difficult to automate. There are no general computer segmentation algorithms that can be applied to medical data. At present, algorithms must be "tailor-made" for medical imaging applications in specific regions of the body. Even so, segmentation of images is often only semiautomated, requiring additional manual editing by an expert (28).
Segmentation is currently an active area of research. There have been many approaches proposed and implemented for distinguishing structures in 3D data in greater detail than that afforded by edge detection alone. These approaches include applications that make use of morphologic parameters or dynamic contour fitting. Our own research has resulted in automated editing applications for specific domains including the abdomen, lung, and liver (28–30).
Whereas the editing procedures described previously are implemented prior to rendering, clip-plane editing is a technique for interactively manipulating the 3D image at the workstation in real time. This technique defines planes that segment the volume into regions of displayed or suppressed voxels. Voxels that obscure the region of interest may be suppressed by setting the opacity of the data to 0. The intersection outline of a clip plane and the volume is displayed on the 3D image and provides a visual cue for editing the volume (Fig 5). Multiple clip planes can be positioned at any orientation or depth within the volume by selecting and moving intersection outlines on the active 3D image display. Clip-plane editing is a simple, flexible tool that enables the user to visualize structures within the volume that would otherwise be obscured. A diagnostically useful image can be created in seconds. In effect, clip planes create images that combine features of volume rendering with features of multiplanar reconstruction.
View larger version (126K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 5. Clip-plane editing. A clip plane (polygon indicated by straight arrow) can be used to interactively "cut away" the abdominal wall or other superficial structures to allow visualization of anatomic structures and pathologic conditions within the volume. The image demonstrates transitional cell carcinoma of the right renal pelvis (curved arrow).
|
|
Rendering Parameters
Rendering parameters are applied to the full volume data set and affect the appearance of the image to be displayed. The window width and level functions are similar to windowing parameter settings on standard CT scanners or workstations. Algorithms that define how a data set will be modified for display are called transfer functions. A transfer function maps the input data values (eg, Hounsfield units for CT) to lighting properties needed by volume rendering algorithms. These properties are opacity, brightness, and color. Opacity and brightness are unique functions of 3D rendering that allow the user to selectively reveal structures that would otherwise be obscured. The percentage classifier combines these functions with the color function and enables more discrete interactions with the volume data set.
Window Width and Level.—Volume rendering typically segments data on the basis of voxel attenuation. We use window width and level controls similar to those used for display of conventional axial CT images. The window can be adjusted to standard settings used to display soft tissue, liver, bone, or lung; however, real-time rendering also permits the user to interactively alter the window setting and instantly see the changes reflected in the displayed 3D image. This interactivity allows the user to rapidly customize the display to specific cases with varying levels of contrast enhancement and explore a variety of attenuation ranges.
The width and level settings define a transfer function that maps the measured attenuation of each voxel to a corresponding gray-scale value, which in turn is used to create the 3D image (Fig 6). The transfer function used in volume rendering also segments the data on the basis of voxel attenuation; unlike thresholding, however, it accurately models volume averaging of multiple materials within a voxel. It is our experience that giving the user complete control over the transfer function works well for making qualitative observations (eg, determining the location of fractures, aneurysms, and tumors) or for understanding complex 3D structures such as overlapping vasculature. In quantitative measurements (eg, degree of stenosis), however, user control can lead to variability. A standardized approach to selecting this transfer function is needed to ensure accurate, reproducible results for such applications as measuring vascular stenoses. Different rendering parameters can alter the apparent diameter of the normal vessel and the stenotic segment. More work is needed to permit quantitative assessment from volume-rendered images.
View larger version (16K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 6. Diagram illustrates the effect of adjustments in window width and level with respect to the voxel histogram of a CT data set. Such adjustments alter the attenuation of structures displayed in a reconstructed 3D image. A change in width (slope) alters image contrast, whereas a change in level (shift on the x axis) alters data inclusion and the attenuation of voxels in the resulting image.
|
|
Opacity.—Opacity refers to the degree with which structures that appear close to the user obscure structures that seem farther away. Opacity can be varied from 0% to 100%. High opacity values produce an appearance similar to surface rendering, which helps display complex 3D relationships clearly. Low opacity values allow the user to "see through" structures and can be very useful for such applications as seeing thrombus within the lumen of a vessel or evaluating bone abnormalities (eg, tumors located below the cortical surface).
These properties of opacity are intuitive; however, varying the opacity also has a second, less intuitive but very important effect on the image: It changes the apparent size of objects. Higher opacity values make objects appear larger, whereas lower opacity values make them appear smaller. This property has important implications for applications that rely on measurements, including measuring degree of stenosis from CT angiography data.
Brightness.—Brightness affects the appearance of the image by scaling the value of every pixel by the selected percentage. This does not affect accuracy; unlike changes in opacity, changes in brightness do not alter the apparent diameter of rendered structures. Like opacity, brightness can be varied from 0% to 100%. Brightness settings are largely subjective and are based on the preferences of the individual user. A setting of 100% works well for nearly all applications. This transfer function was implemented in our development software but was not included in the commercial release because it has proved to be of limited use.
Percentage Classification.—In early versions of volume rendering software, percentage classification was a necessary prerequisite to rendering. Now, it is primarily a means of selectively applying transfer functions to attenuation value ranges within a volume data set. This can be useful in differentiating disease from tissue on the basis of color or decreasing the opacity of obscuring structures such as bone.
The concept of percentage classification rests on the assumption that voxels representing a given tissue form a gaussian distribution of intensities around a central peak value, which theoretically represents 100% of that tissue type. Above and below this peak value are ranges of intensities representing a probability distribution between 0% and 100% of voxels containing the tissue of interest. It is also assumed that additional regions of interest will demonstrate this pattern of data distribution. Each distribution is approximated by a trapezoid in the software that can be manipulated interactively to alter the display. Multiple trapezoidal distributions can be displayed simultaneously. Changing the length or slope of the sides of the trapezoid alters the image in several ways. Decreasing the upslope of the trapezoid increases the gray scale of the image, whereas increasing the upslope increases image contrast. The top of the trapezoid specifies voxels representing undifferentiated tissue, all of which are assigned the same value for display by the transfer function. Voxels are not limited to an all-or-nothing contribution to the rendered image as in SSD and MIP. For example, an expert user can refine an image such that vascular structures with a different attenuation can be distinguished from calcification. Small-diameter vessels can be more readily visualized because volume-averaged voxels, which might otherwise fall outside a threshold value, are included in the rendering.
Image Display
Image display relates to the process by which a "virtual" 3D representation of a volume data set is "flattened" onto one or more 2D planes and to how the resulting images are made accessible to the user. A number of techniques have been implemented in both software and hardware for reducing an image for display from volume data. Although these techniques are usually compared on the basis of speed, the method used also has an effect on image quality. The results of further research to optimize image accuracy will be forthcoming in the medical literature. The image display method also defines what the user actually sees and interprets on the workstation screen. A well-designed user interface enables the radiologist to interact dynamically with images of the "virtual" volume represented in computer memory. Higher-level functionalities of 3D volume rendering workstations include methods for "flying through" and "flying around" the volume, displaying multiple views, and representing depth in a volume. Carefully crafted design and implementation of the computer display interface is indispensable to a truly interactive and useful imaging workstation.
Ray Casting and Other Techniques.—Ray casting is a basic technique for displaying a 3D volume of data in two dimensions. In this technique, an array of parallel lines (rays) are mathematically projected through the volume in alignment with each pixel within a desired display area. The amount of intersection between the ray and a voxel in the volume determines the contribution of that voxel value. A weighted sum of these voxel values encountered by the ray is calculated for each pixel and then scaled to the particular range of gray-scale values in the display, after which an image is constructed. A new image representing a different vantage point can be constructed by altering the relative angle at which the rays pass through the volume. The effects of perspective can be incorporated by passing nonparallel rays through the volume that radiate from a location approximating that of the viewer and calculating the resulting intersections. Ray casting through a typical volume data set to create a 512 x 512 image requires over 250,000 sequential ray calculations. Although this approach can be slow, its implementation requires no specialized graphics hardware.
Two other techniques should be mentioned as alternatives to ray casting. The first is a method called "splatting" in which the voxel values in a volume are projected forward in parallel onto a 2D plane. This technique has some computational advantages over ray casting and includes refinements relating to the spread of voxel values on the 2D plane. The other technique, which is used in our software, takes advantage of optimized hardware routines for applying 2D or 3D textures to planar objects. This method simplifies the initial computations necessary for projection by first realigning the volume data with the desired image plane. Next, sections are extracted with texture mapping hardware and a composite of these sections is generated with blending hardware. Performance of this technique with new specialized hardware has made real-time interaction possible for users, who can render more than 5 images per second from the whole data set. In terms of clinical utility, this capability has made the difference between a system geared toward long-term surgical planning and one used routinely for patient review.
Fly-through and Fly-around.—The special "fly-through" function was developed from computer technologies used in the entertainment industry. A fly-through reproduces the complex rotation and dollying required for an extended camera shot (Fig 7). The computations that guide the motion of a robot-controlled camera sweeping over a scale model also guide the dynamic point of view representing a virtual volume. The line on which the virtual camera is traced can be determined either by automatic routines that follow anatomy or by manual or semiautomatic point placement. A curve fit to these points permits intermediate points to be calculated and can be saved for future playback of the fly-through. The viewing angle relative to the curve can also be varied along the line. An equivalent to focal length is set with variations in perspective. Unlike an actual camera, a clip plane can be positioned to remove obscuring data from in front of the "lens." Rendering parameters are determined for each discrete point defined on the curve, and each sequential image represents a single use of the general routines for 2D projection from the 3D volume. The fly-through function has been shown to be effective in clinical applications such as virtual colonoscopy and the evaluation of stent placement. Future applications for robot-assisted surgery are also under investigation.
View larger version (29K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 7. Diagram illustrates fly-through and fly-around display functions. A fly-through reconstructs an image sequence with rotation and dollying similar to what is used for a complex camera shot. A fly-around projects a sequence of views from the surface of a sphere that has the region of interest at its center.
|
|
The "fly-around" is a function that can effectively isolate a structure within a volume for protracted viewing. The interface we developed requires that the user first place an anchor point in a region of interest within the volume. The distance from the region can be adjusted, and as in the fly-through, the clip plane and perspective are also variable. The image projected on the display represents a view from the surface of a sphere that has the region of interest at its center (Fig 7). Moving the mouse enables the user to rotate the sphere and thereby gain a different vantage point. This presentation of volume data has proved useful for the evaluation of complex local anatomy such as carotid artery stenosis and vascular anomalies.
Multiple Views.—Additional functionality allows multiple views of the volume data with independent parameters in equal segments of the display window ("panes"). For example, the user can maintain a reference image in one pane while simultaneously displaying differing opacities or an MIP image in others. The need for multiple views was first demonstrated in our early implementation of multiplanar reconstruction (31). Current software also adds functionality to features such as the fly-through by showing the dynamic view in one plane and a reference image with graphic representation of the camera picture plane in another. This has proved to be essential to the understanding of a fly-through because the structures displayed are often unrecognizable apart from their context. The user can alter any view by first clicking in the pane to make it active and then adjusting parameters as usual. The software automatically updates to the parameter settings currently in the interface as each pane is selected. When multiple panes are selected, their settings are synchronized.
Obscured Structure and Shading Depth Cues.—The perception of 3D relationships in a rendered image on conventional computer monitors and on hard copies can be achieved with several techniques. Depth can be simulated by hiding or obscuring "distant" voxels with "closer" voxels. This technique is implemented by weighted summation of opacity levels from the back of the data set to the front. Voxels encountered in this procedure contribute their opacity values in a way that reflects their position in the volume; therefore, an opaque voxel in the middle of the data set will obscure all data "behind" it. A nearly opaque object will convey better depth cues than a transparent object. However, this tradeoff between depth perception and the medical necessity to see through transparent superficial structures indicates that the best approach is to view a data set with both high and low levels of opacity.
Depth perception is greatly influenced by a learned set of cognitive "rules" concerning object shading (ie, light and shadow on a surface). These rules have been implemented in computer imaging as a family of related lighting models that vary widely. Simple lighting models construct shading algorithms from computations of the orientation of surfaces relative to a single, fixed light source. More complex models account for multiple light sources, reflections, and shadows within the rendered view. In medical imaging, these complex models are seldom used because they result in marginal improvement in comprehension relative to the significantly increased costs for the computing power required.
Two fairly simple shading techniques that are used in medically based volume rendering applications are depth shading and enhanced surface shading. Depth shading makes "distant" structures appear darker than structures that are "closer" to the observer. However, depth shading by itself provides the user with very little information about 3D relationships within the volume. Enhanced surface shading, which is implemented in our software, brightens voxels at the location of material surfaces on the basis of the local gradient of the voxel composition. Planes through the volume are shifted relative to a user-defined light source. Scaled values of the shifted planes are subtracted from the original volume. The effect of this subtraction is to create a map representing enhanced edges and regions of inhomogeneity. This map can be used to change the brightness values of voxels within the volume, giving the effect of a single light source. This technique is fairly simple and can be implemented in cost-effective computer configurations. However, surface enhancement and shading can create artifacts and make image interpretation more difficult. Obscured structure and simple shading techniques are commonly used in combination in medical 3D volume rendering software to augment the perception of depth in rendered images.
Kinetic and Stereo Depth Cues.—The need to visualize spatial relationships in medical images sometimes calls for methods of display on the computer that are not feasible with other media. Two such techniques are object rotation and stereo viewing. The cues to spatial interpretation provided by viewing an object in motion are called the kinetic depth effect (32). Early volume rendering techniques were characterized by lack of real-time capabilities and reliance on the kinetic depth effect for dynamic interaction with the data set. The resulting output consisted of a cine loop of precalculated images at equal angles around an axis that were displayed in rapid succession. Real-time interactive image processing allows the user to control or alter the image as he or she reviews the case.
Stereo display conveys perspective and depth cues by presenting two separate renderings from slightly different points of view to the left and right eyes. This results in an immediate perception of depth owing to the inherent integrating capability of the brain (stereopsis). Image separation on a single computer display is achieved with left and right shutter devices incorporated into eyewear that open and close to alternate frames. Specialized hardware and additional rendering requirements involve added hardware costs. However, the resulting dramatic 3D effect can be very helpful in understanding complex anatomy. Application of other new technologies holds promise for spatial understanding of medical data sets. For example, a head-mounted device tracks parallax due to head motion. Rendering can then be synchronized to head motions with simultaneous display of images from the viewer's current point of reference. When combined with a stereo display and real-time rendering, this technique can provide a realistic representation of 3D relationships. Stereo viewing is not practiced widely by clinicians; however, preliminary experiments in our laboratory suggest that both radiologists and nonradiologists prefer it to conventional display.
|
ACCURACY OF 3D VOLUME RENDERING
|
|---|
Over the years, 3D rendering techniques have been compared on the basis of a wide range of criteria. Initially, SSD and MIP clearly had advantages over 3D volume rendering in terms of rendering speed and hardware costs, which were often seen as prohibitive in a clinical setting. Several generations of computers have come and gone since the introduction of volume rendering, and now it is legitimate to focus on the accuracy of each of the rendering techniques in clinical applications. This is clearly the research agenda seen in the literature: Of the more than 500 articles that present the merits of SSD, MIP, or 3D volume rendering, more than 50 contain direct comparisons of these techniques. Artifacts and limitations remain for each of the techniques. In addition, interoperator variability of user-determined rendering parameters calls for more careful review and suggestions for 3D protocols. Our studies have demonstrated that significant levels of accuracy are achievable with volume rendering and that 3D volume rendering has some potential advantages over other rendering techniques.
Our first quantitative analyses of 3D volume rendering on Pixar hardware focused on applications in orthopedics. We were able to demonstrate by way of simulated femoral neck fractures that percentage classification of tissue more accurately represents small fractures than does binary classification of voxel data: "Gaps were faithfully reproduced in terms of detectability, size, and edge characteristics. There was no evidence of spurious fusions or of bone dropout or artificial 'holes'" (14). In a further refinement of this study, the role of scanning parameters was found to be significant in determining accuracy in fracture gap imaging (33). CT scanning protocols for optimizing volume visualization were recommended, and we continue to refine and develop our scanning techniques with the quality of rendered images in mind.
Our current studies of accuracy in 3D volume rendering focus on the clinically significant measurements of arterial stenosis in precisely fabricated phantoms (15,16). Three phantoms were constructed from tubes with an approximate density of 125 HU. The tubes were filled with contrast material having a density of nearly 325 HU. The three phantoms modeled a luminal diameter of 9.5 mm and degrees of stenosis of 33%, 67%, and 83%, respectively. The phantoms were scanned in three orientations relative to the scanning plane: in-plane, perpendicular to the plane, and at a 45° angle to the plane. In the initial study (15), five radiologists with clinical experience in 3D volume rendering measured the luminal diameters on a randomized unit value grid overlaid on rendered images of the phantoms. The radiologists had control over four rendering parameters: window width, window level, brightness, and opacity. The results of this study indicate a wide interoperator variability (SD ±10.9% from the actual phantom measurements). There was also significant correlation between the degree of stenosis and the particular radiologist taking measurements, with the largest SD occurring in cases involving 83% stenosis. Scanning angle was not found to be a significant indicator for measurement error. These results suggest that display strategies for selection of accurate transfer function parameters need to be developed to ensure consistent and accurate measurements in volume-rendered images.
To this end, in our most recent study (16) we looked more closely at achieving accurate measurements with fixed rendering parameters. The attenuation of the tubes and contrast material was known, so that fixed classification of the phantom model data set assumed that voxels with an attenuation less than or equal to that of the phantom wall contained 0% contrast, whereas voxels with an attenuation equal to or greater than that of the contrast material contained 100% contrast. Voxels between these values were considered to represent partial contrast and were mapped linearly to percentages between 0% and 100%. The brightness transfer function does not affect measurement in 3D volume rendering and was fixed at 100% for this study. Results showed that phantom orientation has a significant impact on the accuracy of measurements of diameter and degree of stenosis. The mean error was much lower for phantoms at a 45° angle or perpendicular to the axial plane than for phantoms parallel to the axial plane.
Although it is difficult to extrapolate results from phantom studies to clinical applications, these findings "suggest that CT angiography should be significantly more accurate for evaluating vessels oriented relatively perpendicular to the axial plane, such as the carotid arteries and the peripheral vascular system, than for vessels with a more in-plane orientation such as the renal arteries" (16). Clinical studies will also be needed to validate this method as a standard technique for producing accurate volume renderings in individual patients.
|
APPLICATIONS FOR 3D VOLUME RENDERING
|
|---|
There are many applications for 3D volume rendering in medicine. Perhaps the most important long-term application is for routine reading of volume data. Although such an application may not get the attention that other, more novel emerging research areas generate, it is likely to be the development that affects the radiologist most profoundly in daily practice. What this means to our daily practice is not yet clear. However, the attention this focuses on the analytic abilities the radiologist brings to other areas such as diagnosis and treatment planning underlines his or her vital role in the medical community. Indeed, surgeons will someday have their own interactive volume displays, which suggests how the role of radiology will rest more firmly on shared resources and expanded communication. The role of 3D volume rendering in radiology continues to grow rapidly. Areas that have received a great deal of attention to date include evaluation of musculoskeletal trauma, vascular imaging, and numerous applications in oncology.
Since the inception of 3D reconstruction of CT data, applications in skeletal anatomy have provided an opportunity for demonstrating the efficacy of new techniques. The rendering technique developed by Herman and Liu (21) modeled well-differentiated surfaces. Their study included scans of a plastic ventricle cast (performed with a knowledge that the heart would become accessible for rendering) as well as an actual CT data set of the knee. Evaluation of complex craniofacial anomalies played a major role in early clinical applications of 3D reconstruction of CT data as pioneered by Marsh, Vannier, and Warren (33,34). The advantage of 3D reconstruction for surgical treatment of craniofacial malformations was demonstrated in the accurate evaluation of complex spatial relationships, which represented a dramatic improvement over use of axial CT scans alone. Our first applications of 3D volume rendering (mid-1980s) were in the evaluation of skeletal fractures and planning for orthopedic implants (26,35,36). Because preclassification of volume data was necessary at that time, applications in orthopedics provided an excellent opportunity for demonstrating the efficacy of 3D volume rendering despite rendering times that precluded interactivity. More recently, 3D volume rendering has been shown to have advantages over other rendering techniques, particularly in resolving subcortical lesions and minimally displaced fractures and in visualizing hidden areas of interest while creating few artifacts (Figs 8, 9) (37). Three-dimensional volume rendering is also valuable in the evaluation of suspected infectious and neoplastic disease with musculoskeletal CT. Other constructive uses of 3D volume rendering include determining disease extent and requisite therapy planning in accordance with anatomic information contained in the 3D display of volume data. Postoperative review of patients with orthopedic hardware can also be accomplished more effectively because 3D volume rendering eliminates the vast majority of streak artifact and clearly delineates the relationship between hardware, bone, and bone fragments (Fig 10).
View larger version (114K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 8a. Multiple pelvic fractures. Anterior (a), outlet (b), and oblique (c) 3D reconstructions demonstrate fractures of the right femoral neck and left acetabulum. The extent of the fractures is clearly defined. The ability to edit images interactively is particularly helpful in visualizing displacement of the medial wall of the left acetabulum (c).
|
|
View larger version (97K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 8b. Multiple pelvic fractures. Anterior (a), outlet (b), and oblique (c) 3D reconstructions demonstrate fractures of the right femoral neck and left acetabulum. The extent of the fractures is clearly defined. The ability to edit images interactively is particularly helpful in visualizing displacement of the medial wall of the left acetabulum (c).
|
|
View larger version (129K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 8c. Multiple pelvic fractures. Anterior (a), outlet (b), and oblique (c) 3D reconstructions demonstrate fractures of the right femoral neck and left acetabulum. The extent of the fractures is clearly defined. The ability to edit images interactively is particularly helpful in visualizing displacement of the medial wall of the left acetabulum (c).
|
|
View larger version (150K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 9a. Fracture of the tibial plateau. Three-dimensional reconstructions clearly demonstrate a fracture of the lateral aspect of the tibial plateau. (a) Anterior view defines the joint space and its relationship to the fracture. (b) Superior view better defines the orientation of the fragments to the main components of the plateau.
|
|
View larger version (127K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 9b. Fracture of the tibial plateau. Three-dimensional reconstructions clearly demonstrate a fracture of the lateral aspect of the tibial plateau. (a) Anterior view defines the joint space and its relationship to the fracture. (b) Superior view better defines the orientation of the fragments to the main components of the plateau.
|
|
View larger version (131K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 10. Loosening of a hip prosthesis. Three-dimensional reconstruction (anterior view) clearly depicts a left hip prosthesis with loosening of the femoral cup and superior displacement of the femoral component despite substantial artifact on source images.
|
|
A compelling impetus for the development of 3D volume rendering is its potential for providing information that overlaps with or could replace that provided by other, more expensive or invasive radiologic studies (eg, conventional angiography) (Fig 11). Today, CT angiography is assuming a crucial role within the diagnostic repertoire for vascular studies (38– 40). Spiral CT in combination with 3D volume rendering provides information at a lower cost than does conventional angiography. There are many new, cost-effective applications of volume rendering in vascular imaging, including the evaluation of carotid artery stenosis and pulmonary embolism (Figs 12, 13) (41). Surgically treatable carotid arteriosclerosis accounts for 60% of all strokes in the United States. The risk and expense of conventional angiography have contributed to a shift to noninvasive imaging modalities including ultrasonography, MR angiography, and, more recently, CT angiography. CT angiography with 3D volume rendering has the advantage of being a noninvasive "5-minute" examination that can provide all the information needed for many applications. The accuracy of the study in a variety of applications (eg, thoracic and abdominal aneurysm, renal artery stenosis, intracranial aneurysm) is equal to that of conventional techniques. Moreover, the studies can be performed at less than one-third the cost without any of the complications associated with classic angiography.
View larger version (100K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 11. Aortic dissection. Three-dimensional reconstruction with an angioscopic perspective shows the intimal flap (arrow) and involvement of the great vessels. The intravascular angioscopic view is ideal for evaluating aortic dissections.
|
|
View larger version (121K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 12. Figures 12, 13. (12) Carotid artery stenosis. Three-dimensional reconstruction demonstrates significant (>90%) stenosis of the internal carotid artery just past the bifurcation (arrow). (13) Pulmonary embolism. Three-dimensional reconstructions with standard (a) and angioscopic (b) views demonstrate a large thrombus in the right main pulmonary artery (arrow) that subsequently proved to be tumor thrombus from metastatic breast cancer.
|
|
View larger version (136K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 13a. Figures 12, 13. (12) Carotid artery stenosis. Three-dimensional reconstruction demonstrates significant (>90%) stenosis of the internal carotid artery just past the bifurcation (arrow). (13) Pulmonary embolism. Three-dimensional reconstructions with standard (a) and angioscopic (b) views demonstrate a large thrombus in the right main pulmonary artery (arrow) that subsequently proved to be tumor thrombus from metastatic breast cancer.
|
|
View larger version (126K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 13b. Figures 12, 13. (12) Carotid artery stenosis. Three-dimensional reconstruction demonstrates significant (>90%) stenosis of the internal carotid artery just past the bifurcation (arrow). (13) Pulmonary embolism. Three-dimensional reconstructions with standard (a) and angioscopic (b) views demonstrate a large thrombus in the right main pulmonary artery (arrow) that subsequently proved to be tumor thrombus from metastatic breast cancer.
|
|
There are many areas in oncology that can benefit from 3D volume rendering. At our institution, CT with volume rendering is routinely used to determine tumor resectability and preoperative planning for tumor resection in patients with a variety of neoplasms including pancreatic cancer, primary and metastatic hepatic malignancies, lung cancer, and renal cell carcinoma (Figs 14, 15). CT angiography can obviate conventional angiography in evaluating tumoral encasement of vascular structures. Since CT is already a standard procedure in the evaluation of these malignancies, the added costs of generating a CT angiogram are only those associated with recomputing the volume data and the time needed for clinical review. Scanners with "dual-phase" capabilities allow acquisition of a data set suitable for CT angiography followed immediately by a standard parenchymal-phase scan. Potential new applications for 3D volume rendering often follow from such advances in scanner technology; thus, applications for "dual-phase" imaging can be expected in areas such as composite image processing, which is already creating new prospects in multimodality imaging. Surgical planning is another area in which 3D volume rendering has made significant contributions. A majority of surgical procedures involve a complex 3D relationship between the affected tissue and adjacent anatomic structures. Interaction with volume displays of the relevant CT data has proved helpful to the radiologist and surgeon in preoperative location of hepatic tumors relative to the anatomy of the liver. Recent advances in liver resection have made the role of 3D volume rendering critical to the surgeon in patient assessment. The 3D display of CT data helps confirm the initial decision regarding tumor resectability and facilitates preoperative planning for the most appropriate type of resection.
View larger version (111K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 14a. Renal cell carcinoma. Axial (a), coronal (b), and lateral oblique (c) 3D reconstructions demonstrate a left-sided renal cell carcinoma. The superolateral position of the tumor makes it accessible at partial nephrectomy.
|
|
View larger version (97K):
[in this window]
[in a new window]
[Download PPT slide]
|
Figure 14b. Renal cell carcinoma. Axial (a), coronal (b), and lateral oblique (c) 3D reconstructions demonstrate a left-sided renal cell carcinoma. The superolateral position of the tumor makes it accessible at partial nephrectomy.
|
|
Top
Abstract
INTRODUCTION
