Augmented Reality Imaging System: 3D Viewing of a Breast Cancer
David B. Douglas, M.D.1,*, John M. Boone, Ph.D.2, Emanuel Petricoin, Ph.D.3, Lance Liotta, M.D., Ph.D.3,
Eugene Wilson, D.O.4
1 Department of Radiology, Stanford University, Palo Alto, CA, USA; 2 Department of Radiology, University of California, Davis, CA, USA; 3 Center for Applied Proteomics and Molecular Medicine, George Mason University, Manassas, VA, USA; 4 Department of Radiology, Fort Benning, Columbus, GA, USA
Objective: To display images of breast cancer from a dedicated breast CT using Depth 3-Dimensional (D3D) augmented reality.
Methods: A case of breast cancer imaged using contrast-enhanced breast CT (Computed Tomography) was viewed with the augmented reality imaging, which uses a head display unit (HDU) and joystick control interface.
Results: The augmented reality system demonstrated 3D viewing of the breast mass with head position tracking, stereoscopic depth perception, focal point convergence and the use of a 3D cursor and joy-stick enabled fly through with visualization of the spiculations extending from the breast cancer.
Conclusion: The augmented reality system provided 3D visualization of the breast cancer with depth perception and visualization of the mass's spiculations. The augmented reality system should be further researched to determine the utility in clinical practice.
Augmented Reality | 3D medical imaging | radiology
State of the art medical imaging systems including Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) now have the capability to generate high-resolution volumetric datasets with over 1000 cross-sectional images. One method to view such a large number of cross-sectional images efficiently is through volume rendered (VR) techniques, which provides a two-dimensional (2D) representation of the 3D volume. VR can define certain complex anatomy, but this technique is limited by overlapping structures1,2.
Recently, an augmented reality system (D3D Enterprise, LLC, Winter Park, FL)3,4 has capitalized on advances in computer processing speed and virtual reality displays to successfully demonstrate stereoscopic 3D visualization on simulated data of breast microcalcifications.5 This system provides a separate image to each eye with head tracking, ability to rotate, translate, zoom and converge the eyes to a particular focal point, change intra-ocular distance and display fly-through viewing of the image datasets (Figure 1)5. Breast cancer is a leading cause of cancer death in women6,7 and pattern classification is important in characterizing microcalcifications8,9 with branching patterns suspicious for ductal carcinoma in situ (DCIS).10,11 The D3D imaging system was initially tested on simulated data of breast microcalcifications and was successfully demonstrated to improve the assessment of the branching pattern of microcalcifications compared to single perspective viewing.5 Similarly, characterization of a breast mass's shape and margins is important in assessing the characteristics of a breast mass. Dedicated breast CT now provides high spatial resolution imaging of breast masses12 and data from the recent I-SPY trial showed the importance of characterizing tumor morphology13. The purpose of this article is to view a known breast cancer mass with the D3D system to assess the shape and margins, which are important factors in characterizing the risk of malignancy.
MATERIALS AND METHODS
Subjects: Dedicated breast CT data from one subject with known infiltrating ductal carcinoma was used in this study. The breast CT data was obtained from a prior Institutional Review Board approved study in a Health Insurance Portability and Accountability Act (HIPAA) compliant method with all patient identifiers removed prior to viewing with the D3D augmented reality imaging system14.
Procedures: The breast CT data included 204 slices each with a 512 x 512 matrix; thus, the total number of voxels in the volume of interest (VOI) in this dataset was 53,477,376. The virtual reality headset used has a matrix of 1080 x 1200 pixels per eye display such that 1,296,000 rays from each eye's viewing perspective were extended into the VOI. In order to maintain a low latency period of less than 0.1 seconds from user roll-pitch-yaw head movement and/or joystick input and to improve visualization of the breast mass, thresholding was performed to remove voxels of non-interest (e.g., fat density). The 3D cursor was used to highlight the breast mass and analysis was performed by one of the authors (E.W.) to include observations of the breast mass's shape, margins and spiculations.
Data collection: A board certified radiologist with 11 years of experience performed analysis using D3D.
Statistical tests: No statistical tests were performed.
The mass's shape, size and spiculations were characterized using the D3D system and the spiculations were found to be more conspicuous when viewed with D3D than on the native CT section images (Figure 2). The spiculations were especially apparent when zoomed in and when actively rotating the viewing trajectory around the mass. The author noticed that the 3D course of the blood vessels was also particularly well seen with D3D.
Figure 1. The D3D system with unique images provided for each eye to achieve stereoscopic 3D imaging with depth perception. Also, note that multiple viewing perspectives of the volume of interest can be achieved.
Figure 2. (A) Contrast-enhanced breast CT demonstrates the mass with small spiculations extending from the margins. (B&C) Same mass from breast CT exam as seen in (A), but viewed with D3D where (B) represents the left eye viewing perspective (LEVP) and (C) represents the right eye viewing perspective (REVP). The red box illustrates the 3D cursor used. (D&E) represent the same mass from the breast CT, but zoomed in and viewed from a different perspective with (D) representing the LEVP and (E) representing the REVP. Red arrows show spiculations extending from the margins of the mass. The red circle represents a spiculation sticking out toward the user, which was well seen when rotating with the D3D system.
Three dimensional augmented reality is a computationally demanding viewing approach that is only recently possible through advances in computer processing and virtual reality displays. The I-SPY trial found that particular phenotypes such as "well-defined margins" correlated with pathologic response to neo-adjuvant chemotherapy, which aided the clinical recommendation of breast conservation therapy as opposed to mastectomy13,15. Advances through D3D augmented reality could help with assessing treatment response through the use of a 3D ghost image recalled from a pre-treatment baseline image where subtle characteristics of the mass (e.g., shape, margins or extent of the spiculations, etc.) could be recognized. This provides a strong rationale for assessing tumor features throughout the course of chemotherapy. Finally, in cases where lumpectomy is performed, D3D augmented reality viewing of the breast mass may help the surgeon in understanding the 3D nature of a multifocal lesion in pre-surgical planning. Continued improvements in breast CT in the form of improved spatial resolution16,17 and accuracy of the breast density measurements18 coupled with viewing with D3D technology will provide new opportunities for improvements in patient care.
We have illustrated, using a case of a breast cancer imaged on a dedicated breast CT, how the D3D system provides an immersive 3D environment and can better depict the breast mass's shape, margins and spiculations compared to conventional 2D viewing. Furthermore, we demonstrated that the use of a joy-stick to "fly inside" the image with real time head tracking for improved human-machine interface. D3D augmented reality holds promise in additional diagnostic and therapeutic applications in medicine and future testing should be performed.
CONFLICT OF INTEREST
Authors EP and LL have a financial interest in D3D technology. DD has a family member with a financial interest in D3D technology. JMB has a financial interest in breast CT. EW reports no conflict of interest.
1. Fishman EK, Ney DR, Heath DG, Corl FM, Horton KM, Johnson PT. Volume rendering versus maximum intensity projection in CT angiography: what works best, when, and why. Radiographics : a review publication of the Radiological Society of North America, Inc. 2006;26(3):905-922.
2. Johnson PT, Heath DG, Kuszyk BS, Fishman EK. CT angiography with volume rendering: advantages and applications in splanchnic vascular imaging. Radiology. 1996;200(2):564-568.
3. Douglas D, Inventor; US Patent Office, assignee. US 8,384,771 Method and Apparatus for Three Dimensional Viewing of Images. 2013.
4. Douglas D, Inventor; US Patent Office, assignee. US 9,349,183 Method and Apparatus for Three Dimensional Viewing of Images. 2016.
5. Douglas DB, Petricoin EF, Liotta L, Wilson E. D3D augmented reality imaging system: proof of concept in mammography. J Med Devices: Evid and Res. 2016;9:277-283.
6. Ferlay J, Shin HR, Bray F, Forman D, Mathers C, Parkin DM. Estimates of worldwide burden of cancer in 2008: GLOBOCAN 2008. International journal of cancer Journal international du cancer. 2010;127(12):2893-2917.
7. DeSantis C, Ma J, Bryan L, Jemal A. Breast cancer statistics, 2013. CA: a cancer journal for clinicians. 2014;64(1):52-62.
8. Howell A. The emerging breast cancer epidemic: early diagnosis and treatment. Breast cancer research : BCR. 2010;12 Suppl 4:S10.
9. Kopans D. Analyzing the Mammogram - Calcifications. Philadelphia, PA: Lippincott-Raven; 1998.
10. Lee KS, Han BH, Chun YK, Kim HS, Kim EE. Correlation between mammographic manifestations and averaged histopathologic nuclear grade using prognosis-predict scoring system for the prognosis of ductal carcinoma in situ. Clinical imaging. 1999;23(6):339-346.
11. Burnside ES, Ochsner JE, Fowler KJ, et al. Use of microcalcification descriptors in BI-RADS 4th edition to stratify risk of malignancy. Radiology. 2007;242(2):388-395.
12. Boone JM, Lindfors KK. Breast CT: potential for breast cancer screening and diagnosis. Future Oncol. 2006;2(3):351-356.
13. Mukhtar RA, Yau C, Rosen M, et al. Clinically meaningful tumor reduction rates vary by prechemotherapy MRI phenotype and tumor subtype in the I-SPY 1 TRIAL (CALGB 150007/150012; ACRIN 6657). Annals of surgical oncology. 2013;20(12):3823-3830.
14. Lindfors KK, Boone JM, Nelson TR, Yang K, Kwan AL, Miller DF. Dedicated breast CT: initial clinical experience. Radiology. 2008;246(3):725-733.
15. Hylton NM, Blume JD, Bernreuter WK, et al. Locally advanced breast cancer: MR imaging for prediction of response to neoadjuvant chemotherapy--results from ACRIN 6657/I-SPY TRIAL. Radiology. 2012;263(3):663-672.
16. Willekens I, Van de Casteele E, Buls N, et al. High-resolution 3D micro-CT imaging of breast microcalcifications: a preliminary analysis. BMC cancer. 2014;14:9.
17. Gazi PM, Yang K, Burkett GW, Jr., Aminololama-Shakeri S, Seibert JA, Boone JM. Evolution of spatial resolution in breast CT at UC Davis. Medical physics. 2015;42(4):1973-1981.
18. Yang K, Burkett G, Boone JM. A breast-specific, negligible-dose scatter correction technique for dedicated cone-beam breast CT: a physics-based approach to improve Hounsfield Unit accuracy. Physics in medicine and biology. 2014;59(21):6487-6505.
* Corresponding Author. David B. Douglas, M.D., Department of Radiology, 300 Pasteur Drive, Room S047, Stanford, CA 94305-5105, USA. Tel: (650) 723-4000 ext 14257. Email: firstname.lastname@example.org
© 2016 by the Authors | Journal of Nature and Science (JNSCI).