Content-Based Image Retrieval from large Medical Image Databases
Content-based Image Retrieval (CBIR) consists of retrieving the most visually similar images to a given query image from a database of images. CBIR from medical image databases does not aim to replace the physician by predicting the disease of a particular case but to assist him/her in diagnosis. The visual characteristics of a disease carry diagnostic information and oftentimes visually similar images correspond to the same disease category. By consulting the output of a CBIR system, the physician can gain more confidence in his/her decision or even consider other possibilities.
In the past, in collaboration with the Department of Radiology at Indiana University and the School of Medicine at the University of Wisconsin, we developed ASSERT, a CBIR system for the domain of HRCT (High-resolution Computed Tomography) images of the lung with emphysema-type diseases. The symptoms of these diseases can drastically alter the appearance of the texture of the lung as can be seen in the following images. Furthermore, the visual characteristics of the diseases vary widely across patients and based on the severity of the disease.
For this domain, it is easy for the physicians to identify the area of the lung that is afflicted by the disease (Pathology Bearing Region - PBR) but it is hard to identify the disease category. In fact, the physicians decide on a diagnosis by visually comparing the case at hand with previously published cases in the medical literature. Our system combines the best of the physicians' and computers' abilities. It enlists the physician's help to roughly delineate the PBR, since this task cannot be reliably accomplished by state-of-the art computer vision algorithms. It uses the computer's computational efficiency to determine and display to the user the most similar cases to the query case. An output of our graphical user interface is shown in the following image.
Our system characterizes the images using low-level features like texture features computed from the co-occurrence matrix of the image. The retrieval is performed hierarchically. At the first level the disease category of the query image is predicted. At the second level the most similar images to the query image, that belong to the predicted class, are retrieved and displayed to the user. To assess the utility of our system we performed a clinical evaluation trial the results of which were published in the Radiology journal. In the trial 11 physicians participated and were asked to diagnose a set of images with and without the help of our system. The physicians were able to double the diagnostic accuracy when using our system. For more information on the technical aspects of our work the reader is referred to the related publications.
CURRENT TEAM
• Josiah Yoder
ALUMNI
• Jennifer Dy
• Christina Pavlopoulou
• Chi-Ren Shyu
Alex M. Aisen, Lynn S. Broderick, Helen Winter-Muram, Carla E. Brodley, A. C. Kak, Christina Pavlopoulou, Jennifer Dy, Chi-Ren Shyu, and Alan Marchiori, "Automated Storage and Retrieval of Thin-Section CT Images to Assist Diagnosis: System Description and Preliminary Assessment," Radiology, Volume 228, Number 1, pp. 265-270, July 2003. [pdf]
Jennifer Dy, Carla Brodley, A. C. Kak, Lynn Broderick, and Alex Aisen, "Unsupervised Feature Selection Applied to Content-Based Retrieval of Lung Images," IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 373-378, March 2003. [pdf]
C. Pavlopoulou, A. C. Kak and C. Brodley, "Content-based Image Retrieval for Medical Imagery" in Proceedings SPIE Medical Imaging: PACS and Integrated Medical Information Systems, San Diego CA, 2003.
Chi-Ren Shyu, Christina Pavlopoulou, A. C. Kak, Carla E. Brodley, and Lynn S. Broderick, "Using Human Perceptual Categories for Content-Based Retrieval from a Medical Image Database," Computer Vision and Image Understanding, Vol. 88, Number 3, pp. 119-151, December 2002. [pdf]
Sean D. MacArthur, Carla E. Brodley, A. C. Kak, and Lynn Broderick, "Interactive Content-Based Retrieval using Relevance Feedback," Computer Vision and Image Understanding, Vol. 88, Number 2, pp. 55-75, November 2002. [pdf]
A.Marchiori, C. Brodley, L. Broderick, J. Dy, C. Pavlopoulou, A. C. Kak and A. Aisen, "CBIR for Medical Images - An Evaluation Trial", in Proceedings IEEE Workshop on Content-Based Access of Image and Video Databases, Hawaii, 2001.
S. D. MacArthur, C. E. Brodley and C. Shyu, "Relevance Feedback Decision Trees in Content-Based Image Retrieval'," in Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, Hilton Head, SC, June 2000. [pdf]
C. R. Shyu, A. C. Kak, C. Brodley, C. Pavlopoulou, M. F. Chyan and L. Broderick, "A web-based CBIR-assisted learning tool for radiology education" in Proceedings IEEE Conference on Multimedia and Expo, 2000. [pdf]
C. R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. M. Aisen, and L. S. Broderick, "ASSERT: A Physician-in-the-loop Content-Based Image Retrieval System for HRCT Image Databases," Computer Vision and Image Understanding (Special Issue on Content-Based Retrieval from Image Databases), pp. 111-131, 1999. [pdf]
J. G. Dy, C. E. Brodley, A. C. Kak, C. Shyu, and L. Broderick, "The Customized-Queries Approach to CBIR using EM," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Fort Collins, CO, June 1999. [pdf]
C. R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. Aisen, and L. Broderick, "Local versus Global Features for Content-Based Image Retrieval," Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries, Santa Barbara, June 1998. [pdf]
• The National Science Foundation, Award No. IIS - 9711535
• The National Institutes of Health, grant 1 RO1 LM0654301A1.
