Biomedical applications of Multi-Modal Learning and Active Learning

Personnel: Yash-yee Logan, Mohit Prabhushankar 

Goal: To incorporate expert diagnostics and insights into the analysis of OCT using multi-modal and active learning.

Challenges: Data diversity is captured within medical metadata but remains un-exploited in existing active learning paradigms and underexploited in multimodal frameworks.

Workflow of the patient aware active learning framework

Our Work: In [1] we argue that injecting ophthalmological assessments as another supervision in a learning framework is of great importance for the machine learning process to perform accurate and interpretable classification. We demonstrate the proposed framework through comprehensive experiments that compare the effectiveness of combining diagnostic attribute features with latent visual representations and show that they surpass the state-of-the-art approach. We also develop a framework [2]  that incorporates clinical insights into the sample selection process of active learning that can be incorporated with existing algorithms. The framework captures diverse disease manifestations from patients to improve generalization performance of OCT classification. We also demonstrate that active learning paradigms developed for natural images are insufficient for handling medical data [3]. 

 Architectures for OCT classification: (A) Singlestream autoencoder (B) Dual-stream autoencoder

References:

  1. Y. Logan, K. Kokilepersaud, G. Kwon and G. AlRegib, C. Wykoff, H. Yu, "Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification," in IEEE International Symposium on Biomedical Imaging (ISBI), Kolkata, India, Jan. 7 2022. [PDF]

  2. Y. Logan, R. Benkert, A. Mustafa, G. Kwon, G. AlRegib, "Patient Aware Active Learning for Fine-Grained OCT Classification," IEEE International Conference on Image Processing (ICIP), Bordeaux, France, Oct. 16-19 2022. [PDF][Code]

  3. Y. Logan, M. Prabhushankar, and G. AlRegib, "DECAL: DEployable Clinical Active Learning," in International Conference on Machine Learning (ICML) Workshop on Adaptive Experimental Design and Active Learning in the Real World, Baltimore, MD, Jul. 2022. [PDF][Code]