Consistent Active Learning through Regression Analysis

 Personnel: Ryan Benkert, Yash-Yee Logan, Kiran Kokilepersaud, Chen Zhou, Mohit Prabhushankar

Goal/Motivation: In active learning, performance deterioration is a substantial risk factor in terms of stopping criteria and deployment decisions. Effective algorithms must provide performance guarantees for practical utilization.     

Challenges: Software updates for neural networks are infamously non-deterministic. Even when updates are overwhelmingly positive, previously correct data can be mispredicted and deployment behavior remains uncertain. For paradigms such as active learning, regression is essential as each active learning iteration represents a software update and can result in performance deterioration. In spite of the high relevance, regression is rarely considered within the context of active learning settings and regression benchmarks are non-existent. 

High Level Description of the Work: Regression occurs when a model mispredicts a previously correct data point after an update. In other words, regression occurs when the model representation shifts from a positive prediction to a negative prediction. In our work, we utilize to design methods that combat regression in active learning settings. Specifically, we propose positive-congruent active learning; a framework for regression reduction. We further instantiate the framework with CuRIE, an algorithm that restricts the data pool to informative data points with low representation shift probability. In our experiments, we show that CuRIE effectively reduces regression while maintaining generalization performance on in-distribution and out-of-distribution settings. 

References:

1. R. Benkert, Y. Logan. K. Kokilepersaud, C. Zhou, M. Prabhushankar, G. AlRegib “On the Ramifications of Regression in Deep Active Learning”, in review 2022