Consistent Active Learning through Regression Analysis

 Personnel: Ryan Benkert, Yash-Yee Logan, Kiran Kokilepersaud, Chen Zhou, Mohit Prabhushankar


Goal/Motivation: Software regression describes the performance deterioration after a model update [1]. Knowledge of regression properties is essential for trust establishment in machine learning paradigms like active learning.  

Challenges: Software updates for neural networks are infamously non-deterministic. Even when updates are overwhelmingly positive, previously correct data can be mispredicted and deployment behavior remains uncertain. For paradigms such as active learning, regression is essential as each active learning iteration represents a software update and can result in performance deterioration. In addition, regression properties are radical and unpredictable in active learning settings. For instance, adding labeled data to the training set results in either an increase or decrease in regression properties (Figure~1).

High Level Description of the Work: Regression occurs when a model mispredicts a previously correct data point after an update. In other words, regression occurs when the model representation shifts from a positive prediction to a negative prediction. In our work, we utilize to design methods that combat regression in active learning settings. Specifically, we propose positive-congruent active learning; a framework for regression reduction. We further instantiate the framework with CuRIE, an algorithm that restricts the data pool to informative data points with low representation shift probability. In our experiments, we show that CuRIE effectively reduces regression while maintaining generalization performance on in-distribution and out-of-distribution settings. 

FIgure 1: Right shows accuracy with respect to training set size; Left shows regression with respect to training set size. Regression can both increase or decrease with additional data points.

References:

  1. Yan, Sijie, et al. "Positive-congruent training: Towards regression-free model updates." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

  2. R. Benkert, Y. Logan. K. Kokilepersaud, C. Zhou, M. Prabhushankar, G. AlRegib “On the Ramifications of Regression in Deep Active Learning”, in review 2022