Task 001/002 - Neuro-inspired Algorithms and Theory

Abstract:

There is a growing demand for an intelligent system to continually learn knowledge from a data stream. Continual learning requires both the preservation of previous knowledge (i.e., avoiding catastrophic forgetting) and the acquisition of new knowledge. Different from previous works that focus only on model adaptation (e.g., regularization, network expansion, memory rehearsal, etc.), we propose a novel training scheme named acquisitive learning (AL), which emphasizes both the knowledge inheritance and knowledge acquisition. AL starts from an elaborately selected model with pre-trained knowledge (the inherited model) and then adapts it to new data using segmented training. The selection is achieved by injecting random noise to various inherited models for better model robustness, which promises higher accuracy in further knowledge acquisition. The approach is validated by the visualization of the loss landscape and quantitative roughness measurement. The combination of the selective inherited model and knowledge acquisition reduces catastrophic forgetting by 10X on the CIFAR-100 dataset.  Furthermore, AL benefits edge computing with 5X reduction in latency per training image on FPGA prototype and 150X reduction in training FLOPs. 

 

Bio:

Xiaocong Du (S'19) received her B.S. degree in control engineering from Shandong University, China, in 2014, and the M.S. degree in electrical and computer engineering from the University of Pittsburgh in 2016. She started pursuing her Ph.D. degree in electrical engineering at Arizona State University from 2016. Her research interests include efficient algorithm design for deep learning, covering model pruning, neural architecture search, continual learning, etc.  Currently, she is working as a summer research intern at Facebook, Inc.