AI Hardware-Algorithm Co-Design

To scale AI models for muti-modal, life-long learning scenarios, the underlying hardware must take a key role in shaping the design of hardware-aware model architectures and learning algorithms, with inspirations from neuroscience.

To scale today’s AI models towards the future life-long learning scenarios with more sophisticated workloads, the underlying hardware must take a bigger role in shaping the design of hardware-aware neural network model architectures and learning algorithms. This motivates us to explore two directions:

  1. AI systems that naturally tolerate hardware non-idealities without incurring overhead.
  2. Tailored or re-designed learning algorithms that directly benefit from certain hardware behaviors not seen on conventional hardware.

ML techniques themselves may be leveraged for co-designing AI hardware and algorithms:

  1. Training statistical machine learning models based on hardware simulations, CAD data, metrology and measurement data, can enable large-scale design space explorations and may uncover hidden mechnisms.
  2. Meta-models and teacher-models can be constructed to reversely generate efficient, hardware-aware model architectures, using techniques such as neural architecture search (NAS) and knowledge distillation (KD).

Lessons and inspirations from theoretical neuroscience, if used judiciously, with unique characteristics offered by nanoscale semiconductor devices, may guide the design of a new class of AI models and hardware.