Research: AI Hardware

Past Projects Patents Publications Funding Agencies

AI Hardware

(Return to Current Research Outline)

Our research focuses on developing specialized systems that efficiently support the growing demands of artificial intelligence workloads. As AI algorithms become more complex and data-intensive, we recognize the challenges that traditional hardware architectures face in terms of performance, power consumption, and scalability. To address these limitations, we explore innovative hardware approaches and technologies that make AI processing faster, more efficient, and secure. Our work spans across Compute-In-Memory and Compute-Near-Memory architectures, neuromimetic devices, co-design, AI-driven secure hardware, and AI for hardware design, each contributing to the next generation of AI systems.

C. Co-designing Algorithms and Hardware

1. System Technology Co-design (STCO)

STCO focuses on the holistic optimization of the entire computing stack, bridging the gap between algorithmic requirements and physical system architecture. By concurrently exploring optimal dataflows, mapping strategies, and memory hierarchy configurations, we aim to overcome the fundamental bottlenecks of modern AI hardware, specifically the latency and energy costs associated with data movement. Our research involves a cross-layer approach, re-evaluating the traditional "von Neumann" computing paradigm by integrating compute-in-memory and specialized accelerators, reconfigurable architectures to maximize throughput and energy efficiency of AI workloads.

Publications:
  1. Negi, Shubam, Manik Singhal, Aayush Ankit, Sudeep Bhoja, and Kaushik Roy. "COMET: A Framework for Modeling Compound Operation Dataflows with Explicit Collectives." IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2026.

    Abstract: Modern machine learning accelerators are designed to efficiently execute deep neural networks, but emerging models increasingly rely on compound operations that introduce significant off-chip memory traffic challenges. As model sizes continue to grow, computation must be distributed across spatial clusters, which requires frequent and complex collective communication. Existing dataflow optimization frameworks and performance models lack the explicit modeling of these collective communication costs, limiting their applicability. To address this, we propose COMET, a framework that introduces a novel representation to explicitly model collective communication across spatial clusters, alongside latency and energy cost models for operation-level dependencies. By enabling collective-aware modeling, COMET allows for broader mapping exploration, achieving up to a 1.42× speedup for GEMM-Softmax and 3.46× for GEMM-LayerNorm and 1.82× for self-attention compared to unfused baselines.

  2. Kim, Dong Eun, Tanvi Sharma, and Kaushik Roy. "HASTILY: Hardware-Software Co-Design for Accelerating Transformer Inference Leveraging Compute-in-Memory." IEEE Transactions on Circuits and SYstems for ARtificial Intelligence, 2025.

    Abstract: Transformers have become the backbone of neural network architecture for most machine learning applications. Their widespread use has resulted in multiple efforts on accelerating attention, the basic building block of transformers. This paper tackles the challenges associated with accelerating attention through a hardware-software co-design approach while leveraging compute-in-memory (CIM) architecture. In particular, our energy- and area-efficient CIM based accelerator, named HASTILY, aims to accelerate softmax computation, an integral operation in attention, and minimize their high on-chip memory requirements that grows quadratically with input sequence length. Our architecture consists of novel CIM units called unified compute and lookup modules (UCLMs) that integrate both lookup and multiply-accumulate functionality within the same SRAM array, incurring minimal area overhead over standard CIM arrays. Designed in TSMC 65nm, UCLMs can be used to concurrently perform exponential and matrix-vector multiplication operations. Complementing the proposed architecture, HASTILY features a fine-grained pipelining strategy for scheduling both attention and feed-forward layers, to reduce the quadratic dependence on sequence length to linear dependence. Further, for fast softmax computation which involves computing the maxima and sum of exponential values, such operations are parallelized across multiple cores using reduce and gather strategy. We evaluate our proposed architecture using a compiler tailored towards attention computation and a standard cycle-level CIM simulator. Our evaluation shows end-to-end throughput (TOPS) improvement of 4.4× ?9.8× and 1.7× ?5.9× over Nvidia A40 GPU and baseline CIM hardware, respectively, for BERT models with INT-8 precision. Additionally, it shows gains of 16× ?36× in energy-efficiency (TOPS/W) over A40 GPU and similar energy-efficiency as baseline CIM hardware.

  3. Negi, Shubham, and Kaushik Roy. "HALO: Memory-Centric Heterogeneous Accelerator with 2.5D Integration for Low-Batch LLM Inference." arxiv, 2025.

    Abstract: The rapid adoption of Large Language Models (LLMs) has driven a growing demand for efficient inference, particularly in latency-sensitive applications such as chatbots and personalized assistants. Unlike traditional deep neural networks, LLM inference proceeds in two distinct phases: the prefill phase, which processes the full input sequence in parallel, and the decode phase, which generates tokens sequentially. These phases exhibit highly diverse compute and memory requirements, which makes accelerator design particularly challenging. Prior works have primarily been optimized for high-batch inference or evaluated only short input context lengths, leaving the low-batch and long context regime, which is critical for interactive applications, largely underexplored.

    We propose HALO, a heterogeneous memory centric accelerator designed for these unique challenges of prefill and decode phases in low-batch LLM inference. HALO integrates HBM based Compute-in-DRAM (CiD) with an on-chip analog Compute-in-Memory (CiM), co-packaged using 2.5D integration. To further improve the hardware utilization, we introduce a phase-aware mapping strategy that adapts to the distinct demands of the prefill and decode phases. Compute bound operations in the prefill phase are mapped to CiM to exploit its high throughput matrix multiplication capability, while memory-bound operations in the decode phase are executed on CiD to benefit from reduced data movement within DRAM. Additionally, we present an analysis of the performance tradeoffs of LLMs under two architectural extremes: a fully CiD and a fully on-chip analog CiM design to highlight the need for a heterogeneous design. We evaluate HALO on LLaMA-2 7B and Qwen3 8B models. Our experimental results show that LLMs mapped to HALO achieve up to 18x geometric mean speedup over AttAcc, an attention-optimized mapping and 2.5x over CENT, a fully CiD based mapping.

  4. Sharma, Tanvi, Mustafa Ali, Indranil Chakraborty, and Kaushik Roy. "What, When, Where to Compute-in-Memory for Efficient Matrix Multiplication During Machine Learning Inference." IEEE Transactions on Emerging Topics in Computing, 2025..

    Abstract: Matrix multiplication is the dominant computation during Machine Learning (ML) inference. To efficiently perform such multiplication operations, Compute-in-memory (CiM) paradigms have emerged as a highly energy efficient solution. However, integrating compute in memory poses key questions, such as 1) What type of CiM to use: Given a multitude of CiM design characteristics, determining their suitability from architecture perspective is needed. 2) When to use CiM: ML inference includes workloads with a variety of memory and compute requirements, making it difficult to identify when CiM is more beneficial than standard processing cores. 3) Where to integrate CiM: Each memory level has different bandwidth and capacity, creating different data reuse opportunities for CiM integration. To answer such questions regarding on-chip CiM integration for accelerating ML workloads, we use an analytical architecture-evaluation methodology with tailored mapping algorithm. The mapping algorithm aims to achieve highest weight reuse and reduced data movements for a given CiM prototype and workload. Our analysis considers the integration of CiM prototypes into the cache levels of a tensor-core-like architecture, and shows that CiM integrated memory improves energy efficiency by up to 3.4× and throughput by up to 15.6× compared to established baseline with INT-8 precision. We believe the proposed work provides insights into what type of CiM to use, and when and where to optimally integrate it in the cache hierarchy for efficient matrix multiplication.

  5. Roy, Arani, and Kaushik Roy. "AlphaBlend: Hardware-Algorithm Co-design with Mixed-Alphabet Set Multipliers for DNN Workloads." IEEE International Symposium on Circuits and Systems (ISCAS), 2025.

    Abstract: We introduce Mixed Alphabet-Aware Training (Mixed AAT), a novel hardware-aware training method that assigns distinct low-complexity multiplier sets (alphabet sets) to individual Deep Neural Network (DNN) layers based on sensitivity to weight perturbations. This method maintains full-precision accuracy while significantly reducing memory and compute costs. Achieving <1% accuracy degradation on ResNet, VGG, and MobileNet across CIFAR-10 and ImageNet, we complement Mixed AAT with two hardware solutions: an approximate near-memory digital architecture and a modified systolic array accelerator. These co-designs - a blend of Mixed AAT and hardware solutions - offer at least 3.5X energy savings and 1.3X speedup, ideal for energy-constrained applications.

  6. Negi, Shubham, Utkarsh Saxena, Deepika Sharma, Jeffry Victor, Imtiaz Ahmed, Sumeet Kumar Gupta, and Kaushik Roy. "Algorithm Hardware Co-Design for ADC-Less Compute In-Memory Accelerator." IEEE Transactions on Circuits and Systems for Artificial Intelligence, 2024.

    Abstract: The increasing pervasiveness of artificial intelligence (AI), particularly deep learning demands high-performing yet efficient hardware resources at the edge. Analog compute-in-memory (CiM) architectures have tremendous potential to accelerate AI at the edge by reducing data movement between memory and compute units and exploiting parallelism. However, to fully reap the benefits of analog CiM, it is imperative to deal with the area, latency, and power overheads introduced by high-precision analog-to-digital converters (ADCs). In this work, we propose a hardware-algorithm co-design approach to reduce ADC overhead in analog CiM architectures. We designed a deep neural network (DNN) quantization framework tailored to analog CiM hardware architectures, integrating essential features such as tiling, bit-slicing, and layer mapping. Moreover, we also developed an ADC-Less hybrid analog-digital CiM hardware architecture HCiM that can efficiently process the DNNs trained using our framework. Additionally, we studied the effects of non-idealities in analog CiM on DNN accuracy. Using our hardware-aware training methodology, we can perform extremely low precision quantization and reduce the required ADC precision to binary (1-bit) or ternary (1.5-bit). Compared to an analog CiM baseline architecture using 7 and 4-bit ADC, HCiM achieves energy reductions up to 28× and 12×, respectively. Furthermore, in the presence of analog non-idealities, DNN mapped to HCiM exhibits a minimal drop in accuracy.

  7. He, Kang, Indranil Chakraborty, Cheng Wang, and Kaushik Roy. "Design Space and Memory Technology Co-Exploration for In-Memory Computing Based Machine Learning Accelerators." IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2022.

    Abstract: In-Memory Computing (IMC) has become a promising paradigm for accelerating machine learning (ML) inference. While IMC architectures built on various memory technologies have demonstrated higher throughput and energy efficiency compared to conventional digital architectures, little research has been done from system-level perspective to provide comprehensive and fair comparisons of different memory technologies under the same hardware budget (area). Since large-scale analog IMC hardware relies on the costly analog-digital converters (ADCs) for robust digital communication, optimizing IMC architecture performance requires synergistic co-design of memory arrays and peripheral ADCs, wherein the trade-offs could depend on the underlying memory technologies. To that effect, we co-explore IMC macro design space and memory technology to identify the best design point for each memory type under iso-area budgets, aiming to make fair comparisons among different technologies, including SRAM, phase change memory, resistive RAM, ferroelectrics and spintronics. First, an extended simulation framework employing spatial architecture with off-chip DRAM is developed, capable of integrating both CMOS and nonvolatile memory technologies. Subsequently, we propose different modes of ADC operations with distinctive weight mapping schemes to cope with different on-chip area budgets. Our results show that under an iso-area budget, the various memory technologies being evaluated will need to adopt different IMC macro-level designs to deliver the optimal energy-delay-product (EDP) at system level. We demonstrate that under small area budgets, the choice of best memory technology is determined by its cell area and writing energy. While area budgets are larger, cell area becomes the dominant factor for technology selection.

  8. Sharma, Deepika, Aayush Ankit, and Kaushik Roy. "Identifying Efficient Dataflows for Spiking Neural Networks." ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED), 2022.

    Abstract: Deep feed-forward Spiking Neural Networks (SNNs) trained using appropriate learning algorithms have been shown to match the performance of state-of-the-art Artificial Neural Networks (ANNs). The inputs to an SNN layer are 1-bit spikes distributed over several timesteps. In addition, along with the standard artificial neural network (ANN) data structures, SNNs require one additional data structure ? the membrane potential (Vmem) for each neuron which is updated every timestep. Hence, the dataflow requirements for energy-efficient hardware implementation of SNNs can be different from the standard ANNs. In this paper, we propose optimal dataflows for deep spiking neural network layers. To evaluate the energy and latency of different dataflows, we considered three hardware architectures with varying on-chip resources to represent a class of spatial accelerators. We developed a set of rules leading to optimum dataflow for SNNs that achieve more than 90% improvement in Energy-Delay Product (EDP) compared to the baseline for some workloads and architectures.

  9. Negi, Shubham, Indranil Chakraborty, Aayush Ankit, and Kaushik Roy. "NAX: neural architecture and memristive xbar based accelerator co-design." ACM/IEEE Design Automation Conference (DAC), 2022.

    Abstract: Neural Architecture Search (NAS) has provided the ability to design efficient deep neural network (DNN) catered towards different hardwares like GPUs, CPUs etc. However, integrating NAS with Memristive Crossbar Array (MCA) based In-Memory Computing (IMC) accelerator remains an open problem. The hardware efficiency (energy, latency and area) as well as application accuracy (considering device and circuit non-idealities) of DNNs mapped to such hardware are co-dependent on network parameters such as kernel size, depth etc. and hardware architecture parameters such as crossbar size and the precision of analog-to-digital converters. Co-optimization of both network and hardware parameters presents a challenging search space comprising of different kernel sizes mapped to varying crossbar sizes. To that effect, we propose NAX - an efficient neural architecture search engine that co-designs neural network and IMC based hardware architecture. NAX explores the aforementioned search space to determine kernel and corresponding crossbar sizes for each DNN layer to achieve optimal tradeoffs between hardware efficiency and application accuracy. For CIFAR-10 and Tiny ImageNet, our models achieve 0.9% and 18.57% higher accuracy at 30% and -10.47% lower EDAP (energy-delay-area product), compared to baseline ResNet-20 and ResNet-18 models, respectively.

  10. Kosta, Adarsh, Efstathia Soufleri, Indranil Chakraborty, Amogh Agrawal, Aayush Ankit, and Kaushik Roy. "HyperX: A Hybrid RRAM-SRAM partitioned system for error recovery in memristive Xbars." Design, Automation & Test in Europe Conference & Exhibition (DATE), 2022.

    Abstract: Memristive crossbars based on Non-volatile Memory (NVM) technologies such as RRAM, have recently shown great promise for accelerating Deep Neural Networks (DNNs). They achieve this by performing efficient Matrix-Vector-Multiplications (MVMs) while offering dense on-chip storage and minimal off-chip data movement. However, their analog nature of computing introduces functional errors due to non-ideal RRAM devices, significantly degrading the application accuracy. Further, RRAMs suffer from low endurance and high write costs, hindering on-chip trainability. To alleviate these limitations, we propose HyperX, a hybrid RRAM-SRAM system that leverages the complementary benefits of NVM and CMOS technologies. Our proposed system consists of a fixed RRAM block offering area and energy-efficient MVMs and an SRAM block enabling on-chip training to recover the accuracy drop due to the RRAM non-idealities. The improvements are reported in terms of energy and product of latency and area (ms×mm2), termed as area-normalized latency. Our experiments on CIFAR datasets using ResNet-20 show up to 2.88 × and 10.1 × improvements in inference energy and area-normalized latency, respectively. In addition, for a transfer learning task from ImageNet to CIFAR datasets using ResNet-18, we observe up to 1.58 × and 4.48 × improvements in energy and area-normalized latency, respectively. These improvements are with respect to an all-SRAM baseline.

2. Device Technology Co-design (DTCO)

DTCO addresses the physical foundation of next-generation AI hardware by optimizing the device characteristics of beyond CMOS technologies (such as memristors, spintronics, and ferroelectrics) to meet the specific demands of neuromorphic and in-memory computing (IMC). This exploration allows us to mitigate device-level non-idealities through circuit-level techniques and algorithmic adaptation, enabling robust, ultra-low-power, and highly dense AI hardware.

Publications:
  1. Kim, Dong Eun, Tanvi Sharma, Anushka Mukherjee, Mainakh Mukherjee and Kaushik Roy. "MemRaptor: Magnetoresistive Array as Matrix Vector Multiplication and Transcendental Function Operator for NLP Applications." IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), 2025.

    Abstract: Compute-In-Memory (CiM) is emerging as a promising paradigm to design energy-efficient hardware accelerators for AI, addressing the processor-memory data transfer bottleneck. The popularity of CiM can be attributed to their ability to perform massively parallel in-situ matrix vector multiplications (MVMs), the dominant computation in neural networks (NNs). However, NNs used in NLP applications such as Long-Short Term Memory (LSTM) and transformers also frequently perform other operations such as transcendental functions (tanh, sigmoid, and softmax). To that effect, we present MemRaptor, utilizing CiM with magnetoresistive random access memory (MRAM) technology, that can perform both MVM and transcendental functions in the same memory array. MemRaptor overlays a read only memory (ROM) on an MRAM array through hard-wiring the connection of bit-cell with an additional bitline (a bit-cell connected to either of the bitlines but not to both), incurring no array area overhead and a minimal peripheral area overhead. Note, the bitline connection of bit-cell stores the ROM value while the magnetic tunnel junction (MTJ) in the bit-cell stores the RAM data. Particularly, the magnetization state of the 1T-1MTJ bit-cell in the array stores the weight value (RAM data) of the neural network, and the bitline connection of the bit-cell stores the look-up table (ROM data) used for computing transcendental functions. We demonstrate the working of our proposed design through circuit-level simulations for a 64×64 array, using a compact model of CoFeB/MgO PMA MTJ with 120% tunnelling magnetoresistance, 5k? RON , in 65nm technology. Further, we showcase the advantage of MemRaptor over standard MVM-based CiM accelerator architecture, PUMA, through comprehensive system-level evaluations for LSTM, BERT, and GPT models. Our results show up to 30% and 5.3% improvements in terms of throughput and energy-efficiency, respectively, on an average across different workloads.

  2. Yu, Eunseon, Gaurav K, Utkarsh Saxena, and Kaushik Roy. "Ferroelectric capacitors and field-effect transistors as in-memory computing elements for machine learning workloads." Scientific Reports, 2024.

    Abstract: This study discusses the feasibility of Ferroelectric Capacitors (FeCaps) and Ferroelectric Field-Effect Transistors (FeFETs) as In-Memory Computing (IMC) elements to accelerate machine learning (ML) workloads. We conducted an exploration of device fabrication and proposed system-algorithm co-design to boost performance. A novel FeCap device, incorporating an interfacial layer (IL) and (HZO), ensures a reduction in operating voltage and enhances HZO scaling while being compatible with CMOS circuits. The IL also enriches ferroelectricity and retention properties. When integrated into crossbar arrays, FeCaps and FeFETs demonstrate their effectiveness as IMC components, eliminating sneak paths and enabling selector-less operation, leading to notable improvements in energy efficiency and area utilization. However, it is worth noting that limited capacitance ratios in FeCaps introduced errors in multiply-and-accumulate (MAC) computations. The proposed co-design approach helps in mitigating these errors and achieves high accuracy in classifying the CIFAR-10 dataset, elevating it from a baseline of 10% to 81.7%. FeFETs in crossbars, with a higher on-off ratio, outperform FeCaps, and our proposed charge-based sensing scheme achieved at least an order of magnitude reduction in power consumption, compared to prevalent current-based methods.

  3. Sharma, Tanvi, Cheng Wang, Amogh Agrawal, and Kaushik Roy. "Enabling robust SOT-MTJ crossbars for machine learning using sparsity-aware device-circuit co-design." IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), 2021.

    Abstract: Embedded non-volatile memory (eNVM) based crossbars have emerged as energy-efficient building blocks for machine learning accelerators. However, the analog computations in crossbars introduce errors due to several non-idealities. Moreover, since communications between crossbars are usually done in the digital domain, the energy and area costs are dominated by the Analog-to-Digital Converters (ADC). Among the eNVM technologies, Resistive Random-Access-Memory (RRAM) and Phase-Change Memory (PCM) devices suffer from poor endurance, write variability and conductance drift. Whereas magneto-resistive technologies provide superior endurance, write stability and reliability. To that effect, we propose sparsity-aware device/circuit co-design of robust crossbars using Spin-Orbit-Torque Magnetic Tunnel Junctions (SOT-MTJs). Note, standard MTJs have low ROFF/RON and low RON, making them unsuitable for crossbars. In this work, we first demonstrate SOT-MTJs as crossbar elements with high RON and high ROFF/RON by allowing the read-path to have thicker tunneling-barrier, leaving the write path undisturbed. Second, through extensive simulations, we quantitatively assess the impact of various device-circuit parameters such as RON, ROFF/RON ratio, crossbar size, along with input and weight sparsity, on both circuit and application level accuracy and energy consumption. We evaluate system accuracy for Resnet-20 inference on CIFAR-10 dataset and show that leveraging sparsity allows reduced ADC precision, without degrading accuracy. Our results show that an SOT-MTJ (RON=200k? and ROFF/RON=7) crossbar array of size 32X32 could achieve near-software accuracy. The 64X64 and 128X128 crossbars show an accuracy degradation of 2% and 9.8%, respectively, from the software accuracy and an energy improvement of upto 3.8X and 6.3X compared to a 32X32 array with 4bit-ADC.

  4. Yu, Eunseon, Amogh Agrawal, Dongqi Zheng, Mengwei Si, Minsuk Koo, Peide D. Ye, Sumeet K. Gupta, and Kaushik Roy. "Ferroelectric FET Based Coupled-Oscillatory Network for Edge Detection." IEEE Electron Device Letters, 2021.

    Abstract: Coupled-oscillatory networks are an emerging paradigm for efficiently solving optimization problems. In this work, we demonstrate the application of ferroelectric field-effect transistor (FeFET) for energy-efficient coupled-oscillatory networks. A CMOS-compatible FeFET was fabricated having > 1V of hysteresis window and 57 mV/dec of minimum subthreshold swing. With our proposed FeFET oscillator circuits and optimized biasing schemes, a 2× wider synchronization range and up to 276× lower energy per cycle were achieved compared to previous FeFET-based oscillators. Moreover, we employ FeFET coupled-oscillatory network for an edge detection task. Our simulations considering FeFET non-idealities and process variations with a 5-bit quantized image show that the edge detection output closely follows the ideal output.

Click on the expansion arrows in each section to read the publication abstracts.

<<< Return to Current Research Outline