Research: Learning Algorithms and Optimization

Learning Algorithms and Optimization

D. Video Processing Algorithms

Our research develops efficient learning algorithms for understanding complex video data by capturing spatial and long-range temporal information. We explore action recognition, temporal localization, and procedural activity understanding using transformer-based architectures and self-, weakly-, and unsupervised learning to reduce annotation and computational costs. These methods enable robust and efficient video understanding across challenging domains such as surgical workflows, egocentric videos, and human activity analysis.

Publications:

1. Soufleri, Efstathia, Deepak Ravikumar, and Kaushik Roy. "Advancing Compressed Video Action Recognition through Progressive Knowledge Distillation." arXiv preprint arXiv:2407.02713 (2024).

   Abstract: Compressed video action recognition classifies video samples by leveraging the different modalities in compressed videos, namely motion vectors, residuals, and intra-frames. For this purpose, three neural networks are deployed, each dedicated to processing one modality. Our observations indicate that the network processing intra-frames tend to converge to a flatter minimum than the network processing residuals, which in turn converges to a flatter minimum than the motion vector network. This hierarchy in convergence motivates our strategy for knowledge transfer among modalities to achieve flatter minima, which are generally associated with better generalization. With this insight, we propose Progressive Knowledge Distillation (PKD), a technique that incrementally transfers knowledge across the modalities. This method involves attaching early exits (Internal Classifiers - ICs) to the three networks. PKD distills knowledge starting from the motion vector network, followed by the residual, and finally, the intra-frame network, sequentially improving IC accuracy. Further, we propose the Weighted Inference with Scaled Ensemble (WISE), which combines outputs from the ICs using learned weights, boosting accuracy during inference. Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively. Additionally, WISE improves accuracy by up to 4.28% and 9.30% on UCF-101 and HMDB-51, respectively.

2. Chandra, Soumyadeep, Sayeed Shafayet Chowdhury, Courtney Yong, Chandru P. Sundaram, and Kaushik Roy. "ViPeR: Vision-Based Surgical Phase Recognition." IEEE Access, vol. 13, pp. 122408-122419, (2025). CODE AVAILABLE HERE

   Abstract: Surgical phase recognition is a critical, yet challenging, problem in computer vision, with significant implications for automated surgical training, intraoperative assistance, and workflow optimization. However, the development of robust models is hindered by the scarcity of well-annotated medical datasets and the complexity of surgical workflows, which exhibit substantial inter- and intra-procedural variations. To address these challenges, we introduce UroSlice, a novel complex dataset that focuses on nephrectomy surgeries, capturing both radical and partial procedures performed using robotic-assisted techniques. In order to address the task of phase recognition in these videos, we propose a novel model named "ViPeR" (Vision-based Surgical Phase Recognition). Our model incorporates hierarchical dilated temporal convolution layers and inter-layer residual connections to capture temporal correlations at both fine and coarse granularities. Experimental evaluations validates that our approach achieves state-of-the-art performance of 91.7% on Cholec80 and 66.4% on UroSlice - a more challenging dataset due to its irregular phase durations, non-standardized phase order, and a smaller sample size.

3. Chowdhury, Sayeed Shafayet, Soumyadeep Chandra, and Kaushik Roy. "Opel: Optimal transport guided procedure learning." Advances in Neural Information Processing Systems (NeurIPS) (2024): 59984-60011.

   Abstract: Procedure learning refers to the task of identifying the key-steps and determining their logical order, given several videos of the same task. For both third-person and first-person (egocentric) videos, state-of-the-art (SOTA) methods aim at finding correspondences across videos in time to accomplish procedure learning. However, to establish temporal relationships within the sequences, these methods often rely on frame-to-frame mapping, or assume monotonic alignment of video pairs, leading to sub-optimal results. To this end, we propose to treat the video frames as samples from an unknown distribution, enabling us to frame their distance calculation as an optimal transport (OT) problem. Notably, the OT-based formulation allows us to relax the previously mentioned assumptions. To further improve performance, we enhance the OT formulation by introducing two regularization terms. The first, inverse difference moment regularization, promotes transportation between instances that are homogeneous in the embedding space as well as being temporally closer. The second, regularization based on the KL-divergence with an exponentially decaying prior smooths the alignment while enforcing conformity to the optimality (alignment obtained from vanilla OT optimization) and temporal priors. The resultant optimal transport guided procedure learning framework ('OPEL') significantly outperforms the SOTA on benchmark datasets. Specifically, we achieve 22.4% (IoU) and 26.9% (F1) average improvement compared to the current SOTA on large scale egocentric benchmark, EgoProceL. Furthermore, for the third person benchmarks (ProCeL and CrossTask), the proposed approach obtains 46.2% (F1) average enhancement over SOTA.

4. Chandra, Soumyadeep, and Kaushik Roy. "REMAP: Regularized Matching and Partial Alignment of Video Embeddings." arXiv preprint arXiv:2509.24382 (2025).

   Abstract: Real-world instructional videos are long, noisy, and often contain extended background segments, repeated actions, and execution variability that do not correspond to meaningful procedural steps. We propose REMAP, an unsupervised framework for procedure learning based on Regularized Fused Partial Gromov-Wasserstein Optimal Transport. REMAP relaxes balanced transport constraints, allowing non-informative or redundant frames to remain unmatched through partial transport. The formulation jointly models semantic similarity and temporal structure, while incorporating Laplacian-based smoothness and structural regularization to prevent degenerate alignments and reduce background interference. We evaluate REMAP on large-scale egocentric and third-person benchmarks. The method consistently outperforms state-of-the-art approaches, achieving up to 11.6% (+4.45pp) F1 and 19.6% (+4.73pp) IoU improvements on EgoProceL, and an average 41% (+17.15pp) F1 gain on ProceL and CrossTask. These results highlight the importance of partial alignment in handling real-world procedural variability and demonstrate that REMAP provides a robust and scalable approach for instructional video understanding.

Click on the expansion arrows in each section to read the publication abstracts.