Asim Unmesh, Rahul Jain, Jingyu Shi, V. K. Chaithanya Manam, Hyung-Gun Chi, Subramanian Chidambaram, Alexander J. Quinn, Karthik Ramani

IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 451-458, Jan. 2024

Scene Samples

Object-Object Interaction Annotations

Abstract

Dynamic environments in factories, surgical robotics, and warehouses increasingly involve humans, machines, robots, and various other objects such as tools, fixtures, conveyors, and assemblies. In these environments, numerous interactions occur not just between humans and objects but also between objects themselves. However, current scene-graph datasets predominantly focus on human-object interactions (HOI) and overlook object-object interactions (OOIs) despite the necessity of OOIs in effectively representing dynamic environments. This oversight creates a significant gap in the coverage of interactive elements in dynamic scenes. We address this gap by proposing, to the best of our knowledge, the first dataset* annotating for OOI categories in dynamic scenes. To model OOIs, we establish a classification taxonomy for spatio-temporal interactions. We use our taxonomy to annotate OOIs in video clips of dynamic scenes. Then, we introduce a spatio-temporal OOI classification task which aims to identify interaction categories between two given objects in a video clip. Further, we benchmark our dataset for the spatio-temporal OOI classification task by adopting state-of-the-art approaches from related areas of Human-Object Interaction Classification, Visual Relationship Classification, and Scene-Graph Generation. Additionally, we utilize our dataset to examine the effectiveness of OOI and HOI-based features in the context of Action Recognition. Notably, our experimental results show that OOI-based features outperform HOI-based features for the task of Action Recognition.

Download

You can download the dataset here.

The paper is available at IEEE RA-L page.

Cite

If you use this dataset in your research, please cite it as follows:

@article{unmesh2023interacting,
  title={Interacting objects: A dataset of object-object interactions for richer dynamic scene representations},
  author={Unmesh, Asim and Jain, Rahul and Shi, Jingyu and Manam, VK Chaithanya and Chi, Hyung-Gun and Chidambaram, Subramanian and Quinn, Alexander and Ramani, Karthik},
  journal={IEEE Robotics and Automation Letters},
  volume={9},
  number={1},
  pages={451--458},
  year={2023},
  publisher={IEEE}
}

Contact

For any questions or further information, please contact us at ramani@purdue.edu.