To address this problem, we have developed an innovative system to involve touchless interfaces for telesurgery. This type of solution, when applied to robotic surgery, has the potential to allow surgeons to operate as if they were physically engaged with the surgery in-situ (as standard in traditional surgery). By relying on touchless interfaces, the system can incorporate more natural gestures that are similar to instinctive movements performed by surgeons when operating, thus enhancing the user experience and overall system performance. Sensory substitution methods are used as well to deliver force feedback to the user during teleoperation.

Publications:
Zhou, Tian, Cabrera, Maria Eugenia, Low, Thomas, Sundaram, Chandru & Wachs, Juan (2016). A Comparative Study for Telerobotic Surgery Using Free Hand Gestures. Journal of Human-Robot Interaction, 5, 1-28.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan. (2016). A Comparative Study for Touchless Telerobotic Surgery. In Computer-Assisted Musculoskeletal Surgery (pp. 235-255). Springer International Publishing.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan (2015, January). Touchless telerobotic surgery-is it possible at all?. In AAAI (pp. 4228-4230).

Zhou, Tian., Cabrera, Maria., & Wachs, Juan, (2015) Communication Modalities for Supervised Teleoperation in Highly Dexterous Tasks – Does one size fit all?. In 2nd Workshop on the role of Human Sensorimotor Control in Surgical Robotics, in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan, (2014) Touchless telerobotic surgery – A comparative study. In 3rd Workshop on Telerobotics for Real-Life Applications, Opportunities, Challenges and New Developments, in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE.

Videos:

Incision task with Omega sensor

Peg transfer with Leap Motion

Threading task with Leap Motion

US hospitals are facing great shortage of registered nurses, which could lead to an increment in mortality rate. One solution to this challenge is to bring Robotic Scrub Nurse (RSN) into the Operating Room (OR) to free nurses from mundane and repetitive tasks such as instrument delivery and retrieval.

As an important building block for RSN, this paper presents an accurate and robust surgical instrument recognition algorithm. Surgical instruments are often cluttered, occluded and display specular light, which causes a challenge for conventional recognition algorithms. A learning-through-interaction paradigm was proposed to tackle the challenge, which combines computer vision with robot manipulation and achieves active recognition. The unknown instrument is firstly segmented out as blobs and its poses estimated, then the RSN system picks it up and presents it to an optical sensor in a determined pose. Lastly the unknown instrument is recognized with high confidence.

Experiments were then conducted to evaluate the performance of the proposed segmentation and recognition algorithms, respectively. It is found out that the proposed patch-based segmentation algorithm and attention-based recognition algorithm greatly outperform their benchmark comparisons, proving the applicability and effectiveness of a RSN to perform accurate and robust surgical instrument recognition tasks.

Publications:
Zhou, Tian., & Wachs, Juan. Finding a Needle in a Haystack: Recognizing Surgical Instruments through Vision and Manipulation. In SPIE/IS&T Electronic Imaging, no. 9, pp. 37–45, IS&T, 2017. Best Student Paper

Zhou, Tian, and Juan P. Wachs. “Needle in a haystack: Interactive surgical instrument recognition through perception and manipulation.” Robotics and Autonomous Systems 97 (2017): 182-192.

Videos:

Students are nominated for the award by the faculty based on numerous factors, including peer-reviewed publications, awards received for presentations or publications, potential contribution to the field and society, leadership in a research group or community, participation in professional societies, and overall academic achievement.

She was recognized at the 2018 College of Engineering Graduate Student Research, Teaching, and Service Awards Luncheon on April 24th.

Glebys Gonzales and Natalia Sanchez Tamayo demonstrated ABB YUMI, with 3D-printed surgical gripper, performed suture needle manipulation.

Maria Cabrera demonstrated the telementoring system through Augmented Reality on the Hololens.

Rojas gave a talk regarding the current state of the System for Telementoring with Augmented Reality (STAR). The talk included a comparison between two different STAR versions: a Tablet-based and a Head-Mounted Display.

A poster summarizing the talk can be found here.

Prof. Okamura’s research focuses on developing the principles and tools needed to realize advanced robotic and human-machine systems capable of haptic (touch) interaction, particularly for biomedical applications. Haptic systems are designed and studied using both analytical and experimental approaches.

As a part of her visit, Prof. Okamura visited the ISAT lab where graduate students presented some of the research projects currently active.

Tian Zhou, a PhD Candidate, showed his work involving the WAM Robot picking and delivering surgical instruments and his research regarding turn-taking in human-robot collaboration.

Natalia Sanchez and Glebys Gonzalez presented the collaborative robot YuMi (ABB) which has been adapted through rapid prototyping, with extended grippers for research in robotic assisted surgery. Additionally, Prof. Okamura tested the HTC Vive virtual reality headset and controllers.

To finalize the visit, Prof. Okamura participated as an impromptu trainee to showcase the functionality of the STAR telementoring system, wearing the HoloLens (Microsoft), through which annotations are overlayed using augmented reality in the user’s field of view.

Zhou, T., Cha, J. S., Gonzalez, G. T., Wachs, J. P., Sundaram, C., & Yu, D. (2018, March). Joint Surgeon Attributes Estimation in Robot-Assisted Surgery. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (pp. 285-286). ACM.

Proceeding
Poster

Cabrera, M. E., Voyles, R. M., & Wachs, J. P. (2018, March). Coherence in One-Shot Gesture Recognition for Human-Robot Interaction. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (pp. 75-76). ACM.

Proceeding
Poster

Sanchez-Tamayo, N., & Wachs, J. P. (2018, March). Collaborative Robots in Surgical Research: a Low-Cost Adaptation. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (pp. 231-232). ACM.

Proceeding
Poster

A detailed post can be found here: https://engineering.purdue.edu/IE/news/2018/zhang-2nd-place