To address this problem, we have developed an innovative system to involve touchless interfaces for telesurgery. This type of solution, when applied to robotic surgery, has the potential to allow surgeons to operate as if they were physically engaged with the surgery in-situ (as standard in traditional surgery). By relying on touchless interfaces, the system can incorporate more natural gestures that are similar to instinctive movements performed by surgeons when operating, thus enhancing the user experience and overall system performance. Sensory substitution methods are used as well to deliver force feedback to the user during teleoperation.

Publications:
Zhou, Tian, Cabrera, Maria Eugenia, Low, Thomas, Sundaram, Chandru & Wachs, Juan (2016). A Comparative Study for Telerobotic Surgery Using Free Hand Gestures. Journal of Human-Robot Interaction, 5, 1-28.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan. (2016). A Comparative Study for Touchless Telerobotic Surgery. In Computer-Assisted Musculoskeletal Surgery (pp. 235-255). Springer International Publishing.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan (2015, January). Touchless telerobotic surgery-is it possible at all?. In AAAI (pp. 4228-4230).

Zhou, Tian., Cabrera, Maria., & Wachs, Juan, (2015) Communication Modalities for Supervised Teleoperation in Highly Dexterous Tasks – Does one size fit all?. In 2nd Workshop on the role of Human Sensorimotor Control in Surgical Robotics, in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE.

Zhou, Tian., Cabrera, Maria., & Wachs, Juan, (2014) Touchless telerobotic surgery – A comparative study. In 3rd Workshop on Telerobotics for Real-Life Applications, Opportunities, Challenges and New Developments, in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE.

Videos:

Incision task with Omega sensor

Peg transfer with Leap Motion

Threading task with Leap Motion

US hospitals are facing great shortage of registered nurses, which could lead to an increment in mortality rate. One solution to this challenge is to bring Robotic Scrub Nurse (RSN) into the Operating Room (OR) to free nurses from mundane and repetitive tasks such as instrument delivery and retrieval.

As an important building block for RSN, this paper presents an accurate and robust surgical instrument recognition algorithm. Surgical instruments are often cluttered, occluded and display specular light, which causes a challenge for conventional recognition algorithms. A learning-through-interaction paradigm was proposed to tackle the challenge, which combines computer vision with robot manipulation and achieves active recognition. The unknown instrument is firstly segmented out as blobs and its poses estimated, then the RSN system picks it up and presents it to an optical sensor in a determined pose. Lastly the unknown instrument is recognized with high confidence.

Experiments were then conducted to evaluate the performance of the proposed segmentation and recognition algorithms, respectively. It is found out that the proposed patch-based segmentation algorithm and attention-based recognition algorithm greatly outperform their benchmark comparisons, proving the applicability and effectiveness of a RSN to perform accurate and robust surgical instrument recognition tasks.

Publications:
Zhou, Tian., & Wachs, Juan. Finding a Needle in a Haystack: Recognizing Surgical Instruments through Vision and Manipulation. In SPIE/IS&T Electronic Imaging, no. 9, pp. 37–45, IS&T, 2017. Best Student Paper

Zhou, Tian, and Juan P. Wachs. “Needle in a haystack: Interactive surgical instrument recognition through perception and manipulation.” Robotics and Autonomous Systems 97 (2017): 182-192.

Videos:

The design and assessment of three surgical tools in two mock surgical procedures are showcased in this work. The surgical tasks involved tissue removal with the designed cutting tools, where their effectiveness and completion time are assessed. We conclude from these results, that the perpendicular scalpel tool is preferred for faster completion time, but the scissors are preferred for small tissue removal in terms of effectiveness.

Publication:

Sanchez-Tamayo, N., & Wachs, J. P. (2018, March). Collaborative Robots in Surgical Research: a Low-Cost Adaptation. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (pp. 231-232). ACM.

This framework includes the human processes associated with gesture perception and production. From the single gesture example, key points in the hands’ trajectories are extracted which have found to be correlated to spikes in visual and motor cortex activation. Those are also used to find inverse kinematic solutions to the human arm model, thus including the biomechanical and kinematic aspects of human production to artificially enlarge the number of gesture examples.

Leveraging these artificial examples, traditional state-of-the-art classification algorithms can be trained and used to recognize future instances of the same gesture class.

Brief Description: Humans tend to create the gestures on the fly and conventional machine learning systems lack adaptability to learn new gestures beyond the training stage. This problem can be best addressed using Zero Shot Learning (ZSL), a paradigm in machine learning that aims to recognize unseen objects by just having a description of them. ZSL for gestures has hardly been addressed in computer vision research due to the inherent ambiguity and the contextual dependency associated with the gestures. This work proposes an approach for Zero Shot Gestural Learning (ZSGL) by leveraging the semantic information that is embedded in the gestures.

The primary objective of this project is to research, develop, and validate an augmented reality system that would improve the effectiveness of telementoring between surgeons. Telementoring involves procedural guidance of a trainee surgeon by an expert surgeon from afar using telecommunication.

The STAR project is an innovative platform that relies on table and touchscreen displays, transparent screens, tablets, and color and depth sensors to increase the quality of the communication between mentor and trainee.

Institutions involved:

• Purdue University, School of Industrial Engineering
• Purdue University, School of Computer Science
• Indiana University, School of Medicine

With STAR, we want to increase the mentor and trainee sense of co-presence through an augmented visual channel that will lead to measurable improvements in the trainee’s surgical performance.

Our project has four specific aims. First, we will research, develop, and assess a transparent-display augmented-reality system that allows the seamless augmentation of a trainee surgeon’s natural view of the surgical field with annotations and illustrations of the current and next steps of the surgical procedure. Second, we will research, develop, and assess a patient-size interaction platform where the mentor can mark, annotate, and zoom in on anatomic regions using gestures performed over a projected image or on a multi-point touch screen. Third, we will validate and refine the proposed STAR platform in the context of practice cricothyroidotomy procedures on a human-patient simulator in a controlled environment. Fourth, we will validate the refined STAR platform in a simulated austere environment. The environment will match, for example, a military echelon Role 2 medical facility, consistent with a forward surgical team with limited resources. A hemorrhage-control procedure will be done within a damage-control laparotomy on a porcine model.

The STAR project has a military significance. There are three more areas where our proposed system has the potential to improve quantitative and qualitative outcomes in the military healthcare setting. First, telementoring can avoid or diminish the loss of surgical skills through retraining and competence. Second, it will be useful to provide instructional material in a simulation form, to support doctors serving in Iraq and Afghanistan in traumatic care in the battlefield, in a portable and dynamic style. Finally it will allow recent combat medic graduates to reinforce surgical techniques that were not conceptualized during their training curricula.

An innovative 3D joystick was developed to enable quadriplegics due to spinal cord injuries (SCIs) to more independently and efficiently operate a robotic arm as an assistive device. The 3D joystick was compared to two different manual input modalities, a keyboard control and a traditional joystick, in performing experimental robotic arm tasks by both subjects without disabilities and those with upper extremity mobility impairments. Fitts’s Law targeting and practical pouring tests were conducted to compare the performance and accuracy of the proposed 3D joystick. The Fitts’s law measurements showed that the 3D joystick had the best index of performance (IP), though it required an equivalent number of operations and errors as the standard robotic arm joystick. The pouring task demonstrated that the 3D joystick took significantly less task completion time and was more accurate than keyboard control. The 3D joystick also showed a decreased learning curve to the other modalities.

Methodology

The multimodal robotic arm user control systems consisted of three parts: a PC workstation, the different controller types (default joystick, keyboard or 3D joystick), and the actuated robotic arm (JACO) TM Robot Manipulator from Kinova Technology as shown in Fig. 1). The default controller for the JACO arm is a traditional joystick to control the movement of certain elements (i.e. arm, wrist) in two dimensions (see top of Fig. 2). Movement of the robotic arm in the 3rd dimension requires rotation of the joystick knob. This motion is extremely difficult or even impossible for individuals to perform with complete high-level (Cervical levels 1-8) SCIs.

Fig. 1.  JACO robotic arm ready to grasp a water bottle. It can also be
mounted to a wheelchair.

Two alternative modalities were developed in this project to serve as superior user controllers for this robotic arm for quadriplegic users. The first alternative input method developed was through keyboard control (top of Fig. 2). Keyboards are widely used as a direct selection device for efficient and naturally intuitive operation. For keyboard operation, all the functions for robotic control were mapped to specific keystrokes (i.e up, down, left, right, forward, backward, change mode). Three keyboard input control modes were programmed: discrete, continuous and hybrid (a combination of discrete and continuous) modes. During discrete mode, the robotic arm moved in small increments
every time a key was pressed. During continuous mode, the arm would move continuously until stopped or another key to change directions was pressed. During hybrid mode, subjects could toggle between discrete and continuous modes at their discretion.

Fig.2. Subject with a SCI using the 3D joystick to perform the pouring task.

The other alternative control modality was a 3D joystick (Fig. 3) which was originally designed for haptic video game playing by Falcon Technology®. It was reprogrammed and adapted as a 3D joystick controller for the robotic arm. A handle developed for users with no finger gripping ability was positioned in the center of the joystick. The 3D joystick provides users a method of directed selection to control the robotic arm elements to move in 3D Euclidean space. The handle of 3D joystick was positioned at the center of the joystick as a home (or rest) position if not used by the user.

Fig.3.  3D joystick with adapted handle for quadriplegic users.

A force feedback control with a proportional and differential (PD) controller force the handle back to the center after each manipulation. The control diagram for 3D haptic joystick is shown in Fig. 4. A JACO API was used to fictionalize the haptic joystick to achieve 3D control of the robotic arm.

Fig.4.  3D joystick control diagram.

An integrated, computer vision-based system was developed to operate a commercial wheelchair-mounted robotic manipulator (WMRM). In this paper, a gesture recognition interface system developed specifically for individuals with upper-level spinal cord injuries (SCIs) was combined with object tracking and face recognition systems to be an efficient, hands free WMRM controller. In this test system, two Kinect cameras were used synergistically to perform a variety of simple object retrieval tasks. One camera was used to interpret the hand gestures to send as commands to control the WMRM and locate the operator’s face for object positioning. The other sensor was used to automatically recognize different daily living objects for test subjects to select. The gesture recognition interface incorporated hand detection, tracking and recognition algorithms to obtain a high recognition accuracy of 97.5% for an eight-gesture lexicon. An object recognition module employing Speeded Up Robust Features (SURF) algorithm was performed and recognition results were sent as a command for “coarse positioning” of the robotic arm near the selected daily living object. Automatic face detection was also provided as a shortcut for the subjects to position the objects to the face by using a WMRM. Completion time tasks were conducted to compare manual (gestures only) and semi manual (gestures, automatic face detection and object recognition) WMRM control modes. The use of automatic face and object detection significantly increased the completion times for retrieving a variety of daily living objects.

System Architecture

The architecture of the proposed system is illustrated in Figure 1. Two Kinect® video cameras were employed and served as inputs for the gesture recognition and object detection modules respectively. The results of these two modules were then passed as commands to the execution modules to control the JACO robotic arm (Kinova, Inc., Montréal, Canada). Briefly, these modules are described as follows:

A. Gesture Recognition Module

The video input from Kinect camera was processed in four stages using for gesture recognition based WMRM system control; foreground segmentation, hand detection, tracking, and hand trajectory recognition stage. Foreground segmentation was used to increase computational efficiency by reducing search range for hand detection and later stage process. The face and hands were detected from the foreground which provided an initialization region for hand tracking stage. The tracked trajectories were then segmented and compared to the pre-constructed motion models and classified them as certain gesture groups. The recognized gesture was then encoded and passed as command to control the WMRM.

B. Object Recognition Module

The goal of the object recognition module is to detect the different daily living objects and assign a unique identifier for each of these objects. A template was created for each object being recognized. These templates were compared to each frame in the video sequence to obtain the best matching object. The results were then encoded and passed as commands to position the robotic manipulator.

C. Automatic Face Detection Module

A face detector was employed in this module to perform automatic face detection. The goal was to provide a shortcut for the subjects to position the objects to the front of the face by controlling the robotic arm.

D. Execution Module

The robotic arm was programmed as a wrapper using JACO API under C# environment which was then called by the main program. The JACO robotic arm was mounted to the seat frame of a motorized wheelchair. The robotic arm wascontrolled by the encoded commands from gesture recognition, automatic face detection and object recognition module.

 

Fig. 1. System Architecture

I. Introduction

From the 2011 National Health Interview Survey (NHIS) Preliminary Report, it is estimated that 21.2 million adult Americans, namely more than 10% of all adult Americans have trouble seeing. Among the  6.6 million working-age adults with BVI, 64% did not finish high school and approximately only 6% earned a Bachelor’s or higher degree [1]. The lack of proper and effective assistive technologies (AT) can be considered as a major roadblock for individuals that are BVI to actively participate in science and advanced research activities [2]. It is still a challenge for them to perceive and understand scientific visual data acquired during wet lab experimentation, such as viewing live specimens through a stereo microscope or histological samples through light microscopy (LM). According to Science and Engineering Indicator 2014 published by NSF, no more than 1% of blind or visually impaired people are involved in advanced science and engineering research and receive doctoral degrees [3].

By using current single-modality human-computer interfaces (HCI), only limited visual information can be accessed due to different limitations of each sense. Although, tactile-vision sensory substitution (TVSS) technologies, such as Tongue electrotactile array [4], and tactile pictures [5], have been demonstrated capable of conveying visual information [6] of spatial phenomenology [7], the low resolution of somatosensory display arrays have always been a limitation of these methods to convey complex image information. Auditory-vision sensory substitution has also been studied in image perception [8], [9]. Trained early blind participants showed increased performance in localization and object recognition [10] through this substitution. Auditory-vision substitution always involves the memorization of different audio forms and training is required to map from different audio stimulus to visual cues. In addition, the focus on auditory feedback can decrease subjects’ ability to get information from the environment [11]. The current gap is that existing solutions cannot help conveying the richness, complexity and amount of data available to users without disabilities. In this study, a real-time multimodal image perception approach is investigated that incorporates the feedback to multiple sensory channels, including auditory, haptics and vibrotactile. Through the integration of multiple sensorial substitutions, participants supported using the proposed platform showed higher analytic performance than when using the standard interface based on one sensory feedback only.

]]> https://Engineering.Purdue.Edu/isat/multimodal-image-perception-system-for-blind-or-visually-impaired-people/feed/ 0