Home

homepage of the project sponsered by the agency for Healthcare research and quality(ahrq)

Currently, picture archiving and communications systems (PACS) in the OR available in computer workstations in operating rooms (OR) are accessible through traditional interfaces, e.g., touchscreens, keyboards, and mice. However, these can compromise sterility and spread infection. Gestures offer an attractive alternative for surgeon interaction, without compromising the sterile environment. The main obstacle with hand gesture based interfaces is the lack of analytical tools to determine the most efficient and useable gestures that are best fitted to surgical tasks. Currently, the most common practice is for a single individual (usually a system developer) to decide which gestures should be used by all surgeons. The disadvantage is that designers may not consider usability factors specific to image manipulation during surgery, or may only consider gestures suitable to a single surgeon. We are developing a highly efficient, usable methodology for designing suitable gesture-based interfaces for the operating room. Using a rigorous methodology that includes input from an expert panel of surgeons on PACS requirements and operations, vocabulary acceptability criterias (VAC) have been identified. Now we are developing a gesture recognition interface to measure the usability metrics associted with gesture lexicons. In addition to offering an innovative sterile alternative that will promote a decrease in the spread of infection, which represents a substantial savings in medical costs, the proposed research will also improve the understanding of how natural means of interaction affect doctors’ performance in the OR. Improved understanding of the factors affecting design and use of hand gesture interfaces, are key to introducing such interfaces in demanding, time-sensitive tasks, like the OR.

 

Institutions Involved

 

Acknowledgement


The team acknowledges the support and sponsorship of our research by the Agency for Healthcare Research and Quality (AHRQ), under award no. 1R18HS024887.

Studies

Study I: Finding functions of PACS

In this study, typical functions of the Picture Archiving Communication Systems (PACS) were determined. Synapse, a popularly used PACS radiology image browser was used. Three medical image manipulation tasks encompassing most of the functionalities of Synapse® (the PACS system utilized) were considered. Initially, each neurosurgeon was asked to accomplish these tasks using keyboard and mouse interfaces. By tracking the menu choices, mouse motions and selections, all functions required to complete the tasks using the PACS system were collected. The outcome of this study was a list of 34 PACS commands.

Protocol for information acquisition from the three imaging tasks



Experimental setup for study I



List of 34 PACS commands



Study II: Finding surgeon gestures for PACS commands

In this study, we obtained the gestural preference of the neurosurgeons for the 34 commands found in the study I. Initially, the subjects were asked to follow three predetermined steps in the same order: 1) Gesture design on a drawing sheet: required subjects to design and draw the gestures corresponding to each of the 34 commands of Synapse on a drawing sheet. 2) Gesture illustration: required the subjects to perform each of the chosen gestures in front of Microsoft Kinect v2. 3) Manipulation task using Wizard of Oz setup: required the subjects to perform a tumor identification task using the chosen gestures, following a Wizard of Oz experimental setup. Outcome of the experiment was 9 gesture lexicons each containing 28 gestures.



Gesture designed by a subject on drawing sheet



Wizard of Oz Setup



Study III: Develop Vocabulary Acceptability Criteria for gestures

The goal of this study was to develop Vocabulary Acceptability Criteria (VACs) that can potentially explain the qualitative aspects of the gestures. Further, the obtained gestures in study II were compared, ranked and evaluated using these VACs in order to obtain a best gesture lexicon. This study consists of three major parts: Part I – Creation, Part II – Evaluation and Part III – Validation. Six VACs were created in thsi study: Iconicity, Saliency, Efficiency, Compactness, Complexity, Economy of Movement.



Six Orthogonal VACs



Web interface for gesture evaluation based on VACs



Web interface for validation of VACs



Average VAC score per lexicon





Publications

Gestures for Picture Archiving and Communication Systems (PACS) operation in the Operating room: Is there any standard?

Abstract: Gestural interfaces allow accessing and manipulating Electronic Medical Records (EMR) in hospitals while keeping a complete sterile environment. Particularly, in the Operating Room (OR), these interfaces enable surgeons to browse Picture Archiving and Communication System (PACS) without the need of delegating functions to the surgical staff. Existing gesture based medical interfaces rely on a suboptimal and an arbitrary small set of gestures that are mapped to a few commands available in PACS software. The objective of this work is to discuss a method to determine the most suitable set of gestures based on surgeon's acceptability. To achieve this goal, the paper introduces two key innovations: (a) a novel methodology to incorporate gestures' semantic properties into the agreement analysis, and (b) a new agreement metric to determine the most suitable gesture set for a PACS.

Madapana N, Gonzalez G, Rodgers R, Zhang L, Wachs JP (2018) Gestures for Picture Archiving and Communication Systems (PACS) operation in the operating room: Is there any standard? PLoS ONE 13(6): e0198092.

Supplementary Material: Dataset and code associted with the paper can be found at - https://github.com/glebysg/gestural_PACS



Looking Beyond the Gesture: Vocabulary Acceptability Criteria for Gesture Elicitation Studies

Abstract: The choice of what gestures should be part of a gesture language is a critical step in the design of gesture-based interfaces. This step is especially important when time and accuracy are key factors of the user expe-rience, such as gestural interfaces in vehicle control and sterile control of a picture archiving and communi-cation system (PACS) in the operating room (OR). Agreement studies are commonly used to find the gesture preference of the end users. These studies hypothesize that the best available gesture lexicon is the one pre-ferred by a majority. However, these agreement approaches cannot offer a metric to assess the qualitative aspects of gestures. In this work, we propose an experimental framework to quantify, compare and evaluate gestures. This framework is grounded in the expert knowledge of speech and language professionals (SLPs). The development consisted of three studies: 1) Creation, 2) Evaluation and 3) Validation. In the creation study, we followed an adapted version of the Delphi’s interview/discussion procedure with SLPs. The pur-pose was to obtain the Vocabulary Acceptability Criteria (VAC) to evaluate gestures. Next, in the evaluation study, a modified method of pairwise comparisons was used to rank and quantify the gestures based on each criteria (VAC). Lastly, in the validation study, we formulated an odd one out procedure, to prove that the VAC values of a gesture are representative and sufficiently distinctive, to select that particular gesture from a pool of gestures. We applied this framework to the gestures obtained from a gesture elicitation study con-ducted with nine neurosurgeons, to control an imaging software. In addition, 29 SLPs comprising of 17 ex-perts and 12 graduate students participated in the VAC study. The best lexicons from the available pool were obtained through both agreement and VAC metrics. We used binomial tests to show that the results obtained from the validation procedure are significantly better than the baseline. These results verify our hypothesis that the VAC are representative of the gestures and the subjects should be able to select the right gesture given its VAC values.

Gonzalez, G., Madapana, N., Taneja, R., Zhang, L., Rodgers, R. B., Wachs, J. P. (2018). Looking Beyond the Gesture: Vocabulary Acceptability Criteria for Gesture Elicitation Studies. In press of the Human Factors and Ergonomics Society Annual Meeting.



Hard Zero Shot Learning for Gesture Recognition

Abstract: Gesture based systems allow humans to interact with devices and robots in a natural way. Yet, current gesture recognition systems can not recognize the gestures outside a limited lexicon. This opposes the idea of lifelong learning which require systems to adapt to unseen object classes. These issues can be best addressed using Zero Shot Learning (ZSL), a paradigm in machine learning that leverages the semantic information to recognize new classes. ZSL systems developed in the past used hundreds of training examples to detect new classes and assumed that test examples come from unseen classes. This work introduces two complex and more realistic learning problems referred as Hard Zero Shot Learning (HZSL) and Generalized HZSL (G-HZSL) necessary to achieve Life Long Learning. The main objective of these problems is to recognize unseen classes with limited training information and relax the assumption that test instances come from unseen classes. We propose to leverage one shot learning (OSL) techniques coupled with ZSL approaches to address and solve the problem of HZSL for gesture recognition. Further, supervised clustering techniques are used to discriminate seen classes from unseen classes. We assessed and compared the performance of various existing algorithms on HZSL for gestures using two standard datasets: MSRC-12 and CGD2011. For four unseen classes, results show that the marginal accuracy of HZSL - 15.2% and G-HZSL - 14.39% are comparable to the performance of conventional ZSL. Given that we used only one instance and do not assume that test classes are unseen, the performance of HZSL and G-HZSL models were remarkable.

Madapana, N., Wachs, J. P. (2018). Hard Zero Shot Learning for Gesture Recognition. In press at 2018 24th International Conference on Pattern Recognition (ICPR)



ZSGL: zero shot gestural learning

Abstract: Gesture recognition systems enable humans to interact with machines in an intuitive and a natural way. Humans tend to create the gestures on the fly and conventional systems lack adaptability to learn new gestures beyond the training stage. This problem can be best addressed using Zero Shot Learning (ZSL), a paradigm in machine learning that aims to recognize unseen objects by just having a description of them. ZSL for gestures has hardly been addressed in computer vision research due to the inherent ambiguity and the contextual dependency associated with the gestures. This work proposes an approach for Zero Shot Gestural Learning (ZSGL) by leveraging the semantic information that is embedded in the gestures. First, a human factors based approach has been followed to generate semantic descriptors for gestures that can generalize to the existing gesture classes. Second, we assess the performance of various existing state-of-the-art algorithms on ZSL for gestures using two standard datasets: MSRC-12 and CGD2011 dataset. The obtained results (26.35% - unseen class accuracy) parallel the benchmark accuracies of attribute-based object recognition and justifies our claim that ZSL is a desirable paradigm for gesture based systems.

Madapana, N., & Wachs, J. (2017a). ZSGL: zero shot gestural learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction (pp. 331–335).



A semantical & analytical approach for zero shot gesture learning

Abstract: Zero shot learning (ZSL) is about being able to recognize gesture classes that were never seen before. This type of recognition involves the understanding that the presented gesture is a new form of expression from those observed so far, and yet carries embedded information universal to all the other gestures (also referred as context). As part of the same problem, it is required to determine what action/command this new gesture conveys, in order to react to the command autonomously. Research in this area may shed light to areas where ZSL occurs, such as spontaneous gestures. People perform gestures that may be new to the observer. This occurs when the gesturer is learning, solving a problem or acquiring a new language. The ability of having a machine recognizing spontaneous gesturing, in the same manner as humans do, would enable more fluent human-machine interaction. In this paper, we describe a new paradigm for ZSL based on adaptive learning, where it is possible to determine the amount of transfer learning carried out by the algorithm and how much knowledge is acquired from a new gesture observation. Another contribution is a procedure to determine what are the best semantic descriptors for a given command and how to use those as part of the ZSL approach proposed.

Madapana, N., & Wachs, J. P. (2017b). A semantical & analytical approach for zero shot gesture learning. In Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference on (pp. 796–801).



Pictures



People

Investigators


    Juan Wachs
Principal Investigator
    Richard B. Rodgers
Co-Investigator
    Lingsong Zhang
Co-Investigator

Research Assistants


Glebys Gonzalez
Naveen Madapana
Rahul Taneja

Contact Us

Please direct general inquiries about the GestureClean project to the project’s principal investigator:


Juan P. Wachs, Ph.D.
Regenstrief Center for Healthcare Engineering
Associate Professor, School of Industrial Engineering
Purdue University
315 N. Grant Street
West Lafayette, IN 47907
765 496-7380 (tel)
765 494-1299 (fax)
e-mail: jpwachs (at) purdue (dot) edu