Our research is motivated by recognition that design of effective computing systems requires understanding of the relationships between people and technologies. A primary goal of human-centered computing is to strike the appropriate balance between human and machine capabilities and to integrate them in such a way that the system naturally supports the human end user. We focus on research towards intelligent human-computer interaction-using computers to enhance human performance involving complex cognitive activities, for example during visual reasoning and decision-making. Intelligent human-computer interaction combines human cognitive, physical, and affective factors to model human behavior for the design of adaptive systems.

There are several research threads in the HCM3 lab at RIT. These share a common experimental framework based on the idea that human behavior is both observable (e.g. motor, verbal, visual) and hidden (mental states) and that we can capture and model representative data via a variety of physical and physiological sensors. Our interdisciplinary group integrates methods and approaches of human-centered design, visual perception, imaging science, and computational linguistics to study the influence of perceptual expertise and domain knowledge on complex visual tasks. The HCM3 closely collaborates with the Multidisciplinary Vision Research Lab (MVRL) in RIT’s Center for Imaging Science.

1. Multimodal Interaction for Semantic Image Retrieval

The goal of this cross-disciplinary research project is to develop a multi-modal, explorable, retrieval system for biomedical images to allow users to interactively retrieve and analyze images that span a range of visual and semantic variation. We use an interactive human-centered approach to define visual regions of interest, and their relationships to image content  and verbally-expressed domain knowledge in order to define new categorizations of images and to create semantic metadata. System design (database organization, metadata, user interface design..) is then informed by the expert-derived visual and verbal data, inferred user models and data (images/annotations) models. 

-Supported by NSF grant IIS-0941452 CDI. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

 

Research tracks:

(1) Multimodal System Design

Ongoing studies analyze interaction styles for usable, semantic image retrieval.

 

(2) User Modeling

A hierarchical computational cognitive model based on eye movement data distinguishes observers’ visual behaviors based on expertise and type of image and is used for real-time interaction in the multimodal system.

(3) Semantic annotation and linguistic mining

Techniques of NLP and computational linguistics are used for annotation of transcribed verbal narratives to identify terms and concepts for semantic metadata and infer models of domain-knowledge driven image organization.

 

2. Biomedical Image Understanding and Image-based Diagnostic Reasoning

Image understanding, from the human cognitive perspective, is a key limiting factor in advancing use of medical images. To understand the capabilities of the human visual system with respect to biomedical imaging and how visual reasoning strategies are influenced by domain knowledge and expertise we study the relationships between visual strategies (eye movements), related image content and users' perceptual categorizations as expressed through natural language.  This research has provided proof of concept of the value of eliciting tacit knowledge from domain experts through multiple modalities to integrate data and knowledge models for better image understanding.  Study of medical personnel with different levels of training has revealed expertise-related image-based diagnostic strategies and different reasoning styles.

-Supported by NIH 1 R21 LM010039-01A1. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

 

Research tracks:

(1) Expertise-dependent medical image inspection strategies

Level of training/domain knowledge and experience influence visual behaviors in medical image-based diagnostic reasoning.

(2) Gaze-based analysis of image features influencing human medical image understanding

(3) Relationships of diagnostic confidence and correctness

 

(4) Difference in diagnostic reasoning styles and their relationships to learning and medical error prevention

 

3. Undergraduate Research Activities

The "Research Experiences for Undergraduates (REU) for computational sensing" program is a yearly activity (2016-2018) for undergraduate students to acquire research experience related to human computation and data modeling regarding particular real world research problems. It is sponsored by the National Science Foundation (NSF) and Rochester Institute of Technology (RIT).

Below is a list of publications and presentations by undergraduate researchers (the list is still growing!). Undergraduate researchers are labeled '*'. The publications with undergraduate authors in the above research projects are also labeled.

2014:

Bullard, J.*, et al. (2014) Towards multimodal modeling of physicians' diagnostic confidence and self-awareness using medical narratives. In Proceedings of COLING, the 25th International Conference on Computational Linguistics: Technical Papers (pp. 1718-1727).

Womack, K.*, et al. (2014, June) Toward Automatic Extraction of Semantic Units of Thought in Physician Reasoning Processes. 15th International Clinical Phonetics and Linguistics Association Conference. Stockholm, Sweden. June 2014.

2013:

Womack, K.*, et al. (2013, August) Markers of Confidence and Correctness in Spoken Medical Narratives. Proceedings of Interspeech 2013, Lyon, France. In print.

Womack, K.*, et al. (2013, August) Using Linguistic Analysis to Characterize Conceptual Units of Thought in Spoken Medical Narratives. Proceedings of Interspeech 2013, Lyon, France. In print.

Bullard, J.*, et al. (2013, August).Multimodal predictors of diagnostic confidence in medical narratives. RIT Undergraduate Research Symposium, August 2, 2013, Rochester, NY. 

Womack, K.*, et al. (2013, August). Markers of confidence and correctness in spoken medical narratives. RIT Undergraduate Research Symposium, August 2, 2013, Rochester, NY. 

Womack, K.*, et al. (2013, August). Using linguistic analysis to characterize conceptual units of thought in spoken medical narratives. RIT Undergraduate Research Symposium, August 2, 2013, Rochester, NY.

2012:

Womack, K.*, Male, J.*, et al. (2012, October). Towards understanding diagnostic cognitive reasoning of physicians. 2012 Mellon Foundation Symposium—The Multilingual Mind: Language Development and Methodology, Syracuse University. 

Male, J.*, et al. (2012, August). Extracting diagnostic information from dermatological narratives. RIT Summer Undergraduate Research Symposium, Rochester, NY 

Womack, K.*, et al. (2012, August). Disfluencies in the medical reasoning process. RIT Summer  Undergraduate Research Symposium, Rochester, NY. 

McCoy, W.*, et al. (2012, July) Annotation Schemes to Encode Domain Knowledge in Medical Narratives. Proceedings of the 6th ACL Linguistic Annotation Workshop, Jeju, Korea. Ed. Nancy Ide, Fei Xia. Stroudsburg, PA: ACL, Web.

McCoy, W.*, et al. (2012, July) Linking Uncertainty in Physicians' Narratives to Diagnostic Correctness. Proceedings of the Extra-Propositional Aspects of Meaning in Computational Linguistics, Jeju, Korea. Ed. Roser Morante, Caroline Sporleder. Stroudsburg, PA: ACL, Web.

Womack, K.*, et al. (2012, July) Disfluencies as Extra-Propositional Indicators of Cognitive Processing. Proceedings of the Extra-Propositional Aspects of Meaning in Computational Linguistics, Jeju, Korea. Ed. Roser Morante, Caroline Sporleder. Stroudsburg, PA: ACL, Web.

2011:

Engelman, C.*, et al. (2011, May) Evaluating Usability of Multimodal Interaction. Proceedings of the Novel Gaze-Controlled Applications, Karlskrona, Sweden. Ed. V. Sundstedt and C. C. Sennersten. New York: ACM, 2011. Web.

Engelman, C.*, et al. (2011, May) Exploring Interaction Modes for Image Retrieval. Proceedings of the Conference on Novel Gaze-Controlled Applications (NGCA11), Blekinge Institute of Technology, SE-371 79, Karlskrona, Sweden. Ed. Veronica Sunstedt and Charlotte Sennersten. ACM: n.p., Web.