We are investigating how novel AI tools, including large language models (LLMs), can augment clinical mental healthcare, including psychotherapy sessions, or provide support at opportune moments surrounding clinical visits.

Published work:

• Yuewen Yang, Thalia Viranda, Anna R. Van Meter, Tanzeem Choudhury, and Daniel A. Adler. Exploring Opportunities to Augment Psychotherapy with Language Models. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems

This project aims to investigate how sensing devices (e.g. smartphones, wearables) that collect data to measure behaviors associated with mental health symptoms can be implemented in different contexts. We have conducted qualitative work to understand how these tools can be used in high stress workplaces and clinical setttings. Our findings aim to improve programs that use these technologies. Image source.

Published work:

• Jodie Nghiem, Daniel A. Adler, Deborah Estrin, Cecilia Livesey, and Tanzeem Choudhury. Understanding Mental Health Clinicians’ Perceptions and Concerns Regarding Using Passive Patient-Generated Health Data for Clinical Decision-Making: Qualitative Semistructured Interview Study. JMIR Formative Research
• Daniel A. Adler, Emily Tseng, Khatiya C. Moon, John Q. Young, John M. Kane, Emanuel Moss, David C. Mohr, and Tanzeem Choudhury. Burnout and the Quantified Workplace: Tensions around Personal Sensing Interventions for Stress in Resident Physicians. Proceedings of the ACM on Human-Computer Interaction

This project aims to analyze the reliability of machine learning tools that use data collected from personal devices to measure mental health symptoms. Specifically, we aim to analyze the reliability of these tools across different types of individuals, over time, devices, and platforms.

Published work:

• Daniel A. Adler, Caitlin A. Stamatis, Jonah Meyerhoff, David C. Mohr, Fei Wang, Gabriel J. Aranovich, Srijan Sen, and Tanzeem Choudhury. Measuring algorithmic bias to analyze the reliability of AI tools that predict depression risk using smartphone sensed-behavioral data. npj Mental Health Research
• Daniel A. Adler, Fei Wang, David C. Mohr, and Tanzeem Choudhury. Machine learning for passive mental health symptom prediction: Generalization across different longitudinal mobile sensing studies. PLOS ONE
• Daniel A. Adler, Fei Wang, David C. Mohr, Deborah Estrin, Cecilia Livesey, and Tanzeem Choudhury. A call for open data to develop mental health digital biomarkers. BJ Pscyh Open

Rhythms guide our lives. Our biological clocks tell us when we need to sleep, eat and wake. But our use of technology can interrupt and obstruct these rhythms, making it difficult for our bodies to get what they need to stay healthy and balanced. MoodRhythm helps individuals to live a more naturally rhythmic day, ensuring that they can maintain balanced and healthy lives. The key goals of MoodRhythm is to use patients' smartphones to both actively and passively track daily rhythms and to provide affective feedback that can help patients to maintain a regular daily rhythm, while feeding this clinically valuable information back to their physicians.

Recently in mobile and wearable computing, we have seen a rise in continuous sensing across different types of sensors and in device processing. Combination of these array of sensors and corresponding inference have capability to generate detailed non-trivial multidimensional information about the environment and the individual carrying the device. Also, there is a growing number of third-party applications that use these information for context-aware-computing, personal well-being, and just-in-time health care.

In this paper, we propose SAINT, a scalable sensing and real-time sensing and inference toolkit for mobile devices. SAINT uses a client-server architecture. The server can support clients with different sensing (e.g., accelerometer and location) and inference (e.g., human speech) streams on-demand. If more than one client ask for the same sensing or inference data then the server performs necessary computation (e.g., recognizing human speech) once and serves all the clients needing such data with inter process communication. Furthermore, internally SAINT server enables easy sharing of sensing and inference data among different recognition modules (e.g., sleep sensing can use movement and speech inference data) over a simple middle-man $bus$ like structure. This bus architecture makes any sensing or inference streams available for consumption for any other inference module. This sharing enables complex inference possible by reusing components that has been built before. We show that bus and client server architecture incur low overhead while enabling exponential increase in cpu time and power usage on the number of clients. Also, developers found SAINT framework easy to program and deploy.

Stress can have long term adverse effects on individuals’ physical and mental well-being. Changes in the speech production process is one of many physiological changes that happen during stress. Microphones, embedded in mobile phones and carried ubiquitously by people, provide the opportunity to continuously and non-invasively monitor stress in real-life situations. We propose StressSense for unobtrusively recognizing stress from human voice using smartphones. We investigate methods for adapting a one-size-fitsall stress model to individual speakers and scenarios. We demonstrate that the StressSense classifier can robustly identify stress across multiple individuals in diverse acoustic environments: using model adaptation StressSense achieves 81% and 76% accuracy for indoor and outdoor environments, respectively. We show that StressSense can be implemented on commodity Android phones and run in real-time. To the best of our knowledge, StressSense represents the first system to consider voice based stress detection and model adaptation in diverse real-life conversational situations using smartphones.

BeWell is the next generation in mobile health apps. The BeWell app continuously tracks user behaviors along three key health dimensions without requiring any user input. Classification algorithms run directly on the phone to automatically infer the user's sleep duration, physical activity, and social interaction. In addition to classifying activities that influence health, BeWell also computes a weighted score between 0 and 100 for each health dimension. A score of a 100 indicates that the user is matching or exceeding recommended guidelines (averaging eight hours sleep per day, for example). BeWell also promotes improved behavioral patterns via persuasive feedback as part of an animated aquatic ecosystem rendered as an ambient display on the smartphone's wallpaper screen.

The study is aimed at comparing the effectiveness of paper-based survey tools with that of a passive electronic mobile sensing device (MSP) for the measurement of overall physical and social activity of eight senior citizens at Kendal Continuing Care Retirement Community. Before the start of the study, the Yale Physical Activity Survey, SF-36, Friendship Scale, and the CES-D were used to obtain a baseline understanding of subjects’ physical, and social wellbeing. The study length was 15 consecutive days. Each subject was provided a charged, and activated device to wear at the waist during his/her waking hours and was instructed to continue performing their daily activities. Devices were collected at the end of the day for data extraction and charging. We are currently analyzing the results of the device and comparing them with the scores generated by the current paper-based surveys.

The rising popularity of the sensor-equipped smartphone is changing the possible scale and scope of human activity inference. The diversity in user population seen in large user bases can overwhelm conventional one-size-fits-all classification approaches. Although personalized models are better able to handle population diversity, they often require increased effort from the end user during training and are computationally expensive.

In this work, we propose an activity classification framework that is scalable and can tractably handle an increasing number of users. Scalability is achieved by maintaining distinct groups of similar users during the training process, which makes it possible to account for the differences between users without resorting to training individualized classifiers. The proposed framework keeps user burden low by leveraging crowd-sourced data labels, where simple natural language processing techniques in combination with multi-instance learning are used to handle labeling errors introduced by low-commitment everyday users. Experiment results on a large public dataset demonstrate that the framework can cope with population diversity irrespective of population size.

Neural signals are everywhere just like mobile phones. We propose to use neural signals to control mobile phones for hands-free, silent and effortless human-mobile interaction. Until recently, devices for detecting neural signals have been costly, bulky and fragile. We present the design, implementation and evaluation of the NeuroPhone system, which allows neural signals to drive mobile phone applications on the iPhone using cheap off-the-shelf wireless electroencephalography (EEG) headsets. We demonstrate a brain-controlled address book dialing app, which works on similar principles to P300-speller brain-computer interfaces: the phone flashes a sequence of photos of contacts from the address book and a P300 brain potential is elicited when the flashed photo matches the person whom the user wishes to dial. EEG signals from the headset are transmitted wirelessly to an iPhone, which natively runs a lightweight classifier to discriminate P300 signals from noise. When a person’s contact photo triggers a P300, his/her phone number is automatically dialed. NeuroPhone breaks new ground as a brain-mobile phone interface for ubiquitous pervasive computing. We discuss the challenges in making our initial prototype more practical, robust, and reliable as part of our on-going research.

Emotions play a major role in how interpersonal conflicts unfold. Although several strategies and technological approaches have been proposed for emotion regulation, they often require conscious attention and effort. This often limits their efficacy in practice. In this project, we propose a different approach inspired by self-perception theory: noticing that people are often reacting to the perception of their own behavior, we artificially change their perceptions to influence their emotions. We conducted two studies to evaluate the potential of this approach by automatically and subtly altering how people perceive their own voice. In one study, participants that received voice feedback with a calmer tone during relationship conflicts felt less anxious. In the other study, participants who listened to their own voices with a lower pitch during contentious debates felt more powerful. The results of these studies show the potential of automatic and less perceptible emotion regulation systems.

Alertness is a crucial component of our cognitive performance. Reduced alertness can negatively impact our memory consolidation, productivity and safety. To better track the ebb and flow of our alertness, we propose AlertnessScanner, a computer vision based system that collects in-situ pupil information to model alertness in the wild. We conducted two in-the-wild studies to evaluate the effectiveness of our solution, and we found that AlertnessScanner can assess alertness in a passive and unobtrusive way. As such, with AlertnessScanner installed on our phone, it can help track our alertness and we can better schedule our routine.

This project aims to develop new approaches to assess and improve everyday job performance and wellbeing using mobile sensor data, theoretically-grounded models and efficient software implementations.

To see more details, please go to mPerf website.

Photoacoustic effect is a fundamental physics concept which is essentially the generation of sound due to the absorption of intensity modulated light or more generally EM waves by a certain material. We took this fundamental physics concept to build a mobile sensing system that can characterize the quality or nutritional characteristics of liquid food. The long-term vision of this work is to democratize food characterization using such a low cost, easy to use, mobile system which could enable consumers to test food before purchase and to put an indirect pressure on the food industry and government regulators to ensure quality. This work made the following contributions: (1) Proving the fundamental concept of the theory of photoacoustic effect with step-by-step experimentation. (2) Design and Implementation of a low-cost mobile photoacoustic sensing system. (3) Implementation of the signal processing and machine learning algorithm for liquid food characterization. (4) Evaluation of Nutrilyzer for milk protein concentration, milk adulterants, and alcohol concentration characterization.

Food and branding research groups have leveraged illusions to alter individuals perception of serving size, resulting in the individuals serving themselves smaller food portions. One method is using smaller plates or serving utensils, which causes individuals to serve smaller portions of food. This is due to a perception bias that causes individuals to perceive that there is more food on the plate due to the amount of surface area covered or that they have served more. By leveraging one of these illusions we built a persuasive technology that create the illusion that there is more food on a plate than there actually is, which could influence how much food users serve themselves.

We often think of ourselves as individuals with steady capabilities. However, converging strands of research indicate that this is not the case --- our biochemistry varies significantly over the course of a 24 hour period; and consequently our levels of alertness, productivity, physical activity, and even sensitivity to pain can vary according to the time of day. This project focuses on identifying novel measurements and interventions that can leverage these daily variations. Our findings demonstrate that phone usage patterns can be used to detect and predict individual daily variations indicative of temporal preference, sleep deprivation, and the effects of social jet lag.

Being able to unobtrusively measure biological misalignments means there is also an opportunity for technology to play to our biological strengths. Incorporating an awareness of individual biochemical variations could have a significant impact on a wide range of technologies and help support increased well-being, productivity, and higher quality of sleep.

Early detection of mental health changes in individuals with serious mental illness is critical for effective intervention. CrossCheck is the first step towards the passive monitoring of mental health indicators in patients with schizophrenia and paves the way towards relapse prediction and early intervention.

Our results indicate that there are statistically significant associations between automatically tracked behavioral features related to sleep, mobility, conversations, smartphone usage and self-reported indicators of mental health in schizophrenia. Using these features we build inference models capable of accurately predicting aggregated scores of mental health indicators in schizophrenia with a mean error of 7.6% of the score range.

Motivated by the need to support those managing chronic pain, we report on the iterative design, development, and evaluation of Keppi, a novel pressure-based tangible user interface (TUI) for the self-report of pain intensity. In-lab studies with 28 participants found individuals were able to use Keppi to reliably report low, medium, and high pain as well as map squeeze pressure to pain level. Based on insights from these evaluations, we ultimately created a wearable version of Keppi with multiple form factors, including a necklace, bracelet, and keychain. Interviews indicated high receptivity to the wearable design, which satisfied additional user-identified needs (e.g., discreet and convenient) and highlighted key directions for the continued refinement of tangible devices for pain assessment.

DoppleSleep is a contactless sleep sensing system that continuously and unobtrusively tracks sleep quality using commercial off-the-shelf radar modules. DoppleSleep provides a single sensor solution to track sleep- related physical and physiological variables including coarse body movements and subtle and fine-grained chest, heart movements due to breathing and heartbeat. By integrating vital signals and body movement sensing, DoppleSleep achieves 89.6% recall with Sleep vs. Wake classification and 80.2% recall with REM vs. Non-REM classification compared to EEG-based sleep sensing. Lastly, it provides several objective sleep quality measurements including sleep onset latency, number of awakenings, and sleep efficiency. The contactless nature of DoppleSleep obviates the need to instrument the user’s body with sensors. Lastly, DoppleSleep is implemented on an ARM microcontroller and a smart- phone application that are benchmarked in terms of power and resource usage.