Cohort 5 (2023-2027)

This project will focus on how an artificial agent can analyse and explain data patterns in user-collected health data. This will investigate how to build a trustworthy relationship between a user and an artificial agent. The project will conduct a systematic review of health tracking and analysis apps and build an artificial agent that can analyse, explain and interpret user-collected health data (using wearables and apps) to find patterns. Health data will be collected from users via user studies that investigate how these patterns can be best communicated to users. The PhD programme will contribute technical approaches to build trustworthy artificial agents, a better understanding of principles underlying trustworthy social relationships between agents and humans, and the ability to engage in collaborative sense-making of health data. This will be able to improve current approaches to health apps to interconnect personalised healthcare and recommend behaviour changes. In year 1, the focus will be on the taught programme with a start in exploring the relevant literature and technologies. In Year 2, the focus will be on conducting a systematic literature review while making a start to collect sample data from users to develop a data analysis package. In years 3 and 4, an artificial agent will be built that can interact with users to allow for the study of what aspects of interactions result in successful and trustworthy relationships.

References
Richard Brown, Lynne Coventry, Elizabeth Sillence, John Blythe, Simone Stumpf, Jon Bird, and Abigail C. Durrant. 2022. Collecting and sharing self-generated health and lifestyle data: Understanding barriers for people living with long-term health conditions – a survey study. DIGITAL HEALTH 8: 20552076221084456 (link here).

Adrian Bussone, Simone Stumpf, and George Buchanan. 2016. It Feels Like I’m Managing Myself: HIV+ People Tracking Their Personal Health Information. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (NordiCHI ’16), 1–10 (link here).

Jessica Schroeder, Ravi Karkar, Natalia Murinova, James Fogarty, and Sean A. Munson. 2019. Examining Opportunities for Goal-Directed Self-Tracking to Support Chronic Condition Management. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4: 151:1-151:26 (link here).

Lena Mamykina, Daniel A. Epstein, Predrag Klasnja, Donna Sprujt-Metz, Jochen Meyer, Mary Czerwinski, Tim Althoff, Eun Kyoung Choe, Munmun De Choudhury, and Brian Lim. 2022. Grand Challenges for Personal Informatics and AI. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22), 1–6 (link here).

Perepelkina, Olga, and Alessandro Vinciarelli. “Social and Emotion AI: The Potential for Industry Impact.” 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW). IEEE, 2019.

Aims and Objectives
This project will establish key features of just-in-time adaptive interventions (JITAIs; Nahum-Shani et al., 2018) that contribute to habit formation. While JITAIs have been shown to be more effective than statically controlled interventions (Wang & Miller, 2020), little is known about the precise interaction of adaptive intervention features and psychological processes that lead to lasting health habits. The research will be conducted in the domain of hydration, i.e., water drinking. Water drinking is an ideal domain to study the effect of JITAIs, given that water drinking is a relatively simple health behaviour, compared to, for example, eating or physical activity; water drinking needs to happen frequently each day, so that it is susceptible to habit formation; and indeed, healthy water drinkers seem to rely heavily on habits (Rodger et al., 2021). However, many people are underhydrated, with implications for cognitive functioning, mood, and physical health (e.g., risk of diabetes, overweight, kidney damage; see Muñoz et al., 2015; Perrier et al., 2020).

Working with the PUL smartcap and accompanying smartphone app, we will examine how habit formation occurs using an intervention that provides goal setting, monitoring, feedback, as well as situated and personalised reminders. We will address questions such as: Which intervention features predict habit formation, and how can the intervention be optimised to facilitate this process? Given that habits form in response to stable context cues, do reminders at specific, fixed times facilitate habit formation compared to “smart”, adaptive reminders? What is the role of rewarding feedback in habit formation and habit maintenance (cf. Papies et al., 2020)? Does the intervention lead to “specific” habit formation (i.e., drinking water with the PUL device) or to “generalised” habit formation (i.e., drinking water)? Which intervention features (e.g., dynamic goals, smart reminders, visual reward signals) or intervention effects (e.g., reduced dehydration symptoms) predict continued engagement with the app? In addressing these questions, we will conduct research that speaks to JITAI development, as well as to fundamental psychological questions about situated learning and habit change.

Methods, Outputs and Impact
We will work closely with the PUL Hydration team to conduct both qualitative, quantitative, and mixed-methods experimental studies to assess the questions outlined above in real-life settings. We will present and publish our findings as three empirical subprojects (averageing one per year) in both Computer Science and Psychology conferences and journals. In addition to academic and industry users, the findings on how to develop healthy water drinking habits will be of interest to the general public. The Healthy Cognition Lab regularly engages in knowledge exchange activities, which the student would participate in. We also regularly engage with industry and third-sector partners, such as Danone or the British Dietetic Association. Finally, the student on this project would work closely with other ECR lab members working on hydration, and on healthy and sustainable eating behaviours.

References
Muñoz, C. X., Johnson, E. C., McKenzie, A. L., Guelinckx, I., Graverholt, G., Casa, D. J., … Armstrong, L. E. (2015). Habitual total water intake and dimensions of mood in healthy young women. Appetite, 92, 81–86 (link here).

Nahum-Shani, I., Smith, S. N., Spring, B. J., Collins, L. M., Witkiewitz, K., Tewari, A., & Murphy, S. A. (2018). Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine, 52(6), 446–462 (link here).

Papies, E. K., Barsalou, L. W., & Rusz, D. (2020). Understanding Desire for Food and Drink: A Grounded-Cognition Approach. Current Directions in Psychological Science, 29(2), 193–198 (link here).

Perrier, E. T., Armstrong, L. E., Bottin, J. H., Clark, W. F., Dolci, A., Guelinckx, I., Iroz, A., Kavouras, S. A., Lang, F., Lieberman, H. R., Melander, O., Morin, C., Seksek, I., Stookey, J. D., Tack, I., Vanhaecke, T., Vecchio, M., & Péronnet, F. (2020). Hydration for health hypothesis: A narrative review of supporting evidence. European Journal of Nutrition (link here).

Rodger, A., Wehbe, L. H., & Papies, E. K. (2021). “I know it’s just pouring it from the tap, but it’s not easy”: Motivational processes that underlie water drinking. Appetite, 164, 105249 (link here).

Wang, L., & Miller, L. C. (2020). Just-in-the-Moment Adaptive Interventions (JITAI): A Meta-Analytical Review. Health Communication, 35(12), 1531–1544 (link here).

This project explores how neuromodulation based on closed-loop interaction systems and brain stimulation can regulate brain activity responsible for locomotion [1] in human participants. This involves coupling an external stimulator (for transcranial alternating current stimulation, tACS) to specific dynamics of neural circuits, which has shown promising results for interacting with rhythmic brain and motor activity (see e.g. [2] for suppressing tremor in Parkinson; see [3] for an application in locomotion).

Brain stimulation through tACS is a powerful, non-invasive, versatile tool for studying the human brain in vivo. It can alter rhythmic brain activity and associated sensory, cognitive and motor processes, as well as cortical excitability and subsequently neuroplasticity as revealed by the lasting changes in evoked potentials. Beyond application in sensory, motor and cognitive neuroscience, it is under investigation as a non-invasive therapeutic tool in several neurological disorders. In this context, immediate and long lasting effects in synaptic efficacy are imposed to either enhance cortical excitability, or inhibit it, likely by entrainment of underlying brain oscillations [4].

Oscillations are considered to play a fundamental role in the communication of brain cells and hence in the organization of brain networks. In fact, recent findings support a complex communication system based on cross-frequency coupling. The promise of closed loop approaches lays in the fact that it is possible to modulate this oscillatory network activity of the brain reflecting brain states, translating in the modulation of motor and mental processes, including shift of attention, emotions and so on.

Devising a closed-loop system of brain stimulation is a powerful way to provide fine control modulations of brain activity. Human motion data can be acquired and processed online to extract information of repetitive patterns [5,6] and thus help to shape neuromodulation in order to enable brain network entrainment.

References

1. McNamara et al. ‘Stable, interactive modulation of neuronal oscillations produced through brain-machine equilibrium’, Cell Reports, 2021.

2. Brittain et al. ‘Tremor Suppression by Rhythmic Transcranial Current Stimulation’, Current Biology, 2013.

3. Koganemaru et al. ‘Cerebellar transcranial alternating current stimulation modulates human gait rhythm’, Neuroscience Research, 2020.

4. Lakatos et al. ‘A New Unifying Account of the Roles of Neuronal Entrainment’, Current Biology, 2019.

5. Domingos et al. ‘Intention Detection of Gait Adaptation in Natural Settings’, IEEE Symposium Series on Computational Intelligence, 2021.

Cultivating and maintaining mental health is a significant challenge for many residents in care homes. Depression and loneliness, self isolation and low levels of life satisfaction are common among the elderly, who are also more likely to suffer from other physical health problems compared to community-dwelling elderly citizens. Many care homes do not (and can not) provide one-on-one, person-centred care due to lack of resources. This project aims to build data-driven AI models using multimodal machine learning to create a comprehensive picture of care home residents’ mental health. The aim will be to use these models to predict the emergence of potential mental health problems as early as possible, based on analysing multimodal data (audio, video, wearables) collected from care homes.

Main Objectives and Novelty
There has been an increased interest in automatic detection of mental health problems over the past several years, mainly focussing on (1) signals collected from mobile phones and wearables; and (2) younger, tech-savvy populations. However, audio-visual signals provide a vast array of extra cues that can improve inference models (Lin et al. 2021, Zhang et al. 2020), but which are currently underused. The main aims of this project are to:

1. Build a dataset of structured interviews to be collected at care homes using multimodal sensors (audio/video/wearable sensors).
2. Devise novel machine learning models that extend state-of-the-art methods to analyse and use the multimodal signal collected to predict mental health conditions.
3. Validate and evaluate the accuracy of these models via quantitative and qualitative measures of loneliness, depression and anxiety among aged care residents, as well as build a clear picture of aged care residents’ feelings and personal experience with this technology through qualitative interview methods.

Methods
This project will use experimental methods on collecting, validating and evaluating multimodal data related to mental health (Lin et. al. 2021, Laban et. al. 2021,2022). Using the collected data, it will also build on and extend state-of-the-art approaches on multimodal data representation and feature selection to devise inference models to predict and correlate with mental health conditions and risks identified within care homes.

Likely Outcome and Impact
Nearly a half million UK residents currently live in care homes, representing 4% of the population older than 65, and 15% of those aged 85 and over. The onset of the global coronavirus pandemic, and resulting restrictions on face-to-face meetings with friends, family and loved ones has highlighted how fragile human mental health is when faced with even short-term restrictions to socialising, and these effects have been experienced even more acutely by older individuals living in live-in care settings (Cross and Henschel 2020). If we are able to achieve our goals with this project, we will be able to develop tools to assist residents as well as care home staff to identify when individuals are at risk of deteriorating mental health and/or in need of extra 1:1 care or companionship from staff.

References
[1] Lin, W., Orton, I.,Li, Q., Pavarini, G. and Mahmoud, M. (2021). Looking At The Body: Automatic Analysis of Body Gestures and Self-Adaptors in Psychological Distress. IEEE Transaction on Affective Computing.

[2] Zhang, Z., Lin, W., Liu, M. and Mahmoud, M. (2020). Multimodal Deep Learning Framework for Mental Disorder Recognition. IEEE International Conference on Automatic Face and Gesture Recognition.

[3] Laban, G., Ben-Zion, Z. & Cross, E. S. (2022). Social robots for supporting post-traumatic stress disorder diagnosis and treatment. Frontiers in Psychiatry (in press).

[4] Laban, G., George, J.-N., Morrison, V. & Cross, E. S. (2021). Tell me more! Assessing interactions with social robots from speech. Paladyn: Journal of Behavioral Robotics, 12(1), 136-159.

[5] Cross, E. and Henschel, A. (2020) The neuroscience of loneliness – and how technology is helping us (link here).

As robots become more prevalent in our daily lives, there is an increasing need for them to be able to interact with humans in a social and intuitive manner. Even though Robot and AI technologies have rapidly developed in recent years, the ability of robots to interact with humans in an intuitive and social way is still quite limited. An important challenge is to determine how to design robots that can perceive the user’s needs, feelings and intentions, and adapt to users over a broad range of cognitive abilities. In particular, how the robot could react to human partners showing different behaviours, and how human-robot interactive behaviours might potentially reveal novel aspects of human personality and health.

In this project, the student will develop robot control and machine learning methods on the UR3e robotic arm testbed (with haptic sensors) to autonomously control a robot to physically interact with human collaborators (clap hands, jointly move an object, etc.) showing different behaviours. The student will use machine learning methods to extract behaviour patterns and then design robot control methods (PID and continuous learning methods) to control the robotic arm to move with specific behaviour patterns. Furthermore, we will study how to use robot and joint human-robot behaviours to elicit and identify aspects of human personality and health.

The project’s impact will be broad, affecting health, neuroscience, social robotics and fundamental robot research. The output of the project can be published in various journals and conferences, such as IEEE International Conference on Robotics and Automation (ICRA), IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), ACM/IEEE International Conference on Human-Robot Interaction, Science Robotics, Journal of Neural Engineering, and Frontiers in psychology, etc.

References
Hale J. and Pollick F. “Sticky Hands”: Learning and generalization for cooperative physical interactions with a humanoid robot, IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, 35(4), Nov. 2005.

Wiese E., Metta G., Wykowska A., Robots as intentional agents: using neuroscientific methods to make robots appear more social, Frontiers in psychology, Oct. 2017.

Babic J., Hale J. and Oztop E., Human sensorimotor learning for humanoid robot skill synthesis, Adaptive Behavior, 19(4), 2011.

Huang L., Meng Z., Deng Z., Wang C., Li L., et al, Extracting human behavioral biometrics from robot motions, in Proc. 27th Annual International Conference on Mobile Computing and Networking (MobiCom 2021), Oct. 2021.

Context
The food system contributes 34% of greenhouse gas emissions, the majority of which coming from animal agriculture [1] also disproportionately contributing to deforestation, water scarcity, biodiversity loss, and ecosystem pollution [2]. Despite this, most consumers are resistant to substantially reduce their meat consumption, even when considering the accompanying health benefits. Efforts to improve eating habits are traditionally approached through behaviour change counselling sessions with dieticians. Such approaches are time and resource consuming, but digital intervention alternatives lack the essential component of human interaction and social support that drives the effectiveness of behaviour change counselling [3]. Virtual agents hold the potential to fill that gap; however past approaches have typically only been loosely coupled to existing social science in behaviour change [4].

Objectives and Novelty
The project will focus on designing a virtual agent dialogue framework for longitudinal behaviour change interactions rooted in an established psychological theory. The adaptive dialogue agent will be able to guide users through their journey towards dietary change, interspersing activities from behaviour change programmes with social dialogue aimed at reinforcing the user-agent relationship while simultaneously probing users’ preferences and attitudes. These preference-infering exchanges will help maintain and update user models including idiosyncratic sensitivities to key variables identified to be key drivers for transitioning to more plant-based foods [5]: Taste expectations (i.e. meat-based foods are expected to be tastier), Availability (i.e. plant-based foods are less widely available in many settings), Skills (many consumers don’t know how to prepare meat-free meals), Identity (Vegetarian/vegan social identities are not seen as positive by many consumers and contribute to the polarization of perspectives on sustainable eating) and Social Norms (consuming meat is seen as normative, and these norms are communicated through features of the food environment and others’ behaviour). These user models will further impact task-related and relationship-building tasks, altering dialogue such as agents’ food presentation strategies. A key research challenge will consist in designing dialogue policies reconciling concurrent but inter-linked dialogue goals, in this case preference-infering, relationship-building, and delivering task-related dialogue.

Methods and Timeline
After a literature review, the student will extend an existing socially-aware recipe recommender agent framework developed at the University of Glasgow [6] with a baseline rule-based dialogue model for inferring user preferences and attitudes and integrating these variables to alter subsequent dialogue. The model will be used to collect initial data and train further model iterations, considering supervised/reinforcement learning approaches. The resulting dialogue models will be deployed in a series of user experiments to evaluate their effectiveness at promoting user engagement and motivation, infering accurate user models, and driving effective and long-lasting behaviour change.

Outputs and impact
The project is expected to contribute novel dialogue models and policies for human-agent interactions as well as methodological and experimental insights on technologically-mediated behaviour change frameworks. The project’s findings may further feedback into theory formation on habit change and maintenance. The project will have societal impact, both locally through deployments of the resulting behaviour change framework, and further through dissemination with academic and institutional partners.

References
[1] Xu, X., Sharma, P., Shu, S., Lin, T.-S., Ciais, P., Tubiello, F. N., Smith, P., Campbell, N., & Jain, A. K. (2021). Global greenhouse gas emissions from animal-based foods are twice those of plant-based foods. Nature Food, 1–9 (link here).

[2] Poore, J., & Nemecek, T. (2018). Reducing food’s environmental impacts through producers and consumers. Science, 360(6392), 987–992 (link here) / science.aaq0216[4] Graça, J., Godinho, C. A., & Truninger, M. (2019). Reducing meat consumption and following plant-based diets: Current evidence and future directions to inform integrated transitions. Trends in Food Science & Technology, 91, 380–390 (link here).

[3] Schippers, M., et al. “A meta‐analysis of overall effects of weight loss interventions delivered via mobile phones and effect size differences according to delivery mode, personal contact, and intervention intensity and duration.” Obesity reviews 18.4 (2017): 450-459.

[4] Bickmore, Timothy W., et al. “A randomized controlled trial of an automated exercise coach for older adults.” Journal of the American Geriatrics Society 61.10 (2013): 1676-1683.

[5] Papies, E. K., Johannes, N., Daneva, T., Semyte, G., & Kauhanen, L.-L. (2020). Using consumption and reward simulations to increase the appeal of plant-based foods. Appetite, 155, 104812 (link here).

[6] Florian Pecune, Lucile Callebert, and Stacy Marsella. 2020. A Socially-Aware Conversational Recommender System for Personalized Recipe Recommendations. In Proceedings of the 8th International Conference on Human-Agent Interaction (HAI ’20). Association for Computing Machinery, New York, NY, USA, 78–86 (link here).

Human social interaction is increasingly mediated by technology, with many of the signals present in traditional face-to-face interaction being replaced by digital representations (e.g., avatars, nameplates, and emojis). To communicate successfully, participants in a conversational interaction must keep track of the identities of their co-participants, as well as the “common ground” they share with each—the dynamically changing set of mutually held beliefs, knowledge, and suppositions. Perceptual representations of interlocutors may serve as important memory cues to shared information in communicative interaction (Horton & Gerrig, 2016; O’Shea, Martin, & Barr, in press). Our main question concerns how digital representations of users across different interaction modalities (text, voice, video chat) influence the development of and access to common ground during communication. To examine the impact of digital user representations on real-time language production and comprehension, the project will use a variety of behavioral methods including visual world eye-tracking (Tanenhaus, et al. 1995), latency measures, as well as analysis of speech/text content. In the first phase of the project, we will examine how well people can keep track of who said what during a discourse depending on the abstract versus rich nature of user representations (e.g., from abstract symbols to dynamic avatar-based user representations), and how these representations impact people’s ability to tailor messages to their interlocutors, as well as to correctly interpret a communicator’s intended meaning. For example, in one such study, we will test participants’ ability to track “conceptual pacts” (Brennan & Clark, 1996) with a pair of interlocutors during an interactive task where each partner appears (1) through a video stream; (2) as an animated avatar; or (3) as a static user icon. In the second phase, we will examine whether the nature of the user representation during encoding affects the long-term retention of common ground information. In support of the behavioural experiments, this project will also involve developing a range of conversational agents, both embodied and speech-only, and defining appropriate behaviour models to allow those agents to take part in the studies. The defined behaviour will incorporate both verbal interaction as well as non-verbal actions, to replicate the full richness of human face-to-face conversation (Foster, 2019; Bavelas et al., 1997). Insights and techniques developed during the project are intended to improve interfaces for computer-mediated human communication.

References
Bavelas, J. B., Hutchinson, S., Kenwood, C., & Matheson, D. H. (1997). Using Face-to-face Dialogue as a Standard for Other Communication Systems. Canadian Journal of Communication, 22(1).

Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1482.

Foster, M. E. (2019). Face-to-face conversation: why embodiment matters for conversational user interfaces. Proceedings of the 1st International Conference on Conversational User Interfaces – CUI ’19. the 1st International Conference.

Horton, W. S., & Gerrig, R. J. (2016). Revisiting the memory‐based processing approach to common ground. Topics in Cognitive Science, 8, 780-795.

O’Shea, K. J., Martin, C. R., & Barr, D. J. (2021). Ordinary memory processes in the design of referring expressions. Journal of Memory and Language, 117, 104186.

Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.

Aims and Objectives
This project will examine which kinds of data to use to best integrate a digital mobile health intervention into a users’ daily life, to lead to habit formation. Previous research has shown that just-in-time adaptive interventions (JITAIs) are more effective than statically controlled interventions (Wang & Miller, 2020). In other words, health interventions are more likely to lead to behaviour change if they are well situated, i.e., with agency adapted to specific user characteristics, and applied in situations where behaviour change should happen. However, there is limited evidence on how to best design JITAIs for health apps, so as to create artificial agents that lead to lasting behaviour change through novel habit formation. In addition, there is no systematic evidence as to which features of situations a health app should use to support a user to perform a healthy behaviour (e.g., time of day, location, mood, activity pattern, social context). We will address these issues in the under-researched domain of hydration behaviours. The aim is to establish—given the same intervention—which type of contextual data, or which heterogeneous mix of types of data, is most effective at increasing water consumption, and at establishing situated water drinking habits that persist when the initial engagement with the intervention has ceased.

Background and Novelty
Mobile health interventions are a powerful new tool in the domain of individual health behaviour change. Health apps can reach large numbers of users at relatively low cost, and can be tailored to an individual’s health goals and adapted to support users in specific, critical situations. Identifying the right contextual features to trigger an intervention is critical, because context plays a key role both in triggering unhealthy behaviours, and in developing habits that support the long-term maintenance of healthy behaviours. A particular challenge, which existing theories typically don’t yet address, lies in the dynamic nature of health behaviours and their contextual triggers, and in establishing how these behaviours and contexts can best be monitored (Nahum-Shani et al., 2018). This project will take on these challenges in the domain of hydration, because research suggests that many adults may be chronically dehydrated, with implications for cognitive functioning, mood, and physical health (e.g., risk of diabetes, overweight, kidney damage; see Muñoz et al., 2015; Perrier et al., 2020). Our previous work has shown that healthy hydration is associated with drinking water habitually across many different situations each day (Rodger et al., 2020). This underlines the particular importance of establishing dynamic markers of situations that are cognitively associated with healthy behaviours so that they can support habit formation.

Methods
1. We will examine the internal (e.g., motivation, mood, interoception) and external (e.g., time of day, location, activity pattern, social context) markers of situations in which high water drinkers consume water, using objective intake monitors. Then, integrating these findings with theory on habit formation and motivated behaviour (Papies et al., 2020), and using an existing app platform (e.g. AWARE-Light),

2. We will test which types of data or mixes of data types are most effective in an intervention to increase water consumption in a sample of low water drinkers in the short term, and

3. Whether those same data types are effective at creating hydration habits that persist in the longer term.

Outputs
This project will lead to presentations and papers of three quantitative subprojects at both Computer Science and Psychology conferences, as well as a possible qualitative contribution on the dynamic nature of habit formation.

Impact
Results from this work will have implication for the design of health behaviour interventions across domains. This work will further contribute to the emerging theoretical understanding of the formation and context sensitivity of the cognitive processes that support healthy habits. It will explore how sensing and adaptive user modeling can situate both user and AI system in a common contextual frame and whether this facilitates engagement and behavior change.

References
Muñoz, C. X., Johnson, E. C., McKenzie, A. L., Guelinckx, I., Graverholt, G., Casa, D. J., … Armstrong, L. E. (2015). Habitual total water intake and dimensions of mood in healthy young women. Appetite, 92, 81–86 (link here).

Nahum-Shani, I., Smith, S. N., Spring, B. J., Collins, L. M., Witkiewitz, K., Tewari, A., & Murphy, S. A. (2018). Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine, 52(6), 446–462 (link here).

Papies, E. K., Barsalou, L. W., & Rusz, D. (2020). Understanding Desire for Food and Drink: A Grounded-Cognition Approach. Current Directions in Psychological Science, 29(2), 193–198 (link here).

Perrier, E. T., Armstrong, L. E., Bottin, J. H., Clark, W. F., Dolci, A., Guelinckx, I., Iroz, A., Kavouras, S. A., Lang, F., Lieberman, H. R., Melander, O., Morin, C., Seksek, I., Stookey, J. D., Tack, I., Vanhaecke, T., Vecchio, M., & Péronnet, F. (2020). Hydration for health hypothesis: A narrative review of supporting evidence. European Journal of Nutrition (link here).

Rodger, A., Wehbe, L., & Papies, E. K. (2020). “I know it’s just pouring it from the tap, but it’s not easy”: Motivational processes that underlie water drinking. Under Review (link here).

Wang, L., & Miller, L. C. (2020). Just-in-the-Moment Adaptive Interventions (JITAI): A Meta-Analytical Review. Health Communication, 35(12), 1531–1544 (link here).

Main Aims and Objectives
The main goal of this project is to develop approaches for automatic inference of social and psychological phenomena taking place in phone-based conversations. The focus will be on personality traits, conflict management style and interpersonal attraction. The key-assumption is that such phenomena leave physical, machine detectable traces in terms of social signals [Vin09], i.e., verbal and nonverbal behavioural cues such as laughter, tone of voice, lexical choices, turn-taking, pauses, etc. [Pog12]. If such cues can be detected automatically, it is possible to use them for inferring the underlying social and psychological phenomena. The project is highly interdisciplinary and it requires one to take into account both technological and psychological aspects of the problem.

Proposed Methods
The main experiments will be performed over an available corpus of 60 conversations (120 speakers in total) involved in a negotiation (the Winter Survival Task), for a total of roughly 12 hours of material [Vin15]. The data is annotated in terms of psychological constructs (personality traits, conflict management style, interpersonal attraction) and major nonverbal behavioural cues (laughter, hesitations such as “ehm” or “uhm”, pauses, etc.). This will allow the development of multimodal Deep Learning approaches [Bal19] capable to jointly process speech signals and automatic data transcriptions to output information about the social phenomena above. Such part of the project will involve the application of Computational Paralinguistics methodologies (extraction of Low Level Descriptions, etc.) [Sch13] and deep learning models for the analysis of text (word embeddings, BERT, etc.) and speech signals (Recurrent neural Networks, Long Short-term Memory Networks, Convolutional Neural Networks, etc.).

Likely Outputs
This project will lead to the development of multimodal methodologies for the automatic analysis of conversations. While being developed using the data described above, the approaches will be designed to generalize, i.e., to work over other types of data too.

Possible Impact
The project will contribute to endow machines with social intelligence, i.e., the ability to make sense of social interactions in the same way as people do. This is seen as one of the last steps towards the development of AI approaches capable to seamlessly integrate the life of people.

Alignment with Industrial Interests
The project is highly relevant to Conversational AI, one of the key-areas underlying the development of commercially important technologies such as personal assistants (Alexa, Siri, etc.), Human-Robot Interaction, interactive entertainment, etc. In this respect, the project is likely to be of major interest to all industrial sectors in which speech interfaces are of interest.

References
[Bal19] T. Baltrušaitis, C. Ahuja and L. -P. Morency, “Multimodal Machine Learning: A Survey and Taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423-443, 2019.

[Pog12] I.Poggi and F.D’Errico, “Social signals: a framework in terms of goals and beliefs”, Cognitive Processing, Vol. 13, pp. 427-445, 2012.

[Sch13] B.Schuller, A.Batliner, “Computational paralinguistics: emotion, affect and personality in speech and language processing”, John Wiley & Sons, 2013.

[Vin09] A.Vinciarelli, M.Pantic, and H.Bourlard, “Social Signal Processing: Survey of an Emerging Domain“, Image and Vision Computing Journal, Vol. 27, no. 12, pp. 1743-1759, 2009.

[Vin15] A.Vinciarelli, E.Chatziioannou and A.Esposito, “When the Words are not Everything: The Use of Laughter, Fillers, Back-Channel, Silence and Overlapping Speech in Phone Calls“, Frontiers in Information and Communication Technology, 2:4, 2015.