I am a PhD student with SOCIAL AI CDT. My research is concerned with conversationally relevant facial expressions in distinct cultures – East Asian and Western. It is crucial that technological advancements will not be western-centric.
I come from Israel, where I earned a Bachelor’s in Computer Science from the Academic College of Tel-Aviv-Yaffo, and a master’s in Linguistics from Tel Aviv University. In my Master’s thesis, I explored the topic of recovery strategies in an attempt to find what would help senior adults to recover from non-understanding errors.
I believe that social robots will be an inseparable part of our lives in the future, and I am excited to be a part of the SOCIAL AI CDT programme.
Modelling Conversational Facial Signals for Culturally Sensitive Artificial Agents
In spoken interactions, face-to-face meetings are often preferred. This is because the human face is highly expressive and can facilitate coordinated interactions. Embodied conversational agents with expressive faces therefore have the potential for smoother interactions than voice assistants. However, knowledge of how the face expresses these social signals – the “language” of facial expressions – is limited, with no coherent modelling framework (e.g., see Jack & Schyns, 2017). For example, current models focus primarily on basic emotions such as fear, anger and happiness, which are not suitable for everyday conversations or recognized cross-culturally (e.g., Jack, 2013). Instead, signals of affirmation, uncertainty, interest, and turn-taking in different cultures (e.g., Chen et al., 2015) are more relevant (e.g., Skantze, 2016). Conversational digital agents typically employ these signals in an ad hoc manner, with smiles or frowns manually inserted at speech-coordinated time points. However, this is costly, time consuming, and provides only a limited repertoire of, often Western-centric, face signals, which in turn restricts the utility of conversational agents.
To address this knowledge gap, this project will (a) Develop a modelling framework for conversationally relevant facial expressions in distinct cultures – East Asian and Western, (b) Develop methods to automatically generate these facial expressions in conversational systems, and (c) Evaluate these models in different human-robot cultural interaction settings. This automatic modelling will coordinate with the agent’s speech (e.g. auto-inserting smiles at appropriate times), the user’s behaviour (e.g. directing gaze and raising eyebrows when the user starts speaking), and the agent’s level of understanding (e.g. frowning during low comprehension).
We will employ state-of-the-art 3D capture of human-human interactions and psychological data-driven methods to model dynamic facial expressions (see Jack & Schyns, 2017). We will deploy these models using FurhatOS – a software platform for human-robot interactions – and the Furhat robot head, which has a highly expressive animated face with superior social signalling capacity compared to other platforms (Al Moubayed et al., 2013). The flexibility of Furhat’s display system, combined with state-of-the-art psychological-derived 3D face models will also enable exploration of other socially relevant facial characteristics, such as ethnicity, gender, and age (e.g., see Zhan et al., 2019).
The results will be highly relevant to companies developing virtual agents/social robots, such as Furhat Robotics. Skantze, Furhat Robotics co-founder/chief scientist, will facilitate impact of the results. The project will also inform fundamental knowledge of human-human and human-robot interactions by precisely characterizing how facial signals facilitate spoken interactions. We anticipate outputs in international psychology and computer science conferences (e.g., Society for Personality and Social Psychology; IEEE Automatic Face & Gesture Recognition) and high-profile scientific outlets (e.g., Nature Human Behaviour). Jack is PI of a large-scale funded laboratory specializing in modelling facial expressions across cultures.
Year 1 (Master’s): Training in (a) programming human-robot interactions; (b) data-driven modelling of dynamic facial expressions.
Year 2 – 3: Data-driven modelling of dynamic conversational facial expressions in each culture.
Year 3 – 4: Application and evaluation of facial expression models in human-robot interaction scenarios.
- Jack, R. E. & Schyns, P. G. (2017). Toward a social psychophysics of face communication. Annual review of psychology, 68, 269-297.
- Jack, R. E. (2013). Culture and facial expressions of emotion. Visual Cognition, 21(9-10), 1248-1286.
- Chen, C., Garrod, O., Schyns, P., Jack, R. (2015). The face is the mirror of the cultural mind. Journal of Vision, 15(12), 928-928.
- Skantze, G. (2016). Real-time coordination in human-robot interaction using face and voice. Ai Magazine, 37(4), 19-31.
- Moubayed, S. A., Skantze, G., & Beskow, J. (2013). The furhat back-projected humanoid head – lip reading, gaze and multi-party interaction. International Journal of Humanoid Robotics, 10(01), 1350005.
- Zhan, J., Liu, M, Garrod, O.G., Jack, R. E., & Schyns, P. G. (2020, October). A Generative Model of Cultural Face Attractiveness. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (pp. 1-3).
I am a Biomedical Engineering graduate from University College London. Throughout my degree and internships at Francis Crick Institute and ETH Zürich, I have gained experience in a wide range of areas including programming, neural engineering, design, and ethics. I also expanded my interests by taking modules in new areas such as Rehabilitation of Acquired Neurogenic Communication Difficulties in the division of Psychology and Language Sciences.
I have always been fascinated by how technology has evolved, as an aid to improving people’s physical health or mental health. However, I was also intrigued and concerned by how devices could affect the users socially and in the longer term, thinking not only about the functional outcome but also the psychological and ethical outcomes.
While the world has moved towards ubiquitous connectivity, changing how people socially interact, little is known about children and teenagers’ emotional safety and privacy with disruptive technologies like Virtual Reality (VR). My PhD project aims at protecting children and adolescents by creating a framework for parents, moderating and limiting the social VR experience without completely restricting the expression of the children and adolescents.
Facilitating Parental Insight and Moderation for Social Virtual / Mixed Reality
N.B. This proposal is part-funded by Facebook Reality Labs. As affordable, consumer-oriented mixed reality headsets find their way into the home, it becomes increasingly likely such technology will see adoption by children and adolescents, particularly for social Augmented and Virtual Reality (AR/VR). Where a new disruptive technology has entered the market, parental understanding, supervision, and controls have typically lagged, leading to a window (often years wide) where children and adolescents experience unsupervised access. Whilst this can be beneficial (e.g. in terms of technological literacy), historically there have been examples where this lack of safeguards has led to children experiencing new forms of bullying, harassment and abuse, often unbeknownst to parents.
This lack of parental insight and control is particularly important when we consider what new forms of potential misuses and abuses are made possible by embodied social experiences. For example, our own research has shown how VR can be differently affective compared to non-VR  and explored the ethical  and security  challenges posed by these technologies. This project aims to explore potential safeguards for adolescent use of social mixed reality experiences. The main objectives are to:
1) Understand how parents might moderate and limit the social VR experience, informed by prior practice in 2D/social web platforms. For example, Social VR could offer a safe platform for self-discovery and expression when growing up, and there is a tension between protecting children whilst avoiding stifling this growth / freedom of expression that needs to be explored. What moderation approaches are needed; can we adequately sense the circumstances in which they should apply; and how should moderation be enacted (e.g. invisibly to the child, or perhaps via an artificial agent serving as a proxy for the parent’s supervision)?
2) Develop methods to provide parents with insight into current/past social VR experiences. Insight is important because parents/guardians can help children process traumatic or difficult experiences and inform use of parental moderation controls. Given some knowledge of these social, online activities, parents will be empowered to help guide their children through these new worlds. There are challenges here regarding how we can identify, or enable self-report of, key sensitive events, and how to present this information in forms parents can easily manage and understand. We envisage journaling approaches (e.g. video/textual excerpts and descriptions) could enable parents to gain retrospective insights for example, whilst more real-time alerts might be more prescient for younger children, or particularly sensitive events.
This project will explore these challenges both qualitatively (e.g. surveys, focus groups, interviews) and quantitatively (e.g. capturing and interpreting the social signals and metadata made available in social VR experiences such as VR Chat and Facebook Horizon). The successful student will explore the risks of social mixed reality towards children and adolescents from an HCI / usable security perspective, and prototype (e.g. through co-design) tools for parental insight and moderation, evaluating them in terms of safety, efficacy, and the extent to which they support freedom of expression. This project is aligned with the interests of Facebook Reality Labs, which is one of the leading drivers in building the future of connection within virtual and augmented reality. The project has the potential for high impact in guiding the future of social mixed reality for children and adolescents.
 Graham Wilson and Mark McGill. 2018. Violent Video Games in Virtual Reality: Re-Evaluating the Impact and Rating of Interactive Experiences. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’18). DOI:https://doi.org/10.1145/3242671.3242684
 Jan Gugenheimer, Mark McGill, Samuel Huron, Christian Mai, Julie Williamson, and Michael Nebeling. 2020. Exploring Potentially Abusive Ethical, Social and Political Implications of Mixed Reality Research in HCI. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20). DOI:https://doi.org/10.1145/3334480.3375180
 Ceenu George, Daniel Buschek, Mohamed Khamis, Heinrich Hussmann. Investigating the Third Dimension for Authentication in Immersive Virtual Reality and in the Real World. In Proceedings of the 26th IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR 2019).
My interest in human behaviour and emotion science was further cultivated by my time at York, when I was enrolled there for my BSc in Psychology. In my final year, I specialised and designed my dissertation project around the Dangerous Decisions theory and first impressions of faces. Furthermore, I also conducted my literature survey on whether the modulation of human emotions is capable via artificial intelligence. Both of these independent research avenues really honed my interest in on the relationship between human behaviour and artificial intelligence, especially in relation to emotions, micro-expressions and body language.
My PhD project aims to modulate cognitive models of emotional intelligence via the use of Magnetic Resonance Imaging and EEG to build cognitive models that explain modulation of brain activity in regions associated with emotions. The aim of this project is to build data-driven cognitive models of real-time brain network interaction during emotional modulation via said neurofeedback techniques. In my opinion, this would open up new avenues for the current field of wearable EEG technology.
I am enthused for this opportunity within the SOCIAL AI CDT, as the combination of psychology and the computer science behind it really caught my eye. All in all, I hope that my contribution will bring to light novel perspectives and information on the relationship between how the brain functions and possible translations into artificial intelligence.
Modulating Cognitive Models of Emotional Intelligence
State-of-the-art artificial intelligent (AI) systems mimic how the brain processes information to develop systems with unprecedented accuracy and performance in accomplishing tasks such as object/face recognition and text/speech translation. However, one key characteristic that defines human success is emotional intelligence. Empathy, the ability to understand others’ people feelings and emotionally reflect upon them, shapes social interaction and it is important in both personal and professional success. Although, some progress has been achieved in developing systems that detect emotions based on facial expressions and physiological data, a way of relating and reflecting upon them is far more challenging. Therefore, understanding how empathy/emotional responses emerge via complex information processing between key brain regions is of paramount importance to develop emotionally-aware AI agents.
In this project, we will exploit real-time functional Magnetic Resonance Imaging (fMRI) neurofeedback techniques to build cognitive models that explain modulation of brain activity in key regions related to empathy and emotions. For example, anterior insula is a brain region located in deep gray matter and it has been consistently implicated in empathy/emotional responses and abnormal emotional processing observed in several disorders such as Autism Spectrum Disorder and misophonia. Neurofeedback has shown promising results in regulating the activity of anterior insula and it could enable therapeutic training techniques (Kanel et al. 2019).
This approach would extract how brain regions interact during neuromodulation and allow cognitive models to emerge in real-time. Subsequently, to allow training in more naturalistic environments we suggest cross-domain learning between fMRI and EEG. The motivation behind this is that whereas fMRI is the gold standard imaging technique for deep gray matter structures it is limited by the lack of portability, comfort in use and low temporal resolution (Deligianni et al. 2014). On the other hand, advances in wearable EEG technology show promising results in the use of the device beyond well-controlled lab experiments. Toward this end advanced machine learning algorithms based on representation learning and domain generalisation would be developed. Domain/Model generalisation in deep learning aims to learn generalised features and extract representations in an ‘unseen’ target domain by eliminating bias observed via multiple source domain data (Volpi et al. 2018) .
Summarising, the overall aims of the project are:
- To build data-driven cognitive models of real-time brain network interaction during emotional modulation via neurofeedback techniques.
- To develop advanced machine learning algorithm to perform cross-domain learning between fMRI and EEG.
- To develop intelligent artificial agents based on portable EEG systems to successfully regulate emotional responses, taking into account cognitive models derived in the fMRI scanner.
My research with SOCIAL AI CDT focuses on developing an electronic health application for stress management. Through analysing opportunities and challenges in the fast-growing field of AI health applications, I wish to investigate which individual factors, design characteristics, and theoretical underpinnings make mobile health applications most effective.
Prior to my PhD studies, I completed my BSc in Psychology with Neuroscience specialisation at the University of Glasgow. For my undergraduate dissertation, I investigated learning effects in rating individual motives for eating behaviours and COVID-19-related stress, employing novel situated measures. As research intern, I reviewed what types of interventions aid behaviour change. Across multiple clinical positions, I learnt about the needs and possible applications for mobile health interventions. Drawing from all of the above, my PhD project allows me to follow my strong interest in improving individual mental and physical well-being.
Being part of the Social CDT programme, I am mostly looking forward to working with and learning from highly motivated people with unique research backgrounds. Overall, I wish to contribute to the development of large-scale, science-based health applications.
Improving engagement with mobile health apps by understanding (mis)alignment between design elements and personal characteristics
Mobile health apps have brought growing enthusiasm to delivering behavioural and health interventions at a low-cost and in a scalable fashion. Unfortunately, the potential impact of mobile health applications has typically been seriously limited by low user engagement and high drop-out rates. A number of studies have unpacked potential reasons for these problems, including non-optimal fit to a user’s problems, difficulty of use, privacy concerns, and low trustworthiness [TOR18]. Although best practices for developing engaging apps have been established, a growing consensus has concluded that improving engagement further requires personalisation at an individual level. Because, however, the factors that influence individual engagement are complex, individually personalised mobile health apps have rarely been developed.
Psychological literature provides numerous clues about how user interactions can be designed in more engaging ways based on personal characteristics. For instance, it is recommended to highlight rewards and social factors for extraverts, safety and certainty for neurotic individuals, achievements and structure for conscientious individuals [HIR12], and external vs internal factors for individuals with high vs low locus of control [CC14]. Developing and testing personalised mobile health apps based on each personal characteristic would require a long process and many A/B trials, together with significant efforts and costs. Perhaps this explains why personalisation has been limited in practice and why most of mobile health apps have been designed in a one-size-fits-all manner.
Project approach, objectives, and outcomes
Instead of designing and testing each personalised app element, the project here will pursue two novel approaches. First, we will conduct a retrospective exploration of previous app use as documented in the literature. Specifically, we will assess a) personal characteristics of individuals who have previously used mobile health apps, b) design elements (including intervention mechanisms) of these apps, and c) outcomes related to app engagement (e.g., drop-out rates, frequency of use). Of focal interest will be how personal characteristics and app design interact to produce different levels of app engagement. We aim to publish a major review of the literature based on this work.
Second, in a well-established stress app that we continue to develop, we will allow users to configure its design features in various ways. We will also collect data about users’ personal characteristics. From these data, we hope to develop design principles for tailoring future apps and intervention mechanisms to specific individuals. A series of studies will be performed in this line of work, together with related publications.
This project is likely to focus on stress as the primary use-case. In a related project, we are developing and evaluating stress apps that measure and predict stress in specific situations, linking psychological assessment to physiological data harvested implicitly from wearables. In a third project, we are implementing behaviour change interventions in digital health apps to reduce distress and increase eustress. Work from all three projects will be integrated to develop maximally effective stress apps, tailored to individuals, that effectively measure, predict, and alter stress experience.
Alignment with industrial interests
This work will be of a direct interest to a collaboration between Koa Health and the University of Glasgow to develop wellbeing services via digital health apps, including digital therapeutics. Not only does this work attempt to better understand and design health apps, it has the central aim of implementing actual apps for use by clinicians, health professionals, and the general population.
I am excited to be joining SOCIAL AI CDT at the University of Glasgow as a PhD student, researching the application of AI in detecting changes in affective states using human motion analysis.
I graduated with an undergraduate degree in Mathematics and Music and continued with my master’s in Mathematics, both at the University of Leeds. My final year dissertation was titled “Deciphering Turbulent Music” in which the aim was to determine if mathematically defined characteristic of turbulent wave would also be present in a piece of music that was considered to be ‘turbulent’-sounding. During my studies at university, my highlight was exploring the different applications of mathematics to the psychological effects of music, to which I hope to draw upon whilst undertaking my research for my PhD.
I am excited to be focusing once more on research, particularly to explore the opportunities to marry AI techniques with psychological study. Furthermore, I am looking forward to being a part of an interdisciplinary team and working alongside supervisors and students with such diverse backgrounds and talents.
Detecting Affective States based on Human Motion Analysis
Human motion analysis is a powerful tool to extract biomarkers for disease progression in neurological conditions, such as Parkinson disease and Alzheimer’s. Gait analysis has also revealed several indices that relate to emotional well-being. For example, increased gait speed, step length and arm swing has been related with positive emotions, whereas a low gait initiation reaction time and flexion of posture has related with negative feelings (Deligianni et al. 2019). Strong neuroscientific evidence show that the reason behind these relationships are due to an interaction between brain networks involved in gait and emotion. Therefore, it does not come to surprise that gait has been also related to mood disorders, such as depression and anxiety.
In this project, we aim to investigate the relationship between effective mental states and psychomotor abilities with relation to gait, balance and posture while emotions are modulated via augmented reality displays. The goal is to develop a comprehensive continuous map of interrelationships in both normal subjects and subjects affected by a mood disorder. In this way, we are going to derive objective measures that would allow to detect early signs of abnormalities and intervene via intelligent social agents. This is a multi-disciplinary project with several challenges to address:
- Build robust experimental setup of intuitive naturalistic paradigms.
- Develop AI algorithms to relate neurophysiological data with gait characteristics based on state-of-the-art motion capture systems (taking into account motion artefacts during gait)
- Develop AI algorithms to improve detection of gait characteristics via rgbd cameras (Gu et al. 2020) and possibly new assistive living technologies based on pulsed laser beam.
The proposed AI technology for social agents has several advantages. It can enable the development of intelligent social agents that would track mental well-being based on objective measures and provide personalised feedback and suggestions. In several cases, assessment is done based on self-reports via mobile apps. These measures of disease progression are subjective and it has been found that in major disorders they do not correlate well with objective evaluations. Furthermore, measurements of gait characteristics are continuous and they can reveal episodes of mood disorders that are not present when the subject visits a health practitioner. This approach might shed a light on subject variability with relation to behavioural therapy and provide more opportunities for earlier intervention (Queirazza et al. 2019). Finally, compared to other state-of-the-art effect recognition approaches, human motion analysis might pose less privacy issues and enhance users’ trust and comfort with the technology. In several situations, where facial expressions are not easy to track, human motion analysis is far more accurate in classifying subjects with mental disorders.
[DEL19] F Deligianni, Y Guo, GZ Yang, ‘From Emotions to Mood Disorders: A Survey on Gait Analysis Methodology’, IEEE journal of biomedical and health informatics, 2019.
[GUO19] Y Guo, F Deligianni, X Gu, GZ Yang, ‘3-D Canonical pose estimation and abnormal gait recognition with a single RGB-D camera’, IEEE Robotics and Automation Letters, 2019.
[XGU20] X Gu, Y Guo, F Deligianni, GZ Yang, ‘Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement.’, IEEE Transactions on Image Processing, 2020.
[QUE19] F Queirazza, E Fouragnan, JD Steele, J Cavanagh and MG Philiastides, Neural correlates of weighted reward prediction error during reinforcement learning classify response to Cognitive Behavioural Therapy in depression, Science Advances, 5 (7), 2019.
My academic journey began in earnest at Abertay University, where I had the opportunity to volunteer as a research assistant. In the 4th year, my dissertation allowed me to indulge my interest in imagination: I looked at the relationship between people’s self-reported vividness of imagery (how clearly an image forms in their head when they will it to) and the performance of their verbal memory when they were encouraged to imagine elements of a list visually. Through this project, my curiosity grew ever stronger.
After graduating from Abertay University with a BSc in Psychology and Human Resource Management, I secured a PhD position in SOCIAL AI CDT. In my PhD, I will be once again studying the nature of imagination. The aim of the project is to understand what areas of the brain are involved in imagination, and to model their function with the help of machine learning. We will be testing participants using established tests of visual imagery and using fMRI to measure their brain activity while performing certain tasks.
It is difficult to understand what it must be like to imagine differently – the experience of reading a book, listening to a rousing speech, appreciating visually evocative poetry: these are some of the things we all experience differently according to the vividness of our visual imagination.
To say I am excited to be given the opportunity to further study this phenomenon is an understatement – I believe that, in its own invisible way, imagination has a big impact on how we interact with each other and the world around us. I hope that, by studying how different we are, we might come to understand each other better in the end.
Towards modelling of biological and artificial perspective taken
Context and objectives
Visual imagery, e.g. the ability to form a visual representation of unseen stimuli, is a fundamental developmental step in social cognition. Being able to take the perspective of another observer is the focus of classic paradigms in theory of mind research such as Piaget’s landscape task: overturning an egocentric world view is reached around the age of 4 when children learn to simulate another person’s perspective towards a visual screen and imagine what is in sight of that person (Piaget, 2013).
Visual imagery might be one of the cognitive processes supported by extensive feedback connections from higher order areas and other modalities to the visual system (Clavagnier et al., 2004), as evidenced by the fact that sound content can be decoded from brain activity patterns in the early visual cortex of blindfolded participants (Vetter et al. 2014). Preliminary data from Muckli’s lab also suggests that this result cannot be reproduced in aphantasic participants who report an inability to generate visual imagery (Zeman, Dewar and Della Sala, 2015).
Our project aims to further explore the neural correlates of visual imagery and aphantasia by using neural decoding techniques, which allow the reconstruction of perceived features from human magnetic resonance imaging (fMRI) data (Raz et al, 2017). This method will allow us to detect shared representation networks between visual imagery and actual visual perception of the same objects, whether these networks are shared across participants, and whether they differ between aphantasics and non-aphantasics.
Proposed methods and expected results
We will use Ultra High Field fMRI to read brain activity while participants (aphantasics and non-aphantasics) are presented with either single-sentence descriptions of object categories (e.g. “a red chair”) or different visual exemplars from the same categories.
Our hypotheses are that, in the visual system, representations of the same categories: (1) will be generalizable between the auditory and visual conditions for the non-aphantasic group, but not for the aphantasic group, (2) will be less generalizable across aphantasics than non-aphantasics in the auditory condition, (3) that the previous two points will allow us to discriminate between aphantasics and non-aphantasic participants.
In Human-Computer Interaction (HCI), we recently developed computational models capable of representing physical and virtual space, solving the problems of how to recognise virtual spatial regions starting from the detected physical position of the users (Benford et al., 2016). We used the models to investigate cognitive dissonance, namely the inability or difficulty to interact with the virtual environment. In this project, we will adapt these computational models and apply them to cognition processes to test hypotheses 1-3. The end goal is to embed them within AI agents to enable empathic-seeming behaviours.
Impact for artificial social intelligence
Our proposal is relevant for the future development of creating and contrasting artificial agents with and without imagery, not only making AI more human-like, but adding the layer of complexity that is imagery-based representations. We outline a number of key questions where we hypothesize imagery has a function in social cognition, and where imagery-based artificially intelligent machines can be applied to social phenomena. To what extent is visual imagery in social AI an advantage? Simulate the perspective another agent has on a view and being able to match the perspective.
- Clavagnier, S., Falchier, A. & Kennedy, H. (2004) “Long-distance feedback projections to area V1: Implications for multisensory integration, spatial awareness, and visual consciousness”, Cognitive, Affective, & Behavioral Neuroscience 4, 117–126
- Piaget, J. (2013). Child’s Conception of Space: Selected Works vol 4 (Vol. 4). Routledge.
- Vetter, P., Smith, F.W., and Muckli, L. (2014), “Decoding Sound and Imagery Content in the Early Visual Cortex”, Current Biology, 24, 1256-1262.
- Zeman, A., Dewar, M., and Della Sala, S. (2015) “Lives without imagery — Congenital aphantasia”, Cortex, 73, 378-380.
- Raz, G., Svanera, M., Singer, N., Gilam G., Bleich, M., Lin, T., Admon, R., Gonen, T., Thaler, A., Granot, R.Y., Goebel, R., Benini, S., Valente, G. (2017) “Robust inter-subject audiovisual decoding in functional magnetic resonance imaging using high-dimensional regression”, Neuroimage, 163, 244-263
- Benford, S., Calder, M., Rodden, T., & Sevegnani, M. (2016). On lions, impala, and bigraphs: Modelling interactions in physical/virtual spaces. ACM Transactions on Computer-Human Interaction (TOCHI), 23(2), 9.
My PhD research will focus on developing a digital avatar that appears to be alive. More specifically, I will investigate what face and eye movements should be implemented on a digital avatar to produce a sentient appearance.
Prior to joining SOCIAL AI CDT at the University of Glasgow, I completed a BSc in Psychology at the University of Leeds and an MSc in Cognitive Neuroscience at the University of York. During my undergraduate studies, I worked with the Immersive Cognition laboratory (ICON; University of Leeds) on an EEG Brain-Computer Interface project which sparked my interest in the application of computational methods in the field of psychology. This experience led me to undertake the MSc course at York where I was able to develop my programming and neuroimaging skills further. In my MSc dissertation, I used fMRI to investigate the differences in neural representations of faces between healthy people and people with developmental prosopagnosia who suffer from an impaired ability to recognise faces. I am excited to bring my knowledge and experience into SOCIAL AI CDT, and I look forward to developing my skillset further whilst working with people from a variety of different academic backgrounds.
Developing a digital avatar that appears to be "alive"
How should we design a digital avatar so that it appears sentient—i.e. “alive”? Digital avatars can engage with humans to interact socially. However, how should we design such avatars so that they have a realistic appearance that promotes engagement with a human? Building on the strength of digital design avatars in the Institute of Neuroscience and Psychology and the social robotics research on the School of Computing Science, we will combine methods from human psychophysics, computer graphics, machine vision and social robotics to design such a sentient avatar (presented in VR or on a computer screen). We will start with the resting, default state of the avatar. Our research will aim to make it look like a sentient being (e.g. with a realistic appearance and spontaneous dynamic movements of the face and the eyes), who can then engage with humans (i.e. track their presence, engage with realistic eye contact and so forth).
Aims and Objectives
More specifically this project will attempt to achieve the following scientific and technological goals:
- Identify the default face movements (including eye movements) that produce a realistic sentient appearance.
- Implement those movements on a digital avatar which can be displayed on a computer screen or in VR.
- Use tracking software to detect human beings in the environment, follow their movements and engage with realistic eye contact.
- Develop models to link human behaviour with avatar movements to encourage engagement.
- Evaluate the performance of the implemented models through deployment in labs and in public spaces.
Output and Impact
Digital avatars that can engage their users and communicate accurate social information remain a research challenge with potentially broad applications in the digital economy and industry, where the tools developed to animate sentient avatars can also animate suitably designed social robots.
Zhan, J., Liu, M., Garrod, O.G.B., Daube, C., Ince, R.A.A., Jack, R.E. & Schyns, P.G. (2021). Modeling individual preferences reveals that face beauty is not universally perceived across cultures. Current Biology, 31, 1-10.
Chen, C., Hensel, L.B., Duan, Y., Ince, R.A.A., Garrod, O.G.B., Beskow, J., Jack, R.E. & Schyns, P.G. (2019). Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods. 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019), doi:10.1109/FG.2019.8756570.
Jack, R.E. & Schyns, P.G. (2017). Toward a Social Psychophysics of Face Communication. Annual Review of Psychology, 68, 269-297.
I’m delighted to be joining the SOCIAL AI CDT to hopefully improve cognitive training by combining VR, EEG and Machine Learning. This project is a natural extension of my prior experience. For my undergraduate degree, I completed a BSc in Psychology at the University of Dundee. During that time, I volunteered for a year in an EEG lab, which revealed the neuroscientific side of Psychology and motivated me to follow the more quantifiable aspects of it. Following from that, I decided to pursue a MSc in Health Data Science at the University of Aberdeen, which taught me strong foundations in data analysis using R, but also provided training in building Machine Learning models. As part of my final semester, I was lucky enough to undertake a placement with Public Health Scotland, requiring me to develop a data quality pipeline in the form of a R Shiny Dashboard. It taught me a lot about the challenges of developing a product, such as liaising with all the involved parties, but also showed how much joy there is in working on a lengthy project. It felt great to look back over the three months of the placement and see how far the dashboard has come. I can’t imagine what it will feel like after three years of working on the cognitive training delivery.
Evaluating and Shaping Cognitive Training with Artificial Intelligence Agents
Virtual reality (VR) has emerged as a promising tool for cognitive training for several neurological conditions (ie. mild cognitive impairment, acquired brain injury) as well as for enhancing healthy ageing and reducing the impact of mental health conditions (ie. anxiety and fear). Cognitive training refers to behavioural training that results in enhancement of specific cognitive abilities such as visuospatial attention and working memory. Using VR for such training offers several advantages towards achieving improvements, including its high level of versatility and its ability to dynamically adjust difficulty in real-time. Furthermore, it is an immersive technology and thus has great potential to increase motivation and compliance in subjects. Currently, VR and serious video games come in a wide variety of shapes and forms and the emerging data are difficult to quantify and compare in a meaningful way (Sokolov 2020).
This project aims to exploit machine learning to develop intuitive measures of cognitive training in a platform independent way. The project is challenging as there is great variability in cognitive measures even in well controlled/designed lab experiments (Learmonth et al., 2017; Benwell et al., 2014). So the objectives of the projects are:
- Predict psychological dimensions (ie. enjoyment, anxiety, valence and arousal) based on performance and neurophysiological data.
- Relate performance improvements (ie. learning rate) to psychological dimensions and physiological data (ie. EEG and eye-tracking).
- Develop artificial intelligence approaches that are able to modulate the VR world to control learning rate and participant satisfaction.
VR is a promising new technology that provides new means of building frameworks that will help to improve socio-cognitive processes. Machine learning methods that dynamically control aspects of the VR games are critical to enhanced engagement and learning rates (Darzi et al. 2019, Freer et al. 2020). Developing continuous measures of spatial attention, cognitive workload and overall satisfaction would provide intuitive ways for users to interact with the VR technology and allow the development of a personalised experience. Furthermore, these measures will play a significant role in objectively evaluating and shaping new emerging VR platforms and this approach will thus generate significant industrial interest.
[BEN14] Benwell, C.S.Y, Thut, G., Grant, A. and Harvey, M. (2014). A rightward shift in the visuospatial attention vector with healthy aging. Frontiers in Aging Neuroscience, 6, article 113, 1-11.
[DAR19] A. Darzi, T. Wondra, S. McCrea and D. Novak (2019). Classification of Multiple Psychological Dimensions in Computer Game Players Using Physiology, Performance, and Personality Characteristics. Frontiers in Neuroscience, 2019.
[FRE20] D. Freer, Y. Guo, F. Deligianni and G-Z. Yang (2020). On-Orbit Operations Simulator for Workload Measurement during Telerobotic Training. IEEE RA-L, https://arxiv.org/abs/2002.10594.
[LEA17] Learmonth, G., Benwell, C. S.Y., Thut, G. and Harvey, M. (2017). Age-related reduction of hemispheric lateralization for spatial attention: an EEG study. Neuro-Image, 153, 139-151.
[SOK20] A. Sokolov, A. Collignon and M. Bieler-Aeschlimann (2020). Serious video games and virtual reality for prevention and neurorehabilitation of cognitive decline because of aging and neurodegeneration. Current Opinion in Neurology, 33(2), 239-248.
I am delighted to join SOCIAL AI CDT at the University of Glasgow as a PhD student. I am very interested in the relationship between humans and computers – on a personal, societal and cultural level – and the opportunity to use technology to solve some of the wider problems that people encounter today. My research project is focused on understanding how motivation, belief and expectation shape user’s interactions with mobile apps for mental health and how this knowledge can be used to improve the effectiveness of those apps through personalised interventions.
Prior to my PhD studies I completed BSc in Interactive Media Design at Edinburgh Napier University. After that, I worked in a variety of tech companies as a User Experience researcher where I developed a range of qualitative and quantitative research skills. My last role involved working on an AI-driven HR solution that aimed to improve employee performance through personalised recommendations. This challenge also got me interested in the ethical side of AI and the new kinds of tensions that direct and indirect users of those systems experience.
While my main field of expertise is HCI I have an interest in Psychology and Social Sciences which I continue to develop alongside my career. I am also passionate about mental health and wellbeing and in my spare time I am training as a meditation teacher.
Priming expectation and motivation: Unpacking the placebo effect to improve effectiveness of mobile health apps
Numerous reports have highlighted that during the COVID-19 pandemic, poor mental health has been exacerbated globally (e.g., 1,2). There is a broad consensus that the most cost-effective measures for reducing the risk and prevalence of common mental disorders such as anxiety and depression are the implementation of evidence-based preventative interventions. Mobile health apps have brought a growing enthusiasm related to delivering behavioural and health interventions at low-cost and in a scalable fashion. A recent estimate identified in excess of 10,000 health and wellness apps designed for mental or behavioural health (3). Only a very small number of these have been evaluated for clinical efficacy despite claims of being evidence-based.
The effectiveness of a mobile application comprises measures of efficacy and engagement. To enhance effectiveness, it is necessary to deliver evidence-based techniques in an engaging manner which necessitates personalisation to the user–there is no one-size-fits all approach to mental health. Machine learning can create personalised recommendations based on other users and an individual user’s interaction with an app.
An individual’s beliefs, expectation and motivation are known to affect clinical efficacy. Factors such as these are thought to partly account for the placebo effect seen in trials and studies into a range of physical and mental health conditions (4). Yet there is a paucity of research that aims to understand and leverage these for clinical benefit. Improved understanding and application of the placebo effect could enhance the clinical efficacy of treatments including mobile health applications for mental health. From a grounded cognition perspective (5,6), we predict that personalisation and engagement of a mobile health intervention that creates vivid, multimodal expectations of desired outcomes may lead to stronger effects that can’t be attributed to the intervention alone.
Aims and objectives
This project will deepen understanding of how to personalise mobile health apps to user’s expectations, beliefs and motivation aiming to improve the engagement and ultimately the effectiveness of the intervention. The main objectives will be the following:
– Identify key factors that contribute to the placebo effect
– Explore whether manipulation of these factors can improve the effectiveness of interventions for mental well-being.
– Explore how these factors could be objectively or subjectively measured for individual users
– Explore integration of these factors into an AI-driven recommender system to personalise interventions within apps for mental health.
Extensive literature research will be first conducted to characterise the factors that contribute to the placebo effect. This will result in a set of hypotheses on the relationship between measure of, for example, motivation, expectation and beliefs, and a theoretical framework to integrate them and to derive novel predictions for use in mobile health applications. Methods of manipulating these factors will also be identified. Studies may then be undertaken to directly manipulate these factors to determine whether these change the effectiveness of an app for mental well-being. By applying standard statistical methods as well as machine learning (to unpack more complex interplay between factors and content/design elements), these data will be used to identify predictors of effectiveness. The learnings will be used to design and test personalisation in a real-world scenario.
- Czeisler, M. É., Lane, R. I., Petrosky, E., Wiley, J. F., Christensen, A., Njai, R., … Rajaratnam, S. M. W. (2020). Mental Health, Substance Use, and Suicidal Ideation During the COVID-19 Pandemic — United States, June 24–30, 2020. Morbidity and Mortality Weekly Report, 69(32), 1049–1057. https://doi.org/10.15585/mmwr.mm6932a1
- Vindegaard, N., & Eriksen Benros, M. (2020). COVID-19 pandemic and mental health consequences. Systematic review of the current evidence. https://doi.org/10.1016/j.bbi.2020.05.048
- Carlo, A. D., Hosseini Ghomi, R., Renn, B. N., & Areán, P. A. (2019). By the numbers: ratings and utilization of behavioral health mobile applications. Npj Digital Medicine, 2(1), 54. https://doi.org/10.1038/s41746-019-0129-6
- Price DD, Finniss DG, Benedetti F. A comprehensive review of the placebo effect: recent advances and current thought. Annu Rev Psychol. 2008;59:565-90. doi: 10.1146/annurev.psych.59.113006.095941. PMID: 17550344.
- Barsalou, L. W. (2009). Simulation, situated conceptualization, and prediction. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1521), 1281–1289. https://doi.org/10.1098/rstb.2008.0319
- Papies, E. K., Barsalou, L. W., & Rusz, D. (2020). Understanding Desire for Food and Drink: A Grounded-Cognition Approach. Current Directions in Psychological Science, 29(2), 193–198. https://doi.org/10.1177/0963721420904958
I am a current PhD student within the SOCIAL AI CDT at the University of Glasgow. My research will focus on partnering older drivers with semi-autonomous driving technology. Using a driving simulator, I will look at measuring basic driving performance such as steering, gaze and responses to distractors, in order to build a good model for older drivers’ behaviour under different driving scenarios. I will investigate how the car could work with the older driver, augmenting and enhancing their spatial and cognitive abilities, and thus providing support rather than increased distraction/complication.
Prior to joining SOCIAL CDT, I completed my BSc degree in biochemical engineering at the Technion, Israel’s institute of technology. Throughout the years I developed an interest in understanding the interaction between humans and technology. Therefore, I decided to pursue a master’s degree in Industrial engineering – Human factors track – at Ben Gurion University, Israel. During my masters I investigated the effect of fatigue progression while driving a level 2 automation simulator and performing a secondary task on hazard perception ability. Participants navigated through pre-planned driving scenarios and faced various road hazards. I studied their hazard perception ability and fatigue progression under manual, level 2 automation or level 2 with a secondary task driving modes.
After finishing my MSc degree, I looked for opportunities to expand my research. SOCIAL AI CDT was a perfect match to my research interest as the PhD position allows me to expand my research in a way that covers the intriguing group of the elderly.
Being part of the SOCIAL AI CDT programme, I look forward to contributing to the integration of the older population with recent driving technology. I strongly believe that technology should be inclusive, and this program is a great opportunity to look into ways for maximizing the benefit of technology for the elderly.
Partnering older drivers with new driving technology
Main aims and objectives
At present there are over 12million people aged 65+ in the UK, but as yet there has been little investigation of how older drivers cope with advanced driving technologies such as head-up displays, augmented reality windscreens, or semi-autonomous vehicles. As they age, drivers’ cognitive and spatial abilities in particular, change (Maerker et al. 2019; Learmonth et al., 2018) and increased prevalence of diseases, such as stroke further impact these difficulties (Rossit et al., 2012), enhancing accident risks.
In addition, if care is not taken, the increasing amount of information presented by advanced driving technologies can overload older drivers (Learmonth et al., 2017) and make driving even less safe. The aim of this project is instead to ‘partner’ the car to the older driver: we will investigate how the car could work with the older driver, augmenting and enhancing their spatial and cognitive abilities, and thus providing support rather than increased distraction/complication.
Using a simulator, the first stage of the work will look at measuring basic driving performance such as steering, gaze and responses to distractors, to allow us to build a good model for older drivers’ behaviour under different driving scenarios.
Two potential example areas will then be considered to support older drivers:
Augmented reality windscreens: these could adaptively provide an enhancement of the environment, other vehicles or obstacles on the road, thus making them easier to recognise for those with poorer vision. One example would be automatic obstacle detection with the car further enhancing edges to make them stand out more against the background. Additional modalities such as 3D audio and haptic cues could be further areas of investigation.
Semi-autonomous driving: A semi-autonomous NHTSA Level 3+ car could offer to take over the driving if the driver appears to need support, an option most likely to be taken up by older drivers, especially those with age-related diseases such as stroke . However, once in autonomous mode, vigilance is required so that the driver can take back control, if needed. Yet maintaining vigilance is a known issue for older populations, so maintaining enough situational awareness to take over driving may well prove very difficult. We will study this problem and look at how the car can hand back control to the driver in a way that does not cause more problems.
At the same time we will investigate driver acceptance of these new technological aids to ensure that the tools we design are acceptable to older drivers.
We will work with Brewster’s driving simulator which allows us to run controlled studies with inbuilt tools such as eye tracking, gesture tracking and driving performance measurements, along with in-car displays of different types. These can be run in either VR or on a large projection screen. For example, with eye-tracking we can monitor where drivers look in the scene and if they are neglecting particular areas, we can encourage them to move their gaze.