Cohort 3 (2021-2025)
Shaul Ashkenazi
My project will focus on the technological and sociolinguistic aspects of deploying a robot in a real-world context. This context is going to be a cultural centre of an NGO working with refugees and asylum seekers. We will use the Furhat robot, a social robot with human-like expressions and conversational AI capabilities. It will provide essential signposting in a non-English language, showcasing the importance and relevance of social robots.
I come from Israel, where I earned a Bachelor’s in Computer Science from the Academic College of Tel-Aviv-Yaffo, and a master’s in Linguistics from Tel Aviv University. In my Master’s thesis, I explored the topic of recovery strategies in a conversation between senior adults and a speaking chatbot.
I have a great interest in Human-Robot Interaction and marginalized populations, and I am excited to be a part of the Social AI CDT programme.
Deploying Social Robots for Multilingual Accessibility: Supporting Newcomers and Staff in Public Service Spaces
The increasing availability of socially-intelligent robots with functionality for a range of purposes, from guidance in museums [Geh15], to companionship for the elderly [Heb16], has motivated a growing number of studies attempting to evaluate and enhance Human-Robot Interaction (HRI). But, as Honig and Oron-Gilad’s review of recent work on understanding and resolving failures in HRI observes [Hon18], most research has focussed on technical ways of improving robot reliability. They argue that progress requires a “holistic approach” in which “[t]he technical knowledge of hardware and software must be integrated with cognitive aspects of information processing, psychological knowledge of interaction dynamics, and domain-specific knowledge of the user, the robot, the target application, and the environment” (p.16). Honig and Oron-Gilad point to a particular need to improve the ecological validity of evaluating user communication in HRI, by moving away from experimental, single-person environments, with low-relevance tasks, mainly with younger adult users, to more natural settings, with users of different social profiles and communication strategies, where the outcome of successful HRI matters.
The main contribution of this PhD project is to develop an interdisciplinary approach to evaluating and enhancing communication efficacy of HRI, by combining state-of-the-art social robotics with theory and methods from socially-informed linguistics [Cou14] and conversation analysis [Cli16]. Specifically, the project aims to deploy a state-of-the-art HRI system similar to the recent MultiModal Mall Entertainment Robot [Fos16], which was successfully deployed in a Finnish shopping mall for 14 weeks in the autumn of 2019 [Fos19]. Deploying a robot in a public context requires an interaction model which is socially acceptable, helpful and entertaining for multiple, diverse users in a real-world context. As part of the project, a similar social robot system will be developed and deployed in a new sociolinguistic and educational context in The Hunterian, the Museum and Art Gallery at the University of Glasgow. Glasgow is Scotland’s largest, and most socially and ethnically-diverse city, and deployment in The Hunterian provides a unique opportunity to test HRI with users from a wide range of demographic backgrounds. The robot deployments will continue throughout the PhD project in order for the impact of any technical and design modifications to be assessed.
Project objectives are to:
- Carry out a series of sociolinguistically-informed observational studies of HRI in situ with users from a range of social, ethnic, and language backgrounds, using direct and indirect methods
- Identify the minimal requirements (dialogue, non-verbal, other) to optimise HRI in this context, and thereby enhance user experience and engagement, also considering indices such as visitor surveys and attendance
- Implement the identified modifications to the robot system, and re-evaluate with new users.
References
[Cli16] Clift, R. (2016). Conversation Analysis. Cambridge: Cambridge University Press.
[Cou14] Coupland, N., Sarangi, S., & Candlin, C. N. (2014). Sociolinguistics and social theory. Routledge.
[Fos16] Foster M.E., Alami, R., Gestranius, O., Lemon, O., Niemela, M., Odobez, J-M., Pandey, A.M. (2016) The MuMMER Project: Engaging Human-Robot Interaction in Real-World Public Spaces. In: Agah A., Cabibihan J., Howard A., Salichs M., He H. (eds) Social Robotics. ICSR 2016. Lecture Notes in Computer Science, vol 9979. Springer, Cham.
[Fos19] Foster M.E. et al. (2019) MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces. In Proceedings of AI-HRI 2019.
[Geh15] Gehle R., Pitsch K., Dankert T., Wrede S. (2015). Trouble-based group dynamics in real-world HRI – Reactions on unexpected next moves of a museum guide robot., in 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015 (Kobe), 407–412.
[Heb16] Hebesberger, D., Dondrup, C., Koertner, T., Gisinger, C., Pripfl, J. (2016).Lessonslearned from the deployment of a long-term autonomous robot as companion inphysical therapy for older adults with dementia: A mixed methods study. In: TheEleventh ACM/IEEE International Conference on Human Robot Interaction, 27–34.
[Hon18] Honig, S., & Oron-Gilad, T. (2018). Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development. Frontiers in Psychology, 9, 861.
Nicole Lai
I am excited to be joining Social AI CDT at the University of Glasgow as a PhD student, researching the application of AI in detecting changes in affective states using human motion analysis.
I graduated with an undergraduate degree in Mathematics and Music and continued with my master’s in Mathematics, both at the University of Leeds. My final year dissertation was titled “Deciphering Turbulent Music” in which the aim was to determine if mathematically defined characteristic of turbulent wave would also be present in a piece of music that was considered to be ‘turbulent’-sounding. During my studies at university, my highlight was exploring the different applications of mathematics to the psychological effects of music, to which I hope to draw upon whilst undertaking my research for my PhD.
I am excited to be focusing once more on research, particularly to explore the opportunities to marry AI techniques with psychological study. Furthermore, I am looking forward to being a part of an interdisciplinary team and working alongside supervisors and students with such diverse backgrounds and talents.
Brain-Computer Interfaces to Personalise Rhythmic Interventions in Human Motion Analysis
Human motion analysis is a powerful tool to extract biomarkers for disease progression in neurological conditions, such as Parkinson disease and Alzheimer’s. Gait analysis has also revealed several indices that relate to emotional well-being. For example, increased gait speed, step length and arm swing has been related with positive emotions, whereas a low gait initiation reaction time and flexion of posture has related with negative feelings (Deligianni et al. 2019). Strong neuroscientific evidence show that the reason behind these relationships are due to an interaction between brain networks involved in gait and emotion. Therefore, it does not come to surprise that gait has been also related to mood disorders, such as depression and anxiety.
In this project, we aim to investigate the relationship between effective mental states and psychomotor abilities with relation to gait, balance and posture while emotions are modulated via augmented reality displays. The goal is to develop a comprehensive continuous map of interrelationships in both normal subjects and subjects affected by a mood disorder. In this way, we are going to derive objective measures that would allow to detect early signs of abnormalities and intervene via intelligent social agents. This is a multi-disciplinary project with several challenges to address:
- Build robust experimental setup of intuitive naturalistic paradigms.
- Develop AI algorithms to relate neurophysiological data with gait characteristics based on state-of-the-art motion capture systems (taking into account motion artefacts during gait)
- Develop AI algorithms to improve detection of gait characteristics via rgbd cameras (Gu et al. 2020) and possibly new assistive living technologies based on pulsed laser beam.
The proposed AI technology for social agents has several advantages. It can enable the development of intelligent social agents that would track mental well-being based on objective measures and provide personalised feedback and suggestions. In several cases, assessment is done based on self-reports via mobile apps. These measures of disease progression are subjective and it has been found that in major disorders they do not correlate well with objective evaluations. Furthermore, measurements of gait characteristics are continuous and they can reveal episodes of mood disorders that are not present when the subject visits a health practitioner. This approach might shed a light on subject variability with relation to behavioural therapy and provide more opportunities for earlier intervention (Queirazza et al. 2019). Finally, compared to other state-of-the-art effect recognition approaches, human motion analysis might pose less privacy issues and enhance users’ trust and comfort with the technology. In several situations, where facial expressions are not easy to track, human motion analysis is far more accurate in classifying subjects with mental disorders.
References
[DEL19] F Deligianni, Y Guo, GZ Yang, ‘From Emotions to Mood Disorders: A Survey on Gait Analysis Methodology’, IEEE journal of biomedical and health informatics, 2019.
[GUO19] Y Guo, F Deligianni, X Gu, GZ Yang, ‘3-D Canonical pose estimation and abnormal gait recognition with a single RGB-D camera’, IEEE Robotics and Automation Letters, 2019.
[XGU20] X Gu, Y Guo, F Deligianni, GZ Yang, ‘Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement.’, IEEE Transactions on Image Processing, 2020.
[QUE19] F Queirazza, E Fouragnan, JD Steele, J Cavanagh and MG Philiastides, Neural correlates of weighted reward prediction error during reinforcement learning classify response to Cognitive Behavioural Therapy in depression, Science Advances, 5 (7), 2019.
Roberto Luciani
My academic journey began in earnest at Abertay University, where I had the opportunity to volunteer as a research assistant. In the 4th year, my dissertation allowed me to indulge my interest in imagination: I looked at the relationship between people’s self-reported vividness of imagery (how clearly an image forms in their head when they will it to) and the performance of their verbal memory when they were encouraged to imagine elements of a list visually. Through this project, my curiosity grew ever stronger.
After graduating from Abertay University with a BSc in Psychology and Human Resource Management, I secured a PhD position in Social AI CDT. In my PhD, I will be once again studying the nature of imagination. The aim of the project is to understand what areas of the brain are involved in imagination, and to model their function with the help of machine learning. We will be testing participants using established tests of visual imagery and using fMRI to measure their brain activity while performing certain tasks.
It is difficult to understand what it must be like to imagine differently – the experience of reading a book, listening to a rousing speech, appreciating visually evocative poetry: these are some of the things we all experience differently according to the vividness of our visual imagination.
To say I am excited to be given the opportunity to further study this phenomenon is an understatement – I believe that, in its own invisible way, imagination has a big impact on how we interact with each other and the world around us. I hope that, by studying how different we are, we might come to understand each other better in the end.
Towards modelling of biological and artificial perspective taken
Context and objectives
Visual imagery, e.g. the ability to form a visual representation of unseen stimuli, is a fundamental developmental step in social cognition. Being able to take the perspective of another observer is the focus of classic paradigms in theory of mind research such as Piaget’s landscape task: overturning an egocentric world view is reached around the age of 4 when children learn to simulate another person’s perspective towards a visual screen and imagine what is in sight of that person (Piaget, 2013).
Visual imagery might be one of the cognitive processes supported by extensive feedback connections from higher order areas and other modalities to the visual system (Clavagnier et al., 2004), as evidenced by the fact that sound content can be decoded from brain activity patterns in the early visual cortex of blindfolded participants (Vetter et al. 2014). Preliminary data from Muckli’s lab also suggests that this result cannot be reproduced in aphantasic participants who report an inability to generate visual imagery (Zeman, Dewar and Della Sala, 2015).
Our project aims to further explore the neural correlates of visual imagery and aphantasia by using neural decoding techniques, which allow the reconstruction of perceived features from human magnetic resonance imaging (fMRI) data (Raz et al, 2017). This method will allow us to detect shared representation networks between visual imagery and actual visual perception of the same objects, whether these networks are shared across participants, and whether they differ between aphantasics and non-aphantasics.
Proposed methods and expected results
We will use Ultra High Field fMRI to read brain activity while participants (aphantasics and non-aphantasics) are presented with either single-sentence descriptions of object categories (e.g. “a red chair”) or different visual exemplars from the same categories.
Our hypotheses are that, in the visual system, representations of the same categories: (1) will be generalizable between the auditory and visual conditions for the non-aphantasic group, but not for the aphantasic group, (2) will be less generalizable across aphantasics than non-aphantasics in the auditory condition, (3) that the previous two points will allow us to discriminate between aphantasics and non-aphantasic participants.
In Human-Computer Interaction (HCI), we recently developed computational models capable of representing physical and virtual space, solving the problems of how to recognise virtual spatial regions starting from the detected physical position of the users (Benford et al., 2016). We used the models to investigate cognitive dissonance, namely the inability or difficulty to interact with the virtual environment. In this project, we will adapt these computational models and apply them to cognition processes to test hypotheses 1-3. The end goal is to embed them within AI agents to enable empathic-seeming behaviours.
Impact for artificial social intelligence
Our proposal is relevant for the future development of creating and contrasting artificial agents with and without imagery, not only making AI more human-like, but adding the layer of complexity that is imagery-based representations. We outline a number of key questions where we hypothesize imagery has a function in social cognition, and where imagery-based artificially intelligent machines can be applied to social phenomena. To what extent is visual imagery in social AI an advantage? Simulate the perspective another agent has on a view and being able to match the perspective.
References
- Clavagnier, S., Falchier, A. & Kennedy, H. (2004) “Long-distance feedback projections to area V1: Implications for multisensory integration, spatial awareness, and visual consciousness”, Cognitive, Affective, & Behavioral Neuroscience 4, 117–126
- Piaget, J. (2013). Child’s Conception of Space: Selected Works vol 4 (Vol. 4). Routledge.
- Vetter, P., Smith, F.W., and Muckli, L. (2014), “Decoding Sound and Imagery Content in the Early Visual Cortex”, Current Biology, 24, 1256-1262.
- Zeman, A., Dewar, M., and Della Sala, S. (2015) “Lives without imagery — Congenital aphantasia”, Cortex, 73, 378-380.
- Raz, G., Svanera, M., Singer, N., Gilam G., Bleich, M., Lin, T., Admon, R., Gonen, T., Thaler, A., Granot, R.Y., Goebel, R., Benini, S., Valente, G. (2017) “Robust inter-subject audiovisual decoding in functional magnetic resonance imaging using high-dimensional regression”, Neuroimage, 163, 244-263
- Benford, S., Calder, M., Rodden, T., & Sevegnani, M. (2016). On lions, impala, and bigraphs: Modelling interactions in physical/virtual spaces. ACM Transactions on Computer-Human Interaction (TOCHI), 23(2), 9.
Michal Pelikan
My PhD research will focus on developing a digital avatar that appears to be alive. More specifically, I will investigate what face and eye movements should be implemented on a digital avatar to produce a sentient appearance.
Prior to joining Social AI CDT at the University of Glasgow, I completed a BSc in Psychology at the University of Leeds and an MSc in Cognitive Neuroscience at the University of York. During my undergraduate studies, I worked with the Immersive Cognition laboratory (ICON; University of Leeds) on an EEG Brain-Computer Interface project which sparked my interest in the application of computational methods in the field of psychology. This experience led me to undertake the MSc course at York where I was able to develop my programming and neuroimaging skills further. In my MSc dissertation, I used fMRI to investigate the differences in neural representations of faces between healthy people and people with developmental prosopagnosia who suffer from an impaired ability to recognise faces. I am excited to bring my knowledge and experience into Social AI CDT, and I look forward to developing my skillset further whilst working with people from a variety of different academic backgrounds.
Developing a digital avatar that appears to be "alive"
Background
How should we design a digital avatar so that it appears sentient—i.e. “alive”? Digital avatars can engage with humans to interact socially. However, how should we design such avatars so that they have a realistic appearance that promotes engagement with a human? Building on the strength of digital design avatars in the Institute of Neuroscience and Psychology and the social robotics research on the School of Computing Science, we will combine methods from human psychophysics, computer graphics, machine vision and social robotics to design such a sentient avatar (presented in VR or on a computer screen). We will start with the resting, default state of the avatar. Our research will aim to make it look like a sentient being (e.g. with a realistic appearance and spontaneous dynamic movements of the face and the eyes), who can then engage with humans (i.e. track their presence, engage with realistic eye contact and so forth).
Aims and Objectives
More specifically this project will attempt to achieve the following scientific and technological goals:
- Identify the default face movements (including eye movements) that produce a realistic sentient appearance.
- Implement those movements on a digital avatar which can be displayed on a computer screen or in VR.
- Use tracking software to detect human beings in the environment, follow their movements and engage with realistic eye contact.
- Develop models to link human behaviour with avatar movements to encourage engagement.
- Evaluate the performance of the implemented models through deployment in labs and in public spaces.
Output and Impact
Digital avatars that can engage their users and communicate accurate social information remain a research challenge with potentially broad applications in the digital economy and industry, where the tools developed to animate sentient avatars can also animate suitably designed social robots.
References
Zhan, J., Liu, M., Garrod, O.G.B., Daube, C., Ince, R.A.A., Jack, R.E. & Schyns, P.G. (2021). Modeling individual preferences reveals that face beauty is not universally perceived across cultures. Current Biology, 31, 1-10.
Chen, C., Hensel, L.B., Duan, Y., Ince, R.A.A., Garrod, O.G.B., Beskow, J., Jack, R.E. & Schyns, P.G. (2019). Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods. 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019), doi:10.1109/FG.2019.8756570.
Jack, R.E. & Schyns, P.G. (2017). Toward a Social Psychophysics of Face Communication. Annual Review of Psychology, 68, 269-297.
Dominik Szczepaniak
I’m delighted to be joining the Social AI CDT to hopefully improve cognitive training by combining VR, EEG and Machine Learning. This project is a natural extension of my prior experience. For my undergraduate degree, I completed a BSc in Psychology at the University of Dundee. During that time, I volunteered for a year in an EEG lab, which revealed the neuroscientific side of Psychology and motivated me to follow the more quantifiable aspects of it. Following from that, I decided to pursue a MSc in Health Data Science at the University of Aberdeen, which taught me strong foundations in data analysis using R, but also provided training in building Machine Learning models. As part of my final semester, I was lucky enough to undertake a placement with Public Health Scotland, requiring me to develop a data quality pipeline in the form of a R Shiny Dashboard. It taught me a lot about the challenges of developing a product, such as liaising with all the involved parties, but also showed how much joy there is in working on a lengthy project. It felt great to look back over the three months of the placement and see how far the dashboard has come. I can’t imagine what it will feel like after three years of working on the cognitive training delivery.
Evaluating and Shaping Cognitive Training with Artificial Intelligence Agents
Virtual reality (VR) has emerged as a promising tool for cognitive training for several neurological conditions (ie. mild cognitive impairment, acquired brain injury) as well as for enhancing healthy ageing and reducing the impact of mental health conditions (ie. anxiety and fear). Cognitive training refers to behavioural training that results in enhancement of specific cognitive abilities such as visuospatial attention and working memory. Using VR for such training offers several advantages towards achieving improvements, including its high level of versatility and its ability to dynamically adjust difficulty in real-time. Furthermore, it is an immersive technology and thus has great potential to increase motivation and compliance in subjects. Currently, VR and serious video games come in a wide variety of shapes and forms and the emerging data are difficult to quantify and compare in a meaningful way (Sokolov 2020).
This project aims to exploit machine learning to develop intuitive measures of cognitive training in a platform independent way. The project is challenging as there is great variability in cognitive measures even in well controlled/designed lab experiments (Learmonth et al., 2017; Benwell et al., 2014). So the objectives of the projects are:
- Predict psychological dimensions (ie. enjoyment, anxiety, valence and arousal) based on performance and neurophysiological data.
- Relate performance improvements (ie. learning rate) to psychological dimensions and physiological data (ie. EEG and eye-tracking).
- Develop artificial intelligence approaches that are able to modulate the VR world to control learning rate and participant satisfaction.
VR is a promising new technology that provides new means of building frameworks that will help to improve socio-cognitive processes. Machine learning methods that dynamically control aspects of the VR games are critical to enhanced engagement and learning rates (Darzi et al. 2019, Freer et al. 2020). Developing continuous measures of spatial attention, cognitive workload and overall satisfaction would provide intuitive ways for users to interact with the VR technology and allow the development of a personalised experience. Furthermore, these measures will play a significant role in objectively evaluating and shaping new emerging VR platforms and this approach will thus generate significant industrial interest.
References
[BEN14] Benwell, C.S.Y, Thut, G., Grant, A. and Harvey, M. (2014). A rightward shift in the visuospatial attention vector with healthy aging. Frontiers in Aging Neuroscience, 6, article 113, 1-11.
[DAR19] A. Darzi, T. Wondra, S. McCrea and D. Novak (2019). Classification of Multiple Psychological Dimensions in Computer Game Players Using Physiology, Performance, and Personality Characteristics. Frontiers in Neuroscience, 2019.
[FRE20] D. Freer, Y. Guo, F. Deligianni and G-Z. Yang (2020). On-Orbit Operations Simulator for Workload Measurement during Telerobotic Training. IEEE RA-L, https://arxiv.org/abs/2002.10594.
[LEA17] Learmonth, G., Benwell, C. S.Y., Thut, G. and Harvey, M. (2017). Age-related reduction of hemispheric lateralization for spatial attention: an EEG study. Neuro-Image, 153, 139-151.
[SOK20] A. Sokolov, A. Collignon and M. Bieler-Aeschlimann (2020). Serious video games and virtual reality for prevention and neurorehabilitation of cognitive decline because of aging and neurodegeneration. Current Opinion in Neurology, 33(2), 239-248.
Rawan Zreik-Srour
I am a current PhD student within the Social AI CDT at the University of Glasgow. My research will focus on partnering older drivers with semi-autonomous driving technology. Using a driving simulator, I will look at measuring basic driving performance such as steering, gaze and responses to distractors, in order to build a good model for older drivers’ behaviour under different driving scenarios. I will investigate how the car could work with the older driver, augmenting and enhancing their spatial and cognitive abilities, and thus providing support rather than increased distraction/complication.
Prior to joining the CDT, I completed my BSc degree in biochemical engineering at the Technion, Israel’s institute of technology. Throughout the years I developed an interest in understanding the interaction between humans and technology. Therefore, I decided to pursue a master’s degree in Industrial engineering – Human factors track – at Ben Gurion University, Israel. During my masters I investigated the effect of fatigue progression while driving a level 2 automation simulator and performing a secondary task on hazard perception ability. Participants navigated through pre-planned driving scenarios and faced various road hazards. I studied their hazard perception ability and fatigue progression under manual, level 2 automation or level 2 with a secondary task driving modes.
After finishing my MSc degree, I looked for opportunities to expand my research. The CDT was a perfect match to my research interest as the PhD position allows me to expand my research in a way that covers the intriguing group of the elderly.
Being part of the CDT programme, I look forward to contributing to the integration of the older population with recent driving technology. I strongly believe that technology should be inclusive, and this program is a great opportunity to look into ways for maximizing the benefit of technology for the elderly.
Partnering older drivers with new driving technology
Main aims and objectives
At present there are over 12million people aged 65+ in the UK, but as yet there has been little investigation of how older drivers cope with advanced driving technologies such as head-up displays, augmented reality windscreens, or semi-autonomous vehicles. As they age, drivers’ cognitive and spatial abilities in particular, change (Maerker et al. 2019; Learmonth et al., 2018) and increased prevalence of diseases, such as stroke further impact these difficulties (Rossit et al., 2012), enhancing accident risks.
In addition, if care is not taken, the increasing amount of information presented by advanced driving technologies can overload older drivers (Learmonth et al., 2017) and make driving even less safe. The aim of this project is instead to ‘partner’ the car to the older driver: we will investigate how the car could work with the older driver, augmenting and enhancing their spatial and cognitive abilities, and thus providing support rather than increased distraction/complication.
Using a simulator, the first stage of the work will look at measuring basic driving performance such as steering, gaze and responses to distractors, to allow us to build a good model for older drivers’ behaviour under different driving scenarios.
Two potential example areas will then be considered to support older drivers:
Augmented reality windscreens: these could adaptively provide an enhancement of the environment, other vehicles or obstacles on the road, thus making them easier to recognise for those with poorer vision. One example would be automatic obstacle detection with the car further enhancing edges to make them stand out more against the background. Additional modalities such as 3D audio and haptic cues could be further areas of investigation.
Semi-autonomous driving: A semi-autonomous NHTSA Level 3+ car could offer to take over the driving if the driver appears to need support, an option most likely to be taken up by older drivers, especially those with age-related diseases such as stroke . However, once in autonomous mode, vigilance is required so that the driver can take back control, if needed. Yet maintaining vigilance is a known issue for older populations, so maintaining enough situational awareness to take over driving may well prove very difficult. We will study this problem and look at how the car can hand back control to the driver in a way that does not cause more problems.
At the same time we will investigate driver acceptance of these new technological aids to ensure that the tools we design are acceptable to older drivers.
Proposed methods
We will work with Brewster’s driving simulator which allows us to run controlled studies with inbuilt tools such as eye tracking, gesture tracking and driving performance measurements, along with in-car displays of different types. These can be run in either VR or on a large projection screen. For example, with eye-tracking we can monitor where drivers look in the scene and if they are neglecting particular areas, we can encourage them to move their gaze.