Abstract

Though technology has increased our connectivity enormously in recent years (Skype, Zoom, speech-to-speech live translator; social networks, moocs, telepresence, and so on…), emotional connectivity is largely missing from these platforms. This problem is especially pronounced for the elderly living by themselves or in retirement homes, but this is an issue relevant for all of us, as we have witnessed during the Covid-19 pandemic when most of us were forced into social isolation.

We plan to focus on how technology, especially Social Robots with BCI, can promote social inclusion and facilitate human-to-human interaction at an emotional level. We will follow a user-centric, iterative design process that incorporates insights from Emotion AI, Social Robotics, BCI, and Cognitive Science. Initially, we are targeting elderly population, but in our later research we will consider other user groups such as people with autism, families and partners living apart for various reasons, and so on.

Our approach is to use small affective robots such as MiRo-E and Nao, and avatars on mobile platforms as facilitators. We plan to develop a BCI-based feedback architecture to get continuous biometric feedback from the users to assess their emotional and mental states to guide the interaction. The novelty of our approach will be in this BCI architecture that incorporates central nervous system (CNS) measurement devices (like EEG, NIRS) and autonomous nervous system (ANS) measurement devices (like GSR, HR, ECG, EMG). Our goal is to obtain this feedback through non-invasive means as much as possible, and also to develop a version of the system that is completely non-invasive. We aim to develop a multimodal sensory fusion model to combine information from different channels to assess the overall mood of the user, and to deploy deep learning techniques to learn the dynamics of individual user’s mood patterns.

We also plan to study the efficacy of various interaction modalities like games and conversations. In applying the user-centric design approach, we will use in-the-wild methodology to obtain user feedback while conducting our study and developing prototypes. We plan to study both short-term and long-term effects of using this technology-mediated approach on the emotional well being of the users. We also plan to incorporate cultural aspects in our model by studying user groups in different cultures including Poland, Turkey, Japan and India.