Motivation for the study: A growing body of evidence suggests that integrated technologies of brain-computer interfaces (BCI) and virtual reality (VR) environments provide a flexible platform for a series of neurorehabilitation therapies, including significant post-stroke motor recovery and cognitive-behavioural therapy. When immersed in such an environment, the subject's perceptual level of social interaction is often impaired due to the sub-optimal quality of the interface lacking the social aspect of human interactions.

Project objective: We propose a user-friendly wearable low-power smart BCI system with an ecologically valid VR environment in which both the patient and therapist collaboratively interact via their person-specific avatar representations. On the one hand, the patient voluntarily, and in a self-paced manner, manages their activity in the environment and interacts with the therapist via a BCI-driven mental imagery process. This process is computed and rendered in real-time on an energy efficient wearable device. On the other hand, the therapist's unlimited motor and communication skills allow him to fully control the environment. Thus, the VR environment may be flexibly modified by the therapist allowing for different occupational therapy scenarios to be created and selected following the patient's recovery needs, mental states, and instantaneous responses.

Implementation: Careful attention will be paid to balance known neurophysiological evidence of the process with artificial intelligence (AI) within the active BCI protocols to avoid running into conceptual pitfalls. Computed features of EEG signals will serve to monitor the patient's engagement, cognitive workload, or mental fatigue in real-time. These indicators will be combined with observable patient’s performance and behaviours to improve the accuracy of mental state estimation. Exceeding critical mental state levels will signal the therapist to activate appropriate countermeasures in the form of environmental and task changes.

Research and technological challenges: To challenge and overcome existing technologies, commercially available head-mounted VR displays (HMD) combined with miniaturized energyefficient microcontroller units will be employed for EEG signal processing, BCI discrimination and on-board classification implementation, and a full-duplex communication with the HMD controllers. Advanced dry EEG sensors suitable to operate and be placed on the scalp without interfering with the HMD will be developed and tested. A novel patient-to-therapist multimodal collaborative environment augmented through VR immersion and by AI monitored patient’s brain activity will be created. By combining these pieces, a low-power wearable BCI-HMD system will be constructed. A series of clinical studies will validate the system.

Project partners

• Institute of Measurement Science, Slovak Academy od Sciences - Slovakia (Coordinator)
• Laboratory of Intelligent Interfaces of Information and Communication Systems, Technical University Kosice - Slovakia
• Ben-Gurion University of the Negev - Israel
• Sensomedical labs ltd. - Israel
• Institute of Electronics and Computer Science - Latvia
• Lodz University of Technology - Poland
• Eidgenössische Technische Hochschule Zürich - Switzerland