The problem of symbol grounding attempts to associate symbols from language with a corresponding referent in the environment. Traditionally, research has focused on identifying single objects and their properties. Similarly, affordances learning in robotics tend to focus on single objects. These approaches often do not consider the full context of the environment, which contains multiple different objects and their properties as well as relationships among the objects. Furthermore, the state of the environment (e.g., which relationships are true, etc.) affords (i.e., permits executing) certain actions. This project hypothesizes that the grounding process must consider the full context of the environment in order to perform symbol grounding and to better adapt to a new language and a new environment.

This project aims to develop a novel approach to grounding that lifts it to the relational level, where an agent reasons about the relationships between multiple symbols in the language and between multiple referents in the environment and it is able to learn affordances that capture the relationship between objects and their properties, actions, and the environment. Learning such relationships will require combining information from different modalities such as language and perception. We will evaluate our approach with a robot that operates in a kitchen-like environment. The robot will be trained by being presented with a series of demonstrations involving inputs from multiple modalities (language and perception). Then, it will be evaluated by being placed in an unseen environment where it will be forced to adapt to its new setting and interpret, possibly unimodal input (i.e., only language or only perception), in order to correctly carry out the requested tasks.

Project partners

• KU Leuven - Belgium
• Koc University - Turkey
• Örebro University - Sweden