smartTalk: Learning for improved HRI

smartTalk is a learning-based framework for natural-language human-robot interaction (HRI). The primary goal of this framework is to enable non-expert users to control and program a mobile robot using natural language commands. smartTalk is modality-agnostic, and is capable of integrating with both speech and non-speech (e.g., gesture-based) communication. Initially, robots using this mechanism are equipped with a limited vocabulary of primitive commands and functionality; however, through extended use and interaction, the robots are able to learn new commands and adapt to user’s behaviors and habits. This makes the proposed framework highly desirable for long-term deployment in a variety of HRI tasks. Code can be found on smartTalk's  Github page 

Below shows our framework's flow. Given a command, smartTalk will either 1) understand the command, add it to the knowledge bank, and execute. or 2) be unable to understand the command, and prompt the user for further information.

ST System diagram
Field trials on a Husky


We performed various field tests on the outdoor Husky Unmanned Ground Vehicle (UGV) and the indoor Turtlebot, showing generalization across platforms and robots.