Real-time Human-Robot interaction: Unlocking future of robotic learning

Talking with Robots in Real Time

The SHRDLU experiment in the late 1960s was the beginning of a grand vision of robot learning. It is a world of helpful robots who live in human environments and obey dozens of commands in natural language. In the past few years, machine learning (ML), both in simulations and real-world systems, has made significant progress in instruction following. Recent Palm-SayCan research has created robots that use language models to plan behaviors for the long term and reason about abstract goals. Code as Policies demonstrated that pre-trained perceptions systems combined with code-generating models can produce language-conditioned policies for robot manipulation. Real-time interaction with humans is an important feature that robot learning systems lacking in current \”language-in, actions-out\” systems.

In an ideal world, robots would respond in real-time to any task that a human could describe using natural language. In open human environments it is important that end users can customize robot behavior in real time, by offering quick corrections (stop, move your hand up a little) or specifying constraints (“nudge this slowly to the left”). Real-time language can also make it easier to collaborate with robots on long-term, complex tasks. People could guide robot manipulation by providing feedback in real-time.