Exploring Situated Interactions : Training video game AI agents to understand human instructions

Building interactive agents for video game worlds

The human behaviour is complex. Even a simple command like \”Put ball near box\” requires a deep understanding of the context and the language. It can be hard to understand the meaning of words like \”close\” – the speaker may want the ball next to the box, even though technically the ball is inside the box. To correctly respond to a request, a person must understand and judge the context and situation.

Researchers in artificial intelligence (AI), however, believe it is impossible to write computer code that can capture the subtleties of situated interactions. Modern machine learning (ML), on the other hand, has focused on data-driven learning of these interactions. We created a research environment within a video-game environment to explore these learning approaches and build agents quickly that can understand human instructions and safely execute actions in open-ended situations.

We’re publishing today a paper, along with a collection of videos that show our early steps towards building AIs for video games that can grasp fuzzy human concepts and, therefore, begin interacting with people in their own terms.