AI has the power to bring your favorite fictional characters to life, says technologist Kylan Gibbs. Introducing Caleb, an "AI agent" with personality and internal reasoning, he demonstrates how AI-powered characters can interact with people in novel ways, generate unique video game outcomes and augment our ability to tell stories, opening up new worlds of possibility.
Learn more about our flagship conference happening this April at attend.ted.com/podcast
Hosted on Acast. See acast.com/privacy for more information.
[00:00:00] TED Audio Collective Immersive AI is powering the next frontier of gaming. This tech is taking cues from human interaction and interweaving storylines throughout the game based on our own real-time engagement. But let's consider the greater possibilities for this.
[00:00:24] What if an AI-enabled video game can give back to us, too, and teach players empathy and social skills? Right now, lots of games have a more or less static storyline that requires a cause-and-effect loop to move forward.
[00:00:38] But what if, instead, these positive human qualities became an integral part of the gaming experience? This is TED Tech, a podcast from the TED Audio Collective. I'm your host, Sherelle Dorsey. Kylan Gibbs is the CEO of nWorld, an AI-driven video game company.
[00:01:00] In this talk, he takes us inside the immersive experience of shared digital and real-life space. This is a world where humans can talk directly to the game, give commands, and watch the game reply and evolve. And Kylan modeled it live from the TED stage.
[00:01:16] But before we dive in, a quick break to hear from our sponsors. Here at Shortwave Space Camp, we escape our everyday lives to explore the mysteries and quirks of the universe. We find weird, fun, interesting stories that explain how the cosmos is partying all around us.
[00:01:38] From stars to dwarf planets to black holes and beyond, we've got you. Listen now to the Shortwave podcast from NPR. And now, Kylan Gibbs takes the TED stage. When I was 10, I spent a lot of time playing with my Spider-Man action figures.
[00:02:00] I would have done anything for them to come to life. I honestly would take Gandalf over any therapist, but it's been pretty hard to connect with the guy since I long ago finished all of Tolkien's works and movies.
[00:02:11] Why is it that we have to say goodbye on the final page? We may not need to. Stories are changing, moving from static sketches and scripted narratives to living worlds and characters. AI agents are changing the way we can tell stories. These agents aren't just chatbots.
[00:02:30] They're any autonomous system that can take a directive or task and accomplish it without human intervention. Virtual assistants like a chat GPT, autonomous vehicles like a Waymore Tesla, robots moving boxes in an Amazon warehouse. These are all agents. But now they're changing storytelling too.
[00:02:49] We can bring together the active participation of video games with a much more natural social form of interaction enabled by AI agents. Characters that come to life with their own personalities, styles, voices and expression while still being able to interact with one another, the audience and the environment.
[00:03:06] Meet Caleb. Oh, hey, Grandma. Welcome to my secret attic hideout. What's Mr. Moose doing there? That's actually Brother Moose, not Mr. Moose. He's like, um, the wise old moose who gives me advice and stuff. Oh, by the way, thanks for showing me your old superhero action figure, Grandma.
[00:03:34] It's like totally awesome. What's its name? Mighty Squirrel. Really? Yes. Wow, that's so awesome. A squirrel with superpowers? That's nuts. Get it? Nuts? Like what squirrels eat? I bet Mighty Squirrel had some epic battles with evil acorns. By having that real-time responsiveness,
[00:04:06] Caleb makes the audience feel seen and heard, as if they're all inhabiting the shared space. And it isn't just any audience, that's actually my amazing grandma. The characters can also come together with one another and take ideas from the audience and improvise new content on the fly.
[00:04:23] Each of these characters has their own internal reasoning and motivations, but they actually take those ideas from my grandma and then create new content in real time. They can even allow the audience to shape the overall story arc and conclusion of the experience.
[00:04:38] Prepare yourselves, for the Mighty Squirrel is about to take flight. Whoa! Ha ha! Grandma, do you have anything to help me take flight again? There's an eagle outside on a tree. An eagle, you say? Perfect. Time to spread my wings and soar through the skies. Whoa, Mighty Squirrel!
[00:05:08] You're going to be the coolest flying superhero ever. And we'll have epic battles with the evil villains. Okay, so I'm mainly just jealous that Caleb is getting out of the experience I always wanted as a child, but I also love seeing how he takes the story,
[00:05:28] puts it in the audience's hands and turns it into a canvas for their imagination. This is of course a tiny slice of what's possible. With there already being other AI-enabled games and experiences that exist today. Most exciting for me is a brand new genre we're seeing emerge
[00:05:43] that takes social interaction and conversation and puts it as the core mechanic of the experience. So let's say instead of jump and shoot, you actually have to use your social skills to navigate complex social scenarios, immediate conflicts. That's how you win the game.
[00:05:58] I would definitely have benefited more from that type of game than the ones I was playing when I was a teenager. And it's not just pure games per se. Imagine I had an AI tutor who texted me to remind me of a session
[00:06:08] and then jumped with me into a virtual world, walking through the emperor's chambers in ancient China, casually switching between English and Chinese as we reviewed last week's history lesson. Or as my family and I sat down to watch our favorite reality TV show or sports game,
[00:06:23] the agent could actually see the ongoing performance in real time using vision and add hilarious commentary in the background. Games though are unique because they already take us from passive observer and turn us into an active participant. When I get really deep into a game,
[00:06:40] especially immersive role-playing games like Final Fantasy 7, Baldur's Gate 3 or Elden Ring, yes, I'm a massive role-playing game geek, at some point it feels like I'm in the story. But there's still something missing. The core mechanics of these experiences, jump and shoot, hack and slash, point and click,
[00:06:56] feel so distant from the natural ways that I interact with people and things in my own life. And so as immersed as I get in these experiences, it kind of feels like I'm stuck on these rails set up by the developer
[00:07:06] and nudged along from scripted point to scripted point based on the buttons that I press. And immersion requires a sense of real-time responsiveness, a sense that there's a consequence that is immediate and lasting and unique to the choices and actions that I take. However, until now,
[00:07:22] it's been extremely difficult to prescript every combination of consequences to every permutation of player behaviors. Agents overcome this though. It's not just these standalone characters we want to bring to life, but fully agentic interconnected worlds where a choice at one point
[00:07:36] might have dramatic consequences somewhere down the line. And the unique content produced for each user is still grounded in a shared lore and experience so that we have that shared social context that we all need for media. My company, InWorld, helps entertainment houses and game studios
[00:07:52] to build these agents for the next generation of media. And when we begin to build a character like Caleb, we start with the brain, which is crafted in a similar way to instructing an extremely capable improv actor. We start by training custom machine learning models
[00:08:06] that give him a lay of the land, understanding the lore, vocabulary, and grammar appropriate for the experience. And then we actually go in and we tweak his persona, his unique personality, motivations, his flaws and biases. We actually craft his dialogue style and voice,
[00:08:23] the way he verbally expresses himself. We build a unique emotional profile, the way he feels, his social tendencies and relationship habits. And then we preload a bunch of knowledge and memories that give him that robust backstory. Then we actually craft his internal reasoning and mental state,
[00:08:41] kind of the conditions under which he feels certain things or takes certain behaviors. We then decide how he actually gestures and animates those behaviors. And then that completes the brain, which we actually take and we attach to an avatar within a virtual world or game engine,
[00:08:56] effectively bringing it to life. And then we're ready to play. And when we're playing the experience, there's kind of an overarching narrative agent that acts like an AI director, managing or even generating the storyline in real time and ensuring that all the characters and entities
[00:09:12] cohere to their proper places within the story and context. For a specific character like Calem, when he's interacting with the world and characters, he does so in a similar way to how we as humans do, with perception, cognition and behavior. Perception is the multimodal sensory input flowing in,
[00:09:29] the understanding of the world and context and people and objects within it. And then with cognition, he processes that input, kind of like the internal processing of our minds. He filters out irrelevant information to make sure everything is safe and narrative aligned. He extracts out important signals
[00:09:44] like sentiment, intent, emotional cues. He retrieves relevant memories from his past and even generates nuance on the fly, finally reasoning over what to do, both verbally and non-verbally. And finally, with behavior, he actually generates his responses. He decides what to say and how to say it,
[00:10:01] what facial and body gestures to use, what actions to take, and people and objects to interact with. And that's what brings the character to life. And so as amazed as I am by all these task-focused applications of AI that are coming out,
[00:10:15] the more I work with studios and creatives, the more excited I am for the potential for these agents to extend human creative potential and for them to allow audiences to step beyond the script, for them to give audiences agency and allow them to co-create experiences
[00:10:30] that stem from their own imagination. It seems like there's a pretty solid chance my Spider-Man action figures could come to life after all. Thank you. That was Kylan Gibbs at TED 2024. NFTs, GPUs, grokking, capacitance, the tech world is full of a lot of lingo.
[00:10:54] Keep up with the latest acronyms and technology news with TED's new newsletter. TED Talks Tech will bring you tech headlines, talks, podcasts, and more on a biweekly basis so you can easily keep up with all things tech and AI. Subscribe now at the link in our show notes.
[00:11:16] And that's it for today. TED Tech is part of the TED Audio Collective. This episode was produced by Nina Byrd-Lawrence, edited by Alejandra Salazar, and fact-checked by Julia Dickerson. Special thanks to Maria Ladius, Farrah DeGrunge, Daniella Belarezo, and Roxanne Hailesch. I'm Sherelle Dorsey. Thanks for listening in.