The future of AI-generated characters

AI-generated and AI-driven characters may still be in their infancy, with years of development yet ahead before they reach maturity, but already they are making a mark on our lives, and their use is rapidly rising.

They help us get around town, run our smart homes, file our appointments, chat with us on website support systems and social networks, and more. And within the field itself, developers are excited at the potential for these characters to come to life in augmented, mixed, and virtual reality — to advance a new paradigm for human-computer interaction.

But where does the AI-generated character stand right now? What can (and can't) we do with them, and what should we be concerned about with regards to them? And where are they headed next? We reached out to three industry insiders to help us explore the current state of this nascent industry.

A quest for sentience

"When we talk about making an AI character, we're basically saying we want to make characters that can respond in a seemingly sentient way, similar to a living thing or even like a human," said Jacob Loewenstein, head of business at AR startup Spatial.

That apparent sentience seems tantalizingly close to becoming reality. Some AI characters can already recognize facial expressions and a wide variety of objects, speak and listen and answer questions, and process a variety of other inputs in building increasingly-lifelike outputs.

If given a digital body, they can walk around in a headset or on a screen. One thing they can't do well, though, is understand us and the world at large beyond a surface level.

"We're really not that great at that yet," says Loewenstein. These characters can recognize shapes and outlines, but the ability to understand intent and analyze sentiment is much more difficult.

For now, at least. Given enough time and data, Loewenstein adds, we'll get there. Some companies already have solved parts of the problem. Affectiva detects emotional and cognitive states from facial cues and voice analysis to improve driver safety. Spirit AI has used its own character engine authoring tool and SDK to build an AI that can monitor abuse, detect harassment, and take steps to make sure everyone feels safe in an online community.

Greg Cross, co-founder of New Zealand startup Soul Machines, says his company's autonomous "digital humans" have a cognition engine that emulates our nervous system responses to other people at low latency. If you smile at one of their AIs, it'll smile back. If you yawn, it yawns as well. These AIs actively try to understand the context behind each gesture and adjust their own behavior accordingly, much as Magic Leap said its Mica virtual assistant can do at its recent L.E.A.P. conference.

Soul Machines is using these digital humans to reinvent the customer experience for clients like Mercedes Benz, Autodesk, and IBM Watson. It's going well so far, Cross says. They're getting high levels of engagement, and digital humans are quickly becoming the preferred method for engaging with customer service at those companies. That anecdotal evidence is aligned with a recent Salesforce report that claims 69 percent of people prefer chatbots for quick communication with brands.

There's still a long road ahead, however. The nuances of facial expressions, speech, and body language are sometimes lost on today's AIs. And the conversations you can have with chatbots and AI assistants — even those with some face, vision, and emotion recognition — still need to be scripted by humans.

The Google Duplex demonstration notwithstanding, AI agents struggle to create their own dynamic conversational content. Even when you're looking at transactional situations like customer service and restaurant bookings, it may only take a minor segue to trigger an "I'm sorry, I don't understand" response.

If you've ever tried to have a conversation with an AI assistant like Cortana or Alexa, you'll have experienced this first hand. They can look up information for you online, turn on the lights, and book your appointments if you directly ask them to, but they're lost the moment you try to start a discourse about the latest Game of Thrones episode.

When it comes to AI-generated characters, the uncanny valley is no longer a question of appearance. It's now more about behavior and (low-latency) reactivity.

The most jarring thing that can happen in VR and AR, especially, has nothing to do with the visual fidelity of characters and environments. Rather, it's when the characters often have limited (or no) responsiveness to your presence and to other stimuli — like a sudden noise or someone entering/leaving/walking around the room.

Most AI characters have no apparent sentience beyond a limited set of pre-defined routines and basic object and facial recognition. They are robots in the form of digital humans, though there's an energy in the field that suggests they won't remain this way for much longer.

Reshaping interactive entertainment, slowly

Spirit AI's character AI specialist Aaron Reed says that in the realm of video games, where efforts to do interesting things with AI characters have historically taken a backseat to graphics and animation processing requirements, "people are kind of creeping into the circle from different edges, on different fronts."

In Shadow of Mordor, for instance, procedurally generated orcs get promoted if they kill you in battle and then will taunt you the next time you face them. If, on the other hand, you wound but don't kill them, they'll remember what you did and develop fears and hates to match.

The Forza series of racing games uses learning neural networks to generate AI drivers that imitate your driving style after watching you play, so that people around the world can feel like they're racing against other humans — even when they're not.

On the indie side, games like Façade and Prom Week have explored more sophisticated AI characters that respond dynamically and naturally to player input — provided that the player is willing to roleplay to the situation — while Event[0] has players attempt to converse with a possibly self-aware computer intelligence through text input.

Then there are games like Crusader Kings and Dwarf Fortress, which use algorithmic systems and procedurally generated family trees to encode value decisions in simple AI characters that then leads to emergent behaviors — a character might choose to avoid war with a country because they're married to the grandson of the king, for example.

These are all interesting uses of AI to generate compelling characters and narratives, but Reed says they illustrate that "we really haven't seen the breakout game yet that stars an AI-driven character in a narrative sense."

Likewise for entertainment and media more broadly. Digital characters Lil Miquela and Hatsune Miku each have huge fanbases, which shows a willingness from the public — especially younger demographics — to accept the idea of interacting with artificial beings, but it's a stretch to call either of them AI agents. They're controlled by marketers, not algorithms.

There's a sense that a breakthrough could be just around the corner, however. Reed believes we'll see an indie game that stars an AI-generated character make a big splash in the next year or two.

Triple-A game studios, he adds, have steered clear of complex procedural or recombinable text thus far because it's difficult to test these systems in a "vertical slice" demo when (in Reed's experience, at least) their benefits tend to not become apparent until they are nearly finished. Also, big-budget games are expected to be fully voice acted — and text-to-speech technology is only now reaching the point where it's good enough to sound natural.

Big studios appear to be more actively interested in driving that innovation. They have small R&D teams experimenting with Spirit AI's authoring tools with a mind to embracing natural language processing and autonomous AI-generated characters in the next five to 10 years.

This shift could bring dramatic changes in the actions available to gamers. While today they can shoot, jump, and run, future games could focus more on things like character interactions and empathy.

Reed talks about one room-scale VR demo built by Spirit AI where AI characters would respond differently to you depending on your body language. They were designed that way to better gauge the intentions and context behind someone's words and actions.

If you turned your back on them while they spoke or moved into their personal space, they'd react accordingly — perhaps asking you to look at them in the first instance or to back off in the second.

This sort of reactivity shifts digital storytelling into more of a performative space, as opposed to traditional linear or menu-based branching narratives. That, meanwhile, opens up a new form of storytelling that casts you as a sort of actor or improviser who's mentally engaged in the emotions and sociality of the situation.

For his part, Reed can't wait to see what people come up with as the authoring tools get more widely adopted and people start to master the craft of AI-driven storytelling.

The AI field has a diversity problem

There are other problems that need fixing along the way. AI has a serious diversity problem, for one thing.

While Loewenstein says there seems to be a general recognition in the field that inclusivity is important, there's a gulf between where developers are on the issue and where they want to be. And this manifests in a multitude of ways.

Animation libraries for AI characters are built on data from motion capture of developers — who are mostly white men — and a small number of professional actors. That means for now there's an extremely limited range of walking, dancing, and gesturing exhibited by AI characters in virtual, augmented, and mixed reality.

Other behaviors, speech patterns, and decisions are similarly filled with subtle biases toward white men — because even if you make a point of training the AI on a diverse data set, its underlying code may still reflect hidden biases.

It helps that the process of creating AI characters is opening up to a wider audience of non-experts. But all the same, lack of diversity on the developer side limits the breadth of experiences possible with AI characters.

Reuters recently reported Amazon built — but never deployed — an AI to help identify the best job candidates. It discovered that the AI was biased against women because it had been trained on the CVs of successful applicants at the company, which historically were mostly men. Microsoft's experimental learning Twitter bot Tay turned racist after less than a day of interacting with people on the social network.

Other nefarious uses have begun to emerge: Deep fakes and voice mimicry use the voice or likeness of a public figure to create fake content, whether for a prank or to deceive and mislead people.

Companies like Truepic are working on ways to detect this sort of fakery, but that could easily lead to an arms race. Regardless, Loewenstein thinks this technology is getting so good that it'll be a problem we need to deal with far sooner than fully sentient autonomous characters.

Ethical dilemmas

Similarly, the ethics of building AI characters and letting them loose in the world is becoming a bigger issue by the day. And many industry insiders think there needs to a discussion about the rights and wrongs of AI characters now, while the technology is still in its formative stages — because, again, they may be right on the cusp of a breakthrough.

"Increasingly we're getting to a point where you can have an interaction with an entity that you think is human and is really not," says Reed. Lawmakers have started to take notice. A new California bill requires that commercial and political bots identify themselves as non-human.

The rise of AI characters also brings privacy concerns. "Putting an AI character on the face of surveillance can sort of mask the idea that something is watching you and collecting information about you. And that when you interact with it you're actually telling it things about yourself." You're giving it data — contextual and emotional data about your preferences and wants and needs and likes, as well as your general personality.

AI creators need to think about the ethics of this data collection. Some already are. "We have a very clear position [at Soul Machines] that any data that gets collected has to be owned and controlled by the end user and not by the enterprise customer," says Cross.

People working in AI more generally, recognizing that they need to start self-policing if they want to avoid more stringent regulation from outside forces like state and federal governments, are starting projects like The Ethics and Governance of Artificial Intelligence Initiative to research, discuss, and hopefully reach agreements about these sorts of ethical dilemmas. But it's early days here, too, with no universally-agreed-upon manifestos or self-regulation systems.

Looking ahead

Loewenstein notes that people are becoming more comfortable with letting AI characters into their lives. This is especially true for children, but even adults are learning to live with the AI assistants in their phones and computers, as well as in devices like Amazon Alexa and Google Home. "If there's utility and it makes your life easier, people will get comfortable with anything," Loewenstein adds.

While it's unclear at this point exactly to what extent AI-generated characters might replace humans in customer service, Reed says we can be confident that they won't be making writers obsolete anytime soon. "What we're doing is more like giving the existing writers a sort of robot exoskeleton that lets them be way more effective with the content they're creating," he explains.

And that effectiveness, too, is going to take time to develop. Reed points to CD-ROM technology as an example: It took five years from the advent of the technology for Myst to come along and really show the first compelling use for all the extra storage space that CD-ROMs offered to software developers.

AI-generated characters provide a new paradigm for storytelling — a new way to explore and discuss the human condition — and as such there'll be a long period of experimentation across all software fields (including but not limited to games, education, and entertainment). But Reed can't wait to see what "the Nabokov's of AI characters" are eventually able to say about what it means to be human.

In the meantime, we can expect to see AI-generated characters popping up more and more across our lives — with significantly-expanded capacity for dynamic conversations emerging over the next year or two.

Our world is increasingly digital, and our devices are increasingly interconnected, and Cross believes this will in turn drive demand for AI-generated characters that entertain, engage, and help us at home, work, and everywhere else — perhaps even putting us at ease in our self-driving cars. "For the very, very simple reason," he says, "that we like to trust." And if the AI characters look and behave and sound like humans, we'll trust them.

But it won't just be a few AI characters. It'll be hundreds, each built for a different brand, then thousands, and maybe eventually they'll even outnumber us.