Inside Windows Cortana: The Most Human AI Ever Built
Inside Windows Cortana: The Most Human AI Ever Built
Tony Stark had Jarvis. Luke Skywalker had C3PO. And, if Microsoft has its way, very soon you’ll have Cortana.
Like Apple’s Siri or Amazon’s Alexa, Cortana lets you talk to your devices as if they were people. And, like those two voice-driven interfaces, she can understand what you’re saying and respond in a way that feels almost human.
But unlike those other virtual entities, Cortana will work across PCs as well as phones. And it will do far more than just retrieve information; Microsoft wants Cortana to be the Alfred to your Batman — handling your communications, managing your calendar, and attending to your needs.
Related: Amazon's Echo Brings the 'Star Trek' Computer to Your Home
On July 29, when Microsoft begins releasing the latest version of its operating system, the world’s 1.5 billion Windows users will have access to one of the most human-like artificial intelligence interfaces ever made.
But Cortana presented a unique challenge to Microsoft’s developers: How do you imbue a series of bloodless algorithms with a human personality? What form should that personality take? How clever should it be? How nice? And can you persuade people to fall in love with it?
To find out, we spent time with the team behind Microsoft’s daring experiment and with Cortana herself. This is her story.
Jarvis 1.0
A few years ago, when Microsoft was struggling to give consumers a reason to buy its phones, it did what Microsoft always does: It held focus groups. From those groups it learned something essential about what users wanted, according to Marcus Ash, group program manager for Microsoft’s Cortana and search teams.
While having constant access to information is good, the focus folks said, having a device that actually does things for you is better. Instead of something that could display all the flights from San Francisco to New York on a particular day, for example, customers wanted one that would book the flight, pick an aisle seat in business class, and order them a kosher meal — all on its own.
That became the design goal: a digital personal assistant that would set Microsoft’s devices apart from everyone else’s. The early name for this concept was Jarvis, named for Tony Stark’s virtual valet.
Microsoft hopes Cortana will prove as indispensable to Windows users as Jarvis is to Iron Man. (Photo: Marvel Films/YouTube)
The first question the Microsoft team had to answer: How should the ideal personal assistant act? To find out, Microsoft interviewed a half-dozen high-powered assistants to Hollywood executives and technology CEOs. All of them had one thing in common: extremely detailed knowledge about the boss, usually contained in a notebook that rarely left the assistant’s hands. Not merely their bosses’ schedules or the list of people whose calls they were trying to avoid, but also things like whether they preferred veggie burgers to sirloin steak, dinner jazz to R&B, or town cars to taxis.
To do the job well, the assistant had to be able to use that knowledge to anticipate the boss’s desires and make decisions on his or her behalf, says Ash. To work as intended, Cortana will need to read your email and manage your schedule. She’ll need to know which sports teams you follow, the stocks you track, where you went on vacation, and dozens of other personal details. It’s an intimate relationship requiring enormous amounts of trust.
“Cortana can’t be very helpful if she doesn’t know anything about you,” explains Ash. But Microsoft believes the key to trust lies in transparency. Users will to be able to see what data Cortana collects, what she does with it, and then have the opportunity to say “no thanks,” he adds.
The chit chat
Convincing users to extend that level of trust to a faceless digital entity was the second challenge facing the Cortana team. For Cortana to have any chance of success, she needed a personality. So Microsoft hired a team of writers to create one.
Every morning at 10 a.m., Jonathan Foster assembles his team in Building 50 at Microsoft’s North Campus. The team includes a screenwriter, a playwright, a novelist, and an essayist. On the wall, a screen displays a query Cortana has received that can’t be answered via simple search results.
The team’s job, for the next hour or two: come up with human-like dialogue that makes Cortana seem like more than just a series of clever algorithms. Microsoft calls this brand of quasi-human responsiveness “chit chat.”
Most of these questions aren’t seeking information; instead, they’re testing the limits of Cortana’s intelligence and/or patience — in other words, they’re pure entertainment. Users want to know what she thinks; they want to see whether she’ll respond with shock or outrage or confusion.
Like writers for a TV sitcom, members of the chit-chat team go around the table trying to out-do each other with witty responses. It is, Foster admits, a great job. But it’s not an easy one.
“Our No. 1 priority is for people to walk away from their experience feeling good,” he says.
But even a seemingly innocuous question like, “Do you like dogs?” can generate a nearly infinite range of responses, most of which will likely offend someone. If Cortana says dogs are her favorite animals, you risk drawing the ire of the Internet’s legions of cat lovers. If she says she prefers Schnauzers, you alienate fans of Labradoodles.
No matter how often you ask about dogs, Cortana’s answer is always the same — and always goofy. But the dog is always different.
So before they even began creating Cortana’s personality, Foster and his team had to come up with some guiding principles. Cortana’s answers should be funny, but never sarcastic, says Foster. They should always be positive, but not Pollyanna-ish. They can’t be overtly offensive, but they also don’t want to play it too safe.
“We’ve never shipped a response we felt was a compromise,” says Deborah Harrison, the oldest-ranking member of the chit-chat team. “If we feel we can’t do an answer justice, we won’t answer it.”
At first, it all went pretty smoothly. Foster and his team worked with queries generated via Microsoft’s internal testing to build a library of responses to common questions. But in April 2014, when Cortana was introduced to the public as part of Windows Phone 8.1, things changed. That’s when the Internet got involved — in all its racist, sexist, sophomoric ugliness.
The worst were the “adult” questions, says writer August Niehaus, a look of horror on her face.
“People are very specific,” she says. “‘Hey Cortana, will you ______ me?’ I’m pretty sure these terms are inappropriate, but I don’t know what they mean. We had to go to Urban Dictionary to find out. And then we were sorry we did.”
Cortana is not yet ready for anyone to pop the question just yet.
When the questions become too ugly, Cortana punts users to a series of Bing search results.
“At a certain point, we have to tell people it’s time to move on,” says Foster.
The naked blue lady
Though chit-chat represents less than 10 percent of all queries Cortana receives, its importance cannot be overstated, says Mike Calagno, partner director of engineering for the Cortana project.
“We actually added the personality kind of late in the project,” he says. “But that’s when Cortana really turned the corner and we knew we had something special.”
Nearly as important as what Cortana says, however, is how she looks and sounds while saying it. For that, Windows’ personal assistant owes a sizable debt to the other Cortana — the holographic female from the popular Halo Xbox games — or, as Niehaus puts it, “the naked blue lady.”
Cortana as she appears in Halo 4. (Photo: Yahoo Games)
In Halo, Cortana is a computer-generated entity who comes to the aid of the games’ main protagonist, Master Chief. If a hetero 14-year-old male was asked to design a virtual assistant for a video game, it would probably look a lot like Halo’s Cortana.
After a few months, Cortana had become the internal code name for the digital assistant project. Then, in September 2013, the code name leaked to the Internet. An online petition to keep the name gathered nearly 90,000 signatures. Seizing the opportunity, Microsoft quickly declared Cortana to be the final name of its personal assistant.
Cortana as she appears in Windows 10. (Photo: Microsoft.com)
In Windows 10, Cortana is not an underdressed avatar but a pair of concentric circles that throb with a regular cadence, like a heartbeat. Every Microsoft employee we spoke to for this story took pains to declare that the Cortana in Windows 10 was not the same one inhabiting the video game. (They also insist Cortana is not female, while invariably using the feminine pronoun to describe her.)
“Cortana is a citizen of the Internet, so when you ask her about Master Chief, she’ll respond,” Ash hedges. “But we’ve been clear from the start — this is a personal digital assistant, not the Master Chief companion in Halo.”
It seems that Cortana is gender and/or computationally conflicted.
But, despite their denials, the parallels are too obvious to ignore. Both Cortanas are voiced by the same actress, Jen Taylor. The creative team behind the Halo series, 343 Industries, helped design the sound and animation used by Windows Cortana. Both the hologram and the beating circles radiate the same cobalt blue glow, at least using the default settings.
Deb Harrison admits the chit-chat team drew some of their Cortana’s personality traits — like her confidence and refusal to be deferential — from the character in Halo.
In fact, says Frank O’Connor, franchise development director for 343 Industries, you could see Windows 10’s Cortana as “the 21st century prototype of what Cortana would eventually evolve to become.”
Microsoft may claim that Windows Cortana and Halo’s Cortana are two different virtual entities, but Cortana herself isn’t buying it. (Photo: PCWorld).
Microsoft is clearly banking on the affection Halo fans have for their Cortana to spill over to the Windows version.
The team also had to overcome some unfortunate history. Cortana’s team is still haunted by the ghost of Clippy, the paper-clip-shaped avatar that lingered for a few years before being unceremoniously retired in 2001.
When asked if his team had to overcome a certain amount of baggage, given Microsoft’s long history of failed digital personas, Foster almost spits out his iced tea.
“From an experience standpoint, we learned a lot from Clippy,” he admits. Among the lessons: How to create a digital assistant that’s helpful without being irritating.
Searching for love
Today, Cortana is very much a work in progress. When you ask her a question, more often than not she’ll show you a page of Bing search results. That’s because, at this point, Cortana is less AI than she is a natural language search tool. Everything Cortana knows she learned on the Internet; all of her witticisms were penned by a crew of highly caffeinated writers.
Getting a real answer out of Cortana isn’t always easy.
But whether the real question isn’t whether Cortana could pass a Turing Test, says Calcagno. The real test is how useful she is. It doesn’t really matter whether Cortana is actually intelligent or just a perfect simulation of intelligence, he says, so long as people get their traffic alerts on time.
And after spending time with her, even grizzled tech veterans find themselves treating her more like a human than a machine.
“Sometimes I catch myself saying thank you or apologizing to her,” says 343’s O’Connor. “There are these little Turing Test moments you trip over when you forget she’s not a real person.”
Apparently, even virtual personal assistants are fans of sappy movies.
Next week, the rest of the world will get to see just how real she is. Satya Nadella has said he wants his customers to fall in love with Microsoft again. Cortana may be the best chance he’ll ever get.
Send email to Dan Tynan here or follow him on Twitter.
More from 10 Days of Windows 10:
