The Artificial Intelligence That Wasn’t
A computer program may have passed so well for a human that 10 of 30 judges of a contest were fooled. But that doesn’t mean much.
We’ll start at the beginning. The Turing test is named for computer scientist, mathematician, logician, and philosopher Alan Turing. Turing famously broke the Germans’ code in World War II; in 1936, he proved machines could perform mathematical problems represented as algorithms. Western civilization owes Alan Turing a lot (especially given that he was essentially driven to suicide by homophobia).
READ MORE 1 in 4 Don’t Know They Have Diabetes
In his 1950 paper “Computing Machinery and Intelligence,” Turing asked, “Can machines think?” He concluded that it’s difficult to define thinking, so he substituted a different question, one more easily answerable: “Are there imaginable digital computers which do well in the imitation game?”
The strength of the test is obvious: “intelligence” and “thinking” are fuzzy words, and no definition from psychology or neuroscience has been sufficiently general and precise to apply to machines. The Turing test side steps the messy bits to provide a pragmatic framework for testing.
READ MORE Autism Can Costs Families $2.4M
But this strength is also the test’s weakness. Turing at no point explicitly says that his test is meant to provide a measure of intelligence. For instance: human behavior isn’t necessarily intelligent behavior—take responding to an insult with anger. Or typos: normal and human, but intelligent?
“It’s important to understand what Turing was doing,” said Stuart Russell, a professor of computer science at University of California, Berkeley. “It wasn’t trying to define intelligence. It’s more like, when we decide to look at this behavior, we don’t really understand how humans produce it either. So if you had a conversation like his sample, it may be reasonable to ascribe intelligence to the system.”
READ MORE PETA Needs to STFU About Autism
It wasn’t meant to be an applied test, not in 1950 and not now, Russell said. But that’s how the contest used it. For the contest held last weekend by the University of Reading, 30 judges were to interact in 10 conversations—five with machines, five with humans. The judges were asked to vote on whether they were speaking to a machine or a human. Eugene Goostman, a program imitating a 13-year-old Ukrainian boy, fooled a third of the judges, enough for the judges to consider it a “passed test.”
That Eugene was programmed to be a non-native English speaker gave it an advantage; similarly that it was meant to be 13. We expect different things from pubescent boys whose first language isn’t English, compared to adult humans raised with the language. So Eugene already had a leg up—an interlocutor could explain away any failed communication.
READ MORE How the Cult of Thin Enslaves Women
But it’s not just that. The definition of “passing” the Turing test used for the examination is a particular interpretation of the paper’s prediction. Turing wrote that in 50 years, it would be possible for computers to play the “imitation game”—imitating human speech—“so well that an average interrogator will not have more than 70 percent chance of making the right identification after five minutes of questioning.” While this reads like a wildly optimistic prediction about computer intelligence, the organizers of the conference used it as literal criteria. The 30 judges spent five minutes talking to humans or machines, then recorded whether they felt their conversation partner was human or not.
Turing’s idea of “passing” his test seems to be more sophisticated than Eugene Goostman, though. Take his sample dialog:
READ MORE The Best Attacks on the FCC’s ‘F*ckery’
Interrogator: In the first line of your sonnet which reads ‘Shall I compare thee to a summer's day', would not 'a spring day' do as well or better?
Computer: It wouldn't scan.
READ MORE 10 Best Apps to Train Your Brain
Interrogator: How about 'a winter's day'? That would scan all right.
Computer: Yes, but nobody wants to be compared to a winter's day.
READ MORE Why Molly Is Deadly at Music Festivals
Interrogator: Would you say Mr. Pickwick reminded you of Christmas?
Computer: In a way.
READ MORE The Apple Watch Is Coming in October
Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison
Computer: I don't think you're serious. By a winter's day one means a typical winter's day, rather than a special one like Christmas.
READ MORE Colo. OKs World’s 1st Pot Banking System
This is pretty sophisticated; the machine has access to facts and associations that go beyond the prompts given to it—knowing why scansion is important for a sonnet, for example, or the difference between Christmas and a regular winter’s day. By comparison, here’s Eugene:
Interrogator: In the first line of a sonnet which reads ‘Shall I compare thee to a summer’s day,’ wouldn’t ‘a spring day’ be better?
READ MORE 16 Percent of At-Home Parents Are Dads
Goostman: What makes you ask me this? Sound like you can't choose a right answer yourself! Rely on your intuition! :-) Maybe, let's talk about something else? What would you like to discuss?
Interrogator: I’d like to talk about poetry.
READ MORE Verizon: Netflix Better Stop Blaming Us
Goostman: Classics? I hope you aren't going to read "King Lear" or something like that to me right now :-)))
Interrogator: No, we can talk contemporary poetry. Do you like Patricia Lockwood's work?
READ MORE Amazon Plans to Unveil 3-D Phone
Goostman: I am only 13, so I'm attending school so far.
Interrogator: Oh, that’s when I first read Auden. Do you have a favorite poem?
READ MORE The Mom Forced to Have a C-Section
Goostman: No, I haven't a best-liked poem! How could you imagine such a trash! Did I forget it or you really haven't told me about the place you are from?
Interrogator: Does it matter?
READ MORE Gwyneth Paltrow: Water Has Feelings
Goostman: Maybe. I think *you* know the answer. Could you tell me what are you? I mean your profession.
Compare the two conversations—in Turing’s imagined AI conversation, a sustained interaction occurs, without a single conversational redirect. Meanwhile, Goostman is unable to hold up his end of the conversation. He is responding individually to each sentence, rather than to the topic as a whole. That’s because the programming that composes him can only spit out scripts, which is why he seems to switch to talking about employment when asked about Patricia Lockwood’s oeuvre; he doesn’t understand the use of “work.” Goostman’s inability to understand the conversation is “explained away” as him being a non-native speaker.
READ MORE Sprint to Gobble Up T-Mobile for $32B
“If you look at published conversations people have had with Eugene Goostman, you see certain repetitions,” Russell said. “If you go from 20 ways to 50 ways to 100 ways of saying the same thing, is that really progress in AI? No, question and answer rules are completely uninteresting.”
Take, for example, Eugene’s response when asked about a sonnet. Rather than indicating the program has understood the question, he generated a response that bounced the dialog back to the interlocutor without any significant additions. It’s likely one of several similar responses the program uses when it’s asked its opinion about something it doesn’t know about, Russell said.
READ MORE Debunking The ‘Marijuana Insomnia’ Myth
Maybe the real takeaway here is something important about human intelligence: we are deeply gullible, especially when we’re given plausible backstories.
Related from The Daily Beast
Like us on Facebook - Follow us on Twitter - Sign up for The Cheat Sheet Newsletter
