OkCupid’s President on the Site’s Social Experiments: ‘Most People Don’t Care’

(Associated Press)

Earlier this week, OkCupid President Christian Rudder published a blog post revealing that the website intentionally misled its customers for a recent experiment. In the most controversial part of the study, researchers adjusted the match percentage between two people who wouldn’t normally be compatible. Overnight, a person who’d usually appear as a 30 percent match would appear as a 90 percent match, and vice versa.

Before introducing the results, Rudder referenced Facebook’s controversial “emotional contagion” study that was released in June.

“We noticed recently that people didn’t like it when Facebook ‘experimented’ with their news feed,” the co-founder wrote. “But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.”

Rudder’s unapologetic attitude toward this type of research could very well represent a shift in how major websites view their user data. I spoke to him about his views on online social experiments and whether he’d consider more explicit warnings before future tests. Highlights from our chat:

Q: You’ve defended your experiment by saying, “This is how we make a better product.” But isn’t there a difference between A/B testing a website’s design and experimenting with the truth of what you’re seeing on a dating site?

A: I understand why it seems like there’s a difference, but think of it like this: We test our match algorithm often. We might remove a variable, or weigh something differently, or get rid of some option that users had before. We test that against the current status quo in exactly the same way we did with this experiment. So people see different match percentages. This is on that same continuum.

Here obviously we’re suggesting a random number as a match percentage, but it’s still part of that same thing. If you’re developing for any algorithm, whether it’s Google search, or Facebook’s feed, or our match algorithm, your options at that point are, you come up with a draft, and say, OK, this is the best we can do, or you can prove that it is. And the only way to prove that it is, is by running these exact tests. That’s just the scientific method.

Q:How many people in total were involved in the experiment?

A: In the most controversial part of the experiment, where people who had a low match score that they were told [was] high, something around 500 messages were exchanged under that auspice. Very small. Just for scale, we have over 5 million messages exchanged every day.

This was kind of lost in the uproar, but we told these people after the fact, so they weren’t continuing to live under this false piece of information. We said: “Hey, you were involved in a diagnostic test. Your match percentage was misstated. Here’s the correct number. Thank you.”

Q: Why would you use a dating site that you had reason to think was feeding you false information about potential matches?

A: Obviously I want people to think that OkCupid works. That’s what we’re trying to do with this experiment, is make sure that it works. Again, everybody was notified, so it’s not like OkCupid is doing horrible things and making everything false. We are experimenting all the time. We’re changing the text, we’re changing the pictures, we’re changing the colors.

It would be bad if people thought we didn’t do a good job. I think most people understand that part of this was to do a good job.

Q: There’s a real chance that this could blow up in someone’s face. I know that you measure success by initial messages, but are you ever worried that these could result in one horrible date, which might leave someone worn out or discouraged with the service?

A: First, for the most sensitive part of this experiment, it ran literally last week, and we notified people on Friday. So it’s not like a ton of time passed between this experiment and what you want to call the “correction.” Second, I don’t think anybody actually goes on a date with somebody that they don’t like to talk to already.

It is possible — but I think still a long shot — that a few users went on dates with people that answered certain questions differently than they had thought. But it’s not a dangerous situation. Maybe a person likes scary movies and someone thought that they didn’t. We didn’t misrepresent anybody’s answers. We just changed the match score. It’s like rendering a placebo effect for a drug trial. These people had a sugar pill.

Q: I think people expect websites to collect their data, but it never occurs to them that companies would go so far as to intentionally mislead them. Would you ever consider offering a more explicit set of terms about these experiments during the sign-up process?

A: We say in the Terms and Conditions — and I’m not trying to hide behind that document — but we say that we do research, we do experiments — or maybe not experiments; we do research and analysis. And we say that we’re going to publish the results. We’re pretty transparent in terms of the legalese.

Some people like to talk about informed consent in old-school research. They’re not talking about what was really going on in person. When I was in college, I needed 20 bucks, and so I volunteered for an experiment in the psych department. I signed a piece of paper and I sat down in front of a computer. Some words blinked on the screen, or a dot appeared, and I had a button in my hand. I still don’t know what they were testing me for. Maybe they were testing to see if I obeyed their instructions. And to the extent that they would’ve informed me, it would’ve ruined their experiment. I don’t see how that “informed consent” is any different than people knowing terms and conditions exist and clicking through it passively. It’s exactly the same.

Q: Except it’s not taking place in a psych department. Which is why people are comparing your study to Facebook’s. Do you feel that they were similar in any way?

A: They’re similar in some ways. I definitely don’t want to speak for Facebook. I don’t know exactly what goes on there, and I haven’t read that paper.

I think a lot of what Facebook does is kind of annoying. I don’t like seeing ads in my wall feed that pretend that my friends “liked” Dunkin’ Donuts. In this case, however, Facebook was actually trying to benefit the public good with the research they were doing. The level of outrage that this completely modest experiment provoked is just going to prevent them from doing that. They’re just going to go back to selling us Dunkin’ Donuts. That’s just kind of sad.

Q: I’ll admit that I, for one, lack outrage when I hear about these social experiments, maybe because I’m just accustomed to expect companies to take advantage of my data.

A: I think most people don’t care. I think a lot of journalists and pundits are paid to be upset about this. Some people, like privacy advocates, don’t have a job otherwise. It’s just part of a role they play, the way that Rush Limbaugh doesn’t have a job if he’s not mad at Obama. He can’t say that he likes him even if he does. It’s not just Facebook experimenting with us. Every single site does it.

Follow Alyssa Bereznak on Twitter or email her here.