OkCupid’s President on the Site’s Social Experiments: ‘Most People Don’t Care’
Earlier this week, OkCupid President Christian Rudder published a blog post revealing that the website intentionally misled its customers for a recent experiment. In the most controversial part of the study, researchers adjusted the match percentage between two people who wouldn’t normally be compatible. Overnight, a person who’d usually appear as a 30 percent match would appear as a 90 percent match, and vice versa.
Before introducing the results, Rudder referenced Facebook’s controversial “emotional contagion” study that was released in June.
“We noticed recently that people didn’t like it when Facebook ‘experimented’ with their news feed,” the co-founder wrote. “But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.”
Rudder’s unapologetic attitude toward this type of research could very well represent a shift in how major websites view their user data. I spoke to him about his views on online social experiments and whether he’d consider more explicit warnings before future tests. Highlights from our chat:
Q: You’ve defended your experiment by saying, “This is how we make a better product.” But isn’t there a difference between A/B testing a website’s design and experimenting with the truth of what you’re seeing on a dating site?
A: I understand why it seems like there’s a difference, but think of it like this: We test our match algorithm often. We might remove a variable, or weigh something differently, or get rid of some option that users had before. We test that against the current status quo in exactly the same way we did with this experiment. So people see different match percentages. This is on that same continuum.
Here obviously we’re suggesting a random number as a match percentage, but it’s still part of that same thing. If you’re developing for any algorithm, whether it’s Google search, or Facebook’s feed, or our match algorithm, your options at that point are, you come up with a draft, and say, OK, this is the best we can do, or you can prove that it is. And the only way to prove that it is, is by running these exact tests. That’s just the scientific method.
Q:How many people in total were involved in the experiment?
A: In the most controversial part of the experiment, where people who had a low match score that they were told [was] high, something around 500 messages were exchanged under that auspice. Very small. Just for scale, we have over 5 million messages exchanged every day.
This was kind of lost in the uproar, but we told these people after the fact, so they weren’t continuing to live under this false piece of information. We said: “Hey, you were involved in a diagnostic test. Your match percentage was misstated. Here’s the correct number. Thank you.”
Q: Why would you use a dating site that you had reason to think was feeding you false information about potential matches?
A: Obviously I want people to think that OkCupid works. That’s what we’re trying to do with this experiment, is make sure that it works. Again, everybody was notified, so it’s not like OkCupid is doing horrible things and making everything false. We are experimenting all the time. We’re changing the text, we’re changing the pictures, we’re changing the colors.