It’s not just one suspect herpes vaccine trial: Most experimental drugs are tested offshore — raising concerns about data

The clinical trial for a herpes vaccine flouted just about every norm in the book: American patients were flown in to the Caribbean island of St. Kitts for experimental injections. Local authorities didn’t give permission. Nor did the Food and Drug Administration. Nor did a safety panel.

That’s why the trial — run by a startup which has since received funding from billionaire investor Peter Thiel — prompted widespread alarm and censure when it was reported last week by Kaiser Health News.

But in some respects, the herpes vaccine trial isn’t all that unusual. Nearly all drug makers seeking U.S. approval today rely in part on overseas locations and populations to test their drugs, the result of a decades-long push by industry to try to cut costs and speed recruitment of patients. In fact, a STAT analysis found that 90 percent of new drugs approved this year were tested at least in part outside the U.S. and Canada.

The globalization of clinical trials has brought new treatments to historically neglected populations and generated data more representative of the world’s diversity. But the change has not come without side effects: Companies, researchers, and regulators are increasingly grappling with cultural differences — and more seriously, lapses in ethical and scientific standards — that sometimes arise when trials are conducted in countries without a strong tradition of robust clinical research.

And those lapses can have serious consequences for the data that’s used to support medical decision making.

Read more: Failure to report: A STAT investigation of clinical trials reporting

Dr. John Ioannidis, a Stanford professor who studies scientific robustness, looked at nearly 1,300 clinical trials documented in scientific literature for a paper published in 2013. On average, those conducted in developing countries reported more favorable treatment effects than those conducted in developed countries.

Veteran clinicians who have run and evaluated trials abroad say problems can range from hitches translating medical records (which could affect the reporting of side effects) to different standards of care in overseas clinics (which could affect patient outcomes) to mistakes by poorly trained researchers working with little supervision. There have also been cases of outright fraud.

As a result, drugs tested primarily on patients in developing countries could seem more effective, or perhaps even safer, than they really are.

“I think that we may be running into trouble,” Ioannidis said. “The worst thing is that it’s not easy to tell which trial was inflated and which trial was perfectly done. It just casts a cloud of doubt.”

STAT analyzed FDA records on the first 29 new drug approvals this year. (The agency approved two more new drugs in late August, but it has yet to post detailed information about their supporting clinical trials.) The findings:

• 86 percent of approvals were supported in part by data from trials conducted outside the U.S. and Canada.

• A new drug for amyotrophic lateral sclerosis, also known as Lou Gehrig’s disease, was approved in the U.S. exclusively based on trials conducted in Japan.

• In the 11 cases where a detailed breakdown of the data was available, a collective 58 percent of patients were enrolled outside of the U.S.

• The trials were conducted in more than 18 countries outside the U.S., ranging from Russia to South Africa to New Zealand to Italy. (Some countries that hosted trial sites were not named.)

The past decade has seen multiple examples of experimental drugs that generated impressive results in developing countries — but flopped when tested in more mature markets.

The California startup Medivation, for example, thought it had a winner when its Alzheimer’s compound, called Dimebon, produced by some accounts the best results the field had ever seen in a mid-stage study conducted in Russia. But that promise collapsed when, in 2010, Dimebon failed disastrously in a late-stage study conducted in the U.S., Europe, and South America.

Then there was Targacept, a small North Carolina company that generated stunning results for an experimental antidepressant in a mid-stage study conducted in India — but saw that promise flicker when, in 2011, the drug failed in a late-stage study conducted in Europe.

And the problems go beyond industry-sponsored studies.

In a global trial funded by the National Institutes of Health evaluating a diuretic in patients with a type of heart failure, results appeared to have been skewed by malfeasance by investigators at some sites in Russia and Georgia, new evidence suggested earlier this year.

Read more: Robert Califf: ‘The clinical trials enterprise has gone awry’

The FDA is well aware of the concerns: At the start of the Obama administration, the watchdog for the Department of Health and Human Services urged the agency to tighten oversight of overseas trials by inspecting more sites abroad and working more closely with local regulators.

It’s not clear how much has changed since then. The FDA declined STAT’s request for an interview about how the agency is grappling with these issues.

“Often, the FDA has been more reactive than they have been proactive,” said Ken Getz, who researches clinical trials at the Tufts Center for the Study of Drug Development.

Drug makers, meanwhile, say they work hard to carry out the trials responsibly. Pharma companies “take the ethical conduct of clinical trials, including safety of trial participants, incredibly seriously,” said Andrew Powaleny, a spokesman for the Pharmaceutical Research and Manufacturers of America.

Red flags from overseas data spark debate

There are few researchers who’ve spent more time grappling with how to run overseas clinical trials — and how they can go wrong — than Dr. Steven Nissen.

The Cleveland Clinic cardiologist runs cardiovascular and diabetes drug trials that enroll upwards of 10,000 patients and are sponsored by big companies like Pfizer, AstraZeneca, Novo Nordisk, and Novartis. Nearly all of the trials he runs these days include overseas sites, including in Eastern Europe, China, and India. He said he’s encountered some challenges but also had positive experiences.

Nissen also evaluates data from overseas trials when he sits on panels that advise the FDA on whether to approve a new drug. Seeing lots of data coming from places like India and China makes him “a little bit worried,” he said, because it’s not always clear whether their sites meet modern standards of care.

Geographic disparities only comes up as a red flag in meetings with his fellow panelists from time to time, but when they do, “we’ve had very, very vigorous discussions about it in some cases,” Nissen said.

Dr. Sanjay Kaul, a cardiologist at Cedars-Sinai Medical Center in Los Angeles, chaired the FDA advisory committee meeting evaluating a trial where cultural differences appeared to skew results.

The blood thinner, known to scientists as ticagrelor, had been tested for whether it could prevent heart attacks and early death in patients with acute coronary syndrome in both Europe and the North America — but, strangely, the European patients saw their risk reduced on average while the North American patients saw it increased.

What was going on? Kaul and his fellow advisers probed all kinds of possible explanations, he said. The FDA ultimately concluded that the difference came from how a secondary medicine, aspirin, was taken. (It’s given at a higher dose in the U.S. than in Europe.) When the FDA eventually approved the drug, now sold as Brilinta by AstraZeneca, it included a warning to not use the medication with a high dose of aspirin.

Read more: To boost diversity in clinical trials, researchers seek out allies — starting with black pastors

The case offers an example of how Kaul said he thinks about examining apparent geographical differences in trial results: Are they real? And if so, are they due to random variation? Or to a biological or clinical factor?

If he’s ruled out those possibilities, he looks at a clinical trial site’s record. If he sees geographic differences in “data derived from India or China, especially for certain centers that have history of regulatory violations, then my flag goes up,” Kaul said.

For-profit watchdogs and limited FDA scrutiny

The globalization of clinical trials has helped precipitate another shift, in the type of institutional review boards that do most safety and ethics reviews.

Most clinical research was once publicly funded and domestically conducted, often reviewed by a hospital, university, or other nonprofit institutional review board. But in the 1970s and 1980s, drug makers started funding a greater share of clinical studies — and increasingly moved them offshore.

In the years after that, private equity money started flowing into the IRB sector. Now, most of the institutional review boards that oversee trials are private, for-profit operations, said Jennifer Miller, a bioethicist at NYU School of Medicine.

And many of them specialize in monitoring overseas clinical trials. WIRB — part of WIRB-Copernicus Group, the biggest commercial IRB — touts its “experience in more than 70 countries” on its website.

But it can be hard to tell just how thoroughly these companies and their foreign counterparts review trials, especially when the study is conducted abroad in countries with sometimes lax supervision by government authorities. (WIRB-Copernicus declined STAT’s request for an interview about its practices.)

“There are thousands of trials that are run currently in countries where they may have formal IRB approval, but still the level of scrutiny that they get is very, very weak or even nonexistent,” Ioannidis said.

Read more: Now angling for a tax break: Research firms that push drugs through clinical trials

And the FDA is not often there to act as a watchdog.

The FDA inspected just 0.7 percent of the foreign sites that ran trials supporting drugs approved in the 2008 fiscal year. That means foreign sites were less than half as likely to be inspected as domestic sites that year. (The agency inspected 1.9 percent of clinical sites in the U.S. for that crop of drugs.)

The FDA didn’t have analogous data for more recent years. Other statistics do show that the agency picked up the pace of inspections in Eastern Europe, Asia, and the Pacific between fiscal years 2008 and 2011. But it’s not clear what’s happened in the years since.

Asked about any actions the FDA has taken in recent years to regulate offshore clinical trials, FDA spokeswoman Lauren Smith Dyer pointed to a collaboration launched in 2009 with the European Medicines Agency, the agency’s counterpart in the European Union. It involves cooperating on trial site inspections and sharing information.

The agreement, she said, “has permitted a more effective use of limited resources.”

Another challenge in running overseas trials: It can be hard to retain staff — and their institutional knowledge — through multiple years of research abroad.

Lead investigators overseas are more likely to be new to the job than their peers back in the U.S. A study published earlier this year found that lead researchers working outside North America are more likely to be first-time registrants with the FDA. They’re also more likely to choose not to conduct another FDA-regulated trial in subsequent years.

“When you have novice investigators, less experienced investigators, they don’t have the infrastructure [or]…the long term experience,” said Getz, the Tufts researcher and lead author on that study.

An unwritten rule for U.S. participation

When drug makers planning to seek U.S. approval enroll patients overseas, they can go down either of two routes.

Some register their overseas trial with the FDA, under a new drug application, just as they would in the U.S. Others skip that step, but if they expect the FDA to ever consider approving their drug, they still get their trial signed off on by a safety and ethics monitor known as an institutional review board — or an international equivalent. (The herpes vaccine trial raised eyebrows, among other reasons, because its sponsor didn’t do that.)

FDA regulators also often voice an unofficial expectation that global clinical trials enroll a certain percentage of their patients in the U.S., depending on the type of trial and the disease being studied, researchers say. That’s so that if the results looks different between the U.S. and non-U.S. populations, there will be enough data to figure out what’s going on.

Read more: Do cancer clinical trials exaggerate the real-world benefits of drugs?

For example, the FDA expects roughly a quarter of patients to be enrolled in the U.S. to support applications for cardiovascular and diabetes drugs, researchers say. Companies aren’t told they must absolutely meet this threshold if they want to get their drugs approved, but they know they can expect a lot more scrutiny of their data if they don’t reach it.

Companies tend to comply with that expectation: Among the 2017 drug approvals where a detailed breakdown of the data was available, only a few companies enrolled fewer than a quarter of their patients in the U.S.: Genentech’s multiple sclerosis drug Ocrevus (22 percent U.S. patients), Johnson & Johnson’s psoriasis drug Tremfya (20 percent), and the Japanese drug maker Mitsubishi Tanabe Pharma’s ALS drug Radicava (0 percent).

Deborah Etchison, a spokeswoman for Mitsubishi Tanabe Pharma, explained the decision not to test in the U.S. as a matter of expediency. Launching new tests of the drug in U.S. patients “would have added several years to the availability timeline,” she said, “and delayed getting this important new therapy option to people with ALS in the U.S.”