By Mark Zipkin, Staff Writer

Having helped expose the reproducibility crisis about five years ago, Amgen Inc. is staying at the forefront of the issue with its support of F1000Research, an open-science platform to let researchers check each other’s data. BUt as the field wrestles with causes and solutions, some academics are starting to bristle at the intrusion.

 

The issue was sparked by a pair of papers in 2011 and 2012 from Bayer AG and Amgen, respectively, that showed each company could only reproduce a small percentage of published data when the experiments were tried in-house.

 

Although many companies routinely rerun published experiments to verify them, they rarely go

the distance of publishing their findings.

 

Alexander Kamb, SVP of discovery research at Amgen, told BioCentury there has long been a “culture of publication” at the company that has included publishing negative data, but the environment in the scientific community is changing to make that rarer.

 

“The nature of the process is more challenging and tends to favor new findings as opposed to reconsideration of something,” said Kamb. He added that in current climate, “more prosaic, disconfirming results aren’t so easily published.”

 

Now. Amgen has put its weight behind the efforts of life science publisher Faculty of 1000(F1000) to address the problem with the launch of the Preclinical Reproducibility and Robustness channel on the publisher’s F1000Research platform.

 

On Feb. 4, to mark the channel’s launch, Kamb published a joint editorial with Bruce Alberts, former editor in chief of Science and an F1000Research advisory board member, outlining the goal of the channel as an effort “aimed at strengthening the self-correcting nature of science through the widespread, rapid publication of the failures (as well as the success) of attempts to reproduce published scientific findings.”

 

The same day, Amgen posted the channel’s first three papers in which the biotech documented experimental details of internal studies that refute published findings from four different groups, including one paper from its own scientists.

 

Naming Names

The channel was the brainchild of Kamb and Alberts. According to Michael Markie, associate publisher at F1000Research, the increasing unease in the scientific community about the reproducibility crisis prompted the duo to create an alternative publishing outlet to traditional journals, which routinely prioritize the impact of a study over its rigor.

 

According to Markie, publishing in the F1000Research channel differs from publishing in a journal in several key ways.

 

First is speed to publication: submissions are posed after editorial checks - covering ethical standards, methodology and open access to the data - are performed, but before any peer review.

 

In addition, publication does not hinge on the potential impact on the field.

 

Reviewers are solicited once a prepublished study is posted to the channel’s site, where readers can also post comments. F1000Research authors choose their referees, although the publisher screens the choices to prevent conflicts of interest.

 

Once three referees approve a paper, F1000Research indexes it widely on search engines such as PubMed.

 

Markie said that although there’s still a certain stigma attached to openly contesting published findings, the platform is designed for transparency and dialogue.

 

The idea that the preclinical research community could go beyond the stigma came from a dispute around results from the Mouse ENCODE Consortium.

 

In 2014, two papers by the consortium’s scientists published in Nature and the Proceedings of the National Academy of Sciences presented controversial findings on comparative gene regulation data, claiming that the patterns of gene expression data tended to cluster more by species than tissue. The studies contradicted common beliefs in the field.

 

“It was kind of known in the genomics world that a certain study was quite difficult to believe. It was like the whispers of the conference hall,” said Markie.

 

In 2015, two researchers from the University of Chicago refuted the findings in the Genomics, Computational & Systems Biology subject area of F1000Research. After the negative data were posted, four referees — with names attached — approved it, although one approved it with reservations.

 

According to Markie, the widespread attention prompted a healthy discussion.

 

“What happened there was a big lively debate with a lot of the key stakeholders in that area of science who all chipped in, in a constructive way,” he said, adding that even the original authors contributed to the conversation. “Everything was quite civil.

 

NEGATIVE FINDINGS

 

The studies Amgen posted on the new channel address the roles of specific pathways in obesity, neurodegeneration and Alzheimer’s disease (AD). All three are still awaiting peer review. The first challenged a pair of 2012 publications concerning GPR21 in obesity.

 

A paper in Biochemical and Biophysical Research Communications by Amgen researchers reported that GPR21-knockout mice were resistant to diet-induced obesity, and one in The Journal of Clinical Investigation from the University of California San Diego and Pfizer Inc. showed GPR21 knockout improved insulin sensitivity.

 

But in Amgen’s F1000Research findings, new GPR21-knockout mice were generated that did not replicate either of the earlier studies. Instead, the paper suggested the metabolic phenotypes of the original GPR21 knockout mice were due to unintentional changes in expression of a nearby gene, RABGAP1, caused during generation of the knockouts.

 

Because mouse GPR21 is encoded within a RABGAP1 intron, the F1000Research paper’s authors thought the neomycin cassette insertion used to generate the original knockout mice might have altered RABGAP1 expression. In the new knockout mice — which were generated via a 29-base pair deletion in GPR21 using transcription activator-like effector nucleases (TALENs) — RABGAP1 expression was unaffected, and the mice showed no improvements in glucose and insulin metabolism compared with wild-type littermates.

 

The second paper contested a 2010 Nature report from a group at Harvard Medical School that suggested USP14 slows the degradation of proteasome substrates such as tau and TDP-43, which play an important role in neurodegenerative diseases. Whereas the Harvard group’s evidence showed a catalytically inactive form of USP14 decreased tau and TDP-43 levels in HEK293 cells compared with functional USP14 — supporting the idea that functional USP14 prevents their degradation — the Amgen team found no difference between the two. In addition, siRNA knockdown of USP14 by the Amgen team did not affect endogenous tau expression in a different cell line.

 

The Amgen team indicated in its paper that differences in the method, such as the expression vector used, could underlie the discrepancy, and noted that follow-up studies by the Harvard group had failed to show the USP14 effect occurred in vivo.

 

At least one company has been targeting USP14. In 2013, Proteostasis Therapeutics Inc. received a grant from The Michael J. Fox Foundation for Parkinson’s Research to develop a USP14 inhibitor to promote clearance of α-synuclein to treat Parkinson’s disease (PD), aiming for the clinic in 2015

 

In December 2013, Proteostasis announced a partnership with Biogen Inc. to continue development, and in 2014 it received milestone payments from Biogen. Biogen did not respond to requests for comment.

 

The third paper was the only one to receive a response so far — which highlighted how some in the community are viewing the initiative.

 

The study centered on results from a 2012 Science paper from an academic group headed by Gary Landreth at Case Western Reserve University School of Medicine. Landreth’s study used a mouse model of AD, and showed the RXR agonist Targretin bexarotene produced more than 50% reduction of β-amyloid plaque area within 72 hours, reversed cognitive and social behavior deficits, and improved neural circuit function. Landreth is a professor of neurosciences and neurology, and director of the Alzheimer Research Laboratory at Case Western.

 

The Amgen group treated wild-type Sprague-Dawley rats with Targretin, but detected no significant change in β-amyloid levels after three days or seven days. It did not perform behavioral assays or examine neural circuit function.

 

In a response posted on F1000Research, Landreth argued that the reason for the difference is that the Amgen researchers used “the wrong formulation.” In the 2012 study, his group used the clinically approved formulation of Targretin, which is a microcrystalline form of the drug. Amgen’s group used a soluble form of the molecule, which Landreth stated would have a different PK profile that would affect its activity.

 

He added in his response that the importance of the formulation had been well documented in the literature and the FDA filing, and was detailed in a response to four comments on his study published in 2013 in Science. He also noted the use of different species in the Amgen study.

 

“The Amgen scientists (and others) clearly did not make an effort to understand and replicate the original study design,” he wrote in his rebuttal on F1000Research.

 

Landreth concluded by stating a “logical flaw” in the Amgen paper undermines its conclusion. “I think this study is emblematic of the problems associated with reporting ‘failure to replicate’ findings in studies that do not genuinely reproduce the published work,” he wrote.

 

According to Landreth, ReXceptor Inc. licensed options from Case Western on the use of bexarotene in the treatment of AD. ReXceptor did not respond to requests for comment.

 

At least one other company is developing an RXR agonist: Io Therapeutics Inc. has IRX4204 in Phase I testing for AD and PD.

 

WHOSE CRISIS?

 

Since the problem was brought to light by Amgen and Bayer, the academic community has responded with several initiatives to address reproducibility (see “Reproducibility Initiatives”).

 

According to Kamb, to define what constitutes “reproducible,” it’s important to think about what the data need to support.

 

“The key thing is that the clinical hypothesis applies robustly in the maelstrom of the clinic,” he said. “So there is work that might be defined ‘reproducible’ in a very narrow sense, but what we’re trying to do here is find mechanisms and hypotheses that are robust enough to really make it in the clinic.”

 

At the Global Biological Standards Institute (GBSI)’s annual summit on Feb. 9, keynote speaker Judith Kimble said the two papers kicking off the reproducibility crisis were “a bomb” for biomedical researchers. However, she added, “The first question is, is it true? And I think we don’t really know whether or not it’s true yet.”

 

Kimble is a professor of biochemistry at the University of Wisconsin-Madison and an investigator at Howard Hughes Medical Institute (HHMI).

 

Kimble questioned the accuracy of one of the leading points of the summit: a GBSI-backed study published in a June 2015 Perspective in PLoS Biology which calculated the costs of irreproducible clinical research at $28 billion.

 

“Trying to define what is reproducible and what is not — what is a reproducible paper, what is a reproducible panel — is a science in and of itself,” said Kimble. But regardless of the exact number, she added, “There are clearly problems.”

 

During her talk, Kimble pointed to a number of well-known causes of irreproducibility, covering inadequate training, problematic stocks, lack of transparency and occasional fraud or misconduct. But Kimble characterized these as symptoms, adding: “The elephant in the room is hypercompetition.”

 

Hypercompetition, she said, has resulted from ratcheting up the healthy competitive pressures of the field to “the point where something starts to break.”

“I would say that the system is at that point,” she added.

 

One culprit is the push from scientific publishers, the job market and industry to see that results and publications are clinically relevant, she said. Another driver is the increase in the number of researchers while overall financial funding has decreased, with NIH funding down 30% in constant dollars since 2003.

 

Although NIH received a budget increase of 3%, bringing its total for FY16 to $31 billion, Kimble said it would require 5% annual increases for the next five years just to get back to 2003 levels, meaning it is unrealistic to expect the public sector alone to solve the hypercompetition issue.

 

“This bomb that came in 2012 came from the pharma/biotech industry,” said Kimble. She added: “The pharma/biotech industry doesn’t do its own basic research anymore. It relies on research that’s paid for by NIH, and yet it screams ‘bloody murder’ when things go wrong.”

 

F1000Research expects more companies besides Amgen will get involved in the channel as it has actively solicited research from industry, and Markie expects several pharmaceutical companies will jointly publish a paper on the channel in the coming weeks. Still, he said, “We don’t want it to turn into this ‘pharma can’t do it’ channel.”

 

“Obviously Amgen have taken the leadership position here like they did with the Bayer article a few years ago,” said Markie. But he hopes the channel will grow to see 50-100 publications annually, from both industry scientists and academics, “when someone else apart from Amgen has done it.”