OK, Theranos: Here’s the Data the World Needs to See
Last week, one of America’s most prestigious newspapers published a pair of blistering critiques of the country’s hottest biomedical startup. The Wall Street Journal alleged that the blood diagnostics company Theranos wasn’t operating at nearly the level of technological innovation its hype (and $9 billion valuation) had led the world to believe.
After a week of objections across various media platforms, Theranos today released a document offering a bullet-pointed rebuttal of the Journal‘s allegations (not to mention plenty of broadsides at the paper’s ethics and credibility). But the one thing the 6,000 word rebuttal does not do—and the one thing it really needed to—is provide data.
I reached out to Theranos, asking them if they’ll be releasing their primary data. Their response: “All of the information we have released at this time is available online; if we release more, we will be sure to let you know.” Since I’m not holding my breath anymore, let’s talk about data.
Data, data, data. If you’ve been following Theranos long enough, you’ll recognize this refrain: a call for transparency so the medical community can verify that Theranos’ technology does what the company claims. When Theranos deigns to respond, it usually says something along the lines of “We have data! We’ve shown it to the FDA! We post it on our website!”
All technically true, but the FDA data shown in that paper are aggregate results, not primary. What does primary data look like? What does it do? How would you know it when you saw it?
We’ll get into that, I promise, but first let’s review what Theranos is. The company is aiming to offer more than 200 blood tests at prices less than half the Medicare reimbursement rate. Further, rnse tests would be available without a doctor’s prescription. And best of all, many tests would be available via a relatively painless fingerstick instead of a significant blood draw. “When I first heard about Theranos I thought, ‘This is fantastic! How great would it be if everything could be done by fingerstick,’” says Steve Master, director of clinical chemistry at Weill Cornell Medical College.
The Journal‘s article said, in essence, that those fingerstick tests did not work as advertised.
Not surprisingly, Theranos disagrees. And if you read the company’s release detailing all the reasons why, you will probably get confused. It is very long, and very detailed, and deals in the nuances of things like regulatory approval, the testing of tests, why certain tests get tested some ways while other tests get tested in other ways, and why Theranos’ tests are special and therefore get tested in a way that the Journal just didn’t take the time to understand. The document is nothing if not thorough in addressing every allegation brought forth by the Wall Street Journal.
But Theranos does not offer any hard evidence. Despite all the dense regulatory nuance, the release is pretty much a rhetorical combo move. It has none of the data that would make it a proper scientific rebuttal.
So. Data. Developing lab tests is a complicated process, and it involves a lot of working parts. “When I look at the quality of a test, I look at four main things,” says Ed Thornborrow, pathologist at UCSF:
Is it accurate?
Inter-instrument testing compares results (from the same batch of blood) from machine A to the results from a validated machine B. Imagine that instead of a blood testing company, Theranos is a person playing darts in a bar. The proficiency tests would measure how close Terry (if I may) came to hitting the bullseye with each shot (with the bullseye being the targets established by machine B).
Is it precise?
Additional validation tests measure the test’s results against itself. Imagine Terry back at the same dart board. Her shots all come really close to the bullseye, but each lands in a slightly different place. “Precision is how often you can hit exactly the same spot on that dart board, every single time,” says Thornborrow.
The results from these tests would look something like the graphs Theranos has on its web site. The difference would be that the supporting data showing how those dots got lined up so perfectly would be available for scientists to look at.
Does it have a valid reference range?
“For a lab developed test, you actually have to draw a large number of people and run their samples on the device,” says Thornborrow. These patients are supposed to be a representative sample of your patient population. “Basically what that says is if you were to pull 100 people off the street, 95 of them would give you values from this point to this point.”
Imagine that each patient is a dart that Terry has to throw at the board. About 5 percent of those might hit the edge of the board. Maybe the dart was a little lighter or the feathers angled slightly differently. Terry threw the darts exactly the same, they just naturally went outside the target. It’s not the test’s (I mean Terry’s) fault. These darts (er, patients) just have different body compositions (no judgment).
What is the linearity over the reportable range?
Despite the math terms, this one is actually pretty easy to understand. “The linearity is just a range of results that a test will be both accurate and precise,” says Thornborrow. As you get to the extreme values shown in the test results, the level of accuracy and precision break down. This data is critical for qualitative tests that measure values over a range—like a Vitamin D or potassium test—as opposed to quantitative yes/no tests, like “do I have herpes?”
At the dart board, this is really just how close or far away Terry can stand and still hit that bullseye.
Theranos has painted the medical community’s thirst for data as a conspiracy-fueled publicity smear campaign, like a health care version of truthers or birthers—These agitators are really just moles for the company’s competitors, digging for proprietary secrets. Yes, the company has every right to protect its secrets, but it has always seemed strange to make the case that showing primary data would give competitors the edge they need to hack apart Theranos’ black box.
The company’s other line of defense is that their technology is new, with no other tests to compare it with. That might be true, but it’s definitely moot. “Even if you don’t have a peer group, that doesn’t get you out of proficiency testing,” says Thornborrow. Theranos still had to do proficiency tests that pit its technology against some other established of test. “You might not get graded, but your medical director has to review and determine if the results are satisfactory to ensure accuracy and precision.” So Theranos has that data—it’s just not sharing it.
“If they just published then we’d get to look at the data and say we agree or don’t agree, but at least it would be a public discussion,” says Master. “I don’t think Theranos is being treated unfairly in being asked for this.” And if they did release the data, it would be a step toward showing more faith in science than public relations.