Hello, I’m Dr. Jeff Kingsley and welcome to another edition of Riding in Cars With Researchers. Today I want to talk about statistics. This conversation has come up so often over the last few weeks for me with research sites, with Sponsors, and with CROs. These are very intelligent people in the industry and I continually get an AHA moment when people recognize the breakdown of statistics at the research site. Statistics work with very large data sets. If you think about it, if you’ve ever done any work with statistics or if you think about how we design our research protocols for that matter, you have to have a certain number of patients in order to have enough power to be able to say “A caused B”. Statistics work better the larger the dataset. The smaller the dataset, the more inaccurate things become.
Now, let’s make that very simple. Flip a quarter. You can see a quarter – it’s a head, it’s a tail. You know that if it’s a normal quarter, if I flip it enough times, it’s going to be 50% heads, 50% tails. If I flip it just a few times, say four, I can get four tails in a row and that’s not terribly uncommon. I’ve run this exercise in Excel if you want to do it. There are simple formulas you can put into Excel for random number generation and so, you could run a series and then see how the series gives you accuracy the larger the number of rows you use, and gives you inaccuracy the shorter the number of rows that you use. Now, why does any of this matter? Well, because if you’re doing a Phase 3 trial and you’re looking to put 3000 patients in, it’s very possible that over the course of your 3000 patients you have a 50% screen failure rate. It’s also very possible that a very competent site, doing all of the right things, has a 100% screen failure ratio because they’ve only put in 4 patients and all 4 were screen failures. They’ve done nothing wrong as that’s perfectly normal random behavior. That site had 4 patients who came in and one patient was a little anemic and that triggered an exclusion criteria, and another patient just got a diagnosis of a pulmonary nodule suspicious for cancer two weeks ago and needs to be worked up, so that patient has screen-failed for yet another exclusion criteria, et cetera. That is the reality of statistics on a small n (a small sample size), and that’s normal at the site side. Now if you’re a site, the sicker your patient population is, the more this will affect you. The sicker your patient population is on a small n (a small sample size), the greater the likelihood that you’ve got patients who are triggering all of these various inclusion and exclusion criteria.
The healthier your patient population, the more this is muted. If you’re doing vaccine research with young, healthy people, even on a small sample size, you should have a very small screen failure ratio. And on a small sample size, you should still have very predictable enrollment, randomization, and revenue. The sicker your patient population, the more volatility you will have on enrollment, randomization, and revenue. And it’s not that you’re doing anything wrong as it’s normal probability on a small sample size. So if you’re a site, you need to recognize the sources of your volatility to your enrollment predictability and your revenue predictability, and then decide what things that you want to do to help buffer against that, to hedge against that.
You could diversify so that you are in a larger number of medical specialties. I ran an analysis on our company last quarter and our screen failure ratio ran from 8% to 80% across all of our trials, across all specialties. An 8% screen failure ratio in certain medical specialties, certain types of trials, and 80%+ screen failure ratios in others.
Diversification can help you balance all of that. Across all of our trials, they average 50%, so across the entire company we average 50%. But you can see the volatility on a site by site basis. You also need to be able to counter when Sponsors are accusing you of screening inappropriate patients because your screen failure ratio is so high. Maybe you are screened failing inappropriate patients and if so, learn and change your behavior. But it’s also possible that you’re being very appropriate and it’s just probability and then you’re going to have to educate the CRA or the project manager on what you’ve been doing and that it’s completely appropriate.
Thinking in Terms of Statistics
Now if you’re a Sponsor or CRO, begin to think in terms of statistics on small sample sizes. The industry seems to jump to conclusions that are unwarranted because human beings have a hard time thinking mathematically thinking statistically, thinking in terms of probabilities. We can all play better in the sandbox, play better on the same team, if we can begin to recognize the effect of the small sample sizes on sites. And it’s getting smaller! Ten years ago, the average site was randomizing 9 patients per trial. Today it’s fewer than 3 on average based on data that comes out of Tufts University. And so the sample sizes are getting smaller. The industry is running protocols with smaller numbers of patients and the industry is opening up more sites per protocol to get fewer patients per site. All of that means that the sample size per site is going down. And so this effect is being magnified.
Begin to think your way through this because if you want to be a healthy research site. If we as an industry want to have healthy relationships, we needed to begin to recognize the impact of these small sample sizes. Think about it. I would love to talk about it further.
Frankly I’m not a statistician. I think I’ve had one statistics course in my career, but statisticians think about the world in a different way. Their brains work in a different way, and it’s healthy for all of us to begin to recognize the effect of statistics, the effect of probabilities around us.
Send me topics you want me to talk about at firstname.lastname@example.org!
Subscribe to our blog HERE