Allegation: In some medical fields, a quarter of studies may be bullshit, and half contain errors

[SwampNut]

https://www.nature.com/articles/d41586-023-02299-w?utm_source=tldrnewsletter

 

How many clinical-trial studies in medical journals are fake or fatally flawed? In October 2020, John Carlisle reported a startling estimate¹.

 

Carlisle, an anaesthetist who works for England’s National Health Service, is renowned for his ability to spot dodgy data in medical trials. He is also an editor at the journal Anaesthesia, and in 2017, he decided to scour all the manuscripts he handled that reported a randomized controlled trial (RCT) — the gold standard of medical research. Over three years, he scrutinized more than 500 studies¹.

 

For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either because the authors were incompetent, or because they had faked the data.

 

Carlisle called these ‘zombie’ trials because they had the semblance of real research, but closer scrutiny showed they were actually hollow shells, masquerading as reliable information. Even he was surprised by their prevalence. “I anticipated maybe one in ten,” he says.

[RXX]

On 7/19/2023 at 2:16 PM, SwampNut said:

https://www.nature.com/articles/d41586-023-02299-w?utm_source=tldrnewsletter

 

How many clinical-trial studies in medical journals are fake or fatally flawed? In October 2020, John Carlisle reported a startling estimate¹.

 

Carlisle, an anaesthetist who works for England’s National Health Service, is renowned for his ability to spot dodgy data in medical trials. He is also an editor at the journal Anaesthesia, and in 2017, he decided to scour all the manuscripts he handled that reported a randomized controlled trial (RCT) — the gold standard of medical research. Over three years, he scrutinized more than 500 studies¹.

 

For more than 150 trials, Carlisle got access to anonymized individual participant data (IPD). By studying the IPD spreadsheets, he judged that 44% of these trials contained at least some flawed data: impossible statistics, incorrect calculations or duplicated numbers or figures, for instance. And 26% of the papers had problems that were so widespread that the trial was impossible to trust, he judged — either because the authors were incompetent, or because they had faked the data.

 

Carlisle called these ‘zombie’ trials because they had the semblance of real research, but closer scrutiny showed they were actually hollow shells, masquerading as reliable information. Even he was surprised by their prevalence. “I anticipated maybe one in ten,” he says.

 

Go figure. makes sense if you think of studies like other manufactured items. What is the recall rate on the average automobile? Or how many times have you gotten an item that doesn't work? Probably not 44%, but there are lots and lots and lots of parts to studies produced by lots and lots of people of drastically varying competencies. Do you think every medical research facility or faculty has some super power to employ only the best and brightest? A study is only as good as the weakest link in the weakest citation in the study, which is be based on data from even more obscure citations. If you go down the citation research rabbit hole (I have done it just a few times in my life) you can do exactly what this guy did. It is daunting work, but pretty damned satisfying when you find the "AHA" buried in a second-or-third tier footnote.

And of course we have repeated examples of top-tier researchers caught actively falsifying data and finally getting busted.

[Zero Knievel]

So long as there’s an incentive to lie/cheat without serious consequences, it will happen.