9
Breaking Down Cass Review Myths and Misconceptions: What You Need to Know – The Quackometer
(www.quackometer.net)
this post was submitted on 14 Apr 2024
9 points (100.0% liked)
Skeptic
1477 readers
2 users here now
A community for Scientific Skepticism:
Scientific skepticism or rational skepticism, sometimes referred to as skeptical inquiry, is a position in which one questions the veracity of claims lacking empirical evidence.
Do not confuse this with General Skepticism, Philosophical Skepticism, or Denialism.
Things we like:
- Civility
- Thoughtful discussion based on evidence and facts
- Humor
Things we don't like:
- Personal attacks or disrespectful attitude
- Wild speculation on events with no evidence
- Low-effort comments and posts
Other communities of interest:
"A wise man proportions his belief to the evidence." -David Hume
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
A better breakdown https://www.advocate.com/health/hilary-cass-nhs-report-debunked#toggle-gdpr
And edited this one too https://www.tandfonline.com/doi/full/10.1080/26895269.2024.2328249
Ms Reed is not an objective source. Nor does it appear she has much experience with systematic reviews. Indeed she repeats many of the "myths" such as the one about 98% of studies being thrown out.
"A closer inspection of the reviews released alongside the Cass report reveals that 101 out of 103 studies on gender-affirming care were dismissed for not being of "sufficiently high quality," "
That is a lie.
This is another lie from Ms Reed. I am beginning to think this "debunking" article you've shared is actually the original source of most of the myths being peddled on social media.
It's not a lie, they were mostly dismissed even according to your own article, they were dismissed and synthesized into one conclusion. That is still dismissal.
101 of 103 studies were not dismissed. All systematic reviews classify their source studies based on the quality of the work. Of the 103, two were classed as high quality, 58 as moderate quality and the remaining 43 as low quality. For synthesis, only high and moderate quality studies were drawn on. That's more than half, not 2%.
So yes, Erin is lying.
You can't say she's lying until we do a systemic review of why the Cass study dismissed everything but 2 studies for the numbers it used to reach its conclusion. You can't say she's lying without that review no more than I can support Erin by reading each study that was dismissed. What I can tell you is that dismissing that many studies is not normal scientific analysis. It reeks of bias.
This is the lie. They didn't dismiss all but two studies, they actually included 60. More than half of the 103 studies identified for the review.
So yes, Erin, and now yourself, are peddling a lie.
It's key part of synthesising multiple sources into a meta-analysis. Including poor quality studies dilutes the quality of the overall analysis.
By design, it's biased towards higher quality research.
Synthesis is a paragraph summary inclusion ONLY, it means they didn't use data from the study, it is dismissal. I'm done arguing that with you.
They have absolutely used the data from those 60 studies. You can read where they say explicitly that in the report if you cared to.
You are utterly mistaken and firm on your conviction, these are not the qualities of skepticism.
"Don't seek refuge in the false security of concensus"
That's not what synthesis means. I've written synthesis reports before and the data you include from those reports once you have dismissed them as inaccurate, it is an entirely selective process of whatever you want to include from them. We even have a phrase for it in law, Summarily dismissed.
And of the 103 reviewed they included data from 60. It is a lie to say they "dismissed all but two."
Legalese is irrelevant. A systematic review of scientific literature is a different beast to "writing a few synthesis reports".
So you don't know what you are talking about. Gotcha.
Synthesis reports in a scientific study when presenting data, are the parts of the report where you explain why you are dismissing data, so in this case ~98% of the data or studies. So what you just said is ~98% of the data was included in the synthesis report. that's not inclusion of the data. That's selective inclusion to support a conclusion. A normal scientific study can't dismiss 98% of available data. That reveals bias.
You can read the reviews for yourself
https://t.co/82Rjs2L1pA
https://adc.bmj.com/content/early/2024/04/09/archdischild-2023-326669
Let me know where you find the bit where they dismiss 101 out of 103 papers.
Hint, they didn't. It is lying to say they dismissed 98% of the studies they looked at.
Read it. Their only inclusion in the report is to half explain why the were discluded, exactly what I said. Most of the dismissals are unscientific, not supported by a statistical analysis of why it was discluded. Data doesn't become unreliable just because it is incomplete.
That report is absolutely rife with white washing and selection bias, I'd expect a scientific review of trans literature and studies to be a book at this point not 32 pages dismissing 98% of the data. It's frankly insulting to anyone that's read or written any number of scientific studies.
They did not dismiss 98% of the data.
Putting 98% of the relevant available data in a supplementary table like 4 is not including the data.
Supplementary Table 4 (from the first review) is a list of each of the 53 studies included in the review and how they were scored based on the Newcastle-Ottawa scale.
The "data" is in supplementary tables 3, 5, 6 and 7. Only studies that were scored as low quality were excluded from the synthesis.
"They dismissed 98% of data" is a lie.
No it's not. None of the dismissals are statistically/ scientifically supported, and the data they present is blurbed and incompletely presented in a way that isn't inclusive of what those studies actually say.
Nothing was dismissed at all (and "statistics" has nothing to do with it so curious to mention it).
Studies were scored for quality on the well established Newcastle-Ottawa Score. High and Moderate quality studies were included in the synthesis. Low quality studies were not, but their outcomes are still reported.
Outcomes from each study were included in tables 3, 5, 6 and 7.
'They dismissed 98% of the data" remains a lie.
You can't remove a study from a scientific paper without having statistical analysis to back it up. Each of those removed studies all had a statistical analysis of how confident they remained in their data even with the gaps. Because there aren't completed 100% studies in science it just doesn't happen so you use the data you have and test it for a confidence value you obtain using statistics. And the idea that some trans people don't make it to the completion of a study due to personal reasons or even suicide isn't that rare. Not using 98% of the data because of that would be stupid.
You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality.
Studies are also scored low on quality if, for example, they don't control for important sociodemographic confounders. Study that do control these, will have more reliable results.
You can read how the scoring works in supplementary material 1.
"They dismissed 98% of the data" remains a lie. Repeating it doesn't change anything.
"You can of course. Statistics are not required to explain why a self selective Facebook poll is low quality while a multi centre 5 year study with followup and compartor is of a much higher quality".
That's wrong when you are trying to be scientifically correct. A science paper without that math isn't science my dude. And comparing trans healthcare data to Facebook polls is ridiculous
It's remarkably common in systematic reviews, a feature even. You give the impression that this is a new or foreign concept to yourself and are just encountering these ideas for the first time.
Search on pubmed or the bmj or the Cochrane library for other systematic reviews using the Newcastle-Ottawa score. You'll trip over them.
One of the studies reviewed recruited patients over Facebook and polled them.
"They dismissed 98% of the data" remains a lie.
Again I've written these reports. It is absolutely not common practice to disclude data without scientific reason and analysis. It is explicitly taught not to do it that way in college. And it is not scientific to do that without a statistical threshold and confidence analysis of your reasoning.
I am forced to strongly doubt this given your whole misunderstanding of the basic concepts on assessing methodical quality..
Certainly, you've never authored a systematic review for a reputable medical journal.
But don't take my word for it...
https://handbook-5-1.cochrane.org/chapter_13/13_5_2_3_tools_for_assessing_methodological_quality_or_risk_of.htm
You mean such as using a method like the Newcastle-Ottawa score to assess data quality?
If your college course covered systematic reviews and didn't include a review of study assessment methods, ask for a refund.
Statistics are not required to assess that a study without a comparator is weaker than one with.
"They dismissed 98% of data" remains a lie.
The Newcastle method is not seen as a scientific basis for dismissal on its own.
98% of the data was dismissed in the synthesis and was not used to reach the conclusion that there wasn't enough scientific evidence to support transition when 98% of the science says that is wrong.
And every scientific paper is expected to be comprehensive on its subject matter and/or thesis.
It's not used for "dismissal" it's used to score studies on their likelihood of bias. Studies without appropriate controls for example are more susceptible to bias than those with.
Demonstrably false, only low quality studies were excluded from the synthesis which account for less than half of the 103 reviewed. A lie is a lie no matter how often repeated.
That's not what the conclusions say, for example:
And
"They dismissed 98% of data" remains a lie.
https://www.tandfonline.com/doi/full/10.1080/26895269.2024.2328249
That was published a month before Cass came out and so hasn't anything to do with the two systematic reviews being discussed above. It doesn't even mention them.
I'm uncertain what expertise a business graduate can bring to assessing the quality of a systematic review in medicine.
Readers are free to Google the author and subsequently make a judgement on their objectivity on the subject matter.
And yet you have no scientific reason other than an ad hominem fallacy with the author with which to dismiss the criticism with. That like the Cass report are not scientifically sufficient reasons to disclude the criticism or the data respectively.
And I can garuntee you that the Cass report was not peer reviewed like all of the studies they dismissed were because it would have been torn apart. That's the real litmus test of scientific debate.
If they made a scientific argument about these review papers under discussion I might but this is just a polemic using unscientific language like "cis-supremacy" in a low impact obscure journal.
Newcastle-Ottawa scoring is a scientific method for weighting the methodical quality of scientific studies.
It was peer reviewed since thats BMJ policy, unless you have evidence to the contrary. There is even a link on the online edition of both reviews for you to submit a rapid response pointing out all their flaws which I would encourage you to do.
Interestingly some nice fellow DM'd me with a link to "Patient Zero" of the "they dismissed 98% of the data" myth.
https://twitter.com/benryanwriter/status/1779671152148857212
And of course, everyone has doubled down rather than admit they read the wrong paper. A better "litmus test" of scientific debate is humbly correcting yourself when shown to be wrong.
"They dismissed 98% of the data" remains a lie.
98% of the data could be summarized in one sentence. Trans healthcare and hrt works. 98% of the data comes to that conclusion with vast consensus across multi disciplines and fields comes to that conclusion and that was ignored. 98% of the data was discarded. Most of those studies discarded already had a statistical analysis backing up their efficacy while the Cass report doesn't. Nor does the Cass report include a nearly mandatory implicit bias report.
Those peer reviews are most likely selected and not randomized selections or contestations as most peer reviews are required to be, they are ok for initial release irc. But it is an outgoing process that doesn't have an endpoint. They were most likely provided prior to release and the normal peer review process won't be completed for years to undo the damage. But it is not considered peer reviewed yet.
Again you have not proven that the new castle Ottawa scale has any efficacy or scientific merit as a disqualifying tool No one has as far as I know.
Liar.
So now they were peer reviewed but by people you don't trust based on the same evidence you used to assert it wasn't peer reviewed in the first place I.e. zero.
You'd better tell the Cohcrane library to bin every systematic review they've ever done which used this system then. I'd be eager to hear their reply to you
98% of the data was discarded
No I'm just explaining the process and why it isn't complete yet. Or even valid yet
And show me that the Cochrane library ever discarded a study using the criteria even once yet alone with the same level as the Cass report and I'll write them
For something that illustrates the problem with the Cass report read https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/
It was not. All studies that scored high or moderate quality made it into the synthesis. That's 60 out of the 103 looked at, that's not 2%.
You are speculating, based on nothing.
Here's one I found in <7 seconds
https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD013256.pub2/full
If you want to find more simply search the Cochrane library for reviews with "Newcastle-Ottawa" in the main body of text. It seems like this is new to you.
The relevance of a joke paper from 2003 to a systematic review published last week is certainly questionable but it seeks you're trying to imply that Cass discarded anything except RCT's. The didn't and that''s myth #2 from the original Quackometer article.
Where will the goalposts move next?
Goal posts haven't moved and I've already pointed out a dozen of so methodological flaws around the Cass report that you are choosing to ignore.
That's on you