A recent analysis of fMRI data from 2 clinical trials (n=71) argued that a high dose of psilocybin (25mg) leads to a "global increase in brain network integration."

This integration was not found in those who had received a common antidepressant. And thus, the authors argued that this integration in the brain network is a mechanism through which the antidepressant effects happen.

But not so fast - say Manoj Doss, Frederick Barrett & Philip Corlett - there are some big flaws in the analysis. Their commentary - available on a preprint archive - was rejected by Nature Medicine. So, as one would expect - it's getting even more attention now.

Here are the 3 key critiques they have:

1. Picking the measure that showed the biggest difference

A study usually looks at more than one outcome. In this case, at least two measures of depression were used. But, instead of using the main measure (QIDS) - which in the larger study showed no statistically significant difference - the researchers looked at the other measure (BDI).

That in itself is somewhat of a no-no - but this also omits another factor named 'multiple comparisons'. If you look at multiple outcomes, you are more likely to find a significant result. Thus, it's usually required to correct - make your analysis more stringent - for this.

In the analysis, the correction was not done.

2. The antidepressant group already had the lower scores - a floor effect

The fMRI measure being looked at was the brain modularity - the lower the (supposedly) better.

But, "the mean modularity before treatment in the citalopram group [was] nearly equal to the mean modularity in the psilocybin group after treatment."

In other words - the antidepressant group already had low scores. This could mean that there was no more room to lower them further.

Another - just as boring - explanation is 'regression to the mean.' This means that scores that are (by chance) unusually high or low, are more towards the centre the second time you measure it.

3. How we take the fMRI measures may introduce a bias

The fMRI measures are taken when someone is resting (task-free scanning). You wouldn't expect many differences there, right?

Well, not everyone 'resting' is doing the same. An explanation for the increase in network integration could be that participants after psilocybin were engaged in more non-specific cognitive tasks.

Putting aside the anti-depressant effects (convincingly shown in other studies in the days after 25mg psilocybin), participants might have just been more rested.

Reading between the lines - what I think Doss, Barrett & Corlett want to say (scream?) is that psychedelic science should be rigorous - more so than is being done in the analysis they critique.

It's amazing that psychedelic research has made it to Nature Medicine - it's less cool if in 5 years all preliminary findings turn out to be statistical flukes.

I'll end it with the last sentence of their critique: "As a field, we should keep our minds open, but we should also remain vigilant to being misled."