Nov. 14th, 2010

lederhosen: (Default)
Once upon a time, the usual explanation for parapsychology "findings" was bad experimental technique (subjects were getting cues as to the correct answer, that sort of thing). Sometimes due to deliberate dishonesty, often because experimental design is hard.

These days, experimental technique has tightened up a lot, due largely to sceptics and partly to those parapsych researchers who think they've got something and want to be taken seriously.

This has established that if psi effects exist, they're weak*. If we had psychics who could predict the outcome of a coin-toss with 100% accuracy - or even 55% - one of them would long ago have claimed James Randi's money. To detect weak effects, you need to run very large experiments containing thousands or millions of trials, and perhaps apply sophisticated statistical techniques to analyse the data.

The problem with this is that with a large data set and a lot of choices about how to perform your analysis, it's awfully easy to cherry-pick for significance. So, my suggestion is:

Every time a human subject generates a data point (makes a prediction, etc etc), we should use a random number generator to generate, say, a thousand fake versions of the same data point, all consistent with the null hypothesis. From these, we form one real data set, and a thousand fakes.

Stats analysis is then performed blind - the statistician decides on appropriate analysis techniques without knowing which is the real data set. The decision on journal publication is also made blind; only after the article is irrevocably committed to publication does anybody get to find out whether the 'significant' data set identified is the real one.

This wouldn't fix all problems with parapsych research (and depending on the nature of the data, it might take some work to generate fake data that can pass as the real thing) but I think it would be useful in many cases. This needn't be restricted to parapsych; I'm pretty sure there are other fields that would benefit from this too.

*I'm not inclined to dignify the "we have strong psi powers that completely vanish under fraud-proof conditions" argument with a response.


lederhosen: (Default)

April 2017

2345 678

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jun. 26th, 2017 07:09 am
Powered by Dreamwidth Studios