The university of Surrey has recently reported that it is publishing a research paper in the British Journal of Psychology dealing with whether study participants could tell the difference between statements made by convicted rapists and those made by men's magazines. There was one giant glaring problem I immediately saw with this study allowing for giant bias--the researchers were the ones choosing the quotes and thus could choose what quotes participants were reading. Why is this problematic you may ask. The reason is that if you select the proper quotes from two sources you can make any two groups look alike. So long as you keep the quotes short and obscure enough it will be hard to determine who made a statement. Of course, I have not seen the research because it has not been released yet, so I qualify my following arguments with the fact that the research may adequately address my complaints.
So, what did the study find... it turns out participants couldn't accurately guess which statements were made by rapists and which were made by men's magazines. Now, without any further ado I will address the large potential problems I see with this research and the press release by University of Surrey and why I think this conclusion is misleading. I will address 3 main problems I see: 1) guilt by association fallacy, 2) misleading selection of quotes, 3) cherry picking quotes.