SIO150: Ok ESP Isn't Real... with Dr. Lindsey Osterman

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:28 Thomas: Hello and welcome to Serious Inquiries Only, this is episode 150, I'm your host Thomas Smith, and even though you're probably expecting a voicemail episode, I was just so excited about this follow-up to the last episode that I wanted to get this out first, and then on Thursday I'll have that voicemails episode for you. So we get into it in the interview, but there was a lot of negative response to that last episode. A lot of perfectly good responses, a lot reasonable responses, but actually also a lot response that I thought was wrong. People weren't happy with the way we talked about the paper that was published in the APA journal, and I think people had objections that actually weren't even correct, even though ultimately, as you'll see today, I think there are some very good objections to this paper.

0:01:18 Thomas: So I don't know. I think this was a really interesting exercise in skepticism. We go into it a little bit here in the beginning. So let's get right on over to Dr. Lindsey Osterman talking from a real position of expertise on this review paper.

0:01:42 Thomas: Joining me is Dr. Lindsey Osterman. How are you doing, Lindsey?

0:01:45 Lindsey: Fantastic, how are you?, I

0:01:47 Thomas: Good. I'm really, really thankful that you're coming on. You took the time to do a lot of digging into, well, into a lot of different studies that were referenced in this meta study.

0:01:58 Lindsey: Was happy to do it,

0:01:59 Thomas: Yeah, and seriously, I'm really happy to have you. Thank you so much. Lindsey's been on the show previously talking about EvoPsych, right? I don't know how long ago that was, but, oh yeah, that's right, right. You were telling me that the other day. Yeah. It feels like yesterday, but simultaneously everything feels like Yesterday, but it's also five years ago.

0:02:07 Lindsey: Mm-hmm, almost exactly two years ago,

0:02:19 Thomas: So I can Never tell. But do you want to, just for people who might not have been listening two years ago, do you want to go a little bit into your background just to let before us know what kind of expertise you're bringing to this analysis here?

0:02:18 Lindsey: Yes, exactly, sure, so I got my PhD in psychology, not like clinical or counseling, but of the research variety, which Is relevant here, I'm an assistant professor, I teach a lot of methodology courses and statistics courses here, so, so yeah, I have some background into some of the things that are relevant to criticizing this this paper.

0:02:38 Thomas: Right.

0:02:55 Thomas: That's so great. I'm really excited to hopefully learn a little bit more about the statistical analysis required for this kind of Thing because it's frankly it's you know it's not anything I have a background in I'm Actually a super mathy person but I, but not stats I took a bunch of calculus and differential equations and stuff but I never was That good at stats but I Just want to say at the outset that as as for the episode because you know it got a lot of pushback and I get it you know, it's kind of it's a sacred cow or not sacred cow, I guess. And I, and I don't mean to belittle it either. Like I understand there are a lot of people who know no more about certain elements of this and pushback and that's, and I welcome that and it's great. But one thing I want to say is I don't think anyone, I still don't think anyone on the episode said anything wrong. You know,

0:03:02 Lindsey: Yeah. Yeah. Yeah. No.

0:03:45 Thomas: Like there wasn't, there wasn't a whole lot that was wrong. It's more, and that's what' s so interesting. It's, it's being a skeptic, I think, is an identity group. And there's a lot of tone involved, you know? And it's like the fact that we weren't just mocking it or something kind of signals to certain people, I feel like.

0:03:47 Lindsey: I Don't. Either. Yes.

0:04:05 Lindsey: Yeah.

0:04:05 Thomas: But I will say for my part, I do wish I had taken a slightly different tone because I saved a lot of my attitude for the end where I said, you know, like, I really highly, strongly doubt this is real, but as a non-expert, I don't know how, I can't just assert that, you know, like I can't just without, so I wish I had maybe done that more in the beginning with Aaron and Jesse, and that might have affected the tone a little bit, so for my part, but I don't know that there's, and you can correct me because you're absolutely the expert here, if there's anything that I said or that the guest said that is factually wrong, please, by all means, call us out on it. But yeah, that's my impression. What is your impression?

0:04:22 Lindsey: Yeah.

0:04:45 Lindsey: No, I, I thought that the criticism of the episode was was pretty unfair, at least some people's criticisms were pretty unfair. Um, I'm, I'm very sympathetic to Aaron's frustrations with critiques that stop at the prior plausibility of side claims. I think he's absolutely right that in order to legitimately criticize this, you have to engage with the evidence. And that's, that's what I'm here to do. Right.

0:05:10 Thomas: Awesome.

0:05:11 Lindsey: So yeah, no, I mean, and just a second point on that is that I think that Aaron's right now that like, this seems interesting from from what was presented in the article, and I'm going to wait and see what others say is appropriate. If you, if you're just going on the claims in this article, and if you are giving Cardinia the benefit of the doubt that he's representing it honestly, then like, yeah, that's a totally reasonable position. This seems like strong evidence. So that's, that was my impression of the episode.

0:05:41 Thomas: Interesting. I'm sure we're going to dive more into it, but I kept asking, you know, how statistically significant was this result? And maybe that's something you can speak on a little bit. If we were to take it at face value that, you now, like pretend all the data is 100% good for now. We'll go into that that later.

0:05:50 Lindsey: Yeah.

0:06:01 Lindsey: Okay. Yeah, so I mean, that is the meat of what I'm gonna talk talk about, but let's take his claims on face value. If everything that he's saying is accurate and complete about psi evidence, it does seem like pretty persuasive evidence.

0:06:14 Thomas: Wow. Yeah. I mean, and I think that's, I think that's the point because I did see a lot of people who were dismissing this in ways that were not right, in my opinion. I mean, so that I think, that what you said there is really important, you know, because that does put it down to, you know, how much can we trust this peer reviewed article? And a lot of people are highlighting a lot reasons why we can't, and that's great but like the point is something is wrong, you know what I mean, like it's it's either this weird size stuff is real, which none of us really think is true, but it's either that or there's a significant problem with this being published, and I think either way we've we've got to talk about it, so I guess, uh, you want to dive into it?

0:06:43 Lindsey: Yes.

0:06:58 Lindsey: I want to respond To that last point. So there's a parallel here with the publishing of this Review View article and the publishing of a primary research article by Bem in 2011. So this was a 10 experiment study on Psy, and it claimed to show evidence for Psy. And it was published in the flagship journal of social psychology, like the top journal in our area. And the reason that the publishers gave for publishing it was like, look, he is conformed to all the same standards as anybody else in this area as far as we can tell from this article, and so it would be unfair not to publish it. It wasn't an endorsement of what he was claiming, though.

0:06:58 Thomas: Sure. Sure.

0:07:44 Thomas: So does that mean there's the weakness is somewhere in the methods that, you know, like, like, where, where is the, cause that one I'm guessing was, was even more nonsense than this one or the same amount of nonsense.

0:07:49 Lindsey: Yes. They're nonsense in different ways, so I'm going to talk more about them later, but so let's get into it. So The first thing I want to do is just briefly talk about my experience reading this review as somebody who reads a lot of articles like this, um, and by the way, I want to point out this was called a meta-analysis in the last episode and it is not, it's a review paper, um.

0:07:59 Thomas: Okay. Okay. Okay. Sure. Okay. All right.

0:08:20 Thomas: Oh, okay.

0:08:21 Lindsey: Yeah, so I just thought that was worth correcting. Um, he doesn't

0:08:24 Thomas: Sure. Yeah. Can you explain, uh, I'm sure what this is going to keep getting more into the weedsy, but could you briefly explain the difference there?

0:08:29 Lindsey: Yeah, So in a meta-analysis the researchers compile effect sizes from a bunch of primary research articles and then do a re-analyses of the overall effects from those studies, um, but if you, if you notice, if you reread the article, all he's doing is just talking about other meta-analyses and reviewing theory and stuff, so it's, it's a review paper, um, which is fine, you know, yeah.

0:08:54 Thomas: What would a good review paper be? You know, like what would a review paper that shows something significant that's, that's an, that'S not talking, forget Psy, like this is just some other area of science. What, what is that then?

0:09:06 Lindsey: That's, that's actually a question I've been asking myself. What, I mean, no, I mean, let me qualify that, but what I was trying to think about in reading it is like, does this review paper actually differ significantly from other review papers I've read that are good if I'm taking the claims on face value. And yeah, I want to make the argument that it does. So in a good review paper by a scrupulous author, you would expect there to be a fair, comprehensive treatment of the available evidence, right? You would expect no misleading claims, no omission of details that are critical to evaluate the claims. That's what I expect. From a good review article. That's not what I saw in this article.

0:09:21 Thomas: Right. Yeah. Ah, okay.

0:09:56 Lindsey: Yeah. So I'm going to talk about that first. And then in order to structure the discussion of the evidence, I want to organize it around two main questions. So the first is, why is it that the mainstream view is that most Psy findings are due to random chance? And then the second question is going to be, if we were to grant that some of the findings in some of the Psy studies are not chance, that there are real effects, have alternative explanations to Psy been sufficiently ruled out? So that I think is a comprehensive way to address things.

0:10:16 Thomas: Okay. Sounds good.

0:10:39 Lindsey: Alright, cool. So yeah, these are a couple of things that stood out to me about the review article and sort of, you know, made me think it was worth digging into the accuracy of some of his claims. One is that if you read through it, might have had the experience of reading certain claims and feeling like, I don't completely understand the claim that's being made here, or I don't completely understand the point. And if you don't have a background in the stuff, you MIGHT have attributed that to your lack of expertise. Does that resonate with you at all?

0:11:06 Thomas: Lack of expertise certainly resonates with me. Yes. Go ahead.

0:11:09 Lindsey: Yeah. So to anybody else who had that experience, I'd like to say that there were were certain points on which I had the experience of uncertainty, and because I I do have expertise in these things, that makes me think that there's... there's a rhetorical strategy going on here to try to make certain claims seem more complex or impressive than they actually are. Yeah.

0:11:32 Thomas: Oh, interesting. So it's like when I try to read some case law on opening arguments, there's a lot of stuff in there that I'm just like, okay, well, I'll assume the Andrews in the room know what that thing means, but I don't necessarily mean, but you're saying you are an Andrew in this case, and it still made no sense to you.

0:11:34 Lindsey: Yeah. Exactly.

0:11:50 Thomas: Okay. Okay, I

0:11:50 Lindsey: Right. Um. And I mean, you know, there there's some areas where this is more true than others, but but like it's a red flag to me. It's a strategy that pseudoscientists like to use just to make their claims seem more impressive. There was also at least a couple of places in which there was kind of a bait and switch in terms of the evidence that was presented. So as an example, on page 667, if you want to follow along, this is right under the heading of anomalous cognition. And in the beginning of this section, he's describing the difference between two broad sets of paradigms, free response paradigmes and forced choice paradigms. So I'm going to read just a sentence or two where he describes generally what a free response paradigm is before he starts describing specific studies.

0:12:41 Lindsey: So he says, "In free response studies, the target that mass raters will evaluate is not part of a set known to them, such as a deck of cards, but of a large or undetermined data set, such as a photo or film clip from an unknown large or open set." Okay, so this description sets up a specific expectation for me, um, and one that if it were true would actually be pretty impressive. So let me ask you, from that description, like, what what kind of outcomes do you expect are being studied in a free response study?

0:13:17 Thomas: Language, but it sounds like predict something, you know, tell me what I'm looking at or something like that where you have no idea even. It's not like pick a card, any card, it's like what am I holding or something, you know, something that's way more open. Okay.

0:13:28 Lindsey: Yeah. Yeah. That is exactly my expectation, right? Now, if anybody goes back and reads the section, they're going to notice something very curious, which is that not one of the paradigms that are described under this section actually have that kind of outcome. Yeah. So instead, like Gansfeld is included under this and a number of other paradigms. But what you're going to notice is that all of the outcomes, right? So I think where free response comes from is that people have this period where, you know, they're visualizing or they're experiencing these hallucinations and reporting on what they see. But then the actual outcome that they're evaluating for accuracy is their choice out of like four possible choices.

0:13:35 Thomas: Okay. Hmm.

0:14:14 Lindsey: And what they're supposed to do is choose the target based on the things they reported in the visualization. But, like, nonetheless, it's not like the chances are a billion to one like they would be if there was no choices presented, right? It's

0:14:32 Thomas: Like no constraints of any kind just free. Yeah, yeah.

0:14:33 Lindsey: Exactly, exactly. So anyway, like having that expectation set up and then having it followed up with, again, multiple choice outcomes felt like a bait and switch to me. And that could be rhetorically useful, um, if you're not paying attention or you don't know how to pay attention to those kinds of details. So, yeah, um, one one other thing there

0:14:35 Thomas: True. Okay, yeah.

0:14:55 Thomas: That's definitely concerning Do you, so that was a great example, and are you, are there a lot of these, or like is just this one example out of many that you found, or, you know, is this the only one? Okay.

0:15:06 Lindsey: Are a lot of examples of places where he says something that seems to imply one thing, and when you look into it, it's actually another, right? I can give you another one right now. So So, like, all right. Okay, so this is on page 668. He's describing a study that reanalyzed the Storm meta-analysis, which he talks about a lot in this paper. And so let me read a little bit from this. So he says, "Router and colleagues conducted a Bayesian probability analysis of the newer Storm and colleagues' dataset, excluding studies that had used manual instead of computerized randomization." Okay, let me stop there for a second. He didn't exclude studies that had used manual randomization. He did a comparison on the basis of the type of randomization. But yeah, so he goes on to describe the fact that in this reanalysis, Router and colleagues found way more attenuated evidence for psi than the original meta-analysis had, um, and then he ends this passage with the following statement.

0:15:15 Thomas: Gotcha. Sure. Sure.

0:16:09 Lindsey: He says, Storm and colleagues conducted a Bayesian analysis not excluding the studies that had used random tables, as Rauter and colleagues had done, and concluded that the results did support a psi effect. So here's what we're supposed to take away from this. These skeptics came along. They reanalyzed this data with a fancy statistical technique. They excluded some studies for reasons that are not explained to us, and so maybe they're sketchy. But then when Storm came came back and responded to it using the same fancy analysis, but not excluding these studies, they found evidence for Psy, and thus there's evidence for psi, right?

0:16:46 Thomas: Yeah.

0:16:48 Lindsey: I mean, but what we're supposed to take away from it is that, like, we have responded to the skeptics, and even using their same fancy techniques, we find evidence for psi as long as, you know, without the bias against psi, right? So here's a here's the critical details that were left out here that I find highly misleading, right, in their omission. So let's talk about this comparison they did with the types of randomization. They went into the studies in the Storm meta-analysis, and they looked at the methodologies, and they noticed that some studies provided good comprehensive descriptions of the randomization procedures, and they seemed like good randomization procedures. Others either didn't describe how they randomized, which is problematic because for reasons that we'll get into, randomization of target presentation is so crucial in these size studies, right?

But they either didn't describe how they randomized, or they described how they randomize and there were details that were very suspicious. So, for example, some researchers created a random sequence for target presentation. And what you'll notice if you generate a random sequence of any kind is that you'll get these weird sequences of repeating numbers, right? So if you're like determining left-right positions or coin flips or whatever, you'll get strings of like six tails in a row. And that seems weird, but that's what you get in random sequences, right. What some researchers did was generate these random sequences, is, but anywhere where they saw these long strings of repeating values, they just snip them out.

0:17:00 Thomas: Okay. Right.

0:18:27 Thomas: Like, it's that thing where if you're trying to play rock, paper, scissors, humans actually suck at it because we're not good at actual randomness, and so it sounds like these researchers are manually, like, making a random sequence more more humanly random, like more, more visually random to humans, but that's not actually truly random.

0:18:36 Lindsey: Yeah. Yes, exactly. Right. And that's meaningful for a very specific reason that we'll tackle. But suffice it to say, these researchers, Rauter and colleagues, split the studies into whether they used good randomization or whether they didn't provide evidence that they used good randomization.

0:18:51 Thomas: Okay.

0:19:09 Thomas: And so these people you're talking about are the, are the fact checky kind of, this is when the skeptics were kind of going over it. Gotcha.

0:19:09 Lindsey: Right, yeah. Yes. Yeah, and so what they found when they did this comparison is that in the properly randomized studies, the accuracy rates clustered around chance. In the ones where proper randomization had not been demonstrated, though, you got a pattern of accuracy that was somewhat above chance. It seems like it seems relevant to me. So the fact that the fact that details like that are excluded here make me suspicious, right, this does not seem like a, like an honest or comprehensive reflection of what's going on in this literature. So yeah, so this, this has my, my spidey senses tingling about whether this is pseudoscientific. I... And so let's start with this first question of why it is the mainstream view that most psi findings can be attributed to chance, okay?

And the first thing that I want to reiterate here is that in these meta-analyses that are purporting to show psi effects, thing that's fundamental to being able to evaluate these is some assurance that there's not like a selective presentation of effect sizes. You want to make sure that the effect sizes that you're analyzing are representative literature, right? And a lot of these big meta-analysis that have been published and seem to support Psi have been criticized on exactly that basis,

0:19:35 Thomas: I see. Very interesting. That seems like pretty obviously what's happening here. Yeah. Right.

0:20:45 Lindsey: Right, for either, you know, not sufficiently accounting for methodological rigor, or for seeming to selectively exclude effects that don't align with the psi hypothesis. Like for instance, that Rauter paper that we just talked about noticed, when they looked into the studies that STORM had, selectively just dropped certain effects out from studies that they otherwise included that didn't seem to support psi. So right off the bat, we have a sense that these meta-analyses might be biased representations of the effects in the, the literature. Okay, and one other thing that I want to get to before some other stuff is one particularly compelling claim that Cardinia made was that methodological rigor is actually positively correlated with effect sizes. Do you remember that?

0:21:11 Thomas: Okay.

0:21:33 Thomas: Oh, wow. That doesn't seem to be the case from what you've already described, but how does he support that.

0:21:39 Lindsey: Yeah, well, I will tell you. So, so Onerton and Ferrari, this was, he cites this on page 670 as a meta-analysis that found evidence for this correlation. And let me tell you a little bit about how Onerton and Ferrari did this. So they had all these studies in the meta-analysis, and they came up with eight dimensions of methodological rigor that they then coded the studies on. So each dimension that the study satisfied, they got a point. And so you could get scores between zero, for not meeting any of them, to eight, for meeting all of them. Um, and some of the dimensions were pre-specifying the sample sizes that were going to be used. This prevents, um, like a strategy of just collecting data until you get a significant effect and then stopping, which is obviously problematic. Um, another one was pre-planned analyses, so saying up front, I'm gonna do a t-test and nothing else, and that helps to prevent fishing.

0:22:39 Thomas: Sorry.

0:22:40 Lindsey: Randomization, huh?

0:22:40 Thomas: Why am I supposed to know what a T test is? Did I miss?

0:22:43 Lindsey: Nah, just a statistical test. But another dimension was randomization method. They were kind of vague about how they made the distinction between who got the point here and who didn't. But nonetheless, this was a dimension that they coded for. And then they coded for things like, you know, checking to make sure the the randomness worked, that you know, the presentation was truly random, proper control conditions, recording things in a way to minimize error, and then double checking to make sure that there were no errors, so you got two points potentially for doing that, um, and then finally, like, automating the process of calculating your accuracy variable got a point, and if you double checked that to make sure that there were errors in your calculation, you got another point. Okay, how do you feel so far?

0:22:44 Thomas: Okay. Okay.

0:23:32 Thomas: Don't know. It depends on how they applied this. It seems like not that unreasonable of a scoring system, but I don't have experience enough to know for sure if it is.

0:23:41 Lindsey: Well, Let me say this, I mean, they're not unreasonable dimensions, um, but I would say that there are some of these things that if a study is not doing them, that's that's just problematic.

0:23:52 Thomas: Oh, gotcha.

0:23:52 Lindsey: Like they shouldn't get

0:23:53 Thomas: It shouldn't be like, oh, dinged you one point. It should be like this shouldn't even be happening. Happening.

0:23:54 Lindsey: Yeah. Yeah. Exactly. Right, so like automating your calculation of accuracy. Like who the hell is doing that manually? Nobody does that. So anyway, so they gave the studies point per category. We've got scores between zero and eight. And then what they did was correlate this sum with the effect size that the study produced, right? Here's the first problem. In order to, one assumption that you have when you run a correlation is that the two variables that you're looking at are linearly related. And on a conceptual level, there's reasons to think that that's not appropriate here. In other words, like, I don't see study quality increasing by a similar amount for each point you get on their scale. So that's the first problem. But here's the second problem. They don't tell us like,

0:23:58 Thomas: Okay. Sure. Right. Let's

0:24:53 Lindsey: How many studies did specifically, like, pre-specified sample sizes or pre-planned analyses. All they do is give us a summary of the rigor mean. Do you want to guess what the mean was on that scale of zero to eight? Well, that's fair too, yeah.

0:25:07 Thomas: Well, again, I don't know how well they're applying the scoring system, but was it, was it lower than you'd want or was it too high? I don't know.

0:25:14 Lindsey: That's a good guess. Oh, the rigor score was about three, meaning that the average study in this, in this particular meta-analysis, neglected to do five out of these eight things, and when you really look at them, that's potentially a huge problem, depending, like, which five they weren't doing. Right. So Yeah, I mean, like, when I read the claim that that rigor was associated with effect size, that's not what I expected to back it up. Um, so we've got reason to be suspicious of that. Okay, let's also revisit um this idea of the file drawer effect, which Cardinia says has been dispensed with. I think this came up on the last episode.

0:25:36 Thomas: Yeah. I see.

0:25:57 Lindsey: So just briefly, the file drawer effect is the idea that there might be an over-representation of significant findings that are published versus those that are found, to people's reluctance to publish null findings, right? So Cardinia addresses this by saying that in order to dismiss the existing significant findings as due to chance, you would need hundreds and hundreds of null findings that are not published in order to offset them. This is a misunderstanding of the critique and it does not address it.

0:25:56 Thomas: Yeah, Hmm.

0:26:26 Lindsey: So this is a this is an in the weeds part. This a this an Andrew style deep dive.

0:26:31 Thomas: Buckle up. Yeah, here we go. Do I need to get out out my graphing calculator

0:26:35 Lindsey: Okay I'm gonna try my best to break it down but please interrupt me if I do a bad job. So when you conduct a study, the probability that you're going to get a statistically significant result is not determined solely by whether your effect is real, right? It's determined by a lot of other factors. So even if you're studying something that absolutely actually exists out in the world, there is a chance that you'll fail to detect it, right. So the probability of being able to detect a real effect in any given in study is called statistical power. And power is affected by a number of study characteristics. So for example, as you increase the size of your sample, power goes up, more likely to detect the effect if it's actually there. As the amount of like random variability in your outcome goes up your power goes down. Because if you've got a super noisy outcome variable it's harder to see that signal of the true effect through the noise, right?

0:26:39 Thomas: Okay okay right

0:27:41 Lindsey: And also, the size of the effect that you're trying to detect also affects power. So if you're looking at effects that is really large and produces a really large absolute difference, then it's much easier to detect compared to if you are looking at small effect sizes. Okay? So, good so far? Are. Okay. So for instance, let's say that you're testing the effect of Tylenol on pain. And let's say that the true difference in pain ratings between people who take Tylenols versus a placebo is like two full standard deviations different, right? That would be a really large effect.

If you also have an adequate sample size and good methodological control, which cuts down on that error variance, then you can anticipate detecting an effect a lot of the time, like not a hundred percent of the time, but a lot of the time, right? So then if somebody was interested in determining whether there was a reporting bias or a file-drawer effect regarding the effect of Tylenol on pain, here's what they would do.

They would collect the studies for their meta-analysis, they would look at the statistical power of the studies that have been conducted, and they would calculate, which we can do fairly precisely, they will calculate how often you would expect to find a real effect.

0:27:59 Thomas: I

0:28:59 Lindsey: I mean, how often you would expect to found significant effects versus null effects for an effect of that size. Does that make sense?

0:29:09 Thomas: Think so, okay.

0:29:10 Lindsey: Okay. So would compare your expected proportion of significant results to the actual observed proportion of significant results. And if the two don't match, if the number that you're observing exceeds the number that you expect, then that's evidence that there's an over reporting of significant effects in the literature.

0:29:32 Thomas: Okay. I think I see what you're saying, so the file drawer thing is you would expect that a bunch of studies just weren't reported on that didn't show results or something, okay?

0:29:32 Lindsey: Okay. Cool. All right, so Exactly. Right. Um, so the reason that psi has been criticized is that some people have done this analysis, and given the power in the studies, given the small effect sizes of psi effects, the expected proportion of significant findings versus null findings is much lower than what's being reported. And that's evidence that there's a lot of null findings in psi research that we're not seeing.

0:29:44 Thomas: Yeah.

0:30:11 Thomas: I think Aaron said something like maybe the guy in the paper said 400 was a number he threw out there for how many would need to be, and it's like part part of me was, was like, is that even that many? I don't know. There could be 400 studies that people just didn't even bother to publish because they didn't show anything.

0:30:17 Lindsey: Yeah, I Don't think that, I'm not sure exactly what like the number is. I could actually dig that up. They gave a specific proportion that they would expect in the study that I was looking at, um, but the problem that I had with that response is that it wasn't to this critique, it's, it's like, okay, so the, the reason that this is a problem is that in order for a meta-analysis to be interpretable, you have to make sure that there's representativeness in literature, right? Essentially what he's doing is he's taking, he's taken the significant effects in a biased sample and using that to extrapolate how many null results you'd need, so it's like the logic of what he's doing with that number is backwards.

0:30:37 Thomas: Hmm. Oh, it's

0:31:09 Lindsey: Okay.

0:31:09 Thomas: Backwards. Yeah. Okay.

0:31:10 Lindsey: Okay, so that's, that's the first strike. Um, Cardinia claims that this was dispensed with, it wasn't, the thing that he said was not a response to it, and believe me, if somebody had a response of to this criticism he would have have mentioned it because it's huge, so the, the bottom line here is that you can infer virtually nothing from these meta-analyses because all of the thing, all of the null findings have been excluded, well not all of them, but a significant proportion of the null findings has been excluded, it's like the same thing as doing an experiment and excluding any participants that don't align line with your hypotheses. Yeah.

0:31:23 Thomas: Hmm, okay. Right.

0:31:46 Lindsey: So another criticism that Cardinia says was dealt with is this issue of p-hacking, um, and one thing that I, I want to lead off with is that p-hacking is kind of a pejorative term, and it sort of implies that researchers are doing uh like sketchy stuff with evidence. Like, you know, intentionally going into a data set with no preconceived hypotheses and just poking around until they find significance and then lying about it in a publication. That's what p-hacking brings to mind, right?

0:32:15 Thomas: So it's like the worst version of p-hacking.

0:32:18 Lindsey: Yeah, right. But there's a much more sort of subtle and pernicious version of this that is likely a lot more common. So if you imagine that you're You're a researcher and, you know, you have an idea of two things that ought to be related. You design a study and you're not sure, like, how best to measure your outcomes. You include, like three different measures of it. And then when it comes time to analyze, like again, you have an ideal of the conceptual relationship you think should be there. But maybe you don't have, like a specific analysis that should demonstrate that. So you go into your data, you do one analysis. It's not significant. And you're like, ah, stupid. I did the wrong analysis. And so you try another one that should mean the same thing, but maybe it's a better analysis and you don't get it again.

0:32:19 Thomas: Okay.

0:33:06 Lindsey: And you just keep doing that until you find the significant one. And then you're like, oh yeah, that was definitely the correct way to do the analysis. I should have seen that from the beginning. So even if you're trying to be truthful, things like that can still operate and it can be fairly easy to lie to yourself about whether that's acceptable.

0:33:25 Lindsey: I'm not calling researchers who do Psy research dishonest when I say that there might be p-hacking operating here.

0:33:25 Thomas: Okay, You're saying this, that kind of, um, I don't know what to call it, I guess blind spot a little bit, or that kind of a little bit of self-deception that could be present in all kinds of areas, not just Psy research.

0:33:44 Lindsey: Oh, for sure. I mean, actually, that's a nice segue, because the paper that I want to talk about as an example of this is Bem's paper. And one it's significant and that people still talk about it is that it um was one of the big catalysts for these broader discussions in psychology, but you know, social psychology in particular, about whether our standards are appropriate. Um, so yeah, like p-hacking happens across disciplines, um, and it's it's something that at least in psychology we've been really actively talking about and trying to correct, especially over the past, you know, few few years. So,

0:33:45 Thomas: Okay.

0:34:24 Thomas: So in part because of that paper you talked about that was however many years ago.

0:34:29 Lindsey: Yeah, 2011.

0:34:30 Thomas: Interesting. Okay, okay.

0:34:31 Lindsey: Yeah, so credit where credit's due, right. I mean, yeah, so so Bem's paper, this is this is a paper detailing 10 experiments, and nine out of the 10 experiments uh have significant effects supporting psi, right? So right off the bat, you know, for the the power reasons that we talked about a minute ago, um, there's reason to be suspicious of this, because given the power in Bem studies, uh, some people have estimated that even if, again, this is a real effect, he should have only gotten like six out of ten. Um,

0:35:02 Thomas: Where where do they get the The effect size or the power that they're using to calculate that, like, is that based on something he said? Like, you know, like if we're just researching something that, frankly, isn't real, how how do we have like a knowledge of what the power of the effect is? You know.

0:35:24 Lindsey: So what you're assuming is is that the effect sizes that you actually observe in the the studies are estimating the true effect size. So you're sort of like assuming that from the beginning. If that's the case, like say you have an effect size of 0.2, that tells you something meaningful about how often you should detect it in any given study.

0:35:44 Thomas: OK, so you're you're sort of when you're doing this analysis or when whoever's doing it, you're kind of granting for the sake of argument. Like, okay, let's say you did detect something here, and then kind of using that to go re-evaluate on the whole how many of these studies you should expect to see a result in.

0:35:53 Lindsey: Yes. Yes, that's Precisely it. Yeah. Okay, so right off the bat, it like, there's something to suggest that there there might be some selective reporting. Either Bem left out some experiments, or maybe he did more analyses per study than he reported, or something, right? And when you look at the actual effects that are reported across studies, there are some sort of weird inconsistencies. So, for example, there're individual differences or personality traits that appear and disappear, appear as things that predict accuracy. Extroversion shows up sometimes and not other times, and it's not clear whether Bem is just measuring different things across these studies or whether he's just neglecting to report them in Some studies. So that's one thing.

But another thing is, like, if you look at some of the specific hypotheses that he says he created, or that he set forth before analyzing the data, some of them don't make sense. So, for example, let's look at experiment one in this study. So in this study, participants are presented with these images of two doors on a screen, left and right, and they're trying to guess the location of a target picture. And on different targets, I mean, on different trials, pictures have different types of imagery in them, so on some of them they're like neutral, on some of them their romantic, and on some of them there porn, which is fun, so he's got

0:36:03 Thomas: Okay. Hmm.

0:37:35 Lindsey: These erotic images, and he says that he hypothesized prior to data analysis that the accuracy on detecting the erotic pictures would be better than chance, but that on all the non-erotic pictures it would be at chance. Does anything stand out to you as strange about that prediction? Okay.

0:37:54 Thomas: Mean, yes, in that I don't believe that would be the case. But other than that, no, I don'T think I'm picking up what you're hinting at.

0:38:02 Lindsey: So not all size studies use erotic pictures, right? Um, in fact, a lot of them, I mean, I'm aware of the claim that as you increase the like emotional evocativeness of the stimulus that accuracy increases, but nonetheless there are lots of size studies that use neutral targets, right? Right, so why would He I

0:38:19 Thomas: Oh, is this in... I'm sorry. Is this in his meta-analysis, or is it... I thought we were describing one experiment, or am I...

0:38:25 Lindsey: Am, yeah.

0:38:25 Thomas: Oh, okay. I see what you're saying. Okay.

0:38:26 Lindsey: So, but I'm saying, like, why would Bem say that detection of a neutral picture would be at chance? Yeah.

0:38:35 Lindsey: What some have suggested in their criticism of this, um, is that there was these analyses were more exploratory than he was admitting to, that he had a general, general sense that, you know, maybe, know, the erotic pictures would have higher accuracy or something, but not necessarily this specific hypothesis, and that instead he went back and made his claim more specific after he looked at the data and saw the effect. And by the way, he did find that allegedly hypothesized effect. So 53% of the time people were accurate in identifying the location of the porn, and all of the the other categories of stimuli were right at chance. Yeah.

0:39:00 Thomas: Okay.

0:39:17 Thomas: So you're thinking he, or people, have criticized it as taking that result and then going back and kind of reverse engineering a prediction out of it?

0:39:27 Lindsey: Right. And I mean, honestly, that that is no different from the the hype, the hypothetical scenario that I described at the beginning. Just, you know, you know that there's an effect and it has something to do with these porn pictures, and you do the analyses until you find something consistent with that. Okay, so what I hope I've provided evidence for so far is that this body of evidence is a lot less solid and persuasive than it seemed to be after reading this review article.

0:39:27 Thomas: Okay. Right.

0:39:57 Lindsey: There's not a lot to infer from the meta-analyses. These criticisms about file drawer effects and p-hacking have not been dispensed with. A poor set of evidence. And there's every reason to to suggest that this is probably no different from a random effect, right.

0:40:12 Thomas: When you say this, are you talking about about Bem just then? So I want to make sure which thing are you taking about?

0:40:18 Lindsey: Right. No, in general, the evidence that's been prevented, presented for psi, um, seems to be no different than chance. Yeah.

0:40:25 Thomas: Okay. Okay. As on a whole. Gotcha.

0:40:28 Lindsey: Okay. My contention is that most of the time when a size study reports a significant effect, you can be suspicious that nah, it's just it's normal deviations from 50 or 25 or whatever it is, but it is possible that in some subset they're picking up on some type of real effect, and so the question is, is it psi, or do we have some sort of alternative explanation that's simpler? And to talk about this, I want to, I wantto dig into a few relationships between effect sizes and study characteristics that proponents bring up a lot as being valid and interesting relationships. So for one thing, participants are given trial by trial feedback, the effect sizes tend to be larger for those studies. And as it turns out, that's a pretty common practice in psi research. Um,

0:40:28 Thomas: Okay.

0:41:21 Thomas: So like, rather than just guessing them all and having no idea how you did, they're telling you if you got it right or wrong after each one. Okay.

0:41:28 Lindsey: Yeah. Right, um, so you make the prediction, but then you actually get to see what the, what the right answer was. Yeah, so so when participants are given that kind of feedback, you get much higher accuracy percentages than you you see in studies that don't give feedback. Okay, um, and proponents of psi interpret this by saying that participants need to have their psychic intuitions verified in order to pay attention to them, um, so I guess that's one interpretation, um, but hold that relationship in your mind, and and let's return to the randomization issue, okay?

0:42:03 Lindsey: Think about that researcher that generated the random sequence and then snipped out the repeating targets in that sequence, right? So then they have this modified random sequence that sort of aligns with people's intuitions more about where targets should appear given a randomized sequence, right? So so you're a participant, and um, you know, you're getting this trial by trial feedback, and let's, to simplify things, let's say that it's like a a binary choice, so you're predicting coin flips, and the first flip is uh heads, and then you get a tails, and then you get a heads, and a heads, and heads, and heads, right, what is the naive intuition about what the next flip is going going to be?

0:41:53 Thomas: Right. Okay, well, Would be it's got to be you're due for a tails. Yeah.

0:42:52 Lindsey: Due for a tails, absolutely. Okay.

0:42:54 Thomas: Gamblers fallacy or whatever.

0:42:54 Lindsey: Yeah. Yeah, so you're due for a tails, and so, like, if I've just seen like four heads in a row, I'm definitely guessing tails on the next one, right? Because I don't understand probability, and I don't understand independent events, right? Okay. So let's go back to this modification of the randomization. What these researchers have have essentially done is made their sequence more aligned with that participant's expectation. And so when they shift to a tails response on that fourth or fifth trial, they are actually more likely to be right.

0:43:06 Thomas: Yep. Right. Yeah.

0:43:31 Lindsey: When you do this, you make the different trials non-independent, which means that knowing what happened on subsequent trials gives participants some actual information on what's going to happen on the next trial, and if they're paying attention, it's, it's likely that this would increase their accuracy, not by a lot, but maybe by enough to create this like three to five percent increase in accuracy.

0:43:56 Thomas: Definitely, it sounds like it's very important to computer randomized stuff, yeah.

0:44:01 Lindsey: Yeah, um, and it's like, it's not that psi researchers don't know this, but, you know, in looking at these meta-analyses, it does seem like there, there are a lot of studies in which certain aspects of this are neglected. So yeah, um, I actually think that this is kind of an interesting way to interpret some of these effects because, I mean, it is a very subtle source of information. Like it's, it would be pretty impressive to think that this is accounting for, for a consistent effect. And like, who knows if it is, but, um. But That would, that would be interesting. And it also, it's possible that there's something meaningful about some of the, um, the individual differences that have been correlated with accuracy. So if you think about like the fact that people believe in psi are more likely to be accurate, that shouldn't matter, right? Your belief in psi shouldn't mater in terms of how accurate you are, unless there's actually some subtle information there to be picked up on if somebody's motivated to pick up on it.

0:44:33 Thomas: Yeah.

0:45:03 Thomas: Well, how would that help? You pick up on it, is this still the same random thing you're talking about, or, okay.

0:45:04 Lindsey: Yeah. Right, like it would only help if you've got the situation where you're getting feedback, and previous trials give you some information about later trials, um.

0:45:17 Thomas: Interesting.

0:45:18 Lindsey: So personally, I would be really interested for somebody to look at this explicitly, and as far as I know, this analysis hasn't been done, but if you, if you looked at studies that have been done, looked at the conjunction of feedback and these randomization techniques, um, I, I would not be surprised if these participant characteristics are more likely to show up in the ones where the randomization failed and participants are given feedback, yeah.

0:45:43 Thomas: Yeah. This is, this is super interesting, I, I, uh, I love that you pointed this out, and what, another interesting thing about it, I think, is that you've clarified, is this is a different way of being wrong. And I know that might sound, you know, like unimportant, but I think it is important because it gets at where the problem is. You know, it's like you can't dismiss it by saying, as a lot of people have, you know. A lot of people who have dismissed this by saying like, oh, that's just cherry pick studies, cherry pick studies, you don't like the file drawer effect or that kind of thing. That is one problem, but this is a separate one because this problem would survive that, right? Like if you

0:45:44 Lindsey: Yeah. Yeah. Yes. Well.

0:46:27 Thomas: Did the correct statistical analysis and, you know, didn't cherry pick and didn't do all that, but you ignored this problem, you might still see or you would still see this effect because it's a real effect. And I just like how you broke that down in a way that divides this up into two categories of thing.

0:46:47 Lindsey: Thanks, I try, um, yeah, so I think, I think those are the main things, um, that, that I really wanted to dig into, I think that those are probably the primary things that are operating to, to create this apparent effect, um, and so I, I hope that I've sufficiently justified my position on this. On this point I

0:47:04 Thomas: Oh, absolutely. This Is incredibly helpful. I think that, wow, time has flown by, so we don't have too much more time, and maybe it's another episode to do down the line. But I want to know where in the process this should have been caught, you know, because I stand by the fact that as a non-expert of any kind, as a non-psychologist, or even as someone who, like Jesse, does have some expertise, but just not research expertise, I would think, I would hope that I, I ought to be able to pick up this journal and read stuff and be able to trust that elementary mistakes like this are weeded out of the process. So where's this falling short, or do we want to leave this for a later episode?

0:47:50 Lindsey: Think I could be concise, um, one thing I want to say is that not all of these are elementary mistakes, the

0:47:58 Thomas: You're right. I shouldn't have phrased it like that. Sorry about that. That's a good point.

0:47:58 Lindsey: Yeah. I, I mean, I, um, and I think it's important to say that because, like, you, you could be an intelligent person and a reasonable person and look at these things, and, and the patterns here wouldn't necessarily be obvious. I'm able to present this to you right now because I've had the benefit of, you know, looking into this and reading people much smarter than me and piecing these things together. So they're really not obvious. But to the thing about publication, I mean, as far as I know, American Psychologist hasn't issued like a statement about why they published this review, you, um, but I wouldn't be surprised if it wasn't along the lines of JPSP, with them, and you know, their position was like, look, we, we don't believe in sai, or we're not endorsing that conclusion, but he seems to have done the things he needed to do, as well as any other article that we published, so it would be prejudicial of us not to publish this, yes.

0:48:16 Thomas: Right.

0:49:02 Thomas: Is interesting. It's, you know, that makes a certain amount of sense because, because while it's one thing if it's psi that we, you know, I'm guessing 99.9% of us listening are like, yeah, well, that's obviously wrong. When it's something else, some other effect that we don't have that same intuition, it's unlikely. Like if they're saying this would have passed the standards, if we're just talking about some, I don't know, some correlation between taking, as you said, taking Tylenol and some random thing that you like, be able. A statement that you would believe if someone told you there was evidence for it. Like, oh, we found evidence that such and such correlates or whatever. If they're saying if that paper had been published with these same methods, we would have or sorry, had been presented with these Same Methods, we would have published it. So therefore, we can't not publish this one because it shows bias against psi.

0:49:22 Lindsey: Yeah.

0:49:52 Thomas: OK, but who?

0:49:51 Lindsey: Which is kind of funny Because that goes against Cardinia's whole thing. He's like, this field is so biased against psi that they won't, you know, let the evidence see the light of day. I'm sorry, I don't, I know, yeah.

0:49:59 Thomas: Hmm. Interesting. But then I still would wonder what needs to be, what needs to be corrected then. Because I feel like this, the current state of this right now where this paper and any other paper that, as we were just saying, might not be so obviously wrong, but nonetheless would be incorrect. I don't feel like they should be published, right? Or, I mean, is this just a tradeoff of the system of publishing, right,

0:50:31 Lindsey: I think both are true. I mean, if your standards are too rigorous, you're going to publish almost nothing. If your standards aren't too rigorous then yeah, everything gets published and we don't know what's true. So in terms of what you do about this, I can speak at least to what's happening in social psychology and has been for the last few years, which is that more and more journals are starting to value pre-registration in the process of submitting things. So this can mean different things. But one thing that some journals have started doing is researchers submit like a research proposal and then the journal evaluates that quality, gives feedback on changes and stuff to make it the best it can be.

0:51:18 Lindsey: And then if they want to, they can, they can accept it right then, before the data has been collected, before the analyses are done, um, and then, like, it's, it's approved, you get it published regardless of what you find, and that really cuts down on these issues of, you know, researchers feeling like, oh my god, if I don't publish something significant, I'm not going to get tenure, and and all the other things that that contribute to these problems, right, um.

0:51:44 Thomas: And the fact that, like, you're just not going to publish a study if it doesn't show interesting results? Like, does this kind of help cut down on that file drawer effect a little bit potentially? Okay. Yeah, that sounds really smart. I like that.

0:51:46 Lindsey: So Like Yeah. Yes. I Love it, I'm, I'm so excited that the field's moving in that direction, I Think it's great.

0:52:01 Thomas: Well, that's good. I regroup and do a later episode about more of this, you know, because I have a lot more I would want to talk about. Like, how can we, you Know, walk the line between a reasonable breakdown as you've done, you you know, but that requires a ton of expertise, you know. Walk The line between, okay, this result obviously looks not right, but then there's the other side of it, which is a trend among the kind of, I don't know what you would call them, but the kind of center-right faux liberal people who are, or like Jordan Peterson types, who are like, all of humanities and social science is bullshit, shit, kind of thing, like, how do we how do we walk that line in a way that's consistent and and that kind of thing, but I think we should probably leave that for a later date, what do you think?

0:52:02 Lindsey: Yeah. Yeah. Right Right Yeah. Yeah, I think that's reasonable, that'd Be A long discussion, yeah.

0:52:53 Thomas: All Right Yeah But maybe, but we should have it, maybe we can, maybe we can get together, maybe we could work on something like that, because it is, that does strike me as one of the problems I had with the criticism that we got was, if what you're saying is true, then how, how do we trust any of this science, you know, like I, I would be interested to, to know, but okay. Uh, Lindsey, thank you so much for the breakdown. That was incredibly educational. Uh, and I,I really, really appreciate you taking the time to do that.

0:53:13 Lindsey: Yeah.

0:53:24 Lindsey: For sure, this was super fun, as Always

0:53:25 Thomas: Yeah. And is, was there anything we could point people to, or are you not an online-y person much?

0:53:32 Lindsey: I don't know if you, if you Google my name, and in, uh, Google Scholar, some stuff comes Up I guess If you wanna, I Hope

0:53:37 Thomas: There we go. Okay. Well, I just, I tried. I just feel, I feel guilty that, uh, you've put in so much time for this. But I do really think that many people are going to find this immensely valuable because I definitely do. So thank you so much.

0:53:50 Lindsey: So I Hope so, yeah.

0:53:51 Thomas: All Thanks again to Lindsey, and I think we covered it pretty well there, but just a few remaining thoughts. I do have some regrets from that episode, I do think I could have pushed back on some of the stuff Jesse said relating to materialism and quantum mechanics, it's just hard though, because I'm not, I'm out of that, you know, a lot of people pointed out ways that the quantum mechanical explanations couldn't possibly be right. Right. And two things. One, I, you know, that's, it's hard as a non-quantum physicist to to do that in real time for me. And then secondly, also, it didn't seem to be that important to the point that was being made. And maybe you disagree, but the point Jesse and Aaron seem to me making was that this person is kind of damned if he does and damned if she doesn't, like if he doesn't offer some sort of mechanism, nobody will take it seriously.

0:55:00 Thomas: Obviously, uh, and if he does offer some sort of mechanism, people say he's wandering outside of his field, so I, I kind of understood that point, but what I want to point out, I guess a little bit defensively, uh, about this recording with Lindsey, is that in the beginning she said if we were able to take this person at their word for what they, you know, the work they did, and and take their math at their work, then this would be very interesting, like this would, she said that at the beginning, this would be an interesting result, kind of like how Jesse and Aaron said.

And I just want to point out that that is the real problem here. I hope we explore it further in the future, but the problem there is, and the reason why I was fine with airing that first episode was, I think Jesse and Aaron were philosophically justified as non-experts in that specific field to take the journal at its word that they've checked these numbers, even though I was enormously, enormously skeptical of these results, and that was interesting to me, but now having gone through this process, I'm really glad that we did, and I've learned a ton, I hope you have too, and I hope people who are very anti-psi aren't too upset over that first episode where we were entertaining the possibility. But I don't see a real harm in entertaining the probability that a journal did its job, you know, and there are interesting results here.

0:56:32 Thomas: So I think we've gotten pretty well to the bottom of this for the most part. And I enjoyed the journey, to be honest with you. I hope you did, too. So with that said, I will have to see you guys on Thursday for that voicemail episode. Please, still time to call in if you'd like to leave a message about all this or anything else from the month to get your voice heard in this month's episode. So feel free to take that opportunity. The number is in the show description. Thanks for listening. I'll see you very soon.