WTW45: The Brophy Study Is Wrong In Nearly Every Possible Way

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:03 Clip: What's so scary about the woke mob, how often you just don't see them coming.

0:00:11 Clip: Anywhere you see diversity, equity, and inclusion, you see Marxism, and you see woke principles being pushed.

0:00:19 Clip: Wokeness is a virus more dangerous than any pandemic hands down. The woke monster is here, and it's coming for everything, everything, everything, and everything. Instead of go-go boots, the seductress green M&M will now wear sneakers.

0:00:41 Thomas: And welcome to Where There's Woke. This is episode 45. I'm your host Thomas Smith. That over there is Lydia Smith. How you doing?

0:00:49 Lydia: I'm doing pretty well and somehow surviving the craziness of all of this.

0:00:55 Thomas: We are two members of the three-member panel of the master's thesis review. What do you call it?

0:01:02 Lydia: Yeah, panel? Master's panel?

0:01:07 Thomas: Panel.

0:01:09 Lydia: It's been a while, okay?

0:01:11 Thomas: I don't think you actually have your master's. Have you been making it up all this time? It's just called the three, the three, three panel, panel, panel, panel, threes a little three-person panel

0:01:16 Lydia: Yeah, I have to prove it. That's why we had to call in an expert it's

0:01:25 Thomas: Yeah yeah so we're going to be speaking with Dr. Jenessa Seymour who is an expert in exactly what we need for this and oh my god this study is so fucking bad it is so bad and I just can't get enough this drove us insane and I can't get enough of proving how bad at science Jordan Peterson is um because he not only Lee, again, was in charge of this. He also boasts about it. There's so much, though, that we are saving extra for patrons only. Believe it or not, this was me doing my absolute best to limit how much we're talking about this fucking master's thesis. But it's so good. There is so much. When something is so terrible in a complicated way, it takes so much time. So, putting some of it on patreon.com/wherethereswoke for those who are extra keen on scientific dunks on Jordan Peterson. So make sure to check that out if you'd like to. But in the meantime, let's get on over to our expert and we'll hear more about why this thing sucks. I also did want to note, so we, when we first recorded with Jenessa, we didn't really know what we were going to be doing with with it, you know? And so I thought we were just going to get some background information. Like Jenessa didn't even know she was going to be like on the podcast.

0:01:39 Lydia: Crazy yeah yep oh right

0:02:42 Thomas: Um, she was so good. We were like, well, we have to use this, but I only note it because I ended up talking a whole lot. Cause what I wanted to make sure to do was like express all kind of the thoughts I'd been having on this study as a non-expert and just make sure I wasn't saying anything dumb. Um. And And so, again, it was going to be like an on background interview. And so I guess I feel a little self -conscious that like I don't normally get an expert on and then just do like a bunch of talking in front of them. But the reason I was doing is because, again. We were running stuff by her as a background interview, but it ended up being so good. She was excellent. So we decided to use it. Just a comment on that. If it seems like I'm talking too much, that is why. But don't worry. After I stop talking, Jenessa really takes us through some great stuff. So let's get on over to that. If you don't mind introducing yourself and telling us your your qualifications.

0:03:35 Jenessa: So it's Dr. Jenessa Seymour, I former neuroscientist. Um I was working in academia for a while and a very long version or there's the short version that's just like I had tenure track options in front of me that were going to require some sacrifices I had non-tenure track options that were super fun done but um would uh not have job stability and eventually I was just like this is not the life that I would like so now I work at a law office yeah

0:03:40 Thomas: What happened?

0:04:08 Lydia: Yeah, you went to law school, right?

0:04:09 Jenessa: Yeah I am taking the bar in july so yeah it's

0:04:12 Lydia: Awesome.

0:04:12 Thomas: Holy shit. Cool. Well, it's a whole separate conversation. But the point is, yeah, to our discussions today, oh,

0:04:21 Jenessa: Uh I have a phd in neuroscience bachelor's in psychology psychology, cognitive neuroscience. So really, it's, you to this kind of stuff in terms of I've taught stat for behavioral sciences for a while there, which basically, yeah, if you are a psychologist or somebody in the behavioral sciences, how do you pick your statistics? Which ones are appropriate given given the type of study you're doing. So kind of exactly this thing of, um, I have some problems.

0:04:56 Thomas: Yeah, I solicited help some from the Facebook group, and you were kind enough to message me. I'm so glad you did. First thing I want to ask you is, can you please justify to the audience that this is insane and it's not crazy for me to be obsessed with what is happening here. Cause like, I thought I was, I'm reading this. I, what happens. I started really digging into this master's thesis. I decided to, you know, take a bunch of like fucking online courses and stats and factor analysis just because I knew something made no sense about this made absolutely no sense. I came to some conclusions. I consulted you about it. I was I to get a lot of stuff right. It just took me like 20 hours, whereas it took you like three minutes. But then you dug Even deeper into it and I you can, you me for putting this nonsense into your brain because it is truly impossible to stop thinking about because it's It's just either abject lies or like some, I don't know. So I am very curious to hear what you made of this master's thesis from 2015 University of Toronto that we to talk about because it's the basis of all this misinformation. And it's something that Jordan Peterson is leaning on in public Nazi appearances, Nazi YouTube channel appearances to make,

0:06:08 Thomas: You know, just wide ranging claims about us essentially. Eventually and this study is just terrible it's absolutely terrible but here's the here's the other thing and then I'll let you go I'll Let You Go Off I started getting so critical of this study that I was like unlike I think Jordan Peterson I have a bit of humility in thinking like okay if I'm thinking something is this wrong I probably am wrong like if I'm if I're coming in here I don't have a degree in this I don't know what I'm talking You know, I'm coming and I'm looking at this thing and I, just a guy, am like, this so, it's not just like I have some critiques, like this seems so terrible. I like, I have to consult an expert because I'm not going to come out that strong as a non -expert. So what Am I missing something or is this just nuts?

0:06:54 Jenessa: It, bad, man. And I, God, I felt the same way. Like at first I was like, okay, between you and me, it's pretty bad. Cause I I feel terrible, like just publicly dunking on some random master's thesis.

0:07:08 Lydia: That's exactly what I told Thomas too. I was like, you know, I was like, man, is anyone like pulling up my master's thesis and ripping it apart?

0:07:14 Thomas: To which I said, you're not using your master's thesis for Nazi propaganda, hun. So I think you're fine.

0:07:20 Jenessa: This is the problem is as soon as I was like, okay, this fair game once it's being used for nefarious purposes and it's being used in a way that is just not relevant to what it even says. And yeah, definitely Don't blame yourself because I mean, this is my bread and butter. And I spent days looking at this going, wait, where?

0:07:42 Thomas: It's impossible. Yeah, you can't look away.

0:07:47 Jenessa: No, for real. I think I sent you the document where I had to print it and handwrite numbers on it to be like, this ist question number one. This ist question number two. Why is this one question four and five, but on this other sheet, it's 4A and 4B. They have nothing to do with each other why are they parts I don't understand that's

0:08:06 Lydia: Or option C and D, but there's no option A or B. Yeah, I

0:08:13 Jenessa: What makes the count get off and so by the end of it you've lost count and I was losing my mind um I don't think that's a coincidence I think that kind of sloppiness is appearing throughout and it's the kind of thing that shouldn't have made it out the door this looks like the first draft and I I don't know why it's on the internet track

0:08:36 Thomas: And this is where maybe Lydia can help us out because, you Lydia has a master's in psychology and I keep asking her. Reason why I think this is fair comment is like really this reflects poorly on Jordan Peterson's comprehension or teaching skills or whatever, because, what hun, would have happened if you submitted any of this garbage at any process in your getting a master's thesis? Like, wouldn't someone have told you this is garbage?

0:08:59 Lydia: Would have had so many you track? What is that?

0:09:03 Thomas: Track changes kind of thing.

0:09:03 Lydia: Track changes would have been lit up on my paper from my advisor. And you not famous. He's, he's a really, you know, great professor and I learned a lot from him, but he also had expectations. You know, like if you're about to get a degree, it needs to mean something. If you're you're about to present a written publication, a thesis of your time here, he was like, it should reflect what you've actually learned. It should reflect truth. Should uphold those expectations as a researcher, as a scientist in psychology. And especially because a of the folks in my my program were looking at applying to PhD programs so there was this additional thing of like you know like you really need to get your feet wet here and get comfortable with doing this and going to help make you a stronger candidate for PhD programs and I'm curious if there's some element here because this was like a master's to PhD ramp that she was in I like I don't know that it's entirely Jordan Peterson's fault or if it's like a little I don't Know lazy of the program in general, because if you were admitted to the master's program, as long as it was adequate, you know, your performance with your grades and everything, then you continued on to the PhD program, you don't have to apply. So I wonder if it was like less rigorous to an extent as a result. But yeah, anyway, my advisor would not have let me get away with that at any point in the process, even first

0:09:04 Jenessa: Changes well

0:10:31 Thomas: Still

0:10:31 Lydia: Draft probably.

0:10:32 Thomas: Still have a theory that she didn't get her master's because this thing sucks so bad. I don't really think that's true, but like maybe, I don't know. If it was published on the school website, does that mean she successfully defended it or whatever?

0:10:39 Lydia: I that, I don't know for sure. I mean, I imagine maybe every school is different, but I think my school, had I not successfully defended my thesis, I do not think it would have made its way into publication. Like your advisor has to sign off on it so I

0:10:55 Thomas: Yeah, that's probably true. There's more than just Jordan Peterson as an advisor. That's another thing that made me doubt myself and made me want to make sure we consulted another expert because like, how the fuck? Okay, I was prepared to come in here and do. Now, in the first episode, we talked about this. I came in pretty strong that that the PC authoritarians were conservatives and the PC egalitarians were liberal. Now that I have gone into massive depth on this pile of crap, that's not exactly right. So I do want to offer a bit of a correction, but it's also just stupider than that. Like it's worse because here's my interpretation. And fortunately we have Jenessa here to kind be my master's thesis advisor,

0:11:08 Lydia: Would yeah yeah

0:11:40 Thomas: Jenessa, so see if I pass muster here. The way she talked about it in the video, the way both of them talked about it is the PC egalitarians and the PC authoritarians were like two ends of a spectrum. She literally said like, well, on one side, we have the PC egalitarians and what they do is they blah, blah, blah. They're like, you know, a women's studies professor and they and then we have the PC authoritarians and they do all this stuff. And then the PC equalitarians justify what they're doing. And one thing Lydia kept telling me a number of times, to your credit, you're 100% right, hun, was that doesn't make sense as a way to talk about this Like you were saying that like you could classify the questions that way, but it would be weird to classify the people that way. And the problem was I kept saying, well, you know, that doesn'T exactly seem fair to me just as a lay person, because if you did some sort of polling about politics and you found two groups of people and were like, yeah, over here, over here, there's the Democrats. It seemed to me that that would be fine. And I think, correct me if I'm wrong, hon, but I think because it's been, you know, over a decade, I feel like you didn't have the words to say why I

0:12:47 Thomas: Was wrong, but I was definitely wrong. Like, but I was just trying to understand it.

0:12:51 Lydia: I think we had like a communication challenge because it has been a long time since I have lived and breathed this kind of statistics you know not not just sort of like your everyday sort of statistics. And, I was like, I can't figure out how to explain to you what I'm trying to say. And so thankfully we have, Jenessa uh, here to help.

0:13:08 Thomas: Well, well, what you actually told me first, because there's layers and layers and weeks to this fucking story, because again, I can't stop thinking about this. First, you're like, can you just watch some videos on factor analysis? And I was like, yes, I actually was going to do that anyway. It's something I want to know anyway. Like I like being able to read studies better. I want to be able to read studies. So then I went on a bit of a deep dive of learning factor analysis and watching all the courses I could. And so what I was going come and tell you fine folks is, yeah, no, it doesn't at all make sense to classify those groups and talk about those groups of people that way. And I would make a comparison. And would be like saying, okay, over here, you have the extroverts and they are so-and-so and they have these qualities. And then over here, You've got the night people and those people over there that are, that stay up late at night, they have These qualities and to which you might think like, well, couldn't you be both an extrovert and a night person, or by the way, neither an extroverted nor a night person. And I think based on my understanding of factor analysis, that should have been more how she was talking about it because these are two separate factors. One thing that it was hard for me to understand as a lay person is what you do with factor analysis is you're finding, and you can put this into better words, but you you're you're finding the hidden underlying variables that are explaining your data in a way that kind of reduces the number of variables. So if you have like 10 questions and you're able to find kind of one hidden variable that predicts it better, that'd be super cool. And maybe that variable is politics or something. And so what I didn't realize was it's you would have is you would Have someone who's high on your factor or maybe someone who was low on your Factor, but you wouldn't have two different factors be like the two different groups of people. Is that right?

0:14:50 Jenessa: That's exactly right. Like if I the politics, for example, would be a great one if I was giving questions and I was like, oh, this is weird. I'm getting a weird distribution. I wonder what this is. And I ran a factor analysis and all of a sudden I get at this bimodal distribution. And it looks like, oh, geez, people answered a bunch of like abortion and gun control questions this way or that way. And I'm like, Oh, God, it's politics, right? I'm getting the Democrats are over here and the Republicans are over there. So the factor would be politics. And you could either think of it as a spectrum of belief, or sometimes things are are so different that you could probably call it categorically Democrats over here, Republicans over there. But yeah, it's existing on a spectrum. It Might be a Nice, Normal Distribution where the tails are rare, the center is the most common. It could be a bimodal distribution like politics where people are tending to center on everybody's over here or everybody's over there and there's a couple of weirdos. Unfortunately how it tends to be. But yeah, the factor wouldn't be Democrats and Republicans. Democrats is a factor. Republicans is a factor. The factor would be political party and your score or along it would be your, your partisan ranking.

0:16:14 Thomas: Because it wouldn't make sense to have two factors and then you could be high in Democratism and high in Republicanism at the same time. That wouldn't make any sense. And that was something that, again, as a layperson, I just truly wasn't grasping. And despite Lydia's attempts, and she was absolutely right, true, she - and so here's multiple levels of why this thing sucks. There's actually tens of levels of Why This Thing Sucks. And so the challenge here is figuring out where to start. But I want maybe a place to begin here before we get a real detailed glimpse into the just absolute horrors of stats that they're doing here. The war crimes of statistics they're committing.

0:16:49 Thomas: Here's we should talk about what three of us might do if we were trying to do this study, you I think what inevitably do when they hear people talking about a study like this is they are probably thinking in their mind what it should be based on their making sense of it. You I just assumed that the way she would be doing it is doing a questionnaire that's going to test on, you know, politics and it's going to test on political correctness and then you're kind of because there would be an interesting study there. I would think and correct me if I'm wrong, I would think the way to go about this if your theory was political correctness is a thing and I want to measure it. Okay, sure, let's try to measure it. So I would think an important component to that would be a political questionnaire so because Christine Brophy believes that PC equals left. She believes, and Jordan Peterson believes going into this, you can see in her writing, she's like, PC is a product of the left essentially. And so it seems to me, not a scientist, but it seems the first thing you'd want to do is, okay, well, I better be able to distinguish between are people PC or are they just left? Like if that's the same thing, then what am I even additionally testing for? So I would think you'd like, maybe if you have have that two -factor analysis thing going. You form your questions in such a way that you hope you're getting two factors, one of which would be like political views, as we already kind of covered, and the other might be some sort of PC scale, but it would have to be kind of agnostic as to content,

0:18:15 Thomas: I would think. You'd need to test like how sensitive are you to people using words you don't like? And you would want to form a battery of questions that was vague, you know, or that was neutral. Or if you had politically loaded questions, you'd want to make sure you you had enough either way to maybe like still group people differently depending on their politics. Does that sound right? Or what would you add to that?

0:18:36 Jenessa: No, I totally agree. Was I thinking that the whole time, like I was kind of disappointed that, you study that could have been right there, there is, there are so many interesting questions hiding inside of this. And it's worth asking the question of like, like, is it helpful that, you know, we police our language in some ways and not others? And, you know I think you've had people on here to talk about that before. And like, there is research on that. Do people prefer to do it? Who is more likely to police their language in what ways? Do they feel about their own language, other people's language? I not like it when you say it, but I'm okay with me saying it? You know, those are really interesting questions. And instead we got this. I think, I honestly think like I wouldn't even do factor analysis. I'd probably just start with like, I would come up with a list of questions that like you're saying, I Would try to come up With some that are, I'm, know, I'd have to think carefully about what would be like nice and neutral and not activate anybody's political beliefs. I mean, I'd probably try to sit down with some people of different beliefs first to do a like survey type of deal. What do you, let me ask you, what do you think is like

0:19:51 Jenessa: A term you don't like to hear? What's a term that you think other people don't Like to hear, but

0:19:56 Thomas: In fairness, I think she did that, and it says she did that with like some other people in the school. Like we formed this, you battery of questions based on our perceptions of the media and what they think politically correctness, right? It was something like that.

0:20:10 Lydia: And then consulted with staff, uh, other grad students for their thoughts and opinions on that and added to it to get to the 203

0:20:17 Thomas: But boy, did they fail.

0:20:20 Jenessa: Definitely clearly does not pick up or I guess it in a couple of places it does pick that up right actually I I would argue in some places it weirdly picks it up a lot I learned more slurs for irish people in this paper things I've never heard

0:20:35 Lydia: I think

0:20:37 Thomas: Well, struggle in organizing this. Because every time we look more into it, there's more reasons it's wrong. So I don't know where to start. Because on one hand, you've got a couple of things going on. You've got the factor analysis and the fact that that makes no sense what she did. I also want to hear more about that and how you're saying you wouldn't necessarily do factor analysis because that's news to me. I don't know how any of this works. So I would have thought that you always do that. I don't know what that means. So I'd love to learn more about that from you. But we need to separate kind of the problem of also the questions she came up with suck and don't measure the things she thinks she's measuring? So should we start with, okay, what if she were measuring the things? Are you agreeing with that approach, hun?

0:21:12 Lydia: So I um but I also agree with your statement there that that her questions don't measure what she thinks they measure

0:21:20 Thomas: A hundred percent. But like are those two distinct issues that maybe we can tackle one at a time? We could first start with like, okay let's assume her questions are actually capturing And they're well worded and they're actually well formed, which side note, spoiler, they're not. They're absolutely terrible. As you say, maybe some of them are serviceable, but like they're nicht capturing the thing. But let's assume she was maybe asking okay questions. Tell us about this complicated stats stuff, this factor analysis. Like it's just so much jargon. I thought I understood some of it, but it really escaped me eventually.

0:21:54 Jenessa: Okay if we were doing this like the normal way they've got their 203 questions and like let's just pretend they were fine what I would do um I were going to to be as agnostic as possible, was going to not mess with my data. I would just run the analysis and I would let it pull out how many factors computer do you think there are? What do I put these questions into bins in the most optimal way? So I am bins where the questions within it are highly correlated with each other, and they're not highly correlated With questions in the other bins. And if that came out with 203 bins.

0:22:50 Jenessa: Yeah, that would not be good. That would mean all my questions measure completely unrelated different things. If I came out with one bin, I just asked the exact same questions over and over again.

0:22:46 Lydia: Start over

0:22:50 Thomas: Not good. Yeah. Oh, okay. Is that, or, they all correlate, meaning they could be the same question, but they also could be things that like are very highly correlated in some reasonable way.

0:23:15 Jenessa: Yeah so it's not necessarily a bad thing. It could just mean I have hit on on thing. I'm trying to think how this could have happened. Yeah, yeah. One really big construct that is hiding in this data. There's some major thing, there's probably a million factors influencing human behavior all the time, right? But we're trying to model this. Phrase I was was taught about this is all models are wrong. Some models are useful. That, always going to oversimplify it and miss something, but it could usefully tell me something about people. And maybe there's this one factor that is doing a really good job of explaining how people answered these questions. And your score on a spectrum on this factor would tell me how you tend to answer these. And then I got to look at it and look at what questions go on the high end of it and what questions Go on the low end and try to name it. What is it?

0:23:24 Lydia: Construct yeah yeah yeah

0:24:23 Thomas: My god that's such an important point that I meant to say this is another thing that is going to sound so stupid to you two experts that even think about the fact that like yeah what you're doing is you plug all this stuff and the computer you know you You just say, computer, feed me, do factors. And the computer is just running fucking numbers. It's not, the whole part where you're assigning a value and names to these factors. Obviously, this is true if you think about it for five seconds. But I think, again, to a layperson, it's not something that I immediately thought of, which is, oh yeah, she's just naming these factors what she wants to name them. There's no, she could have named it anything. She could have named it, you-handedness and right-handedness. And it would be like, obviously you would want to name it something based on the actual questions, but there's nothing in the stats that makes her pick a name that actually reflects what the factor is that she's measuring. So the raw statistical analysis can tell her hypothetically, oh, you've got these two beautiful factors. Again, hypothetically.

0:24:43 Lydia: Factor and factor

0:25:22 Thomas: Yeah. And then it wouldbe up to judgment to say, well, what are those based on the questions you could kind of. Well, and this is a good question for you, Jenessa. What's that process? Would you, let's say you did this. Would you look at the questions and kind of measure, oh, okay, so this question five is very consistently this factor over here. Question seven is very constantly this factor over here, and would you kind of divide them up and then maybe using that method, put a name to them? Is that how you would try to name them or is there some other method you would use?

0:25:22 Lydia: Two some

0:25:52 Jenessa: That's That's exactly it. And actually, that that of leads me into a perfect example here. It'll also lead me into something I'm going to come back to because so she had either 41 or 40 or 20 or 22 attitude questions. Good. I don't know. I Don't know, I don't know I tried. According to table three, there were 22 and 20 end up surviving the factor analysis, they usefully fit into a factor, right? So these attitude questions, when they forced it to go to four factors, I'll also come back to that. Of the factors that pops out is punitive justice. So when we ask you your attitudes about all these different things, oh, this is interesting. Here's four questions. So like, okay, you had to rate your agreement, right. And I'll give you the four whole questions, create this attitudes towards punitive injustice. That's your right to laugh that's also a problem um right so here are the four see if you can put this together why you might have called this punitive justice the first one was when a charge of sexual assault is brought forth the alleged perpetrator should have to prove his or her innocence

0:27:04 Thomas: It specify if you're in court or does it specify where you are? I can't remember. Oh, okay. I thought there was, wasn't

0:27:09 Jenessa: The whole question there's

0:27:12 Lydia: Was there one that was like focused on the university

0:27:14 Jenessa: There's some that are Yeah, yeah.

0:27:16 Thomas: For that one, it doesn't say like what we're even talking about. It just says, okay, okay, gotcha.

0:27:19 Jenessa: This one, all it says, when a charge of sexual assault is brought forth, the alleged perpetrator should have to prove his or her innocence. The second one that fit in that category, if a student is suspected of sexual insult, he or she should be suspended pending further investigation. Investigation number three an allegation of sexual assault is to be rejected if there is reasonable doubt about the guilt of the alleged perpetrator number four it is wrong to criticize the status and rights of women under islam because it is racist and disrespectful disrespectful of multiculturalism those four correlate highly and

0:27:55 Lydia: What

0:27:57 Thomas: Yeah. What? Wow.

0:28:03 Lydia: One of these things is not like the other wow that's crazy nuts

0:28:06 Thomas: That's not p-hacky at all. That is just totally reasonable grouping there.

0:28:11 Jenessa: That that is called punitive justice.

0:28:14 Lydia: Especially

0:28:14 Thomas: What's interesting is there's also times where she's divided it out into like sexual impropriety categories, which kind of made more sense to me. And so, yeah, got to be pretty certain about what you're testing, right? Like when you're asking it, when you'RE forming a question, You don't want it to be that someone could answer one way for like seven different reasons, right?

0:28:34 Jenessa: Yeah, I would say every once in a while, I like to leave something ambiguous, because I'm trying to think of a good example where I might want a participant to, I don't want to tell them what I think the question means. I want to leave it up to them to you tell me what it means. You know, I could imagine a scenario where that's true. I just, I don't think that happens at all in this study where I would want that to be the case. In study, you really want them to know what you meant by it.

0:28:52 Thomas: Interesting. Yeah.

0:29:08 Lydia: Not including attitudes and behaviors in the same sentence so then like you don't know right it's the compounding information within an item and even if you had an ambiguous question that was worthwhile you still wouldn't want to have compounded because then you can't interpret it at all it's you don't know what they're responding to which part of the question uh

0:29:30 Jenessa: Exactly. Like, I mean, I looked at these questions myself and I was like, well, Well, some of them are like a classic loaded question. Like, I know what you're doing here, you know what I mean? And like, it's a really common issue in social psych that you try really hard to stop participants from knowing what you're up to because they behave differently when they know what you're up to and what they know what you're looking for. And this is like a worst case scenario for that of participants are going to try to behave in a certain way. It's better because they did an online survey. You're not face to face. They're not worried about judgment. That helps a little. But people definitely knew what you were doing. They knew you were asking them about, are you one of those people? And they were going to react a certain way. Like, I'm not shocked about the high correlations they got because people were probably doing that. You know what I mean? Yes

0:30:28 Thomas: My gosh. I just realized it's even worse than what I said. I forgot that the very division and the four factor solution you're talking about was the time where the first factor was sexual propriety. And then all of your punitive justice questions, one of which is just randomly about Islam. I don't even understand the connection, but the other three are all about sexual propriety. You can't have punitive justice be the category. And then you ask, by the way, they aren't even about punishment. They're all about due process, which is, I think, a different thing. And it might be people's misunderstanding. You know, like I could see people easily misunderstanding something to say, well, yeah, you should have to prove you're innocent. And I think that what they're doing is that's a big gotcha because actually it's the burden should be on the other person to prove that you're guilty. And it's like, yeah, okay, but like, what context is this in? Are people just talking about like casually, do they know if you're talking about a court of law? And also, again, this is all, those are three due process questions related to someone accused of sexual impropriety. Why is it punitive justice? None of those talk about punishments at all. They just talk about what you should do while someone is accused of something, which is, I mean, again, maybe it's nitpicking there, but like that's not at all. And you would want to ask, I would think, a range of questions about different topics. And you could even ask some that have nothing to do. Oh, would this be a good idea? Ask some that had nothing to deal with PC, like ask them about punishment,

0:31:53 Jenessa: Perfect

0:31:53 Thomas: Just punishment in general.

0:31:55 Lydia: Think also you you're saying like yeah maybe this is nitpicking but that's the whole point I mean it it is about semantics when you're talking about naming a factor and you do have to be mindful of that and so yeah it's

0:32:10 Jenessa: Me bring you to an it's a piece that's so easy to overlook here so the first one when a charge of sexual assault is brought forth the alleged perpetrator should have to prove his or her innocence sense. Rate your level of agreement. We've And got a positive correlation here. So agreeing with that is considered you're a punitive person. Rate your levelof agreement. If a student is suspected of sexual assault, he or she should be suspended pending further investigation. Slightly lower correlation, but it's positive. So, agreeing with that, yes, you should suspend them. So punitive. The next one, an allegation of sexual assault is to be rejected if there is reasonable doubt about the guilt of the alleged perpetrator. Positive correlation again.

0:33:01 Jenessa: Agreeing with that, agreeing with that is you are for punitive justice? No. And I thought maybe are are they just absolute valuing all of these to positive but then all the egalitarian ones are negative numbers so how

0:33:01 Thomas: What? I didn't think about that.

0:33:12 Lydia: Nitpiking

0:33:15 Thomas: Yeah. God, I'm so glad because I, I, that exact thing I didn't notice, but there are other places I noticed that where I thought, again, I was like, am I an idiot? Am I dumb? Did I, am I just not understanding this? But no, like there's so many places where you're like, I'm sorry, you're saying that's a positive correlation? Yeah, Like that you're exactly right. I mean, an allegation is to be rejected if there's reasonable doubt about, I can't even form an ideological, I Was about to say, well, oh, those are all the conservative answers or the but no that's not even true like the yes answer to the perpetrator should have to prove his or her innocence you would think that oh that's a crazy lefty you just want if anyone's accused of sexual assault you want them instantly banned okay cool so you're saying hypothetically you could say that's the like hardcore lefty quote-unquote pc answer but then if the other one like you said is an allegation of sexual salt is to be rejected if there's reasonable doubt. That's the opposite of the super lefty answer. You would think, you know, you'd want that correlation to go inverse to your point. So like what I can't

0:34:21 Thomas: Put it. Yeah. Politically, it doesn't make sense. Punishment wise, it does not make sense again. None of these are about punishment. I'll point out another thing for the second question. If a student is suspected of sexual assault, he or she should be suspended pending further investigation. I could hold the opinion that, oh yeah, cause you're in a school. So sure you could suspend him, but I could also so hold the opinion that if he was found guilty, we should barely punish him. Like I could hold those two opinions like that. So you're not really testing on like punitive justice. You know, I could hold the opinon that I'm all for like restorative justice and I would want him to work with the person, but I could still believe that he should be suspended while the investigation is pending. Those are two different things.

0:34:21 Jenessa: Can they yeah yeah I mean one that was really telling to me is what they call uh what is the name they've got here egalitarian beliefs and something think something?

0:35:12 Thomas: Are yeah. A fourth factor is egalitarian beliefs and policy. Yeah. Yeah. Those are all the negative ones for some reason.

0:35:16 Jenessa: Egalitarian beliefs and policy. Well, it is because if you agree with the following statement, you are not egalitarian. It is disagreement they are looking for. Racial, ethnic, and gender quotas should exist in employment. Racial ethnic and gender quota should exist education.

0:35:36 Thomas: Wait. Yeah. What?

0:35:38 Jenessa: Government interventions should exist to equality in professions, where historically large inequality. And in academia and corporations, ethnic, racial, and gender diversity are required to have a diverse range of ideas. They would like you to disagree with those statements in order to be considered egalitarian.

0:35:56 Thomas: Wait. What. Oh, now I'm even more confused. Right.

0:36:03 Jenessa: The only reason I can think of that that's negative. I don't know why some of these numbers are negative and positive I

0:36:10 Thomas: Well, now I guess my only worry is now does that affect how she's categorized the PC egalitarians? Cause that makes even less sense. Cause those are all conserved. That's the opposite of what she's saying, obviously. She's in their mind, the egalitarian thing is like, no, you got to be equal in your opportunity. It's like, Like, she's doing the only way to stop discrimination on the basis of race is to stop discrimination onthebasisofrace. She's doing that bullshit and saying that that's more egalitarian. But then elsewhere in the study, I'm pretty sure that's not how she's talking about PC egalitarians, especially, by the way, if she's saying a women's studies professor is a PC egalitarian so that this just makes no sense.

0:36:36 Jenessa: Think I was trying to I was like oh it's got to be like a reverse scoring thing like blah blah blah but then I was, like, no because if they're saying the first one on the censorship rate your level of agreement the words of classic nursery rhymes should be changed to more inclusive lyrics and that's positively scored so, like Like, agreeing with it means you're for censorship. And this is, rate your level of agreement, racial, ethnic, and gender quotas. It's negatively scored for agreeing with it. So they think it's not egalitarian to believe that.

0:37:22 Thomas: Despite my best efforts, we've gone into how stupid these questions are. And I feel like that that is a separate thing. No so we'll put a pin in that there's so much more to talk about with these stupid questions but talk about the the stats weirdness that happened here

0:37:27 Lydia: But that's Oh

0:37:35 Jenessa: Okay. So there's a couple of different things that happen. First, we start with our, like, 203 questions. And I that's not like ridiculous on its face, except there's not 203 questions, because then if I go to the other table, there's either 184 or 191, depending on how you count it in the appendix A. And on table one, there're 200, but it's missing the uncategorized questions of of which there are seven, which would be 207. So I don't know. I don't know how many questions there are, and I don't know where they are, and I don't know what happened to them. But whatever. There's some questions. And the categories, I—okay. So these are categories they made. They looked at the questions. They thought about how they were going to write the instructions. This is not done by statistics. At all, which would be fine if this is just for organizational purposes. They made attitudes, sensitivity to offense. Thing we have to be careful about is they're going to count in the question numbers. They have two like sort of subscales of that. Sensitivity to sexist insults, sensitivity, sensitivity. Sensivity to right wing name insults. There's only two questions about it. Injustice,

0:39:04 Jenessa: Measures, which sometimes gets called punishment, offense, which is somehow different from sensitivity to offense, general PC, which has only four questions, and then there's uncategorized questions, which is seven questions, which are somehow different than general PC. I don't know. So those are the question categories they came up with. But then when they go to run the factor analysis, what I would have done is just put the questions into the damn machine and just run an analysis. You have 200 questions. See what happens. And it kind of feels like they might have done that because at one point they say they did a factor analysis and got like 10 factors out. And they did it again a slightly different way and they also once again got 10 factors out and nowhere in the 400 tables in this paper is there anything with 10 factors as the result so then they said we don't like that for reasons and

0:40:12 Thomas: Like we didn't like we got 10 factors and we did it again. We said, like, um, can you please try again? And the model said, it's still 10 factors. We're gonna ignore that though.

0:40:21 Jenessa: So I'm not quite sure what happened there. So then what happened is they took just attitudes, attitudes, which, again, was either 41 or 20 or 22 or 19 questions. And they took 22 of them. It says 22 attitude subscale items from the original version of the PC scale. I don't know what that means. And, they ran a factor analysis just on that. And it's a forced four -factor solution. They said, do not give me 10. Do not give me whatever. Give me four. I would like four.

0:40:57 Thomas: Does that change the math like that is a thing you can do right but does that change the map or does that that just say, give me your top four factors.

0:41:06 Jenessa: I guess top four is a reasonable way to think of it. Yeah, it's not necessarily not okay. Certainly, if you came in with an a priori, like I have a hypothesis about this, that they're going to be four, yeah, go for it. You just want to say, like, 10 is unwieldy and I just want to see—yeah, kind of like you're saying—top four, why not go for it? Um, see, why four, why not six, why not two, why? Like, I don't see the logic here, especially when we are going through those questions and showing how much overlap there appears to be here if some of the beauty of a factor analysis. If I didn't force the number, if I just hit go, and it grouped together some questions and grouped together these other ones, and you were like, wow, that is so interesting.

0:41:55 Jenessa: I wonder why the way people answer these sexual propriety questions goes together. And the way they answer these other questions, the way that they answer that I would have thought are like sexual propriery, for some reason, the math thinks they're not the same. Have thought that meant something. But when you force it to be four factors, I don't necessarily think it means anything because you told it to before. If you had told it To be three, would it have put those two together? Maybe?

0:42:23 Thomas: But it's giving you—now we can talk eigenvalues and other terms that I don't, it's not as though you can just make something have four factors if it really doesn't, right? Because you're going to see that reflected somehow in terms of like the the scoring of like how well is this actually working, right?

0:42:38 Jenessa: Yes, yes. Okay, that's, that's fair. Yeah, so that if I try to force it to before, like, do it. It'll—computer will cooperate but it will give me back numbers, results that show how poorly correlated these are. So it is true, yeah, the four. And the numbers don't look terrible enough to say that is completely as if I, you know what I mean?

0:43:03 Thomas: But did did they already do some weird stuff to get to the four? Because are we, now are we back to the floor we talked about before, which is sexual propriety, censorship, punitive justice, egalitarian beliefs and policy?

0:43:15 Jenessa: So those four.

0:43:16 Thomas: Yeah, because now I am racking my brain to figure out, again, not just why was the Islam question categorized the way it was, but now I'm wondering like, wait, why did he even correlate with those other questions? That does seem weird, but like if, okay, so this is where your expertise is really key. When you look at that chart, because in one of the million tables, as you say, it gives us those scores as to, and precisely what they are is, you'll have to help me, but I think the first number is, this is how much this question correlated within this factor. So like,

0:43:14 Lydia: Yeah

0:43:49 Thomas: You that's how we went in order. And then the communalities number tells you, I'm blanking. What does that tell you? Oh

0:43:56 Jenessa: This is the tough one. It's like, just in general, how useful is it towards the factors overall? All. So extent to which this item loads on to any factor is useful to any factor at all. A part that also confuses me at times. And I often ignore that line on the table. I use it to X stuff out and I move on. But it I believe, indicate something exactly like you're saying. As you look down, the first three look like pretty similar numbers and then we have a pretty serious dip where yeah that islam one is I mean the math is showing us it's correlating with those other ones not nearly as well as the top three correlate with each other and hmm

0:44:43 Thomas: I will give us the dictionary definition to work with, just It says it's the proportion of each variable's variance that can be explained by the factors, so you would want a higher value because the more variance that's explained by the factor you're actually looking at, the more directly related to that factor it is, right? Like that tells you that it's not being explained by something else. I guess the challenge that I have is that I don't totally get how that's not redundant like in a way, but there is some way that it is not redundant in some complicated stats way. I just don't, I can't articulate what what it is.

0:45:19 Jenessa: So it's kind of like, um, say, like, the variance is explained by—um, one way to think of it is any variance left that's still not explained. If I added a fifth factor, what would still get explained? A sixth factor, a seventh factor, um, know. And eventually I'd have all the factors that explain all of human behavior and there'd be 7 million of them. But that's a way to think about it is for this item and factors that I have picked out, how much of the variability between people as people, some people answer it this way, see it.

0:45:54 Thomas: I See it now. So maybe to put it in terms again that will help other lay people. Let me make sure I'm thinking of this right. Let's say you have a question that everyone answers yes to. It would correlate 1.0 with whatever factor you found, but it would have a communality of like zero. If everyone answers yes, then yes, absolutely. Whatever factor you divided that into, that question would be like, yeah, 100% with this factor, but none of the variance would be explained by that factor alone because every factor would have it. I mean, I'm using extreme examples, but is that right? Maybe that's not right, but it's the amount that other factors also might contain that answer, but the variance is explained just by the factor you're looking at. Could that be it?

0:46:40 Jenessa: Think so. I think for commonalities, you shouldn't think of the individual factors. I think you should think of the whole model as a whole versus like what's.

0:46:47 Thomas: Well, then why is it answered by question though? Because there's a different common...

0:46:51 Jenessa: Yeah. Yeah. So for each question, given that we have like four factors in total, is its variability getting the variability between people getting explained? So yeah, if everyone answered it exactly the same, like you're saying, yeah, the communality would be zero because the factors wouldn't help to explain it. Yeah. So I think that's exactly how to think of it. The easiest, most straightforward way, the commonality tells me the extent to which that factor or any factor is helping to explain how somebody answers this question. I

0:47:30 Thomas: Factor, okay, any factor. So does that mean the communalities is a score of kind of the entire factor analysis in a way? Like under this four -factor analysis, this is how much this question is explained in this model essentially?

0:47:43 Jenessa: Think so, because I'm pretty sure the extent to which it is explained by only this factor is what the other number is.

0:47:50 Thomas: Yeah, okay. That's why I was thinking it was redundant. I guess. And so, okay. Yeah, I think I've made sense of it now. Yeah.

0:47:57 Jenessa: So the first number you'll see is the extent To which this factor helps explain the variability in this question. And the second number at the very end of the page is the communality. How much does this model even existing explain the answer? And if it was very low, that means this question doesn't really fit into this model. And so sometimes they just get tossed. They have nothing to, this model doesn't explain this.

0:48:27 Thomas: I see. So that might not necessarily be an indictment of the model. It could be an indictment of that question. Like if you ask a question that just kind of, okay, that's interesting.

0:48:38 Jenessa: Absolutely. So like they tossed the vegetarianism question because they're like, it's just not doing anything for us.

0:48:43 Thomas: I See. So this is a good way to, again, name your factors, because this tells you, you oh, these are the scores that actually matter within the given factor. And the other ones could have a correlation. But if again, if everybody's answering a certain way or they don't work in this model, they'll have a low communality score. And then it would tell you, OK, even even if it correlated most with factor three, it doesn't really mean that it's like core to that factor. And it wouldn't be really something you would want to name that factor based on. Right. Yeah.

0:49:13 Jenessa: Yeah, yeah, I think so. And I think this is also part of how, like, know how much we want to get into the orthogonal versus, yeah.

0:49:22 Thomas: I don't know that if you, yeah, if you actually made sense of that, sure. But I don'T know if we, I personally never made sense of it enough to say anything about it that would make sense to people. But sure, you go if, if, you did.

0:49:32 Jenessa: Yeah. All right. Let me give you a quick blurb and we can decide whether we want to talk about it. Normally, the way everybody I'm used to talking about it, maybe other fields of psych do it differently. I want to reserve space for the Obloman people, whatever. Normally, I, if I was doing a factor analysis, it feels almost definitional to me that I would do it orthogonally, which means I would want my factors to not correlate with each other.

0:50:04 Jenessa: That items that belong in box A, not only do they all belong in box A because they match each other, but they don't belong in box B because the other box B items are different. And so this is, for example, how the big five personality model was made. I take all the adjectives out of the dictionary that look like they could relate to a person And I ask people to either, you know, describe a person and see which ones they put together or tell me which ones go together and run a factor analysis. And what you'll find is you get these five nice categories in every country. It's really cool. So, for example, you've got extroversion, introversion, and you've Got Openness to Experience. Experience, and I've got a bunch of questions I can ask you that will assess extroversion. I got a bunch OF questions I CAN ask you, that assess openness. Within those bins, the openness questions correlate with one another, the extroversion questions correlate with another, but the extroverse questions don't really correlate with the openness question. And I want them to not correlate with each other because I want these to be independent, different things. That's why we would say these are five unique traits and you can get a score from zero to 100 on all five of the different traits on the big five. That's the orthogonal side of it.

0:51:34 Lydia: She done orthogonal on, you know, what they eventually come up with, which the two-factor analysis, then we would have likely seen questions be grouped differently because right now there is assumed to be relationship across both factors, right?

0:51:53 Jenessa: Exactly. So she's doing the version, um, they're calling it oblimin, but I really liked the other word for it. Um, oblique, thank you. I don't know why, for whatever reason, I think oblique shows up in the, uh, more often. Um. So, you've got the orthogonal there where it's everything has to, like, definitionally, they have to be to a certain degree not correlated. The oblique one, you allowing them to be correlated to a uncertain extent. You might do that if you're like, I just really want to be able to pull out these two factors. I know they're correlated, but I do think they are are different things. You could almost think of them like sub factors of a larger factor maybe. And that's why they're correlated with each other.

0:52:42 Thomas: I can't make sense of what impact this has on, in real terms on the data. You know, it's, it's hard. I don't know. Would you do this otherwise, than like you weren't getting good enough factor analysis going when you didn't do it? Is there any reason you can think of why you would have wanted to do this?

0:52:49 Lydia: It

0:52:57 Jenessa: In this study specifically, or in general, inthis study, no, I got nothing.

0:53:05 Lydia: Could

0:53:00 Thomas: In this study. Yeah, exactly. It just seems, again, lots of p -hacking potentially. Okay, we're going to call it there. Believe it or not, that's about half of what we recorded with Jenessa. And there's some even more complicated stuff. If you want to hear more, and including, if you want to hear some more, it's like complicated advanced statistics stuff. But again, Jenessa does a great job of making it incredibly clear for us. And so, yeah, yeah. And so if you'd like to learn even more smart reasons why Jordan Peterson sucks, hop on over to patreon.com/wherethereswoke and get a bonus Jenessa. You know, it's a, it's a whole, like, another episode. It was that good. And I'm seriously a giant thank you to Jenessa for taking the time. Only with this, but with other stuff, like we, she even looked at another study that I sent her where someone used, like, study wasn't great, but they least used like real methods, like actual methods to kind of categorize people as woke or PC or whatever. And even though we kind of, you know, it wasn't like we agreed with the study, but it was like, oh, this is how you would actually do it. And so that was really useful and interesting too. So just want to say huge thank you to Jenessa. She was amazing. I hope to hear more of her on various stuff, including Serious Inquiries only. So if you'd like to hear mehr, check that out, patreon.com/wherethereswoke. not we'll see you all next episode thanks so much for listening do you want to sign off or

0:53:27 Lydia: Really accessible yeah goodbye

0:54:27 Thomas: Bye now bye so many browsers open, I can't find where Zencaster is, I don't know how to hang up, I don't—I have any idea—I've so many this is the first time this has ever happened Yeah.

0:54:56 Lydia: That's the other part of the study. It's the sucking you in now.

0:54:59 Thomas: I can't find where we're talking. I don't actually know.

0:55:02 Lydia: Horror movie.

0:55:03 Thomas: Oh, wait, I might've found it. Okay. There we go. I Can Stop It. All right.

0:55:08 Lydia: All right. Bye.

0:55:09 Thomas: Bye.

0:55:09 Jenessa: Shit, these are correlated. It's