SIO57: Problems in Academia

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:29 Thomas: Hello and welcome to Serious Inquiries Only. This is episode 57 and I'm your host Thomas Smith,. Today I've got for you a conversation with Matthew Facciani and Jeremiah Traeger. And what we're going to be talking about has to do with publishing and academia. It's something that branched off a bit from the Gender Studies hoax paper. When that happened... I had a lot of people in academia, PhDs left and right, it was pretty cool, emailing me and saying, "Hey, here's how this works." Not having gone to grad school, that's really cool, I didn't know how it worked, so, uh, branching off from that a little bit, we're going to talk about both how publishing in academia works in these journals, how peer review should work, and then also a lot of the pressures that are inherent to the current system and a lot of the problems that come from that.

0:01:19 Thomas: So it's a really interesting topic. It gives us kind of both more perspective on the gender studies hoax, but also reaching beyond that gives us a bigger view of what's going on in academia and some of the challenges. So with that said, let's go over to Matthew Facciani and Jeremiah Traeger. You guys are here to talk about the problems with publishing and academia, right?

0:01:48 Jeremiah: Yeah, I reached out to you because there's sort of a culture that sort of... enables this type of thing to get out there. There's a big publish or perish mentality, so I just ran it by you and Haley and it seemed like something you were interested in, so...

0:02:04 Thomas: Yeah, definitely.

0:02:05 Jeremiah: I'm a graduate student in engineering, which still is kind of like, when you're a grad student in engineering, it's still like science, but I know Matthew has more experience in like social sciences, like what the, what the penis hoax was into. Yeah.

0:02:24 Matthew: Yeah.

0:02:26 Thomas: So there's no there hasn't been any engineering penis hoaxes that you know of, you're saying?

0:02:26 Jeremiah: No But but one thing I want to emphasize is like, when you, when you get into academia and you're reading papers, and I'm not making a false equivalence here, like pretty much every paper you read, you look and you're like, okay, there's a problem here and there's no problem there, and that doesn't invalidate like any given paper, like there's still good stuff to take out of most things, some things are worse than others, and it's sort of the aggregation of like all the data that sort of builds consensus, so.

0:03:03 Thomas: Hmm.

0:03:03 Jeremiah: You know You know, one paper for me is not the biggest deal. So, well,

0:03:08 Thomas: Well, maybe let's go. Well, first, thanks for giving your background. Matthew, you want to give your background, too,

0:03:13 Matthew: Yeah, so I'm also a graduate student. I started off getting a PhD in neuroscience, actually, and ended up doing that for a few years, and I switched majors into sociology, and I'm finishing my PhD in sociology. Yeah.

0:03:26 Thomas: Perfect, yeah. So, that's, so gender studies is a branch of sociology, right? Do I have that? I'm not in academia, so sorry if I get that wrong. Oh.

0:03:34 Matthew: No worries, it's tricky because, um, I think there's some different subsections of gender studies because it's such a huge, broad concept. And the gender studies that I think about may be different than someone who publishes in literature or media studies, which I think there's a lot of conflating going on. So, maybe we can get into that a little bit more. But basically, how I study gender as a sociologist and a social psychologist is more about having gender as as a variable in experiments. And that might be different than someone like writing about different types of theories or ideas that they have that might Be more like literature based.

0:04:07 Thomas: Okay do Okay, so it sounds like there's maybe more to it than just simply gender studies is this, but do you get the sense that the, uh, I don't know if you've heard of the Iyer right now and the penis hoax stuff and all that. Do you get the sense that that's directed at any given subdivision within gender studies specifically, do you think? Or is that just a general, you blanket gender studies is bad kind of thing?

0:04:34 Matthew: Well, I think that's the problem is because they say gender studies, a lot of times from what I've seen, it's like this study, or the conceptual penis thing shows that gender studies has problems. And they're not differentiating the different subcategories within it very explicitly. So, to me, I think they're unfairly criticizing a lot of literature that's not necessarily trying to be scientific. Because if it was, then yeah, I would agree that it's not falling, it's not very rigorous. But from my understanding, and that's not really my field, again, I don't do a lot of the more media studies and stuff like that, but I don't think they're trying to do it from a scientific lens.

0:05:17 Thomas: Yeah, I brought it like one of the sources that James gave me was the, fat men's sexuality paper. I think it was something like that. And it just did kind of a survey of Different movies, shows, whatever it was, different things in the media of, you know, how fat male sexuality is portrayed. And that seemed like a perfectly valid thing to talk about to me. So is that one of those things you mean where it's like, this is just kind of looking at, you know, popular culture. It's kind of doing a survey. It's not meant to be like a study where there's these variables and that, you know, it's, is that what you mean?

0:05:56 Matthew: Yeah, exactly. And I think there's still a lot of utility to that because it can be scientific, like down the road, like those ideas can create, like it can give someone ideas for something they could study and do experiments on. And they're just starting to, like, share people's experiences with, like, for me, a lot of that stuff seems to be just sharing people's, their stories, like what it's like to be a fat person or a queer person or whatever group in whatever area of society, and then if they write about that and share those narratives, then you can talk about them, and then maybe some other group of people, like social scientists, can be like, "Oh, that's an interesting idea, maybe fat people are discriminated against," and then we can run a study on that. And to me, that's like a totally separate kind of thing that we're starting to talk about.

0:06:43 Thomas: Hmm. Yeah, well, okay, actually this is going to sound really stupid, but again, I didn't go to grad school. I went to just regular school. I went to just undergrad, and I'm sure there's plenty of people listening who don't have the experience that you guys do and that other of my guests do who are PhDs and all this stuff. What exactly is going on when you talk about the problem of publishing? So, because these journals are very interesting to me, I never really looked into them at all until this thing blew up. And then all of a sudden I was reading all these different academic journals and trying to get a feel for what they are. So, like, what are they? Give me, give us the basics of what's going on there.

0:07:20 Jeremiah: I think when we were prepping for this, we want to establish sort of the culture of academia and How, you know, when you're a grad student or a professor, you want to get funding for your research, and you have to apply to, you know, grants, you have apply for public and private ones. And, uh, there are different projects that are going to get funded for research. You, journals want, like, sexy results, they want significant things, they want game changers, especially if they're a big journal. Uh, so if you're applying for funding, you're going to get that same type of, uh, emphasis. You want sexy stuff, you want interesting projects, projects. And one other thing that ties into this is that it's not just how, how good of results we project they're going to be, uh, we're also looking at the researcher, so we're looking at the professor.

0:07:29 Thomas: Yeah.

0:08:17 Jeremiah: And seeing, oh, what type of publication history does, does he or she have, like, how many papers do they have. How many times have they gotten into like a high impact journal, stuff like that, that really ties into how many grants you get. It also ties into hiring, because schools and universities, they're hiring faculty, they want to look at their publication history. They want to look at the awards they've gotten. And so it's incredibly competitive, and, and basically your career as a scientist is tied to both how many times you publish and how good the publication is. So, like, that, I think that's really important to emphasize, because it's really, um, it's really important to understand the culture of academia.

0:09:10 Jeremiah: One other thing to bring up is that regarding hiring, the PhDs that have been awarded in the past 30 years has increased, and the faculty positions offered has not increased that much. It's been actually fairly constant. In science and engineering, I have a citation you can put in the notes. Faculty jobs have been around, like, you know, 3,000 a year, roughly. Compare this to an increase from 1980 of 20,000 PhDs awarded to 35,000 PhDs rewarded.

0:09:50 Thomas: Yeah.

0:09:50 Jeremiah: Not like every PhD goes into academia, but you still see an increase when there's no increase to back up the hiring for academia

0:10:02 Thomas: Well, this is a different topic, but that was why my opinion on the whole equity hiring thing at Evergreen was like, look, I know we have way more candidates than there are positions in academia, I had thought. So it seems to me that that would be a reason that prioritizing diversity wouldn't necessarily reduce your candidate.

0:10:02 Jeremiah: Yeah

0:10:27 Matthew: Yeah

0:10:35 Jeremiah: Right but but what I want to emphasize is there's an enormous incentive for you to get published so like you want to get by the peer review process I'm

0:10:49 Thomas: So community that improves your chances of... So when you say get published, so again, I'm sorry, you're gonna have to break it down more basically, I think. So when you're getting your PhD, are you publishing your thesis or is that just within your university or something? Or does that get published in a journal?

0:11:09 Jeremiah: Working on actual papers to put in a journal um I don't know, I don't know how it is with Matthew

0:11:13 Thomas: Okay.

0:11:15 Matthew: I think from my experience a lot of phds do try to publish at least a few times during their their graduate school career. So you'll have your,

0:11:23 Thomas: Oh, okay.

0:11:24 Jeremiah: Like, like in engineering programs, we try, I think I have a minimum of two, like, first author papers I have to be on, and like, second author papers can tie into that too. So yeah it's

0:11:35 Matthew: Yeah.

0:11:37 Thomas: What are the kind, could you give us an example, like what are the kinds of things, because that strikes me as like, wow, it's incredible that a PhD student, someone who hasn't even finished their, you know, program or whatever, is able to publish, like, when I think of these journals, I think this must be groundbreaking stuff. It must be like new, you know, new academia, new topics, new whatever, like what is the kind of thing you're publishing?

0:12:02 Jeremiah: Actually, academia is kind of backwards, because it's a lot of the time the grad students or the postdocs, they're, they, that's what you do after you get a PhD if you want to go into academia, do a postdoctoral position. A lot of the times it's the students doing the research, and your advisor is just there to uh, give advice and encourage you because they have experience, but they often usually don't do the research, from what I can tell. Yeah.

0:12:31 Thomas: Well, that's kind of how the whole world works, actually. So it's not backwards. You get to a supervisory position, then you just have everybody else do stuff. And then you're like, oh, I'll put my name on that. So that's interesting. But but what kinds of give me an example. Have you published anything or are you still pending publishing?

0:12:33 Jeremiah: Yeah Yeah, yeah, yeah. But

0:12:35 Matthew: Yeah.

0:12:48 Jeremiah: Writing, so...

0:12:49 Thomas: OK, OK.

0:12:50 Matthew: Everything you're saying, I agree with and I think it's spot on. I guess, yeah, for me, I published a few times as both a first author and a co-author in different types of journals. So when I was on the neuroscience program, I have a publication in the journal Human Brain Mapping, for example. And I was a co-author for that. I helped analyze and collect the brain imaging data. And I wrote that paper with my advisor and some of our grad students and we submit it to the journal. Usually, almost always, you're going to get tons of criticism from the reviewers. Reviewers and the best case scenario is then a revise and resubmit and then you resubmitt it and then hopefully if you made all those changes that they wanted you to change then they might accept it but the thing is it's very difficult to publish in these high tier journals. So, um, I just recently submitted to probably the top, um, social psychology journal, uh, Social Psych Quarterly, and I got rejected.

0:13:48 Matthew: So that was a bummer. And the acceptance rate is like nine or 10%, even after the revisions. So in academia, it is a lot of just self correcting and getting a lot of feedback from peers, and trying your best to do good science, send it off to a journal and see what kind of feedback you get, and then how to better communicate what you found, or if those results are, are interesting enough to publish. And that's something that I think Jeremiah was was talking about earlier, that is really important is this idea that the sexiness of the content is more important than anything else, unfortunately. And that becomes really problematic because in academia, publications are the currency that it runs on.

0:14:33 Matthew: And the more currency you have tends to relate to how sexy a lot of those publications were and how interesting the results were, not even necessarily if the work was very methodologically rigorous.

0:14:45 Thomas: One thing that was referenced in defense of this gender studies hoax was that the Cogent series boasts some sort of like 60 something percent rejection rate, or approval rate, I can't remember which. So now you're talking about, for a top journal, a 9 to 10 percent exception rate. So that sounds like a big difference to me. I just wanted to point that out. So you're not, now, I'm maybe not necessarily impressed by something like a 60 percent rejection rate.

0:15:08 Matthew: Yeah No, that's, that's really easy actually. Usually the top journals, I mean, 10 percent's even like doable, um, sometimes like Science or Nature, I don't know, some of the journals Jeremiah might be more familiar with, engineering, like, um, it could be four or five percent even. Yeah.

0:15:29 Jeremiah: They have, like, impact factors of 30 or 40, so that's the type of things we're looking at. But it's important to note that, like in the peer review process, for the most part, the reviewers are contributing as a volunteer effort. When you submit your paper, send it to the editor and you say, well, these people, yeah.

0:15:34 Matthew: Sure

0:15:33 Thomas: Right. Oh, yeah. Hold on. This is a great, this is a great point because could you, yeah, can you, you guys could talk about how peer review should work? Because I'm guessing it's not how it worked in the, in the Cogent series thing. So, yeah, great. I should have asked that. Go ahead.

0:15:58 Matthew: Sure

0:16:02 Jeremiah: Oh, yeah, no, no. This is good. We prepared for this. So we, you know, you submit it to the journal, and then you're supposed to get it peer-reviewed, and and they'll ask you you know who are some good people in this field who can review it and and you give some suggestions but you don't know if those are the people.

It's blind, so you're you're not supposed to know who those people are, um, and sometimes uh sometimes they don't know who you are either, so um, but it's a volunteer effort, and again, to to emphasize uh just how pressured, like, professors and postdocs and faculty are, they're doing their experiments or going to meetings all the time, and now they have, on top of that, they have to spend a few hours looking through this in-depth paper and uh make sure it's not full of flaws and and take everything out all without getting paid. Like, my postdoc my post doc I was talking to him yesterday he's like Like, yeah, I'm just going to be up all night reading this paper, like, on his own time. Just, you know, mostly just for the endeavor of science. It's for the love of it. Well,

0:16:05 Matthew: Yeah yeah

0:16:20 Thomas: Oh, yeah.

0:17:10 Thomas: I was going to say, what is the incentive to even do that?

0:17:18 Jeremiah: I mean, we benefit from, yeah

0:17:17 Matthew: I mean you can put on your cv too it can be like, it so there is some some benefit, especially as like graduate students or young researchers, I can say I was a reviewer for X journal, especially if it's pretty good journal, but like the really established researchers, like for them, they're just doing a service at that point. They already are big names in the field. And they don't really gain a lot by volunteering their time to, you know, have a really good review on all these journals that they have to look at.

0:17:21 Jeremiah: That's true, right?

0:17:30 Thomas: Oh, okay.

0:17:48 Jeremiah: So there's a benefit to the science that if you help a paper get better, then, you it lifts the rest of the field because you want good science in your field to build upon. But that's just a barrier to a lot of people. It's just you're putting effort into something you're not getting paid for necessarily. And the higher-ranking journals have a bias, again, towards this groundbreaking research. And what that leaves out is room for replication. Like part of the foundation of scientific inquiry is independent verification, but you're not going to write up a paper that does exactly what another group does, you're not going replicate things. A lot of the time, what replication looks like, at least for me in the science, is somebody else makes a method to, let's say, create some material and say, I want to use that material to work on another experiment because of those material properties. Um, what I can do is try and use their methodology to make that material first and then do an experiment on that material. But a lot of the time, um, that replication doesn't work out, so but but as you can see

0:18:35 Thomas: Yeah. Hmm. Yeah, so so when something is published in a journal, it doesn't it hasn't necessarily been replicated. Is that right?

0:19:15 Jeremiah: There's yeah correct yeah it yeah and

0:19:23 Thomas: Hmm.

0:19:23 Matthew: Yeah, most of the time it's not a replication or anything like that. And again, because it's not sexy to replicate something, it's going to be interesting if there's already this nice study that's already published. And you're like, well, we're going to see if they really found that, because unfortunately, a lot of times we can't replicate a lot of things. And there are flaws with the previous study. So it creates a whole lot of issues.

0:19:45 Thomas: Well, plus I've heard who's going to grant funding for that if you're like, oh, I want to just do the same thing like Okay,

0:19:50 Matthew: Right. Exactly.

0:19:53 Jeremiah: It's it's harder for you matthew right because you do like you study people so you have smaller sample sizes yeah

0:19:53 Matthew: Yeah. Yeah, yeah. I mean, that's the whole thing. All of these other issues apply to science generally. And then social sciences is trickier in a lot ways because they're studying people which have all these different problems to study them. They're not always going to tell you what they're thinking, for example. And there's a lot of problems even within the social sciences about making sure we define our terms really well and have really logical theory that can build on each other. And it's trying our best to make it as rigorous as possible. And there are some really good, rigorous social science out there. But there's also some really bad social science. And usually, but not always, the better science is going to be in the better journals. But as Jeremiah was saying before, like, in science, when you look at these different papers, you're always going to see problems with them because science is self-correcting and it's building off each other.

0:20:49 Matthew: And there's no going to be any perfect paper. It's just hopefully in a body of work in some area, you can see findings that build off each Other and make predictions. And we can see that in some social sciences and other sciences, but there's also a lot of things that we can still learn a lot about.

0:21:10 Jeremiah: So there are a lot of problems that this leaves us. It leaves for poor replication, obviously. I have this other citation from Nature where they did a survey. It's not how many papers were wrong, it's how many scientists found bad issues, but they found a poll that said 70% of scientists in the Nature Journal failed to reproduce results of some kind, like, in their career. So, um, that's an overrepresentation, it's just 70% of scientists, because a lot of times you make mistakes in replication, but still, a lot at the time you're going to have problems reproducing things, you

0:21:52 Thomas: I think I've heard that misquoted is like 70% of experiments haven't been, but you're just saying 70% of scientists of a given journal said they have had replication problems, what, in their career and ever? Okay. But

0:22:06 Jeremiah: So so they've had trouble reproducing, they have bad, they're bad results uh you can find papers with bad methods but they give really flashy headlines it's really good for pr uh there's I don't know if you, yeah.

0:22:23 Thomas: Hypothetically they all could have tried to replicate the same thing and failed and then only one experiment is not that's that is just a weird I don't know it's not what I thought it was I guess that's what I mean to say but anyway go on sure

0:22:25 Jeremiah: Yeah yeah I'll I'll put I'll putting it in the notes so so people can check that, um, uh, there, there's also, I, I'm sure you've heard of p hacking um, right

0:22:42 Thomas: Yeah, yeah, you may need to explain that though, because every time I hear what these p-values are and all that stuff, I remember it for like 10 minutes and then I already forget what it is.

0:22:46 Jeremiah: Yeah it's it's It's a statistics thing. Basically, you can tell how good or how significant a finding is based on a statistical value from your data set, and that's your p-value. And if you want to make sure, there's effect size, but that's different from p- value.

0:23:10 Thomas: Is it how big an effect is? Is it like how... Is that what it is? Oh, okay.

0:23:20 Jeremiah: I don't know if Matthew can explain effect size, but p hacking I sort of an arbitrary cutoff for how good a p value you can get, a statistical value, and most people pick 0.05, and if your data crunches out and you have a value less than 0.5, then people look at it and say, "Oh, see, this is significant," and that's what you're aiming for when you set up your experiment.

You're like, "Okay, I'm going to aim to get 0.05 or less," and if you get more, than a lot of reviewers looking at your paper aren't going to take kindly to that, so you have to try again. But the thing with p hacking is, you know, sometimes you get different numbers from experiment to experiment just due to error and and fluctuation. And so what p-hacking often means is that you're doing an experiment and you're doing multiple studies and you just happen to stop where the gives you something significant to chew on. It's sort of like rolling the dice until you get the number you want and then moving on from there. So that's just one way to sort of cheat getting by peer review

0:24:36 Thomas: Right. Okay, I still don't totally know what p-value means then. I thought it was like... Okay.

0:24:45 Jeremiah: Yeah

0:24:46 Matthew: Is just looking at how likely a result is to occur by chance alone. So when we say 5%, that means that's only a 5% chance that that would happen purely by chance alone. So when we have a 0.05 cutoff, yeah,

0:24:59 Thomas: Oh, I see. So if you're, if you're doing some sort of experiment where you're grabbing a bunch of data that's supposed to be, I'm guessing randomized, there's always the chance that like you just happen to randomly pick all the, if let's say it's people or something, you happen to randomly pick all the people who have a certain characteristic. So by chance, is that what it means? Like by chance your data will be skewed then. Okay. I see.

0:25:22 Matthew: Exactly. So if you have a 5%, cutoff that means one out of 20 experiments is going to be purely due to random chance that you find a significant result. So that's a problem. And also there's ways that you can abuse the system. For example, looking at sample size. So something that is frustrating to me in the social sciences, we do correlations sometime. And if you have a huge sample of people, like tens of thousands of people, you can get a statistically significant difference between two groups, but the correlation might be like 0.01. So is that very meaningful versus significant? Like, you could have a significant difference between these two groups because you had so many people, and because of all these people you found a significant difference of these two correlations, but it doesn't really tell you a lot as far as what's going on in the world. So that's just one example of how you can use this one measure of significance, if it was likely to happen by chance, as the end-all be-all.

0:25:32 Thomas: Okay.

0:26:28 Matthew: And it shouldn't be like that. Like we talked about before, effect size is the magnitude of the effect. So not just looking at if it's likely to happen, but how strong the effect is. And we should always look at that and other types of measurements in tandem with just the p-value. So, just having one p- value and saying, oh, this was significant in one study, like, that doesn't mean a whole lot by itself. But if there's, like a couple studies all looking at the same thing, they all have significance, and they all pretty good effect sizes, meaning they all have pretty strong effects, then we can see that this thing might actually be a real thing.

0:27:04 Thomas: Yeah, so you're saying you need to look at more than just a, it's like, maybe right, currently journals are a little bit simplistic sometimes in that they're just like, is the p-value less than 0.05 or greater than, and that's kind of how they're determining it, but you're saying you kind of need to look at more numbers than that, you need to look at sample size, effect size, and get a bigger picture of just how rigorous it was, I guess, or just how replicable it might be.

0:27:30 Matthew: Yeah. Mm-hmm.

0:27:32 Thomas: Okay.

0:27:33 Jeremiah: So, and other problems involved with, you know, the peer review process, uh, regarding how it's difficult to get anything in that isn't cool or significant. This, as you might imagine, if you're not publishing things that don't work, then that also kind of doesn't help science, because, uh, if you're not publishing things that don't work, then, uh, somebody else

0:27:59 Thomas: Oh, are people always doing the same thing and like not knowing that someone already tried it?

0:28:00 Jeremiah: Yeah That can happen a lot. There's, uh, I have another nature link, um, you can actually go to, for example, clinicaltrials.gov, of uh, they keep a database on all clinical trials, uh, in the U.S., uh, but it's not some it doesn't have the level of detail that you would have in a paper, so, and not every clinical trial is going to be the same, so you it's going to leave out things that are necessary for scientists, and it, and it, it turns out around half of clinical trials go unpublished, just just because everybody's trying to get you know the stuff that works in so it's it wastes time for science it Also, you know, obviously predatory journals will exist to help you publish as an academic you, you want to get your name out there, the predatory journal can send you an email and you're like well

0:28:38 Thomas: Yeah. Yeah.

0:28:57 Thomas: I yeah. I have heard from several people. I got a lot of emails that were like, hey man, I get emailed from these stupid journals all the time. Like, hey, come publish something. Do you guys want to talk about that? Do you get that already?

0:29:00 Matthew: Yeah.

0:29:09 Matthew: Yeah. I would, yeah, I would like to, because that was something that was very frustrating whenever I saw that mentioned with the conceptual penis thing about, like, the automatic transfer, first off, about how that was just, like automatically linked. Like, that's something that isn't recommended necessarily by, like other scientists as like, like, we really think this is a good idea and we're going to actually make a conscious uh thought out decision that this other journal could be a good fit. Like, that wasn't what happened at all. It was just an automatic, like, this uh within that system of publishing, and a lot of these predatory journals are just kind of built in to send out these these emails because I get those. I mean, it seems like once a week I'll get an email for some random journal. I might even got one from that Cogent Social Sciences that's like, "Your work is excellent, please publish in our journal." And, like, they'll, you can tell, because, like, for me, I don't have my doctorate yet, and and they're labeled Dr. Facciani because it's just automated,

0:29:20 Thomas: Yeah. Right.

0:30:03 Jeremiah: Yeah Yeah, I pretty much ignore those, so that's all anyone really does. But but as you can see, like, all these things coming together, you don't really help create helpful results in science. And if people are pointing out all these flaws and you start getting BS papers everywhere, where it's going to start uh losing the public confidence in science as an institution just because everybody's getting all these results out that don't mean anything. And over time, people are going to say, oh well, that that's just a study that doesn't mean nothing when really, like, it's the bulk of the scientific literature that tells you what's going on. So

0:30:03 Matthew: Right?

0:30:03 Thomas: Yeah. Yeah.

0:30:46 Thomas: You know, I don't know that I got a clear answer on how peer review should work. Could you guys talk about that? And if you want to reference what happened in the penis hoax, you can. How should peer review actually work? Is it people, you talked about it being blind and I think we covered that much, but what are the kinds of things they're telling you? How involved should it be and is it?

0:30:53 Jeremiah: Yeah Matthew, you've published more, so I want to hand it off to you.

0:31:13 Matthew: From my experience, it does depend on the journal. So I've published in a couple different tiers of journals. And, like, first one that I published at Human Brain Mapping, to me, that was a, you know, it was a good journal in neuroscience, science, and I thought the review process made a lot of sense. They had valid criticisms and they had people that were motivated to look at the paper critically. And I feel like that, as we talked about before, it is volunteer based, but there is some incentive if people want to have that title on their CV of them reviewing a good paper or a good journal, then it's going to look better for them. So maybe that's why the feedback at top journals are going to be potentially a lot lot better than those in the lower journals. Because if you write that you're a reviewer on Cogent Social Sciences, no one's going to care about that.

0:32:02 Matthew: So you might not have as much of an incentive to actually look at it. I do think peer review is great in theory, like we should have relevant experts look at it. And a lot of times you do get some really great feedback. But there's always this joke about, like, reviewer two is doesn't know what they're talking about, because you'll get a series of a couple reviewers giving you feedback and one of them, it seems like, didn't even read the paper.

And it's possible that they, you know, were up all night trying to get this deadline done and they're like, oh crap, I have five papers in your review, and they'll scribble through them all, or they'll do it, you know, on an airplane to a conference. And sometimes you just won't get good feedback. So ideally, it'd be nice if maybe people were paid to do this so they had an incentive and there was more a systematic way to prevent um, to have more checks and balances of people just not paying attention to it, and also making sure that you do have relevant experts actually reading the paper. Um, so those are just a couple ideas that I have, because I do think, you know, it can work. A lot of times it's just that there's all these problems with it too.

0:32:44 Thomas: Yeah.

0:33:06 Jeremiah: Almost, you never get it back and say this is great. In fact, a lot of the time they'll get back to people in the hard sciences and say, you need to do this experiment or this experiment after you think you've accounted for everything. So it's not, they'll tell you what you need to do again, maybe they'll want another data set. Certainly the formatting stuff too, but like that's that's just fluff so you know

0:33:17 Thomas: Yeah. Hmm.

0:33:30 Matthew: Yeah like Um, the most recent paper that I got rejected from, uh, the main criticism was the data I used wasn't new enough, even though in the paper I said that the theory should apply to any year, and they're like, well, it's not new enough, so that was really frustrating for me because I explicitly said like it doesn't matter what year this takes place that the same stuff should occur. And you might get feedback like that. And again, if you're trying to look at these top sexy kinds of articles, then they might say, well, we want data that's from two years ago instead of 20 years ago. And it just looks better, even though the theory is just as good. And stuff like that can be really frustrating.

0:33:46 Thomas: Yeah. Yeah. How long is it? Can you give me any indication of like how long a peer review an actual peer review should take? Because I'm just trying to get a picture for what that because there was a quote unquote review process for the penis hoax. But you know, the average listener probably has no idea what what in like an actual rigorous peer review or even just an acceptable

0:34:21 Matthew: Yeah.

0:34:45 Matthew: So From my experience, can take a few months, so when you submit it on the website, yeah, it might be three months at least for a good paper, that's a good journal that's getting tons of submissions. Some of the top papers can be up to six months, even maybe even longer, before you get a feedback, and that's the first feedback you might get. So let's say you get, again, they're asking for revisions, they want you to resubmit it, so then you might not after you make those revisions and resubmitt, that might be another three months. So a lot of the journals that are being published, that if you go in Google Scholar and you look 2017, you know, they might have been you, but they're being out right now, but there was the actual experiments might have done, you know, a couple years ago.

0:35:35 Jeremiah: Yeah.

0:35:36 Matthew: Yeah right yeah no seriously that could be an issue

0:35:39 Jeremiah: You can imagine that weight also applies more pressure because like what are they doing they're trying to get more papers out so like it slows everything down yeah They

0:35:40 Matthew: Um right

0:35:49 Thomas: Taking six months or three to six months, that does definitely paint a picture. You, because I think the cogent thing was like no time at all. Are you going back and forth multiple times with peer reviewers? Are you, you know, like just, just describe it if you don't mind. Mm-hmm.

0:35:57 Matthew: Uh, Yeah, in my experience, um, again, in the last paper, uh, there was a flat reject. So usually there's the initial desk reject that can happen, so the editor or someone else will look at, the give a quick glance through the paper and make sure, like, is this even worth wasting people's time. So really, that should have been desk rejected, the conceptual penis paper, if the editor actually looked at it. And they should this is not yeah the first journal.

0:36:23 Thomas: True, Which it was in the first journal. It was in the first one they, yeah, Norma.

0:36:33 Matthew: Yeah. Um, which, you know, Norma, um, I haven't heard of it before. I don't think it's a top.

0:36:37 Thomas: Impact factor of zero. Yeah.

0:36:40 Matthew: Yeah, yeah. That's something that I wrote about in my blog post That it's, you know, not a very top journal anyway, but it's still something that's at least, you know, a predatory journal necessarily. Um. And they desk rejected it. So that shows you just how low on the totem pole you went, down the level of journal, uh, prestige, for this thing to get published.

Um, but yeah, so assuming it gets through the desk reject, then it gets sent to those reviewers who, you know, volunteers that have expertise in this area, and then they might have, you know, several months to look at it, because they know that they're professors and they're super busy and they, you know, are probably going to wait till the absolute last day that they can, they can do it. Um, and then their going to, you know, write their comments. You get an email saying, okay, here are the comments from the reviewers. Um, they're going to either suggest that you revise and resubmit it based on their comments, or is this going to be like a reject, you know, try again, um, and maybe another journal. And that's kind of how the process works.

0:36:43 Thomas: Yep. Yeah.

0:37:39 Thomas: Well, that sounded just like kind of like a flat reject or accept kind of thing. I mean, or is it more, are they giving you changes? Are they giving you, yeah,

0:37:47 Matthew: So after you hear back in like three months, you're going to get comments that say, make these comments and then we'll consider, consider accepting it, or we think that it's still too far off and we're going to reject it. And again, they can only accept so many. So, you know, I heard a story from when I was in my last program, one of the top neurojournals, they were saying to my professor that they're like, all right, this is a really good study, but we have to reject so many papers. So we're gonna reject it it right now, but resubmit it in a few months, and we'll probably accept it. So like, that's kind of stuff happens too. And it's just kind of the game that, again, if you want to have a top journal, you have to have like a 10% acceptance rate or so. So sometimes that means you're going to have to reject pretty decent papers just to get keep that rejection looking pretty, pretty good.

0:38:04 Thomas: Yeah.

0:38:41 Thomas: Seems like now we can, we can maybe transition to your, your solutions, Jeremiah. It's because it sounds like, if we're just trying to summarize quickly or just list, there's a lot of problems going on. There's all this pressure to publish. There's the fact that, I mean, sounds like I had no idea that reviewers really have no incentive to even do, and I mean, it's, it's so, Yeah, yeah, true. They have incentive out of the goodness of their heart and slash wanting to maybe put it on their CV, as you said, and wanting to do a good job in general. And I'm sure that they do, but I'm just surprised that it really does sound like something that last minute, oh shit, I've been so busy, I forgot to do this, make some formatting changes or something, you know? And that's surprising to me. And there's all kinds of problems going on. I guess, what are your, how do you think we can fix this?

0:39:03 Jeremiah: Have some incentive, but yeah.

0:39:32 Jeremiah: As usual, more funding helps. You know, I think you covered that in a previous episode, so I don't really need to go into the problems there, but, you know, it helps ease researchers' careers careers when they know they're still going to be supported, even if they don't happen to get this particular paper in, you know, nature or something, like they're Still going to have a career after this. It also helps, you Know, universities fund faculty, stuff like that. So I mean, but money is obviously, you Now, a solution to a lot of things. So, right,

0:39:34 Thomas: Yeah. Hmm.

0:40:08 Thomas: So Just to illustrate that, though, or to flesh it out more, you're saying so, like, the lack of funding or the tight, you know, kind of availability on funding makes it so the pressure to publish is even higher because those who have published some sexy results in something might be more likely to secure that funding. Is that what you're... Okay.

0:40:28 Jeremiah: I mean the grants you're going to get, Like when you apply for a grant, your background and sort of prestige ties into that a lot, too. So the money you're going to get ties to how good of a researcher you appear to be. Some journals have thought of implementing better standards or at least more transparent standards to publishing. Publishing uh some have proposed publishing like your lab notebook so that's just like you writing down chicken scratch that you're supposed to do in the lab every day um it's not really it's not really easy to read and usually there

0:40:37 Thomas: Okay. Okay. Hmm. Yeah, I was gonna say, what do you think that would add? Like, what is the goal there?

0:41:08 Jeremiah: Uh well, for one thing, uh, let's talk about, like, if you're going to falsify a data point or change a data point's, um, usually that's going to happen in.

0:41:19 Thomas: "Dear journal, today I am falsifying a data point.

0:41:21 Jeremiah: No, No, no. It brings more transparency, because a lot of the time, at least what I'm looking at in my science, I'm trying to determine a trend, see what materials are more likely to produce this behavior in certain chemicals. So, say I do an experiment, and then three weeks later I do another, and three weeks later I do another. And then later I look at all the trends, and they're all discombobulated, but if I want to change a few things here and there after the fact that I know the trend, then I know what's going to happen. But if I've been taking notes about what these trends look like ahead of time, like if I write down what they are on January 1st and then January 22nd and then February 3rd, then it's more transparent and you can make sure that I didn't change anything around in a shady manner. Like, you wouldn't write down that you're gonna you're not gonna write down, obviously, that "Dear nature, I'm gonna falsify some data but it does bring more transparency and you know I don't know if there's

0:42:28 Thomas: Yeah," yeah. At least it makes it take more effort to fake something, I suppose.

0:42:34 Jeremiah: Yeah I don't, I don't know if there's a way to a hundred percent prevent uh faulty data or something but you know Nature has implemented more constraints, like being more transparent about their statistical data and methods. Um, other journals, like Plus, which is an open access journal, have been encouraging the publication of this less sexy or significant work. So sort of what we were talking about, like results that don't necessarily lead to big discoveries, maybe some data that shows things don't work, just being a bit more transparent so people have more information about what has been done and what, what hasn't, and and just things that people should know. So, and and as far as careers, and I think Matthew and I both agree on this, one, more pathways for teaching faculty or other pathways for other jobs in academia besides, besides just research.

0:42:43 Thomas: Yeah. Hmm,

0:43:35 Jeremiah: And I think a lot of the listeners, and maybe you might agree with this, you always had that professor in undergrad who just did not give a damn about the class they were teaching because they had tenure and they were more interested in their research. They didn't really care to host office hours. That's because they're focused on their research and they don't have time to waste on undergrads, and that's sort of, you know, that's the standard pathway in academia, is to go and prioritize research, get funding from university to work on research, and teaching is just sort of a thing you can do on the side, but it'd be really nice if there were more pathways for people who weren't really interested in research, or like Matthew and I are thinking of going into psycom as a career, maybe pathways for that, uh, if that happens then there are going to be more professors and more teachers who aren't filling up space in areas where, uh, other professors are wanting to focus on researching. So it- it opens up the pool a little bit more so

0:44:40 Thomas: Yeah. Well, it sounds like more money would solve everything, but yeah, I guess it doesn't necessarily, or maybe I'll ask the question, is there any reason why research and Teaching should be connected. Like, do you need to be one to do the other? Because it sounds to me like they would be two very different things. Like, I would want, you know, unless you're teaching about your current research, like, I would want someone who's just a professor to be focused on that, to be focused on their students, to be focused on their curriculum, and someone who's researching to be focused on their research. But is there a reason why those need to go together?

0:45:22 Thomas: Other than, you know, funding and like they're trying to get as much out of every person as they can.

0:45:21 Jeremiah: Well, go ahead Matthew, I

0:45:24 Matthew: Yeah. Yeah. I think that's a really good question. And I would say that they don't necessarily have to be connected. I mean, you can be a research faculty and just do research, and you can just be a teaching faculty. And, I think, if you're really focused in your research and you're reading up on the field, then you'll be able to teach it. I guess the only potential issue could be if you're not actively reading up on all the current literature and you're teaching for like 30 or 40 years and you keep teaching like the same stuff, then maybe you're going to be as good as the teacher then. But assuming that you're keep up to date with everything and you're reading the articles in your field, I think you can be a very competent instructor, even if you're not actively publishing. Publishing. But yeah, I think we do have this issue where a lot of researchers or people that are getting all the grants and publishing a lot, like it's a burden to teach.

0:46:21 Matthew: I know that this is something that I've seen where you'll buy out being able to teach a class, for example. So you have a million dollar grant and you can just pay the department money to hire an adjunct so they can teach that class for you, for example. So there's just a lot of professors that are really on the research. They want to be a big name in the field, and they get really frustrated when they have to teach. So they're the worst professors. And I think it would be easier if just they didn't have to do it. And then there was this more teaching jobs, and that was more accepted in academia. Because right now, there still is this idea. I don't know if it's similar in Jeremiah's field, but just the idea that you should be a top researcher and a professor that's publishing a lot.

0:47:07 Matthew: And that's kind of like what's expected generally. And if there was maybe alternative pathways, now that we have millions of PhDs and, you know, there's all these other niches to be filled, like good teaching and good science communication, maybe that would alleviate a lot of different areas if there were just more understanding that there's a lot of different ways that we can all contribute to science.

0:47:29 Jeremiah: Don't know, I, I think most universities justify paying professors by a lot what they teach, so that's just kind of all it's tied to, really. And that's, that's how they justify giving me a stipend, is we TA classes. And so, you know, that's all there is to it. There's a lot more to this topic, but of course we're constrained for time, so.

0:47:52 Thomas: I would love to get more in depth at another time. Maybe if people have questions, if there's more to look into, maybe we can do that. But okay, for the time being, I want to thank you, Matthew Facciani and Jeremiah Traeger, for coming on, giving us non-academics, giving us plebs, kind of an indication of Publishing and how it works, because I really, if you're not in it, you know, it's hard to know. And I think that that puts a lot of context around a lot of what is being talked about lately. So thanks for giving us that perspective.

0:48:09 Matthew: Yeah

0:48:10 Jeremiah: Plebs you plebs yeah you're welcome I'm gonna go right back to my ivory tower right after ending so yeah

0:48:33 Matthew: Thank uh

0:48:35 Thomas: Oh, make sure, yeah, make sure to go ahead and plug where you want people to find you.

0:48:41 Jeremiah: Uh, you can find me on Twitter at gerbivore, uh, and uh, as far as like my science stuff, I contribute to A Tippling Philosopher on Patheos. Thank

0:48:52 Thomas: Awesome. Yeah, I'll have those links in the show notes too.

0:48:55 Matthew: For me, yeah, you can look at uh my Twitter and Facebook, um, Matthew Facciani, for both of them. I have a pretty unique name, so if you just google it'll pop up. Um, I also have a blog on Patheos, According to Matthew, and I just started my own podcast also called According to Matthew. So, I talk about, you know, science a lot and politics and social justice, religion, all those kinds of things. You can find it on there.

0:49:21 Thomas: Excellent. Well, thanks very much for joining us today and we'll keep in touch.

0:49:26 Matthew: All right, great. Yeah, thank you for having us on.

0:49:26 Jeremiah: You thomas

0:49:50 Thomas: All right, very interesting conversation. Thanks so much to Jeremiah and Matthew for that one. It gives a bigger picture of what's going on, more to work with when talking about things like hoax papers and the like and also more in-depth conversation about publishing in general. So thanks for that. That was really fun. And I have to say there's a lot to talk about Thursday. I'm excited to get to your voicemails. I got a lot pending and also Peter Boghossian has lost his mind, so I'm gonna... Get to talk about that a little bit. That's always fun. But no, lots to talk about. I will see you Thursday. Thank you so much for listening. See you then.