SIO477: Debunking Bad AI Research, and Bad Coverage of AI Research

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:12 Thomas: Hello and welcome to Serious Inquiries Only. This is episode 477. I'm your host Thomas Smith and today we're going to be talking about AI. And to some extent we're gonna be covering some basics like what the hell is it, what are these large language models, what aren't they, and things like that. Our guests today are Alejandro and Julia from theluddite.org. You've heard Alejandro on the show before. And they write very critically, very interestingly about AI and about AI studies and how a lot of them are very, very very bogus. And not to mention the reporting, which when a study is good is often misleading, but when a steady is kind of suspect, who knows what the reporting on it will be and how closely it will resemble reality. There's so much interesting stuff here. Seriously, fascinating conversation. I, for one, I'm pretty impressed by AI and there's certainly things that it's very good at. And it's really interesting to think about. It's really interested to think about what it is and can it really ever be considered to be like thinking in a real sense can it be a person in any sense eventually that kind of thing or is there a limit to what this exact kind of technology can even be those are tough questions to answer but where I think Alejandro and Julia are really getting at something is in taking like an anti -capitalist look at it and seeing a lot of danger signs about it and a lot of the misalignment of incentives going on. And also, it's one of my favorite kinds of SIO things, which is debunking some bad studies, or at least badly represented studies about AI and what it can and can't do. All right, well, I'll take the usual break. Please support the show at patreon.com/seriouspod. Seriously, it makes a huge difference. It's the only reason the show continues. And you can avoid those ads by signing on over there. Thanks so much to the patrons who make this happen. All Right, we'll take a break and we'll get on over to Alejandro and Julia from theluddite.org.

0:02:13 Ad: [Ad break 1 — ad music]

0:02:13 Thomas: Alejandro and Julia, welcome to the show.

0:02:15 Ale: Hey, thanks for having me.

0:02:16 Julia: Thank

0:02:17 Thomas: I had so much fun last time we spoke, but it's been a while. Do you mind kind of reintroducing yourself and Plus One?

0:02:22 Julia: You and

0:02:23 Ale: Yeah, sure. I'm Alejandro. I write about tech on TheLuddite.org and I do so with my awesome partner, Julia, who is here with me.

0:02:32 Julia: I'm Julia and I am a PhD candidate at the Complex Systems Institute at UVM in complex systems and data science.

0:02:40 Thomas: So right up your alley.

0:02:42 Julia: Yeah, my thesis that is due imminently is tentatively titled Fundamental Salience in LLMs.

0:02:46 Thomas: That's really cool. I'm excited to have you on. Appreciate you coming.

0:02:52 Julia: Yeah. Thanks for having me.

0:02:54 Thomas: Yeah. So where do we want to start? Do we want to just start with kind of the basics?

0:02:57 Julia: Sure. I can give a quick overview of LLM. So large language models or LLMS are AI or machine learning models that take in huge swaths of text as training data and create new utterances by choosing the most plausible small pieces of text to output next. And those pieces are called tokens and they can look like letters, words, or sub words.

0:03:19 Julia: So these AI models are called deep learning models because they're relatively unsupervised, but unsupervised is kind of a loose term because it sort of sounds like there's no human in the loop. And actually that is very much not the case. Basically they take in lots of input and learn in a relatively unsupervised manner. The parts that go into them are word embeddings and neural networks. And then there's an attention mechanism that allows the word embedding to be contextualized. So you could also think of them as a latent space of information plus a way to get a probability distribution over some kind of output

0:03:56 Thomas: Think

0:03:56 Julia: Pieces out of that space.

0:03:58 Thomas: You've lost us.

0:03:59 Julia: Okay, I'm sorry.

0:03:57 Ale: Why don't you tell the story from how training data becomes an utterance? Maybe tell it that way.

0:04:06 Julia: So we want these models to learn from natural text, but everything they know how to do is situated in numbers and and electrical impulses because they're computers. So we need some kind of bridge between the naturalistic text that we're giving them and what they know how to operate within. So the tokens are sort of the bridge between those two things. So when the LLM takes in input, it parses it into small tokens of text. And then each of those text pieces has a vector corresponding to it. And then the way the model understands what an utterance means, it passes that vector from layer to layer throughout the model and updates the values in the vector as it goes.

0:04:49 Ale: So to summarize, there's a bunch of texts, it crawls the internet, and then it grabs all this text and it tokenizes it. Now the model itself has no concept of a character or a word. It speaks tokens and tokens are something that it sort of invents for its own ability to consume text and turn it into memory. That goes into like a giant matrix, and then that matrix is a bunch of math, and that math spits out utterances.

0:05:17 Thomas: It's crazy because I think back in the day, correct me if I'm wrong, I don't really think this was necessarily the way that we all thought this AI thing was going to go. It seems like back in the day, I was like, someday the computers will get so good, they'll know stuff. But I don't, at least to the layperson, it felt like, oh, wait, is this the thing we were thinking the whole time? Like, is that when we make the future movies and there's the robots that are sentient or whatever, you know, it's like, is this what we had in mind? But on the other hand, it kind of seems like it could be. I mean, what is a human but a thing that learns the tokens and predicts the next token in the series? Series. It's like, is that really so different from our process of learning? I have kids, it's like they at first cannot predict anything that's going to happen. They don't know that something's going fall off the table. They know nothing until they see that happen and they get good at predicting the next thing. It is kind of weird. I have had both those thoughts and I would be curious for your thoughts on them. Can this really be what we consider like sentience or artificial intelligence or is this some sort of just, this is just ones and zeros in a different

0:06:18 Thomas: Way that doesn't really meet that expectation.

0:06:21 Julia: I agree with both of your thoughts. So I think language is somewhat of a beguiling medium because we're so used to encountering it from another person, in which case you're licensed to impute these things about what what their internal state is like to some extent. And that makes it sort of confusing to encounter something that speaks really fluently, but doesn't have a human mind behind it. And I think that can be sort of misleading because I definitely don't think that these models think and understand in the way we do, but I also don't Think it's fair to sort of portray them in the purest stochastic parrot form of the argument where you're just thinking of them as knowing nothing but ones and zeros, sort of like maybe like Searle's Chinese Room. I think that's too far on the other side of it.

0:07:15 Thomas: It is so fascinating to think about this stuff. This feels like an area where philosophy has such a direct, I don't know if impact's the right word, but it's like, there's not a lot of work I imagine where it's, like, this is exactly what the philosophers were talking about. I feel like a lot philosophy is, like divorced from practical reality, but now with this AI, it's ,like, well, I dunno, what does it mean to be a person? What does it means to know something? What is it mean?

0:07:36 Julia: Yeah.

0:07:36 Thomas: It's like all those questions. I mean, how much of that is part of your education you've gone through for this or, you know, how like how present are those philosophical thoughts and arguments or is it more approached technically all the time?

0:07:48 Julia: It's very present in the work I've been doing. It'S something I think is really interesting. And the program I'm in has been fairly flexible in sort of letting me drive the interest that I'm chasing. And it's something that these kind of topics, me and Catherine Kramer have worked on a bunch. And also there's a lot of people at the Complex Systems Institute, like Professor Josh Bongard has like a wonderful class on mind morphology and machine that really dives into exactly the kind of questions you're raising. That was a really, really fun class. But yeah, I think it's like really important to keep in mind with respect to the AI hype stuff that we sometimes run into. At what point can we say that the therefore in between we have a machine that talks, therefore we have a machine that thinks. When do we feel comfortable saying that therefore is licensed? I think right now that therefore is happening not always thoughtfully and sort of just being invoked. I think it's something that's really worth interrogating carefully.

0:08:46 Ale: The fact that philosophy does impact this is going to be like a through line for this entire conversation. One of our major themes for going through these studies is going to be a call for a doing theory on these things. So these are not just black boxes where we pull levers and get data. We should think about, we know how they work, we made them, and we should analyze them as such.

0:09:06 Thomas: Do we know how they work? So you mentioned that and I hear a lot of, you knows, stories about like, oh, well, you know, it's hard to even explain where it comes up with this output kind of thing. And it also feels to me, again, as someone who doesn't know anything about how directly these models are made in a technical sense, it does seem like it would be pretty hard to supervise this kind of thing. Like I know, I know that they, they tried to put, and I'm sure you could tell me this a lot more accurately, but I know they've done things like try to put controls on certain things. And the best they can do is like, well, if it seems like the topic is this kind of thing, then we'll have it override with a message like, Hey, we're not going to talk about that. But it's like in a sloppy way that doesn't really work because that strikes me as very downstream and after the fact. And that I guess I took that to mean that it's kind of would be really hard to practically speaking, influence or supervise LLMs to the degree that you could kind of cut off an entire topic or something like it just seems

0:10:03 Thomas: Like it'd be really difficult. And I thought that was because it's kind of hard to have a person in the room. So what did you mean when you said there are there are humans there are, there is supervision, I guess.

0:10:03 Ale: Before you get into that, I want to draw a distinction between we know what works and we can make it do exactly what we want, right? Like we know a car works, but I can't get it to tie my shoes, right. So people make LLMs, their code is written by people, but because of their nature, it's really hard to get them to do specific things. And I passed to Julia because that's squarely in her court.

0:10:32 Julia: Yeah, I think you're right in being skeptical of alignment methods that are glued on top of the model because the innovations that have been incredibly powerful in creating human -like language performance in LLMs. One sort of key component of them is reliance on generating plausible text that's very baked into the goals and the objective functions in the model itself. And as long as your goal is to generate plausible text, you're not ever going to, I think, be able to control, like you were saying, the output all Because it's perfectly plausible to say things which are not true or which are harmful or rude or offensive. Those things are still perfectly plausible in a lot of ways. So I think something like RLHF, reinforcement learning from human feedback, would be an example where the model you end up interacting with very much has people in the loop. Because there's an objective function added onto the model that incorporates human judgments of the model's responses. And then it penalizes or rewards the model for how good that response was. But those things, even though they're trying to modulate the importance of the plausible text generation objective function that's already in the model, they're not overriding it. As long as that's still the core, as long as plausible text generation is still the score of what the model learns to do and tries to do, I think you're never going to have like a hundred percent ability to have that align with saying something true, saying something accurate, saying something polite, because that's a different goal.

0:12:12 Thomas: Yeah, I'm curious. Maybe this is a little field, but But OK, so something like we've throughout the past however many decades we've been trying to do different things with computers or I guess AI in a way like can it beat the best chess player and that kind of thing. And then like recently, I remember somewhat recently, the bots got so good at the game Go, which was thought to be like way more complicated and more difficult to master for a computer than chess is. And I remember hearing that like it was doing things we couldn't really even explain, but they were better moves. Is that the same underlying thing, but just the token is Go moves instead of words? Or is that a different technology or is this irrelevant in a different topic?

0:12:20 Julia: I don't know anything about how the Go model works, but the problem it's trying to solve, I would think is fairly different because with human -like language performance, it's open -ended and there's no complete map of the combinations that are going to be valid in the same way that there would be for a chess game.

0:13:17 Thomas: Theoretically, yeah, yeah. It's super, super, súper vast and complicated. So it's not like they can map out every possibility. And then Go was, you know, like I said, it's thought to be even more so. So it wasn't the kind of thing where they could mechanically, like the earliest chess computers were mechanically programming in lines of play and stuff and like logic to to it, but that had a limit. Okay, so when you talk about rewarding a model, does that just mean like upping the probability it places on a certain thing, or what does that actually mean?

0:13:30 Julia: Not directly. A lot of times the way the models work, there's some sort of cost function that's assessing the quality of the answers. So you have like some examples of what good answers look like. And when the model gets close to that, then that's like a relatively good score. And when the model gets far from that, it gets penalized according to the cost function. That feedback helps the model set the parameters. So the biases and the weights connecting the nodes in the neural network, but they're not like directly on the output probability of the tokens, like at the final output layer, at least not like typically.

0:14:25 Thomas: I think we kind of got some of the basics there. It's really complicated, so it's a hard thing to have the basics, but maybe one more question along those lines. Does it feel like there's a limit to this sort of, I don't know, for lack of a better word, kind of technology, like this kind of intelligence? If all it's doing at its heart is just taking in massive data and predicting the next thing, essentially, does that sort of preclude it in a way evolving deeper intelligence? Does that fundamentally limit what it's capable of, do you think?

0:14:51 Julia: I think so.

0:14:51 Ale: I just want to jump in. I agree very strongly.

0:15:00 Julia: Yeah, I think so. But the shape of that space, I Think is not known like the upper bound of of what that looks Because the way I think of it, the LLM can get pretty good at learning enough of the meaning of words that it can use them in a fairly human -like way. That's not to say that it learns the meaning of the word in the same way we do. It doesn't have a body, or at least not a very similar one to ours. So it's not associating sensory input or memory of its past experiences with words or anything like that. But when we see such convincing and impressive linguistic competence as it displays, I think that's compelling evidence. And when we look at the sort of information that the tokens themselves can encode in their structure, it's pretty detailed and pretty fine -grained.

0:15:54 Julia: It doesn't seem, I mean, I can't say 100%, but it doesn't seem like it's the absolute minimum amount of information you would need to encode in order to just say a good thing that makes sense. It seems to reflect more meaning than that. But it's also possible that any structure that's sufficiently performant would encode meaning. So it's a little I'm not sure. But there is evidence that LLMs let's say they can establish sort of this base level of meaning sufficient to be able to produce linguistic output that sounds reasonably human -like, it seems like they can actually step beyond that to some extent too, because once they have that level of performance, it does seem like they learn from approximations that are provided to them in text. So if you provide new information to the LLM, it does seems to be to take that in, which sort of implies that that whatever the minimum level needed for speaking in a reasonably human -like way is, they've met that. And then they can go at least some level beyond that. But then there's a ceiling of what are all the things that you could provide explanations in text for, right? Because that's not going to be equivalent to sensory experience and being a person. And there's all sorts of information you couldn't encode in a textual approximation, but you can get arbitrarily good probably, if

0:17:22 Julia: That makes sense.

0:17:23 Thomas: It does to me. I don't, it does make sense. I guess the one thing I didn't totally understand there was what was the part where they're taking in more, more somehow in a way that's different, gone

0:17:31 Julia: I think there's reason to think that they can keep learning even once they've sort of, yeah, and achieved like they sound like a person, but they seem to be able to keep learning beyond that. It's all combinations of, I mean, maybe not all, but the things I'm aware of are combinations of architecture and more training data.

0:17:40 Thomas: Through all their teaching materials. Right.

0:17:57 Julia: So giving it more data, but also how you structure providing that data to the model also matters to how much information it can absorb out of that data.

0:18:07 Thomas: Interesting. Is there any like spooky stories about what these models have done? I mean, is there anything that sticks out in your mind that has like been hard to explain or that people have been thinking a lot about in terms of their performance.

0:18:17 Julia: I think their spookiest performance is in conjunction with people using them in ways I find alarming. The way people invoke them and use them is much more chilling to me than anything I've seen the actual model do itself.

0:18:25 Ale: So I have a lot of studies here that I want to go over with you. What I'd like to do, can I paste these headlines for you to read?

0:18:40 Thomas: Oh, this is the Washington Post. I've heard of that one. Chat GPT is a poet. A new study shows people prefer its verses. Hmm, sure.

0:18:49 Ale: Why don't you give me the next one?

0:18:50 Thomas: Let's look at the next one. The Guardian. AI poetry rated better than poems written by human study shows. Wow.

0:18:56 Ale: Yeah. Okay. So there were a ton of these recently. This was just like a few weeks ago, maybe a month ago, this study came out. We're gonna have all these headlines, and we're going to look at the study behind them. So I just pasted that for you to read. I can read this headline for you. So you don't have to keep working so hard. AI generated poetry is indistinguishable from human written poetry and is rated more favorably. So that's okay, it's just

0:19:06 Thomas: Kind of seems like both those things would be hard to be true, but I guess not. Yeah, I mean, I guess you could say, like, if someone's a really good human poet, they could both be preferred on average and indistinguishable in terms of, like... They blend in with the rest of the humans, I suppose.

0:19:20 Ale: Ah, that was Yeah, you've got it. So it's a good dunk. That's the dunk I was going to do also. But so what they did was this is a study where they got they did two, they kind of did two studies in the study. So the first thing they did was they created a data set. So they had chat GPT, they gave it 10 famous poets, like, you know, Shakespeare, Chaucer, like these sorts of things.

0:19:48 Thomas: Did they do any good ones, too, or... No, I'm just kidding. Take that!

0:19:51 Ale: Ah, got him. I'm sure Chaucers really feeling that from the so they got these 10 poets, and they told chat GPG to generate poems in the style of these poets, right. And then they went and they did, they got some people, and we're going to explain how they got those people in a second. And they asked them to say, okay, which one of of these was written by a person. And then they did a second batch of studies, which is they got those same poems, and they asked people to rate them. And then what they concluded is that people can't tell the difference, but they do rate the poems more favorably. So I want to talk about a few claims they make here. So the first big claim they make is one that Julia already touched on, which is human out of the loop. So they really harp on this in the paper. And the idea is that generative AI has in so many fields, it's now more human than human, which is the other concept that I want to talk about. It's become more human -than -human in all of these fields, and it can produce these outputs that are more human then human without a person in the loop, right? So that's like sort of like the big claim of the paper. So let's talk about it. First, I just want to throw this right off the bat. In their dataset, one of their poems is just the opening of the the prologue of the Canterbury Tales, the one that ChatGPT generated. So just like right off the bat, their data set is fundamentally flawed. ChatGT just like vomited the first thing that Chaucer wrote. Yeah, it it's sort of quote unquote plagiarized.

0:21:08 Thomas: Oh. Oh, so, like. So, like stole something. Like. Okay. OK, that'd be interesting, because, yeah, just thinking theoretically, if they're like, I got this computer that will be a better poet than all the humans and it just selected the best poetry and copied it, you'd be like, all right. But it's not really like that's just gathering the best stuff we've done. That's not Really like coming up with stuff in a new artistic way.

0:21:17 Ale: So that's let's just put that to one side.

0:21:37 Ale: Yeah. Totally. So I'll leave that point aside because I want to get to it later. So in the first half of the claims, right, the idea that the poetry is indistinguishable, there's this really wonderful paper by Ernest Davis that I guess I can give you for show notes or something. I'm going to quote actually, the simple rule that a poem was written by ChatGPT, if it is either a Shakespearean sonnet, a single couplet, or consists of four stanza verses, and follows both an AABB or ABAB rhyme scheme, will have an accuracy of like 88%, right? So the idea that it's indistinguishable, it's not indistinguishable because it's very easy to come up with a heuristic. So now we have to ask, okay, why did the people find this to be indistinquishable? Right so if you actually look into the study what they did was are you familiar with this service called prolific do you know what mechanical turk is okay prolific is like so what they did was they got these underpaid click

0:22:21 Thomas: I can't remember. Yes. OK, I was thinking it was going to be like related to that. Yeah. Right.

0:22:29 Ale: Workers and they paid them prolific says they can make like up to eight dollars an hour or something so this is just like like you know yeah

0:22:35 Thomas: Oh, shit. I'm out of here. See you guys later. Yeah.

0:22:37 Ale: Yeah this is like the like worst like underpaid quick work people they get paid by the task and they just like ask these people. So, I mean, I think it's pretty obvious why those people wouldn't be able to distinguish between the two, right? Like these people aren't taking, that's not what poetry is. Yeah. So they, they do get into that. They do get into that. And they say, they say that, you know, this is like normal people, but I even think that is like kind of a ridiculous claim because it's people under very specific conditions cannot distinguish between like, like if you are, if you are

0:22:52 Thomas: And not to be like elitist or anything, but when you're talking about any sub branch, like you would want to know, can the experts differentiate? Not can just people. Okay.

0:23:12 Thomas: Yeah. Yeah, yeah. People getting paid not enough to do like the kinds of people who need to do that. Yeah as quickly as possible to jam it in for sure.

0:23:18 Ale: To do tasks as quickly as possible, as quickly as possible. And then so I think right there, I think the distinguishability claim is sort of nuts, right?

0:23:26 Thomas: Yeah hmm.

0:23:26 Ale: And then the other claim where they prefer it also like just directly follows from the conditions of the study, right. Like if you pay people and you ask them, do you like this poem? If people are in a hurry, they're not going to take the time to study something that has a complex form, right, they are not going take the times to to really understand, they're just going to be like, just doesn't make sense. I hate it. You know, next thing, right?

0:23:32 Thomas: Hmm. Yeah.

0:23:51 Ale: Which is like, if you think about what poetry is, or even like a good song or any like piece of art, you don't just like go to the museum and like speed run the paintings, right. That's just like a silly, a silly concept. So I think that what I really want to highlight from this study is that what they've done is they've taken this thing that like, they sort of black boxed it, and then they got this gig work thing. And they sort sort of like bolted these two together. And then they concluded that because of this like unholy contraption of an experiment that like isn't really anything, right? Like, like, what does this mean? Really? This doesn't mean anything. You just proved that like people in a hurry don't like poetry. Of course, they don't. And dann from there, they went on to tell a story. And the story they tell is the AI is just as good at poetry as people. And than they went so far as to claim that the poetry AI can make poetry with a human out of the loop, right.

0:24:42 Ale: And that's like a really really bold claim. Like the human is all over. They literally put humans in the loop in the study, right? They asked Chatubiti to do a poem in the style of a human. That human is in that loop. This is just like a, this is like such a complex fiction that they've created. That's like purely just like their own like ideological imposition. This is like just a failure of theory, like a profound failure of history. So then they write this as a paper. It inexplicably gets through peer review. I mean, not inexplicably, we can go down this path.

0:25:12 Ale: But like, academic publishing is a nightmare, sort of It's Scientific Reports. It's a nature subsidiary. So it is like a real peer review journal, but it's not like a top tier journal. But then that gets that goes out. And then all these outlets pick it up. And then we have like, you know, everybody's feed is filled with like AI poetry is better than humans.

0:25:14 Thomas: Is it a major journal though? Okay. Yeah. Yeah. Ah, that is the, that is the information ecosystem we live in.

0:25:44 Ad: [Ad break 2 — ad music]

0:25:45 Ale: I want to do another study. This is we're going to be on a medical theme for a minute here.

0:25:50 Thomas: So is this, I'm, this is not a headline then, right? This is this actual study.

0:25:53 Ale: No, this is we are going to go straight to the study sometimes. Yeah, yeah.

0:25:54 Thomas: Okay. GPT versus resident physicians, a benchmark based on official board scores. Huh?

0:26:01 Ale: So I really like this one. Because so remember how in the AI poetry study, they tell the story as like a human out of the loop entirely. But in reality, like they literally put a human in the loop. I mean, in the one case, the AI did literally nothing but like regurgitate a Chaucer poem. But like, even for the other ones, like the human is in the group, because the AI didn't make its own style, right? Like poets are supposed to have their own style to put another poet style in. And then that's not even to talk about the fact that like a A human was in the loop because a human posted all the poems into Prolific. Like people had to do a ton of physical labor to make this work, right? So I want to really like nail that home here. So if you look at this study, here's a quote from it. Given the maturity of this rapidly improving technology, the adoption of LLMs in clinical medical practice is imminent, which is like, yeah, really strong language. I'm going to do one more. GBT -4 marks a critical milestone in which LLMS achieved physician level performance. Performance these findings underscore the potential maturity of llm technology urging the medical community to explore its widespread application and so so how they did this study was they they just got a bunch of like medical exams like I think they're like board exams and they just gave them to chat gpt and they gave them two like resident physicians and they concluded that chat gbt did better but here's this like damning little sentence buried deep in like like the, like, I forgot what section is, but it's like halfway through the study because GPT models cannot interpret images questions, including imaging analysis, such as those related to ultrasound, electrocardiography, X-ray magnetic resonance imaging, computed tomography and positron emission tomography,

0:27:36 Ale: Computed tomography imaging were excluded.

0:27:38 Thomas: It's like the side effects at the end of a medication commercial

0:27:41 Ale: Totally. Totally. Yeah. It's it's I think this, I really want to talk about this one because if you read, I mean, come on. Like, yeah.

0:27:48 Thomas: Yeah especially yeah once you distill it down to that it's almost like well okay the answer key also does that you know it's once you distil it down we're just going to take the stuff where we have a textbook that has a teacher's edition and the chat gpt does really well at just telling us what's in that it's like okay the soda is the book like that's that's not really feels a little bit

0:28:08 Ale: Right. Right, exactly. It's I had a friend tell me that it's sort of this is the equivalent of saying like pencils can do well on exams. It'S like, yeah, because you were holding it. Yeah.

0:28:19 Thomas: Underwhelming minimal human intervention a pencil can do

0:28:24 Ale: And this is so this is this is going to be a theme that we're going to run into. Julie, why don't you talk about language games?

0:28:29 Julia: Yeah, so one thing we've noticed with a lot of AI research is that, so we talked earlier about the objective function of the model, which for LLMs is often generating plausible text. So at their core, we've set them up, we've trained them, we're taught them to be really good at generating plausible texts. So one of the strategies that a lot of these researchers do, I think sometimes on purpose and sometimes without noticing what they're doing, is they set up a complex task. They set up representation of that complex, often embodied task as a very simple language game of generating plausible text. Then they give it to the thing that is very good at generating plausible texts.

0:29:09 Thomas: Great at that yeah wow

0:29:17 Julia: Then when it does, in fact, generate plausible text, they conclude it can actually succeed at the very complex task, you know? I think this study is a really good example of that.

0:29:29 Ale: There's one study that we covered on the blog about it was back in like the kind of really early days of chat GPT. And the headlines were chat GPT can be an entire tech company. It can do the whole thing. It could run a whole company for under $7. And then if you like, if you actually went in to look at what the study was doing, it was like, they were setting up all these different chatbots to like talk to each other. It was just like absolutely bananas. But then of all of the problems with it, the one that really spoke to me as like a software developer was they had this study that was like like 85 % of the code that I generated like executed perfectly. And I just like, I want to explain to non software developers, like, that's, that's so they described it like we got this code. And we did all the work of setting up all these dependencies. And then we ran it and like 85% of it executed and like there is no software repository in the world where 15 % of the code like cannot be executed. And no one knows what like, That's, That'S like the most horrific. That's like a nightmare. I like can't imagine anyone coming to the conclusion that that. So you know, it really shows like, like, if you set up the way I described it in that post is like, the way these studies are set up, it's the equivalent of like, you design battleship, the game battleship gets a game for children to play, give them battleship. And then when they can play it, you conclude that like children can run navies, right? That's like sort of what they're doing here.

0:30:06 Thomas: Useless useless. Yeah. Okay.

0:30:47 Ale: This will be a quick one. Here's a New York Times.

0:30:49 Thomas: Here we go. New York Times, AI chatbots defeated doctors at diagnosing illness. Okay. I mean, I find that plausible because like the way we diagnose stuff, the differential diagnosis -y stuff does seem like an algorithm really. So it seems like plausible to me that, especially because the AI can take in so much information and know every possible possible disease and ailment and whatever. It seems plausible to me.

0:31:14 Ale: I agree that it is plausible that you can give an LLM a set of symptoms, and it will be very good at diagnosing things. But we'll get into that in a minute. For this specific study, I just want to point out New York Times runs the headline AI chatbots defeated doctors at diagnosing illness. Exactly.

0:31:31 Thomas: Doctors had to turn in their badge and gun and stethoscope.

0:31:35 Ale: Yeah, this study was titled large language model influence on diagnostic reasoning, a randomized clinical trial. And then here is you don't even need to go into the study. It like has like a key point summary. And in the findings, it says the use of an LLM did not significantly significantly enhanced diagnostic reasoning performance compared with the availability of only conventional resources. And then it also has, it also has, you know, it remains unknown whether use of such tools improves physician diagnostic reasoning. In the test, it like did a little bit better than doctors, but like it has like all of these like caveats and other things happening in it that just like get instantly flattened, right? When like the media runs a thing.

0:32:07 Thomas: It sounds like it's saying doctors plus basic, you know, whether it's a reference book or the internet can also do that is what it sounded like it was saying, but I don't,

0:32:20 Ale: Yeah, yeah, but if I remember correctly, one of their things was when you give doctors an LLM, they actually do if I remember correctly they actually do worse than they did before

0:32:29 Thomas: That kind of makes sense. I'll be honest. I've tried to use chat GPT for some things and it's mixed results. It is like hard to get good. Yeah. It can be really useful sometimes, but it just really depends. But it's also interesting because I do think, well, branching off, or we could have talked about this in the beginning, but I do Think that it may be that we need to evolve a certain skill or learn a certain Skill, which is using LLMs. That'll be kind of its own unique thing. And it'll be about knowing their uses, their proper uses, and when they're actually going to be good. And And that in itself, in the same way that, like, everyone has Google, but not everyone can do good research. You know what I mean? Like, maybe it's going to be just another iteration of that.

0:32:38 Ale: That's also my experience

0:33:15 Julia: I think that's true to some extent, especially because there's a really big space to be explored now that LLMs have proven the efficacy of their basic architecture. There are actually many parts of that architecture that were initially under -constrained. We chose them in a relatively arbitrary fashion. So there's now now a lot of space to explore what sort of juice we can get by changing those things within the model. And I think also concatenating different models together, like some people have, you know, attached retrieval augmented generation. And I think that has a lotof promise to be helpful.

0:33:53 Thomas: What is that? I don't know what that means.

0:33:55 Julia: Like when you're Google searching something, you want it to bring up a document that's relevant, and an LLM lets you search in sort of like much more natural language, theoretically, you can you can kind of describe what you're talking about.

0:34:03 Thomas: Mm -hmm. Right, right, right. Yes.

0:34:06 Julia: But if it could also return to you a list of relevant documents, that would be great. So that sort of strategy.

0:34:13 Thomas: Finally, I don'T have to use Ask Jeeves anymore.

0:34:16 Julia: And there's also tools being developed that are, the idea is that you can make a pipeline where they can help you check expectations for what the object outputted by the LLM is going to look like. I think there's promise in that as well. Like if you know, because it's hard to take in a lot of of information and get something useful and formatted reasonably out of it. And LLMs can be really helpful for that. So if you have another step after that, that helps you verify that the thing which was now produced is what you wanted, then I think that can be really good. I use chat GPT a lot for like formatting citations that are in plain text into BibTeX.

0:34:56 Thomas: Oh, God. Yeah. It's so great at that kind of thing.

0:34:59 Julia: I do think they're They're really useful. And, you know, they're really cool because we've spoken language for, you know, 100 ,000 to a million years, something like that. And that whole time, it's always been other people, pretty much.

0:35:10 Thomas: And the occasional parrot.

0:35:14 Julia: Right. So now, like, LLMs come along and computers can speak pretty human -like language, too. Like, that's pretty incredible.

0:35:21 Ale: Yeah and I will say on the on the like usefulness thing I think there is a little bit of cold water to throw on it which is this is something julie and I have written about mostly mostly julia actually one of the problems with lms as opposed to like other tools is that because they're so general purpose and like can be used like right like for example kids are using it to like write their essays or whatever ever.

0:35:40 Ale: There's kind of like two problems with that, right? One, there's like a massive like de -skilling happening, right, like writing is like an important skill. And then on the other end is it's useful for these things, but like at what cost, right. And these are like, besides the de -killing, LLMs are only useful if you like fully externalize like the destruction of like the information ecosystem, the destruction Of the literal material ecosystem of the world, right they like use a ton of energy.

0:36:03 Ale: So like, they're sort of like useful, but only if all the negatives or like externalities to you. And I think the usefulness quickly goes down once you realize their destructiveness.

0:36:12 Thomas: Yeah. Well, that's got to be another whole part of the conversation. I don't know a lot about that. If you did want to discuss the environmental impact, I don't know. I've heard obviously about that, but I haven't looked into too much detail on that.

0:36:25 Julia: Yeah. So from a paper by Shumaila Vidal in 2024, they found that, or they mentioned that, this was not the main point of their paper, but that that training a single moderately large model produces twice the American lifetime's worth of CO2.

0:36:40 Thomas: So if you try to be fair to it, do we get more usage out of it or something that we, you know, like how do you actually apples to apples that comparison compared to, I don't know, we're all going to use a ton of Wikipedia in our lifetime or something. I don't know.

0:36:52 Julia: Yeah, I think a lot of these models are being trained relatively single use. I don't know that for sure, but I think because of the way we have this obsession with the aesthetic of experimentally verified science, I think there's a lot of models being trained for one-off experiments that probably don't have a long lifespan after that. Yeah.

0:37:13 Thomas: Okay I thought the training was like the whole coming up with it like a version like a new oh the new

0:37:17 Ale: When Julia says a moderately sized model or a medium, I forgot what word you used, Julia, but like that's like, that's not like ChatGPT is like jumbo enormous. We're talking like you're like State University Research Department is spinning these up.

0:37:31 Julia: There's a lot of them and it's a bit depressing These new models for generative AI video production that just dropped recently. It is such cool technology, but the unconscionable amount of largely unacknowledged and invisible resources that are going to be mindlessly consumed to create unnecessary AI output that's probably being made, at least according to even the commercials put out by some of these companies. It's created in the service of this like pointless capitalist make work. And then after that, it's discarded and forgotten. And for that, we're like accelerating our climate apocalypse.

0:37:46 Thomas: Gotcha yeah

0:38:09 Julia: It's depressing.

0:38:11 Ale: You compared how much is it compared to say, how much energy does it use compared to, say, like a Wikipedia lookup or a Google lookup? And it's, you know, we don't know for sure because OpenAI is paradoxically and ironically private and actually extremely closed, but we know it's orders of magnitude.

0:38:24 Thomas: I want to make very clear obviously I'm concerned and fully willing to believe that these things are using a ton of energy I guess my skepticism comes from like I don't know how much fucking shit this is always using I DON'T KNOW HOW MUCH THE CLOUD IS USING I DON'T KNOW how much the just the whole internet any of that stuff I don't know what to compare it to exactly. And I don't know, like it would have to be that they made a whole bunch of new infrastructure to be using this additional energy, right? Or is it that the, okay,

0:38:54 Julia: They are doing that.

0:38:56 Thomas: So it's that, oh, nice. Well, that's good. Not even ironically, I think if we actually started using nuclear, that would help a lot of the greenhouse gas part of it.

0:38:56 Ale: They are. Yeah, a Microsoft Baga nuclear plant.

0:39:05 Ale: Well, for this case, just to throw cold water on this one, Microsoft Bada nuclear plant, but the nuclear plant... My understanding from reading an article about this way back is that not only can it not actually provide the energy that it needs, like the scale is off, both the time and the energy scale is on, so it can't provide the energie it needs. And then on top of that, it takes like 10, 20 years to get a nuclear plant going. And like, maybe in 20 years, it'll power their data centers. But like, they're running out of energy like now.

0:39:33 Thomas: I think I read that same article and I will say that's a separate topic, but there's things we could do to make nuclear more easily accessible. And that's something that I think separately we should be doing. But point taken, they're making new infrastructure and that's taking a ton more energy.

0:39:47 Ale: And water, don't forget water.

0:39:47 Thomas: Yeah. Oh, sure. These things drink water, huh? Shit. Okay.

0:39:51 Ale: They drink a ton of water and there's like a ton of like local community conflict because they, you know, you want to put them in places that have access to groundwater, but then it evaporates easily because that's how you're cooling servers. But that means you're using up the groundwater.

0:40:03 Thomas: Wow. I would think if anything could use water that didn't need to matter that much, it would be a cooling system. Like can it use water that we aren't really using?

0:40:11 Ale: Yeah, they can, but it's more expensive.

0:40:15 Julia: Don't have the exact stats to back up oh I think a lot of this is sort of alarming and wasteful you know I don't have the numbers like at my fingertips but I think if it helps sort of illustrate why I think that there's a lot of arguments right now that for LLM or generative AI like development to continue models are going to need access to more information than exists in the universe and I think that sort of the appetite of the companies and creators behind these models, it's boundless. And I think it's sort of insane to be looking at, we need more data than exists in the universe in sort of an arms race of trying to outscale and outperform your competitors instead of trying to say, well, people read the same stories over and over and get new information out of them. Why are these models not able to take more information out of the content that's provided to them? It's sort of like this headlong, fairly irresponsible race in the direction that right now, like a greedy optimization of the space, like this direction looks most promising. We're dashing headlong into it, burning whatever we can along the way.

0:41:15 Ale: What

0:41:30 Julia: That's my impression.

0:41:31 Ale: We've argued in the past is that this like sort of like boundless, like greedy, single-minded appetite is sort of proof that they're like not serious. Like they're not actually like, they're serious about making large language models, but they're Not serious. This is not like a serious, like engineering tasks are constrained by reality. Right. So like to start complaining that you're running out of data, that's not the problem. The problem is your model is stupid.

0:41:53 Ale: Like if you're like running out of data is not a thing. We just have a certain amount of data. This Is like this resource that like is usable. Yeah.

0:41:53 Thomas: I'll admit I didn't even understand what was meant by that. Like what, I don't even understand like how, if the task is taking in and being able to learn from a bunch of data that we've produced, it seems nonsensical to say I need more than exists because that's what you have. That is the thing. I don't, yeah, it's nonsensesical to me.

0:42:04 Ale: Like they're exactly. Yeah, I think it's a mindset that you can only have if you have sort of like in the same way that earlier, you know, I was saying that the researchers for the AI poetry, sort of, like, they're so like, there in this, like mind space where they like, don't even see the human in the loop, right? They're like, so sort of the ideology of the AI hype, right, is sort of behind it. I think if you take that perspective, and you look at the thing, and you like truly believe that these things are going to become sentient or whatever, which Julie and I think are silly, is silly. If you believe that can happen, then you must, if you take that assumption and you go into it, then while you conclude that you don't have enough data, when like, actually the constraints of the problem, that's just a constraint of the program.

0:43:00 Ale: Like you have the amount of data that you have in the universe. You can't blame that, right? It's like getting mad at gravity because your thing fell. Like, it's silly. It's bad engineering.

0:43:16 Ad: [Ad break 3 — ad music]

0:43:16 Thomas: Well, we're actually pausing here because there's so much more to to talk about. There are a lot more bogus studies. And in part two, we actually get more into this really, the studies go from just like kind of bad to like sinister in a way. And the capitalist incentives are really bad. And then we also talk big picture about what all this means. So, so excited for that. Thank you so much for listening. And thanks to our guests, Alejandro and Julia. And I'll see you all for part two.

0:44:01 Thomas: Are you worried about getting sued by the large language model?

0:44:03 Ale: Yeah. Big, big AI. If I was going to be sued by Big AI, that would have happened already, I think. Julia, any thoughts?

0:44:11 Julia: It's not good.