SIO478: This AI Is Not My Girlfriend, It's My... Social Agent...

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:12 Thomas: Hello and welcome to Serious Inquiries Only. This is episode 478. I'm your host, Thomas Smith. All right, this is a part two of the really fascinating AI discussion with Alejandro and Julia from theluddite .org. Make sure you caught part one. If not, go back a tick in your podcast player. We're going to get more into some terrible studies, including probably the ultimate silliest and worst AI study, and really dig into the big picture of what's going on here from an anti -capitalist perspective or capitalist critical at the very least. Alejandro and Julia will also give us some key things to look for in AI coverage to make sure we don't kind of fall for the hype and the bogus stuff. A really useful little toolkit. All right, well, we'll take our break and then we'll get on over to it. Hey, please support the show. And thank you so much to the patrons who make the show happen. Patreon .com slash SeriousPod. Go there right now. patreon.com/seriouspod. Pledge as little as a dollar and you don't have to hear ads ever again. No more ads. All right. Thank you so much. And let's get on over to part two with Alejandro and Julia.

0:01:25 Ad: [Ad break 1 — ad music]

0:01:26 Ale: Study.

0:01:27 Thomas: Yeah, let's do it.

0:01:27 Ale: I love this one. This is one of my favorites. This one's really good because it really brings together a bunch of the pieces that we've talked about in like a couple of the other studies. So nature .com from almost exactly one year ago to the day, actually, I just pasted it for you. If you want to do us the honor.

0:01:43 Thomas: It says Google AI has better bedside manner. Oh, I saw this one. Yeah. Google AI is better bedside manner than human doctors and makes better diagnoses. Okay. Yeah.

0:01:53 Ale: Yeah, it's a bold claim. If we're going to come away with one thing from this discussion is like, you can just click through and read the study. They're not you know, it'S not in a foreign language. It'S not magic. Most of these studies, they're benchmarking studies, you understand what a benchmark is, everyone knows what it is, right? They're asking two people to think and you can just read it. And you can decide both if it's a reasonable thing to do, and if it's being reasonably reported. So let's talk about the study behind this one. So this is a Google study. It's on archive, which is a preprint server. And it's called towards a conversational diagnostic AI. So if you look at the actual study, one thing you'll notice is that everyone who works, every author, there's like, I don't know, a dozen or more, two dozen maybe authors on this. All of them work at Google. Yeah. And here, so this is being reported as a study, but this is product research.

0:02:47 Ale: And this is something that we're kind of seeing a lot of, and I don't see a ton of other people calling it out because the AI hype is so, it's kind of like so exciting. This is what I do for a living, product research, and it's not the same as science. You know, we kind of use the word research for both. But like, when you make a product, you don't want other people to reproduce it sort of like, but whereas when you Make a science you do, right.

0:03:12 Ale: So I just want to like highlight that the political economy of AI publishing, we've talked about that sort of, like media, right, the media loves this, like kind of benchmarking things. And then we've talked about these like, kind of like benchmarking studies, which seems to for whatever reason, they get through peer review, they get lots of clicks, that's really good for the status of the researchers. And then there's this other thing, which is this product research, this is like really good way at getting like your product in the press and getting a lot of attention.

0:03:35 Thomas: Think funding.

0:03:37 Ale: Yeah, and getting funding and getting you know, and you think about I mean, think about what this this nature .com article, which is about a paper. So nature is a peer reviewed journal. But it's also a company that owns a a magazine and a bunch of other journals. And then so nature, the mag nature, not the journal, but the magazine wrote an article about a Google, not a paper that's actually product research.

0:04:01 Ale: So that's what we're currently talking about. Yeah. And you think about what that does right now, a Google sales, someone who's, you know, whose product is based on this thing can go to your local, I don't know, your local like hospital technology acquisitions officer and tell them, oh nature wrote an article about it right and like that's just like a really like problematic dishonest way of doing it like product research and media so I really want to just call that out up front

0:04:25 Julia: And I think it gets a bit even more sort of tangled and pernicious than that because when we talk about models that are like state-of-the-art a lot of them are so big like the resources involved are so huge that only someone like Google or OpenAI is gonna even be able to do that so there's an allure to working with them even beyond like the cushy job crossover that could happen even just to have access to cutting -edge state - to the art models. I think that makes it a really gray area that I think a lot of scientists end up participating in what is essentially product research without sort of even realizing that they're doing that. A good quote from this paper by Templeton et al. 2024 just, I think, illustrates some of the problems with this. They said, Cloud 3 Sonnet is a proprietary model for both safety and competitive reasons, some of the decisions in this publication reflect this. So I think like right away, that's a red flag about sort of the ecosystem people are working in. The model itself can also be paywalled even to access it. And then there's a strategy that these companies have like Google that engage in what recap and Lundvall called predatory practices of rent seeking and subordination. So for example, by 2019, Google had published 6 ,447 scientific papers co -authored with almost 2 ,400 organizations. Yet none of those organizations were among the top 100 co -owners of Google's patents. And before 2017, Google has only shared three patents with the university, which was Stanford. And similarly, although Amazon owned over 10 ,000 patents by 2018, it only shared 13 with other companies and none with universities, even though Amazon frequently touts their collaboration with the scientific community. So there's public funds and training and resources going into this research, but the parts that could actually make people money, I mean, even beyond that, the knowledge, the work product itself, those things are being captured by these larger companies. And I think that's a very bad, bad way to do that.

0:06:39 Ale: But is it clear what's happening here so the researchers at universities are working with google at all amazon whatever and what's happened is they have a second affiliation at google say they kind of work together on projects and maybe they work part -time at the university part -term at Google. But their work on Google obviously affects their work at the university, right? I'm a researcher, I'm thinking about these things. But then Google or whoever will have like an NDA and that that research can never come back to the university. Right. So these, these are they're sucking IP out of our public institutions.

0:07:10 Thomas: Yeah it's like the old privatizing the gains and what is the other word?

0:07:16 Julia: Yes, 100%, 100%.

0:07:17 Ale: 100%. Yeah.

0:07:17 Julia: And they get reputational benefits and coverage in the media. They get authority by participating in an ecosystem of academic and scientific publishing, right? By putting out these papers, they also get to claim, like, look at all these good things we're contributing to. And then when it gets when that work gets cited in the media, the media thinks it's disinterested, fully disinteressed. We sort of tore into one particular paper for seemingly just regurgitating straightforwardly content from one of these companies in their paper that was then peer reviewed, which is essentially just laundering that company's image for them. Them.

0:07:53 Ale: Don't worry, that's our next paper. Okay, let's get back to this one. So this is sub member, this is Google's product research, pretending to be a paper that's whole stack of absurdities. Let me read you a quote here. It's a randomized double blind crossover study of text based consultations. So you know, I think everybody knows what a double randomized double one study is. So what they did was they had people talk to chat GPT or a doctor through a chat box, right text based consultation. So we're already seeing that game. But then they wrote this 50 page paper about it. And they really touted you know they have this like complex rubric. It's like super long. But if you actually look, if you read the paper, they have a screenshot that I'm going to share with you. They have a screenshot of what these chats look like. And if you look at it, it sort of becomes immediately clear that this entire thing is a farce because you can instantly tell when it's chat GPT and when it is a person. Because like humans don't format things like this in text, right?

0:08:34 Julia: She

0:09:00 Ale: So it's just like there it's this is you know julia alluded to this earlier but this is like purely the aesthetic of science this is like truly we are cargo culting our way through this there's no come on but you can even see in the patient here you can see like there's a typo right there's typo it's all lowercase whereas like the chat gpt one is like understood throbbing mainly like come on right like people don't talk type like that so just like the very like just I got like a very fundamental I think when we really tore into this on the blog the way we described it is putting Putting doctors in a chat interface and concluding that the chatbot is better than the doctor is kind of like putting a car in the water and concludging that fish are faster than cars. Like, it's just like, what are you like? What are you even doing? Like, you know what I mean? This is not like a meaningful thing to do. But like, ultimately, like, if we're going to do this research, this is what it's going to look like. And the way that the New York Times ran it, I think is like really damning of the sort of like media ecosystem that sort of like creates this like media clickbait style of of like, it incentivizes this like empty AI study, right? Like the poetry one, like this one.

0:09:00 Thomas: Honestly, there's almost in a way nothing special about the AI part of this because this is the same bullshit we've been covering on the show for a decade. Study shows Shows that drinking turmeric actually cures cancer. And then it's not really that. It's a way worse version of that. So in a way, it's kind of expected that this would be the same exact thing happening with AI.

0:10:22 Ale: Yeah, totally. I think with the small distinction, which is like when you do a study that's like turmeric cures cancer or whatever, we do have some insight into how the body works and sort of like what all these things are. But in reality, that's actually like a really difficult thing to like study. Like there's a reason we do these like longitudinal studies, right. But for LLMs, like what does it mean? If turmeric were to stop cancer, it would mean a lot. It would be like a huge deal. But what does it mean if chat GPT is better than doctors at diagnosing illnesses over like text boxes? Like that's not how illnesses get diagnosed. You know what I mean?

0:10:58 Ale: Like what's the how you operationalize this? We haven't even talked about the failure mode. Okay, so it's better at diagnosing illness is when in a text box, but like I don't interact with my doctor in a text box. I interact with My doctor in person. Sure.

0:10:58 Thomas: Sorry to quibble, but I do interact with my doctor in a text box sometimes, you know, for quick stuff. Sure, yeah.

0:11:14 Ale: But if it's serious, you go in right. And then like on on top of that, when your doctor doesn't know they like consult another doctor, but chat TPT has like no concept of not knowing it'll just like vomit, whatever, every time. Right. So it's sort of like, I would draw a distinction between those kinds of stuff in this, which is the results of those like could potentially be useful if they were like proven or whatever. Whereas like there kind of is no use for this other than like shilling a product. Like, yeah, yeah.

0:11:39 Thomas: I'd tell that to Big Turmeric. It's the same. Yeah. Yeah, I mean, I'll hold out that I do message my doctor stuff. And if it turned out that chatbots were better at that, that's definitely useful. If it's your only health care, that'S not good. But if it's if they can handle, you know, some part of health care. Well, I feel like that's useful.

0:12:00 Ale: Yeah, sure. I just worry about, you know, we don't really know, like the failure mode is just so egregious, right? They have no concept of I don't know, which I think is like a really scary failure mode. But then, you Know, the last thing I want to say about this paper is that I know I talked about the rubric and the length. I think that's pretty cynical. So if you actually read the paper, I actually have several typos. And I think what's happening here is that what they wanted to do is get this in front of the media. I don'T think this is this is, it's not not well written. It has typos. It's very long. It is full of jargon. It takes paragraphs of like passive voice text to explain extremely simple concepts. And I think that what they're doing and this is conjecture, but I think what they are doing is they are making media bait and they know that the journalists are not going to read a 50 page AI paper. What they want the journalists to do is to see the headline, read the abstract and give them a call and then they can give them some pull quotes, right?

0:12:51 Thomas: Yeah, totally.

0:12:53 Ale: So I think this is what is happening here and people should be aware of that when they read these articles that these journals probably haven't read it. And if you know if they click through it, and you see this article is like really hard to read, that might be what's happening.

0:13:03 Thomas: Yeah, that's so true.

0:13:04 Ale: Okay, this is my personal favorite AI paper ever written. This one's really good. I really like this one. I'm going to send you the paper there was a bit so this has been we're going to skip the headline straight. I think everybody knows that's what happens. But this has got coverage all over the place. It was in all your big media.

0:13:15 Thomas: Do you want me to read the article?

0:13:21 Ale: It's more fun when you read the headline. Yep.

0:13:22 Thomas: Okay. Well, yeah. So we're skipping the media part. We're just going straight to like the abstract and the title of the thing. So loneliness and suicide mitigation for students using GPT -3 enabled chatbots. All right.

0:13:34 Ale: Do you know what replica is?

0:13:38 Thomas: Remind me.

0:13:39 Ale: No, I'll tell you later. It's more fun if you don't know. So here's what what this paper did. They recruited some people and they asked them to do a study, right? This is how all these papers work. And so what they did was they contacted replica is a chatbot app, like a mobile app, and they contacted users on it. And they said, Hey, we'll give you a $20 gift card to do 60 minute survey or 45 minutes or greater than the actual length. And then they did some very simple stats on the survey. Like they said do you find replica makes your mood better and they sort of like parse that out like that, right, they put out these surveys, people filled them out and then they had like a you know they they they did some statistics that were like 60 I don't remember the actual numbers 60 of people agreed that it made them feel a little bit better and like these things but then they also they had this like text box that's like write other stuff and then from there they can this is where they get the suicide mitigation part of the study so three percent of respondents told them that were it not for replica you know know, they might have they might have died by suicide.

0:14:36 Ale: That's like a thing that people wrote in. And so they put that right at the top of the study. And now I want to tell you a little bit about replica. So replica actually, let's do this. I've not done this before. Just Google Images replica ads.

0:14:51 Thomas: Images. Oh, is it going to come up? Replica is it? How am I going to make sure it's the brand?

0:14:55 Ale: Oh, r e p l I k a.

0:14:58 Thomas: Ah, that would do it, wouldn't Okay. Your AI girlfriend?

0:15:03 Ale: Uh huh. Yeah.

0:15:03 Thomas: That's one. Need a friend? Oh, it's an AI girlfriend. Okay.

0:15:08 Ale: So are these if I remember what these ads look like? They're extremely horny.

0:15:12 Thomas: Yeah, I'm just seeing a bunch of fake -looking AI images of girls, and it just says replica, basically, and AI girlfriend, okay.

0:15:21 Ale: Yeah so replica's an online horny girlfriend AI app. And so this paper, yeah, so here's what's so damning about this paper. It doesn't say that. So remember, these are these are users who were already replica users. And then they asked them to fill out a survey and they asked him some questions about mental health. If you just read this study, you would think that replica was maybe a mental health tool, maybe just a friendship tool, you wouldn't have any idea like what it is. There's like this like two sentence throwaway line about like the selection bias but like holy moly what a selection bias right like that is exactly

0:15:24 Thomas: And it saved my life! I'm sorry. Hmm. Yeah. Yep. Yeah, yeah, for sure.

0:15:56 Thomas: Yep, yep. So, and, you know, not to cast any shade, but you're asking the kind of people who need an AI girlfriend, hey, how much do you like this AI girlfriend? That's, I mean, it's fine, but that is not data about chatbots or, you You know, just, yeah.

0:16:08 Ale: Yeah yeah right to conclude that gp3 enable chatbots can mitigate suicide from that date is like is it is an insane thing to do right that is that is a wild claim but it sort of gets worse I think because if you like think about I think if you really think about what is that population like right so this is very very online, kind of lonely young men, they're trolls, right?

0:16:19 Thomas: That is really funny. Wow.

0:16:36 Ale: And this is something that I, it's conjectures. They could be, right. But you know, if you think about like, do you ever do any online gaming? These days are behind me, but I remember very often being told KYS. Do you remember this? KYS, kill yourself.

0:16:32 Thomas: Could be. Sure. Yeah, it's pretty ubiquitous.

0:16:52 Ale: This population has very specific problems of interpretation behind it. Right. And I think, to report this survey result without like even disclosing that that selection bias and then not doing any, any sort of follow up like at all, right? They didn't contact, it's just like wildly irresponsible, I would say.

0:17:10 Thomas: That's amazing.

0:17:11 Ale: But it's actually like, I think even worse. So I don't know how many how many studies you read, but you're actually required to disclose any conflicts of interest in the paper. So this paper disclosed no conflicts of interests, but But its author, Bethany Maples, is also the CEO of an AI education company.

0:17:24 Thomas: Great. Not great.

0:17:31 Ale: Not great at all. The other kind of like two little things I'd like to say about it is, so there's a second conflict of interest there. So she's the, she's a CEO of her own AI education.

0:17:39 Thomas: And she is an AI girlfriend.

0:17:41 Ale: And she is not an AI girlfriend, but on her board is Siobhan Zillis. Do you know who Sioban Zillus is?

0:17:47 Thomas: No.

0:17:48 Ale: Siobon Zillas is an Elon Musk sort of extended, extended universe person. She's like the mother of two of his children. Yeah.

0:17:56 Julia: Works at Neuralink, or at least she did at some point.

0:18:00 Ale: Yeah. So Siobhan Zaliz is someone who's like in the Elon Musk sort of open AI orbit, which I think is if not a conflict of interest, I think something that's like pretty close, right? Like this is a pretty, pretty sketchy. And then the third thing is the data, I forgot the exact timeline, but the data for this study was collected in like 2021, but The Paper came out about a year ago. And I think The Paper became out right when Replica launched a mental health product. So So Replica has been sort of adamant that it's not an AI girlfriend, which is like a Trumpian lie, right? Like, it's just like it's like, just so it's so bold. But one thing they, they, that, you know, when this paper came out, the, I forgot the CEO of Replica's name, but she went on like a big press tour and she was like, you know, this paper shows that we're in a coincided with, you know, the launch of the new mental health product. So here we sort of see, I think like the culmination of like all these things we've talked about, right. Right? There's like horrible media ecosystem is

0:18:08 Thomas: Yeah yeah oh huh right yeah yeah yeah

0:18:56 Ale: Like really sort of motivated reasoning. It's like, this like, you know, dishonest reporting of the information. And in this case, like just like the purest and most disgusting of corporate interests, right? This study cannot happen without intimate access to replicas user bait, like app, right. You need to work with the company to think that they would allow this to be published without like agreeing with it to some capacities.

0:19:18 Ale: Like it's like pretty, like, it's just like not true. You know, like that's not how companies work.

0:19:18 Thomas: Well From the abstract, Replica is merely an intelligent social agent so yeah

0:19:28 Ale: Exactly, exactly. Just truly is this is like, you know, this is one of the most egregious papers that I think I've ever read.

0:19:33 Julia: I don't know the exact details of Siobhan Zillis and the AI education company Bethany Maples started and their relationship with Replica and their relationships with each other. But I think it's at least a lot of red flags that these people are sort of all connected to each other, given that this paper coming out served them really well. Bethany Maples got a bunch of quotations in the media. She got listed as an AI expert in a million places. Replica got to point to this study over and over and over again.

0:20:05 Julia: And I think that's just suggestive at the very minimum of an ecosystem that's really not ideal.

0:20:08 Thomas: Critically three percent reported that replica halted their suicidal ideation yeah that is yeah wow

0:20:17 Ale: Halted their suicide. It's just like such a bold claim that like violates just like obvious intuition to like then go through and like put that in a paper and sort of like launder that through all the selection biases and corporate interests. It's just like such wildly irresponsible malpractice of science.

0:20:31 Thomas: AI

0:20:31 Julia: Well, and I mean, it's really easy to imagine somebody seeing this headline, the headlines about it, where they were like, well, you know, this just giving students access to this chatbot reduce their rate of suicidal ideation. It's a relatively easy and cheap thing for something like a college to then provide to every student, right? And people who use Replika, you know, some people have reported like, I basically started using Replika and then it got incredibly uncomfortably sexual with me right away and I didn't know why. You know, so it's easy to imagine this sort of like going awry very quickly just based on people interpreting what What's being presented to them at face value in a context that's authoritative?

0:20:44 Thomas: Girlfriends for everyone yeah yeah and if it's being interpreted as like oh this is just a chatbot therapist you wouldn't necessarily consider that there could be those

0:21:25 Julia: No, no, definitely not.

0:21:35 Ad: [Ad break 2 — ad music]

0:21:35 Ale: There's a couple of themes I think we've seen through this, right? So the first one I want to to say, I'm actually going to quote myself because I already put a lot of thought into how I said it and I'm going to stick to it. So I wrote this in one of the Luddites posts. These so -called studies are purposefully, almost explicitly designed to reach the result that workers are dispensable. Even when they obviously demonstrate exactly the opposite, the researchers in the media dutifully report that we workers are replaceable because AI is coming for our jobs. It is whether consciously or not capitalist propaganda that exists to discipline labor, to convince us that we should be grateful for what we have, and to force us to think twice before demanding more. So I know, I think if you go back, think about the GPT versus residents one, right? Like, anytime you see a paper that's framed that way, which is like, we doctors, we put AI against a kind of labor that we've deemed like lower status, right residents are like the lowest status doctors, you know, the result just from reading the headline, right, like, you Know, you knows what happened. And I think there's a it's sort of they can be silly. But I think There's a really really important. You know, AI can't do the things that people, I think Cory Doctorow put it really well.

0:21:36 Thomas: Downsides yeah right

0:22:41 Ale: Oh no, I know Thomas, I Know You and I have a long history with Cory Doctoraw, but he put it really well. Yeah. But, um, he put a really well a while back that he said like, you know, it's AI can't the thing that, you don't remember, I don't have a quote, but AI can do the thing that it says it can do.

0:22:42 Thomas: You do yeah I'm sure you can quote yourself, but you can't quote Dr. Al. Quoting yourself, stealing your quote.

0:23:01 Ale: Yeah. So this is me quoting him, quoting me, quoting him, paraphrasing that, that, you know, AI can't do the thing that it says it can do, but it can doing it well enough that your boss can fire you, right? Like if it can convince your boss that can do it well, enough at the end, even if they can't fire you to discipline you, right. To threaten you and just say like, Oh, you, maybe you shouldn't. And I think when you take all these things in their totality, I think they're like a really, and you know it, the average person, isn't me like, you now reading all these, they're just like absorbing these in the, in the general vibe, right?

0:23:05 Thomas: Right. Oh, yeah.

0:23:32 Ale: And I think there's like a really strong labor disciplining aspect to this research. And I just want to call that out.

0:23:38 Thomas: Interesting.

0:23:38 Ale: I think it's like really bad. I think this is not at all what science is for. Science is not for making sure that corporations can extract more profits from its workers.

0:23:47 Thomas: Yeah, there really it does feel like one thing that might have been an illusion for a lot of all time was that, well, if something produces worse outcomes, surely we just won't do that because that's worse outcomes. When it feels more and more like it might just be that we just have worse outcomes. And we're like, shit, why, why did that? I don't know. Okay. Worse outcomes. People forget it'll become the new normal. So it might be that the output isn't as good. But as you say, like, it's not as good, but, uh, kind of close enough. And then that's just the new norm.

0:24:20 Thomas: That feels like the way the internet is going right now.

0:24:19 Ale: Yeah, I mean, yeah, no, I I mean just look at like, I feel like that is the output of like our entire culture. It seems like like I mean Just like think about like a great, like homogenization of all things, everything from like Marvel movies to like, Oh, I don't know if I don'T want to pick a fight, but like, you know, how these, these sort of like homogenous mass produced cultural products, if they can kind of like reproduce a thing, this is in that tradition, right? If they can make something cheaper and faster than they still, and they can Make, you Know, if it can make three times as much, then they kind of don't care if it's worse than it sells less because it's still three times as much.

0:24:55 Ale: You know? Okay. So there's the labor disciplining aspect. And I think that ties in to you know in this way that they don't value human labor we see them writing out the human labor in these stories right these like human out of the loop we took out all the images from these things oh the chat gpt can run the whole damn tech company right there's just sort of like I don't know where this is you know I don't want to like psychoanalyze like our our culture writ large of these researchers but I donT know where THIS like anti -human ideology is coming from but it's bad and we shouldn't do it.

0:25:25 Julia: Was a really good example of this in commercials I saw recently for some of these AI products, and I don't remember which company exactly was behind it. But the gist of the commercial was like a loving family moment where children were presenting their father with something special for his birthday and the wife had forgotten it was So she asked her phone to use generative AI to make her something that looked meaningful quickly. And then she gave that to the husband. And this is like the pro, pro this product commercial. And it's missing the point of, it's kind of like that movie Click with Adam Sandler, where like, you, me neither, but I know what the movie is about. Out.

0:25:54 Thomas: Oh, wow. Yeah, never saw that. Yeah. I just want to make very clear, you cannot casually reference that as though that's a reference that we should have seen.

0:26:14 Julia: I haven't seen it either, but the premise is that you have this thing that lets you fast forward through your life to get out of unpleasant situations, but then one day you wake up and you fast forward through everything meaningful as well. And I think we're doing, yeah, I'm sorry. Some of the people making and selling these products, for some reason, it's not striking to them as an, in fact, horrific outcome to skip the part where you made something meaningful for your family because you love them. And the same with the poetry. You skip the part where you engage with the poetry in any meaningful way because you're just trying to get through it to the next poem as fast as possible. It's this sort of perverse, I think, reflection of the incentives that are maybe felt on a daily basis, but which we should reflect

0:26:28 Thomas: Sure. There's got to be a better way to reference that concept. No, I'm just kidding. Just teasing. Yeah yep yeah yeah mm

0:27:07 Julia: On and maybe decide if that's the world we want to set these things up to create and to co -create with us.

0:27:13 Ale: Yeah, the AI poetry paper specifically had this like wonderful articulation of whatever this like deep horrific pathology is. They asked people how much they liked a poem, but then sometimes they wouldn't tell them if it was AI generated. And sometimes they would. And what they found was that people have a bias against AI poetry. And this is how they like talk about that. But it's like, yeah, of course they do. Like people have have a bias against robot hugs like if you if I give someone a hug and then I tell them it wasn't me it was like some stupid machine they're like oh that hug sucks now I'm sad

0:27:45 Thomas: Well yeah also if it's like okay how good is this drawing I'm like oh okay and they're like actually that just happened by chance because someone dropped a can of paint or whatever it'slike oh so you've taken out the part where a human being wanted to to express something. It's art. Well, then I don't like it as much, obviously.

0:28:03 Ale: Yeah it's it's you know it's the thing the thing about poetry is that it's poetry like you know like like you're supposed to like sit with it and read it and think about it and like sort of like feel the sort of transcendental experience of like feeling emotion from like someone else's reading text if I know that like a machine made it it's like well yeah I like it less now you know It's not a bias. Calling that a bias is just a weird articulation of this impoverished world outlook that you need some sort of critical intervention.

0:28:31 Julia: It reminds Reminds me of the cloth mother, wire mother study.

0:28:36 Ale: Yes. Do you know this one?

0:28:37 Thomas: I don'T know what this is, but I already love it. Like just, just, just guessing.

0:28:40 Ale: No, you're going to be horrified.

0:28:41 Julia: It's incredibly sad, but it's one of these, it's from a long time ago, but they had these little baby monkeys and then they deprived them of contact with a parent and then gave them options of hugging a wire mother or a cloth mother and then concluded that the baby monkeys prefer the cloth Mother.

0:29:01 Thomas: Now that's just good science.

0:29:04 Julia: There's a lot going on. I think for one thing, just this sort of neglect of what it is that would make a hug meaningful. The material conditions of the monkeys not being taken into account, like they're deprived of any sort of nurturing contact. They're seeking that out desperately. And why do we need to use science to verify this?

0:29:24 Julia: Like we could reason about it from first principles. We don't need to test this on the little innocent baby monkeys. I think sort of all of those things are going on at different metaphorical levels within current AI research, including this fetishization of the experimental setup and neglecting engaging with story and engaging with first principles and reasoning as something valuable. The part where you think about the research and then produce results that other people can think with, that's why we're doing the science. If you cut that out, what are we here for? And some of these these people are here for product research. And that sucks.

0:29:24 Thomas: Yeah, hmm, hmm.

0:30:05 Ale: To take that further, I like that way of putting it, the fetishization of this experimental setup in a way that, I mean, we showed how, for example, when I showed you that screenshot of the discussion between ChatGPT pretending to be a doctor when it's like a double blind, but it's very clearly like chat GPT, right? This sort of total lack of engagement with the theory of what you're doing and what you'RE doing and why you're trying to prove and what does it mean? And can we argue this in first principles if we instead just procedurally reproduce the aesthetic of science over and over? This theoretical void that we've left, it seems like product research is just consuming it. By sort of retreating and abandoning, by leaving a space, by retreating from the place where we think about why we're doing this research, it seems like these very motivated interests have moved into it to take its place. One thing we really want to say here is that like science has theory. That's an important part of doing things, right? Science isn't something you don't pull a lever, get data and report statistics. Give me something to think with. What does it mean to be a part of the world? Like, what does it means to you? If we're just like mumbling statistics to no one in particular, like

0:31:08 Ale: What is that for? You know?

0:31:08 Julia: There's something we've been calling green eggs and ham research happening, which is, you know, we established something like that LLMs hallucinate in a box.

0:31:21 Thomas: What?

0:31:21 Julia: And then there's new research coming out the next day saying, but will they hallucinate with a fox? And the answer is like, obviously, yes. But producing this sort of research is easier than actually thinking about what you're doing and doing real science. And I think like that siren call of the heuristic is always alluring to all of us. Like I'm not necessarily trying to come down too hard on the individual people involved, but I think it's worth critiquing, like, does this thing just match the aesthetic you're expecting? And, or, you know, is, is the experimental setup and the structure of the argument meaningful to the end result? Are the statistics, do they match the level of precision you can actually ascertain? Do you need this incredible level of precision when you're trying to talk about something we only know imprecisely in the first place? If we're talking about meaning and emotional relationships between people and AIs and things at that level, what are the statistics functioning for in that case? And I think sometimes what they're functioning for is as an aesthetic, as a signal to being in the in -group rather than providing value or meaning. I don't want to be too harsh on people because there's also a lot of really good work being done, but I think it would be good to encourage critical engagement with what we're doing and why we'redoing it rather than just sort of barreling headfirst as fast as possible, which I think there's a lot of pressure to do especially in this area because things are changing so quickly I think people are are almost afraid sometimes to slow down and engage with what's happening and why it's happening because someone else might do something newer and faster and better that negates the thing they've already put time into

0:33:11 Thomas: I think there's also been a lot of this thought of like the singularity or the like this has been labeled as the next thing it's sort of like if in 2005 for something like that someone from the future was like hey something called the iPhone is going to be the next biggest thing and then everyone's like shit we got to make that then like we got figure out what that is you know like it's like Like, I feel like a lot of this is they just know, and they might not be wrong. Like they know this is the TM next big thing. So everybody is trying to like corner the market because once the iPhone was made, it basically did take over for the standard of phone. Like that became the new standard of what phones are. And I, I do feel like there's just so much competition to become that.

0:34:00 Thomas: Like there's this feeling that like whoever gets that first is going to eat everybody else.

0:34:04 Julia: The line between creation and manifestation is being blurred.

0:34:09 Ale: Yeah, yeah. And especially from like the people who are saying the tech industry, I think we talked about this last time, the tech industry is the tech industry and capital, like venture capital are sort of like indistinguishable. Exactly.

0:34:18 Thomas: Yeah. Yeah. It's already, well, it's already bad to begin with, let alone with these other bad incentives. You know, like it's already so, yeah.

0:34:25 Ale: Exactly, it's really bad. And then because these people who are saying it's the next big thing, as our wealth has become so unbelievably concentrated, there are also the people who get to decide what we do like as a society, right? So like, you know, just because just come up with like a very silly analogy, we might not all want to eat mashed potatoes. But if that's the only thing on offer, then whoever makes the best mashed potatoes is going to win, right? The next big thing is sort of a thing that they're dictating kind of from the top down and the rest of it just forced to eat whatever they serve.

0:34:54 Thomas: And yet, Julia's getting a doctorate in this. Checkmate. No.

0:34:59 Ale: Checkmate liberals.

0:35:00 Thomas: I'm actually curious though, how is it that you want to look at this going forward, Julia? Because this is definitely something you're putting a lot of time and work into understanding. And I'm curious, just on a personal level, where you're looking to apply that or what you're working to do with it.

0:35:14 Julia: I don't know. I would know. So what happened was my undergraduate degree is in linguistics. When I started going back to school for my advanced degree, only people basically, more or less, you mentioned parents, there's a few like edge cases, but basically only people spoke language. And then during that time, machines learned to speak language. And that's incredible. It's amazing.

0:35:18 Thomas: I was told it was the next big thing, so... No, I'm just kidding. Oh, yeah.

0:35:45 Julia: And so I sort of couldn't help but get sucked into it because for a long time, language has come bundled up with all these other things about human cognition. And this is the first time we might be able to disentangle those things that historically have been impossible to disambiguate. And LLM can show us what does the minimum viable viable product version of language as a technology look like.

0:36:12 Julia: And I think that's like a really, really interesting area. You know, there's a really good paper by Levin, 2024, on StigmaG and the self as a tool of, yeah, so like basically like communication and message passing. And you could think of those, you know, all the way from maybe cells interacting through electrical impulses or coordinating very basic things through electrical impulses. You can sort of see that all the way through language and LLMs and writing as like one continuum of information technology and trying to build structure in opposition to a constantly degrading dispersive medium that we live in with the universe. So I think that's sort of, that's like a really incredible, fascinating,

0:36:23 Thomas: Sorry, Stigmergy. Yeah.

0:36:59 Julia: Even like noble pursuit of life. And so that's why I really like LLMs despite all these sort negatives we've pointed out.

0:37:07 Thomas: It's endlessly fascinating for sure.

0:37:18 Ad: [Ad break 3 — ad music]

0:37:18 Thomas: So you guys have outlined how bad a lot of this coverage is. What have you learned and what can we apply to try to look out for this in the future?

0:37:26 Julia: If people want just little pieces to be looking out for when they're encountering reporting or studies about this, the language game aspect of: is this a complex task being represented as a game of generating plausible text when we know the thing is good at generating plausible texts? And then there's a backwards induction to: okay, then it's good at the complex task. Watching out for that is valuable. And then watching out for benchmarking, because there's just so much that gets consolidated down into a statistic about benchmarking, you know, are the artifacts from the model really telling you what would be something that would be relevant to the thing you're trying to ascertain, you know, because there's different models, there's different scales, there are different training data, there are different kinds of objective functions like kludged on top of the sort of basic text generation ones. You know, there could be reinforcement learning with human feedback, there There could be sort of all these other things going on, but when you only see like one benchmarked number about the performance of the model, that can be sort of misleading, especially if it could be benchmarked against any sort of human output, like people scoring things like the poetry thing on essentially some kind of mechanical Turk context, like the material conditions that generate both ends of the benchmark can be deeply misleading. So

0:38:46 Julia: Those are like things I would say people can watch out for when trying to evaluate this information for themselves.

0:38:52 Ale: The one thing I would add to that is who's writing the paper. So you know, I encourage if people see a paper, I encourage to click through them or a headline encourage to click through that to the link to the paper and see who the authors are and see where their affiliations are. It's not uncommon to see 20 authors all at Google, in which case, you know, be skeptical of it. Um, and even, and you know, that that's not to say that, that, uh, that you should trust trust is the wrong word, but that's Not to say they should be satisfied when you see that they're not that, but you know it's, it's something to check.

0:39:23 Thomas: For sure. This has been really fascinating. I really appreciate the time. So interesting. And I know I'd be honored to have you guys back in the future on this topic because it's not like it's going away and there's so much all the time on this. I always like to ask, where do you want to send people to find more of your work or is there anything you want to plug? That kind of thing.

0:39:41 Ale: So we write about tech at the Luddite.org. It's an anti-capitalist tech blog.

0:39:44 Julia: And I wanted to acknowledge we've been working on this topic with Catherine Kramer, and then also some of these topics that I'm kind of covering like off the cuff and maybe making some mistakes. And we talked about with more precision in our recent paper on tokenization, which is cross posted on the Luddite and on archive, and then also in my forthcoming dissertation.

0:40:09 Ale: Yeah, I encourage people if you know we, we kind of like dunked on a lot of AI research, but if you go to Luddites and you read the paper around tokenization you'll find what we think is good AI research there's tons of good AI research but there's way more bad AI research

0:40:19 Thomas: There's got to be. I would hope there's good AI research.

0:40:26 Thomas: Well, maybe we could have a sequel that is good AI research because that would be that actually it is really interesting what's going on with what these models are able to do so maybe uh who knows maybe that's an idea for another time but yeah absolutely thank you so much for coming on really appreciate it so yeah maybe just uh transition as though we did it right yeah

0:40:37 Julia: Thank you.

0:40:41 Ale: Yeah thanks for having us yeah got it all right I'll be cool um be cool man okay so

0:41:07 Thomas: Be cool AI girlfriends for everyone