SIO530: Did AI Actually "Solve" That Famous Math Problem?

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:14 Thomas: Hello and welcome to Serious Inquiries Only. This is episode 530. I'm Thomas and I'm joined by Jenessa in the pre-intro. How are you doing?

0:00:22 Jenessa: I am doing pretty good hanging in there.

0:00:25 Thomas: That's it? That's all you're doing? Because I feel like you should be extra excited because we have a guest.

0:00:30 Jenessa: Yes. I am here not in my expert capacity, which is super fun.

0:00:35 Thomas: Jenessa will be saying nothing intelligent, she told me. There's no... She's committed. Zero Insight. Sight, you're just another me. Would you

0:00:39 Jenessa: Yes! I'm switch into just me voice. I can, uh, introduce my lovely friend, if you'd like, um So Yeah.

0:00:52 Thomas: What I was getting at, yep. The

0:00:54 Jenessa: Um, you'll have heard chuckling in the background, my friend David Souther. Uh, might have heard me credit him in the past for helping me with technical things on, uh, understanding computers with, uh, various OA and SIO things. Uh, we've been friends for a long while. Um, relevant to the podcast, not, not an expert on me. But David has his bachelor's in mathematics and computer science. He's a software engineer, and an LLM researcher and educator. And we want to talk about this thing that's been going on with some of thing with AI and math, and I didn't get it.

0:01:32 Thomas: AI solved math they figured it all out they

0:01:37 Jenessa: And I had to call David.

0:01:35 David: They did not, close,

0:01:37 Thomas: Found what Every number equals They solved all that. They're like, this week we saw Yeah. I'm excited about this, people.

0:01:42 David: Almost, hi,

0:01:39 Jenessa: Yeah. David also built the, uh, program that did my echolocation maze experiment. To like

0:01:51 Thomas: I remember that when you reference that anyway. Yeah.

0:01:52 David: That was so fun,

0:01:52 Jenessa: Yeah it Like produced all kinds of different pinging noises, and I could change, like, not just the pitch but, like, the decay and, like, the overtones and undertones, and it was It's, like, really cool for being able to find just the right frequency to echolocate.

0:02:09 David: The better part of a week at the time, and today, I could have done it in like 10 minutes with a nice prompt on the LLMs, it would have been totally different, hi., thank you so much for having me, Thomas and Jenessa.

0:02:16 Thomas: Welcome to the show. And I'm used to reading your name on the OA patrons.

0:02:24 David: It's been wild ride. I've been a long-time listener, first-time caller.

0:02:29 Thomas: So nice to have you. I'm really excited, uh, because of course, uh, have probably seen that headline that AI, uh, solved some sort of, uh, really hard math problem is what everyone will have seen.

0:02:37 David: Yeah, the Navier-Stokes Millennium Problem.

0:02:42 Thomas: Yeah, that's how the Millennium Falcon flies. They had to say so we can have a millennium Falcon now. That's awesome.

0:02:46 David: Solve this problem, and the Millennium Falcon uses field flow equations for its hyperdrive. Absolutely true, 100%. Yeah,

0:02:52 Thomas: You go. And yeah, I'm really excited to get the real story of that. I was, I was, uh, teased a little bit by Jenessa that there's maybe, uh, something more to this than, than the public is getting.

0:03:04 David: Just another instance of the rigmarole that we've been seeing with the AI companies companies, desperately outbidding one another to see who can have all of the money so that there is no money for anyone else. And along the way, we get to destroy the fabric of communication while we're at it. That's a little bit of editorialization. I would love to step back and just kind of start with maybe a bit of overview of this is a math problem that we've been looking at since, uh, well, 80 years really, and we've got some really exciting developments that the AI companies want to, to take credit for, and I'm really curious if that's going to work.

0:03:43 Thomas: What a great teaser, uh, so excited. Why don't we take our first break, and people can skip the ads: patreon.com/seriouspod. And, uh, I am, I'm really excited to dig into this, so we'll take a break and then we'll get into it.

0:04:07 Jenessa: So, I, yeah, saw a headline. Just, you know. My feed is constantly feeding me nonsense. And first it was just the, oh yeah, AI did a thing, and I'm like, whatever, I don't care. And it was like math, I don't care. And then all of a sudden I got the feed of, hey, wait a minute, somebody's accusing them of maybe not crediting them for something. So I sent it to David and I was like, hey, what's this about? And I'm thinking like, I'm telling him something he hasn't already heard, and I, I got back, like, all caps, oh my god, um

0:04:21 Thomas: Dare you.

0:04:39 David: Something

0:04:41 Thomas: And he wasn't just saying, how dare you insult math in the way you just did it. Wasn't that because I, I think the news headlines should just be like, math, and I'd be like, fucking A, yeah, that's awesome. And then it should tell me like some math thing that I forgot about, and I'll be, be like, yeah, that's good, that's news. Good job news, because that would be my dream world.

0:04:45 David: That I didn't know already. I

0:04:58 Jenessa: I One love in math is statistics, and that's not math apparently. So, um, mostly when it gets to the theoretical stuff, I'm like, whatever. Uh, apparently this This one, this one got interesting because it became a social science problem for me to wonder what are all these people yelling at each other about? Yeah, David started giving me some of this story and I was like, oh, Thomas is going to love this. So that's kind of how we got here.

0:05:28 David: My response was a series of all caps messages and then a GIF of Madeline Khan saying, flames on the side of my face in anger and frustration at the way that this was turning out.

0:05:37 Thomas: Flames. Flames, all Right, well, I'm excited to dig into it. Uh, so why the flames, flames on the side of your face, and

0:05:48 David: Let's back up a few steps. Let's talk about this math problem and a few of these math problems kind of from the history here. If go way, way back to the turn of the 20th century, we get a gentleman by the name of David Hilbert, who is looks around and says, there's a whole lot of things we've kind of figured out.

And we've really only got four or five, like, yeah, about 20 things that if we can figure out these 20 things, we'll really have solved math. And that that was 1900. Uh, and then Einstein comes along and he's like, yeah, by the way, there's this general relativity and the quantum mechanics people are like, Yeah, there there's, this uncertainty principle by the middle of the century, we had pretty much figured out all of the math that David Hilbert had complained about.

And then we found out that we had a whole lot more math that we needed to learn about. Um, and we got there by talking to each other and we come to the turn of the 20th, 21st century, uh, a group called the clay Institute puts out what we call the millennium challenge, which is basically doing the same same thing. Hey, this worked really well to kind of focus what we were talking about. Let's do that again.

Let's put together a whole bunch of these, you know, end of 1999, really hard problems in math. This can kind of be where we are going to focus our research.

0:06:54 Thomas: Then a janitor by the name of Matt Damon came up to the blackboard

0:06:58 David: Has happened. That's great. Because janitor by the name of Matt Damon comes to the blackboard and sees this and starts talking about this, we have more people in the mathematics. We have more different angles and ideas here. And I view mathematics as really more of a communication and collaboration effort. I'm not a professional mathematician, right? I want to put that out there. I mean, engineer, I'm on the software side. I can follow it, but I'm not in the field myself. The aren't really, it's kind of like a jigsaw puzzle, right. Like I don't think anyone goes to put it together a jig saw puzzle to see the picture. The picture is already there. It's putting the puzzle together. That's the fun part. It is working with other people on the puzzle that brings you together to do that. So

0:07:42 Thomas: Like, if somebody gave you, like, a finished jigsaw puzzle, you'd be like, what is this? Yeah.

0:07:48 David: Cool. All right. Yeah. So one of these is a problem called the Navier -Stokes problem, which is talking about how fluids move and interact. And you'll see some of the reporting is like, oh, hey, now that we have this Navier Stokes solution, we can do really really cool fluid mechanics. Engineers already do really cool fluid mechanics, we've got programs that use what we call discrete mathematics to break down into any engineering size challenge to build dams or to build spaceships or to do anything else with that that you'd want to do.

0:07:51 Jenessa: Is The same thing as the, like, oh, we don't know how the fluid dynamics work, but, like, Van Gogh painted it. Like, I don't even know if that's true, but, like, the thing about, like, the swirlies. So

0:08:31 David: Yeah, I think that's super, super reasonable. Like, we don't know, in this analytic sense, well, this actually is a really good way to look at discrete versus versus analytic. We

0:08:40 Thomas: Let me add one more before you get to that. Wasn't there a thing where the movie Frozen 2, like, something about the, no, I'm serious, something about the CGI of how they did the way, like, the water when it flooded, that I saw something that was like, yeah, that gave this insight to how this works.

0:08:48 David: Yeah, yeah.

0:08:58 David: I would totally believe that I haven't come across that. Um, interstellar did that with black holes, but I, yeah, I totally believe that. So engineering discrete side, we, and in Van Gogh's intuition, we just kind of know enough to do it in the bounds that we're interested in. At the scale of a dam, at the scale of the Hoover Dam, we can do the engineering really well. We We don't don't need the full mathematical solution to do that. The full mathematical solution is kind of an art form. It's kind of its own art piece. But as mathematicians explore that, they get to push the bounds further and further to infinity. And the hope is that you'll find something new in there. On the way, on the path to pushing towards this end solution, you'll fine neat things as you go that have implications elsewhere. Where that's, if you were to say there's a goal of mathematics, that's the goal. It's not to solve it, it's to find all of the interesting questions on the way and then get more interesting questions.

0:09:00 Thomas: Okay, I'll Google it while you explain. Gotcha.

0:09:55 Jenessa: It doesn't really change what, like, an engineer who does fluid dynamics is not, like, oh, I'll build a damn different now. Okay.

0:10:02 David: No, because we didn't learn anything new about the Navier-Stokes equation the way that OpenAI did it, and that leads us to maybe some of our problem.

0:10:06 Jenessa: Oh Oh, okay.

0:10:11 Thomas: Is This, like, the three-body problem, where it turned out that they're like, yeah, I mean, the whole premise is kind of dumb, because, like, we, we can do that if we have the three bodies, like, if we can get enough. It was like, I'm going from memory, but it's like we just don't have a general solution for, like, every possible three but or Or something. Is it sort of like that where it's like we can do it when we pin it into the barriers or the boundaries that we're looking at or something?

0:10:31 David: Yeah. Exactly that. We can do three-body problem for the James Webb telescope or the Nancy Grace telescope that are up there orbiting at the L3. The Nancy Roman Grace telescope is a brand new telescope that just launched.

0:10:44 Thomas: Sorry, Nancy Grace Telescope. What? How did I not hear about this telescope? Okay, we got more SIO to do. Anyway, so carry on.

0:10:51 Jenessa: What the hell? I

0:10:55 David: On, for those problems, for this constrained case, yeah, we...

0:10:58 Thomas: So it's not the lady who comments on legal stuff. Just to be very clear, when you said Nancy Grace telescope, is that the like lady who comes on? Yeah. I was like, the Trump administration is weird, but I don't think they did that.

0:10:59 David: No,

0:11:06 Jenessa: Saw the haircut immediately.

0:11:14 Thomas: Did they?

0:11:14 David: Grace Roman. Nancy Grace Roman was probably pivotal in the Hubble telescope getting launched and funded. And so we put her name, so she's got her name on the latest telescope that's going out to deep space to do all sky research.

0:11:28 Thomas: Gotcha.

0:11:29 Jenessa: David's also a secret space nerd.

0:11:31 David: It's not secret.

0:11:31 Thomas: Yeah.

0:11:32 Jenessa: Yeah, that's true. Yeah, he worked at Starlink. I forgot to mention that.

0:11:36 Thomas: Oh, wow.

0:11:33 David: I did. Yeah. I worked at Space Chai, Space XAI, Space Chai.

0:11:39 Thomas: Space, what? Oh, wait, I don't, do I know what that, what is that? SpaceX?

0:11:45 David: Remember that Elon bought Twitter and then renamed it X and then renamed to XAI and then merged it into SpaceX. So now it's Space X AI.

0:11:54 Thomas: My God. I didn't know that.

0:11:58 David: X is the Greek sound, so it's space chai This is definitely, that's

0:12:01 Thomas: Oh, nice. Sounds delicious. Love it. Okay. OK, sorry, we derailed you with me imagining Nancy Grace looking through a telescope, but setting that aside.

0:12:08 Jenessa: Okay.

0:12:10 David: Great. The other Grace. The other Nancy. Grace Nancy Race Roman is who that was about. Yeah, this is exactly the same as the three body problem where we know it well enough to run space missions, but are there interesting things when you really push those boundaries any further? That's what we're trying to get at. And for the last, what, 25, 26 years, several of mathematicians, not like any formal group, just some mathematicians have been working on this.

And maybe about 10 years ago, maybe about five years ago—I didn't quite look up this part well enough—we Diego Cordoba and Louis Martinez Zarro, adding a couple of very clear contributions to the field. And they've really pushed the problem forward. Trying to do some historical research, like what was the state of Navier -Stokes research in like 2025? Of course, the news headlines came out. And so it's almost impossible to find the old quanta articles, which is a pain.

As far as I could gather, it was kind of unclear where the research was at the time. And so it's a bit surprising that we had new results in 2026, but also people were working on it. Define So Navier -Stokes really quick.

Navier-Stokes is this problem in fluid dynamics that says, hey, if you start with the mathematical, mathematical spherical cow of a perfect homogenous fluid, can you slosh it around in such a way that you can either always know what it's going to do or slosh around in a way that you break it so that some particles or some part of it moves infinitely fast, right? You've got a discontinuity and infinity. Mathematicians don't like actually finding infinities. They just like knowing it's out there.

So we've got these two options of we can either solve it perfectly, or we can show that there's what we call an instability. And the current approach that OpenAI announced is, hey, we found this way to build an instability and you can do it using these forcing functions. And it gets a little bit technical. Also turns out to get kind of uninteresting if we're looking at some of the follow -up research or the followup conversation. We haven't had follow -ups research yet.

This is where we start to get into the, who, who's on first or who found this first. Uh, so the OpenAI post says on Tuesday, September 1st, we heard rumors that two Millennium Prize Problems had been solved. Inspired by these rumors and the step change in performance of our internal model, we launched an effort to evaluate it on all open Millennium Prize Problems. Now, if we go back to that, there were like a dozen, 25 Millennium Price Problems.

Like this isn't a small set of research. This is a whole lot. These these rumors. I first heard about these rumors on September 8th or 9th, when I got linked to a Mathstodon post from Terence Tao. Terence Tai is well known in the field. He's kind of this like star child mathematician.

And this post is this delightful, it's posts, like eight paragraphs of vague booking about wrote recently about how the collection of good, fruitful, open problems is now being mined in a non-renewable fashion, leading to the potential scenario of these problems becoming scarce. And he goes on to talk about how the AI companies are doing so much math so quickly that the mathematical community doesn't have time to

0:12:14 Thomas: Yeah. Gotcha.

0:15:33 Thomas: Oh,

0:15:33 David: Talk about them, doesn't some math for reals or just talk about it with us. Like if If you want to do the math, that's great, just talk about it, just share it.

0:15:34 Thomas: Leave some math for us. Well, not us, but show work or something.

0:15:47 David: Your work, and more importantly, show what didn't work. Show the interesting stumbles on the way. A couple days later, we get a post from a by the name of Tristan Buckmaster who works at NYU. And he writes that for the most of the past year, progress was slow. On August 15th, we obtained the initial blow -up results with smooth forcing for a of sub -problems. I can say the first LLM generator proof Levant sent me was the most horrendous I have ever read.

We verified it on August 22nd. Since this point, we have been working around the clock to understand the proof and turn it into something human -readable. Now, there's an important thing in there. They say the LLN generated the proof. They were using large language models. They wereusing AI to find it.

And this other gentleman that we'd mentioned, uh, Levant Opoge, uh, at Anthropic, and he was involved with another mathematical discovery from August on a set of problems called the Raimon Zeta problem. And what they did for the Raimon Zeta problem is they took a one piece of established research from the last couple of years, which uses this very a thing called the the narrow box conditions to move the Riemann zeta problem forward.

And based on their prompting, the LLM found some research from the seventies on how to remove narrow box conditions from other similar problems. And their LLN combined that together and came up with a proof on this other thing. Difference here is that Levant is consistently working with other mathematicians. He's consistently talking about things and keep this number in mind. He He uses, for this problem, they use 15 million output tokens. For reference, as a software engineer, not the top of my token maxing leaderboards.

I try to like stay third or fourth place when I go token maxting. I want to spend a few hundred thousand tokens on most weak -sized problems that I work on, right? I want it to be focused. I want It to be intentional. I want To know what I'm trying to find with the LLM and kind of direct it in that area. I don't just go, yeah, find everything. Good luck.

We go back to the OpenAI post about the Navier -Stokes, they claim, this is in their post, they're very proud of this. The process of resolving, the agent spent 2 .7 million messages and used approximately 130 billion output tokens. So we've got a couple of mathematicians doing LLM -assisted research, search finding 15, 20 million tokens. Open AI says, Hey, we heard a rumor that you might be working on this.

We are going to spend $15 million in a week to see if we can scoop you in one week in 88 hours of compute time. They took an entire, not an entire vertical rack of 40 servers. They took an entire strip, 2000 servers, full time for 88 hours, $15 million at API list cost to get their results, that they heard a rumor that someone else had already found

0:19:03 Jenessa: They knew somebody else was either already found it or was about to, or...

0:19:08 David: Rumors, rumors happen. If we, if we look at everybody's statements, uh, Buckmaster says that that they first found their proof on August 22nd and they were spending the time to like make it presentable, make it something that they weren't ashamed.

0:19:09 Jenessa: Okay. Okay.

0:19:22 Thomas: OK.

0:19:24 Jenessa: Just works at some university. He's not with any AI company.

0:19:28 David: Isn't some university, but yes, he's the university mathematician. Correct. And he was working with, uh, Levant at the time who is at anthropic. And this is important.

0:19:30 Jenessa: Right. Well, but it's just like two guys doing it.

0:19:40 David: This is just two guys. Yeah, this is Levant is doing this as his hobby project. He is into math. He using his skills as an AI researcher as a hobby project, working with his mathematician friends and, and making, you know, state-of-the-art advancements on these problems. That's pretty cool. Open AI says that on September 1st, so a week and a half later, they first hear about this and that's when they start their process. Buckmaster, the NYU professor, sends some emails back and forth with Open AI. And he says, "Hey, I heard that you're working on this. Let's coordinate, because, like, maybe I can save you some time. Maybe we can work together maybe we can publish jointly that we already did for 150 dollars. So if we, like, combine resources, so, and buck master says, "Let's talk next week."

This is like on a Thursday. Opening I was like, "No, we need to talk now." And he's like, "No, no, I'm a math professor. I don't do things now. We'll deal with this next week." Open AI finally gets him on the phone on the Sunday and kind of starts like being super cagey. So Buckmaster's like, well, so how'd you get there? Like, what was your approach? And they're like, we're not going to tell you. He's like you're sure like you're not gonna tell us at all what it found. And they'll like, oh, well we the agents work on it themselves. So like we didn't give them any instructions. So what did they come up with? Well, maybe we told them to use forcing functions.

And Buckmaster's like, so like the things that I was working on that no one else seems to be using in my conferences, like we talk, the entire point of mathematics is that we talk to each other. And I haven't heard anybody else going this direction. You just like share your prompts? No, absolutely not. We will not tell you how we did this. We will not tell you what we did.

0:20:18 Thomas: Yeah. Maybe I can save you 15 million dollars of pointless computing.

0:21:31 Thomas: How we did what? So at that point, you're saying they already solved it? It, or because I thought at this point you're saying there's, like, rumors, and then we don't know okay gotcha

0:21:37 David: We don't know. So OpenAI isn't telling us. After the fact, it seems like at this point, they did have what we call a Lean verifiable result. Basically, the advanced proofs at this point are so complicated that we have written an entire subset of computer programming languages that that only demonstrate that a math proof is correct. So that's kind of cool. And when you have a hundred-page-long math proof, you kind of need to do something like that.

0:21:41 Thomas: Yeah

0:22:06 David: So the only thing OpenAI is admitting is that their system gave them one of these lean proofs, but they won't tell us how they won't tell, uh, how they did it. And they say, but you know what? No, no, no. Here, here's the thing. We We don't want to win the millennium prize. We're not doing this for the glory. We're just doing this to, to see if our model can, we would love to share credit with you. OpenAI Let's says let's announce together and we'll, We'll we'll put put your name on it, but we we will not put Alpoga's name on it. And Buckmaster says, well, I worked with Alpaga. So obviously his name has to be on

0:22:05 Thomas: Just

0:22:36 David: It too, just

0:22:37 Thomas: Because he's affiliated with the rival AI company, what

0:22:39 David: Because he's with, with Anthropic entirely that. Buckmaster's letter, he says, if open AI goes for without attribution to both a pro game, myself, quote, I will go public with what happened. Open AI. Why would you ruin your career? Buckmaster? I'm an academic. Why would this ruin my career?

0:22:55 Thomas: Yeah my career's already ruined, so

0:23:02 David: Is like, this isn't my career. Like, okay. Open AI. AI. Well, if you want to play mean, we can play mean. Question mark, question mark, question mark. We hang up.

0:23:12 Thomas: This is a correspondence that's been released, that, you, yeah, I was kind of, kind of ask what your source is for all this. Is, is it because they came clean with it or came out with that? Oh

0:23:19 David: Source for this is a self -written letter that Buckmaster posts on, his university website. This is on, I think the September 8th is when we get get this published. Sorry. September 9th is when we get this publish. September 8th is when Terrence Tao starts a vague booking about this being a thing. And, um, to, to end Buckmaster's letter here, I think it's, it's very emblematic of what's going on here. Uh, if indeed an opening, I model did close the gap to Navier Stokes. That is a remarkable thing. And it should be said loudly by them with the history intact. I would much rather be talking about mathematics, Louis and DeVay goes ideas, and about what that means for the rest of us. So no one on the math side, do I get the impression, or in this for the glory, they're, they're, they're all just doing the math. That's

0:24:04 Jenessa: Wait, what is this, this thing? I, like, caught this for a moment. I caught this for a minute. I thought it was like, I thought he was like when he was saying like something about like forcing equations. And that is it. That sounded suspicious to my brain, so

0:24:15 Thomas: It was that, like, he was using a certain approach that other people weren't using, and it was like, "That can't be a coincidence," kind of thing. Sure.

0:24:22 David: Two things in here on the maths side. Forcing function is, uh, okay, for this problem, uh, actually, go back a ways on the Navier-Stokes problem for this. Maybe we're going too deep. Back in high school, you did cannonball trajectory, right? Cannon shoots the cannonball and based on the initial angle and how much charge it has, you have the initial velocity and it lands somewhere else. One form of the Navier -Stokes problem implies that you have, basically, you have that setup. You set up your fluid, you let it slosh around, and you see what happens. A second form, and that's with these forcing functions, lets you kind of continuously add energy. This would kind of be like saying, okay, well, what if we have gravity?

Well, if you're sloshing around on a planet with gravity, you're going to have different behavior than than on in space without gravity, with microgravity. And those are just kind of different ways to look at it. And the problem didn't really specify one way or the other. What the forcing function is going to do is it's going to be like gravity, except that instead of always pointing towards the planet, you can basically make it do whatever you want.

So if I'm the discussion about the OpenAI and the Buckmaster proofs, they're kind of like spinning it. They're kind of like adding a little agitator that's constantly spinning it a little bit. And that's okay, sort of, except that there's a couple of posts that I've read since then that say, "Well, I guess we shouldn't have let the Navier-Stokes problem go that way. We should have said that that wasn't allowed, because it's not interesting." Actually, yeah

0:24:36 Thomas: Wait

0:25:59 Jenessa: It's like if the cannonball had a rocket attached to it

0:26:02 David: That's very reasonable, but this isn't if the cannonball had a rocket attached to it, this is if the canon had a fictitious warp drive attached to if had a bracket drive, that'd be fine, because that makes sense, but if you have a, an unobtainium magic device then cool. That could work the math is correct and consistent, but you're the real world, in the physical world, you're not going to have that. So you don't generate any new insights about the physical world with this.

0:26:02 Jenessa: Oh Oh, I get It but

0:26:34 Thomas: So are you saying they didn't actually solve it then? Or I don't understand. Oh

0:26:39 David: By The Clay Institute for the Millennium Problems, this solution seems totally reasonable. Again, it's very new. I am not an expert in it, and it's problem. I says, yes, I confirmed it, and this is correct and reasonable. That said, this meets what the problem statement was. The follow -up discussion is, hey, maybe that's not what the Problem Statement should have been. We should have, in the Problem Statement, said you can't have unobtainium magic devices for the force side of the equation. Sorry,

0:27:22 Thomas: We got, like, two different things going on here, I. Well, several. But when I know this is going to be pretty hard for not, not just me but any listener to try to follow along with the math on the best of days. And when it's a, you know, problem that is, like, obviously hard for a reason, but what does it mean to solve this thing? Then I thought, I thought it's conditions that you laid out, and then you're just saying that the was kind of a cheat on it. And then, if so, is that something that was necessarily the only way the LLM could have solved it, or is that kind of a separate issue? Like, I guess it's getting out a couple different things, like, is any reason why the, the LLN couldn't have solved this in a way that makes, I don't know, that's more satisfying? Yeah

0:28:07 David: Second half of that question opens a great part that I'd love to continue on how I think about LLMs and how Ithink LLMS affect the broader conversation here. The specific part to are there other ways to do it? The answer is still question mark, question mark question mark.

We mean by this is, um, the Millennium Problem says, "Here is a set of bounding functions. Can you write another function that fills in the inside in such and such a way?" You can, you can think about this as, um, Millenium Problems, or the statement, the Millennial Problems gives you a, a coloring book with the lines drawn in. We know these lines are the lines that we want to figure out, and you need, need to color them in in such a way that is aesthetic. And yes, there's more rules on it, but I think for, for this level of conversation, that's, that's really what we're going for. Did you color them in consistently and aesthetically? One that the LLM found worked. It was fine. It's probably close to the way that Buckmaster and Opoga did it.

Are there other ways? Probably. We don't know, actually. If they spent 150 billion tokens, I bet you there were a couple of other very close solutions that look different in there that just didn't quite match their final verification phase. But we won't know. We won't know because OpenAI won't tell us. They won't share this information. They have no reason to, they have no motive to. The, that I'm seeing here for people who aren't mathematicians is the way that a company is going about approaching this problem. Million would fund the entire Harvard mathematics department. You could have just built a new math department and you would

0:29:54 David: Have fed you a dozen professors and a hundred grad students and a thousand undergrads for a year with this week's worth of money.

0:30:03 Thomas: Just have them work on this one problem

0:30:06 David: That you could have told them, just work on this problem, and would have gotten there. Would they have also made friendships along the way. Yes. And that's The point.

0:30:15 Thomas: Yeah And that's more important. Yeah.

0:30:18 David: Yes. It? Is I hope that that is not a controversial thing to say. It

0:30:18 Thomas: Does

0:30:19 Jenessa: I Yeah, there's clearly, like, a values statement here that, I would agree, I don't think that's controversial. That would have been really good to

0:30:32 David: Would be nice. Just if you wanted it this much, just buy a math department and put a math apartment together. You could have done that too, but then you couldn't have said that our model is so much cooler than their model.

0:30:44 Thomas: I don't know, it's so hard with this kind of math because it's just like, I know in other Millennium Problems, or, you know, another of those really tough-to-solve problems problems, it's stuff that's a little more, I think, graspable, where it's like, what was that one? It's like, is there any, or if you're trying to find, like, a higher prime number or whatever, something like that, where you're like, "Well, I found it, that's the prime number," and I can kind of grasp onto that. Like, and when it comes to that, I don't have any precious feelings, like, well, I don't really care if a machine finds a higher prime number or some poor soul who has nothing else to do is just, like, like, doing long division or something, that's fine. But what's the difference here? And how do we know how, how is this used for? If, if it's so ambiguous like this,

0:31:03 David: Yeah.

0:31:28 David: Prime example, I'm I'm going to change it a little bit, but I think that's perfect. We that can find the next highest prime or digits of pi. We have problems that can fine the next digit of pi, if you want the millionth digit of pi, we'll give it to you. If you want a billionth digit to pi, there's a Wikipedia page that just lists all of the advancements in finding digits of Pi, you can give a computer a program that will just continuously give you the next digital Pi and we will it will run forever, giving you more digits of PI. That's not interesting.

What's interesting is finding new ways to get digits of pi, because then you can learn new things about algebra, about circles, you can get new ways of thinking about things, and that's the goal of most mathematicians, or that's the the fruit of most mathematicians here is new ways of thinking about, and that is valuable in itself. That's the thing that we're going for, because once you find a new way to think about something thing. Maybe that applies somewhere else. Jenessa talks about this all of the time in some of the advancements that she's made, where she feels that, I apologize for speaking to you, Jenessa, we've

0:31:36 Thomas: Sorry,

0:32:36 David: Talked about this so many times, just the you've shared with me of, hey, I read this paper and I read that paper and they didn't say the same thing, but it really looks like if you if you take them this way, they kind of hint at the same thing. So I went to find that and I couldn't find anyone else who said that. And then I found it and I wrote it up. And just that is so cool when you can make those connections. And sometimes the AI does that and sometimes it doesn't. And if you don't share that, you don't get them.

0:32:36 Jenessa: That's fine. So Curious how much of the problem is just, like, straight up, if an LLM does it, you must lose that, or if it's like, no, you could show your work and it's more corporate culture not showing it? Oh

0:33:19 David: Think it's entirely the second. I I use LLMs in my day -to -day work all the time and I share my results all the time. I talk with my peers and my colleagues about the challenges that my company is facing, about the challenge that our customers are facing and the problems that they're having. And we say, this is a good way to solve it. Or we say I don't know what a good way to resolve it is. Let's do some research. The language models, these AI things are are just phenomenal at doing language. They just take large amounts of language and run through them. Turns out that language isn't thinking, but if I just need to say, go find me 37 academic papers and find the summary so that I can go the next step, that's great. It doesn't do that on its own. It only does that because I said, this is what I want to find. Go see if you can get me that.

0:33:21 Jenessa: So Because I guess, like, in my head, if it was just like, I, okay, like. Like, if it was like classic sci-fi AI, which is like essentially just the equivalent of, I just ask some other human to do it. I just say go, and then the thing comes back, and then it's like, here's your answer. And then there's like intermediate steps. But you're talking like about intermediate steps where you could actually reveal that.

0:34:31 David: Yeah yeah So the way that I do research with the LLM is I let the LLLM hallucinate a set of questions. And then instead of having the LLLM give me an answer, which may or may not be correct, I have it go and do a Google search and bring back a bunch of documents that are about that. Then it says, okay, the, part of this this document and that part of that document are talking about the same thing. That's probably close to the right answer.

These things, they're saying differently, that's probably an interesting piece. And I can use that to do a large amount of research very quickly and focus in on the parts that I'm actually interested in and might actually be novel versus the parts that are already solved. Someone else already wrote about it. I can just work with their work, and we can communicate with them about what they've already done to project forward. I take that, now I haven't just said, hey, AI, go generate this answer for me.

I've said, Hey, AI go help me sort through all of the papers about this already so that I can go back to my team and say, here are the interesting pieces about what we're trying to do next and the interesting ways to solve the problems that we have.

0:35:47 Jenessa: Okay, if you just, like, did that front to back, you were trying to solve this problem, would you also use, like, what was it, like, 340 bazillion tokens, or is it, and it's just over 10 steps, or is that, like, it would ultimately be more efficient? Oh

0:36:03 David: A couple of scales here. I do this, I usually work kind of one -on -one with an LLM instance. I'll send it out. It'll send maybe a couple of what they call sub -agents to go and do like, hey, I'm going to do a Google search. I'm going to go search DuckDuckGo, just get some variety in there. It comes back to to me and I evaluate their answers. I evaluate the output material.

0:36:25 Jenessa: And that saves a whole bunch of tokens, right?

0:36:28 David: Keeps me in the loop. Yeah. It saves tokens because I did the thinking. It didn't save tokens. It just used much cheaper tokens. My tokens, my brain tokens are way more efficient than the LLM thinking tokens. And yeah, I mean that, that search will take me, I don't know, um, at a couple of nearby answers. That'll take take me 10, 15 ,000 tokens. And at billing rates, the current billing rates are $50 per million tokens. So of searches to get to $50 in API spend or to

0:36:31 Jenessa: Right, of course, kind of the important part Right. Oh

0:36:31 Thomas: So

0:36:54 Jenessa: So I was cheap. Okay Okay, throw

0:37:02 David: Get to $250 in billing spend, which, hey, for the amount that I'm getting paid as a software engineer, that is a very reasonable expense by an employer. OpenAI is doing is very differently. OpenAI is not using humans to evaluate the output of the computer. They are using other computers to evaluate, the output, and they are just basically racing them against each other. You go figure it out. You decide if it's correct. If you've based on your feedback, you go do something else.

And they're just arguing, literally, they're using these AI tokens to generate arguments against each other until finally one of them finds something that that outside independent program that we call Lean, the proof solver, actually says, yes, this is the proof. The way that you could think about this is, let's say you have, like, a field of grass, and every blade of grass is a different written page, and, I don't know, it's a couple couple of acre field, right?

Like that's a lot of pages that we could get through. What I'm going to do is I'm gonna go and say, I think that my page is probably on that hill over there. And so I'm to go over there and I'm be very careful to look around that area. I'm going to tend to that area of the garden very well. Versus what I think OpenAI did is they just kind of flooded the whole thing and saw what floated to the top. And then everything else just just got lost and just wasted and just washed away.

0:38:40 Thomas: Um, happened in terms of, I mean, it's not that important, but, like, there was a prize associated with this. Is Just trying to get a sense of, like, what happened here. I can't even understand it, and so who's getting the friggin' prize? That That'll help me understand it. We don't know. Okay. Hmm.

0:38:54 David: Don't know yet. This was September 8th. Uh, clay Institute says we aren't even going to evaluate whether the problem has been tackled for three years for

0:39:06 Thomas: Yeah. Why, I thought

0:39:06 Jenessa: That feels long. Is that normal? Oh

0:39:09 David: Clay institute. It is, they've I went back and I read there, how do we evaluate these things? And it's, you can claim that you have a proof. We don't know that that, that is a reasonable claim until the mathematical community has had a couple of years to work with it. Years of other people responding to it, evaluating it, critiquing it. What if somebody finds a flaw in. And what if somebody says, oh, no, you didn't think about this piece of it here? OK, cool. That wasn't the right proof. So we can't do that right away. Clay Institute doesn't want to like have people on staff just to review every claim. They're going to wait and they're going to let the research community do that evaluation. Not They're evaluating whether the paper was true or false. They're evaluating whether the mathematical community thinks and agrees that that paper did solve the problem. So that's really what they're looking for

0:39:11 Jenessa: Okay, okay, okay.

0:39:26 Jenessa: Sense. Wait Is also weird to me because it's like, okay, on the one hand it could go to this Buckmaster, and what's the other guy's name?

0:40:14 David: And Alpoga.

0:40:14 Jenessa: Alpoge. It could do that, it could go to those two or OpenAI, like, how does that corporations or people for the purposes of this award, like? Oh

0:40:24 David: Jenessa, as an academic, you're going to love this. The attribution Line on the OpenAI paper is "OpenAI" one.

0:40:27 Jenessa: God What? The, like

0:40:35 Thomas: It would have to be Sam Altman or something, or

0:40:38 David: Would assume it should be Sam Altman or Sebastian Bubeck, the one of the engine, like It It should, right?

0:40:41 Thomas: Whoever was like doing the Yeah. Huh.

0:40:44 David: No, they're just like, no, OpenAI. OpenAI did this. So that's a problem. Answer to that question will have to wait for a couple of years, and we might need to do a follow-up episode in 2030 to see where that is.

0:40:57 Jenessa: Saw, like, okay, I read, like, some of this and then told you, and then we were planning this. The most controversial claim I saw was, like, hey, is it possible you just did what Buckmaster did because it was, I don't know, in their data set or they had, like, did

0:41:03 David: Yeah.

0:41:12 Thomas: I saw he alleged that maybe they accessed, like, his Codex chat history, or maybe the model used it, you know, that kind of thing. Is that what you're saying?

0:41:22 Jenessa: Know what Codex is, but yes

0:41:24 Thomas: Is the, um, Chat GPT Uh coding model

0:41:25 Jenessa: Okay. Yeah. But

0:41:27 David: Yeah, it's one of their interfaces. Um, is a loud voice in the mathematical community is, uh, that that is possible. My take is we don't know and we won't know. My take is also that it is quite plausible and possible in three or four different ways that that could have happened, to varying varying levels of nefarious, malicious, or just, we don't care. Um, I don't think that Buckmaster accused them of stealing their work. Buckmaster claims in his letter that he asked them whether they looked at his work and they said, "No, we didn't look at your user account."

0:41:28 Thomas: Yeah. Yeah.

0:42:04 Jenessa: Is his account in the training set even?

0:42:07 David: Question. Did they use some of his older work in the training set was enough of the general mathematical work in the training set. It seems plausible by their terms of service, almost certainly. The conversation in the machine learning field is mixed. Says, yes, it's very possible. The reply to that is, yeah, but at the volume of data that they're training on, it is unlikely that one user or a couple of user sessions would have that much influence on the model weight.

0:42:40 Thomas: 15

0:42:42 Jenessa: After 15 billion dollars worth of tokens, though? Oh Right.

0:42:45 Thomas: Million. Well

0:42:47 David: Billion tokens, $15 million of tokens. The training and the inference are different steps. The training is where they spend way more than, is where they send $150 billion of tokens of all of the language in the world that they can get their hands on.

0:42:50 Jenessa: Right. Right. Which would probably maybe include this. Okay

0:43:11 Thomas: Maybe, yeah, yeah.

0:43:12 David: It exceedingly implausible that it wouldn't. Their of service say, we may use this to improve our product. That mean they train with it? Does that means they look at it for after the fact, did you cuss out our chat bot? We don't know. The implication is that, yes, they use this for training. Yes, you can opt out, and they pinky promise that they don't use your conversation after that if you've opted out. Is there any way to prove that? That's probably an opening arguments episode about the OpenAI New York Times lawsuit.

0:43:44 Thomas: Yeah.

0:43:45 David: The answer is we don't know. OpenAI certainly aren't forthcoming, and in the absence of information, there are are many plausible possibilities.

0:43:41 Jenessa: Right. So

0:43:53 Thomas: I mean, they've, so they've said that there's no way any user inputs past July 3rd could have influenced the thing, but, like, maybe he was working on it before that, I don't. Hmm.

0:44:05 David: They very clearly said, like, we started working on this in August. Some of their work was on OpenAI. Some of their work was Anthropic. They've probably used local models as well. They've definitely had some of this work on

0:44:20 Jenessa: Maybe it could be if you just spend enough tokens you will eventually pull to the surface the guy who already did it. Oh

0:44:29 Thomas: Yeah.

0:44:29 David: That the problem was actually very close to being solved, and they were just the only ones working on it, right? Like, it could be that even if you didn't have their research notes, the rest of the writing in the mathematical community was pretty close to solving this, and the published papers were nearly there, and we just didn't realize, or the mathematical committee didn't realise it.

0:44:43 Thomas: Yeah, yeah. My opinion for what it's worth would be that I doubt it's that it literally took like their, you user files or any of that. Like, I feel like it's what you're saying now, or it's like, if you're putting that much money and that many tokens into it, that means you're, you're everything you just said, like all the current research, all the currently everything.

0:45:07 Thomas: And it doesn't strike me as implausible that that would just kind of brute force it. But I think that's kind of maybe a little less important than the broader point you're making, right? Which is like, this is not really a valuable thing to do. Like it's not a good use of resources, right.

0:45:06 David: Yeah, well, I they had just looked at their notes and used their notes, because then they wouldn't have spent $15 million dollars of natural gas to find this, like

0:45:32 Jenessa: They collaborated, like, they weren't willing to collaborate with the because of the other guy being a Anthropic, but if they had collaborated, they would have got there so much cheaper and faster because

0:45:42 David: They hadn't just collaborated, they wouldn't have spent 15 million dollars. They could have spent 15,000 or 1500, probably. Well

0:45:51 Jenessa: They, they were already almost there.

0:45:52 Thomas: Like pick one that someone else isn't already imminently solving. Oh,

0:45:57 David: They said they did, and they spent another 150 billion tokens, and they don't have on the other Millennium problems, and they don't have a solution for them, so I

0:46:06 Thomas: That's interesting.

0:46:08 Jenessa: Also that. That also makes it suspicious to me too. Like, oh, you tried all of them. And the one you solved is the one that somebody else was about to solve.

0:46:15 Thomas: Yeah. But, but it also has the other answer of like, that'S why these people were about to solve of it. It's like the most attainable one, but yeah.

0:46:18 Jenessa: Seems. Sure, sure. Yeah. Like a little bit of Yeah.

0:46:24 Thomas: Yeah.

0:46:25 David: All of these are possible. I don't know. I have been

0:46:26 Jenessa: Yeah, yeah.

0:46:26 Thomas: Yeah. Wow. What a, what an interesting story. Really strikes me as like kind of a, either way, a bullshit thing to do. And they're just really trying to get the positive press of like, look at our model. It'S solving these impossible problems kind of thing.

0:46:38 Jenessa: Yeah?

0:46:43 David: Playing with this analogy for this where Imagine there's a club of people. Who want to. Parks in New York City, cool. Lovely parks in New York. A couple of them have said, you know what? It would be really cool if someone visited every park in New York city. That would be pretty cool. We're going to take e-bikes and we're gonna use our e-bikes to do it in a year. And they, you know, this weekend they're doing it. That weekend's rainy. They don't do anything. They're pedaling around. They visited. They've got three parks left. They are so close. They got four parks. They were about to do, they're on their last weekend, they're coming up on, on their last little, you know, rinky-dink, you know, five-square-foot park, because New York has little five-square-foot parks. They're almost there, and OpenAI is like, you know what? We're gonna do, we're gonna do this, and we are going to roll coal over the finish line. Why? Yep.

0:47:31 Jenessa: Like, missed the entire higher point of why anyone was doing this in the first place. Oh.

0:47:38 Thomas: Right. There's no good reason to do it. There's, there's reasons that might make sense for their PR, but it's not like the, it's, the point is, it's not like a value. They're the park people were about to get a bunch of good press and they wanted that good press. Uh, it's not like a good reason. Yeah. Because

0:47:52 David: It's collaboration, the teamwork, the humanity, doing things together this

0:47:56 Thomas: To your point, when you started this off, it's not like, oh, this is the last equation we need to build our time machine, and if anybody you know, like, solves it, we can plug in X and then we get a time machine. It's like, like we can do everything pretty much that involves this. This is like, this is just like a cool math thing, right?

0:48:03 Jenessa: Interesting.

0:48:17 David: Is a cool math thing, and maybe we find other cool math things on the way.

0:48:22 Thomas: And to bring it all back around, the thing I was vaguely remembering was this article from Penn Engineering that said the snow graphics in Frozen can predict the mechanics of real avalanches. And it was, It's a case of like the science going back and forth. It was like the science fueled the CGI, which then fueled the science again. Again, and so I thought it was kind of cool, and I guess it was Frozen one. I was older than I remembered, right?

0:48:34 David: Great. Yeah. Well, Take that to the next step. What can we do when we know the of avalanches better? We can better rescue people from avalanche and We can make a better Frozen three. All of these things, all of the things that we can do do.

0:48:53 Thomas: Yeah. Make a better Frozen movie. Yeah. Oh, no. Okay, sorry. Yeah. Interesting. Yeah.

0:49:03 Jenessa: I guess just like the joy, I, for me, always in science was the getting there. Like, the answers are cool, but, like, you learn so much on the way to the answer, and that's, like, the whole point. And if some machine machine just

0:49:18 Thomas: I also think there's like different kinds of problems Right, there's plenty where it's like, oh, you, if the, the ChatGPT wants to solve, like, curing this, this particular kind of cancer, I don't care about the journey to get there. Yeah, it's like, burn those tokens, man. That's fine.

0:49:19 Jenessa: Yeah. Sure. Whichever Yes. The answer is good. Yes. The answer I want it. Yes. The

0:49:34 Thomas: But yeah, but this kind of thing is like, that was point. It's like it did a bunch of Sudoku for you.

0:49:39 Jenessa: Was the point. Yes.

0:49:40 Thomas: Yeah, It's Like that we could have just done Sudoku and had a good time that I know you can solve it.

0:49:34 David: Yeah, yeah, Yeah!

0:49:41 Jenessa: I wasn't doing the Sudoku because I wanted the answer. That was definitely, I was trying to fall asleep. Actually, this

0:49:50 Thomas: Yeah, I wanted to know, ah, what is this corner? It's seven. I needed to know that seven goes in this corner, or, thank you, machine. Yeah.

0:49:53 David: That felt

0:49:56 Jenessa: Is why I quit all those math classes. Oh,

0:50:01 Thomas: That's not the takeaway message from this, uh, leave it to Jenessa to ruin our good time. No, I'm just kidding. Well, Well, this was a lot of fun and yeah, David, I'd be happy to have you back to talk more AI sometime, but I think overall top line, and you give me your overall top line, it sounds like this was kind of dumb. And they should have done this and blown all these tokens

0:50:05 Jenessa: You're right. Oops, excuse

0:50:25 David: Yeah, there a couple of problems that AI can be very effective at helping with. This is not one of them.

0:50:33 Thomas: All right, well, thanks so much. And Jenessa, uh, make, uh, Venmo your finders fee for, uh, David. Excellent. I'm

0:50:40 Jenessa: Just gonna keep bringing you random people I know.

0:50:43 Thomas: Cool with it. They seem pretty cool. All right, thanks so mucho. I really appreciate it.

0:50:47 David: My pleasure.

0:50:48 Jenessa: Talking to you. I'm David's also a secret space nerd.

0:51:09 David: It's not secret.

0:51:11 Thomas: This is the last equation we need to build our time machine, and