SIO407: Entropy and Information

Transcript

This transcript was generated automatically and may contain errors — the audio is the record.

0:00:17 Thomas: And welcome to Serious Inquiries Only. This is episode 407. I'm your host, Thomas Smith, and we've got some physics for you again. Dr. Brian Gillis joining us for the umpteenth time. Whatever number that is, it's not enough. Always love chatting with Brian, learning more about our universe. This time, we're going to be talking about entropy. Entropy. I'm sure that's a term you've heard, or maybe you know a bit about.

I not only have we never really just covered it straight up on this show, I don't think, but also Brian is coming at it from an interesting angle, talking about entropy and information and computation and thinking. He was actually inspired by the food episode about, you taking energy, burning calories, but to come on and explain, maybe in more fundamental terms, why that is, why that's necessarily true. And we can learn more about physics and cool stuff like that along the way. So looking forward that conversation.

Before we get there, as and I would be absolutely honored if you You supported the show on Patreon.com/seriouspod. You'd get access to ad -free episodes to these seriously awful movies. Oh, that should come out any moment. One was a lot of fun. Oh, I can't wait to get you that one. That monthly hangout. That has been a good time. We actually put out a poll for everybody.

That monthly hang out, by the way, that's for anyone in the top two patron tiers for either show, either Serious Inquiries Only or Where There's Woke, which you should definitely check out. And we're putting out a poll to see when people can make it because it's been so much fun. We want to make sure everybody can make It next time. Been playing a bit of Codenames and it has been an absolute blast. Those and more great reasons to join patreon.com/seriouspod. So if you have, thank you so much patrons. And if you haven't, sorry, well, you're probably going to hear an ad right now, or maybe not. Not if you live in a place where they don't serve ads. I don't know. Let's get to Dr. Brian Gillis. Here we go. Dr. Brian Gillis, welcome back to the show. How are you doing today?

0:02:16 Bryan: Thank you, Thomas. I'm doing great. How are you?

0:02:18 Thomas: Am so pleased to be speaking with you. It feels like it's been too long. Any amount of time without some physics in my life is honestly too long, though. So, you know.

0:02:22 Bryan: Yeah. It's pretty hard to live without physics. Um, I tried it once. Didn't go very well.

0:02:29 Thomas: Yeah. I was going to make it like an abstract joke about that, but you can't even really. Like anything, anything. Oh my God, my brain. Any object, any reality would be described by something we would call physics. Astrophysics. Oh, my God. There's no way. I can't make it. I was like going to say, oh, I was just floating like one dimension. No, even that would have some sort of physics to it, right?

0:02:32 Bryan: Yeah. Yeah. Yeah. I, I mean, the best would be something where their physics is completely disconnected from ours and Doesn't share any rules, but there would still be something.

0:02:57 Thomas: Yeah. So, you're right. There has been plenty of physics in my life. There hasn't been as much astrophysics, but I was super excited. You left a really great comment about the recent, it episode, and we were talking about brains using energy, and there's a little bit of an interesting conversation.

I'm still, this is not the topic of today's show, but signposting, if anyone has any knowledge or like knows any way to answer this question or anyone who could answer this question, I am still interested in a question that I don't think really came across correctly in that episode, which is, does it use more energy to be a chess chess player struggling really hard to do it, which feels like it takes a lot of energy to me, or to be a grandmaster who's got a lot of it, like it inhabits an instinct.

I'm curious about that. If anyone could send an email, any way to answer that, I don't know. But you took more of like an entropy related kind angle on it. And I realized like, oh, shoot, I don't think we've ever just covered the basic concept of entropy. And there's so much to talk about there. So I'm excited. What do you have for us today?

0:03:05 Bryan: Oh, yeah. Yeah.

0:03:58 Thomas: Day.

0:03:59 Bryan: So I think this is a good way in, as there are entropy -related reasons to answer the fundamental question, why does thinking use energy at all, or why does it have to use energy? Entropy is really the answer to that. So maybe let's just start at the basics. Thomas, if you were to define entropy, how would you define it?

0:04:19 Thomas: I Think it's like layman's, the definition seems to usually be like, well, things tend to only get more disordered. Like you can't spontaneously, in a closed system without, you other, uh, influence, I guess, over time things go from like more ordered, maybe, to, to more disoriented, and you can't go backwards really without some outside influence. So the example I think I usually hear is like, once you pour that milk in your cup of coffee, the first state of that is like maybe, well, I guess you have just coffee, but then you have the cream or whatever is just kind of in one little bit, and then it just starts to spread, and it's never going to spontaneously go back to being like the one little blob in your coffee when you poured it in. Did I not do a good job.

0:05:03 Bryan: Yeah. So disorder versus order. Yeah. You've

0:05:07 Thomas: Your answer was like, oh, I've got work to do.

0:05:07 Bryan: Got it. Yeah, disorder versus order is actually the way people most commonly talk about entropy. It's not really how I like to describe it. And I think it can actually be, can give you the wrong impression of what entropy actually is in some circumstances.

0:05:20 Thomas: Sure. Sure. Right. It, as I said, that I was like, oh, that's probably more of the like theological way people frame it to make it be like, see, you need a god to have created the order, but I didn't have a better word so feel

0:05:39 Bryan: Yeah. Yeah. Well, it's hard to define it simply. And I think one good way to show why order isn't the right way to think about it is imagine you take a standard deck of playing cards. If you buy one at the shop, it'll usually be ordered like ace-two-three-four of one suit, then of the other suit, then of another, and intuitively you'd say, oh, this is a very well -ordered deck of cards.

0:06:04 Bryan: But from an entropy standpoint, every possible ordering of cards is entropically equivalent, because it has an equal probability to end up any specific way. Way. And so what we as humans call order doesn't really correspond to entropy very well. Fact, so with a deck of cards, since they all end up having the same entropy, it isn't very useful metaphorically to even talk about a deck of cards. I think the better way to think about entropy is instead of cards you've got a bunch of coins that can be heads or tails.

0:05:57 Thomas: Mm hmm. Mm hmm, right. Right.

0:06:40 Thomas: Can What

0:06:40 Bryan: Let's say we've got 100 coins. You flip Flip them all randomly, lay them out on a table, and count how many heads or tails they are. Let's say you flip them all, you count, you said, okay, I've got 54 heads, 46 tails. Now, that number, how manyheads, howmanytails, is going to be associated with one entropy value for the set of coins. It's specifically the total number. It doesn't matter for the entropy which coin is heads and which is tails. It's just the overall number.

So if by chance all the ones on the left are heads and all the runs are right are tails, mater for this entropy calculation. Are, of course, some other circumstances, like if the way you're calculating entropy is whether a particle in the box is randomly on the right or left, then it matters. But I the point of entropy is that we treat the particles as indistinguishable. And so it doesn't matter which ones have a given energy state.

It's just how many are in one state or another. I say energy state here, this is if we're being really careful about the physics. For entropy, it has to be particles in a different energy state, like some are higher energy than others. And for this metaphor, we'll just say heads is high energy, tails is low energy.

0:08:03 Thomas: Uh, okay, I want to make sure I'm, I'm going along with you. What, what is that, could you explain that a little more? Why does it need to be that, that you're saying, to, to properly be an example of, like, defining entropy, the, anything you're talking about, if it's an analogy with cards or with with coins, there has to be like a higher energy state and a lower one or more.

0:08:21 Bryan: That's how it works out physically. And part of the reason for that is that there are an infinite number of ways you can define and decide upon properties of objects. Project, like, we can decide, okay, what matters is whether the coin landed heads or tails. Or we could say, oh, no, what matter is the angle the coin is oriented.

0:08:39 Thomas: Yeah, what year of the coin? Yeah.

0:08:43 Bryan: Yeah. Or how close to the center of the table did the coin land? There are so many different ways. And each of those ways would give us a different value of entropy. And

0:08:51 Thomas: See. So are you saying it needs to be something that's like, for lack of a better word, objective to the universe? Like there's no human value judgment on it. It's like nope, this is something and this is nothing else. And there's not way we could order it that would like change that. Or no way we could look at it that would really change that component of it. Oh,

0:09:09 Bryan: And that's what the the energy level does, because the total energy of the system is conserved, and it gives us an objective way to look at it. And entropy is also very closely related with energy. The energy. In fact, probably the best definition I have of entropy is it, you, the amount of energy in a system that can't be used to do any productive work.

0:09:34 Thomas: That's interesting. Don't think I've ever heard that before.

0:09:38 Bryan: To think about this is, like, we all talk about using up energy in day-to-day parlance, like this example of thinking using up Energy. But physically speaking, Energy is conserved. There's a little asterisk there, and for more, go back to the episode we did on Portal. But that doesn't come into play here.

0:09:44 Thomas: Hmm. Yeah. Not until I shoot you with my portal gun, it doesn't.

0:09:59 Bryan: So, properly speaking, energy is conserved. So what are we actually saying when we're using up energy? What we mean is that there are certain types of energy which we can use to do work and certain types which we cannot. Of this This is gravitational potential energy. If we have something up high, we could do work with it. We could let it fall on a lever and cause that to push things, or that lever pushes something else up.

0:10:22 Thomas: Hmm. Say, or just thinking about the body, use energy in certain chemical forms. You eat food, you can enjoy certain chemical form there. It doesn't, like I say, it don't go away, but it may turn into body heat and we can't really use the body heat to like do more things. Is that kind of what you mean?

0:10:43 Bryan: Right, yeah. And heat is kind of the classic example of the type of energy we can't do anything with. So once we've, quote-unquote, used up energy, we turn it from some form like gravitational potential, kinetic energy of linear motion, into temperature, which is just random kinetic energy. And once it's in that form, it's much harder to use. Now, if you have different areas with different temperature, you can use that to get some energy out of it that you can do something with. But once everything has the same temperature, which will eventually be the case as you use up more and more free energy, then there's basically no productive work you have left that you can possibly do. All that's going to happen is that that random motion will just cause the universe to expand. That's about the only thing non -free energy is good for.

0:11:41 Thomas: Non -free energy is good?

0:11:43 Bryan: Yeah. So free energy is the energy you can use to do work. Like, yeah.

0:11:47 Thomas: Oh, oh, I mixed those up. Okay, yeah, yeah. Sorry.

0:11:49 Bryan: And non -free energy is energy that you can't. It'll contribute to the expansion of the universe, but that's about it.

0:11:57 Thomas: Mean, it's not nothing. Like give it some credit, you know. It's not that easy to expand the universe, Brian.

0:11:58 Bryan: Yeah. Entropy is basically the way of measuring how much energy in a system is in one of these states.

0:12:11 Thomas: Interesting.

0:12:11 Bryan: And in particular, entropy measures how much non -free energy there is in the system.

0:12:16 Thomas: Oh, okay. So earlier I was saying, oh maybe I was leaving the door open like, oh two or more energy states. But is it really just two? It's really just these two? Or does it get more complicated than that?

0:12:27 Bryan: More complicated than that. Are like a lot of ways energy could be free and lot OF ways that can go into entropy. And we're just trying to simplify things down here. It helps to think of there are just two ways here. At once. Way entropy relates to free energy is just a pretty simple equation. You take the total energy and subtract off entropy times the temperature of the system multiplied by a constant to properly convert units. Does So mean that there's an interesting relationship here between the energy, the entropy, and the temperature. That as you cool things down, the non -free energy in a system decreases.

0:12:28 Thomas: Sure. Okay. Sure. Okay. Yep. Hmm. Okay, okay.

0:13:16 Bryan: This has some interesting implications. That basically means that you quote-unquote lose less energy, or less energy goes from free to non-free, when things are operating at lower temperatures. One of many reasons that a lot of high -performance computers try to operate at low temperatures as you get lower operating losses.

0:13:29 Thomas: Wow. Right. Yeah, that's what I was thinking too. Yeah, everything seems like it needs to be cold. I think I asked, jokingly, Quantum Anne one time, like, man, is there any computer that can to be hot? Because that would be nice. You know, that would make it easier. I feel like everything needs to to be frozen in these weird, you know, containers that make it really difficult.

0:13:38 Bryan: Yeah, yeah.

0:13:52 Bryan: I think for quantum computers, the reason they do it is more related to getting superconductors, but there might also be an entropy -related reason for that thing superconduct more at lower temperatures. I'm not sure. It's not my specialty. We've gotten what entropy is, what free energy is, and non -free energy.

0:13:58 Thomas: Yeah. Yeah, sure. Okay. Would you mind putting in terms, because I feel like maybe this more technical definition, maybe I've, I'm not sure if I've not followed or if I, could you put it in terms of the coffee? Was that a bad example or would you be able to cast that in kind of those energy terms? Because I'm sure I would know how to do that.

0:14:19 Bryan: Yeah.

0:14:30 Bryan: So the coffee cup, it's a bit difficult to put into energy terms. I think it's easier to think of gases for this rather than liquids, since gases are a bit more flexible. So, let's just say you've got this aquarium full of air. Let's say one state is that all the air in the aquarium is smoothly distributed throughout.

0:14:37 Thomas: Okay. Sure.

0:14:52 Bryan: It's equally likely to be found at any point. Now if you were to try to get any energy out of the thermal motion of the air, you'd find it's basically impossible. Like you put a little propeller in there, it's not going to spin because, Yeah, all the area is moving in each direction equally likely. Now let's go to another case where you've got an aquarium where you push all the air into one half of it and then you put the propeller in between the two and you take out the divider that was pushing it and now the propellor is going to spin.

0:14:51 Thomas: Kind of just sits there. Yeah gotcha.

0:15:28 Thomas: Yeah, all of a sudden super fast vacuum action.

0:15:33 Bryan: Yeah, so happening is that you've got all the same particles at the same temperature. You've just decreased the entropy of the system by moving them all into one specific state, like the left half of the equation. And then once you remove the divider, the system gradually, through the random motion, will go to a higher and higher entropy state. And we've set up the propeller in a way that, that as the entropy increases, it converts that loss of free energy into useful work that we do somewhere else.

0:16:09 Thomas: Gotcha, okay, that makes sense.

0:16:13 Bryan: Now maybe we can get to why thinking and computation uses up energy. So, I think the important part to take away from what we've just talked about is is that free energy is equivalent to negative entropy, or entropy is equivalent to a loss of free energy. And this relationship goes both ways, that if you decrease the free energy, you increase entropy. You increase entropy, you decrease the free energy, just a tight equation. It's just this is how they're defined. And so the reason thinking uses up energy is because thinking increases entropy. This is the point, which is why I don't like to think of entropy as disorder, as thinking, would think of thinking or computation as intuitively adding order, figuring out new Things. Things, in fact, from an entropy perspective, it's increasing the entropy. Yeah.

0:17:12 Thomas: I mean, I was just thinking, like, what if you're really bad at thinking and you produce just disordered thoughts? That's still gonna take energy. Yeah.

0:17:25 Bryan: It's still, well, um, the the increase in entropy doesn't really depend on how good you are at thinking. And, fact, uh, speaking, the part that increases entropy isn't the thinking itself, but it's preparing space to think into, or that is, like, erasing parts of your brain where you, you can store new information.

0:17:31 Thomas: Interesting.

0:17:49 Bryan: Know, if you think of like a computer, let's say you want to do a new calculation, like out your calculator, you want to add five and two, you're going to need some place to store the results. So you clear up some space, then do the calculation. And it's actually the clearing up of space in the first bit, it. That's what increases the I think probably the best way to go into why this happened is go back to the metaphor of coins, because coins with heads and tails, we can talk about like bits on a computer. So, like, heads can be one, tails can be zero. And

0:17:48 Thomas: Got It, okay, yeah, that makes sense, okay.

0:18:29 Thomas: Oh.

0:18:30 Bryan: Imagine you have information you want to store and you've got a coins bunch that are randomly distributed, like you've got a hundred coins on a table, you can do a little drawing in them, you want to make a big U shape. Well, before you draw the U, you have to erase whatever else is there, and the coins are already going to be randomly distributed, so you're gonna have to go through, check each coin, you know, if its heads, flip it over, if it's tails, leave it alone.

0:18:42 Thomas: Man.

0:19:00 Thomas: Is this gonna piss off the computer engineers? Because I don't think hard drives work that way. Don't they just leave whatever's there there and then only overwrite the parts that matter?

0:19:09 Bryan: If you tell it to erase the file, it just deallocates it. But if you want to write to something first, you have to write over every bit there. It might happen that when you wantto save something, where you're saving it to already has something, But you can't just turn all the proper zeros to ones. You have to turn some of the ones to zeros there.

0:19:13 Thomas: Hmm. Okay. Right.

0:19:33 Bryan: Depends on the specific operation, whether you zero it out first and then add things, or you go over and adjust each bit. I'm not a computer engineer. I can't speak to which is done. But in some sense, you do have to erase over what's there in order to add something new.

0:19:51 Thomas: And in some sense, it wouldn't matter because like on average, you know, I was just I'm just trying to think it through. It's like, oh, well, this coin happens to be already what we need it to be. But like on average, that's all going to kind of cancel out because, you you're going to have to change a certain number of coins on average to to make the thing you're trying to make. Right.

0:20:07 Bryan: Yeah. Yeah. It's all going to average out in the end. And this is the part that scientists believe theoretically for a long while, it took them a while to prove this, but this erasure is what increases the overall entropy of the system. And the what happens is you have to erase things, take it down to zeros, which initially in this step actually brings the computer to a lower entropy state.

0:20:34 Bryan: But by the laws of thermodynamics, the overall entropy can't decrease. So you have radiate that away as waste energy when you zero something out. So that goes down into lower entropy, and you've radiated away that entropy. So total entropy is equal. Then within the computer, you move some of the bits up to ones again to store whatever you want to store. That increases the entropy in the computer. But then you've also got that bit that you radiated it away in the first part. So now the total entropy of the system has increased.

0:20:09 Thomas: Right. Right, and does matter that the total entropy of the system, like, do we need to keep track of that? Because in this computer example, I mean, we're plugged into a wall, right? So doesn't that constantly, like mean that the entropy could kind of do lots of things because we have that external energy source?

0:21:27 Bryan: Yeah, and have energy, you can increase the Entropy.

0:21:31 Thomas: We are masters of entropy. We can decrease it at will.

0:21:34 Bryan: Yeah, and is going into the explanation of why we need the energy in the first place, why can't we do computations without using up energy? And you know, there actually are thoughts about how to do this. This. So talking about here is what's called the Landauer limit, named after Rolf Landauer, who proposed this in 1961, I think. And he basically said that if you go down into the physics of the system, you can calculate very simply how much energy is the minimum that you you need to, quote unquote, use up in order to overwrite one bit of data. And it ends up being like the Boltzmann constant, that's the big constant that connects temperature and energy, times the operating temperature, times the natural logarithm of two. And that comes in because a bit has two states. And the is how entropy works. Because otherwise, the number of possible states is like each bit is 2 times 2 times 2 times two times two. But entropy is supposed to be additive, not multiplicative. You take the natural logarithm. And if you didn't understand of any of that, don't worry. Just Yeah.

0:21:39 Thomas: Right, right. Oh, Hey, everyone. Math and then thing. Yeah, okay.

0:22:51 Bryan: Just yeah. We've got this equation. That's the minimum amount of free energy that needs to be up used to do one bit of operation, at least theoretically. This is a very tiny amount of energy. The most efficient computers we currently have use about a million times more energy per bit of operation, according to sources I found. It was actually a bit contradictory. Some said millions, some said billions. I bet there's just uncertainty some between which computers and stuff.

0:23:01 Thomas: Yeah? Huh? That's interesting. That would imply we have a lot of room to improve efficiency of computing hypothetically. That's interesting.

0:23:28 Bryan: Well, Well, if you just look at the land hour limit, that's true. But there are a lot of other things that get in the way.

0:23:36 Thomas: There's the other guy from The Odd Couple. It's not just Martin Landau, it's... I don't remember the other guy, but he must have a limit too. There's got to be more than one.

0:23:39 Bryan: Yeah, yeah. There are a bunch of limits. And yeah, this is one we know we can't go below. But we're actually, even though we're so far off of it, we're already facing another one which is causing difficulty, which is quantum tunneling.

0:23:46 Thomas: I don' t know what that is. That's a limit on computing.

0:24:02 Bryan: Yeah, it actually is. So quantum go to a metaphor again. Let's say you're the wilderness and there are two neighboring valleys. And in one of the valleys is a ball. And this ball is rolling a little, but it doesn't have enough energy to get up over the side of one valley into the other. So in normal physics, that ball's just going to stay there. If you gave it more energy, it might be able to get from one valley to another, but it doesn't. Going to say there. In quantum physics, when not looking, the ball might sneak over to the other valley.

0:24:21 Thomas: Oh, ok.

0:24:37 Thomas: Yeah, doing all kinds of crap when we're not watching. It

0:24:39 Bryan: That's basically what quantum tunneling is. And if you go through the equations of quantum mechanics, you can see how this bizarrely makes sense, that the ball doesn't move as a particle. It moves as a wave. And the wave function will be well -bounded within the valley. But when you calculate it out, it turns out the wave function can leak out the sides of the valley a bit and into the other valley. And only when you look at the ball in some way does it turn into a particle again. Yeah, not really turn into, but you perceive it as a particle.

0:25:17 Thomas: Yeah. Got it. Don't look at the ball. I'm just kidding.

0:25:19 Bryan: So if you don't look at the ball for long enough, eventually it becomes like a coin flip, which of the two valleys it's in. Miniaturize this into a computer. You're trying to store one bit of information somehow in a computer, and no matter how you store it, There's always that minuscule quantum tunneling chance that it'll flip from one value to the other.

0:25:21 Thomas: Yeah. Oh. I see.

0:25:44 Thomas: So, and we need to expend some amount of energy to like prevent that. Is that kind of what the limit is? Nice.

0:25:49 Bryan: Yeah, there are a lot of ways around it. You know, one of the ways is, just you make it bigger. Quantum mechanics happens on a small scale. If you go to a larger scale, it gets less likely for something weird and quantum to happen. So if the bit is covered by, you not just one electron, but a few hundred, it's a lot less likely they'll all quantum tunnel over to the other state. Or you might be able to perceive which has more. Or I think what actually they do in most systems, it depends on how you store the bit. Different mediums do it different ways. In an active computer, I think it's an electric current. But there's still the possibility of quantum tunneling switching that around. Stronger the current is, the less likely that is to happen because there's more of it. Like, there's a decent chance one electron will skip over. But if you've got a flood of 100 electrons going through, yeah, yeah.

0:25:51 Thomas: Yeah. Sure. Right. Yeah. And it's a binary thing. So yeah, it's not going to flip this bit essentially if some want small. Part. That makes sense. So there's that, that limit is essentially like whatever energy or inefficiency that causes, we can't really get around that, otherwise we'd lose, lose the ability to even store bits of data, kind of thing.

0:27:00 Bryan: And so there are other ways around this as well. Like keeping it big will decrease the chance, but it doesn't eliminate it. And they're also, you know, like with 100, they could all go over. It's less likely, though. But, you Know, there are also, You inefficiencies and that we just can't make things that behave perfectly. And there's the environment around us can cause issues like since things aren't at a zero temperature, things are vibrating around. This could cause things to flip over inadvertently. And here's where the astrophysics comes in. There are what we call cosmic rays bombarding everything.

0:27:07 Thomas: Hmm. Right. Ah, yeah.

0:27:39 Bryan: Yeah, rays are basically ridiculously high energy photons coming in from outer space and from somewhere in the universe. Yeah, just like there's an alien holding up a big light, just trolling us, just watching the data on our computer slowly be corrupted.

0:27:46 Thomas: From the aliens. No, Yep.

0:27:57 Bryan: Disrupted and And These cosmic rays are actually a very big deal and people not understanding them have uh uh pretty big chaos um so essentially what happens if you have a high energy photon and you know depending on how the computer or works um a bit different but it's a known thing in most computer systems that it can flip a bit at any given time. It's just a you have to live with. Are a lot of things that have been blamed on cosmic rays. Like is that some player was trying to speed run the game Super Mario 64, and all of a sudden Mario jumped up like a thousand meters in the air. And he He thought, oh, this must be an amazing new glitch I've discovered. And, you a whole thing of, like, a bounty was put out for anybody who could figure out how to recreate the glitch. And people tried and tried. Had no luck. And eventually somebody looked into the code and said, okay, maybe if this bit was flipped at just this time, it would cause this effect.

0:27:57 Thomas: Yep, yep.

0:29:02 Bryan: And one of the low stakes. There was one time this happened in a computerized election.

0:29:02 Thomas: That's crazy. On a newer version of Mario. Oh no.

0:29:09 Bryan: No, it happened in an election. Somebody got like, I the exact number, but it was like 1 ,024 more votes than they they actually did. And it exceeded the number of people registered in that district by a fair margin. And the people investigating it saw that and were convinced somebody was trying to commit fraud. And eventually, they had to be convinced that this was a risk of how computers operate, that this could happen sometime. And yeah.

0:29:11 Thomas: Oh, God.

0:29:39 Thomas: There's no way we're convincing Trump voters that it was a cosmic ray. I'm just saying it doesn't matter how many facts there are, whatever. Want to go back to the thing of like the entropy is from storing the calculation or from Preparing the storage, because that actually makes a ton of sense to me, just thinking in terms of, you know, I know a little bit about basic computing. It makes sense because if you think of a circuit, and I want to ask you if this is because of this or just, I don't know, coincidental or something.

Thing, but if you think of how a basic, like, kind of logic circuit works, and I may use the wrong terminology, but I know the things I'm talking about, you've got those logic gates, and you, the basically the fucking fundamental nature of all of our computing, everything at its core is these little logic gates that you're talking about, the binary Ones And zeros, and if you line a bunch of those up, you can do all kinds of calculations, you can doing any, you know, math, all, all that kind of stuff. But like, if you aren't storing it, then really what you're doing is you're setting up a circuit, and maybe it goes through some logic gates, and maybe goes to enough of them that you've input something that's the equivalent of two plus five, like you're saying, and the output is the binary for seven.

But if you don't store it anywhere, that's just a circuit. And the energy goes out the other side, possibly, depending on the gates and stuff set up, but like you haven't done anything, right? Is that kind, kind of what you mean? Because in order to store it, you'd have to clear out some storage and put it in there. And then that takes a little bit of energy versus if you just turn the circuit on and back off.

Like, yeah, theoretically, I think the answer can come out the other side without having used really any energy, possibly, on how the logic gates work, but you won't have done anything. Once the switch goes off, it's over. You haven't done anything, okay?

0:29:49 Bryan: Yeah, sure. Yeah. Yeah.

0:31:16 Bryan: What's going on is you have to zoom in on the logic gates themselves as the gates themselves are actually where we lose energy, at least in the current methods of computing. So let's think about a logic gate, one of the most basic, the AND gate. That is, you have two currents going in, one coming out. And so the AND AND gateway, if both currents are active, you'll get an active current out. If either or both currents are inactive, you'll an inactive current out, and if look at the two states of the system before and after, you can see that before you've got two bits of information, two currents that can be either on or off, and

0:31:25 Thomas: Right. Yep. Yep. Yeah. Yeah. You're

0:32:01 Bryan: After you've one bit of information.

0:32:04 Thomas: Totally right, that makes way more sense, I overthought it. No, that is, yeah, because you definitionally you can't get back that energy with an AND gate, because you have to do two on to get one on, like if you have two on inputs, which take two charges, as you're saying, you, the most sort of energy you can get back out is the one on, which is or one, you know, the number one, I guess, whatever you want to call the bit, and that's definitionally essentially half the energy, give or take, and you cannot do that any other way, right?

0:32:33 Bryan: Also works from an information perspective that you've got two bits of information in, one out. And for the AND gate, if you see one on coming then you know that they were both on. But if you you see it off, there's three possibilities of what could have gone into it. So, you can't narrow down any further from that. You can't recreate the input. And, so, from an entropy standpoint, we look at the information in the two states. So, going in, you've got the first one could be on or off. The second could be one or off, so that's in four states. And coming Coming out, you've got on or off, you got two states. So the entropy calculation, so you can tune out if you don't like math, but you basically the two to get the decrease in entropy. So four divided by two is two, then take the natural logarithm, then multiply by the temperature, multiplied by Boltzmann's constant, and you get KT ln 2 is how much entropy is changed in that operation. So here we go from, yeah,

0:32:48 Thomas: Yeah, yeah.

0:33:20 Thomas: I was going with you until whatever those words were. KT long... What?

0:33:46 Bryan: K is constant, K stands for constant, T stands for temperature, ln, natural logarithm, and then 2 is the number that comes after 1.

0:33:49 Thomas: Oh, okay. Yep. Ah, I think I'm familiar with two. Thank you. Yeah, yes.

0:33:59 Bryan: Okay, yeah. So that's how we calculate the entropy change. And so in this model of computing, if you want to do anything with these gates, you're free energy with every gate you go through. This applies to any of the gates that take in two inputs and output one. There's also the not gate, which inverts any current. And that one, since it has the same amount of information in and out doesn't necessarily cause any increase in entropy, though since you have maybe an current going to inactive or inactive to active, something's happening with energy

0:34:04 Thomas: Interesting.

0:34:36 Thomas: Yeah, has to go somewhere.

0:34:38 Bryan: There. Yeah, so there's some energy, but if we're just thinking about entropy, there's no change there.

0:34:43 Thomas: That's so cool. Yeah. The information component of that is really interesting. You go from, like about the and gate, you've got two, would that be bits? Yeah. You've got 2 bits going in and they could, as you said, there's, so there's four possibilities there. That's easy, right? Two times two or whatever. And then once you have that answer, it's either one or zero, it's on or off or whatever you want to say. And you've crunched it down to one bit. And like you say, you can't, if it's one, you know the answer. If it's zero, you don't know what all the, what all the inputs were. And that's like a loss of information. That's interesting to think about in that, in those kinds of mathematical terms. I'm sure there's some, you numbers or formula way of saying like what to, to quantify the amount of information lost there. Well, But I guess it would just be one bit, right?

0:34:52 Bryan: Well, bit is the technical term for, it's the fundamental unit of information that, you know, if you want to define it in words a bit, is the amount of information that reduces the uncertainty by one half. So just zero or one. It's a binary piece of information. You This gives us the fundamental limit, but specifically this is the limit for this type of computing. Let's say that we could figure out a way to get around the quantum stuff. Maybe a quantum computer or something could handle this well enough that we can do a lot of calculations and not have to worry about quantum tunneling getting in the way. And let's say we figure out the technology and we come up against this limit. You think it might be possible at all to figure out a way for us to get below this limit? The

0:35:33 Thomas: For For information, yeah. Gotcha.

0:36:22 Thomas: Sorry, which limit?

0:36:23 Bryan: Land hour limit, the limit to entropy or the energy because of this loss of information.

0:36:24 Thomas: Oh, the, yeah, yeah Hmm trying to make sure I've got the hypothetical correct Because I just know that quantum computing takes so much fucking energy That I don't know if that complicates things But I guess just hypothetically A qubit, it's gotta be I know it's like than a bit Like it's more Just based on those episodes

0:36:45 Bryan: Yeah, don't worry about quantum computing. Let's just think about the logic gates. The problem we're facing is that we've got two bits of information in and one out. And let's say we want to add two numbers. You can set this up to do, I think it takes five logic gates per bit to perform addition. Um, that's, you're losing entropy five times because each logic gate cuts down, uh, information. What if you could change the way you do computing so that instead of this type of logic gate, you have another type, which is always, okay, you Have two current in, it's always two out. You have one in, it's always one out. You have three in, it's Always three out. If you do something like that? And yeah.

0:36:46 Thomas: Okay Okay. Okay. Oh, that's interesting.

0:37:33 Bryan: And so this is an idea some people have had. It's not practical to put into use yet as we have so much other energy loss anyway, but maybe in the far future we'll get to this point. Idea is we do what's called reversible computing. Computing and you can set up a type of logic gate. And it turns out what's most useful is a input, three output logic gate, which depending on how you set it up, you can have a pair of them acting like in of the other fundamental types of logic gates. You can have an AND gate, OR gate, NOT AND, NOT OR, exclusive OR, or exclusive NOR. I think that covers them all. Could set it up so that any two of those will operate as one of those. And then you have the third current out, and the that went in will have two currents out. And by combining all all three. You can always uniquely describe what went into it.

0:37:49 Thomas: Yeah.

0:38:33 Bryan: So,

0:38:34 Thomas: That's interesting is what I'm thinking of is if all your inputs are on, how could you do anything other than lose information? Because if all your inputs are on, anything less than all your outputs being on would lose energy or whatever, but then that doesn't tell you anything, that would just be same. Does that make sense? I don't know.

0:38:54 Bryan: Well, I think that you always have three bits of information in and one bit out is how it works. So I'm just, three bits in, three out always.

0:38:54 Thomas: Oh.

0:39:03 Thomas: So you're still losing, well, then you're losing more energy, aren't you? Three bit in and one, maybe I'm not understanding it.

0:39:11 Thomas: Yeah. Yeah, okay, so what I'm saying is if all three of those are on, in like all three bits are turned, are ones going in to the circuit, how could that work? Because anything, any answer is either going to be three on, or anything, anything less than that, you're kind of losing, losing, sorry, you, you're increasing entropy. Right?

0:39:30 Bryan: How it work is that you have three inputs, We'll call them A, B, and C. two inputs, A and B, are always just copied into the output. We'll just say A prime, B prime, and c prime are the outputs. So A prime is always equal to A, A, b prime is Always equal to b, and c prime gives you some of information out. And it does this complicated operation on A, B, and C to determine which of one and zero it is. But the result is that you can always take all three of those and invert this operation. You can what C to get the result of any of the other logic gates you want.

The operation here, I and think it's some combination of XOR and XNOR on, you do like operation on A and B, and then the other operation on that and C, and so the output of that always preserves the total amount of operation. So if you think back to the types of logic gates we have, there's and and or are things up into a one and three, like there's one possibility corresponds to one specific input and the other possibility to three inputs.

0:39:46 Thomas: Okay. To, to a, huh?

0:40:52 Thomas: Right.

0:40:52 Bryan: So, it's harder to go back that way as one of those could be three and one extra bit isn't going to tell you enough, enough. So the XOR and XNOR gates, which is just the opposite of it, are more useful. So the XOR gate is exclusive OR. So OR means or, but not and, which means if the two things are different, you get a one result. If they're the same, you'd get a zero result. So there are two ways to be different. It could be one, zero or zero, one, and two ways to be same, 1, 1 and 0, 0. So the XOR gate splits it evenly 2, 2 and

0:40:52 Thomas: Gotcha. Yeah.

0:41:14 Thomas: Right. Right. Oh,

0:41:32 Bryan: The XNOR gate, it does the same thing. It just flips which is which. And so by doing those two, like you apply it once to A and B and then another time to that result and C, you're not losing any information there. So all all in all, A prime, B prime, and C prime will always have the same information. They've just arranged it a bit differently.

0:41:32 Thomas: I see. Yeah.

0:41:58 Thomas: Interesting. Okay. I, I, sure that's a bit hard to follow over audio, so we'll leave it there. But that, terms of that explanation. Um, but yeah. Okay. So I hypothetically anyway, I think I kind of see the concept going on.

0:42:01 Bryan: Yeah.

0:42:10 Bryan: And this is something that, you hasn't been too much research into it because it's nowhere near needed. The efficiency of computers is still millions of times greater than this limit, but it's something that we would need need to take seriously if we ever get to the point where we might be able to get below that. And it looks like from the little research I've done that some quantum computers are already doing this type of thing and the way they work. And I think the reason this is done for quantum computers is that you want to allow the wave functions to propagate and do logic with them without quote-unquote collapsing the wave function.

So it collapses when you do an observation on it, which shoves it into it's definitely one state or the other. And if you want to not collapse it, you can't lose information anywhere, because losing information forces the collapse. So in order to do work with any quantum computer, at least as I understand it, I'm not but an expert on this, requires, to some extent, reversible computing in all of the middle parts so that you don't prematurely collapse any of the wave functions.

0:42:19 Thomas: Yeah. Huh? Yeah, sure.

0:43:23 Bryan: Right, so you think you understand all of that?

0:43:26 Thomas: I a hundred percent definitely understand.

0:43:28 Bryan: Good, because it's all going to be on the test.

0:43:28 Thomas: No, Yeah, I was going to say, let's re roadmap how we got there for, everybody. If you don't, mind if so that we can reset where we are. So we're talking about the limits on computing based on entropy And other things. And you were were kind of saying, you know, if we did computing a little bit of a different way than just the basic ones and zeros and those gates, uh, could we prevent what, energy loss or entropy, or is that the same thing, right?

0:43:37 Bryan: Yeah, yeah.

0:43:52 Bryan: Remember, free energy is the same as negative entropy, in prevent entropy from increasing by doing reversible computing, which never gets rid of information as we're going along. Maybe at the final state at the end of it. I mentioned offhandedly that you can do addition with five bitwise operations. You need five logic gates per bit in order to add two binary numbers. With reversible computing, maybe we could set this up to to instead of going through five gates which destroy the information, maybe we can get it down to effectively just one gate that we keep the information as we go along and only destroy at the end, turning it into an addition gate.

0:43:56 Thomas: Right, right, mm. Mm-hmm, wow.

0:44:39 Thomas: But How would one gate be able to hold all that information? Oh,

0:44:44 Bryan: That's the thing. The actual gate does a lot in the middle. It's just that none of what it does actually destroys information. And there There are a lot of ways you can do this with, like, propagations of waves and quantum mechanics. You can still have the same wave going. Like, quantum mechanics is weird. You can have the same particle travel two routes at once.

0:44:52 Thomas: Yeah, yeah. Okay. Yeah. Right. Boy, that'll, that'll save you some gas money, yeah.

0:45:07 Bryan: Yeah. It all together, that's why thinking does take energy, but there might be a bit of a loophole we can exploit if it ever becomes a real concern.

0:45:16 Thomas: Like if we ever run into that limit. We may be able to kind of change the way. Wow, that's super cool, I love

0:45:18 Bryan: Yeah. Yeah. Rethink how computers work almost from the ground up as so much is built on these gates. But yeah, if we do need to make computers better than that and are able to get close enough, which is doubtful, but you never know, there is a potential loophole.

0:45:24 Thomas: Yeah. It Yeah. I mean, I know the great driving force in innovation is usually porn, so I think if porn demands get to a certain level of quality on online, they'll find, you know, they'll find, the nerds will find a way to get that HD, even more HD. Okay. So I know we just dipped our toes in, but that's really fascinating. I particularly love the information theory part of it is I've not looked too much into. But every time I do, I'm like, oh, this seems super fascinating. You know, information being like that necessarily has to, you require some amount of energy to produce and all that. That's just super fascinating to me. Maybe we'll do more on that another time. Well, Dr.

0:45:51 Bryan: Yeah, yeah. Yeah. Yeah, sounds good. But,

0:46:22 Thomas: Brian Gillis, as usual, I know you've been super bogged down. So I don't know if you have anything to plug or do you?

0:46:28 Bryan: Well, I've got a bit going on on my YouTube channel, actually playing through a game coincidentally called The Entropy Center, which is, um, it's got the structure of Portal, except instead of working with Portal guns, you've got an Entropy gun, which rewinds time. And this came out before Tears of the Kingdom, so that actually beat Zelda to the punch with having a time rewindy device. It's interesting, and I actually get into a discussion of this very subject when they They try to explain how they're generating energy with entropy, which makes no sense in -game, and I explain why. So look up Brian Gillis on YouTube, B-R-Y-A-N-G-I-L-L-I-S, and you can find me there.

0:46:48 Thomas: Wow.

0:47:07 Thomas: Right. As always, it was a pleasure. Hope to see you again very soon.

0:47:11 Bryan: Right, thank you very much, Thomas. Talk to you later.