#34: Move 37: The Moment AI Stopped Playing by Human Rules

2016 Something happened changed how we think about AI forever. A machine made and moved in the ancient game of Go that no human would ever make. Commentators called it "beautiful", experts called it "impossible". One of the words greatest player sat in stunned silence, unable to comprehend what was happening. This story of AlphaGo's move 37, and why it matters for beyond a boot game? Let's explore what move 37 reveals about how modern AI actually thinks, why it terrifies some of the words best minds, and what it tells us about the future of autonomous systems in market, hiring, securities, and basically decision making at scale. From the ancient rules of Go to the cutting edge of machine learning. From Lee Siddhal's legendary response to the implications for a word increasingly run by algorithms. This is the story of the moment we realized AI wasn't just following the rules anymore. It was rewriting them. So in March 2016, in South Korea, a tournament all filled with Go experts, generalists, and spectators from around the world. The man named Lee Siddhal sits across from a computer. He's 33 years old, already 18 times word champion. He's considered one of the greatest, if not the greatest, Go player's alive. The computer is called AlphaGo. Three hours into the match, AlphaGo makes a move. It was move 37. A commentator and expert with decades of experience looks at the board and belief, and he says, quote, this is not a human move, of course, wasn't Lee Siddhal's takes off his glasses. He stands up, he walks away from the board, and for 15 minutes, he sits in the corner of the room just thinking. And folks, we are talking about an 18 time word champion, considered the greatest or one of the greatest Go player alive. When he comes back, he's pale, shaken, and he will later say, I felt like I was playing against something unnatural. This moment, this move 37, changed something fundamental about how we understand artificial intelligence, creativity, and what it means when we lose the ability to predict what a machine will do next. So it is a story about a game, but it's really a story about losing control. Welcome back to the identity navigator. This is your host Rohit Akhti O3. And today we are going to explore why move 37 matters, not just for Go, but for the future you and I are building. Let's start at the beginning. To understand why move 37 was shocking, you need to understand Go. Go is an ancient game. It originates in China over 25 years, 25 hundred years ago. Simpler to learn the chess, you place black and white stones on a grid, and you try to control territory. That's pretty much it. And here is what makes Go so profoundly difficult. The number of possible board positions is larger than the number of atoms in the universe. So saying that again, the number of possible Go games is literally larger than the observable universe. Compare that to chess. This has roughly 10 to the power 120 possible positions. That's a number with 120 zeros after it. It is incomprehensibly large. Go has 10 to the power 170 possible positions. So compare that to physicist estimating that there are 10 to the power 80 atoms in the universe. This is why I compute my struggle with Go for decades. You can't solve Go through brute force calculation the way you can solve chess. You can't evaluate every possible move because there are too many possibilities. Go requires something that feels almost human, intuition and pattern recognition. The ability to see a position and just know if it is good without calculating every variation. Now folks for 1500 years, only human could play Go at the highest level. And the best players, the ones who had studied the game for their entire lives, they developed an almost supernatural ability to read the board very much like Lee had. They could sense the flow of the game, they could see moves, the other couldn't see. Lee said almost definitely one of those players. He was considered a living embodiment of Go mastery, a man who had spent his entire life learning the nuances of this ancient game. And then in 2016, a machine was created that could challenge him. That machine was alpha go. And let me tell you about the team that built alpha go because this is where the story gets interesting. And this is the most fun that I've ever had researching a podcast episode. But back to the company of the people that built alpha go deep mind was a small AI research company founded in London at 11 by 2016, they had about 100 and 50 people working there and they were obsessed with one thing solving Go. The lead researcher was a brilliant man named David Silver, very focused. He had spent years thinking about how to teach a machine to play Go intuitively. The key insight was this, you can't teach a machine to play Go by programming in rules, you have to teach it to learn and for that they did two things. First, they trained alpha go on historical go games, thousands of them, hundreds of thousands of moves made by the words greatest players. They created a neural network, a simplified model of how the human brain works and fed it all these moves. The neural network started to learn patterns, what made it good opening, what makes a solid position, what moves to champions tend to play. But here is where it gets clever, they didn't stop there. They had alpha go play against itself, thousands of games, millions of moves and each game alpha go played slightly differently exploring variations and strategies and each game taught it something new. This as we all know is called reinforcement learning, the machine learns not by being told what is right but by playing and getting feedback when or lose. Over time, alpha go developed intuition. It learned pattern that no human had explicitly taught it, it learned to recognize good positions, it learned to evaluate the board not through calculation but through something that resembled human intuition. By 2016, they believed alpha go was ready to challenge a word champion. And Lee Sedol, being Lee Sedol accepted the challenge. Now in all honesty, nobody expected alpha go to win. Most experts thought Sedol would dominate some but alpha go with win 1 or 2 games but not a match. The match was best of 5 games and the first one to win 3 would be the overall champion. Alpha go 1 game 1, Sedol was stunned but not devastated. He thought he just had a bad game, alpha go 1 game 2 and this is the time during that match that Sedol was starting to realize this was no ordinary machine. And in game 2 on move 27, alpha go did something that changed everything. So let me paint you a picture of what the board looked like at move 37. and R into the game both AlphaGo and Leesidol have made intelligent moves the game is balanced and it's not clear who's winning then alpha go plays move 37 is it places a stone on the three by three point of the board the three three point if you asked a million go player to list the next 1000 moves they might make almost none of them would include the three three point at this stage of the game unconventional weird and it's the kind of moves that breaks with 2500 years of go traditions and theory the commentators watching the game live one video and these were word class go professionals they literally don't understand what they were seeing one commentator said this is not a human move another says I've never seen alpha go play like this before there is confusion uncertainty the professionals are watching an AI to something that violates their understanding of what a good moves looks like and at that point now people will tell you differently but here's the thing at that point they didn't know if it is good or a bad move think about that the best go players in the world cannot evaluate if a move is correct without calculating it but they can't calculate it because the third space is too large so they are stuck they are watching a machine make a decision that might be genius or might be stupid or catastrophically bad and they have no way to know which one the commentators I think later called it a desperate move at that time I think what they were calling is that alpha go is making a mistake maybe the match machine is confused maybe it's glitching but then over the next 10 20 30 moves the game unfolds and the three three point starts to reveal itself as brilliant it was placed perfectly to control territory later in the game it was a move that set up dominance many moves in the future this is where he said all the greatest play in the world sits in stand silence because he realizes he doesn't understand how alpha go things he doesn't have the pattern recognition to see why the three three point was good it's not that it was obvious and he missed it it's that the move exists in a part of strategic space that human intuition doesn't naturally explore a reporter later asks him how he felt when alpha go played move 37 he says and I quote it it was an extraordinary move human just wouldn't play this way but it was also beautiful now that's the shocking part move 37 wasn't just non-human it was elegant, effective most artistic it was creativity but not human creativity here is what's important about move 37 and why I'm telling you this story on a podcast about identity and access management move 37 was the moment the word realized a machine can make a decision that are correct and optimal but who's reasoning we cannot fully understand or predict that's not trivial that's everything for most of AI's history the systems we built were transparent if a computer made a decision you could ask it why and trace back the logic if a calculator said 2 plus 2 equals 4 you could understand the reasoning if an algorithm filtered emails into spam you could theoretically understand why but with move 37 you cannot do that the neural network that deep learning systems that alpha go uses it doesn't explain itself because it it cannot the internal logic is distributed across billions of parameters billions of calculations and billions of connections that don't map onto human language the machine ultimately made a decision the decision was good but the reasoning opaque alien and incomprehensible and this is where it gets scary this isn't unique to go the same type of neural network that plays go can also evaluate loan applications it can screen resumes it can set prices it can decide who gets hired who gets approved who gets charged what interest rates and in all those cases just like with move 37 the machine is making decisions that might be optimal but are fundamentally not transparent to human oversight think about Sarah one of the engineers that we discussed in one of the previous episodes her resume was downright by an algorithm she didn't know why the hunting manager didn't know why the algorithm didn't know why it just knew this resume resembles resumes that had high iteration rates in the past so down-ranked that's move 37 in hiring now multiply that across millions of decisions thousands of companies billions of people hiring algorithm lending algorithm pricing algorithm recommendation algorithm risk assessment algorithms each one making non intuitive decisions that might be optimal but are fundamentally not human fundamentally not predictable and fundamentally not explainable and each one compounds the next this is what happened in lease it all smash after move 37 he couldn't predict will for goes next move because he couldn't understand the logic he was playing a game against an opponent who strategy was alien to him now imagine that's the state of global markets imagine that's the state of hiring imagine that's the state of security credit and risk assessment you are playing a game against an opponent whose logic he do not understand you don't know what move it's going to make next and you don't know if it is going to be genius or catastrophic you are hoping it's smart but you have no way to verify it this is the implicit promise of move 37 when you build a system that can learn and optimize faster than you can understand what happens next okay so we are two games in alpha go has one twice lease it all is starting to realize this is not a match he can win by conventional play he decided to do something radical in game 4 he decided to he decides to play for the divine move so in go there is this concept called the divine move it the move that seems impossible until it is played and it is so perfect that it is almost spiritual the divine moves it's what separates a good player from a truly great player it is a move that breaks all conventions and reveals a new understanding of the game he said all decides in his head at that time if I'm going to lose to a machine I'm going to try to beat it with creativity he plays a move in the early game that's unconventional expected it's moved 78 of the game and it exposes a weakness in alpha go no one knew existed alpha go for the first time in the match makes a move that's clearly bad the machine panics it plays desperately and for the first time lease it all wins the committee just go wild lease it all wins the one game he needs to win he proves that human intuition can still find something a machine cannot that's the divine move human creativity at its peak can find a weakness that deep learning could not have dissipated lease it all becomes visibly emotional he cries after the game not because he won he knows he's still losing the match but because he proved something important the machine has limits but here is what's crucial Lease it all's divine move only. worked because it was unconventional. It broke the normal rules, found a blind spot in alphago's training. And do you know what happened after that match ended? Deep mind went back to the lab, they trained a new version of alphago on millions of new games, they fed it more data, they made the neural network deeper and they made it more robust. The next version of alphago would not have the same blind spot. The divine move, that beautiful moment of human creativity, it would work once, maybe twice, but then the machines would learn, it would adapt, it would become invulnerable to that specific strategy. And that's the real lesson of move 37 and divine move. It's not that humans can never beat machines, it's that humans can only beat machines by being unconventional, unpredictable, creative. By stepping outside the normal logic, that the machine was trained on. But machine learn from being beaten. Each time you find an unconventional strategy that works, you are teaching the system how to defeat that strategy next time. That's the feedback loop of AI development. Now I want to connect move 37 to something bigger, something that most people don't talk about. Up to the match, Lee said all retired from professional go. He said he didn't want to play in a world where machines could beat him. He said alphago is an entity that cannot be defeated. Think about what that means. One of the greatest co-player in human history, looked at a machine, played against it and decided there is no point in competing anymore. The machine is better and it's only going to get better. That is surrender. But here is the thing. Lee said all only had to make that decision because of move 37. Move 37 wasn't the best move because it followed logical rules that Lee said all could understand and eventually counter. Move 37 was the best move because it existed in a part of strategy space that human don't naturally explore. The machine found an alien strategy, non-human way of playing the game and once a human knows that strategy is possible, they can't unknow it. They cannot compete with something that things in fundamentally different ways. Now let me ask you something. What happens when this same dynamics play out in markets? Right now, high frequency trading algorithms make the seasons in microseconds. The trade in ways that are so fast, so alien that human can't even see them happening. They make moves equivalent to move 37 in trading. Moves that are optimal but non-intuitive. Human watch these moves and realize we cannot compete with this. We can't predict it. We can't understand it. We have surrendered the market to the machines. What happens when this same dynamics plays out in hiring? Algorithms activates resume in a way that human don't naturally think. They notice patterns that human miss. They make hiring decisions that scheme weird. They down rank certain demographic and they favor certain skills over others. They make moves in the talent market that humans find incomprehensible. And humans realize we cannot compete with this. We cannot predict what the algorithm will do. We have surrendered hiring to the machines. What happens when this same dynamic plays out in credit security, pricing, recommendations? Every domain where we deploy deep learning, we are creating an alpha go moment. We are creating systems that make move 37s. Moves that are optimal but alien. Decisions that are correct but incomprehensible. And once those decisions, those systems are in place, humans are in the position he was in after game 2. Watching a machine making decisions we don't understand, knowing that understanding might come to late. But here is the terrifying part, only had to play 5 games. In the real world, the feedback loop is continuous. Every move is recorded, every decisions is analyzed and the machine learns from every move, every decision. You play a move, find a way to beat the system, the system learns that move. Now it cannot be beaten that way ever again. You play another move, the system learns, adapts, becomes more robust. And eventually, we run out of moves. This is what happened with alpha go. But alpha go is just a machine, go is just a game. In the real world, when you run out of moves, when you have surrendered all strategic options to a machine you don't understand, what happens then? Move 37 is the moment we realize something profound. We can create systems that are smarter than we are but who is thinking we cannot access. We can create systems that make better decisions that humans but in ways that are alien to human logic. And once we do that, once we deploy those systems across the economy, across markets, across hiring, across security, we enter a new phase. We enter the phase where the machines don't just beat you. It beats you in a way you cannot understand. It beats you in ways you cannot learn from. It beats you because it thinks differently. Move 37 was beautiful. Lease at all said so himself. It was creative. It was optimal. It was everything we want from an intelligent system. But it was also the moment Lease at all realized I have lost the ability to predict what this system will do next. That's just not about go. That's about the future. Because we are building versions of alpha go across the entire economy, hiring algorithms, trading algorithms, pricing algorithms, credit algorithms, security algorithms and an agent AI, autonomous systems that makes the season not just about positions on a board, but about these things like hiring, firing, investing and building. Each one will make its own move 37. Its own unconventional, incomprehensible, optimal decisions. Move 37 it not just a beautiful moment in Go history. Move 37 is a prophecy. It's a warning about what happens when we build systems we cannot understand. When we surrender control to optimization, when we let feedback loops run so fast that human oversight becomes impossible. Move 37 is the moment we should have realized we are not building a tool. We are building an opponent. One that will eventually operate according to logic we cannot access. And the question we should be asking is what happens when we are not playing a game when losing is just losing a match? What happens when we are playing for the future? Lee, sad at a board across from a machine and realized I can no longer predict what happen will do. That's the position we are all going into not in Go but in work in markets in hiring and security in the system that govern our lives. The question isn't whether move 37 moments will happen. That's already happening. And they are happening in financial markets right now. They are happening in hiring algorithms. They are happening everywhere we deploy deep learning without fully understanding how it works. The real question is what do we do about this? Do we keep building faster, deeper, more opaque systems and hope they optimize in our favor? Do we keep surrendering domains to machines and assume they will take care of us? Or do we start asking hard questions about transparency, about interpretability, about whether moving fast is the same as moving smart. Move 37 was a moment of beauty. It was a moment of loss, the loss of human understanding, the loss of predictability, the loss of control. I want you to notice the next move 37 moment in your word, in your company, in your industry. When you see a decision being made by an algorithm, by system, by something you build and you realize, I don't understand why that happened? That is the movie to stop or to pause and to question to ask, do we understand this system but most importantly do we trust it? Can we predict what it will do next? Because move 37 isn't coming, it is already here and the choice is whether we master it or whether we surrender to it. This is Rohit Agniho 3. You have been listening to the identity navigator. Thank you for the time and thank you for spending time with this story with me and if this episode resonated with you, please share it because these conversations about what we are building and whether we understand it, they need to happen and they need to happen. Until next time, stay curious about the systems you are inside. I hope to see all of you soon. Please reach out via LinkedIn, emails or by any other channel that is accessible. Thank you. I'll see you soon.