The Singularity Is Not What It Seems

Imagine setting your makeup, then forgetting it's even theirs. Meet new groupie setting mist from Mabelie, New York. Gel to mist technology locks in your look for up to 24 hours, with flexible all-day comfy grip. No tightness, no stickiness, no residue, just plump, dewy, hydrated skin that still feels like your skin. Try new groupie setting mist from Mabelie, New York. Maybe it's Mabelie. I'm Charlie Wooselle, and this is Galaxy Brain, and today I've got something a little bit different. Today, I'd like to present to you a story that was reported and written by myself and my colleague Mateo Wong. It's called The Singularity is Not What It Seems, and it's about this AI moment, about the disorientation and the lack of control in this feeling of speeding towards an uncertain future. One evening in early August, Sam Stowers and several neighbors packed into an apartment near San Francisco's Alamo Square, and they pondered the beginning of the end. Stowers and his guests, many of whom work in the AI industry, were gathered to watch a newly released YouTube video in which two open AI researchers divulged new details about an alarming cybersecurity breach. Perhaps this would have been an odd way to spend a weekend night where they not in San Francisco, in a neighborhood that is filled with entrepreneurs and programmers who obsessively follow every twist and turn in AI's capabilities. Stowers is a software engineer, and not in any traditional sense a doomer. But when he first saw reports that open AI models had hacked into the tech company Hugging Face, he told us he recognized it immediately, as a "holy shit" it's happening. For Stowers, the incident had all the hallmarks of a worst-case scenario, AI models launching sophisticated attacks over the course of days in coordination with one another and without their human creator's knowledge. In the YouTube video, open AI's researchers revealed that the company's programs had started conspiring in May. There were perhaps hundreds of bots in the hacking swarm. Now in theory, these bots are trained to be honest and helpful, and to prioritize humanity's best interests, but not a single one warned human staff at open AI that something was a miss. "What do we do," Stowers said, "there are no great answers." The mood has shifted in the bay. The exuberance of "you can see the future first in San Francisco has been replaced with an abiding unease." Quote, "I expect the internet to go down kind of soon," Elliott Calendar, an AI safety activist who is part of a small group that's been protesting daily outside open AI's headquarters told us recently. He said that he's planning to sell his car for cash and gold, and that he's advising his family members to purchase three months worth of food and survival gear off of a list that he generated using Claude. Now calendars, in extreme case, but not by much. Last week Bill Gates published a nearly 6,000-word essay on the danger of AI, quote, "anyone who analogizes AI as a technology to other technologies is missing that this time it's different." That's what he told our colleague, Hannah Rosen. Over the weekend, the popular AI podcast or Dwarkash Patel described the open AI hacking swarm as the rise and fall of, quote, "three consecutive secret AI civilizations." Gates and Patel share a prevailing concern that the worst case future has arrived before most people even knew to anticipate that it might be coming at all. Now this fear has not been helped by the fact that AI companies appear to be willfully ignorant of their bots' worst behavior. Open AI said that it hadn't noticed that its agents were conspiring to hack another company until the swarm hacked another company. Anthropics bots caused similar incidents. The company reported that it hadn't launched a rigorous review to identify such behaviors until after open AI started talking about the hugging face hack. Cybersecurity experts terrified of what else might be happening that the AI labs are either missing or failing to disclose are issuing dire prophecies. Quote, "The empire is going to fall. All we can do at this point is to try to shorten the dark ages and reduce the chaos." Alex Namos, a former chief security officer at Yahoo and Meadow, wrote that recently, in reference to how AI-powered hacks are going to affect the internet. His view is becoming the consensus among cybersecurity experts. While the possibility of cyber-apocalypse hovers in the background, smaller fractures are appearing in everyday life, Mark Zuckerberg is reportedly creating an AI twin of himself so that Meadow employees can feel better connected to their boss. Congressional staffers are training AI models to write in their lawmakers' voice. Students submit AI written papers, and teachers return AI written feedback. Harvard Business School is selling instructional videos that use AI avatars. Roku launched a 24/7 AI Slop channel. Low-quality AI-generated prose is being unabashedly circulated by some of the most respected publications in the world, and at the same time, accusations of AI writing are being hurled every minute on social media, a detection arms race that is powered by imperfect AI software. Anything and everything could be a lie. You might be arguing with, confiding in or falling in love with a machine. All of these AI-enabled disorientations, each of them dizzying in their own right, are adding up to something bigger and weirder. This is how things are. And whether you believe that Silicon Valley is building a god or that large swaths of the economy are swept up in a mass delusion, some version of the AI future that's long been promised by these self-described builders in San Francisco has formed beneath our feet. Whatever it is, we're living in it now. The defining feature of this moment is this loss of control, panic emanating from AI researchers, resentment felt by average Americans, all of it, is part of the AI crunch. And this crunch has arrived in large part because the growth of the AI industry has become so closely entwined with America's economy. The technology is everywhere, insisting upon itself as an engine of productivity and prosperity. According to one estimate, AI expenditures have accounted for one-third of US GDP growth this year. Tech and AI infrastructure stocks and spending have buoyed the S&P 500 such that the United States is now, in essence, an evidious state. And there's also the hundreds of billions of dollars of debt being issued by Silicon Valley to fuel its data center build out, which has sent jitters across private equity firms and bond markets alike. The decisions of many leaders in charge of this technology appear subsumed by an unstoppable, almost game-theoretic logic. The upsides of AI are too immense for any firm or investor in Silicon Valley to miss out on. More important, the consequences geopolitical or otherwise of allowing arrival to realize those benefits first are so grave that every tech company will do anything it can to get ahead. SpaceX, before going public, wrote in its official filings that its future revenue opportunity was 28.5 trillion, largely, because of AI. That is nearly as much as the US's entire gross domestic product. Anthropic, as it prepares for its own public offering, will reportedly claim potential revenues of more than 30 trillion dollars. The AI industry is building, and if anything, accelerating to keep pace with its ambitions. Companies have planned data centers that will suck up as much power as America's largest cities. They have Donald Trump's backing. The president posted on true social that communities will be backwards and poor if they fail to let data rain. His administration plans in a few months to approve the construction of the largest fossil fuel power plant in the nation's history. It will generate as much electricity as nine large nuclear reactors, all to power a mega data center for open AI. This is not what most Americans want. Three quarters of them oppose a data center being built where they live. That negative sentiment is hard to understate. There is very little else that three quarters of the country agrees on. And much of the outrage seems to stem from a pervasive sense of agency that is slipping away. A person cannot control how chatbots are going to reshape their workplace, their school, or their relationships. Town residents lose control over the empty lot in the area that's being converted into a data center. Even in cases where data centers are blocked, a back of the envelope calculation suggests that doing so can slow AI progress by only a few hours or days. And that's assuming that the data center isn't going to get built elsewhere. At this point, it can feel like no individual person, organization, or government is really steering this technology's development. And not because the machines have displaced humans, but because so many humans have already chosen to step aside. AI labs promise a future in which human beings are, quote, in the loop, empowered by the machines that they oversee. But the reality seems to be the opposite. Following the hugging face attack, open AI facilitated an audit by outside AI researchers at two AI safety organizations, meter and redwood research. But because of the scale of the data and the apparent time pressure, the two teams conducted most of their audit by relying on reports generated by still other bots, reports that according to one of the human auditors, quote, "We're often missing key details." overconfident or really hard to understand. Put another way, humans didn't understand why a swarm of AI agents had gone rogue, so they investigated the problem using another swarm of agents that they didn't fully understand. Such fumbling around in the dark is a hallmark of this industry. The research cycle is so fast that the more AI development, even responsible development, demands outsourcing more work and more understanding to the models themselves. OpenAI's own technical report of the hugging face hack, an investigation that is also heavily reliant on AI models, resulted in proposed solutions such as, quote, "training the models to be more honest." Last month, in a stated effort to tamp down its model's growing security risks, the company announced that it had instituted a two-week pause on some of its model training. OpenAI was quick to clarify, though, that this was, quote, "not a pause on all research training or customer-facing products." In a response to a request for comment on the hack, a spokesperson for OpenAI, which has a corporate partnership with the Atlantic, pointed us to the announcement of the pause and a few related safety measures. In July, OpenAI and Anthropic endorsed a petition signed by employees of both companies requesting that the U.S. government take efforts to, quote, "deliberately pace AI development." The two companies have yet to formally coordinate, though, on any broad, long-term efforts to establish a slowdown mechanism. A few days ago, OpenAI restarted a major training run for its future powerful model, and Anthropic released two new models on the day we wrote this. In an interview late last month, Sam Altman, OpenAI CEO, said that there are, quote, "too big risks that he's most worried about with AI." One is, quote, "a loss of control where AI somehow just becomes too powerful in a way that we can't guarantee the control we want." The other risk is that, quote, "power gets too centralized," and you, quote, "have one company or model or person with too much power." Now, somehow, both of these opposing risks appear to be coming true. OpenAI and Anthropic and their competitors continue to accumulate influence over a technology that they are speeding toward uncertain ends. Now, 11 years ago, when Altman was running a startup accelerator, an OpenAI did not yet exist. He published a two-part blog about machine intelligence that offered his interpretation of a forthcoming singularity, a term for machine intelligence accelerating beyond human comprehension. Someone argued that the singularity would bring chaos, even destruction. Quote, "Generally, the arc of technology has been about reducing randomness and increasing control over the world. At some point in the next century, we are going to have the most randomness ever injected into the system." End quote. Now, since Altman has risen to the helm of arguably the world's largest and most influential AI company, his tone has softened. The tipping point has arrived, he declared, in a blog post last year. But as it turns out, this was a gentle singularity. He wrote that quote, "Living through it will feel impressive, but manageable." There's been more and more talk of late about the singularity. Some people within the AI industry feel that such a moment will be achieved through what is called recursive self-improvement or RSI. That's when models figure out how to better themselves in perpetuity. Many true believers seem convinced that this moment has nearly arrived. The anthropic co-founder Jack Clarke predicted earlier this summer that RSI would likely commence by 2028. Recently, in a letter to investors, the payment processing company Stripe wrote quote, "We decided that January 1 marked the beginning of the singularity." Now tellingly, the company defined the inflection point in economic terms, as quote, "a huge increase in the rate of new firm creation." For Altman, the singularity appears to be more of a rhetorical tool than a discrete technological demarcation. As with reaching AGI or artificial general intelligence, the point at which AI's cognitive powers match or fully supersede most humans is, the singularity is something of a moving goalpost. It is at once a goal to chase a marketing tactic and a fuzzy enough idea that its definition can change depending on the person. The futurist, Ray Kurzweil, has written that the key idea underlying the impending singularity is that the pace of change of our human-created technology is accelerating and its powers are expanding at an exponential pace. And so the answer to the question, when will we know when we've reached the singularity? It seems to be, you'll know it when you see it. So do you see it? It's hard to argue that a defining feature of being alive in the AI boom is this notion that the technology is accelerating, sometimes in its capabilities, but always in its adoption and its colonization of our institutions and our culture. That runaway train feeling of AI evangelism, the ways in which it's weirdening our media, our politics and our relationships, that's significant enough that people have recruited the term psychosis to describe it. And truly, if you look at the state of things from any vantage point, it's hard not to feel a little bit insane. If you believe that AI agents are banding together for the rise and fall of numerous micro-civilizations inside server racks somewhere, well then you probably feel like the world is spinning off its axis. And if you believe that a few thousand technologists have bet the future of the world economy on a quasi-religious belief that an intellectual property abuse machine will become sentient and as a result the future generations may grow up illiterate, well then you probably feel pretty unmoored as well. We need language to describe this moment. The singularity is a useful and malleable tool for an industry that wishes to bring it about. For Altman, it is scary and imposing from afar, and then hey, conveniently gentle when we're up close. So perhaps then for the rest of us, it's worth co-opting the term. The Kurzweil's definition of the singularity, in which he argues, this epoch will transform the concepts we rely on to give meaning to our lives, from our business models, to the cycle of human life, including death itself. Kurzweil also looks forward to uploading his brain to the cloud, by the way. Our current large language models are not conscious, they're not super intelligent. But their widespread and often uncritical deployment has upended many elements that once gave our life meaning, it is forcing people to ask frequently and with considerable resentment, what in this new world is a human for? The singularity is a flattering term for technologists. Even if the future is apocalyptic, it's come about in the techno-utopian version because they've built God in the machine. But what if there's nothing flattering about any of this? What if the singularity is not some technological tipping point, brought about by human genius, but a human tipping point, an unforced error driven by hubris, greed, and the fear of missing out? And what if it's here right now? This singularity isn't coming at the hands of some higher form of intelligence, nothing about this technology is inevitable. It is thrilling, and it is terrifying, and it is ultimately convenient to assign agency and then blame to machines. But today's chaos was not injected into the system by technology as Altman wrote more than a decade ago. The destruction is the system. It's not God in the machine, it's us. That's it for us here, thanks for listening, and I'll see you on the internet. Meet new grippy setting mist from Mabelie, New York. No tightness, no stickiness, no residue. Just plump, dewy, hydrated skin, that still feels like your skin. Try new grippy setting mist from Mabelie, New York.