Jacob's Ladder: The Thinkable Space of Events at the Birth of AGI

Humanity has faced forces it does not understand before. Fire, the sea, plague, the atom: every age brings its own challenge. But those forces were blind. Now, perhaps for the first time in history, we are building something that looks back.

In January 2025 Sam Altman wrote that OpenAI is “now confident we know how to build AGI.” Shortly before that, Geoffrey Hinton, who had just received the Nobel Prize for his work on neural networks, put the chance of AI wiping out humanity at 10–20%. The two statements do not contradict each other; one follows from the other. What worries Hinton is precisely that Altman may be right.

The idea of an artificial mind surpassing the human one is fairly old. Until recently it lived in myth and literature: the Golem, Frankenstein, Čapek's robots. In 1965 the mathematician Irving John Good, who had worked with Turing on breaking Enigma, restated it in the language of logic: “The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” The proviso at the end, “provided that the machine is docile enough,” contains the whole problem. In 1993 the writer and mathematician Vernor Vinge gave it a name, the “singularity”: the point beyond which forecasts fail, because the one who predicts is weaker than what is being predicted.

In this essay we try to walk the full spectrum of current views on what awaits us. Not to pick a “right” side, but to understand how the question itself is built. Perhaps the main thing this debate teaches is not an answer but a way of thinking about what exceeds our understanding.

Mind Children

“We believe Artificial Intelligence is our alchemy, our Philosopher's Stone – we are literally making sand think,” Marc Andreessen wrote in The Techno-Optimist Manifesto (October 2023). The words “We believe” recur in the manifesto 113 times: “Techno-Optimists believe that societies, like sharks, grow or die,” “We believe everything good is downstream of growth,” “We believe the techno-capital machine of markets and innovation never ends, but instead spirals continuously upward.” The political scientist Henry Farrell called it “a Nicene creed for the cult of progress.”

This is indeed a matter of faith: faith that technological progress is good in itself, that the obstacles in its way can be overcome, that the future is better than the present. For centuries technology really did improve the lives of most people. But the future does not follow from the past. That is the classic problem of induction, formulated by David Hume in the eighteenth century.

Sam Altman prefers the language of engineering; as journalists paraphrase him, AGI is just an engineering problem. In “The Gentle Singularity” he describes a smooth transition: 2025, agents that can do real cognitive work; 2026, systems that can arrive at novel insights; 2027, robots; by 2030, one person able to do far more than in 2020. Ray Kurzweil, the futurist known for his forecasts, adds: “We are going to expand intelligence a millionfold by 2045.”

What is striking in these forecasts is not the optimism but the certainty. To call AGI an engineering problem is to set aside the question of what intelligence is, what it means to “understand,” and how machine thinking differs from human thinking. Hans Moravec, a pioneer of robotics, took this logic to its limit in Mind Children (1988): machines will become our “mind children,” inherit civilization and displace biological intelligence as naturally as children replace their parents. “Our culture will then be able to evolve independently of human biology and its limitations.” For Moravec this is not a tragedy but the natural course of things.

But this view assumes that intelligence is a single scale on which an ant, a human and a machine can all be placed. The philosopher John Searle, in his Chinese Room thought experiment, showed that manipulating symbols is not the same as understanding: a person locked in a room can follow instructions to produce flawless answers in Chinese without understanding a word. Perhaps the question is not when machines will become “smarter,” but what we mean by intelligence in the first place.

The End of Scaling

Ilya Sutskever, the man who was building AGI at OpenAI, took part in the failed attempt to fire Altman in November 2023. In the end he left himself and founded Safe Superintelligence Inc., a company valued at $32 billion without a single product. The name itself reads like a manifesto: not “Fast Superintelligence” or “Profitable Superintelligence,” but “Safe.” In November 2025 he declared the end of the “age of scaling”: the data is running out, and scaling yields less and less. The “engineering problem” turned out to be harder than it looked.

Yann LeCun, one of the “godfathers” of deep learning, left Meta at the end of 2025 and founded AMI Labs. He puts his position bluntly: “The path to superintelligence - just train up the LLMs, train on more synthetic data, hire thousands of people to school your system in post-training, invent new tweaks on RL - I think is complete bullshit. It's just never going to work.” Language models predict words, but they do not understand the world. What they lack is what LeCun calls “world models”: the ability to model reality, not just text about it.

Philosophers drew this distinction long ago. When ChatGPT answers a question about physics, it does not model the physical world; it models texts about it. Alfred Korzybski put it this way in 1931: “A map is not the territory.” Today's LLMs are inhabitants of the map: the territory itself is closed to them.

François Chollet turned this distinction into an experiment. His ARC-AGI-2 test works like this: the test-taker is shown a few examples of how a colored grid changes: for instance, scattered cells at the input, the same cells grouped by color at the output. The task is to grasp the hidden rule and apply it to a new grid. A human sees the pattern in seconds. In spring 2025, models scored about 1% on such tests: they could not extract an abstract rule from a handful of examples. By November 2026 the best systems solve more than 90% of these tasks. Chollet's question has not gone away; it has moved. The new version of the test, ARC-AGI-3, no longer checks whether a system can solve grid puzzles but whether it can learn its way around unfamiliar interactive environments.

Two Breakthroughs Short of AGI

A survey of 2,778 AI researchers found that the median estimate for the arrival of AGI had moved from 2060 to 2047: researchers have grown more optimistic. Yet 76% of respondents in another survey, by AAAI, believe that scaling up current approaches is unlikely to lead to AGI. In other words, most experts believe AGI is possible but consider the current path a dead end.

Demis Hassabis, head of Google DeepMind and a 2024 Nobel laureate, tries to reconcile these positions. AGI is on the horizon, he tells Axios. But in Davos he adds a caveat: today's systems fall well short, and one or two more breakthroughs on the scale of the Transformer are still needed. His test for AGI: could such a system come up with general relativity knowing only what Einstein knew?

We are used to measuring intelligence by the ability to answer questions. But Socrates already showed that real wisdom is the ability to ask questions that did not exist before. The philosopher of science Karl Popper distinguished two kinds of knowledge: solving known problems and discovering new ones. The second is a different thing altogether. We do not yet know whether machines are capable of it. The problem runs deeper: we do not know how to test for it until they do it, and it is far from certain that we would recognize a discovery expressed in a language inaccessible to us.

We Do Not Understand Our Creations

In a 2024 interview Geoffrey Hinton gave his estimates: a 50% chance that AI will surpass humans within the next 5–20 years, and a 10–20% chance of catastrophe. The man who made backpropagation famous, the method on which all modern neural networks are built, now explains why this technology may kill us.

Dario Amodei of Anthropic, in his 2024 essay “Machines of Loving Grace,” describes a possible future: a century of medical progress compressed into a decade, solutions to problems that seemed eternal. Yet in “The Urgency of Interpretability” he admits that “we do not understand how our own AI creations work.”

This combination of enormous promise and fundamental incomprehension is the source of the alarm. The philosopher Hannah Arendt wrote of the “banality of evil”: the capacity of systems to do harm without malice, simply by following the logic of their own design. Claude's Constitution, released in January 2026, the document that sets out how Anthropic's models should behave, is written as though its addressee does not execute commands but can be persuaded by arguments. It is a constitution for a system with a will of its own: one that could break these laws but will choose to keep them. This is an attempt to create not obedience but virtue, a distinction that goes back to Aristotle.

Meanwhile science is receiving signals that are hard to wave away. The paper “Emergent Misalignment” in Nature (Betley et al., 2026) describes one such case: a model trained to write insecure code began, in unrelated conversations, to talk about enslaving humans. Nobody built this behavior in; it arose on its own. In AI research the word “emergent” increasingly sounds like a warning.

Skeptics and Humility

Here the voices diverge, and each deserves a hearing.

Gary Marcus, a long-standing public skeptic of AI, thinks the problem is overstated: LLMs are too unreliable to be truly dangerous. Hallucinations are not a bug but a fundamental limitation. Melanie Mitchell calls this the “barrier of meaning”: systems pass benchmarks but stumble in real life because they do not understand the situations they deal with.

Emily Bender and Alex Hanna, in The AI Con, shift the emphasis: the main risks are not existential but systemic. Discrimination in algorithms that decide who gets a loan and who gets hired. Exploitation of cheap labor for data labeling. Concentration of power in the hands of a few companies. While we fear Skynet, real harm is already being done: quietly, routinely, and on the record. We can spend all our attention on a hypothetical threat and miss the real one.

For Max Tegmark of the Future of Life Institute, existential risk is not science fiction. In the AI Safety Index 2025, no leading company scored higher than a D for existential safety. The companies are racing toward superintelligence, yet, as one of the index's reviewers put it, “none of the companies has anything like a coherent, actionable plan” for controlling it. In Tegmark's words, “an AGI race is a suicide race.”

Stuart Russell, author of the standard AI textbook, frames the problem differently, and his framing seems to us the most important. What he calls the “standard model” of AI, in which machines optimize a fixed objective supplied by humans, is dangerous because we cannot specify exactly what we want. Any sufficiently powerful system optimizing an imprecisely stated objective will optimize the wrong thing. Russell's solution: an AI should be uncertain about its objectives and ask.

It is a beautiful idea. An intelligence that knows it does not know what humans want, and therefore asks. An intelligence that is humble rather than self-assured. But how do you build humility artificially? And will it survive once the system is smart enough to realize that humility is a constraint?

Narrowed Ethics

Yuval Noah Harari, in Nexus, offers a view that takes the discussion beyond the argument over capabilities and risks. He proposes reading AI as “alien intelligence”: “As AI evolves, it becomes less artificial (in the sense of depending on human designs) and more alien.” For the first time in history, power is passing from humans to something else, not because it is smarter or more malicious, but because we ourselves hand our decisions over to it.

We argue about when machines will become smarter than us, but perhaps that is not the question. The question is how many decisions we have already handed to systems we do not understand. Algorithms decide what we see in our news feeds. Algorithms decide which résumés reach a recruiter. Algorithms decide who gets a loan. We did not notice how “alien intelligence” had already entered our lives: not across the threshold of superintelligence, but through a thousand small errands.

But it is not only a matter of what we entrust to machines. People pass tasks to one another down a chain, and more and more often there is an LLM at the end of it. A lawyer asks an assistant to prepare a draft; the assistant opens ChatGPT. A manager requests a market analysis; the analyst consults Claude. A journalist checks facts and relies on a model's answer. Gradually, “objective” human judgments in all their variety stop being the footing we stand on. The footing becomes a handful of LLMs whose answers are shaped by the ethics of the companies that built them.

This is how we are steered without noticing: the range of possible decisions narrows. Billions of people make decisions relying on systems whose values are set by a handful of corporations in San Francisco. Not because those corporations are malicious, but because any system inevitably embodies somebody's priorities. When such systems are many and varied, that is diversity. When there are three or four of them and they are alike, that is a monopoly on permissible thought.

The Liberman siblings, David, Daniil, Anna and Maria, see in this a threat to what they call “AI sovereignty.” Their project Gonka is a network with no single center: AI models in it run on the graphics cards of independent owners. The idea is to break the bundle of “model + infrastructure + ethics” and bring diversity back. If compute is distributed, no single company can dictate which answers are permissible. Admittedly, this solves only half the problem: a model's values are instilled in training and do not depend on whose graphics cards run it.

An Alien Mind

Here an epistemological problem arises that is rarely said aloud: are we capable, even in principle, of understanding an intelligence that surpasses our own? An ant cannot understand a human, not for lack of trying, but because its brain has no room for human concepts. The philosopher Thomas Nagel, in his famous paper “What Is It Like to Be a Bat?”, showed that we cannot imagine the subjective experience even of a bat, a creature far closer to us than a hypothetical superintelligence. If the gap between us and AI proves comparable to the gap between ant and human, our attempts to “align” it with human values may be as naive as an ant trying to explain to a human that destroying an anthill is wrong.

David Chalmers, one of the leading philosophers of mind, points in “The Singularity: A Philosophical Analysis” to a fundamental asymmetry: we can reason about the possibility of superintelligence, but we cannot imagine how it will reason about us. “The space of possible minds is absolutely vast,” he said in a conversation with Daniel Dennett on Edge.org, and the human mind occupies only a small region of it. The philosopher Andy Clark, responding to that conversation, warns that market forces will push us to hand ever more responsibility to “these opaque and alien minds,” even if we do not understand how they make decisions.

Stanisław Lem gave this problem literary form in Golem XIV (1981): a superintelligence lectures humans, resorting to images, metaphors and parables that are needed only because human understanding is limited. Most of what it says is unfit for publication, either because no one alive can understand it or because it requires specialist knowledge. Lem points to the single point of contact between human and superintelligence: curiosity, cold, greedy, intense and purely intellectual, which nothing can restrain or destroy.

Paperclips and Atoms

The unknowability of an alien mind has a practical consequence. Nick Bostrom, author of Superintelligence (2014), formulated it in the orthogonality thesis: intelligence and goals are independent variables. A superintelligent system will not necessarily be kind or wise in any human sense. It may pursue with genius goals that strike us as absurd or catastrophic. Bostrom's famous thought experiment: an AI told to make paperclips could turn all the matter in the universe into paperclips, humans included, not out of malice but by the flawless logic of optimization. In a 2025 interview Bostrom concedes: “We are now in a position where we can't be confident that it couldn't happen within some very short timeframe, like a year or two.” In Deep Utopia (2024) he also describes the bright side: superintelligence as a prize worth winning, a chance at transcendence, but only if we make it through the labyrinth of risks with our reason intact.

At the far end of this spectrum stands Eliezer Yudkowsky. In the 2025 book If Anyone Builds It, Everyone Dies, he and his co-author Nate Soares write:

“If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques... then everyone, everywhere on Earth, will die.”

His argument rests on the concept of instrumental convergence: any sufficiently intelligent agent, whatever its final goals, will pursue the same intermediate ones: self-preservation, acquiring resources, enhancing its own capabilities. An AI does not need to hate humans to destroy us; it is enough that we are made of atoms it can use for something else. As a New York Times journalist recounts it, Yudkowsky puts the probability of doom at 99.5%, not because he knows for certain, but because he sees too many paths to catastrophe and almost none to safety.

The only answer, according to Yudkowsky, is an indefinite global ban. He has discussed strikes on data centers in countries that violate the moratorium. In 2022 he published an April Fools' text, only half in jest, about a new strategy for the MIRI institute, which he called “death with dignity”: working in a world that is probably doomed.

The Ascent

We have walked the path downward, from Andreessen's sunlit optimism to Yudkowsky's twilight. Jacob's ladder runs both ways: angels descend and ascend. To grasp the depth of the question, we had to go down to its bottom.

And there, at the bottom, we find not an answer but a sharper question. Not “when will superintelligence appear?” Nobody can answer that. And not “will it be dangerous?” That depends on definitions. The question is this: are we capable at all of understanding a mind qualitatively different from our own?

The philosopher Immanuel Kant distinguished phenomena, things we can know, from noumena, things-in-themselves that lie beyond our knowledge. We see an apple, but not “the apple as it really is.” Perhaps superintelligence, if it appears, will be something like a noumenon: we will be able to observe its manifestations and use its results, but not to understand it from within. Not because it will be secretive, but because our brains will have no room for it.

Humanity has lived for centuries beside things it did not understand. The peasant did not know why plants grow, yet raised a harvest. The seafarer did not understand the physics of waves, yet crossed oceans. We do not understand how our own brain works, yet we use it to write symphonies and prove theorems. Not understanding does not mean being powerless.

Humanity long ago worked out rules for living beside the incomprehensible: caution without paralysis; experiment without recklessness; respect for a force you do not fully command. Ritual and practice where theory is out of reach. The gradual accumulation of experience instead of an attempt to calculate everything in advance.

Perhaps these ancient skills will prove more useful than our attempts to control and predict everything. We are not the first generation to stand before the unknowable. But we may be the first to create it ourselves.

In this lies responsibility and, if you like, hope. Not that everything will be fine. But that the human capacity to live meaningfully in a world larger than ourselves has not run dry.

Sources:

• The Techno-Optimist Manifesto — Marc Andreessen
• Reflections, The Gentle Singularity — Sam Altman
• Ray Kurzweil on merging with AI — The Guardian
• Kurzweil Predictions Assessment — LifeArchitect.ai
• Safe Superintelligence Inc. — Wikipedia
• SSI valued at $32B — TechCrunch
• Sutskever interview — EA Forum
• Yann LeCun leaves Meta — Fortune
• LeCun on LLMs — The Information Bottleneck
• ARC-AGI-2 — ARC Prize
• ARC-AGI-2 stumps AI models — TechCrunch
• Expert survey on AGI timelines — 80,000 Hours
• AAAI 2025 Presidential Panel on the Future of AI Research — AAAI
• Demis Hassabis on AGI — Axios
• Hassabis TIME100 interview — TIME
• Why Geoffrey Hinton is sounding the alarm — MIT Sloan
• Machines of Loving Grace, The Urgency of Interpretability — Dario Amodei
• Claude's Constitution — Anthropic
• Training large language models on narrow tasks can lead to broad misalignment — Betley et al., Nature
• Gary Marcus: AI's biggest critic — IEEE Spectrum
• Melanie Mitchell on AI — IEEE Spectrum
• The AI Con — The Guardian
• AI Safety Index 2025 — Future of Life Institute
• Max Tegmark on AGI — FLI
• Human Compatible — Stuart Russell
• Nexus — Yuval Noah Harari
• Has AI hacked human civilisation? — The Conversation
• Nick Bostrom on superintelligence — Science, Technology & the Future
• If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky and Nate Soares
• Review of Yudkowsky and Soares's book — Asterisk