WritingArticle preview
Article #2

Does Intelligence Require Consciousness?

What subjective experience might contribute to intelligence, where arguments about computation fall short, and how a machine could show us that it understands a world.

A machine could discover a new law of physics, explain why it works, and design an experiment that confirms it. Would that establish that the machine understands something? Would it establish that there is anything it feels like to be that machine?

Those questions can sound almost interchangeable. In a person, understanding often arrives as an experience: a confusing collection of facts suddenly makes sense. It is natural to assume that the experience and the intellectual achievement belong together. But whether they must belong together is precisely what we need to establish.

The distinction matters for how we think about AI. If subjective experience supplies an essential ingredient of reasoning, a machine that lacks it might encounter limits that more computation cannot overcome. If the intellectual work can happen without experience, machines could surpass us while leaving the mystery of consciousness unresolved. There is another possibility too: the machines doing that work might themselves be conscious.

I want to separate those possibilities, then ask how much of the argument we could turn into an experiment. Scientific discovery seems like a particularly useful place to try. It demands more than producing a convincing explanation of something somebody else already understood.

What would have to be explained?

By intelligence, I mean capacities such as learning, reasoning, planning, and developing useful explanations. By consciousness, I mean subjective experience: the felt character of seeing a color, hearing a note, being confused, or having an idea. A system could also monitor its own errors or describe its internal states. Calling that self-awareness would still leave open whether those operations feel like anything.

Understanding is the troublesome word between them. We usually expect someone who understands a phenomenon to predict what will happen, explain why, and recognize when the explanation breaks. But a person might insist that those abilities are evidence of understanding while something further constitutes understanding itself. That disagreement cannot be settled by quietly choosing a definition.

There are at least four questions here:

  1. Can a machine perform the intellectual work humans perform?
  2. When should we count that performance as understanding?
  3. Can a machine have subjective experience?
  4. Does some essential part of intellectual performance depend on that experience?

A positive answer to the first question would put pressure on some answers to the second. It wouldn't automatically answer the third or fourth. In particular, observing an intelligent machine wouldn't show that intelligence can exist without consciousness unless we also knew the machine was unconscious.

That last qualification is easy to lose. It is also central to interpreting any experiment we might run.

The difficulty with explaining experience

Consider a complete account of what happens when someone sees a red object. It might describe light entering the eye, neural signals, the discrimination of colors, the formation of a memory, and the person's ability to say what they saw. David Chalmers's formulation of the hard problem of consciousness asks why those processes should be accompanied by experience at all. Explaining what a system does can appear to leave unexplained why doing it feels like something.

This is a philosophical challenge to the sufficiency of a functional explanation. It isn't a measurement showing that a nonphysical ingredient exists.

There is a serious reply. Perhaps we are asking for an additional thing after the thing to be explained has already been described. On a physicalist account, experience could be identical with certain physical processes, encountered through the very different perspectives of introspection and external observation. An inability to recognize the identity intuitively would not make it false. Daniel Dennett challenges the separation of a further hard problem from the explanation of cognitive functions. On that view, setting experience aside while explaining everything the mind does risks building the mystery into the question.

The empirical case for studying the brain is strong. In a controlled propofol study, researchers tracked changes in responsiveness and electrical brain activity as volunteers entered and emerged from anesthesia. This connects a physical intervention to reproducible changes in the markers used to assess consciousness. It doesn't give the experimenter direct access to a volunteer's experience, but it gives research something considerably more useful than an intuition alone.

Still, dependence and identity are different claims. Learning how a physical process changes experience doesn't, by itself, settle what experience is. Conversely, difficulty explaining that identity doesn't establish that it is impossible. The physicalist owes an account of experience; the critic owes more than the observation that such an account is hard to imagine.

The first-person and third-person perspectives create a practical problem too. I have direct access to my own experience. I infer yours from what you say, how you behave, and the fact that we are similar organisms. With machines, the structural similarity is much weaker. Fluent reports of experience deserve scrutiny, but the absence of a human brain cannot simply serve as proof of its absence. We need reasons for whichever standard we apply.

Computability is a separate claim

Even if consciousness is entirely physical, it doesn't follow by definition that its relevant processes can be reproduced by a digital computer. That requires a further claim about physics. And even if those processes are computable, reproducing them might require impractical amounts of time or memory.

These are three different issues: what exists, what an algorithm can calculate in principle, and what a finite device can calculate in practice. Confusing them makes both the optimistic and the skeptical arguments about AI look stronger than they are.

Roger Penrose offers a particularly ambitious argument that human mathematical understanding goes beyond computation. Its starting point is Gödel's incompleteness theorem. Roughly, a sufficiently expressive, effectively specified, sound formal system cannot prove every arithmetic truth. Under the relevant assumptions, reasoning about the system from outside can establish a statement that its own rules cannot prove.

Penrose argues that the human ability to recognize such truths cannot be fully captured by an algorithm. His reply to critics of Shadows of the Mind sets out the argument in terms of idealized mathematical understanding. He also proposes that explaining consciousness will require physical processes beyond ordinary computation. The logical argument and the proposed physics are separate steps; accepting a limit on a formal system does not establish a mechanism in a brain.

The disputed step is the passage from formal systems to human minds. People make mistakes, and recognizing the truth of the relevant statement depends on assumptions about the system's soundness. We cannot simply grant the human an infallible view from outside every possible system. An algorithm can also reason about another formal system and adopt stronger assumptions; standing outside one system is not automatically standing outside computation. In his technical critique, Solomon Feferman rejects Penrose's Gödelian argument even while expressing his own skepticism about computational accounts of mind. The disagreement concerns what the theorem licenses us to conclude, rather than whether the theorem is true.

Ordinary quantum computation doesn't remove this distinction. It may make some calculations much more efficient, but it does not generally turn uncomputable problems into computable ones. David Deutsch's paper on the universal quantum computer helps distinguish the physical implementation of computation from an abstract specification of its powers. Invoking quantum mechanics is not yet an explanation of an ability that algorithms cannot reproduce.

For the consciousness question, the useful consequence is modest. We should neither treat the computational reproducibility of human thought as a theorem nor treat Gödel's theorem as a proof that human thought escapes computation. A claimed difference needs a more specific argument or an observable consequence.

What irreducibility would imply for machines

Suppose consciousness does resist a fully reductive explanation. Several very different futures would remain possible.

Experience might accompany cognition without contributing any additional causal power to it. Then everything required to solve a scientific problem could still be implemented computationally. Whether the resulting system experienced anything would be a further question.

Alternatively, experience might be a fundamental feature of nature, organized into minds by the right physical structure. That possibility does not, on its own, privilege biological tissue. We would need an account of which structures support experience before deciding whether a silicon system could have it.

The possibility with the clearest implications for a limit on digital AI is stronger: experience participates in cognitive work that ordinary computation cannot reproduce. To make that claim useful, we would need to identify the work. Does it enable a particular kind of abstraction? The recognition of mathematical truth? The selection of an experiment? Saying that consciousness supplies understanding names the proposed difference without explaining its operation.

Even discovering such a physical difference would not necessarily establish a permanent human advantage. If nature contains a useful process that digital computers cannot reproduce, an engineered device might eventually exploit that process directly. A limit on a kind of computation would not necessarily be a limit on everything we could build.

Nonphysical or theistic accounts introduce further possibilities. Consciousness as a basic feature within nature is a different claim from consciousness connected to something beyond nature. Neither claim alone tells us which machines could think or have experiences. Specific consequences would require additional assumptions. If two accounts predict the same observations, the experiments considered here would have no basis for choosing between them.

This is why the practical AI question is narrower than the whole metaphysical dispute. What, if anything, does consciousness contribute to intellectual performance that a computational system could not supply?

Novel answers and the source of understanding

There are already examples worth taking seriously. In its 2025 AlphaEvolve report, Google DeepMind said that the system improved the best-known solutions in about 20 percent of a set of more than fifty open mathematical problems. AlphaEvolve combines language models, automated evaluators, and evolutionary search. It proposes programs, checks them, and uses the results to guide further exploration.

These are reported advances with checkable mathematical content. Dismissing them as the repetition of known answers misses what makes them interesting. But attributing the whole achievement to an independent machine mind also skips part of the explanation. People chose the problems, arranged the search process, and supplied ways to recognize an improvement.

The attribution question is difficult because human discovery also depends on inherited infrastructure. Einstein had access to mathematical notation, earlier theories, experimental results, and an intellectual community. Requiring a machine to invent everything it uses would apply a standard no human scientist has met. The fact that people built a system is not enough to explain away whatever intellectual work the system subsequently performs.

The distinction I want to investigate is where the system can revise the terms of its own inquiry. Can it decide which discrepancy matters, invent a measurement, and replace a representation that has become an obstacle? Can it recognize that an objective supplied by its designers is too narrow to capture the phenomenon it has found?

AI Feynman, a physics-inspired method for discovering equations from data, illustrates another part of this territory. Its authors tested it against known physics equations. Recovering a compact relationship from observations is a substantial ability. Choosing which observations to obtain and which quantities to invent would extend the investigation further.

There need not be a sharp boundary at which recombination becomes understanding. Humans recombine ideas too. We can still ask whether a system's concepts support explanations that survive new tests, and how much of the conceptual work people supplied along the way.

Give it a world it hasn't seen

Our world makes that investigation unusually hard. AI systems have learned from enormous amounts of human writing, including descriptions of nature and the mathematics used to explain it. When a model supplies a good scientific explanation, tracing the origin of its concepts can be difficult. Simply asking a familiar question in unfamiliar words doesn't remove that problem.

Here is a more useful experiment. Build several simulated worlds with coherent rules unlike the physics we know. The rules should produce enough structure for sustained inquiry: observable regularities, interactions participants can manipulate, and phenomena that become accessible only when their instruments improve. The designers know the rules. The participants do not.

Give separate communities of people and AI systems comparable observations and actions. Both can manipulate their surroundings, store results, communicate, and build tools. Neither receives a list of scientific concepts to discover or a score indicating that its latest theory is getting warmer. The initial task can be broad: learn enough about this place to predict and control what happens in it.

That instruction still supplies a purpose. There is no experiment without choices about what participants can do and what researchers will count as evidence. The aim is to avoid supplying the particular abstraction whose discovery is under investigation.

The interface would matter enormously. If a world is presented through named quantities such as mass, charge, and temperature, much of the scientific decomposition has already been done. Participants should instead encounter observations from which useful quantities might be inferred. They should be able to devise procedures that reveal something the initial display conceals. Machine participants must not gain access to the simulator's source or hidden state through tools that human participants lack.

This would test a community rather than a lone genius. Discoveries accumulate through shared records, criticism, specialization, and better instruments. That is the modest meaning of civilization here. A community might invent notation, divide experimental labor, and preserve a failed theory so another investigator doesn't repeat the same mistake. No one needs to build a city before being allowed a notebook.

A hybrid group would also be useful. If people and machines together outperform either alone, we would want to understand which parts of the process each contributes. The answer could be more informative than declaring a winner.

Watch the path to an explanation

A final leaderboard would lose much of the evidence. I would want to see which observations each group considered surprising, which measurements it invented, and which experiments it chose when two explanations fit the same data. Does its explanation predict a new effect? Does a concept developed in one setting help it understand another?

Consider a world where objects exchange something that can be stored and transported. Early experiments suggest that the amount held by each object is enough to predict the exchange. A community builds a theory around that quantity, and the theory works across many situations.

Later, better instruments reveal reliable exceptions. The group discovers that the exchange also depends on how the contents are distributed internally. It could attach a separate correction to every troublesome object. Or it could invent a higher-level quantity that captures the relevant internal organization and makes the exceptions part of a broader rule.

The second approach earns its status if it does more than tidy old observations. It should predict what will happen when researchers build a new kind of object or combine familiar objects in a previously untested arrangement. If the prediction succeeds, the revised representation has survived an opportunity to fail.

The new concept need not be a primitive in the simulator's code. Temperature can describe a population of particles even when a simulation assigns no temperature field to any individual particle. A machine community could discover a similarly useful collective quantity that its world's designers never named. Recovering the implementation's variable names would be a poor substitute for this kind of abstraction.

An unfamiliar vocabulary would not be sufficient evidence either. Renaming familiar ideas is easy. A strange representation would become interesting when it explains observations, supports interventions, or predicts something other representations missed. Human scientists could first investigate the same world independently, then examine the machine community's theories. That comparison could reveal useful abstractions the human group had overlooked without making human surprise the final criterion of truth.

There is a further extension: allow the machine community to improve its own research tools. It could write analysis programs, develop instruments within the world, or change how its agents share results. Researchers would then be able to examine whether a crucial discovery method was supplied by people or developed during the inquiry. The original system would still have human designers. The question is what their design enabled it to originate.

What would make the comparison fair?

From scratch is necessarily an approximation. Humans arrive with perceptual habits shaped by evolution and experience. AI systems arrive with architectures designed by people and concepts learned through training. A synthetic world can withhold facts about its own rules; it cannot remove every prior assumption from either group.

Even unfamiliar physics may favor one participant. A world built around manipulating symbols could suit a machine's interface, while another organized around spatial interactions could suit human perception. Researchers would need several kinds of worlds, varied interfaces, and enough replication to see whether an apparent advantage survives those choices. One clever simulator would not settle the issue.

Resource budgets would need the same care. Equal clock time is not equal opportunity when one group can copy itself or exchange records almost instantly. Researchers could report observations, experimental actions, communication, computation, and elapsed time separately, then compare trajectories under several constraints. There is no obvious conversion that turns all those resources into a single fair score.

Some phenomena should be withheld until after theories are proposed. Rival explanations could then be tested against fresh interventions. It would also help to record when a person supplied a hint, repaired a tool, or changed an agent's instructions. An apparently autonomous conceptual advance might otherwise depend on a small but decisive human contribution.

A limited version could begin with a world containing only a few discoverable regularities. That could test experiment selection and concept revision. The stronger proposal, involving sustained cumulative science across rich worlds, would demand much more development. This is an experimental design to investigate, not an established benchmark with known results.

There is also a limit built into simulation itself. A world running on an ordinary computer has computable dynamics. Success there would demonstrate abilities in that class of worlds. It would not settle whether real brains exploit some physical process the simulator cannot represent. For that stronger question, investigations of actual physical systems would remain essential.

How much would the result tell us?

If machine communities repeatedly developed productive explanations without receiving the crucial concepts or tests from people, it would become harder to maintain that machines can only operate inside explanatory frameworks humans have already supplied. We would have evidence about scientific cognition under those conditions, including evidence about where particular concepts originated.

We would still not know whether the machines were conscious. Their success would challenge an argument that digital systems necessarily lack a required intellectual capacity. It would not establish that subjective experience is unnecessary, because the successful systems might possess it. A theory connecting experience to observable structure would be needed to take that further step.

A persistent failure would need just as much care. Suppose machine groups became excellent at prediction and optimization but required a human hint whenever an explanation needed to be reorganized. That would identify a specific capability worth studying. Before invoking consciousness, researchers would need to investigate memory, training, incentives, experimental access, and the organization of the community.

Repeated failure across improved systems would make those explanations more demanding. It would not prove a limit on every possible algorithm. No finite series of experiments exhausts the space of programs, and a limit of current architectures is different from a limit of computation. The useful result would be a reproducible boundary whose causes could be investigated.

This is the part of the consciousness debate I want to make more concrete. We can ask a machine to choose what needs explaining, build the means to investigate it, and revise its concepts when the world refuses to cooperate. We can examine the record of that work instead of judging only its final answer.

Whether there is experience behind the work may remain unresolved. But a claim that experience is essential to scientific understanding should become more specific: somewhere between noticing an anomaly and confirming a new theory, what does consciousness make possible?