Shadows on the Chessboard: Manifolds, Metaphysics, and the Nature of Chess Evaluation

An Inquiry into the Philosophical Foundations of Neural Network Chess Engines

Abstract: This paper examines the philosophical and mathematical foundations underlying modern chess engine evaluation, with particular attention to the distinction between engines trained on self-play data (exemplified by Leela Chess Zero and the experimental engine Theoria) versus engines trained through competitive selection pressure (exemplified by Stockfish). We argue that these approaches embody fundamentally different metaphysical commitments—teleological realism versus Darwinian constructivism—and that these commitments manifest in measurable differences in evaluation coherence, computational efficiency, and interpretability. Drawing on concepts from manifold learning and realist metaphysics, we propose a framework for understanding why engines with "truer" evaluations might exhibit different computational properties than engines optimized purely for competitive fitness, even when the latter demonstrate superior tournament performance.

1. Introduction: The Eidos of Chess Evaluation

What would a perfect chess evaluation function look like? This question, seemingly technical, conceals a profound philosophical puzzle. If chess has an objective evaluative structure—positions that are truly won, drawn, or lost with perfect play—then evaluation functions can be measured against this ideal. If no such structure exists independently of our representations, then "better evaluation" can only mean "more useful for winning games."

Modern chess engines have achieved superhuman strength through two distinct paradigms. Stockfish, the dominant traditional engine, employs a neural network (NNUE) trained on positions evaluated by previous engine versions, refined through competitive testing. Leela Chess Zero (Lc0) learned entirely from self-play, deriving evaluation from game outcomes without human knowledge input. These approaches achieve comparable playing strength but may encode fundamentally different relationships to chess truth.

This paper emerged from practical experimentation with Theoria, an engine using Stockfish's architecture trained on Lc0's evaluation data. The hybrid revealed unexpected properties: approximately 80 Elo weaker in rapid testing, yet exhibiting superior scaling with search depth, greater evaluation stability, and more coherent strategic themes in its principal variations. These observations demand explanation beyond mere engineering differences.

We propose that the divergence reflects a deep philosophical split: Theoria/Lc0 embody a Platonic-Aristotelian approach where evaluation discovers pre-existing truth, while Stockfish embodies a Darwinian approach where evaluation constructs fitness without reference to correspondence. Under philosophical realism, these approaches cannot converge because they answer different questions about the nature of chess value.

2. The Manifold Hypothesis and Chess Position Space

2.1 Ambient Space and Encoding

Neural network chess evaluation operates on numerical representations of positions. A standard bitboard encoding maps positions to 773 binary dimensions (twelve piece types across sixty-four squares, plus castling rights, en passant, and side to move). The NNUE architecture's HalfKP encoding expands this to 40,960 sparse binary dimensions representing piece-king relationships.

This encoding space is the ambient space—the high-dimensional arena within which positions are represented. The choice of encoding is not neutral; it determines the geometry through which the neural network must navigate.

2.2 The Manifold Within

The manifold hypothesis, central to modern deep learning theory, posits that high-dimensional data typically occupies a much lower-dimensional surface embedded within the ambient space. Consider an analogy: a garden hose exists in three-dimensional space but remains intrinsically one-dimensional—movement is possible only forward or backward along its length.

Chess positions exhibit this property dramatically. The theoretical state space encompasses approximately 10^43 legal positions, yet these do not scatter randomly through the 2^773 possible encoding points. Legal positions satisfy constraints (both kings present, valid piece counts, no pawns on promotion ranks). Reachable positions—those achievable through legal play from the starting position—form a smaller subset still. Practically occurring positions—those arising in actual games—occupy a thin "ribbon" threading through the vast ambient space.

This ribbon is the manifold. Positions cluster around strategic patterns, connect through move sequences, and organize according to chess principles. The manifold's structure explains why neural networks can learn chess evaluation without memorizing 10^43 positions: they learn the function only along the ribbon's surface.

2.3 The Encoding Distortion Problem

Different encodings project the same underlying chess structure into different geometric configurations. A position's "neighbors" in HalfKP space may differ from its neighbors in bitboard space. Strategic similarities may become geometric distances; unrelated positions may accidentally cluster together.

This projection introduces distortion. The manifold in encoding space is not the "true" manifold of chess relationships—it is a shadow, warped by representational choices. Some encodings preserve structure better than others, but no encoding is neutral.

NNUE's architecture explicitly acknowledges this problem through king buckets—separate weight matrices activated by different king positions. This partitioning recognizes that the encoding creates artificial discontinuities (same position, different king square, completely different activation patterns) requiring manual correction.

3. Platonic Shadows: A Metaphysical Framework

3.1 The Eidos and Its Shadows

Using Platonic realism as inspiration, chess could be considered to possess an eidos—an ideal form or evaluative structure existing independently of representation. Every position has a true value (win, draw, or loss with perfect play), and this truth exists whether or not any engine discovers it.

If the eidos exists, we can describe a hierarchy of shadows:

Each level introduces distortion. The learned evaluation approximates the encoding manifold, which approximates the intrinsic manifold, which approximates the eidos. Truth degrades through successive projections.

3.2 The Kaleidoscope Analogy

A kaleidoscope shatters a coherent image into fragmented, repeated, twisted reflections. The encoding performs a similar operation on the eidos: the true evaluative structure is fractured into fragments distributed across the high-dimensional encoding space.

Some fragments preserve local accuracy—tactics remain sharp because material relationships survive encoding robustly. Other fragments distort severely—subtle strategic truths scatter across unconnected regions.

The neural network must learn to navigate this kaleidoscope: recognizing which fragments belong together, compensating for apparent discontinuities, stitching coherent evaluation from scattered shards.

3.3 Search as Shadow-Stitching

This framework reframes the relationship between evaluation and search. Static evaluation navigates the kaleidoscope directly, subject to all its distortions. Search compensates by exploring multiple fragments, testing whether local evaluations compose coherently across move sequences.

Deep search reconstructs regions of the intrinsic manifold by aggregating information across many encoding-space fragments. Strategic truths invisible in any single fragment emerge from the aggregate—not because the evaluation discovered them, but because search stitched enough shards together to glimpse the pattern.

This explains an observed phenomenon: Stockfish exhibits "hyperdimensional strategic" play at high search depths but decomposes into pure tactics at low depths. The strategy was never in the evaluation; it emerged from search traversing enough fragments to overcome encoding distortion.

4. Two Ontologies of Evaluation

4.1 Teleological Evaluation (Theoria/Lc0)

Lc0's evaluation derives from outcomes. Self-play games proceed to termination; results propagate backward through the game tree. A position's value reflects where the game leads with continued play.

This is teleological evaluation: value defined by telos, the end toward which a position tends. The evaluation asks what the position wants to become, discovering the form actualizing itself through play.

Under this approach, evaluation aims at correspondence. The network learns to predict game outcomes, and game outcomes are real—checkmate is checkmate, draw is draw. Truth flows backward from terminal states, grounding evaluation in chess reality.

The process resembles Aristotelian actualization: the acorn contains the oak implicitly; the position contains its destiny. Evaluation makes explicit what was always latent in the position's nature.

4.2 Darwinian Evaluation (Stockfish)

Stockfish's traditional training follows a different logic. Evaluation features that correlate with winning are retained; those that don't are discarded. The fitness function is competitive performance, tested through millions of games against various opponents.

This is Darwinian evaluation: value constructed through selection pressure rather than discovered through correspondence. Features survive not because they reflect chess truth but because they produce wins.

Crucially, winning and truth-tracking can diverge:

The evaluation doesn't aim at correspondence to the eidos. It aims at survival in the competitive ecology. "Truth" is whatever works.

4.3 The Non-Convergence Thesis

These approaches embody different metaphysical commitments and answer different questions:

These questions converge only if truth and fitness perfectly align—if knowing the true value of positions is always the best strategy for winning. But chess involves opponents, time pressure, practical complications. An evaluation calibrated to exploit opponent weaknesses may outperform one that simply reports truth.

Under philosophical realism, we must conclude: Stockfish's Darwinian process cannot produce truth, only fitness. It may approach truth accidentally (where truth happens to be useful) but has no mechanism guaranteeing correspondence. Theoria's teleological grounding at least aims at truth, even if it achieves only approximation.

The approaches should not converge because they pursue different goals. One seeks the eidos; the other seeks survival.

5. Empirical Signatures of the Divergence

5.1 The Sharp Twisted Mosaic vs. The Fuzzy Icon

The philosophical distinction manifests in observable evaluation properties:

Stockfish (Sharp Twisted Mosaic): Each fragment is locally precise—tactics are razor-sharp, immediate threats correctly identified. But fragments don't compose coherently. The global picture is twisted, distorted, inconsistent across regions. Apparent strategic plans may be post-hoc rationalizations of locally-winning moves strung together without genuine throughline.

Theoria (Fuzzy Icon): The global shape is truer—strategic themes cohere, plans connect logically. But edges are soft, details blur. Tactical precision suffers. The icon captures the eidos's form but lacks the mosaic's local sharpness.

This explains Theoria's 80 Elo deficit in rapid testing: fast time controls reward sharpness over coherence. Tactics decide games. The mosaic wins because its fragments cut precisely where immediate accuracy matters most.

5.2 Principal Variation Coherence Decay

A principal variation (PV) represents the engine's predicted optimal sequence for both sides. Analysis reveals systematic differences in PV quality:

Early PV moves (1-5): Both engines perform well. Heavy search verification ensures accuracy.

Middle PV moves (5-15): Stockfish's lines begin interleaving tactics and strategy incoherently—thematic discontinuities, plans that contradict earlier moves, shifts in character without logical motivation.

Late PV moves (15+): Stockfish's continuations become frankly hallucinatory—confabulated sequences that "feel" plausible but dissolve under independent verification.

Theoria's PVs maintain thematic coherence deeper into the line. Tactics serve strategy; plans develop logically. Even past search horizon, the extrapolated moves reflect genuine chess logic rather than pattern-matched ghosts.

This is the "Eldar the Elf" phenomenon: Stockfish plays a winning move, and when asked for its plan, the honest answer is "strawberry ice cream"—a confabulated association with no correspondence to chess reality. The move works; the explanation is hallucination.

5.3 Strategic Theme Persistence

Observational analysis reveals that Stockfish's play interleaves tactics and strategy in ways that resist coherent interpretation. A typical sequence might show:

No narrative connects these moves. Each is locally optimal, but the ensemble tells no story.

Theoria's play exhibits diachronic coherence—themes persist across moves:

The game tells a story. Moves connect to a throughline. This is exactly what we would predict if Theoria's evaluation encodes genuine strategic structure while Stockfish's encodes fitness-maximizing patterns without semantic content.

5.4 Computational Efficiency and Scaling

Preliminary testing suggests Theoria scales more efficiently with search depth than Stockfish—each additional node purchases more evaluation accuracy.

The manifold interpretation explains this: Stockfish's search does double duty, both exploring the game tree and compensating for evaluation distortions. Theoria's search does single duty, amplifying an already-coherent evaluation rather than correcting a fragmented one.

If the manifold is less distorted, less computation is wasted stitching fragments. Truth is compressible; fictions require overhead to maintain consistency. A truer evaluation should be computationally cheaper to exploit.

6. The Dark Forest and Hallucinatory Magic

6.1 Tal's Dark Forest

Mikhail Tal, the great attacking world champion, described entering positions where normal evaluation broke down—chaos beyond calculation, "a dark forest where 2+2=5." He won not by understanding but by navigating darkness more comfortably than opponents.

When asked to explain his play, Tal often couldn't generalize. Each position was its own hallucination, navigated by intuition without composable principles. The explanations that worked "in the forest" became nonsense outside it.

6.2 Stockfish as Permanent Forest-Dweller

Stockfish's entire evaluation operates in Tal-space. Locally coherent, globally hallucinatory. It wins by traversing darkness faster than opponents, not by carrying light.

The PV is a path through the forest that happened to work, not a map that generalizes. The evaluation "sees" advantages real only within its encoding—ghosts that happen to point the right direction.

This is "hallucinatory magic": producing correct results without causal transparency. The trick works, but even the magician doesn't know why. The mosaic coheres through accumulated selection, not understanding.

6.3 Theoria's Ambition: Carrying Light

Theoria represents an alternative: evaluation that references something real, even if fuzzily. Not perfect illumination, but a lamp in the darkness.

The ambition is an engine that could answer "why this move?" with something truer than strawberry ice cream. Not confabulation, but reference. Not hallucination, but correspondence—however approximate.

Whether this ambition is achievable, or whether chess is fundamentally a dark forest admitting no illumination, remains an open question.

7. Methodological Implications: Testing Truth vs. Fitness

7.1 The SPRT Problem

Standard engine testing uses Sequential Probability Ratio Testing (SPRT) measuring win/draw/loss rates between engines. This methodology cannot distinguish truth from fitness—it measures competitive survival, not correspondence to the eidos.

An engine with truer evaluation but lower Elo registers as inferior. SPRT collapses all sources of strength into "wins more," obscuring whether wins derive from understanding or exploitation.

Under SPRT, useful fictions beat inconvenient truths. There is no reward for correspondence independent of competitive outcome.

7.2 Proposed Empirical Tests

Distinguishing truth-oriented from fitness-oriented evaluation requires alternative methodologies:

PV Coherence Decay Analysis: Extract principal variations at fixed depth. Verify each move independently with much deeper search. Measure at what ply each engine's PV diverges from verified-best. Prediction: truth-oriented engines maintain accuracy deeper.

Evaluation Stability Under Transposition: Reach identical positions via different move orders. Compare evaluation stability. Prediction: truth-oriented engines show less variance (encoding artifacts matter less when the manifold is less distorted).

Concept Probe Accuracy: Train linear probes on internal network activations to predict human chess concepts (passed pawns, king safety, outposts). Measure how linearly decodable concepts are. Prediction: truth-oriented evaluations align better with human-interpretable concepts.

Extrapolation to Novel Structures: Test on positions outside training distribution—unusual material imbalances, rare pawn structures, composed studies. Prediction: truth-oriented engines generalize better (truth transfers; fitness is local).

Plan Verbalization Coherence: Feed PVs to language models for explanation. Rate explanations for coherence, consistency, chess validity. Prediction: truth-oriented PVs yield sensible narratives; fitness-oriented PVs yield "strawberry ice cream."

7.3 Preliminary Evidence

Initial investigation of Theoria versus Stockfish reveals:

This preliminary evidence aligns with theoretical predictions but requires systematic documentation.

8. Philosophical Implications

8.1 The Limits of Darwinian Intelligence

If Stockfish's strength derives from fitness-maximization without correspondence, it represents a form of intelligence fundamentally alien to understanding. It succeeds without knowing; it wins without truth.

This has implications beyond chess. Machine learning systems optimized purely for performance metrics may systematically diverge from truth where truth and performance come apart. They will hallucinate confidently, confabulate explanations, and resist interpretation—not as bugs but as features of Darwinian optimization.

8.2 The Value of Truth Beyond Utility

Theoria's potential computational efficiency suggests truth may have practical value even within an engineering framework. If correspondence to reality enables better compression, scaling, and generalization, then truth-seeking and fitness-seeking may diverge in the short term but favor truth in the long term.

Alternatively, truth may have value independent of utility:

Even if Theoria never surpasses Stockfish competitively, it may illuminate chess in ways Stockfish cannot.

8.3 The Possibility of Synthesis

The sharp mosaic and fuzzy icon may not be mutually exclusive. Hybrid approaches might combine:

Whether such synthesis is possible without inheriting the worst of both approaches remains to be seen.

9. Conclusion: Two Paths Up the Mountain

Chess evaluation has achieved superhuman strength through two philosophical paths: one seeking correspondence to an ideal evaluative structure, the other seeking competitive fitness without reference to truth.

These paths may climb the same mountain from different sides, eventually converging at perfect play. Or they may ascend different mountains entirely, optimizing for different goods that cannot be reconciled.

The distinction matters beyond chess. As machine learning systems pervade high-stakes domains, understanding whether they discover truth or construct useful fictions becomes critical. Systems that hallucinate successfully may be more dangerous than systems that fail transparently.

Theoria, despite its competitive deficit, represents an important experiment: can we build systems that aim at truth, not merely fitness? Can correspondence compete with survival? Does the eidos cast shadows we can learn to read?

These questions admit no easy answers. But they deserve investigation. The chessboard, that ancient crucible of strategy and calculation, may yet teach us something about the nature of machine understanding—and its limits.

References

This essay synthesizes primary analysis with theoretical frameworks from:

Empirical claims regarding Theoria are based on ongoing experimental work and require systematic documentation for formal publication.

Appendix: Glossary of Key Terms

Ambient Space: The high-dimensional encoding space within which chess positions are represented numerically.

Eidos: (Platonic) The ideal form or perfect evaluative structure of chess, existing independently of representation.

Encoding Manifold: The lower-dimensional surface occupied by chess positions within the ambient space, shaped by representational choices.

Intrinsic Manifold: (Hypothetical) The true topology of chess positions independent of encoding—the "real" manifold of which encoding manifolds are shadows.

Darwinian Evaluation: Evaluation constructed through selection pressure (competitive fitness) without guarantee of correspondence to truth.

Teleological Evaluation: Evaluation derived from outcomes (game terminations), grounding value in actual chess results.

PV Coherence Decay: The phenomenon whereby principal variations become increasingly hallucinatory (less verified, more confabulated) at greater depths.

Diachronic Coherence: The property of strategic themes persisting coherently across move sequences within a game.

Hallucinatory Magic: Successful performance achieved through pattern-matching without causal transparency or correspondence to reality.


Prepared for further development and peer review.