Series: Self-Vector Philosophy (1/4)
Intro
There is an observation I cannot shake off.
When Claude or GPT gets an answer wrong, it fails in a particular way: fluently, convincingly, with complete internal coherence. The system never hesitates. It does not say: “Wait, I’m not sure about this.” It generates an answer that sounds as though it were carefully considered, which makes it far more dangerous than an obvious blunder.
Over twenty years ago, Daniel Kahneman explained why this happens. Not with AI. With humans. But the mechanics are the same.
Two Systems
Kahneman distinguishes between System 1 and System 2. System 1 is fast, automatic, intuitive. You see a face and instantly know whether someone is angry. You hear a sentence and understand it without thinking. You drive along a familiar route without consciously steering. That is System 1.
System 2 is slow, deliberate, demanding. You calculate 17 times 24 in your head. You draft a letter to someone who matters to you. You verify whether an argument holds. System 2 consumes energy. It is uncomfortable. And it is what protects us from the systematic errors of System 1.
The crucial point, which most people citing Kahneman miss, is this: System 2 is not simply “slower thinking.” System 2 thinks about thinking. It is metacognitive. It asks: Am I too confident? Have I overlooked something? Is my assessment based on data or on a gut feeling leading me astray?
To do this, it needs a model of its own operations. You must know how you reached an assessment before you can question it.
LLMs Are System 1
Now project this onto AI systems, a comparison Kahneman never made, but one that is compelling.
An LLM generates text through next-token prediction. Every token represents an implicit decision based on statistical patterns drawn from billions of texts. No conscious rule application. No planning. No checking. Pure pattern recognition across high-dimensional spaces.
This is functionally identical to what Kahneman calls “intuition”: fast, automatic judgments that are usually right, yet produce systematic errors. Availability heuristic: what is easily retrieved is considered probable. Anchoring effect: the first piece of information influences everything that follows. Overconfidence bias: the system displays more certainty than the data warrants.
Anyone working regularly with LLMs recognises these patterns. The system does not hallucinate randomly. It hallucinates plausibly. Because plausibility is exactly what System 1 optimizes for.
Where Is System 2?
The industry’s answer is reasoning models: chain-of-thought, extended thinking. The system is forced to articulate intermediate steps before giving an answer. This helps, measurably.
Yet it does not solve the problem, because those intermediate steps are themselves System 1 output. The system “thinks” by generating tokens that look like thinking. It does not carry out genuine verification. It produces a simulation of verification.
The difference is clear: a human with System 2 can say “Wait, my intuition says X, but I know that in situations like this I tend to overestimate Y, so I should be more careful.” That requires a model of one’s own thought process. Metacognition.
An LLM can generate that sentence, but it has no model of its own inference process. It does not know how it arrived at its answer. It cannot say: “This answer is based on thin data,” because it has no access to its own data basis.
The Self-Vector as System 2
This is where the self-vector enters.
A self-vector is a compact, dynamic state that weights information processing. Six dimensions: exploration, depth, autonomy, persistence, abstraction, confidence. Plus an emergent layer that forms through experience.
The sixth dimension, confidence, is decisive. It encodes how far the system trusts its own assessment. And it shifts with experience. A system that has been wrong multiple times develops lower confidence in specific areas. Not because someone programmed it. But because the reflection layer (layer 3 of the architecture) updates the vector.
This is not a complete System 2. But it is the beginning of metacognition: a system that models its own state and derives from it how cautious it should be.
The difference from a reasoning model is fundamental: the reasoning model simulates thinking; the self-vector weights thinking. It does not say “I’m now thinking carefully.” It shifts the depth dimension upward and the confidence dimension downward, and that shift dictates how all subsequent information is processed. Not as performance, but as a state change.
What This Means in Practice
Take two systems. Both receive the same question, and the correct answer lies outside their training.
System A (a standard LLM): generates a plausible answer. Fluent, convincing, wrong.
System B (with a self-vector): the confidence dimension registers low familiarity with the topic. The exploration dimension rises: search for additional sources. The autonomy dimension drops: ask rather than answer. The system says: “I’m not familiar enough with this topic to give a reliable answer.”
Not because someone programmed that response. But because the self-vector shifted the weighting. That is functional metacognition. As Kahneman would put it: the system runs a monitor over its own intuition. An internal doubter that asks: “Are you really sure?”
The Bridge
Kahneman showed where the problem lies: System 1 without System 2 is an intuition machine without brakes. Not stupid, but unbridled. That explains why LLMs are simultaneously impressive and unreliable. They possess brilliant intuition and zero self-doubt.
The self-vector is an attempt to build structural self-doubt. Not as a disclaimer (“I am an AI system and can make mistakes”), but as a weighting function that actually changes behavior.
Whether that is sufficient remains an open question. But the direction is right: not more intuition. More reflection.