The Technical Research Lineage

4. Technical research lineage

A map of the technical literature you should know exists. Each numbered subsection corresponds (roughly) to a chapter in Part III of this book, where the mechanism, the code, and the trade-offs are worked out in full. This section gives you the names, the significance, and the pointer — enough to recognise each idea, know why it earns a place in the stack, and find where it's built out. It is a reading map, not the build manual: this section stays at the level of names + significance + pointer, and the Part III chapters own the mechanism and code. The → at the end of each subsection is where the depth lives.

4.1 Base models

The substrate every companion runs on. You will almost certainly not train one — but the shape of these results decides what you can afford to run, which is the whole ballgame for an always-on character.

• Transformer (Vaswani et al., 2017). The architecture under everything since: attention replaced recurrence and made scale tractable. You inherit it; you don't reinvent it. Worth knowing only so the later acronyms have a home.
• Scaling laws (Kaplan 2020; Chinchilla 2022). Kaplan showed loss falls predictably with parameters, data, and compute. Chinchilla then corrected the recipe — most pre-2022 models were badly undertrained, and you get more from data-proportional training than from raw parameter count. This is why a well-trained 8B can embarrass a sloppy 70B, which is the entire economic case for running a small model locally instead of renting a giant one.
• Mixture of Experts (Mixtral, DeepSeek, Qwen3-Coder, GPT-4-class). Route each token to a few of many expert subnetworks: you buy the capacity of a huge model while paying inference for a fraction of it. For a companion that has to be present 24/7 — where inference cost, not training, is the dominant operating expense — this is the lever that makes continuous presence affordable.
• Open-weights families to know in 2026: Llama 3.x / 4, Qwen 3, DeepSeek-V3, Mistral, Gemma 3. The sovereignty thesis this project is built on (own the runtime, own the friend; → ch. 18) is only real if the weights are yours to hold. These are the families that make it possible. → ch. 13.

4.2 Adapting a model to a persona

How a generic base model becomes your character. Listed cheapest-first, which is also the order you should reach for them.

• System prompts and persona injection. What everyone does — and further than newcomers expect. The character-card community (§3) gets startlingly far on prompt craft alone, which is why the field's working wisdom is "the limit is the card, not the model."
• In-context learning / few-shot. Drop example exchanges into the context and the model imitates the voice. Cheap, instant, no training run; the fastest way to dial in tone before you commit to anything heavier.
• LoRA / QLoRA (Hu 2021, Dettmers 2023). Parameter-efficient fine-tuning — train a small adapter instead of the whole network, and (with QLoRA's quantization) do it on a single consumer GPU. This is the point where "fine-tune the persona" stops being a datacenter project and becomes a weekend one.
• DPO (Rafailov 2023) and ORPO. Preference tuning without standing up a separate reward model: show the model preferred vs dispreferred replies and it learns the gradient directly. The cheap, stable way to sand off behaviors (manipulation, off-character drift) you couldn't fix in the prompt.
• Distillation. Teacher → student: a big capable model generates training data that teaches a small cheap one to behave like it. How nearly everyone serious about cost actually ships.
• Note: the canonical sequence is prompt → RAG → fine-tune → distill, and skipping ahead is the #1 trap — fine-tune too early and you bake in choices you should still have been iterating on in a prompt, at ten times the cost to undo. → ch. 20.

4.3 Memory

Active subfield. The 2026 state of play:

• Vector RAG (Pinecone, Weaviate, Qdrant, pgvector). Default starting point for "long-term memory": embed past turns, retrieve the ones nearest the current message. Necessary but not sufficient — it finds text that's similar, not facts that are true or current, which is exactly the gap the higher-level frameworks exist to fill.
• Mem0, Letta (formerly MemGPT), Zep. Higher-level memory frameworks that handle the part raw vector search ignores — what to remember, what to forget, what to summarise. Mem0 emphasises lightweight; Zep emphasises temporal/graph (when a fact was true); Letta emphasises an agent that self-manages its own context window.
• Graphiti / GraphRAG (Microsoft). A knowledge-graph layer on top of retrieval, so the system can answer who/when/relation questions ("who did she mention last week, and how are they connected?") that naive vector similarity fumbles.
• Anthropic "Dreaming" (May 2026). An async, hippocampal-replay-style process that consolidates memory between sessions rather than during them — the model "sleeps on" the day's conversations. Recent enough that production patterns are still emerging, but directly load-bearing for a persistent companion that should wake up changed by yesterday.
• Google Memory Bank (I/O 2026). A managed memory primitive for Gemini agents — the same problem, packaged as hosted infrastructure for teams who don't want to own the state layer.

→ ch. 15, ch. 16.

The single most-cited research observation that builders keep ignoring: stuffing everything into the context window degrades quality even when it fits. You need retrieval and summarisation, not just a bigger context. The 2025 "Lost in the Middle" line of work and its successors keep confirming this.

4.4 RAG

• The original Lewis et al. RAG paper (2020). Coined the pattern: retrieve relevant documents, put them in the prompt, generate grounded in them instead of in the model's frozen memory. Background, but the whole vocabulary starts here.
• Hybrid search (BM25 + dense). Keyword search and embedding search miss in different directions — one nails exact names and IDs, the other catches paraphrase — so combining them beats dense-only across most benchmarks. A cheap, near-free win; make it the default.
• Rerankers (Cohere, bge-reranker). A second-pass model re-scores your top retrievals for actual relevance before they hit the prompt. Big quality lift for a small latency cost — typically the highest-ROI single component in a retrieval stack.
• 2026 framing. "RAG is the knowledge layer; memory is the state layer." Knowledge is what the character knows about the world (stable, factual, shareable across users); state is what has happened between you and her (personal, accumulating, hers alone). Conflating the two is a top-tier design error — it's why §4.3 and §4.4 are separate sections. → ch. 16.

4.5 Agents

• ReAct (Yao 2022). Interleave reasoning and tool calls in one loop — think, act, observe the result, think again. The mental model underneath nearly every agent built since.
• Tool use / function calling. The model emits a structured call, your code runs it, the result returns to the context. Now native to all major model APIs; this is the mechanism by which a companion does things — checks the weather, remembers a date, sends a message — rather than only talking about them.
• MCP — Model Context Protocol (Anthropic, 2024). A standard wire format for exposing tools and data to any model. 2026 reality: it's the lingua franca — build your companion's capabilities behind MCP once and they stay portable across models and runtimes instead of being welded to one vendor.
• LangGraph and similar. Explicit-graph orchestration for multi-step flows. The counter-position is worth holding onto: Anthropic's "Building Effective Agents" (2024) argues a fixed workflow usually beats a freeform agent loop — more reliable, cheaper, far easier to debug. For a companion, most of what looks like "agency" is better built as a workflow you control.
• Multi-agent. Several specialised agents collaborating. Real gains on narrow, decomposable tasks — but usually overkill, and a latency-and-cost tax, for a companion, which is meant to feel like one character, not a committee deliberating behind the curtain.
• Personal-agent runtimes (OpenClaw, 2025–). Agent frameworks reconceived as products people live with rather than task-runners you fire and forget — which is the exact bridge from "agent" to "companion." See the agent-lineage section of §1 for the history and why the two converge.

→ ch. 17.

4.6 Multimodal / embodiment

Giving the character a voice, ears, and a body. The stack splits cleanly along the senses.

• TTS (the voice): ElevenLabs (closed, gold-standard prosody), StyleTTS2, XTTS-v2, Kokoro, Orpheus. The thing that separates these is prosody — rhythm, emphasis, emotion — not word accuracy. This is where the uncanny valley lives or dies; a flat read undoes a perfect persona.
• STT (the ears): Whisper / faster-whisper, Distil-Whisper, Nvidia Parakeet/Canary. Largely a solved problem in 2026 — choose on latency, language coverage, and whether it runs local, not on accuracy.
• Real-time conversational stacks: OpenAI Realtime API, Hume EVI, Pipecat, LiveKit agents. These own the genuinely hard part of spoken presence — turn-taking, barge-in/interruption, sub-second latency — that a plain request/response text loop cannot fake into feeling alive.
• Avatars (Live2D, VRM). Live2D = 2D rigged (the VTuber standard); VRM = 3D humanoid format from Pixiv/Vket. Both have mature web runtimes, which means the body is solved-enough that the remaining work is art and rigging, not engineering — see the avatar-protocol notes in ch. 25.
• Image gen. SDXL, Flux, SD3 for the base render; ControlNet for pose and composition control; LoRAs to lock a specific character's identity. Together they're how you keep one recognisable face across a thousand generated images — the consistency problem that makes or breaks a visual companion.

→ Part IV of this book.

4.7 Safety & alignment

• Llama Guard, NeMo Guardrails, Granite Guardian. Off-the-shelf moderation classifiers that screen inputs and outputs against a policy before they reach the user or the model. Drop-in, and — like everything in this subsection — primarily a hosted-operator concern; see the framing in the third bullet.
• Jailbreak literature. GCG, PAIR, many-shot jailbreaking — the adversarial-prompt attacks that defeat the guardrails above. Mostly relevant when defending a hosted product you're accountable for; a user "jailbreaking" their own sovereign, locally-run companion is not an attack, it's just using the thing they own.
• Companion-specific safety. Crisis-response prompts, age-verification flows, escalation pathways — the obligations the Character.AI lawsuits (2024–2026) crystallised. Like the jailbreak literature above, these attach to hosted operators who control the service and assume a duty of care; they are not enforcement machinery a user-owned, locally-run runtime can or should impose on its owner. This book treats them as operator duties, distinct from the sovereignty case (→ the duty-of-care discussion in §2 and the ethics chapter, ch. 5). → ch. 22, ch. 41.

4.8 World models and the situation model

The newest gap to close, and the one most often mistaken for memory. A companion needs a live, structured model of the situation she is in right now — not the past (memory, §4.3), not world facts (knowledge, §4.4), but the present state: who and what is present, their current states, the active threads, the time and social context, and what she expects next. In agent theory this is the "B" in BDI (beliefs); most LLM agents leave it implicit, and the result is a companion that is subtly amnesiac between turns — reacting to each signal in isolation, never to a situation.

• Situation models (Kintsch, 1998) → Grounded Situation Models (Mavridis & Roy, 2006). Cognitive science's name for the structured representation a mind builds of "what is going on." GSM made it computational: a sensor-updated "structured blackboard" — a theatrical stage in the agent's mind — fusing linguistic, visual, and proprioceptive evidence into one current picture. The direct ancestor of what a companion needs.
• BDI beliefs (Bratman; Rao & Georgeff). The belief base an agent updates from perception and reasons over — the home the tick loop's "fold events into beliefs" step actually requires (→ ch. 18).
• Temporal knowledge graphs (Zep / Graphiti, 2025–26). The 2026 way to implement it for an LLM agent: an LLM extracts entities and relations from each episode into a bi-temporal graph (valid-time + system-time), so "what was true when" is a first-class query, not an inference over a pile of episodes. It beats vector memory on multi-hop and temporal reasoning (~15 pts on LongMemEval). This is the structured world model made practical — and local-ownable.
• LLM-as-world-model (model-based planning, 2024–26). The pragmatic frontier view: the LLM already contains a broad world model, so you prompt it to roll out "what's likely next / what does this imply" to plan, rather than training a dynamics model. Cheap, native, and the right source of prediction for a companion — and prediction-error (surprise) is a gift of a salience signal.
• Learned latent world models — the other meaning, and a false friend here. JEPA / V-JEPA 2.1 (LeCun), Dreamer, Nvidia Cosmos 3: neural nets that learn an environment's dynamics in latent space and plan by imagined rollout. This is the headline "world models" research of 2026 — and it is built for embodied / physics / video agents, not a chat-and-life companion. It earns a place only if she gets a body in a real or simulated environment (→ ch. 43); for the desktop companion it is the wrong tool, and conflating the two is how you over-build.

The boundary that keeps this from collapsing into §4.3/§4.4: knowledge is timeless and cites a document; memory is past and cites a conversation turn; the world model is present-tense and cites a live, time-stamped belief about the situation now. The companion-specific upgrade is belief-tracking, not just state-tracking — modelling the user's believed state and where it diverges from hers (theory-of-mind / common ground), which is what turns "she remembers facts about me" into "she understands my situation." → ch. 18 (the loop step, prediction and surprise), ch. 19 (the WorldModelStore surface + contract).