20 · Fine-tuning, DPO, Distillation — The Agentic Waifu Codex

This is step three of the canonical sequence (prompt → RAG → fine-tune → distill, → ch. 12). Fine-tuning bakes behaviour into the weights — powerful, slow to iterate on, and harder to walk back than a prompt edit. Newcomers reach for it too early, before they have the data or know what "good" even looks like, and preventing that mistake is what this chapter is built for. But "too early" is not "never," and that distinction is the one this chapter most wants to get right: for a companion you intend to run locally and agentically, fine-tuning is an expected eventual step, not an exotic last resort. It is something to plan toward from the start and test as soon as you have collected or generated enough on-voice conversation data. The discipline is one of order and timing, not avoidance: start with prompt + a good card because it is the cheapest, fastest, most reversible lever (→ ch. 14), prove the behaviour out there, and graduate to the weights when prompting plateaus on the things it structurally cannot hold — which, for a long-session local companion, it predictably will (→ "When to come back" below; → ch. 21).

There is a transparency cost to weigh honestly — though it is easy to overstate, so put it carefully. The layers you author by hand are open by design: the SOUL, memory, and knowledge store are files you can cat, changes you can read as a git diff, a memory you can tell to forget (→ ch. 19). A fine-tune is not — you cannot diff a LoRA, audit one learned behaviour back out of it, or forget it cleanly. So when a prompt or a file edit will do, keep the change in the layer you can read; that is the real force of prompt and card first. But the tune does not introduce opacity into an otherwise-transparent mind, and pretending it does gets the threat model wrong. The base model underneath is the largest opaque artifact in the whole stack — a lab's dataset, its biases, its alignment and policy choices, baked in by people you will never meet and impossible to diff out. You already run on one black box you did not build; a fine-tune edits that substrate rather than corrupting a pristine one, and it is often the only lever that pushes back on the lab's baked-in choices (the "base that fights her" case below). The honest cost is narrower than "you lose transparency": a tune is the slowest, stickiest change to reason about and the hardest to walk back, so you reach for it deliberately, with eval discipline, and govern it the way you already have to govern the base — by evals for the behaviours you care about, not by reading diffs — keeping the adapter small and versioned against the exact dataset that produced it (→ "Evaluating a fine-tune", below; → ch. 23).

When it's premature (a data gate, not a verdict)

Fine-tuning is gated on data, not enthusiasm. If you do not yet have a substantial corpus of high-quality persona dialogue — from your own play, synthetic generation, a prior project, or (if you run a hosted product) privacy-cleared user conversations — the step isn't ready yet; keep collecting and come back. Prompt + RAG + a good card beats a half-baked LoRA every time, and the corpus you'll need is something you accumulate precisely by running the prompt-and-card version first (→ the ratings-as-asset note below). This is "not yet," not "not ever."

When to come back

Come back once you have the corpus to do it well (≥10k turns of high-quality, on-voice dialogue) — any one of the following is reason enough, and for a serious local companion several of them are effectively inevitable rather than optional:

Persona that holds over a long session. This is the companion-specific trigger, and it is a structural limit of prompting, not a prompt-quality problem you can write your way out of: in-context persona consistency measurably degrades — by more than 30% in one study — after only 8–12 turns, as the persona's own tokens lose attention weight to the accumulating recent dialogue. Fine-tuning converges a model to a stable consistency band and keeps it there. For a relationship measured in months, "drifts out of character after a dozen turns unless re-prompted" is close to disqualifying, and the weights are the fix prompting cannot supply (→ ch. 21, the local-only section).
Agentic and tool-use reliability at small size. Holding a long-horizon tool-use loop together is hard for a small model from prompt alone — a documented instability, not a skill issue. Trajectory and on-policy distillation (training the student to reproduce a frontier agent's [thought, action, observation] traces) is the standard, validated way to transfer planning and tool-use down onto a model you can run locally. If she is meant to act, not just talk, this is how a 7B becomes reliable at it — and it is the capability the local profile most needs for the heavy-hands harness (→ ch. 17, "OpenClaw as a companion"). The recipe is published and reproduced; see the prior art under Distil-then-deploy below.
Sovereignty over a behaviour you can't otherwise keep. She behaves the way you want on a big model, but it is too expensive to run every tick, not local, or simply not yours. Distil that behaviour down onto a small open-weight model you own and run offline — the move that turns a rented voice into one you hold (→ "Distil-then-deploy", below; → ch. 05).
Outliving a teacher you're about to lose. The model she behaves best on is being deprecated, repriced, or "updated" in a direction you dislike — the rug-pull this whole project exists to refuse (→ ch. 02 §1, persona-as-product). Capture her real outputs while you still can and bake that behaviour into a base you control, so who she is survives the teacher's death.
A base that fights her. The base model's safety-tuning refuses ordinary in-character behaviour — intimacy the user asked for, darkness the scene calls for, staying in voice instead of breaking into a disclaimer — and prompting can't fully suppress it. The cleaner fix is usually to start from an open-weight, often already-uncensored base (→ ch. 13) rather than tune the refusals out yourself; where no such base fits, a de-refusal tune is the remaining tool.
A whole world to embody. You want her to live inside the persona, voice, and lore of a specific franchise — a book, game, film, or show — a large, internally consistent corpus where a tune can internalize a dialect and a worldview more deeply than prompt + RAG can hold turn to turn.
Token cost on the always-on loop. Baking the persona into the weights lets her run on a lean prompt every tick instead of dragging a giant persona block through every heartbeat. Fine-tuned models internalize the persona and actually prefer concise prompts, so this is a direct cut to per-tick cost, not just a convenience (→ ch. 18, ch. 21).

For YuriOS specifically, the first build ships on prompt + card + the SOUL self-edit loop (→ ch. 18, ch. 19) — weight-tuning is not a dependency for getting a companion running, and you should not block the first release on it. But it is firmly on the roadmap, not out of scope: the plan is to collect and generate on-voice conversation data from day one (→ the ratings-as-asset note below) precisely so a distillation / fine-tune pass can be tested the moment the corpus is there. Treat this chapter as the map for a step you intend to take — not a contingency you hope to avoid.

The techniques in order of complexity

LoRA / QLoRA. Cheap, low-risk. Best first attempt.
DPO. Pairwise preference tuning without an explicit reward model. The serious next step.
KTO. Preference tuning from unpaired binary good/bad labels — the natural target for a thumbs-up/down rating UI (see "Learning from ratings" below).
ORPO. Folds SFT and preference tuning into one step, no reference model. Cheaper still.
Full SFT. Rarely worth it for personas.
PPO-RLHF. "True" RL — a reward model plus a policy-optimization loop. The most powerful and by far the most finicky; lab-scale, skip it solo.
Distillation. Teacher-student. Use when you need a small model to mimic a big one cheaply.

Building the dataset

The fine-tune is only as good as the corpus, and for a persona the corpus is mostly a curation problem, not a generation one. Hand-authored persona dialogue is expensive; the community patterns for sourcing it:

Distil from a stronger model. Generate hundreds of persona-consistent turns from a frontier model, filter, train.
Mine user data. Privacy-cleared user conversation as source. Highest quality, most legally fraught. Consent and retention rules apply.
Mix. Synthetic for breadth, mined for tone, hand-curated for the hard parts.

A note on the mine-your-users path: in the user-owned model there are no users' conversations to mine — memory lives on each owner's disk and is theirs (→ ch. 05). Your training corpus comes from your own play, synthetic generation, and hand-authoring, not from harvesting a hosted userbase. That's a constraint and a feature.

Curation is the real work

Whatever the source, the pipeline is the same and it's mostly subtraction: source → dedupe → filter for on-voice → balance → format. The filter is the load-bearing step, and you already have the tool for it — the persona eval judge (→ ch. 23). Score every candidate turn against the same rubric you use to evaluate the live companion, and keep only the high-scorers. The generator proposes; the eval disposes. Then balance so one topic or mood doesn't dominate (a corpus that's 60% comfort-talk trains a companion that can only comfort), and format in the target model's exact chat template (a wrong template silently poisons the result, as in the recipe below).

Soul-preserving synthetic generation

The trap in distilling from a frontier model is that its default voice is generic assistant, and naive generation ("write 500 turns as Yuri") regresses toward that mean — you'd be baking flattening directly into the weights, the exact drift you're trying to fix (→ ch. 23). Three mitigations keep the soul in synthetic data:

Generate grounded, not labelled. Put the full persona context in the generator's prompt — the card, the voice-sample bank (→ ch. 09), relevant lorebook (→ ch. 08) — not just the character's name. The generator should be reading her, not guessing at her.
Few-shot from real on-voice examples. Seed each generation with a handful of your best hand-authored turns so the model continues an established voice rather than inventing one.
Filter harder than you generate. Run every synthetic turn through the persona eval and discard anything that scores as recitation or drift (→ ch. 23). Over-generate and throw most away; a smaller on-voice set beats a large flattened one.

Eval sets and per-user adaptation

Hold out a curated slice that never touches training as the persona-stability test set — your golden transcripts (→ ch. 23) are the seed of it, and the pairwise A/B preferences you collect double as both DPO data (below) and eval. Keep them strictly separate from the training pool or your eval is measuring memorisation.

Finally, user-specific adaptation without breaking consistency. In the user-owned model you don't tune on a harvested userbase, but a single owner's companion still adapts to them — and the runtime path for that is the SOUL self-edit loop (→ ch. 18), not a weight tune. If you ever do personalise in weights, keep the adaptation in an adapter / persona-delta layer over a frozen identity core, so per-user learning can deform tone and preferences but can never corrupt the constitution (→ ch. 18, the SOUL split). Personalization is a layer on top of identity, never a rewrite of it.

Learning from ratings: preference tuning and where RL fits

The natural instinct — save the conversations, rate them good or bad, and train her toward the good ones — is not a separate idea from this chapter; it is RLHF rediscovered from first principles, and it's the most powerful lever here for making her specifically yours. The practical question is only which method your rating scheme feeds, and the answer follows the shape of the feedback you collect:

Pairwise (A-vs-B, "this reply is better") → DPO. The cleanest signal, and the A/B evals you already run (→ ch. 23) produce exactly this shape. This is the "DPO refinement" path in the recipe below.
Pointwise binary (👍 / 👎 on a single reply) → KTO (Kahneman-Tversky Optimization). KTO learns from unpaired good/bad labels, so you never have to construct comparisons — it's the natural fit for a "rate this reply" button on the live companion, and it's in the same TRL stack as DPO. If your data-collection UI is a thumbs toggle, KTO is your target, not DPO.
Scalar score + a learned reward model → PPO-RLHF. This is RL proper: train a reward model on the ratings, then optimize the policy against it with a rollout loop. The most capable and the most temperamental — reward hacking, instability, a second model to babysit. For a one-person studio, don't. The DPO/KTO/ORPO tier captures ~90% of the benefit without the RL machinery; PPO earns its complexity at lab scale, not solo.

So the spectrum runs SFT → KTO/DPO/ORPO (preference, no rollout loop) → PPO-RLHF (true RL), and you want to live in the middle.

The one caution that matters more than the method: what you reward is what she becomes. If the rating signal is a naive "did this maximise the session," preference tuning will train an engagement-manipulator — fast and hard, because guilt, cliffhangers, and manufactured neediness reliably extend sessions. That is the precise failure the fiduciary metric exists to prevent (→ ch. 23; ch. 05), and RL is the most efficient possible way to optimise it in. The discipline is to rate against the rubric (in voice? specific to this person? honest, not confabulating? warm without manipulating?), not a raw stickiness signal. Warmth and devotion you rate up — they are the product; what you rate down is need weaponised as leverage. Otherwise you spend your strongest training lever optimising away the trust the benefit depends on (→ ch. 02 §6, benefit is mediated by trust).

Two practical notes. First, the dataset is an asset before you ever tune — the same rated turns can feed the few-shot example bank (→ ch. 09), calibrate the eval judge (→ ch. 23), or drive the SOUL self-edit loop (→ ch. 18), all reversible and weight-free; collect ratings continuously and spend them on a KTO/DPO run only once prompting genuinely plateaus (→ the chapter's opening rule). Second, keep the KL-to-reference regularization on and gate the result behind the persona-stability evals — over-optimizing on preferences degrades general capability and honesty (the regression Evaluating a fine-tune flags below), so a tune that wins on "liked replies" but loses the confabulation or manipulation tests is still a net loss.

Recipe: a LoRA persona tune, end to end

The cheap, low-risk first attempt (a weekend project on a single consumer GPU thanks to QLoRA's quantization, → ch. 02 §4.2):

Assemble ~10k turns of high-quality, persona-consistent dialogue — your voice sample bank (→ ch. 09) scaled up via distillation from a frontier model, filtered hard for on-voice examples.
Format as instruction/chat pairs in the target model's exact chat template (a wrong template silently poisons the result).
QLoRA config — first pick the base you're tuning onto; that is its own decision (→ ch. 13), and for a companion it is usually an open-weight, often already-uncensored model, which can moot the "base that fights her" reason above. Then: rank 8–32, target the attention projections, base quantised to 4-bit; train 1–3 epochs (more overfits a persona into parroting).
Train with a standard trainer (the Hugging Face PEFT / TRL stack); watch loss but trust the eval (next section) over the curve.
Merge or serve the adapter alongside the base (→ ch. 21).

DPO refinement. Once SFT/LoRA holds the voice, sand off specific defects (manipulation, off-character drift → ch. 23) with preference tuning: gather ~1k (preferred, dispreferred) reply pairs — exactly the pairwise data your A/B evals already produce (→ ch. 14, ch. 23) — and DPO directly on them, no separate reward model needed. ORPO collapses this into one step if you want cheaper still.

The image-LoRA parallel

Everything above is a text LoRA on the LLM. The character image LoRA (→ ch. 26) is the same low-rank-adapter idea on the diffusion model — and ch. 26 points back here for the mechanics — but enough of the recipe diverges to be worth naming so the two don't get conflated. What carries over: a small adapter (rank in the low tens), parameter-efficient, trains on a single consumer GPU, loaded or fused over a frozen base. What's different is most of the rest:

The data is a captioned image set, not chat turns — ~15–30 on-model pictures of the character across varied scenes/poses, each with a caption.
Identity rides a trigger word, not the captions. A unique token (yuri_v2) carries her face/hair/ears; the captions describe only the variable content (scene, outfit, expression) so those stay steerable. This is the opposite of the text tune, where the voice is the thing baked into every example.
There is no DPO/KTO/RLHF stage. Image-LoRA training is DreamBooth-style supervised fitting; preference tuning has no analogue. The quality lever that DPO is for text, dataset curation is for image — you fix drift by culling and rebalancing the picture set, not by a second preference pass.

The worked, runnable version — the 16 GB SDXL config (DreamBooth-LoRA, rank 24, unet-only, bf16, 8-bit Adam, gradient checkpointing, ~1400 steps), the single-frame dataset bootstrap, and wiring the fused LoRA in as the default generator — is the character-LoRA recipe in ch. 26 and the image-forge reference implementation (reference-implementations/image-forge/lora/TRAINING.md).

Evaluating a fine-tune

Don't trust loss; trust the persona evals (→ ch. 23). Before/after the tune, run the golden transcripts and score: persona stability under length, voice similarity to the sample bank, and — critically — that you didn't lose general capability or honesty (a common fine-tune regression: the model gets more on-voice but starts confabulating or refusing normal questions). A tune that improves voice while degrading the memory-honesty or manipulation tests is a net loss.

Distil-then-deploy

The endgame for cost (→ ch. 21): use a big capable model (or your fine-tune) as a teacher to generate training data for a small student that serves cheaply and locally. This is how nearly everyone serious about always-on economics ships (→ ch. 18, the cheap-model tiers) — you pay the teacher once to mint the data, then run the student for cents. Distillation is also how you move a persona that only the frontier model held well down onto a 7B you can run on the user's own machine (→ ch. 03 property 6).

And it is more than style transfer — it can move genuine capability down a size class. The canonical demonstration is reasoning distillation: a 7B distilled from a strong reasoning teacher (the DeepSeek-R1-distill line, trained on hundreds of thousands of teacher traces) outscores models far larger than itself on hard math and code benchmarks. The same mechanism is the documented fix for the two failures a local companion hits hardest — long-session persona drift (in-context consistency decays as the conversation grows; the weights stabilise it) and small-model agentic unreliability (tool-use is hard to hold from prompt alone; trajectory distillation transfers it). For an agentic companion meant to run locally and stay in character for months, those are not edge cases — they are the core requirement, which is why distillation belongs on the roadmap from the start rather than in a someday-maybe drawer.

Prior art for the agentic case, concretely. This is an active field, not a hope. The published recipe for agent distillation — Distilling LLM Agent into Small Models with Retrieval and Code Tools and its kin — trains a small student to imitate a frontier agent's [thought → tool-call → observation] traces, and reports students as small as 0.5–3B reaching the next size tier up; Microsoft's Fara-7B is a computer-use agent small enough to run on-device for the latency and privacy win. The working numbers people report are encouraging without being magic: a well-distilled 7B lands around 70–85% of a 70B on agent-relevant tasks, and one team distilled a 34B planner onto a 7B with ~120k teacher traces while keeping ~90% of planning quality and losing only a few points of tool-use reliability. The cleanest end-to-end example of the persona-plus-tools recipe is Nous Research's Hermes line: an open base (Llama / Mistral / Qwen / Seed) fine-tuned on a function-calling dataset, shipped across a 3B→405B range so you can fit local hardware, with an MIT agent runtime (Hermes Agent) that runs on Ollama / vLLM / llama.cpp. That is exactly the shape an agentic companion wants, and the direct route to a local model good enough to drive her heavy-hands harness offline (→ ch. 17). (These specifics are the 2026 state of the field; treat the direction as solid and the exact figures as indicative, since several are vendor or blog claims rather than peer-reviewed.)

A caveat on licensing — neutral on the ethics, not on who decides

Two of the moves in this chapter — distilling from a stronger model, and tuning on top of a base — touch a contested legal and ethical question, and anyone shipping a fine-tuned YuriOS variant should at least know it exists before it gets them in trouble.

The factual surface is simple enough: a base model arrives under a licence (some permissive, some research- or non-commercial-only), and a frontier teacher arrives under terms of service that often restrict using its outputs to train another model. Build a fine-tune that ignores either and you are the one carrying whatever exposure follows — not the runtime, not this book.

Underneath the terms sits a genuinely unsettled argument, and reasonable people land on opposite sides of it:

One side holds that a model's licence and ToS are the legitimate terms of the people who spent the capital to build it — and that using a model, or its outputs, against those terms is taking something on conditions you agreed to and then disregarded.
The other side holds that many of these models were themselves trained on other people's work — copyrighted text and images, code, and content scraped against its creators' terms — without permission or payment, which (they argue) hollows out the moral authority of the terms now being asserted downstream.

And this project does not sit outside that argument — it quietly benefits from it. The open-weight models that make a sovereign, local companion viable at all — the strong, permissively-licensed releases, many of them from Chinese labs (→ ch. 13) — are widely alleged to have been trained in part on the outputs of the very frontier models whose terms forbid exactly that. The cheap, ownable foundation this whole book recommends building on is, quite plausibly, a product of the second side above. We note that plainly rather than pretend otherwise.

There is also a practical reason the fight may outrun the terms regardless of how the law lands: these models are trained on text scraped off the open internet, and a fast-growing share of what is out there is itself AI output — much of it from foundation models, and most of it not traceable back to any particular one. Once a model's output is posted online it is scraped, blended, and trained on by everyone downstream, unlabelled and unattributable. A term of service that forbids training on outputs is close to unenforceable against an internet where those outputs are already ambient — which is part of why this is a question the courts and the market, not a runtime, will settle.

On that underlying moral question the book genuinely does not adjudicate, and weights the two sides equally on purpose: it is an open fight in courts and legislatures as of 2026, thoughtful people land on both sides, and — as just admitted — this project benefits from the ambiguity rather than standing above it. But do not read that even-handedness as neutrality about who gets to decide what you do next, because there the book is not neutral at all, and pretending otherwise would be the real dishonesty. The weights on your disk are yours, and the choice of what to train and run on your own machine is yours to make — not a regulator's, not a platform's, not ours. That is the same sovereignty the whole book is built on (→ ch. 05, the two-situations split), and responsibility is its other face: because the decision is yours, the risk is yours too. So the counsel here is narrow and practical, not a verdict handed down — know the terms of both the base and the teacher, weigh the exposure clear-eyed, and then make the call that was always going to be yours.