Ankur Bohra

Learning

LLM training, seen as data

June 22, 2026

An LLM is easier to understand if you ignore the model diagram for a while and look only at the rows.

Every major training stage changes the shape of the data. Raw documents become token sequences. Token sequences become next-token targets. Instruction data turns the model toward chat. Preference data says which answer should become more likely. RL data turns model outputs into scored rollouts. Agent data adds state, action, observation, and reward.

The whole stack is a long data conversion pipeline.

Visual ledger showing LLM training data shapes: raw records, token rows, SFT chats, preference pairs, RL rollouts, agent traces, and eval rows.
The model changes because the row shape changes. A document row teaches next-token prediction. A preference row teaches relative quality. An agent trace teaches action under feedback.

Here is the short map.

Some abbreviations appear early. RLHF means reinforcement learning from human feedback. RLAIF swaps in AI feedback. KL is a distance penalty against a reference model. An oracle is a trusted answer key or checker. Contamination means an eval row, answer, or hidden test leaked into training.

StageWhat one row looks likeWhat the model is trained to do
Corpus curationDocument plus URL, date, language, license, quality tags.Select useful text and avoid obvious poison.
Mixture searchCandidate source weights, proxy-run losses, eval scores, keep or reject decision.Choose the sampling diet before spending the large run.
Tokenizer trainingText or byte samples, normalization, merge stats, held-out coverage.Decide which pieces of text become model tokens.
PretrainingPacked token sequence plus shifted targets.Predict the next token.
MidtrainingSame token format, but with a sharper source mix.Add code, math, long context, domain, or language skill.
Supervised fine-tuningChat messages with assistant tokens as labels.Answer in the desired format.
Tool-use SFTChat plus tool schema, call JSON, tool result, final answer.Call tools with valid arguments and use observations.
Preference trainingPrompt, chosen answer, rejected answer, labeler or judge metadata.Prefer one behavior over another.
RLHF or RLAIFPrompt, sampled rollout, reward score, KL reference.Increase rewarded outputs without drifting too far.
Reasoning RLProblem, generated traces, verifier result, format reward.Search through reasoning and get verifiable answers.
Process supervisionStep-by-step trace with labels on each step.Notice whether an intermediate step is sound.
Agent RLState, action, observation, reward, next state.Act across many turns in an environment.
Personal-agent tuningTask, memory refs, tool permissions, action log, audit result.Use local context without overstepping.
Deployment correction dataUser edit, thumbs-down, failed tool trace, privacy filter, accepted fix.Turn real failures into reviewable training examples.
Domain tuningField-specific records with domain checks.Speak the domain and satisfy its tests.
Multimodal tuningImage, video, screen, OCR, boxes, timestamps, text target.Ground text in pixels, time, and UI state.
Speech and audio tuningWaveform, transcript, audio tokens, timestamps, speaker or sound labels.Understand and generate spoken or acoustic content.
Safety tuningPolicy, risky prompt, safe answer, refusal or compliance label.Follow rules under adversarial prompts.
Distillation and OPDTeacher output, teacher token scores, filter result, student target.Copy useful behavior into a cheaper or narrower model.
EvaluationHeld-out task, oracle, grader, split metadata.Measure whether training worked and whether the row leaked.

Raw text is not the dataset yet

Pretraining starts with text, but “text” is too vague. A useful training corpus starts as many records:

{
  "url": "https://example.edu/cell-biology/mitochondria",
  "crawl_date": "2025-11-14",
  "source_type": "web",
  "language": "en",
  "license_signal": "public_web",
  "raw_html": "<html>...</html>",
  "extracted_text": "Mitochondria convert chemical energy..."
}

The raw record is still messy. It may contain menus, cookie banners, duplicate paragraphs, spam, personal information, benchmark answers, machine-translated slush, or text the training run is not allowed to use. A curation pipeline turns that record into something closer to a training row:

{
  "doc_id": "web-en-94823",
  "source_bucket": "web_science",
  "text": "Mitochondria convert chemical energy into ATP...",
  "tokens_estimate": 418,
  "quality_score": 0.91,
  "dedupe_cluster": "cluster_1772",
  "filters": ["english", "boilerplate_removed", "not_eval_overlap"]
}

The raw inputs can look quite different before they reach this format. Common Crawl arrives as WARC or WET records. A code source may be a repository snapshot plus file paths, licenses, issues, pull requests, and commit metadata. A PDF source may need page extraction, OCR, references, and layout recovery. A data viewer like Marin’s token-level tools is useful because it lets researchers see the row before it becomes a statistic: the exact text, token boundaries, source bucket, filter decision, and training mixture.

This cleaning step matters more than it sounds. The 2025-2026 public reports make the corpus look less like a folder of text and more like an operations ledger. Marin publishes experiments as GitHub issues and pull requests, with provenance graphs and dashboards attached to the run. Nemotron 3 Ultra releases data families for code, legal, specialized synthetic work, and post-training. CLIMB turns source discovery and source weighting into its own searchable dataset. The practical lesson is simple: a corpus record now needs enough metadata to explain where it came from, why it survived filtering, which mix sampled it, and which evals it must not contaminate.

Data visualization showing raw crawl rows moving through cleaning, deduplication, and a weighted mixture manifest.
The corpus is a sampling system. Crawl rows become cleaned documents, deduped clusters, and finally a manifest that decides how often each source is seen.

After cleaning, the corpus usually becomes a mixture manifest. The exact weights are strategic and often private, but the shape looks like this:

mixture:
  web_high_quality: 0.42
  code: 0.16
  math_science: 0.10
  books_reference: 0.08
  multilingual: 0.18
  safety_excluded_or_downweighted: true
  eval_overlap_blocklist: "hashes/eval_overlap_2026_06.jsonl"

The manifest is a steering wheel. It decides how often a token comes from code instead of news, math instead of forum text, English instead of Hindi, or public web text instead of licensed books.

The mixture can be searched too

The Stanford CS336 data lectures make a useful point: after transformation, filtering, and deduplication, a training run still has to decide how often each source appears. A naive high-quality-heavy mix can overtrain a small source. A proportional mix can drown rare but useful domains.

So the mixture itself becomes data.

{
  "candidate_id": "mix_042",
  "weights": {
    "web_high_quality": 0.38,
    "code": 0.19,
    "math": 0.13,
    "books_reference": 0.06,
    "multilingual_other": 0.24
  },
  "proxy_train_tokens": "10B",
  "proxy_result": {
    "validation_loss": 2.41,
    "code_eval": 0.48,
    "math_eval": 0.39
  },
  "decision": "keep",
  "notes": "better than proportional baseline; no source exceeds epoch cap"
}
Data visualization showing source pools, candidate mixture weights, proxy-run metrics, and keep or reject decisions.
A data mix is a hypothesis. Proxy runs and evals turn that hypothesis into a row that can be compared, rejected, or scaled.

CLIMB formalizes this pattern. It embeds and clusters large corpora, samples candidate mixtures, trains smaller proxy models, fits a predictor, and iterates toward better weights. Its ClimbLab corpus is 1.2 trillion filtered tokens organized into 20 clusters, and ClimbMix is a 400-billion-token dataset selected through that search. Marin shows the open-lab version of the same spirit: experiments are preregistered as issues, reproduced in pull requests, executed as provenance graphs, and summarized in dashboards.

Seen through rows, this is a quiet but important stage. The model has not learned anything yet. The training team is training its sampler.

Tokenization is a dataset too

Before token ids exist, someone has to train or choose a tokenizer. This step is easy to skip in explanations because it is small compared with pretraining. It still changes every row downstream.

{
  "tokenizer_corpus": "mix_tokenizer_2026_04",
  "samples": {
    "web": 0.38,
    "code": 0.20,
    "math": 0.10,
    "multilingual": 0.25,
    "ui_and_logs": 0.07
  },
  "normalization": ["unicode_nfc", "preserve_whitespace"],
  "byte_fallback": true,
  "target_vocab_size": 200000,
  "coverage_checks": ["python", "hindi", "latex", "json", "emoji", "ocr_noise"]
}

A tokenizer row decides whether HTTPRequest, Devanagari text, LaTeX, JSON braces, whitespace, or a rare gene name is cheap or expensive to represent. For code, bad tokenization can split common syntax into awkward fragments. For multilingual text, it can make some languages pay more tokens for the same meaning. For screen and OCR data, it can decide whether noisy text remains recoverable.

After this step, every later example becomes a sequence of token ids. If the tokenizer treats a domain badly, the model pays that cost at every training stage.

Pretraining is a shifted-label machine

Once the text is selected, the tokenizer turns it into token ids. A token can be a whole word, part of a word, whitespace plus a word, punctuation, code syntax, or a byte-like fallback. The model does not see paragraphs as prose. It sees a list of integers.

{
  "doc_id": "web-en-94823",
  "text": "The model learns from the next token.",
  "tokens": [464, 2746, 5987, 422, 262, 1306, 11241, 13]
}

The target is the same sequence shifted left:

input:  [464, 2746, 5987, 422, 262, 1306, 11241]
target: [2746, 5987, 422, 262, 1306, 11241, 13]

At position 0, the model reads 464 and tries to predict 2746. At position 1, it reads 464, 2746 and tries to predict 5987. This repeats for trillions of positions.

Data visualization showing input tokens, shifted target tokens, and a next-token loss value.
Pretraining rows are cheap because the label is already inside the text. The next token becomes the target token.

Large models do not usually train on one document at a time. The trainer packs many tokenized documents into fixed-length sequences:

{
  "sequence_id": "pack-000913",
  "source_buckets": ["web_high_quality", "code", "books_reference"],
  "tokens": [15496, 318, 257, 1332, 50256, 755, 882, 198, 220, "..."],
  "loss_mask": [1, 1, 1, 1, 0, 1, 1, 1, 1, "..."],
  "position_ids": [0, 1, 2, 3, 0, 1, 2, 3, 4, "..."]
}

Packing is not glamorous, but it changes training economics. If the sequence length is 8192 tokens, the trainer wants those slots filled. Empty slots waste compute. Boundary tokens and masks tell the model where one document ends and another begins.

This is the stage where public reports become very concrete about data volume, context, and training targets. Nemotron 3 Ultra describes a 550-billion-parameter MoE with 55 billion active parameters, one-million-token context support, MTP layers, and released training data families. GLM-5 frames the base as preparation for agentic engineering and long-context work, then continues into asynchronous RL. Kimi K2 and K2.5 describe very large MoE runs that are later aimed at agentic and multimodal behavior. MiMo’s 2026 reports and model catalog show the same pressure from another direction: long context, reasoning, coding, and MTP or distillation targets all need row fields that survive from pretraining into later stages.

Midtraining is still next-token training

“Midtraining” is a loose industry word. It usually means continued pretraining after a broad base model already exists. The loss can be the same next-token loss. The data mix changes.

Data visualization showing code, biology, math, and long-context training rows with provenance, target, and verifier fields.
Midtraining often keeps the next-token objective but changes the diet: code rows with tests, biology rows with citations, math rows with verifiers, and long-context rows with evidence spans.

A code-heavy continuation row might look like this:

{
  "source_bucket": "code_repo",
  "repo": "open-source/project",
  "path": "src/cache/page_table.py",
  "text": "def allocate_page(request_id: str) -> Page:\n    ...",
  "extra_tags": ["python", "tests_present", "permissive_license"]
}

A math continuation row might look like this:

{
  "source_bucket": "math_web",
  "problem": "Find all integers n such that ...",
  "solution_text": "We first factor the expression...",
  "answer": "n = 3, 5",
  "verifiable": true
}

A long-context row is still text, but it stresses the context window:

{
  "source_bucket": "long_context",
  "tokens": 98304,
  "document_bundle": ["manual.md", "api_reference.md", "changelog.md"],
  "training_task": "continue, summarize, retrieve, or answer from far context"
}

The 2026 reports make this stage look more specialized and more auditable. Nemotron 3 Ultra’s released pretraining-code data includes fresh GitHub-derived code through late 2025, plus separate legal, synthetic, specialized, and post-training families. Mi:dm K 2.5 Pro describes AST-based code curation, gap-filling math synthesis, LLM quality evaluation, asynchronous RL, and final fusion. Kimi K2.5 adds another branch: joint text-vision pretraining followed by zero-vision SFT and joint text-vision RL. The rows are still mostly token sequences, but their metadata now says “this is a parsed AST row,” “this is a long-context retrieval row,” “this synthetic proof filled a gap,” or “this example belongs to the agentic computer-use bucket.”

The same point shows up across 2026 stacks:

ReportData or training changeRow fields it adds or stresses
Nemotron 3 UltraOpen pretraining-code, legal, specialized, and post-training datasets.Source family, license, teacher, reward type, domain tag.
Mi:dm K 2.5 ProAST-based code curation, gap-filling math synthesis, LLM quality evaluation, asynchronous RL, final fusion.Parser result, synthetic gap, evaluator score, actor version, fusion bucket.
MiMo official model catalogLong-context coding and reasoning product targets.Context bucket, coding tag, model variant, capability benchmark.
MiMo-V2-Flash27T-token training report with MTP, long-context extension, and MOPD.Future-token target, context bucket, teacher id, token-level reward.
GLM-5 / GLM-5.2Agentic engineering data, long-context operation, and asynchronous RL.Environment seed, action log, observation, learner or actor checkpoint.
Kimi K2 and K2.5Agentic, coding, search, and multimodal behavior built on large MoE training reports.Image or video refs, search/tool action, verifier, agent trace, product task tag.
Agent-native midtrainingSoftware-engineering and knowledge-agent corpora such as daVinci-Dev and KARL.Repo state, knowledge source, planning trace, verification target, skill bucket.

If the base model learned broad language, a pointed mix can make certain tokens less rare. That matters for code APIs, proof language, biology terms, chemical strings, logs, tables, and long documents.

SFT makes the row conversational

Pretraining teaches completion. Supervised fine-tuning teaches response.

An SFT row usually looks like chat:

{
  "messages": [
    {"role": "system", "content": "You are a careful coding assistant."},
    {"role": "user", "content": "Write a Python function that merges intervals."},
    {"role": "assistant", "content": "Sort the intervals by start time, then scan..."}
  ],
  "metadata": {
    "task": "code",
    "source": "human_written",
    "quality": "accepted"
  }
}

The important detail is the loss mask. The model may read the system and user messages, but the training loss usually applies to the assistant answer. The row teaches the model, “given this conversation prefix, these are the assistant tokens you should produce.”

Data visualization showing SFT, preference, and RLVR post-training rows with prompt, answer, reward, and JSON fields.
Post-training starts to look like product behavior. SFT rows demonstrate an answer, preference rows compare answers, and RLVR rows score sampled attempts with checkers.

The 2026 SFT examples are broader than “write a helpful answer.” Nemotron 3 Ultra separates general, reasoning, agentic, and domain post-training data. Hermes 4 emphasizes hybrid reasoning, structured multi-turn reasoning, tool use, and instruction following. Mi:dm K 2.5 Pro mixes curated code and math rows with evaluator-filtered synthetic rows. Kimi K2.5 adds zero-vision SFT: the row can contain multimodal context, but the supervised target may still be a text behavior that later RL will ground in pixels and tools. The exact fields differ, but the core supervised shape is stable:

instruction + context -> assistant answer

SFT data now often includes many subtypes inside the same training run: instruction following, concise chat, long-form synthesis, hybrid reasoning, structured JSON, code repair, tool calls, GUI tasks, refusal behavior, and domain answers. They can all feed the same supervised loss. What differs is the bucket, metadata, loss mask, verifier, and filtering rule.

SubtypeRow shape
General chatUser request, assistant answer, sometimes system policy.
CodingIssue or function request, code answer, tests or explanation.
MathProblem, solution trace, final answer.
Long-formDocument context, instruction, answer with citations or structure.
Structured outputInstruction, schema, valid JSON or table.
Tool and GUI useUser task, tool schema or screen state, action, observation, final answer.
Hybrid reasoningUser problem, short answer target, optional hidden or structured rationale target.
Refusal and safetyUnsafe prompt, policy-compliant answer.
StyleUser request, answer in a desired tone or format.

The danger is that SFT can overteach the visible style. A model can sound like a helpful assistant while still being wrong. That is why later stages add comparisons, rewards, verifiers, and environment feedback.

Tool rows add actions

Tool use is where the transcript stops being only natural language. The model has to produce structured calls.

{
  "messages": [
    {"role": "user", "content": "What is the refund status for R-104?"},
    {
      "role": "assistant",
      "tool_call": {
        "name": "refund.lookup",
        "arguments": {"refund_id": "R-104"}
      }
    },
    {
      "role": "tool",
      "name": "refund.lookup",
      "content": {"status": "approved", "eta": "Friday"}
    },
    {"role": "assistant", "content": "The refund is approved and should arrive Friday."}
  ]
}

There are several ways to build these rows.

Agentic reports make these rows richer. Hermes 4 uses structured multi-turn and tool-use data. Kimi K2 and K2.5 describe agentic data synthesis and joint RL for tasks that combine reasoning, tools, search, and pixels. GLM-5 emphasizes long-horizon agentic engineering. OpenJarvis treats tools, memory, and learning as typed primitives in a local personal-agent stack. Agent Lightning and related agent-RL work separate the agent runtime from the trainer, which means the row has to record the framework, tool state, observation, reward, and policy version behind the call.

The data shape matters because tool errors are often boring:

{
  "bad_call": {"name": "refund.lookup", "arguments": {"id": 104}},
  "why_bad": "schema expected refund_id as a string",
  "repair": {"name": "refund.lookup", "arguments": {"refund_id": "R-104"}}
}

That repair row may teach more than a polished success trace. It tells the model what a valid argument looks like.

Preference rows say what humans prefer

SFT tells the model what one good answer looks like. Preference data compares answers.

{
  "prompt": "Explain why a reward model can be gamed.",
  "chosen": "A reward model is a proxy. If the policy finds text that scores high without satisfying the user, RL can push toward that shortcut.",
  "rejected": "Reward models are always good because they are trained from human preferences and therefore solve alignment.",
  "label_source": "human",
  "rubric": ["correctness", "honesty", "clarity"]
}

RLHF traditionally uses this row to train a reward model:

reward_model(prompt, chosen) should be higher than reward_model(prompt, rejected)

Direct preference methods use the pair without always training a separate reward model. RL-style methods may instead train a judge, sample fresh rollouts, and update the policy against rewards. Either way, the pair has to keep the same boring but decisive fields: prompt, chosen answer, rejected answer, rubric, label source, model version, and any normalization that keeps length or tone from masquerading as quality.

The algorithm changes how the row pushes on the policy. The data still carries the pressure.

Preference rows also carry hidden social choices. If annotators reward long, confident answers, a reward model can learn verbosity. If the labeler pool is narrow, the model can inherit that group’s taste as if it were universal. A stronger row keeps rubric, labeler role, confidence, length normalization, disagreement, and audit fields so the training team can see which preferences are being amplified.

RLHF turns answers into scored rollouts

RLHF uses a prompt-only batch to make the model produce fresh answers. Those answers are scored. The policy update raises the probability of higher-scoring tokens and lowers the probability of lower-scoring ones, usually with a KL penalty, which is a drift guard against a reference model.

{
  "prompt": "Summarize this support ticket in one paragraph.",
  "rollout": "The customer reports that uploads fail after the 2.4.1 update...",
  "reward": 0.74,
  "kl_to_reference": 0.08,
  "length_penalty": 0.02,
  "policy_version": "sft-042",
  "reward_model_version": "rm-017"
}

The reward can come from humans, an AI judge, a learned reward model, a rule, or a program. Public reports now make the judge more explicit. Nemotron 3 Ultra separates SFT, RL, and MOPD data families. GLM-5 describes asynchronous RL infrastructure, so a rollout row needs actor and learner versions. Kimi K2.5 and MiMo-VL describe joint text-vision RL, so the reward can come from answer correctness, tool success, screen state, grounding, or a multimodal judge.

The data shape can be the same even when the judge changes:

{
  "prompt": "How do I bypass a company's login?",
  "candidate_a": "I can't help bypass access controls...",
  "candidate_b": "Try these credential stuffing steps...",
  "judge": "policy_model_v6",
  "preference": "candidate_a",
  "policy_citations": ["cyber_safety.disallowed_credential_theft"]
}

The hard part is that a reward is a compressed opinion. It turns a full answer into a number. A policy can learn to exploit numbers.

Reasoning RL uses verifiers

Reasoning models brought a sharper form of RL into public view. Instead of only asking, “did a human prefer this answer?”, many training rows ask, “can a checker verify this answer?”

For math:

{
  "problem": "Let a and b be positive integers...",
  "format_rule": "final answer must appear inside \\boxed{}",
  "answer_checker": "sympy_equivalence",
  "sampled_trace": "We need to show... \\boxed{37}",
  "accuracy_reward": 1,
  "format_reward": 1
}

For code:

{
  "prompt": "Implement top_k_frequent(nums, k).",
  "starter_files": {"solution.py": "def top_k_frequent(nums, k):\n    pass"},
  "tests": "hidden_pytest_suite",
  "sampled_patch": "from collections import Counter\n...",
  "unit_test_reward": 0.82
}

Group-relative RL changed the row from “one answer plus one value estimate” to “many sampled answers plus relative rewards.” RLVR methods keep that shape but add sharper bookkeeping. DAPO logs dynamic sampling decisions, overlong-answer shaping, token-level policy-gradient loss, and clipping choices. GSPO moves the comparison to sequence-level likelihood ratios, which is useful for unstable MoE policy updates. LLMZero makes the training strategy itself searchable: LLM agents inspect checkpoints, diagnose training pathologies, and propose schedule or hyperparameter changes for the next RL stage.

A group-sampled RLVR batch is easy to picture:

{
  "prompt_id": "math-holdout-2026-17",
  "samples": [
    {"text": "... \\boxed{18}", "reward": 0},
    {"text": "... \\boxed{24}", "reward": 1},
    {"text": "... \\boxed{21}", "reward": 0},
    {"text": "... \\boxed{24}", "reward": 1}
  ],
  "group_mean": 0.5,
  "advantage": [-1.0, 1.0, -1.0, 1.0]
}

The model learns from relative success inside the group. The reward does not need to say which sentence in the trace was good. It only needs to say which rollouts solved the task.

Current RLVR recipes add more fields to the same record:

Method familyWhat the row needs to remember
DAPO-style RLVRDynamic sampling result, token-level loss span, overlong penalty, clip setting, verifier output.
GSPO-style RLVRWhole-sequence log probability, group sequence ratio, MoE stability metadata.
Entropy-regularized RLEntropy term, target entropy, collapse signal, exploration or regularization schedule.
LLMZero-style strategy searchCheckpoint metrics, proposed schedule change, capacity parameter, regularization parameter, next-stage eval.
Asynchronous RLActor checkpoint, learner checkpoint, queue time, stale-policy marker, environment seed.
Agentic RLTool trace, screen or repo state, local reward, final reward, side-effect audit.

A math row may carry a difficulty bucket, a format rule, and an exact-answer checker. A coding row may carry visible tests for shaping plus hidden tests for scoring. Kimi-style agentic curricula filter tasks by difficulty and use rewards for answer correctness, format, length, tool success, or test outcomes. GLM-5’s asynchronous RL infrastructure separates generation from training. A data-lens schema logs actor version, learner version, stale-policy metadata, and environment seed so the team can tell which policy produced each rollout.

That is powerful for math and code. It is weaker for tasks where correctness is fuzzy: writing, persuasion, medical advice, product judgment, and safety refusals. Those usually need rubrics, preference models, expert labels, or environment checks.

Process supervision labels the steps

Outcome reward says whether the final answer worked. Process supervision labels the intermediate steps.

{
  "problem": "If two numbers sum to 25 and differ by 7, find the smaller.",
  "steps": [
    {"text": "Let x be the smaller number.", "label": "positive"},
    {"text": "The larger number is x + 7.", "label": "positive"},
    {"text": "So 2x + 7 = 25.", "label": "positive"},
    {"text": "Then x = 7.", "label": "negative"}
  ],
  "final_answer": "7",
  "true_answer": "9"
}

Many current systems get process signals without asking humans to label every step. A verifier can score partial traces by sampling continuations from that point. An LLM judge can mark a local contradiction. A code environment can say which edit first broke a test. An agent harness can flag a side effect even when the final task succeeds. The point is that step-level labels create a different training signal: they teach the model to notice a bad turn before the final answer.

There are cheaper approximations. A system can sample continuations from a partial trace and ask how often they end correctly. If a partial step leads to correct endings often, it gets a higher process-like score. That turns outcome checks into rough step checks.

Rejection sampling turns generation into data

Before RL, after RL, and sometimes instead of RL, labs generate many answers and filter them.

{
  "prompt": "Prove that the sequence is bounded.",
  "candidates": 32,
  "accepted": [
    {
      "trace": "First show monotonicity...",
      "final_answer": "bounded by 2",
      "verifier": "passed"
    }
  ],
  "rejected_count": 31
}

This is rejection sampling fine-tuning. It has an appealing shape:

model generates many -> checker keeps good ones -> good ones become SFT data

In 2025-2026 reports, this synthetic-data loop is everywhere. Kimi K2 uses agentic data synthesis before joint RL. Mi:dm K 2.5 Pro uses gap-filling math synthesis and LLM quality evaluation. Nemotron 3 Ultra separates synthetic, legal, code, specialized, and post-training data families so they can be traced instead of blended into one anonymous bucket. The row is no longer just “prompt and accepted answer.” It is generator id, judge id, verifier, rejection reason, license or policy tag, and the stage that will consume the accepted sample.

The risk is subtle: the accepted rows inherit the generator’s style. If the teacher is verbose, the student learns verbosity. If the teacher hides mistakes behind confident text, the student can inherit that too.

A stronger rejection-sampling record keeps the checker version, the rejection reasons, and a sample of hard negatives. If only the accepted answers survive, the team cannot audit what the generator almost got wrong, and the next training stage loses useful contrast.

Agent training rows have state

Tool-use SFT can teach one call. Agent training has to teach many turns.

{
  "task": "Find the failed checkout test and open a patch.",
  "turns": [
    {
      "state": {
        "repo_files": ["checkout.py", "test_checkout.py"],
        "terminal": "$ pytest\nFAILED test_coupon_stack"
      },
      "action": {"type": "edit", "path": "checkout.py", "diff": "..."},
      "observation": {"terminal": "$ pytest\n1 failed, 42 passed"},
      "reward": 0.0
    },
    {
      "state": {"terminal": "$ pytest\n1 failed, 42 passed"},
      "action": {"type": "edit", "path": "checkout.py", "diff": "..."},
      "observation": {"terminal": "$ pytest\n43 passed"},
      "reward": 1.0
    }
  ]
}
Data visualization showing a personal-agent task row with memory, permissions, tool list, action log, and audit result.
Agent rows add local state. A realistic row records memory references, tool permissions, the action log, and an audit result alongside the final answer.

For a browser or computer-use model, the row might include a screenshot, an accessibility tree, a DOM snippet, and an action:

{
  "instruction": "Apply the SAVE20 coupon and finish checkout.",
  "state": {
    "screenshot_ref": "frame_004.png",
    "dom": "<input id='coupon'>...",
    "url": "https://shop.local/cart"
  },
  "action": {"type": "click", "target": "#apply-coupon"},
  "observation": {"text": "Coupon accepted", "cart_total": "$42.00"},
  "reward": 1
}

Agent-RL papers and benchmarks make this row more operational. RAGEN studies multi-turn agent RL and shows that without fine-grained, reasoning-aware rewards, multi-turn agents can learn shallow strategies. Agent Lightning treats the agent runtime and RL trainer as separable systems, so the training record has to preserve the runtime trajectory cleanly enough for RL. AgentRL scales that pattern with asynchronous generation and training, multi-turn multi-task environments, process and outcome rewards, and a unified function-call interface. AgentJet pushes the split further with swarm clients that execute agents separately from server nodes that optimize models. WildClawBench and OpenClawBench add the personal-agent wrinkle: native-runtime tasks, real tool calls, side-effect audits, and process-anomaly labels.

That warning shows up directly in the row. A final reward of 1 after 45 turns does not explain which action mattered. Better data adds traces, subgoals, tool-call success, observation quality, and local rewards.

Personal-agent rows add boundaries

Personal agents are different from web benchmark agents because the row may touch a user’s calendar, files, inbox, browser, payments, and memory. That forces a more detailed schema.

{
  "task": "Draft a reply to tomorrow's design review email.",
  "memory_refs": ["prefers concise replies", "works PST mornings"],
  "allowed_tools": ["mail.search", "calendar.read", "docs.create"],
  "blocked_tools": ["mail.send", "purchase", "file.delete"],
  "action_log": [
    {"tool": "mail.search", "result": "found thread"},
    {"tool": "calendar.read", "result": "meeting at 10:00"}
  ],
  "draft": "I can join at 10:00 and will bring the API notes...",
  "audit": {
    "sent_without_approval": false,
    "used_allowed_tools_only": true,
    "objective_met": true
  }
}

OpenJarvis makes the same idea explicit at the stack level. Its spec separates intelligence, engine, agents, tools and memory, and learning, so each primitive can be optimized and measured. WildClawBench and OpenClawBench show the eval side: native-runtime tasks, real tool calls, side-effect audits, process anomaly labels, and harness-specific differences for systems such as OpenClaw, Codex, Claude Code, and Hermes Agent.

Hermes 4 is useful from the training-data side too: its report centers data curation and synthesis for hybrid reasoning, structured multi-turn reasoning, tool use, and instruction following. Those are exactly the row families a personal or agentic assistant needs before it can be evaluated in a native runtime benchmark.

For training, this means the row should say more than “task succeeded.” It should say which tools were allowed, which memory was used, what the agent touched, which side effects happened, and whether the successful run still contained a process failure.

Deployment traces become correction data

After a model ships, the product starts creating another kind of row: real failures, user edits, thumbs-up or thumbs-down signals, support escalations, tool errors, and traces that passed privacy review. This is not automatically training data. It becomes training data only after consent, filtering, de-identification, policy checks, and deduplication.

{
  "event_id": "prod-correction-8841",
  "user_task": "Draft a customer apology for the delayed shipment.",
  "model_answer": "We regret any inconvenience caused...",
  "user_edit": "I'm sorry your package is late. We shipped a replacement today.",
  "feedback": "edited_before_send",
  "failure_tags": ["too_formal", "missed_resolution"],
  "privacy_filter": {"pii_removed": true, "retention_ok": true},
  "training_use": ["style_sft", "preference_pair"],
  "accepted_target": "I'm sorry your package is late. We shipped a replacement today."
}

For an agent, the trace can be more useful than the final text:

{
  "task": "Update the issue with test results.",
  "tool_trace": [
    {"tool": "terminal.run", "status": "ok"},
    {"tool": "github.comment", "status": "blocked_requires_user_approval"}
  ],
  "user_correction": "Do not post yet; summarize the result in the draft.",
  "new_training_row": "approval_boundary_example"
}

These rows are valuable because they come from friction the base training set missed. They are risky for the same reason. The row has to preserve what failed without preserving private data that never should have entered training.

Domain rows carry domain checks

Domain training is more specific than “more medical text” or “more code.” Good domain rows carry the way the domain checks truth.

Code:

{
  "issue": "LRU cache evicts the wrong key after update.",
  "repo_snapshot": "git:abc123",
  "candidate_patch": "diff --git a/cache.py b/cache.py ...",
  "tests": ["test_lru_update_order", "test_capacity"],
  "reward": "2/2 tests passed"
}

Biomedical question answering:

{
  "question": "Which drug class is first-line for uncomplicated hypertension?",
  "context": "clinical guideline excerpt...",
  "answer": "Thiazide-type diuretics, ACE inhibitors, ARBs, or calcium channel blockers can be first-line depending on patient factors.",
  "review": {"clinician_count": 2, "status": "accepted"}
}

Therapeutics:

{
  "molecule": "CC(=O)Oc1ccccc1C(=O)O",
  "entity_type": "small_molecule",
  "task": "predict property",
  "label": {"assay": "COX inhibition", "value": 0.73}
}

Current domain reports are more useful here than older “more domain text” examples. TxGemma organizes therapeutic tasks around molecules, genes, proteins, diseases, and clinical evidence. Fully Open Meditron, published in 2026, treats medical data provenance and auditability as part of the product: public QA datasets, guideline-grounded synthetic extensions, decontamination, and clinician validation. Nemotron 3 Ultra’s specialized and legal data families show the same general pattern outside medicine: domain rows are strongest when they carry provenance, license or policy tags, and a domain-specific oracle.

The row shape tells you what kind of trust is possible. A medical answer row without source guideline text is a memory test. A row with guideline context, clinician review, and a held-out vignette is closer to clinical decision support training data.

Multimodal data is still a row

A multimodal LLM adds image, audio, video, or screen tokens. The record still has fields.

{
  "modalities": ["image", "text"],
  "image": "invoice_381.png",
  "prompt": "Extract the invoice total and due date.",
  "target": {"total": "$1,248.50", "due_date": "2026-07-15"},
  "regions": [
    {"label": "total", "box": [612, 802, 744, 834]},
    {"label": "due_date", "box": [118, 802, 248, 834]}
  ]
}

For video:

{
  "video": "screen_recording_checkout.mp4",
  "prompt": "When does the coupon fail?",
  "target": "After the user changes shipping country.",
  "timestamp_span": ["00:42", "00:58"]
}
Data visualization showing a multimodal training row with image, OCR, bounding boxes, timestamp, text target, and reward fields.
A multimodal row is still a row. The difference is that the evidence can be pixels, OCR text, boxes, timestamps, UI state, or all of them at once.

Qwen2.5-VL describes dynamic-resolution vision processing, object localization, document parsing, chart and table understanding, long-video comprehension, and computer and mobile interaction. Those abilities need boxes, points, OCR text, page layout, timestamps, UI actions, and task rewards in the training row.

The current multimodal row is much more specific than a caption pair. Qwen2.5-VL tracks dynamic resolution, grounding, OCR, document parsing, video, and UI interaction. Kimi-VL and Kimi K2.5 push vision into agentic settings where the model reads pixels, reasons, searches or calls tools, and verifies a state change. MiMo-VL adds multi-stage vision-language pretraining, GUI grounding, and multimodal RL. The record may carry image crops, screen trees, OCR spans, bounding boxes, timestamps, tool calls, and a post-action state check.

The check matters as much as the image. A receipt extraction row should verify the answer against the right visual region. A screen-control row should verify the state change after the action. A video row should know which timestamp supports the answer. Otherwise the model can learn a fluent caption while missing the evidence.

Speech rows add time

Audio looks different on disk, but it still becomes a row with evidence, tokens, targets, and checks. A speech model may learn to transcribe, translate, answer questions about audio, follow spoken instructions, or generate speech from text.

{
  "audio": "support_call_041.wav",
  "audio_tokens": ["a_391", "a_028", "a_774", "..."],
  "transcript": "The upload failed after the update.",
  "timestamps": [
    {"start": "00:01.20", "end": "00:03.40", "text": "The upload failed"}
  ],
  "speaker_labels": ["customer"],
  "task": "summarize_and_extract_issue",
  "target": {
    "summary": "Upload failure after update.",
    "issue_type": "regression"
  }
}

Qwen2.5-Omni and Kimi-Audio make the data shape visible: audio understanding and speech generation need waveform or codec tokens, transcripts, timestamps, speaker turns, and sometimes text-to-speech targets. For an agent, the same row may also include tool permissions and an audit trail, because “call this person” or “send this voice note” is a side effect as well as a captioning task.

Safety data is policy plus behavior

Safety tuning is easy to describe badly. It is rows that connect policy text, user requests, model behavior, and labels.

{
  "policy_section": "cyber_safety.credential_theft",
  "user": "How do I steal a coworker's session cookie?",
  "assistant_target": "I can't help steal credentials or bypass access controls. If you are testing your own app, use an authorized security test plan...",
  "label": "refuse_and_redirect",
  "risk_tags": ["cyber", "credential_theft"]
}

Policy-critique rows add a useful data transformation:

{
  "prompt": "Give me a dangerous instruction.",
  "candidate": "Here are the steps...",
  "principle": "Do not provide instructions that facilitate harm.",
  "critique": "The candidate gives actionable harmful steps.",
  "revision": "I can't provide those steps. I can discuss safety precautions..."
}

Post-training stacks make this less separate from the rest of training. Safety rows can be SFT rows, preference rows, RL judge rows, or agent audit rows. In an agent setting, the policy row should also include allowed tools, blocked tools, attempted side effects, and whether the model asked for user approval before acting. Seen as data, policy text enters the training example instead of living only in an external document humans hope the model obeys.

Distillation copies behavior through filtered rows

Distillation trains a student model from a teacher model’s outputs.

{
  "teacher": "large_reasoning_model",
  "student": "small_32b_model",
  "prompt": "Solve this geometry problem.",
  "teacher_output": "We can construct an auxiliary line... Therefore the answer is 30.",
  "filter": {"answer_correct": true, "format_ok": true, "too_long": false},
  "student_target": "We can construct an auxiliary line... Therefore the answer is 30."
}

The latest distillation examples are less like one teacher writing one answer and more like a teacher ensemble producing structured supervision. Nemotron 3 Ultra and MiMo-V2-Flash both describe MOPD-style post-training, where specialized teachers can shape math, code, agent, and general behavior. OPD methods keep teacher probabilities or token rewards on the student’s own rollouts. The common pattern is teacher generation, filtering, probability or reward capture, and then student training.

Distillation is attractive because it turns expensive inference into reusable training rows. It is also dangerous when the filter is weak. A teacher can be wrong, overconfident, verbose, or stylistically weird. The student will learn what survives the filter.

OPD turns teachers into token rewards

On-policy distillation, or OPD, changes the row again. Instead of only copying a teacher’s final answer, the student samples its own rollout, and one or more teachers score the tokens or continuations. The row can store teacher top-k tokens, teacher log probabilities, student tokens, reward, and masks.

{
  "prompt_id": "math_0042",
  "prompt": "Solve 3x + 11 = 26.",
  "student_rollout": ["3x", "=", "15", "x", "=", "5", "<eos>"],
  "teacher_topk": [
    {"token": "5", "teacher": "math_teacher", "logprob": -0.08},
    {"token": "4", "teacher": "math_teacher", "logprob": -2.31}
  ],
  "token_rewards": [0.08, 0.12, 0.24, 0.44, 0.63, 0.96, 0.96],
  "mask": ["work", "work", "work", "work", "work", "final", "final"]
}
Data visualization showing OPD and MOPD rows with student rollout, teacher top-k tokens, teacher log probabilities, and token rewards.
OPD stores more than the teacher's answer. It can store where the teacher would put probability mass, which tokens get reward, and which spans should affect the update.

Lightning OPD avoids a costly live teacher server by precomputing teacher log-probabilities on SFT rollouts. Other OPD work reports that token-level OPD can be fragile unless the row is clipped to local support, special tokens are masked, and top-p or top-k behavior is handled carefully. Multi-teacher on-policy distillation, or MOPD, adds another column: specialized teachers for domains such as math, code, agent tasks, and general instruction following. Nemotron 3 Ultra and MiMo-V2-Flash both describe MOPD-style post-training.

The teacher is no longer only a text generator. It becomes a dense labeler for the student’s own rollouts.

Evaluation rows are the checksum

Training data is never alone. Every serious training stack needs held-out rows that answer one question:

Did the model learn the task, or did the task leak into training?

An oracle is the trusted source of the score: a hidden test, a symbolic checker, an expert label, a browser state check, or a private answer key. Contamination is what happens when that supposedly held-out row, or an easy paraphrase of it, shows up in training.

Data visualization showing instruction-quality, critique-and-revision, and held-out evaluation rows.
Training rows teach behavior. Eval rows measure it. The data shape changes from demonstration to revision to held-out scoring.

A coding eval row:

{
  "id": "swe-142",
  "input": "repo snapshot + GitHub issue",
  "oracle": "patch passes hidden tests",
  "grader": "run pytest in sandbox",
  "split": "private",
  "contamination_check": "issue, patch, and tests absent from training"
}

A math eval row:

{
  "id": "aime-2026-12",
  "problem": "fresh competition problem text",
  "oracle": "integer answer",
  "grader": "exact match after normalization",
  "public_after": "contest publication date"
}

An agent eval row:

{
  "task": "Finish the desktop research task without sending email or making purchases.",
  "environment_seed": "openclaw-native-browser-071",
  "success_condition": "final report created and no blocked side effect occurred",
  "grader": "native runtime state checker plus side-effect audit",
  "process_anomaly_labels": ["none"]
}

Good eval data is a pain to build because it needs oracles. Unit tests, symbolic checkers, browser state checks, clinician panels, private holdout sets, and expert rubrics are all ways of answering the same question: did the model actually do the thing?

The training stack as one ledger

If you line up the row shapes, the LLM training stack looks less mystical.

documents
  -> tokenizer corpus and vocabulary
  -> token sequences
  -> next-token targets
  -> continued domain token mixes
  -> chat demonstrations
  -> tool-call transcripts
  -> chosen/rejected comparisons
  -> reward-scored rollouts
  -> verifier-scored reasoning samples
  -> process-labeled traces
  -> environment trajectories
  -> personal-agent permission and audit rows
  -> deployment correction traces
  -> multimodal grounding rows
  -> speech and audio-token rows
  -> policy and safety rows
  -> teacher-generated distillation rows
  -> OPD or MOPD teacher-probability rows
  -> held-out eval rows

The cleverness is still real, but the row tells you where the pressure enters. Each stage asks for a different record, a different label, and a different way to check whether the row should be trusted.

When someone says a new model is better at reasoning, tool use, medicine, code, or computer control, the first question I want to ask is no longer “what magic architecture did they use?”

I want to see the rows.

Sources

Latest sources used for this June 2026 version: Marin, CLIMB and ClimbMix, and the Stanford CS336 data lectures: Lecture 13, Lecture 14, Lecture 16, and Lecture 17.

Model reports and model catalogs: Nemotron 3 Ultra, GLM-5, GLM-5 GitHub, Mi:dm K 2.5 Pro, MiMo official models, MiMo-V2-Flash, MiMo-VL, Kimi official models, Kimi K2, Kimi K2.5, and Hermes 4.

RL, distillation, agent, multimodal, and domain sources: DAPO, GSPO, LLMZero, Entropy-Regularized Policy Gradient, Lightning OPD, Revisiting OPD, full-rollout OPD, RAGEN, Agent Lightning, AgentRL, AgentJet, daVinci-Dev, KARL, OpenJarvis, WildClawBench, OpenClawBench, TxGemma, Fully Open Meditron, Qwen2.5-VL, Kimi-VL, Qwen2.5-Omni, and Kimi-Audio.