Why Domain-Specific Fine-Tuning Fails

When you're new to LLMs, it's easy to think of fine-tuning as a straightforward feature. The intuition goes: feed in company docs, medical terminology, game lore, or legal case law, run a few epochs, and the model learns that knowledge and answers from it. Even the name suggests it — fine-tune, as in taking an already-good model and making minor adjustments with your data.

In practice, it's a far more violent operation.

An LLM is a massive pattern learner trained to predict the probability distribution over the next token. Saying it "understands" text is convenient shorthand, but what's actually happening inside the model is computing a next-token distribution over an enormous space of contextual patterns. By the time a model has been post-trained, its distributions have been painstakingly calibrated across vast amounts of text, instruction, and preference data. Pushing a small domain dataset on top of that is nothing like writing new content in a blank notebook. It's closer to re-tensioning the strings on a finely tuned instrument to match a single piece of music.

Touch one string, and the whole instrument goes out of tune.

LLM fine-tuning is less about adding knowledge to a store and more about re-calibrating the next-token distribution.

What Fine-Tuning Actually Changes

A language model takes a sequence of input tokens and outputs a probability distribution over the next token. Given a context like "The capital of Seoul is," the model scores every candidate token for that position. Applying softmax to those scores gives a probability distribution. Training is the process of nudging parameters so that the probability of the correct token increases.

A simplified picture looks like this:

Context: "The patient's blood pressure is"

Distribution before training:
  "normal"    0.31
  "high"      0.18
  "low"       0.11
  "measured"  0.06
  ...

One batch of domain data:
  "The patient's blood pressure meets the criteria for stage 2 hypertension"

After gradient update:
  "stage"   0.27
  "normal"  0.19
  "high"    0.12
  ...

The problem is that this change is not confined to that single sentence. Transformer parameters are shared across enormous numbers of contexts. Weights that shift when processing "The patient's blood pressure is" also affect "The patient's condition is," "Blood pressure is normal, but," and "programming documents where the word 'stage' appears." The model is not a database that stores knowledge in key-value rows. Boost one pattern, and the surrounding distributions it's entangled with move with it.

This is where the misconception of fine-tuning as a "insert new knowledge" button comes from. Training logs show loss going down. Eval accuracy goes up. Certain prompts produce the desired answers. So it feels like learning has occurred. But that success is usually observable only within a narrow distribution. Rephrase the same question slightly differently and the answer degrades. General query quality drops. Instruction following becomes sluggish. This shows up more with smaller models, and feels more wasteful the better the post-training was on the base model.

If you run enough fine-tuning experiments yourself, the first thing to break isn't the knowledge itself — it's the model's verbal habits. Responses get shorter, or fixate on a particular format, or start mimicking the style of the training dataset in places where the model should instead admit uncertainty. If the domain corpus is written like formal reports, the model hardens into that register. If the QA data repeats "The answer is A" over and over, the model drags that surface pattern into complex reasoning tasks. This is what makes fine-tuning dangerous. You think you're inserting knowledge; the model is re-fitting the entire token pattern space.

The Pattern-Recognizer Perspective

Viewing LLMs as pattern recognizers makes many phenomena less mysterious. The model compresses enormously high-dimensional rules of the form "after this context, these tokens tended to follow." Pre-training covers the rough distribution of web documents, code, books, and papers. Instruction tuning adds patterns for answering questions and following directions. Preference optimization via RLHF or DPO nudges the distribution toward responses humans prefer.

A model that has gone through all this isn't simply in possession of a lot of knowledge. It has also learned behavioral distributions: how long to make a response when asked a question, how to express uncertainty, how much explanation to attach to a code request, whether to show intermediate steps on a math problem. The model quality users actually perceive comes from these behavioral distributions.

Domain fine-tuning drops a narrow, strong signal on top of these finely calibrated distributions. The smaller the dataset, the more biased that signal is. Five thousand hospital consultation logs contain the phrasing patterns and elisions common in clinical settings. Twenty thousand sentences of internal company documents contain internal acronyms, team-specific expressions, and the formatting conventions of particular document templates. From the model's perspective, "domain knowledge" and "the writing habits of whoever created this dataset" arrive as a single undifferentiated thing. It all comes in as patterns to match for next-token prediction.

This is the most common blind spot for newcomers. The model doesn't learn in the semantic units humans attach to content. There's no internal labeling system that automatically categorizes "this item is factual knowledge, this item is style, this item is incidental noise." The loss function only sees the probability of the correct token. So if the format of the domain data is consistent, the model earnestly learns that format too. If every response in the data starts with "Of course," the model learns to say "Of course." If the data has verbose disclaimers, the model learns verbosity.

Why Catastrophic Forgetting Happens

Catastrophic forgetting is the phenomenon where a neural network loses performance on previous tasks as it learns a new one. It's an old problem. McCloskey and Cohen documented it in 1989, describing how connectionist networks in sequential learning showed sudden collapse of existing knowledge — they called it catastrophic interference. The quality degradation seen in LLM fine-tuning belongs to the same family of problems.

Model parameters are a shared resource. Moving parameters to fit new data also changes the capabilities those parameters served before. The same problem that shows up in small MLPs doesn't disappear in transformers with billions of parameters. Scale adds some buffering capacity, but a well post-trained model has already struck a careful balance across many objectives — so even small updates produce visible behavioral changes.

EWC (Elastic Weight Consolidation) is one of the canonical approaches to reducing forgetting by making important parameters harder to move. In their 2017 paper Overcoming catastrophic forgetting in neural networks, Kirkpatrick et al. used Fisher information to estimate which parameters mattered for previous tasks and penalized large deviations in those parameters. The idea is sound. The problem is that real LLM fine-tuning environments are far messier. Task boundaries are blurry, the "previous task" is effectively the entire internet plus all of instruction behavior, and what needs to be preserved is hard to express as a single score.

LoRA doesn't eliminate this problem either. LoRA freezes the original weights and trains only low-rank adapters, improving efficiency. QLoRA trains adapters on top of a quantized base model, dramatically cutting memory costs. Both matter in practice. I reach for LoRA-family methods before full fine-tuning — they're easier to roll back, cheaper to experiment with, and don't directly corrupt the base model.

But at inference time, the adapter is added to the base model's output distribution. If the adapter pushes logits toward particular tokens, the final next-token distribution changes. The fact that the base weights were preserved and the fact that response quality was preserved are two different things. The operational advantage is that you can remove the adapter to get the original model back — but with the adapter on, distribution degradation still has to be measured.

"Domain Knowledge" Is Doing Too Much Work

When people say "domain-specific," several distinct goals get lumped into one phrase. Some want the model to know internal company acronyms. Some want it to respond in a specific JSON schema. Some want it to answer legal queries grounded in case law. Some want it to avoid prohibited phrasing in medical intake conversations. These goals should not all be collapsed into a single fine-tuning strategy.

My breakdown:

What fine-tuning does well is change behavioral patterns. Always responding in JSON, conforming to a specific tool call schema, maintaining a "analyze first, then answer" response structure, adopting a concise customer service tone — these work because they layer a repeatable surface pattern on top of capabilities the model already has. Success rates are high.

Inserting large quantities of new facts is the unfavorable case. Something like "Our pricing plan changed in July 2024" is better served by retrieval. Baking it into model parameters makes it hard to update, obscures the source, and makes hallucination harder to control for similar questions. Above all, the cost of perturbing the entire distribution just to encode that one fact is too high. The moment you try to use an LLM as a knowledge store, operational complexity shoots up.

Why Pattern Injection Like <think>...</think> Works

The fact that certain reasoning models consistently produce <think>...</think> blocks, or follow a structured internal reasoning format before answering, is a good example of what fine-tuning is well-suited for. The reason is simple: this task is far more local and repetitive than memorizing new facts.

Suppose many samples in the dataset have this structure:

User: [problem] ...
Model: <think>
[intermediate reasoning] ...
</think>
[answer] ...

The model learns a strong format pattern: "when a problem comes in, open <think>, write intermediate steps, then answer after </think>." This pattern repeats identically across enormous numbers of inputs. Even when the correct knowledge differs per sample, the wrapper structure is constant. From a next-token prediction standpoint, the signal pushing up the probability of the <think> token is extremely consistent.

JSON output works the same way. When every sample repeats the structure {, "answer", "reason", the model learns those token transition patterns reliably. Tool calling is the same: seeing a user request, selecting a function name, filling in arguments, emitting a termination token — this is a pattern fine-tuning captures well. Because the model already has underlying language and reasoning ability, what we're injecting is closer to "which form to use when expressing that ability."

Domain fact injection is a different story. "Acronym A means B," "Policy C has an exception under condition D," "Product E is only deployed in region F" — these items share weak surface patterns with each other. Each is a separate fact. From a distribution standpoint, it requires many fine-grained adjustments: nudging specific tokens upward in specific contexts. With enough repetition, the model can memorize them. But that memorization conflicts with generalization. In similar contexts the model may pull in the wrong fact, or blend old pre-training knowledge with new fine-tuning knowledge.

So the working distinction — "pattern injection is viable; knowledge injection is risky" — is useful in practice. Don't over-generalize it. With large-scale continued pre-training, rigorous data curation, sufficient compute, and broad evaluation, domain adaptation is achievable. In fields like medicine, law, and code where corpus size is large and style is stable, domain-adaptive pretraining shows real gains. But what startups and product teams typically mean by "fine-tune on our documents" is not that kind of undertaking.

A Falling Loss Does Not Mean a Better Model

When train loss drops cleanly in a fine-tuning run, it feels reassuring. Add a falling eval loss and it feels even better. But if the eval set was drawn from the same data generation process as the training set, the model has been calibrated to that process. Consistent templates, style, question length, and answer vocabulary make loss go down. Real user inputs arrive differently.

To actually see this gap, you need at minimum two eval sets. One is the domain-target eval: does the model know the company acronyms, does it conform to the required schema, does it fill tool call arguments correctly? The other is a preservation eval: cover the things the base model already handled well — general Q&A, coding, summarization, multilingual responses, safe refusals, long-context processing. If fine-tuning raises the first score while dropping the second, the model has failed in product terms.

When reviewing domain fine-tuning results, I look at regression samples before success samples. Success cases are easy to find — ask a question close to the training data and one will almost always appear. Regressions start subtly. A question the model previously answered with "I don't know" now gets a confident wrong answer. Code blocks go unclosed. Korean questions get answered in mixed English. The prompt says "briefly" but the model follows the verbose format from the fine-tune data. When I see regressions like these, I won't ship the model no matter how clean the loss curve looks.

A minimal preservation eval table is non-negotiable:

Models: base vs domain-lora-r16
Sample count: 100–300 per category

Eval categories:
- General instruction following
- Korean Q&A
- Code generation
- Summarization
- Questions that should be refused
- Domain QA
- Output format compliance

Metrics to record:
- Accuracy or pass rate
- Format violation rate
- Hallucination judgment rate
- Mean response length
- 20 manually reviewed regression examples

Mean response length matters more than it looks. If response length drops 30% or doubles after fine-tuning, the model's behavioral distribution has shifted. Changes that accuracy alone won't catch show up in length first. The same applies to format violation rate. If domain QA goes up 5 points but JSON-breaking increases on general instruction following, that model will cause failures on other paths in production.

When Fine-Tuning Is Actually the Right Tool

Arguing against fine-tuning entirely would be wrong in practice — I actually like fine-tuning. I just think the risk-to-cost ratio gets unfavorable the moment you use it as a knowledge injection tool.

Good fine-tuning targets change model behavior in a narrow, well-defined way. Examples:

• Stabilizing tool calls to a specific schema.
• Aligning responses to a product's tone.
• Drilling a specific task's input/output format into the model.
• Compressing few-shot examples that used to live in long prompts into model behavior.
• Making a smaller model mimic a larger model's response style.

Even in these cases, data quality comes before data quantity. Two thousand clean samples beat a hundred thousand bad ones. With instruction-response data in particular, consistency often matters more than raw answer quality. If what you want for a given input type — the length, the format, the refusal style, the tool-call pattern — keeps shifting across samples, the model's behavior will shift too.

Resist the temptation to fix training failures by cranking up the LoRA rank or adding more epochs. Sometimes the problem really is capacity, but more often the objective is poorly defined. If you want factual question answering, attach a RAG pipeline. If you need policies to stay current, build a searchable policy store with validation logic. If you want the model to follow a specific procedure, generate procedure samples and fine-tune on those. Mix all three into one dataset and none of them will be stable.

The Risk Hidden in "Train on Our Data"

When someone says "we can just train on our data" in a product meeting, the first question to ask is: what exactly do you want to change? Factual recall? Response format? Domain terminology intuition? Tool use? Prohibited behaviors? Breaking the goal down this way draws a clear line between what fine-tuning should handle and what it shouldn't.

Factual recall is handled by retrieval and context injection. Response format is a job for prompting, constrained decoding, or SFT. Tool use can be learned from trajectory data. Prohibited behaviors require looking at a rule layer alongside preference optimization. Domain terminology intuition — when you actually have a sizable corpus and a proper evaluation set — warrants considering continued pre-training. Skip this decomposition and fine-tuning looks like a universal fix right up until deployment, when general-capability regression trips you at the finish line.

LLMs are pattern learners. That's not a dismissal — it's probably the most accurate description of why they're so powerful. Language, code, reasoning, conversational norms, tool use: all of it could be compressed into token patterns, and that's what made today's models possible. By the same logic, you can't arbitrarily stuff new patterns in after the fact. The next-token distribution that already exists is a thin equilibrium point where countless capabilities overlap.

If I'm designing a fine-tuning run today, my first document is not a training script — it's a list of failure conditions. I decide upfront which general capabilities, if degraded, will cause me to stop; which format violations are unacceptable; how much domain score improvement justifies accepting some damage. Then I build the dataset. Then I compare base model, prompt-only, RAG, and LoRA against the same evaluation rubric.

Domain fine-tuning is a card you play last. First, check whether retrieval solves it. Check whether prompting is enough. Check whether output constraints can block the bad behavior. Only when you still need to bake a repeated behavioral pattern into the model itself do you reach for fine-tuning — and even then, the goal is safer framed as "make capabilities the model already has surface in a specific pattern," not "teach the model a new world."