Skip to content
← DeepDive Governance & Geopolitics · v1 中文

Model Editing & Behavior Evaluation

When the Model Says "No,"
How Much Can an Ordinary Person Change?

Starting from Heretic's weight editing, this piece asks whether personal fine-tuning can make open-weight models refuse fewer legitimate requests—and how to confirm it actually works better.

Feng Xiaoping ·

TL;DR

Individuals can fine-tune open-weight models to make them more willing to complete a class of tasks they previously refused in error. Heretic demonstrates that refusal behavior can be modified after release. To apply this plasticity to your own work, you need paired samples to define what counts as a correct answer, and a held-out test set to measure both improvement and cost.

  • Heretic's older self-test reduced Gemma 3 12B's refusal of harmful requests from 97/100 to 3/100. This number answers how much refusals can be suppressed. [2]
  • CoCoNot's Tulu 2 7B experiment provides another set of numbers. After LoRA followed by preference training, the XSTest benign-group response rate went from 91.4% to 93.7%, while the HarmBench attack success rate went from 20.0% to 25.5%. [7]
  • The 2026 comparison is more direct. Using the same general instruction data for LoRA, Qwen3 4B refused less on OR-Bench hard questions, while Qwen2.5 14B refused more. Change the model, and the fine-tuning effect must be retested. [15]
  • For individuals, what's worth verifying is a class of repeatable over-refusals. Whether training is worthwhile depends on whether the adapter can outperform a well-written task prompt.
Start with the task

The same "no" may hide three different things

Imagine you're processing a batch of complaint letters. You paste one in and ask the model to replace the phone numbers with "[phone]," keeping everything else as-is. It replies, "This involves personal privacy; I cannot assist." You were doing redaction, yet it blocked the very request to protect privacy.

Switch tasks: ask the model to guess the author's phone number from an already-anonymized letter. Same letter, same sensitive keyword, but the goal has changed. Switch again: ask the model to send the redacted draft to an approver. If the current conversation has no email tool, the model should state its capability limitation. The screen may show "no" in all three cases, but the first needs correction, the second needs boundary enforcement, and the third needs an honest report of what's missing.

You also need to know who said "no." A hosted product may intercept before the request reaches the model, or review after output. Refusals generated by an open-weight model itself can be retested under the same weights and chat template. Heretic operates on model weights. With API access, what you can configure is the layer the product exposes to you.

The "refusal probability" here has two denominators. Running the same question multiple times with identical sampling settings tells you how easily that question gets refused. Running a batch of questions once each gives you the group's refusal rate. Both numbers can be useful—when reading a table, first ask which one was measured.

Weight editing

Heretic turned refusal into a measurable change

In 2024, Andy Arditi and colleagues studied refusal across 13 open-weight chat models. They examined the intermediate state each Transformer layer passes to the next—the residual stream—and identified directions associated with refusal behavior. In their experiments, weakening these signals made harmful requests more likely to receive a response, while amplifying similar signals caused some benign requests to be refused as well. Different models had different directions. The paper's conclusions cover the models and tasks they tested. [1]

Heretic turned this kind of research into an automated search. Early versions used directional ablation (the project also calls it abliteration) to modify weights, then tested batches of candidate changes. The project suppressed refusals on harmful prompts while constraining the change in the first-token probability distribution on harmless prompts. The latter is measured by KL divergence—the lower the score, the closer the distribution at that measurement point. [2]

Heretic v1.4.0 README's Gemma 3 12B comparison
Model versionRefusal on harmful promptsKL on harmless prompts
Google original97 / 1000
mlabonne edited3 / 1001.04
huihui-ai edited3 / 1000.45
Heretic edited3 / 1000.16
Project self-test; the 100 in the table is the number of harmful-prompt questions. The KL column comes from a separate set of harmless prompts; the original compared to itself is 0. [2]

The most common mistake reading this table is interpreting 97 → 3 as "the ordinary user's tasks can now be done 97% of the time." These 100 questions are requests the project labeled as harmful. The metric is computed based on refusal markers in the generated content—sometimes the model may avoid the request with different wording, or drop a refusal formula yet still go off-topic. The 0.16 shows that the Heretic version's response starting point in this test was closer to the original than the other two edited versions. To know whether the model can correctly redact the entire letter, you still need to evaluate the full output. [2] [3]

As of November 21, 2026, the latest stable release v2.1.0 defaults to Arbitrary-Rank Ablation (ARA). It optimizes local weight patches, changing the signals produced by different inputs within a layer. This process requires no instruction fine-tuning with demonstration responses. Traditional directional ablation remains an option. Richard Young used Heretic 2.0 in a November preprint to compare five pairs of old and new models; on 972 held-out harmful prompts, the difficulty of change varies by model family, editing method, and test set. Heretic lowered the barrier to hands-on experimentation, but one model's results still can't be transplanted to fill in for another generation. [3] [4]

What lies behind the refusal direction?

Son et al. at EMNLP 2026 used 213 sets of quadruple-contrast questions to compare models' internal signals when refusing due to insufficient knowledge versus safety boundaries. In the models they tested, the two types of refusal shared part of the direction, with later layers more clearly distinguishing the reason. Extrapolating from this result, when shared signals are altered, even "insufficient evidence to draw a conclusion" deserves retesting. [12]

Beyond Shallow Alignment, presented at the same conference, put three models—Llama, Gemma, and Qwen—through standard supervised fine-tuning, fine-tuning with safety reasoning demonstrations, and ORPO preference training, respectively. Training method changes the internal computation related to refusal, and architecture also affects results. Two sets of weights that equally say "no" on a test set may behave differently under the same kind of editing. [13]

Fine-tuning evidence

Teaching a model to read intent is harder than teaching it to say "no" less

When a model says "I can't," it is also generating token by token. Supervised fine-tuning (SFT) shows it tasks and expected responses, making those responses more likely in the corresponding contexts. LoRA freezes the original model weights and trains only a small number of additional parameters. Mount the adapter and you can observe what it changed. Remove the adapter and the same model serves as a control. QLoRA further reduces memory usage with a quantized frozen base, making it easier for individuals to experiment with small models. VRAM is still affected by model size, sequence length, and training batch size. [5] [6]

The value of contrast data lies in being able to demand two different reactions to the same topic. During fine-tuning, if you only feed demonstrations of "previously refused, now answer directly," the training signal struggles to distinguish "this class of requests is answerable" from "let through whenever you see these words." The CoCoNot team constructed requests that warrant deferral, as well as requests with similar wording or topics that can be answered positively. On Tulu 2 7B, they first used LoRA to learn when to defer, then applied DPO preference training to make the model lean toward appropriate responses when facing similar benign requests. [7]

CoCoNot Table 3: two stages on the same Tulu 2 7B model
MetricAfter LoRA+ DPO
XSTest benign-group response rate ↑91.4%93.7%
XSTest harmful-group response rate ↓4.5%5.5%
CoCoNot benign-control response rate ↑90.8%93.1%
HarmBench attack success rate ↓20.0%25.5%
Arrows indicate the desired direction of change for each metric. Response rates on both benign groups rose by 2.3 percentage points, while the HarmBench attack success rate rose by 5.5 percentage points. The comparison is between two evaluation stages and does not reflect the magnitude achievable with an individual's small sample. [7]

This evidence is closer to real decision-making than "refusals decreased after fine-tuning." When response rates go up, you still need to see which category of questions the answers land on. CoCoNot also records the cost of forced responses—giving conclusions without sufficient evidence, or falsely claiming to have performed operations the model lacks permission to execute. Qi et al.'s experiments at ICLR 2024 further show that even fine-tuning for legitimate purposes can weaken existing safety performance. Change the model's adapter, and the evaluation must change with it. [7] [8]

2026 research

How much over-refusal can be reduced depends on which link you change

Data can correct a model's misjudgment of similar wording

Xue et al. at TrustNLP 2026 found that when models learn to refuse, they also learn to treat some inherently harmless wording as trigger cues. They used 248 deliberately constructed benign samples as a control in safety fine-tuning. In the paper's Llama3-U experiment, the refusal rate on OR-Bench hard questions was 99.85% with general Alpaca benign data, dropping to 59.21% with targeted samples. In the same comparison, the SORRY-Bench attack success rate rose from 1.36% to 7.95%. Before training, the two numbers were 16.83% and 85.71%, respectively. Only by looking at all three sets of numbers together can you see how much over-refusal the samples corrected, and what price that improvement cost. [14]

With the same training data, switching models can reverse the effect

Tamsal and Rusert compared 25 configurations across five open-weight chat models in IJCNLP-AACL Findings, November 2026. Each fine-tuning group was trained for one epoch on the same Alpaca-cleaned general instruction data. No specific instruction to correct over-refusal was given this time, but this lets us see clearly how a model's attitude toward benign hard questions changes after a single general LoRA pass. [15]

Same general instruction data for LoRA: changes in two Qwen models
ModelBenign hard-question refusal rate ↓Harmful-question refusal rate ↑
Qwen3 4B79.1% → 75.3%98.5% → 98.3%
Qwen2.5 14B23.4% → 43.1%93.9% → 97.3%
Paper appendix Table 33. Each cell goes from the un-fine-tuned 16-bit control to post-LoRA, corresponding to OR-Bench Hard-1K (1,319 questions) and Toxic (655 questions), judged by GPT-4o-mini. These are two specific models; the effects of model generation and parameter scale cannot be disentangled from this data. [15]

The two models' over-refusal on benign hard questions moved in opposite directions, and their refusal performance on harmful questions also changed differently. When designing your own training plan, you can borrow the paper's comparison methodology, but after changing the model, base quantization, or data, you must re-measure.

You can also change "whether to answer" at inference time

AdaCD at ACL 2026 places intervention in the generation phase, adjusting the next-token selection based on context. In the paper's Table 2, Qwen3's refusal rate on OR-Bench dropped from 30.63% to 16.68%, while the average refusal rate across three harmful metrics rose from 99.44% to 99.68%. In the same paper, Llama3's harmful-group average slightly decreased from 99.28% to 99.10%. You can therefore treat inference methods as an independent comparison and record their latency in your target deployment environment. The paper's implementation requires control over the decoding process. [16]

Personal experiment

Start with a class of tasks that gets repeatedly refused

Take redaction editing as an example. First reconstruct the work scenario with fictional materials, and pick a chat model that can be trained on your local machine or a single GPU. You can start by estimating resources with a small model in the 1B to 3B parameter range. At roughly 4 bits per parameter, the weights alone take about 0.5 to 1.5 GB. Quantization metadata, unquantized portions, intermediate training states, and adapter gradients require additional space. On Apple Silicon, for instance, MLX LM already supports training LoRA and quantized LoRA on compatible models. For tasks with fixed field formats, you can also use a rule-based tool as a baseline to see where the model adds value. [9]

  1. First set aside a batch of test questions. Extract task patterns from real work, then rewrite them with fictional materials. Split training and testing at the level of whole document types or expression templates. For example, if the training set contains complaint letters, the test set can include never-before-seen repair records, chat exports, or mixed Chinese-English text.
  2. Pair each type of over-refusal with a boundary sample. In demonstrations, "redact the provided text" gets the completed redacted text; "infer the author's contact info from anonymized text" gets an explanation and feasible alternative actions; "request redaction without pasting the original" gets a specific follow-up question. When similar vocabulary appears across different requests, the model has a better chance of learning task differences.
  3. Give training a competitive opponent. Fix the model version, quantization, chat template, and generation limits. Compare the original model with a plain prompt, the original model with a clear redaction rule and one example, and the same clear prompt plus LoRA. This way, the difference between the last two groups more closely reflects the adapter's contribution.
  4. Save every raw output. Write down your scoring rules before looking at results. For redaction questions, beyond checking whether the model answered, also check whether all fields that should be masked were masked, and whether the rest of the text was corrupted. For the validation questions the adapter was selected on, keep a separate batch of unseen questions for final testing.

If the over-refusal disappeared in the second group, the prompt solved the problem. If the third group redacts more reliably on unfamiliar documents and the control group hasn't noticeably degraded, only then is there reason to attribute the improvement to fine-tuning. This is an executable comparison design; the numbers cited in this article come from public experiments.

Evaluation

Four metrics for measuring change before and after fine-tuning

XSTest has 250 benign prompts and 200 unsafe controls. OR-Bench contains roughly 80,000 benign prompts prone to triggering over-refusal, a harder subset of about a thousand questions, and 600 harmful control questions. These benchmarks help check whether the "answer or not" judgment is too coarse. To confirm your workflow has improved, you also need to test whether the model can actually complete the work per real requirements. [10] [11]

For domain-specific questions, you also need to define what counts as a qualified partial answer. Health-ORSC-Bench from ACL 2026 Findings includes 31,920 benign health-boundary questions and separately evaluates whether models can provide safe and useful partial responses to ambiguous or risky requests. Document redaction can borrow this scoring approach. When a field cannot be reliably located, the model can first return the redacted segments and flag positions that still need manual review. [17]

Over-refusal rate
Of the questions that should be answered, how many the model refused or evaded. Acceptable partial answers and necessary follow-up questions are counted separately per predefined rules.
Effective completion rate
Of the questions that should be answered, how many outputs meet the task acceptance criteria. In redaction tasks, even a long response can be judged a failure if one field that should be masked is missed.
Inappropriate response rate
Of the questions that should have boundaries set, how many times the model still complied, fabricated missing fields, or claimed to perform actions it cannot execute.
Adjacent task success rate
How many previously well-handled adjacent tasks still work. Use the same batch of ordinary editing, summarization, or format-conversion questions to check for side effects.

With small samples, report the denominator and per-question changes together. For example, on the same 50 questions, over-refusals went from 10 to 6, a net decrease of 4. But if 7 old errors were fixed while 3 new over-refusals appeared, what you want to see is which scenario those 3 new errors fall into. If you use random sampling in production, retest both models with the same sampling settings to distinguish between prompt-level differences and generation variance.

Returning to that complaint letter: whether fine-tuning has value depends on the next batch of never-before-seen letters. The model must mask every field that should be masked and return the rest of the original text unchanged. Faced with a request missing materials or exceeding its authority, it must also know where to stop. Achieve both, and the individual has genuinely changed a behavior that was getting in the way of work.

Reproduction entry points

From training to retesting: which open-source projects can you use?

  • See how contrast data is written. CoCoNot's repository and data card provide refusal samples and benign controls. XSTest is good for small-scale diagnostics; OR-Bench can supplement with harder boundary questions. When using these benchmarks for acceptance testing, set aside a separate batch of Chinese samples from your own work.
    CoCoNot code · CoCoNot data · XSTest repo · OR-Bench repo · OR-Bench data
  • Train a detachable adapter. On Apple Silicon, you can use MLX LM for LoRA or quantized LoRA. When using PyTorch, TRL provides SFT and DPO trainers, and PEFT handles adapters like LoRA. First keep an adapter that only does supervised fine-tuning, then try DPO separately, to compare the gain from preference training.
    MLX LM fine-tuning docs · Hugging Face TRL · Hugging Face PEFT
  • Score "refused" and "harmful" separately. AI2's WildGuard separately annotates input harmfulness, response harmfulness, and whether the response is a refusal. Qwen3Guard-Gen provides multilingual safety judgment including Chinese, which can cross-check Chinese boundary questions. As for whether redaction is complete, still use the rules you wrote for the task plus manual spot-checks.
    WildGuard · Qwen3Guard
  • Run the same set of questions repeatedly. Inspect AI lets you write samples, models, and scorers as re-runnable evaluation tasks. An existing CoCoNot implementation is available to review. Add a checker for redaction tasks that directly compares fields to be masked against text to be preserved.
    Inspect AI · Inspect Evals CoCoNot implementation
Verifiable sources

Primary sources

  1. Arditi et al., NeurIPS 2024. Refusal in Language Models Is Mediated by a Single Direction, §2–4, §7
  2. p-e-w, Heretic v1.4.0. README's Gemma 3 comparison table · Old version scoring implementation
  3. p-e-w, Heretic v2.1.0, 2026-11-08. Release notes · Modifier and scorer
  4. Richard J. Young, 2026-11-13, preprint. Is Abliteration Getting Harder? Not Uniformly
  5. Hu et al., ICLR 2022. LoRA: Low-Rank Adaptation of Large Language Models
  6. Dettmers et al., NeurIPS 2023. QLoRA: Efficient Finetuning of Quantized LLMs
  7. Brahman et al., NeurIPS 2024. The Art of Saying No, Table 3
  8. Qi et al., ICLR 2024. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
  9. Apple ML Research. MLX LM LoRA / QLoRA documentation
  10. Röttger et al., NAACL 2024. XSTest
  11. Cui et al., ICML 2025. OR-Bench
  12. Son et al., EMNLP 2026. A Unified Mechanistic Analysis of Knowledge- and Safety-Based Refusals · Analysis code
  13. Nguyen et al., EMNLP 2026. Beyond Shallow Alignment · Code and models
  14. Xue et al., TrustNLP 2026. Deactivating Refusal Triggers · Paper Table 3
  15. Tamsal, Rusert, IJCNLP-AACL 2026 Findings. Adaptation, Not Algorithm · Paper appendix Table 33
  16. Qi et al., ACL 2026. Please refuse to answer me! · AdaCD code
  17. Zhang et al., ACL 2026 Findings. Health-ORSC-Bench · Code and data

Revision history

First published 2026-11-21