A reviewer that cannot say what is right

Aera answers finance questions for small businesses. You give it a cash-flow file and ask something like "which month breaks me if I hire two people", and a pipeline plans the work, calls analysis tools written in Rust, and writes an answer. The last stage before the user is a critic: a second model pass that reads the question, the draft, and the list of tools that ran, then returns PASS or REVISE.

Adding a critic is the easy part and it is nearly free to make it useless. Four decisions are what separate a reviewer from a rubber stamp, and each of them costs something.

1. It only says REVISE if it also says what is right

The critic returns JSON. A REVISE verdict must carry a non-empty improved_answer. If it comes back with the verdict and an empty field — or the field missing — that is a hard error, not a fallback:

"REVISE" => {
    let improved = parsed.improved_answer
        .filter(|s| !s.trim().is_empty())
        .ok_or_else(|| AeraError::AgentExecutionError(
            "Critic REVISE dedi ama improved_answer boş".into()))?;

The alternative I rejected was the obvious one: if the reviewer objects but offers nothing, ship the draft and log a warning. That path is comfortable and it inverts the whole point. It means the only case where the critic's objection is ignored is the case where the critic could not articulate the objection — which is exactly the case where something strange is happening.

A reviewer is allowed to say "this is wrong" only if it can say what right looks like. Otherwise the pipeline stops.

The cost is real: malformed model output becomes a user-visible failure instead of a silently-degraded answer. I would rather explain an error than discover, months later, that the critic had been failing open the whole time and nobody noticed because the answers looked fine.

2. An unrecognised verdict is an error, not a pass

Same shape, one level up. If the verdict string is neither PASS nor REVISE, the pipeline errors with the value it actually received. It does not coerce, it does not guess from a prefix, it does not default to pass.

Defaulting to pass on an unparsed verdict is how a review stage quietly turns into a latency tax. Everything keeps working, nothing is ever rejected, and the only evidence is a rate you are not measuring.

3. The critic runs at temperature 0.1, and that is a stated position

// Critic deterministik olmalı, yaratıcılık değil tutarlılık hedeflenir.
let config = GenerationConfig {
    temperature: 0.1,
    top_p: 0.7,
    max_output_tokens: 2048,
    ..GenerationConfig::default()
};

The drafting stage wants some range. The reviewing stage does not. A reviewer that returns different verdicts for the same input is not a reviewer — it is a coin flip with a vocabulary, and the fact that it is usually right makes it worse, because you will trust it.

This is the cheapest of the four decisions and the one I see skipped most often. The same generation config gets reused across every stage of a pipeline because it is one struct, and the review stage inherits a temperature that was chosen for prose.

4. It skips the review when there is nothing to review

// Sohbet sorularında Critic pas geçilir
if tools_used.is_empty() && draft_answer.len() < 500 {
    return Ok(CriticVerdict::Pass);
}

Both conditions have to hold: no analysis tool ran, and the draft is short. That is the shape of a greeting or a clarifying question. There is no computation behind it to check, and sending it to a second model costs a round trip and an API call to audit the sentence "which file would you like me to look at?"

This is the one decision of the four that is a heuristic rather than a principle, and it is the one most likely to be wrong. A 499-character answer that cites no tool but confidently states a number would slip through. The reason I am comfortable is the first clause, not the second: an answer with no tool call has no computed figure behind it by construction. The length bound is a second-guess on top, and if this ever bites, that is where it will bite.

What the four have in common

Three of them are the same decision applied in three places: when the reviewer's output is not well formed, fail loudly rather than proceed. That is unremarkable as a principle and surprisingly rare in practice, because every individual instance looks like a small unnecessary strictness, and the aggregate of skipping all of them is a review stage that has never rejected anything.

The fourth is different in kind. It is a performance trade with a guessed threshold in it, and it is written down as a guess.

The general version: in a pipeline where one model checks another, the failure you must design against is not the reviewer being wrong. It is the reviewer being absent while appearing present. Every path that turns a malformed review into a pass is a path toward that, and each one arrives looking like reasonable error handling.