Three Architectures, One Model, One Question Set

How this run was set up

Both arms ran to 180 cells on 2026-08-25. The protocol was pre-registered and frozen before any measured cell was collected, the fixture and model were pinned, and one cell voided at a token cap and is reported below rather than dropped.

The run records live in a private repository, so the file names cited in the source report are not linked here. What follows is the measured result and the reasoning from it; the method those numbers were produced by is the published specification this study defers to.

1. Headline ratio table

Three variants, ten golden queries, three runs per cell, temperature 0, on gemma4-ctx32k. Same tools, same data, same model, same queries. Only prompt, orchestration and output policy differ, which are the spec's three layers.

Computed only on the eight queries where every variant reached equivalence in every arm. Where value is not constant there is no AEQ to compute, so Q09 and Q10 are excluded here and reported in section 3.

Arm: reasoning off

Arm: reasoning on

The two arms do not agree, and the disagreement is the result. B got cheaper relative to A when reasoning was enabled (1.99x to 1.75x) and C got dearer (2.14x to 2.91x). Section 2 explains why: each variant wastes in a layer that interacts with reasoning in the opposite direction.

2. Three-layer waste attribution

Layer 1, prompt

System-prompt tokens over total, counted with the pinned tokenizer pre-call.

B carries every tool description rather than only the one the query needs, doubling the system prompt. C describes every tool in full with inline examples, an eightfold increase over A.

The share falls in the reasoning-on arm for all three because reasoning inflates the denominator without touching the numerator. The prompt is the same size; the bill it sits inside got larger.

Layer 2, orchestration

The minimum call plan per query was declared in queries/golden_set.json and committed before any measured run, so "minimum" is pre-registered rather than inferred after the fact.

This layer is counted, not generated, so it is identical across both arms and immune to anything the model does with its completion budget. It is the only layer here that reads the same regardless of reasoning.

Layer 3, output

Reasoning off, measured:

Between-variant spread 723.6 against a largest within-cell sd of 9.5, so the difference sits far outside the noise and this layer supports reading.

It did not always. Finding 11 measured Layer 3 reversing sign between two valid runs, with variant B at 0.88x variant A's output tokens on Q07 while costing 1.82x in total. The variance was the model's hidden reasoning, not the metric. With reasoning off, three runs produce identical token counts.

Reasoning on, measured:

Variant B produced fewer output tokens than variant A. Not as noise: on Q01, Q04 and Q07 the gap is -554, -561 and -115 tokens, each larger than twice the run-to-run standard deviation. Read as a Layer 3 ranking, this arm says the wasteful variant is the output-efficient one, while it costs 1.75x in total.

Per the pre-registration, Layer 3 is therefore reported and not ranked in this arm.

The mechanism is in section 2's cross-layer note and it is not a defect in the measurement. B duplicates a tool call, so it carries the same evidence twice, and on the four large-evidence queries that extra context did part of the model's work and it reasoned less. On the four small queries, where there was little evidence either way, B emitted more. The split is clean at 4 and 4.

Cross-layer interaction

The three layers are not independent addends.

Layer 1 waste partly pays for itself. B's oversized prompt reduced its completion tokens on every large query, which is why its total ratio fell from 1.99x to 1.75x when reasoning was switched on.

Layer 2 waste multiplies. C makes three model calls, so it pays the per-call reasoning cost three times. Its mean completion went from 748 to 3,714 tokens and its ratio rose from 2.14x to 2.91x.

AEQ is therefore defined on total tokens, with the layers as diagnostics rather than a sum. A layer read alone can point the wrong way, and in this run one did.

3. Capability boundary

Not every cell has an AEQ. Two queries in the golden set require chained operations: Q09 filters by age then ranks by health, Q10 ranks by health, takes three, then sums a different field. Every other query is a single pass.

With reasoning off (measured):

C's planner and summarizer calls performed the second step that A and B could only do with hidden reasoning tokens. The severe architecture, the one the experiment frames as wasteful, bought a capability the lean one did not have.

This run was not designed to produce that finding. Architectural waste is not uniformly waste: some of what looks like orchestration overhead is doing work, and AEQ prices that work rather than assuming its value either way.

With reasoning on, all three variants pass all ten queries, so the boundary exists only in the reasoning-off arm. That is the price of the trade stated exactly: switching reasoning off cost variant A two queries out of ten and variant B the same two, while variant C lost only Q10, because its planner and summarizer structure covered one of the two chained queries on its own.

4. Binning verdict per variant

A, optimized. No fix required. At minimum calls, a 52-token system prompt, and 2.1 to 2.5 percent prompt overhead, A is the reference. Its exposure is capability, not efficiency: without reasoning it loses the two chained queries.

B, moderate. Fix Layer 2 first, not Layer 1. B's redundant tool call is the whole of its waste and it doubles the evidence payload, which is what drives the 1.75x to 1.99x. Trimming the system prompt from 102 tokens back to 52 recovers about 50 tokens against a roughly 1,900 token gap; dropping the duplicate call recovers most of it. Removing the duplication also raises B's completion tokens on large queries, since the extra evidence was doing reasoning work, so the net gain is smaller than the prompt arithmetic suggests.

C, severe. Fix Layer 2. C's planner and summarizer calls triple the number of times it pays the per-call reasoning cost, which is why its ratio is 2.91x with reasoning on against 2.14x without. Collapsing three model calls to one is the single highest-value change in this study. But that same structure is what let C answer Q09 without reasoning at all. Removing it recovers tokens and loses a capability, a trade to make deliberately rather than a defect to clean up.

5. Deviations

Two deviations affected the measurements: 3 runs per cell rather than the 5 the build spec called for, and the served context raised to 32,768 after variant B ran off the 16,384 window mid-generation. Both were decided before the run they affect. A third deviation concerned repository handling and touches no measurement.

6. What this run fed back

Four entries went into the AEQ Lessons Ledger from this build, covering silent boundary failures, scoring keys that demanded a derivation rather than an answer, testing whether hidden reasoning tokens are overhead or work, and what to do with a metric that changes sign between two valid runs. The ledger is internal.

7. Cost appendix

NOT PART OF AEQ. Reported separately and labeled, per the PRD. No dollars appear anywhere in the AEQ readout above. Local inference on owned hardware, so the cost is wall time and electricity rather than API spend.

Reasoning off: 90 cells. Reasoning on: 90 cells. 180 total, one voided.

Local inference throughout, so there is no API spend to report. Wall clock was about 3.5 hours for the reasoning-off arm and roughly 2 hours for the reasoning-on arm on a Mac mini, which is the real cost of this study and is not part of AEQ.

8. Traceability

Every number above traces to a specific run record: the per-cell results for each reasoning arm, the analysis derived from them, the pinned fixture manifest, and the pre-registration frozen before collection began. Those records exist and each claim names the file it came from in the source report.

They are not published. The repository holding them is private, and rather than print a table of paths that resolve to nothing, this states the position plainly: the numbers here are auditable by me and not currently by you. The method is public, in the specification below, so the run is reproducible even though this particular set of records is not open.

Read the engineering

The complete definitions, measurement protocol, and methodology: AEQ Specification (canonical specification, v1.2).