# Benchmarks

> Every key of benchmarks/cases.yaml, the exact pass and fail rules, the eight validation gates, hardware cases, how results enter the track record and what the workflow page shows, with the lint code for each mistake.

Source: https://qsolve.omlabs.co/docs/build/benchmarks
Updated: 2026-11-13

A benchmark case is a known-answer input with the result you expect, stored in `benchmarks/cases.yaml`. Qsolve runs every case on `qsolve test`, on submission, and again after publication as ordinary platform runs that build the track record. Each case compares the quantum method with the baseline, your `reference` method's result. The file is required: without it lint reports `E050`. Lint codes are listed in [Lint codes](/docs/build/lint-codes).

## The file

```yaml
cases:
  - name: ring-6-with-chord
    input_file: examples/ring_6.json
    seed: 1
    expect: {max_gap: 0.0}
  - name: noise-model-ring-6
    input_file: examples/ring_6.json
    target: fake:fez@1.6
    seed: 1
    expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 40}

hardware_cases:
  - name: hardware-ring-6
    input_file: examples/ring_6.json
    seed: 1
    target: ibm:ibm_fez
    expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 40}
```

This is from the `max-cut` workflow. The file has two top-level keys, `cases` and `hardware_cases`. Any other top-level key is ignored. A missing `cases` key means zero cases, which fails the `benchmark_cases` and `cases_pass` gates. A misspelled key inside a case or inside `expect` is rejected: the unknown key makes `qsolve test` stop with a validation error, and the platform's gates stage fails. Lint does not check the content of `cases`, only of `hardware_cases`, so run `qsolve test` to find such mistakes.

## Case keys

| Key | Required | Default | Meaning |
|---|---|---|---|
| `name` | Yes | | Text shown in the track record. Uniqueness is not checked, but the track record keeps one current row per name and target, so two cases with the same name and target hide each other: keep names unique. |
| `input_file` | One of the two | | Path, relative to the workflow folder, to a JSON file holding one object. The resolved path must stay inside the workflow folder. |
| `input` | One of the two | | The input object written inline. If both are given, `input` wins and `input_file` is ignored. |
| `params` | No | `{}` | Parameter overrides, for example `{shots: 20000}`. Checked like any run's parameters against the manifest. |
| `seed` | No | `0` | Integer seed of the run. |
| `target` | No | the run's target | For example `fake:fez@1.6`. Without it the case runs on the target `qsolve test` was given, `sim-cpu` by default, and on the platform `sim-cpu`. |
| `expect` | No | all defaults below | What the result must satisfy. |

At submission, before the sandbox gates, every case (`cases` and `hardware_cases`) is checked and the submission is rejected, naming the case, if: it has neither `input` nor `input_file`; `input_file` is absolute, contains `..`, leaves the workflow folder (including through a link) or does not exist or is not a JSON object; the input does not match your input schema; or `params.shots` is above 100,000. One bad case therefore rejects the submission; it never silently skips the others. Put large inputs in `benchmarks/inputs/` and refer to them with `input_file`.

A case that raises before it produces a result is listed as `FAIL <ExceptionType>: <message>`.

## Expectations

Every `expect` key is optional and the object may be empty.

| Key | Type | Default | The case passes when |
|---|---|---|---|
| `max_gap` | number | `0.0` | `gap <= max_gap + 1e-9`. The gap is `(quantum - baseline) / max(abs(baseline), 1e-12)`. A quantum result that beats the baseline has a negative gap and passes. A gap of 0.05 allows a quantum objective 5% worse. |
| `max_abs_error` | number | none | `abs(quantum - baseline) <= max_abs_error`. When set, `max_gap` is ignored entirely. Use it when the objective is large and the interesting difference small (total energies), and for classifiers, where it bounds the difference of two error rates. It is two-sided: a result better than the baseline by more than this also fails. |
| `min_fidelity` | number | none | The mitigated fidelity against the ideal output is at least this, and at least 90% of the quantum method's shots could be scored. Values outside 0 to 1 are not rejected. |
| `min_postselection` | number | none | The share of shots kept by the error checks is at least this. Values outside 0 to 1 are not rejected. |
| `max_two_qubit_gates` | integer | none | No circuit the quantum method sent has more two-qubit gates than this. |

The objective is lower-is-better, so a maximisation problem returns a negative number from `evaluate`; see [Writing the workflow](/docs/build/writing-the-workflow).

### What fails a case whatever `expect` says

The checks run in this order and the first one that applies gives the message. A case fails when:

| Message | Condition |
|---|---|
| `no headline metric: quantum and reference could not be compared` | The quantum method or the reference produced no usable value. |
| `method x failed: ...` | A method's unavailable reason starts with `failed:` (a classical method crashed). A method that is merely unavailable (`MethodUnavailable`) does not fail the case by itself. |
| `method x claimed A, its solution gives B` | A method's reported objective differs from `evaluate(solution)` by more than `1e-6 * max(1, abs(value))`. |
| `quantum method x sent no circuit to the backend` | The quantum method made no dispatch through `ctx`. |
| `no mitigation numbers reported` | `min_fidelity` or `min_postselection` is set and the run reported none. |
| `only N% of the quantum method's shots could be scored against an ideal output ...` | `min_fidelity` is set and coverage is under 90%. Circuits over 20 qubits or with mid-circuit measurement cannot be scored. |
| `fidelity F < B` | The mitigated fidelity is below `min_fidelity`. |
| `post-selection P < B` | The share of shots kept is below `min_postselection`. |
| `no circuit size reported` | `max_two_qubit_gates` is set and no circuit was recorded. |
| `N two-qubit gates > M` | A circuit exceeds `max_two_qubit_gates`. |

Missing `max_gap` or `max_abs_error` fails without a message; the table shows the gap. The same rules decide a case in `qsolve test`, in the platform's gates stage, in the history runs and in hardware runs.

## Comparison with the baseline

Every result carries a gap, the absolute error, and a headline accuracy with a verdict, "Matches the best classical answer" or "Below the best classical answer". The baseline is the `reference` method. Qsolve recomputes every method's objective with your `evaluate` where a method returns a solution; the platform, not your code, produces each number. Classical methods run with a backend that refuses circuits.

You choose one of three metric kinds in the manifest under `metric`. The default is `relative_gap`.

| `metric.kind` | Accuracy | Use for |
|---|---|---|
| `relative_gap` | `1 - max((q - r) / abs(r), 0)`. Only a worse result loses accuracy. | Optimisation with a lower-is-better objective. |
| `relative_error` | `1 - abs(q - r) / abs(r)`, with `stderr / abs(r)` as its plus-or-minus. | Measured quantities such as energies. |
| `error_rate` | `1 - q`, where `q` is a share of mistakes. The plus-or-minus is a 95% Wilson interval for `n_test` cases (the quantum method's `details["n_test"]`). | Classifiers. |

`q` is the quantum objective, `r` the baseline's. A result matches when any of these holds:

- `relative_gap` or `relative_error`: accuracy is at least `1 - tolerance`.
- `relative_gap` or `relative_error`: `abs(q - r)` is at most `tolerance_abs`, in the objective's units.
- `relative_gap` or `relative_error`: the difference is at most twice the quantum method's `stderr`. This counts only while `stderr` is at most a quarter of `abs(r)`.
- `relative_gap`: the quantum objective is no worse than the baseline's.
- `error_rate`: `q <= r + tolerance + tolerance_abs`, or the lower end of the 95% interval of `q` is at most `r + tolerance + tolerance_abs`.

`tolerance` is between 0 and 1 (default 0), `tolerance_abs` is at least 0 (default none), `note` is at most 600 characters and is shown beside the number. Set a tolerance only when exact equality is the wrong standard, for example chemical accuracy of 1.6 mHa, written `tolerance_abs: 0.0016`. The manifest fields are documented in [Manifest](/docs/build/manifest).

This verdict is separate from `expect`: a case can pass its `expect` and still show "Below the best classical answer", and the other way round.

## The validation gates

`qsolve test` and the platform's gates stage run these gates on `sim-cpu` with no network. A case with its own `target` keeps it.

| Gate | Passes when |
|---|---|
| `manifest` | The manifest and both schemas load and validate. |
| `example_inputs` | At least one example exists and every example validates against the input schema. |
| `benchmark_cases` | At least one case is defined. |
| `methods_declared` | The manifest declares at least two methods. |
| `fidelity_checked` | At least one case sets a non-zero `min_fidelity` and runs on a `fake:` or `ibm:` target. |
| `cases_pass` | At least one case ran and every case meets its `expect`. |
| `reproducible` | Re-running the first case that ran, same input, params, target and seed, gives the same result hash, or a result equal up to relative 1e-8 (absolute 1e-12) on every number. Wall-clock method times are ignored. |
| `estimate_honest` | Your `estimate()` is within a factor of two of the circuits and the shots that case sent (ratio from 0.5 to 2.0, each checked only when the run sent some). Passes with a note when you give no `estimate()` or the run sent no circuits. |

`reproducible` and `estimate_honest` are added only when at least one case ran to a result, so a workflow whose every case raises shows six gates. All gates must pass to publish. Gate messages and fixes are in [Troubleshooting](/docs/build/troubleshooting).

## Noise-model cases

Include one case on a noisy target with `min_fidelity` set; the `fidelity_checked` gate needs it. Use `fake:fez@1.6`, a device noise model with errors scaled up by 1.6 (the scale is between 1 and 10). It runs on a CPU and costs nothing. This is from `generic-qubo`:

```yaml
- name: noise-model-qubo-5
  input_file: examples/qubo_5.json
  target: fake:fez@1.6
  seed: 1
  expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 30}
```

`qsolve test --target fake:fez` also satisfies the gate for a `min_fidelity` case without its own target, but the platform always validates on `sim-cpu`, so declare the target in the case.

Do not put `ibm:` targets under `cases`. Validation has no network and no device access, so such a case cannot run and fails. Lint does not flag it. Real-device cases go under `hardware_cases`.

## Hardware cases

A device run is slow, scarce and costs money, so Qsolve never starts one for you and validation ignores them. List the cases you want measured on a real IBM device under `hardware_cases`. This is from `electrolyte-solvent-screen`:

```yaml
hardware_cases:
- name: hardware-water-only
  input:
    basis: sto-3g
    solvents: [water]
  params: {shots: 1000}
  seed: 1
  target: ibm:ibm_fez
  expect: {max_abs_error: 0.02, min_fidelity: 0.97, min_postselection: 0.9}
```

A hardware case has the same keys as a case, and `target` is required.

| Rule | Value | Lint code |
|---|---|---|
| `hardware_cases` must load: valid YAML, keys known, types right | | `E052` |
| `target` matches `ibm:<device>`: letters, digits, `_ . -`, 1 to 60 characters | for example `ibm:ibm_fez` | `E052` |
| At most this many entries | 3 | `E053` |
| The manifest lists `ibm` under `targets.qpu` | | `E054` |
| All hardware cases together, inputs written out, are at most | 600 KB | none: publication stops with "The hardware cases are too large" |
| Each case has `input` or an `input_file` that exists inside the workflow folder | | none: the submission is rejected with a message naming the case |
| Each case's input matches your input schema | | none: the submission is rejected with the schema path that failed |

Lint itself does not check these last two; the platform does, right after lint and before the gates.

Keep them few and small: a case a device can handle at a price you accept.

### How they run

1. Publish the workflow. At publication Qsolve copies each hardware case, with its input file written inline, into the version. Hardware cases of a version are fixed at that moment.
2. Open the workflow page as its developer (admins can too) and choose **Get a quote**. The platform creates one run per case in your account and prices each. Nothing runs until you confirm each quote. Only the latest published version's cases are started. If a previous batch is still being priced or run, the request is refused as in progress.
3. The run needs the target to be in the manifest's `targets` and the device to be available, or it is refused with `target_not_supported` or an availability error.
4. The runs are ordinary runs paid from your own credits, each capped by the platform's per-run limit (50 credits unless the operator changed it).
5. A maintenance pass adds each finished run to the track record, 20 runs per pass, so it can take a minute. A run that failed for platform reasons (`platform_error`, `backend_unavailable`, `run_too_large`, `insufficient_credits`, `unsupported_target`, `cancelled`) says nothing about the workflow and is not recorded. A run that failed because of the workflow (a crash, a timeout) is recorded as failed.
6. The pass or fail uses the same rules as any case.

A newer result of the same case name on the same target replaces the earlier one, which stays as history.

## How results enter the track record

| Source | When | Target | Billed to |
|---|---|---|---|
| Gates stage | On every submission | `sim-cpu` and each case's own target | Nothing recorded; a check only |
| History stage | After publication of a new version | Each `cases` entry, on its own target or `sim-cpu` | The platform's system account, at most 20 credits and 900 seconds per case, three cases at a time |
| Hardware cases | When the developer confirms a quote | Each `hardware_cases` entry | The developer |

The history stage runs each case through the real platform pipeline (estimate, credit hold, sandbox, settlement), so the recorded cost is what a researcher would pay. For each run Qsolve stores the case name, target, device, pass or fail, gap, absolute error, both objectives, shots, run time, credits, the seed, the params and the quality numbers. A case the platform could not run is not recorded and is counted in the submission report. If the history stage fails as a whole, the stage shows as skipped and the version stays published. Resubmitting identical content (the same version and digest) does not run the history stage again when the version already has recorded results, so the track record is never duplicated.

Resubmitting identical content of an existing version changes nothing. A new version starts with its own empty track record.

## What the workflow page shows

| Part | What it contains |
|---|---|
| Track record | Latest result of each case on a `sim-` target for the current version, with accuracy, verdict, cost and circuit numbers. The headline accuracy is the median over these rows. The workflow cards count passed and total cases on `sim-` targets only. |
| Noise-model estimate | Latest results on `fake:` targets, with fidelity and share of shots kept. Shown apart, not in the headline. |
| On quantum hardware | Latest results on every other target (`ibm:`). Shown apart, not in the headline, because the hardware measures the device as much as the workflow. |
| Earlier results | Older results of the same case and target, kept as history. |

Versions and track record: only the current version's `sim-` and `fake:` results are shown. Hardware results of earlier versions stay on the page, labelled with their version, until a newer run of the same case replaces them, because they are scarce. See [Versioning](/docs/build/versioning).

## Lint codes for benchmark mistakes

| Mistake | Code |
|---|---|
| `benchmarks/cases.yaml` missing | `E050` |
| `hardware_cases` unreadable or invalid, or a hardware case without an `ibm:<device>` target | `E052` |
| More than three `hardware_cases` | `E053` |
| `hardware_cases` without `qpu: [ibm]` in the manifest | `E054` |

Everything else in the file is checked by `qsolve test`, not by lint.

## Writing good cases

- Use inputs where you know the answer, or where the reference is exact.
- Cover the range: a tiny case, one near your envelope, a degenerate one. `generic-qubo` has cases with and without its cardinality constraint, up to 18 variables.
- Include a case where the quantum method is expected to do worse, with an honest tolerance. `portfolio-optimization` allows `max_gap: 0.05` and `0.10` on its larger synthetic cases; `max-cut` includes a complete graph, where low-depth QAOA struggles.
- Keep cases quick. The whole gates run, including the second run of the first case, is one sandbox job. Its time limit is your manifest's `resources.timeout_s` raised to at least 300 seconds and capped at 7200; its memory is `resources.memory_mb` capped at 8192.
- For classifiers, use test sets of at least 200 points. On fewer, one point moves accuracy by several percent and the interval is wide. `quantum-kernel-classifier` bounds the error-rate difference with `max_abs_error: 0.06`.
- Set noise-model bounds from measured values and leave a margin; the `electrolyte-solvent-screen` workflow sets its tolerances at about three times the shot noise.