Benchmarks

A benchmark case is a known-answer input with the result you expect, stored in benchmarks/cases.yaml. Qsolve runs every case on qsolve test, on submission, and again after publication as ordinary platform runs that build the track record. Each case compares the quantum method with the baseline, your reference method's result. The file is required: without it lint reports E050. Lint codes are listed in Lint codes.

The file

cases:
  - name: ring-6-with-chord
    input_file: examples/ring_6.json
    seed: 1
    expect: {max_gap: 0.0}
  - name: noise-model-ring-6
    input_file: examples/ring_6.json
    target: fake:fez@1.6
    seed: 1
    expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 40}

hardware_cases:
  - name: hardware-ring-6
    input_file: examples/ring_6.json
    seed: 1
    target: ibm:ibm_fez
    expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 40}

This is from the max-cut workflow. The file has two top-level keys, cases and hardware_cases. Any other top-level key is ignored. A missing cases key means zero cases, which fails the benchmark_cases and cases_pass gates. A misspelled key inside a case or inside expect is rejected: the unknown key makes qsolve test stop with a validation error, and the platform's gates stage fails. Lint does not check the content of cases, only of hardware_cases, so run qsolve test to find such mistakes.

Case keys

At submission, before the sandbox gates, every case (cases and hardware_cases) is checked and the submission is rejected, naming the case, if: it has neither input nor input_file; input_file is absolute, contains .., leaves the workflow folder (including through a link) or does not exist or is not a JSON object; the input does not match your input schema; or params.shots is above 100,000. One bad case therefore rejects the submission; it never silently skips the others. Put large inputs in benchmarks/inputs/ and refer to them with input_file.

A case that raises before it produces a result is listed as FAIL <ExceptionType>: <message>.

Expectations

Every expect key is optional and the object may be empty.

The objective is lower-is-better, so a maximisation problem returns a negative number from evaluate; see Writing the workflow.

What fails a case whatever expect says

The checks run in this order and the first one that applies gives the message. A case fails when:

Missing max_gap or max_abs_error fails without a message; the table shows the gap. The same rules decide a case in qsolve test, in the platform's gates stage, in the history runs and in hardware runs.

Comparison with the baseline

Every result carries a gap, the absolute error, and a headline accuracy with a verdict, "Matches the best classical answer" or "Below the best classical answer". The baseline is the reference method. Qsolve recomputes every method's objective with your evaluate where a method returns a solution; the platform, not your code, produces each number. Classical methods run with a backend that refuses circuits.

You choose one of three metric kinds in the manifest under metric. The default is relative_gap.

q is the quantum objective, r the baseline's. A result matches when any of these holds:

• relative_gap or relative_error: accuracy is at least 1 - tolerance.
• relative_gap or relative_error: abs(q - r) is at most tolerance_abs, in the objective's units.
• relative_gap or relative_error: the difference is at most twice the quantum method's stderr. This counts only while stderr is at most a quarter of abs(r).
• relative_gap: the quantum objective is no worse than the baseline's.
• error_rate: q <= r + tolerance + tolerance_abs, or the lower end of the 95% interval of q is at most r + tolerance + tolerance_abs.

tolerance is between 0 and 1 (default 0), tolerance_abs is at least 0 (default none), note is at most 600 characters and is shown beside the number. Set a tolerance only when exact equality is the wrong standard, for example chemical accuracy of 1.6 mHa, written tolerance_abs: 0.0016. The manifest fields are documented in Manifest.

This verdict is separate from expect: a case can pass its expect and still show "Below the best classical answer", and the other way round.

The validation gates

qsolve test and the platform's gates stage run these gates on sim-cpu with no network. A case with its own target keeps it.

reproducible and estimate_honest are added only when at least one case ran to a result, so a workflow whose every case raises shows six gates. All gates must pass to publish. Gate messages and fixes are in Troubleshooting.

Noise-model cases

Include one case on a noisy target with min_fidelity set; the fidelity_checked gate needs it. Use fake:fez@1.6, a device noise model with errors scaled up by 1.6 (the scale is between 1 and 10). It runs on a CPU and costs nothing. This is from generic-qubo:

- name: noise-model-qubo-5
  input_file: examples/qubo_5.json
  target: fake:fez@1.6
  seed: 1
  expect: {max_gap: 0.0, min_fidelity: 0.97, max_two_qubit_gates: 30}

qsolve test --target fake:fez also satisfies the gate for a min_fidelity case without its own target, but the platform always validates on sim-cpu, so declare the target in the case.

Do not put ibm: targets under cases. Validation has no network and no device access, so such a case cannot run and fails. Lint does not flag it. Real-device cases go under hardware_cases.

Hardware cases

A device run is slow, scarce and costs money, so Qsolve never starts one for you and validation ignores them. List the cases you want measured on a real IBM device under hardware_cases. This is from electrolyte-solvent-screen:

hardware_cases:
- name: hardware-water-only
  input:
    basis: sto-3g
    solvents: [water]
  params: {shots: 1000}
  seed: 1
  target: ibm:ibm_fez
  expect: {max_abs_error: 0.02, min_fidelity: 0.97, min_postselection: 0.9}

A hardware case has the same keys as a case, and target is required.

Lint itself does not check these last two; the platform does, right after lint and before the gates.

Keep them few and small: a case a device can handle at a price you accept.

How they run

1. Publish the workflow. At publication Qsolve copies each hardware case, with its input file written inline, into the version. Hardware cases of a version are fixed at that moment.
2. Open the workflow page as its developer (admins can too) and choose Get a quote. The platform creates one run per case in your account and prices each. Nothing runs until you confirm each quote. Only the latest published version's cases are started. If a previous batch is still being priced or run, the request is refused as in progress.
3. The run needs the target to be in the manifest's targets and the device to be available, or it is refused with target_not_supported or an availability error.
4. The runs are ordinary runs paid from your own credits, each capped by the platform's per-run limit (50 credits unless the operator changed it).
5. A maintenance pass adds each finished run to the track record, 20 runs per pass, so it can take a minute. A run that failed for platform reasons (platform_error, backend_unavailable, run_too_large, insufficient_credits, unsupported_target, cancelled) says nothing about the workflow and is not recorded. A run that failed because of the workflow (a crash, a timeout) is recorded as failed.
6. The pass or fail uses the same rules as any case.

A newer result of the same case name on the same target replaces the earlier one, which stays as history.

How results enter the track record

The history stage runs each case through the real platform pipeline (estimate, credit hold, sandbox, settlement), so the recorded cost is what a researcher would pay. For each run Qsolve stores the case name, target, device, pass or fail, gap, absolute error, both objectives, shots, run time, credits, the seed, the params and the quality numbers. A case the platform could not run is not recorded and is counted in the submission report. If the history stage fails as a whole, the stage shows as skipped and the version stays published. Resubmitting identical content (the same version and digest) does not run the history stage again when the version already has recorded results, so the track record is never duplicated.

Resubmitting identical content of an existing version changes nothing. A new version starts with its own empty track record.

What the workflow page shows

Versions and track record: only the current version's sim- and fake: results are shown. Hardware results of earlier versions stay on the page, labelled with their version, until a newer run of the same case replaces them, because they are scarce. See Versioning.

Lint codes for benchmark mistakes

Everything else in the file is checked by qsolve test, not by lint.

Writing good cases

• Use inputs where you know the answer, or where the reference is exact.
• Cover the range: a tiny case, one near your envelope, a degenerate one. generic-qubo has cases with and without its cardinality constraint, up to 18 variables.
• Include a case where the quantum method is expected to do worse, with an honest tolerance. portfolio-optimization allows max_gap: 0.05 and 0.10 on its larger synthetic cases; max-cut includes a complete graph, where low-depth QAOA struggles.
• Keep cases quick. The whole gates run, including the second run of the first case, is one sandbox job. Its time limit is your manifest's resources.timeout_s raised to at least 300 seconds and capped at 7200; its memory is resources.memory_mb capped at 8192.
• For classifiers, use test sets of at least 200 points. On fewer, one point moves accuracy by several percent and the interval is wide. quantum-kernel-classifier bounds the error-rate difference with max_abs_error: 0.06.
• Set noise-model bounds from measured values and leave a margin; the electrolyte-solvent-screen workflow sets its tolerances at about three times the shot noise.