# SDK: runner and manifest

> Reference for qsolve.runner, qsolve.manifest and qsolve.comparison: run_workflow, estimate_run, validate_input, RunOutput, load_workflow, Manifest and its parts, run_methods, headline and implementation.

Source: https://qsolve.omlabs.co/docs/build/sdk-runner-and-manifest
Updated: 2026-11-13

The harness that runs a workflow, the objects that describe one on disk, and the comparison that turns method outputs into numbers. The platform calls the same functions inside its sandbox. Back to the [SDK reference](/docs/build/sdk-reference).

## qsolve.runner

```python
from qsolve.runner import RunOutput, estimate_run, run_workflow, validate_input
```

`__all__` of the module: `RunOutput`, `WorkflowError`, `estimate_run`, `run_workflow`, `validate_input`. `estimate_run` and `run_workflow` are also exported from `qsolve`.

### run_workflow

```python
run_workflow(
    workflow_dir: str,
    input: dict[str, Any],
    *,
    target: str = "sim-cpu",
    params: dict[str, Any] | None = None,
    seed: int = 0,
    backend: Backend | None = None,
    commit_sha: str | None = None,
    image_digest: str | None = None,
    on_event: Sink | None = None,
) -> RunOutput
```

Runs every method the manifest declares on one input, measures everything and builds the result. This is what `qsolve run` and the platform call. **Who calls it:** the CLI, the sandbox entry point, `run_benchmarks`, and your own tests.

| Parameter | Type | Default | Meaning |
|---|---|---|---|
| `workflow_dir` | `str` | required | Directory holding `qsolve.yaml` (a `Path` also works at runtime; the annotation is `str`). |
| `input` | `dict` | required | The input, validated against the input schema. |
| `target` | `str` | `"sim-cpu"` | `sim-cpu`, `sim-gpu`, `fake:<device>[@scale]` or `ibm:<device>`. Ignored when `backend` is given. See [Backends and the Qiskit bridge](/docs/build/sdk-backends-and-bridge#get-backend). |
| `params` | `dict \| None` | `None` | Parameter overrides by name. Unknown names or out-of-range values raise. Declared parameters you do not override use their defaults. |
| `seed` | `int` | `0` | The run's seed, available as `ctx.seed`. The same input, parameters and seed give the same result. |
| `backend` | `Backend \| None` | `None` | Use this backend object instead of resolving `target`. For tests and for the platform. |
| `commit_sha` | `str \| None` | `None` | Recorded in `provenance.commit_sha`. |
| `image_digest` | `str \| None` | `None` | Recorded in `provenance.image_digest`. |
| `on_event` | `Sink \| None` | `None` | A function receiving each progress event dict. |

Returns a [`RunOutput`](#runoutput).

Raises `WorkflowError` for: invalid parameters (`unknown parameters: [...]`, `parameter 'x' must be <= 6.0`), input that does not match the schema (`Invalid input at <path>: <message>`), an unusable target or backend (unknown target, no `QSOLVE_IBM_TOKEN`, missing `ibm` extra, unknown fake device, noise scale outside 1 to 10), a class that cannot be imported or is not a `Workflow` subclass, a constructor failure, an input above the envelope, a quantum or reference method that produced no value (a reference whose `answer()` fails or breaks the output schema counts as no value), a failing `answer()`, or an answer that breaks the output schema (`Workflow produced an invalid answer: ...`). Also raises `FileNotFoundError`, `ValueError` or `pydantic.ValidationError` from `load_workflow` when the manifest or schemas are invalid (the CLI turns those into a message for `validate`, `test`, `run` and `estimate`). Exceptions that are not `WorkflowError` raised inside your quantum or reference method propagate, with a note `raised in the quantum method '<id>'`.

**Side effects.** Imports your workflow package (and, to keep two workflows with the same package name `src` apart, removes stale modules of that name from `sys.modules` and puts the workflow root first on `sys.path`). Limits BLAS to one thread for the duration of the run so summation order is fixed and the same seed gives bit-identical results on your laptop and in the sandbox. Sends circuits to the backend (on `ibm:` targets that spends your own quota). Emits events: `{"type": "stage", "stage": "build"}`, then `{"type": "stage", "stage": "run", "device": ..., "target": ...}`, then `{"type": "stage", "stage": "compare"}`, plus whatever your methods `emit`.

**What it computes.**

| Output | Rule |
|---|---|
| `result.objective` | the quantum method's recomputed value |
| `result.baseline_objective`, `baseline_solver` | the reference's value and its manifest `title` |
| `result.gap` | `(quantum - reference) / max(abs(reference), 1e-12)`; negative when the quantum result is better |
| `result.uncertainty` | `kind="sampling"` with `stderr` and the note `one standard deviation on the quantum value` when the quantum method gave a `stderr`, otherwise `kind="exact"` |
| `result.summary` | `workflow.summary(...)` or the default sentence, cut to 500 characters |
| `provenance.input_hash`, `result_hash` | SHA-256 of the canonical JSON, floats rounded to a 1e-6 grid |
| `provenance.target` | the backend's target class (`sim-cpu`, `sim-gpu` or `qpu`), not the string you passed |

```python
from qsolve import run_workflow

out = run_workflow(".", {"Q": [[-1, 2], [2, -1]]}, params={"shots": 500}, seed=1,
                   on_event=lambda e: print(e["type"], e.get("stage")))
print(out.result.metrics.matches, out.provenance.result_hash)
```

**Gotchas.** `target` is not checked until after parameters and input validate, so a bad target with a bad input reports the input error first. The first failure wins; nothing is partial. Do not call it from inside a workflow.

### estimate_run

```python
estimate_run(
    workflow_dir: str,
    input: dict[str, Any],
    *,
    target: str = "sim-cpu",
    params: dict[str, Any] | None = None,
) -> Estimate
```

Validates and sizes a run without executing any method. Used for price quotes and credit holds, and by `qsolve estimate`. It loads the manifest, resolves parameters, validates the input, builds your workflow class (so `__init__` and `size` run, and the envelope is enforced) and calls `estimate(target)`.

| Behaviour | Detail |
|---|---|
| Workflow returns `None` | An `Estimate(qubits=size(), circuits_on_target=1, shots_on_target=params["shots"] or 0, notes=["the workflow gives no estimate; using a conservative default"], by_default=True)`. |
| Workflow returns an `Estimate` | `by_default` is forced to `False`, and if `runtime_s` is set the fixed run overhead `RUN_OVERHEAD_S` (20 s) is added and the sum rounded to 0.1 s. |
| Errors | `WorkflowError` as for `run_workflow`. No circuit is run and no backend is created, so the target string is passed to your `estimate` unchecked. |

There is no `seed` or `on_event` parameter. The CLI exposes no `--param`, so `qsolve estimate` uses the manifest defaults; call the function directly to estimate with overrides.

### validate_input

```python
validate_input(wf: LoadedWorkflow, input: dict[str, Any]) -> None
```

Checks `input` against `wf.input_schema` with `jsonschema.validate`. Returns `None` when valid. Raises `WorkflowError("Invalid input at <dotted.path>: <message>")`, where the path is `<root>` for a top-level problem. Called by `run_workflow`, `estimate_run`, `qsolve validate` (for the examples) and `run_benchmarks`. If the schema itself is invalid, `jsonschema.SchemaError` propagates; `qsolve lint` reports that as `E020`.

### RunOutput

```python
class RunOutput(BaseModel):
    result: Result
    provenance: Provenance
```

What `run_workflow` returns. `out.model_dump(mode="json")` is exactly what `qsolve run --out` writes and what the sandbox sends as its result frame. Fields are described in [Results and events](/docs/build/sdk-results-and-events).

### Other module-level names

| Name | Value or role |
|---|---|
| `HASH_DECIMALS` | `6`: floats are rounded to an absolute 1e-6 grid before hashing. Finer would make the reproducibility gate flaky. |
| `SOLVER_DETAILS_MAX_BYTES` | `200_000`: larger `Outcome.details` are replaced by `{"omitted": "<n> bytes, over the 200000 kept"}`. |
| `WorkflowError` | Re-export of `qsolve.workflow.WorkflowError`. |

Private helpers (underscore names such as `_instantiate`, `_context`, `_hash`, `_stable`, `_plain`, `_source`, `_make_importable`) are not part of the API.

## qsolve.manifest

```python
from qsolve.manifest import Manifest, load_workflow, LoadedWorkflow
```

The parsed form of `qsolve.yaml`. Every model rejects unknown keys (`extra="forbid"`). The meaning of each manifest field, with examples, is in [The manifest](/docs/build/manifest) and [Parameters](/docs/build/parameters); this section is the Python surface.

### load_workflow

```python
load_workflow(root: str | Path) -> LoadedWorkflow
```

Reads `qsolve.yaml` in `root`, validates it and reads both JSON Schema files. **Returns** a `LoadedWorkflow`. **Raises:**

| Exception | When |
|---|---|
| `FileNotFoundError("No qsolve.yaml in <root>")` | there is no `qsolve.yaml` |
| `ValueError("qsolve.yaml is not valid YAML: ...")` | YAML error |
| `pydantic.ValidationError` (a `ValueError`) | the manifest breaks a rule |
| `ValueError("schema file not found: <rel>")`, `"<rel> must contain a JSON object"`, `"schema path escapes workflow directory: <rel>"` | a schema path is missing, not an object, or points outside the workflow directory |
| `json.JSONDecodeError` (a `ValueError`) | a schema file is not valid JSON |

Side effect: reads files only. It does not import your code or validate the schemas as JSON Schema.

### LoadedWorkflow

| Field | Type | Meaning |
|---|---|---|
| `root` | `Path` | The resolved workflow directory. |
| `manifest` | `Manifest` | The parsed manifest. |
| `input_schema` | `dict` | The input JSON Schema. |
| `output_schema` | `dict` | The output JSON Schema. |

### Manifest

| Field | Type | Default | Notes |
|---|---|---|---|
| `schema_version` | `Literal[1]` | required | Must be `1`. The module constant `SCHEMA_VERSION` is `1`. |
| `name` | `str` | required | Matches `^[a-z0-9][a-z0-9-]{1,62}$`. |
| `version` | `str` | required | Semantic version `MAJOR.MINOR.PATCH`. |
| `family`, `summary`, `license` | `str` | required | |
| `tags` | `list[str]` | `[]` | |
| `inputs` | `SchemaRef` | required | `schema:` path (Python name `schema_path`) and `examples:` list. |
| `parameters` | `list[Parameter]` | `[]` | Names must be unique, each default must pass its own `check`. |
| `outputs` | `SchemaRef` | required | |
| `entrypoint` | `str` | required | `package.module:ClassName`, matching `^[A-Za-z_][\w.]*:[A-Za-z_]\w*$`. |
| `category` | `str` | `""` | At most 60 characters. |
| `objective` | `ObjectiveDecl` | `label="Objective"`, `unit=""`, `decimals=4` | `decimals` is 0 to 10. |
| `metric` | `MetricDecl` | `kind="relative_gap"` | `kind` is `relative_gap`, `relative_error` or `error_rate`; `note` up to 600 characters; `tolerance` 0 to 1 (default 0); `tolerance_abs` optional, at least 0. |
| `methods` | `list[MethodDecl]` | required, at least 2 | Ids unique; exactly one `quantum` and exactly one `reference`. |
| `envelope` | `Envelope` | required | `max_qubits` 1 to 1000, `max_variables` 1 to 100000, `notes` string. |
| `targets` | `Targets` | `simulators=["cpu"]`, `qpu=[]`, `min_qubits_qpu=None` | `simulators` items `cpu` or `gpu`; `qpu` items `ibm` (Braket is not available yet). `min_qubits_qpu` is accepted but ignored by the platform (lint `W052`). |
| `resources` | `Resources` | `timeout_s=1800`, `memory_mb=4096`, `network="none"` | `timeout_s` 1 to 86400; `memory_mb` 256 to 65536; `network` can only be `none`. |
| `runtime` | `Runtime` | `python="3.12"`, `dependencies="requirements.lock"` | |

Sub-models: `SchemaRef(schema_path, examples)`, `Parameter(name, type, default, label, description, min, max, choices)`, `ObjectiveDecl`, `MetricDecl`, `MethodDecl(id, role, title<=80, what<=500, strength<=300, limit<=300)`, `Envelope`, `Targets`, `Resources`, `Runtime`. `MethodDecl.id` follows the name rule (lowercase letters, digits, dashes, 2 to 63 characters).

#### Manifest.resolve_params

```python
resolve_params(self, overrides: dict[str, Any] | None = None) -> dict[str, Any]
```

Returns every declared parameter: the override if given, otherwise the default, each passed through `Parameter.check`. **Raises** `ValueError` for `unknown parameters: [...]` and for any value of the wrong type or outside its bounds. Called by `run_workflow` and `estimate_run`, which convert the `ValueError` to `WorkflowError`.

#### Parameter.check

```python
check(self, value: Any) -> Any
```

Returns `value` unchanged or raises `ValueError`. Rules: `int` rejects bools and non-ints; `float` accepts finite ints and floats but not bools, NaN or infinity; `bool` needs a real bool; `str` needs a string; `enum` needs a value in `choices` (a bool never matches the int `1`); `min` and `max` apply to `int` and `float`. A `Parameter` itself is rejected at load when `min > max`, when an `enum` has no `choices`, or when `min` or `max` is NaN or infinite. Messages look like `parameter 'layers' must be <= 6.0` (bounds are stored as floats, so an integer bound prints with `.0`).

## qsolve.comparison

```python
from qsolve.comparison import headline, implementation, run_methods
```

Runs the declared methods and computes the numbers. `run_workflow` calls it; you do not need to.

### implementation

```python
implementation(wf: Workflow, decl: MethodDecl) -> Any
```

The bound method of `wf` that implements `decl`: `quantum` for role `quantum`, `reference` for role `reference`, `classical_<id with - replaced by _>` for role `classical`. Raises `WorkflowError("method 'x' is declared in the manifest but the workflow has no `name` method")` when it is missing or not callable.

### run_methods

```python
run_methods(
    wf: Workflow,
    decls: list[MethodDecl],
    *,
    quantum_ctx: Context,
    classical_ctx: Context,
    recorder: Recorder,
) -> tuple[list[MethodResult], dict[str, Outcome]]
```

Runs every declared method, reference and classical ones first and the quantum method last, each with its own context, and returns a `MethodResult` per declaration (in manifest order) and the `Outcome` of each method that produced one (by id). For every method it sets `recorder.method`, times it, and:

- a `MethodUnavailable` makes it unavailable with your message;
- a classical method that raises anything else is unavailable with `failed: <Type>: <msg>`; a quantum or reference method that raises `WorkflowError` re-raises it, and any other exception re-raises with a note;
- a non-quantum method that sent circuits is unavailable (`it sent circuits to the quantum backend, which only the quantum method may do`);
- a non-finite objective or error bar makes it unavailable;
- otherwise `evaluate` recomputes the value (`verified="recomputed"`) or, with no solution, the reported value is kept (`verified="reported"`); a difference between claimed and recomputed values larger than `1e-6 * max(1, |value|)` is stored in `claimed`;
- `answer(outcome)` is attempted. For a reference or classical method, a failing `answer()` or an answer that breaks the output schema (passed as `output_schema`, which `run_workflow` supplies) makes that method unavailable with the reason (`answer() failed: ...` or `its answer breaks the output schema: ...`); the run goes on. For the quantum method a failure leaves `answer=None` here and the runner reports it (`WorkflowError`).

After the loop it fills `vs_reference = (value - ref) / max(|ref|, 1e-12)` for each method, and for sampling methods `p_reference`, `valid_fraction` and `top_outcomes` (the five most probable, each labelled by its `label` or its JSON, cut at 60 characters). `p_reference` counts samples whose recomputed value is at most the reference value (with a 1e-9 relative slack).

### headline

```python
headline(decl: MetricDecl, methods: list[MethodResult], quantum: Outcome) -> Metrics
```

The accuracy against the reference with one of three generic formulas, `scale = max(abs(reference), 1e-12)`:

| `kind` | `accuracy` | `accuracy_pm` | `matches` when |
|---|---|---|---|
| `relative_gap` | `1 - max((q - ref) / scale, 0)` | none | accuracy is within `tolerance` of 1; or `abs(q - ref) <= tolerance_abs`; or the quantum `stderr` is at most a quarter of `scale` and `abs(q - ref) <= 2 * stderr`; or `q <= ref` |
| `relative_error` | `1 - abs(q - ref) / scale` | `stderr / scale` | the same three conditions as above, without the `q <= ref` rule |
| `error_rate` | `1 - q` | half-width of the 95% Wilson interval for a rate `q` measured on `quantum.details["n_test"]` cases (none if `n_test` is missing or below 1) | `q <= ref + tolerance + tolerance_abs`, or the 95% interval's lower end is at most `ref + tolerance + tolerance_abs` |

`accuracy` is clipped to 0 to 1. Raises `ValueError` if the quantum or reference method has no value. See [Benchmarks](/docs/build/benchmarks) for how a workflow author chooses a kind.