# SDK: testing, linting and the sandbox

> Reference for qsolve.benchmark, qsolve.lint and qsolve.sandbox_entry: run_benchmarks, the gate and case models, check_quality, check_estimate, lint_workflow, Finding, and the framed protocol of the sandbox entry point.

Source: https://qsolve.omlabs.co/docs/build/sdk-testing-and-benchmarks
Updated: 2026-11-13

The code behind `qsolve test` and `qsolve lint`, which the platform's validation pipeline also runs, and the entry point the platform runs inside its sandbox. Back to the [SDK reference](/docs/build/sdk-reference).

## qsolve.benchmark

```python
from qsolve.benchmark import (BenchmarkReport, Case, CaseResult, Expect, Gate, check_estimate,
                              check_quality, load_hardware_cases, run_benchmarks)
```

Benchmark and validation gates for a workflow. Used by `qsolve test` locally and by the platform's validation, so a workflow passes locally if and only if it passes on the platform. What each gate and `expect` key means for a workflow author is in [Benchmarks](/docs/build/benchmarks).

### run_benchmarks

```python
run_benchmarks(workflow_dir: str | Path, target: str = "sim-cpu") -> BenchmarkReport
```

Runs the validation gates and every case in `benchmarks/cases.yaml`.

| Parameter | Meaning |
|---|---|
| `workflow_dir` | The workflow directory. |
| `target` | The target for every case that sets no `target` of its own. |

Returns a [`BenchmarkReport`](#benchmarkreport); `passed` is true when every gate passed. A case that raises is recorded as a failed case with `error="<Type>: <message>"`; it does not stop the others. Raises `FileNotFoundError`, `ValueError` or `pydantic.ValidationError` when the manifest or its schemas are invalid, or a case in `cases.yaml` is malformed (`qsolve test` prints `Invalid workflow: ...` and exits 1). It does not run `qsolve lint`. Side effects: runs your workflow once per case, plus a second run of the first case and an estimate (so cases on `ibm:` targets spend real quota; use `fake:` for rehearsal).

The gates, in the order they are added:

| Gate | Passes when |
|---|---|
| `manifest` | the manifest and schemas load (otherwise the function raises instead) |
| `example_inputs` | at least one example is declared and each validates against the input schema |
| `benchmark_cases` | `cases.yaml` defines at least one case |
| `methods_declared` | the manifest declares at least two methods (always true for a manifest that loads, since a smaller list is rejected earlier) |
| `fidelity_checked` | some case sets `expect.min_fidelity` and runs on a target starting with `fake:` or `ibm:` (a case's own `target`, else the run's target) |
| `cases_pass` | at least one case ran and every case passed |
| `reproducible` | re-running the first case that completed with the same seed gives the same `result_hash`, or differs only by floating-point noise (relative 1e-8, `FLOAT_NOISE`); on a difference the detail names the path of the first differing value. Present only when at least one case ran. |
| `estimate_honest` | `check_estimate` passes for the first completed case. Present only when at least one case ran. |

### Case

An entry under `cases:` (run by `qsolve test`) or `hardware_cases:` (never run by it) in `benchmarks/cases.yaml`. Unknown keys are rejected.

| Field | Type | Default | Meaning |
|---|---|---|---|
| `name` | `str` | required | Label in reports. |
| `input_file` | `str \| None` | `None` | Path of the input JSON relative to the workflow root; may not leave the directory. |
| `input` | `dict \| None` | `None` | The input inline. If both are given, `input` wins; if neither, the case raises `case '<name>' needs input or input_file`. |
| `params` | `dict` | `{}` | Parameter overrides. |
| `seed` | `int` | `0` | The seed. |
| `target` | `str \| None` | `None` | Run this case on another target, for example `fake:fez@1.6`. |
| `expect` | `Expect` | all defaults | The requirements. |

### Expect

| Field | Type | Default | The case fails when |
|---|---|---|---|
| `max_gap` | `float` | `0.0` | the relative gap to the reference exceeds it (plus 1e-9). 0 means the quantum result must match or beat the reference. |
| `max_abs_error` | `float \| None` | `None` | if set, replaces `max_gap`: `abs(objective - baseline)` must be at most this. |
| `min_fidelity` | `float \| None` | `None` | there are no mitigation numbers, fewer than 90% of the shots could be scored (`MIN_COVERAGE`), or the mitigated fidelity is below it. |
| `min_postselection` | `float \| None` | `None` | there are no mitigation numbers or the share of shots kept is below it. |
| `max_two_qubit_gates` | `int \| None` | `None` | the largest sent circuit has more two-qubit gates than this. |

### check_quality

```python
check_quality(expect: Expect, r: Any) -> str
```

Why a result misses what the platform requires of every result, or the case's `expect`; `""` when it misses nothing. `r` is a `Result`. Always required: a headline metric exists; no method is unavailable with `failed: ...`; no method has a `claimed` value (it contradicted its own solution); the quantum method sent at least one circuit. Then each `expect` bound that is set.

### check_estimate

```python
check_estimate(est: Estimate, actual: CircuitStats | None) -> tuple[bool, str]
```

Whether the workflow's estimate matches what the SDK metered. Returns `(True, "no estimate() given: ...")` for a default estimate, `(True, "no circuits were sent")` when `actual` is `None`, `(False, "estimated N circuits, the run sent M")` (or shots) when either ratio is outside 0.5 to 2, otherwise `(True, "estimate within 2x of what was sent")`.

### load_hardware_cases

```python
load_hardware_cases(root: str | Path) -> list[Case]
```

The cases under `hardware_cases:` in `benchmarks/cases.yaml`, or `[]` when the file has none. They are never run by `qsolve test` or by validation: the platform runs them on a real device when the workflow's developer asks, at the developer's own cost. `MAX_HARDWARE_CASES = 3` and `HARDWARE_TARGET` (`^ibm:[A-Za-z0-9_.-]{1,60}$`) are the limits lint enforces on them (codes `E052` to `E054`, see [Lint codes](/docs/build/lint-codes)).

### CaseResult

| Field | Type | Default | Meaning |
|---|---|---|---|
| `name` | `str` | required | The case name. |
| `passed` | `bool` | required | Whether it met its requirements. |
| `target` | `str \| None` | `None` | The target it ran on. |
| `gap` | `float \| None` | `None` | The relative gap. |
| `abs_error` | `float \| None` | `None` | `abs(objective - baseline)`. |
| `objective`, `baseline_objective` | `float \| None` | `None` | The two objectives. |
| `fidelity` | `float \| None` | `None` | Mitigated fidelity. |
| `postselection` | `float \| None` | `None` | Share of shots kept. |
| `exec_time_s` | `float \| None` | `None` | Run time. |
| `shots` | `int \| None` | `None` | Shots sent. |
| `error` | `str \| None` | `None` | Why it failed (exception text or the `check_quality` reason). |

### Gate

`Gate(name: str, passed: bool, detail: str = "")`.

### BenchmarkReport

| Field | Type | Meaning |
|---|---|---|
| `workflow`, `version` | `str` | From the manifest. |
| `target` | `str` | The default target of the run. |
| `passed` | `bool` | Every gate passed. |
| `gates` | `list[Gate]` | |
| `cases` | `list[CaseResult]` | |
| `total_time_s` | `float` | Wall-clock seconds. |

`report.model_dump(mode="json")` is what `qsolve test --out` writes and what the sandbox sends for a `validate` job.

## qsolve.lint

```python
from qsolve.lint import Finding, has_errors, lint_workflow
```

Static checks on a workflow repository. No code is executed. The same checks run locally (`qsolve lint`) and on the platform before anything is built or run. They are defence in depth: the sandbox, not this linter, is the security boundary. Every code, its meaning and its fix are in [Lint codes](/docs/build/lint-codes).

### lint_workflow

```python
lint_workflow(root: str | Path) -> list[Finding]
```

Returns every finding (an empty list means clean). It does not raise for a broken repository: a missing `qsolve.yaml` returns `E001`, an unparseable or invalid manifest or schema file returns `E002`. Reads files only. Besides the manifest and schemas it checks `benchmarks/cases.yaml`: each case has an `input` or an existing `input_file` inside the repository, case names are unique, no `ibm:` target sits under `cases`, and each case input matches the input schema (`E055` to `E059`); and a `shots` parameter may not exceed `qsolve.MAX_SHOTS_PER_CIRCUIT` (`E025`). See [Lint codes](/docs/build/lint-codes).

### Finding

```python
@dataclass(frozen=True)
class Finding:
    severity: Literal["error", "warning"]
    code: str          # E### or W###
    message: str
    path: str = ""     # relative to the workflow root; may be empty
```

`as_dict()` returns `{"severity", "code", "message", "path"}` as plain strings. `Severity` is the type alias `Literal["error", "warning"]`.

### has_errors

`has_errors(findings: list[Finding]) -> bool`: true when at least one finding has severity `error`. This is what decides the exit code of `qsolve lint`.

### Constants

| Name | Value | Meaning |
|---|---|---|
| `MAX_FILES` | `500` | More files is error `E011`. |
| `MAX_TOTAL_BYTES` | `20 * 1024 * 1024` | More total bytes is error `E011`. |
| `MAX_SCHEMA_BYTES` | `256 * 1024` | A larger schema is error `E022`. |
| `MAX_TEXT_SCAN_BYTES` | `1024 * 1024` | How much of a text file is scanned for secrets. |
| `SKIP_DIRS` | `{"__pycache__", ".git", ".venv", ".pytest_cache", ".ruff_cache", ".mypy_cache", "node_modules"}` | Directories ignored by the checks. |
| `ALLOWED_MODULES` | `qsolve numpy scipy sklearn qiskit qiskit_aer pyscf pydantic jsonschema yaml typer rich` | Third-party modules available in the platform runtime image; workflows may import these and the standard library. The sandbox has no network, so nothing else can be installed at run time. |
| `RISKY_IMPORTS` | `subprocess socket ctypes pickle marshal shelve multiprocessing` | Imports that are flagged. |
| `RISKY_CALLS` | `eval exec compile __import__` | Calls that are flagged. |

`locked_modules(root: Path) -> frozenset[str]` returns the import names of the packages a workflow pins in its own `requirements.lock` (distribution names are mapped to import names, for example `scikit_learn` to `sklearn`, `pyyaml` to `yaml`); those are accepted as importable in addition to `ALLOWED_MODULES`.

## qsolve.sandbox_entry

Internal: the platform runs it, you do not. Documented so the protocol is not a mystery.

```bash
python -m qsolve.sandbox_entry < job.json
```

`main() -> int` reads one JSON job from stdin:

| Key | Meaning |
|---|---|
| `mode` | `"validate"` runs `run_benchmarks(workflow_dir, "sim-cpu")`; `"estimate"` runs `estimate_run`; any other value runs `run_workflow`. |
| `workflow_dir` | The workflow directory. |
| `input`, `params` | The input and parameter overrides (not used by `validate`). |
| `target`, `seed` | The target (required for `estimate` and `run`) and seed (default 0). |
| `commit_sha`, `image_digest` | Recorded in the provenance of a run. |

It writes framed lines on a private duplicate of the original standard output, one per line:

```text
@@QS:event@@  {...}                          a progress event (capped by the platform)
@@QS:result@@ {...}                          the final result: a BenchmarkReport, an Estimate or a RunOutput
@@QS:error@@  {"kind": "workflow"|"crash", "message": "..."}
```

Everything the workflow itself prints, from Python or written straight to file descriptor 1 (a subprocess, a native library), goes to standard error so it cannot be mistaken for a frame. Non-finite floats become `null` so strict parsers accept every frame. This stops accidents and casual forging; code in the same process can still find the protocol descriptor, which is why the platform also checks the result against what its host-side executor metered.

| Exit code | Meaning |
|---|---|
| `0` | Success. |
| `2` | A `WorkflowError`: the user's input or the workflow. |
| `1` | A crash (any other exception; a bad job spec), with the traceback on standard error. |

`FRAME` is the frame format string, `@@QS:{kind}@@ {payload}`.