SDK: testing, linting and the sandbox

The code behind qsolve test and qsolve lint, which the platform's validation pipeline also runs, and the entry point the platform runs inside its sandbox. Back to the SDK reference.

qsolve.benchmark

from qsolve.benchmark import (BenchmarkReport, Case, CaseResult, Expect, Gate, check_estimate,
                              check_quality, load_hardware_cases, run_benchmarks)

Benchmark and validation gates for a workflow. Used by qsolve test locally and by the platform's validation, so a workflow passes locally if and only if it passes on the platform. What each gate and expect key means for a workflow author is in Benchmarks.

run_benchmarks

run_benchmarks(workflow_dir: str | Path, target: str = "sim-cpu") -> BenchmarkReport

Runs the validation gates and every case in benchmarks/cases.yaml.

Returns a BenchmarkReport; passed is true when every gate passed. A case that raises is recorded as a failed case with error="<Type>: <message>"; it does not stop the others. Raises FileNotFoundError, ValueError or pydantic.ValidationError when the manifest or its schemas are invalid, or a case in cases.yaml is malformed (qsolve test prints Invalid workflow: ... and exits 1). It does not run qsolve lint. Side effects: runs your workflow once per case, plus a second run of the first case and an estimate (so cases on ibm: targets spend real quota; use fake: for rehearsal).

The gates, in the order they are added:

Case

An entry under cases: (run by qsolve test) or hardware_cases: (never run by it) in benchmarks/cases.yaml. Unknown keys are rejected.

Expect

check_quality

check_quality(expect: Expect, r: Any) -> str

Why a result misses what the platform requires of every result, or the case's expect; "" when it misses nothing. r is a Result. Always required: a headline metric exists; no method is unavailable with failed: ...; no method has a claimed value (it contradicted its own solution); the quantum method sent at least one circuit. Then each expect bound that is set.

check_estimate

check_estimate(est: Estimate, actual: CircuitStats | None) -> tuple[bool, str]

Whether the workflow's estimate matches what the SDK metered. Returns (True, "no estimate() given: ...") for a default estimate, (True, "no circuits were sent") when actual is None, (False, "estimated N circuits, the run sent M") (or shots) when either ratio is outside 0.5 to 2, otherwise (True, "estimate within 2x of what was sent").

load_hardware_cases

load_hardware_cases(root: str | Path) -> list[Case]

The cases under hardware_cases: in benchmarks/cases.yaml, or [] when the file has none. They are never run by qsolve test or by validation: the platform runs them on a real device when the workflow's developer asks, at the developer's own cost. MAX_HARDWARE_CASES = 3 and HARDWARE_TARGET (^ibm:[A-Za-z0-9_.-]{1,60}$) are the limits lint enforces on them (codes E052 to E054, see Lint codes).

CaseResult

Gate

Gate(name: str, passed: bool, detail: str = "").

BenchmarkReport

report.model_dump(mode="json") is what qsolve test --out writes and what the sandbox sends for a validate job.

qsolve.lint

from qsolve.lint import Finding, has_errors, lint_workflow

Static checks on a workflow repository. No code is executed. The same checks run locally (qsolve lint) and on the platform before anything is built or run. They are defence in depth: the sandbox, not this linter, is the security boundary. Every code, its meaning and its fix are in Lint codes.

lint_workflow

lint_workflow(root: str | Path) -> list[Finding]

Returns every finding (an empty list means clean). It does not raise for a broken repository: a missing qsolve.yaml returns E001, an unparseable or invalid manifest or schema file returns E002. Reads files only. Besides the manifest and schemas it checks benchmarks/cases.yaml: each case has an input or an existing input_file inside the repository, case names are unique, no ibm: target sits under cases, and each case input matches the input schema (E055 to E059); and a shots parameter may not exceed qsolve.MAX_SHOTS_PER_CIRCUIT (E025). See Lint codes.

Finding

@dataclass(frozen=True)
class Finding:
    severity: Literal["error", "warning"]
    code: str          # E### or W###
    message: str
    path: str = ""     # relative to the workflow root; may be empty

as_dict() returns {"severity", "code", "message", "path"} as plain strings. Severity is the type alias Literal["error", "warning"].

has_errors

has_errors(findings: list[Finding]) -> bool: true when at least one finding has severity error. This is what decides the exit code of qsolve lint.

Constants

locked_modules(root: Path) -> frozenset[str] returns the import names of the packages a workflow pins in its own requirements.lock (distribution names are mapped to import names, for example scikit_learn to sklearn, pyyaml to yaml); those are accepted as importable in addition to ALLOWED_MODULES.

qsolve.sandbox_entry

Internal: the platform runs it, you do not. Documented so the protocol is not a mystery.

python -m qsolve.sandbox_entry < job.json

main() -> int reads one JSON job from stdin:

It writes framed lines on a private duplicate of the original standard output, one per line:

@@QS:event@@  {...}                          a progress event (capped by the platform)
@@QS:result@@ {...}                          the final result: a BenchmarkReport, an Estimate or a RunOutput
@@QS:error@@  {"kind": "workflow"|"crash", "message": "..."}

Everything the workflow itself prints, from Python or written straight to file descriptor 1 (a subprocess, a native library), goes to standard error so it cannot be mistaken for a frame. Non-finite floats become null so strict parsers accept every frame. This stops accidents and casual forging; code in the same process can still find the protocol descriptor, which is why the platform also checks the result against what its host-side executor metered.

FRAME is the frame format string, @@QS:{kind}@@ {payload}.