API Reference
The API reference is generated from the implemented Python package using MkDocs and mkdocstrings.
Most application code should import from the public package or from the documented application, domain, and infrastructure modules.
AI Hardware Benchmark Suite public package.
AssistantInsight
dataclass
One grounded assistant finding with a confidence label and source metric.
AssistantResponse
dataclass
Explainable benchmark intelligence for a session or session pair.
to_markdown()
Render a portable assistant report suitable for tickets or reports.
BaselineResult
dataclass
Reference-backend baseline used to validate suite plumbing, not model accuracy.
BenchmarkAssistant
Generate deterministic, metric-grounded benchmark intelligence locally.
explain(session, baseline=None)
Explain results, anomalies, hardware deltas, and optimization opportunities.
BenchmarkDefinition
dataclass
One official, runtime-neutral benchmark workload definition.
dataset_manifest()
Return a deterministic synthetic-input manifest for this benchmark.
BenchmarkIntelligenceProvider
Bases: Protocol
Optional provider protocol for an externally hosted AI explanation service.
BenchmarkLifecycleState
Bases: StrEnum
Lifecycle states for benchmark orchestration.
BenchmarkOutcome
dataclass
Result returned by the benchmark service.
BenchmarkRequest
dataclass
Validated benchmark request accepted by the benchmark service.
BenchmarkSchedule
dataclass
Pollable interval schedule; execution is owned by a dedicated runner.
BenchmarkService
Coordinate benchmark execution over a configured backend.
run(request)
Run one benchmark session and return the final immutable session record.
DashboardArtifact
dataclass
Generated standalone dashboard artifact.
DashboardService
Build a responsive dashboard from a bounded page of stored sessions.
build(store, output_dir, *, title='AIHW-Bench Dashboard')
Generate a dashboard without embedding an unbounded history payload.
ExecutionPlan
dataclass
Ordered execution steps for a benchmark session.
measured_steps
property
Return the measured steps only.
ExecutionScheduler
Build a deterministic execution plan for warmup and measured iterations.
MarketplacePlugin
dataclass
Marketplace listing metadata; installation remains an explicit user action.
MonotonicTimingEngine
High-resolution timing engine backed by the system monotonic clock.
OfficialBenchmarkSuite
Discover official workloads and materialize deterministic dataset fixtures.
definitions()
Return official workloads in stable release order.
definition(identifier)
Return one benchmark definition or raise a structured validation error.
materialize_dataset(identifier, output_dir)
Write one deterministic dataset manifest and return its path.
materialize_all_datasets(output_dir)
Write manifests for the full official suite in stable order.
baseline_results()
Return immutable reference-backend baseline fixtures.
write_baselines(output_path)
Write deterministic reference baseline results.
PluginContext
dataclass
Context passed to plugin lifecycle callbacks.
PluginDiscovery
Discover and normalize plugin registrations from Python entry points.
discover_entry_points()
Load plugin registrations from installed package entry points.
discover_objects(objects)
Normalize in-memory plugin objects for tests and embedded composition roots.
registration_from_object(plugin_object)
Normalize a plugin object, factory, manifest, or registrar into a registration.
PluginLifecycleState
Bases: StrEnum
Lifecycle state tracked for each discovered plugin.
PluginManager
High-level plugin loader that combines discovery, registry, and lifecycle.
load(registrations=None, *, activate=True)
Discover, register, and optionally activate plugins.
PluginManifest
Bases: BaseModel
Validated plugin manifest returned by simple plugin packages.
reject_self_dependency(value)
classmethod
Normalize dependency names and reject blanks.
to_metadata()
Convert the manifest to the domain plugin metadata model.
PluginMarketplace
Read and write a versioned local marketplace index.
PluginProviderKind
Bases: StrEnum
Provider categories accepted by the AIHW-Bench plugin registry.
PluginRegistration
dataclass
Registration payload returned by an AIHW-Bench plugin entry point.
from_manifest(manifest, *, providers=None, validate=None, activate=None, deactivate=None)
classmethod
Create a registration from a manifest and optional provider objects.
PluginRegistry
Validate plugins and expose registered providers by category.
register(registration)
Validate and register one plugin registration.
register_many(registrations)
Register plugins after resolving same-process plugin dependencies.
activate_all(context=None)
Run validation and activation callbacks for registered plugins.
deactivate_all(context=None)
Run deactivation callbacks for active plugins in reverse registration order.
providers_for(kind)
Return registered providers for a provider category.
provider(kind, name)
Return one registered provider by category and provider name.
plugin_metadata()
Return plugin metadata in deterministic order.
ProjectConfiguration
dataclass
Project-scoped configuration and selected workspace profile.
RemoteBenchmarkRequest
dataclass
Transport-neutral remote benchmark request; caller supplies transport security.
ResultComparator
Compare matching numeric metrics from two persisted sessions.
ScriptedTimingEngine
Deterministic timing engine for tests and smoke validation.
StatisticsEngine
Compatibility facade for the canonical :class:StatisticalAggregator.
Historical callers receive duration-specific field names while all numeric calculations are delegated to the single metrics implementation.
summarize(durations)
Return duration statistics using the stable benchmark-engine schema.
WorkspaceManager
Persist workspace profiles and project configurations as readable YAML.
resolve(project_name)
Resolve profile defaults with project configuration taking precedence.
WorkspaceProfile
dataclass
Named workspace defaults applied before project configuration.
BackendRegistry
Bases: Protocol
Resolve and validate benchmark backends.
register(backend)
Register a backend implementation.
resolve(name)
Return a backend by registry name.
select(configuration, hardware, workload=None)
Select the best backend for the resolved configuration and hardware.
validate(backend, configuration, hardware, workload=None)
Validate a backend for the requested configuration and hardware.
BenchmarkBackend
Bases: Protocol
Contract for a backend that can prepare, execute, and clean up a workload.
supports_device(device)
Return whether the backend can target the requested device string.
validate(configuration, hardware, workload)
Validate the backend against the selected configuration and hardware.
prepare(configuration, hardware, workload)
Prepare backend resources before the benchmark starts.
execute(phase, iteration, configuration, hardware, workload)
Run one warmup or measured iteration and return backend observations.
cleanup()
Release any resources acquired during preparation.
BenchmarkResult
Bases: FrozenModel
Aggregate result for a workload/backend/device combination.
is_successful()
Return whether the benchmark result completed successfully.
primary_metric(name)
Return a primary metric by name.
BenchmarkSession
Bases: FrozenModel
Immutable top-level benchmark session record.
require_session_timezone(value)
classmethod
Require timezone-aware session timestamps.
validate_completion_time()
Ensure completed sessions include completion time.
finalize(status, *, completed_at=None)
Return a finalized copy of the session.
add_artifact(artifact)
Return a copy of the session with a new artifact.
Configuration
Bases: FrozenModel
Resolved AIHW-Bench configuration.
The model is immutable and records the sources used to build it.
explain_sources()
Return readable source descriptions in precedence order.
to_resolved_yaml()
Serialize the resolved configuration as YAML.
Diagnostic
ExecutionResult
Bases: FrozenModel
One backend execution event.
require_timezone(value)
classmethod
Require timezone-aware timestamps.
validate_time_order()
Ensure execution end time is not before start time.
ExportArtifact
Bases: FrozenModel
Generated artifact metadata and integrity information.
require_created_at_timezone(value)
classmethod
Require timezone-aware artifact timestamps.
HardwareInspector
Bases: Protocol
Collect host and target hardware metadata for a benchmark session.
inspect()
Return a hardware profile for the current machine.
HardwareProfile
Bases: FrozenModel
Host and target hardware metadata captured for a session.
summary()
Return a compact hardware summary.
capability_report()
Return normalized capabilities for backend selection and reports.
LoadedModel
dataclass
Resolved model handle and extracted metadata.
Metric
Bases: FrozenModel
Measured, derived, estimated, or metadata metric.
format()
Format a metric for human-readable output.
compare_to(other)
Return numeric delta from another compatible metric.
ModelError
ModelLoader
Bases: Protocol
Contract implemented by format-specific model loaders.
can_load(source)
Return whether the loader can handle the given file path.
load(request)
Load a model file and return metadata plus the resolved handle.
ModelLoaderCatalog
Bases: Protocol
Registry interface used by application services.
load(request)
Load a model from an explicit load request.
load_workload(workload)
Load a model from resolved configuration.
ModelLoadRequest
dataclass
Request describing a model source to be loaded.
ModelMetadata
Bases: FrozenModel
Metadata for a benchmark workload or model.
fingerprint()
Return a stable fingerprint for comparison and caching.
to_dict()
Return a JSON-serializable model description.
summary()
Return summary fields suitable for reports.
PluginMetadata
Bases: FrozenModel
Validated plugin descriptor metadata.
is_compatible(core_api_version)
Return whether the plugin API version matches the core API version.
provider_names()
Return provider names declared by this plugin.