Skip to content

Software Requirements Specification

Document status: Baseline
Standard alignment: IEEE 29148-style requirements specification
Product: AI Hardware Benchmark Suite (aihw-bench)

1. Introduction

1.1 Purpose

This Software Requirements Specification defines the functional, non-functional, interface, data, security, performance, quality, installation, release, and acceptance requirements for AIHW-Bench.

This document is the engineering contract for future implementation. Architecture, code, tests, documentation, packaging, release automation, and community workflows must trace back to these requirements.

1.2 Scope

AIHW-Bench is a Python 3.12+ benchmarking and profiling framework for AI workloads across CPUs, GPUs, embedded platforms, RTL simulators, FPGA prototypes, ASIC accelerators, and future custom hardware. It provides a CLI, Python API, plugin interface, configuration system, benchmark engine, profiler, metrics engine, hardware abstraction layer, session store, visualization layer, and reporting/export system.

1.3 Definitions

  • Backend: An implementation that executes a workload on a runtime, device, simulator, or accelerator.
  • Benchmark session: One immutable benchmark execution record containing configuration, runs, metrics, hardware metadata, and artifacts.
  • Metric: A measured, derived, estimated, or metadata value with units and provenance.
  • Plugin: A separately installed package that contributes providers such as backends, metrics, reports, visualizations, exporters, or CLI commands.
  • Profiler: A component that samples or records execution characteristics around benchmark scopes.
  • Resolved configuration: The final configuration after applying defaults, file values, environment variables, and CLI overrides.
  • Workload: A model, graph, executable, simulator task, or AI computation being benchmarked.

1.4 Acronyms

  • API: Application Programming Interface.
  • CLI: Command Line Interface.
  • CI/CD: Continuous Integration and Continuous Delivery.
  • FPGA: Field-Programmable Gate Array.
  • GPU: Graphics Processing Unit.
  • JSON: JavaScript Object Notation.
  • ONNX: Open Neural Network Exchange.
  • PRD: Product Requirements Document.
  • RTL: Register Transfer Level.
  • SBOM: Software Bill of Materials.
  • SRS: Software Requirements Specification.
  • TFLite: TensorFlow Lite.
  • YAML: YAML Ain't Markup Language.

1.5 References

  • Project Constitution.
  • Product Requirements Document.
  • Architecture Specification.
  • Software Design Document.
  • API Specification.
  • Plugin Architecture Specification.
  • Unified Release and Distribution Strategy.
  • IEEE 29148 principles for requirements structure and traceability.

1.6 Document Conventions

Requirement keywords use RFC 2119-style meanings:

  • shall: mandatory requirement.
  • should: recommended requirement.
  • may: optional capability.

Priorities:

  • P0: Required for the first stable release.
  • P1: Required for broad production usefulness.
  • P2: Planned enhancement.
  • P3: Future or experimental requirement.

Requirement IDs are stable and must be used in implementation tasks, tests, documentation, and release checklists.

1.7 Intended Audience

  • Maintainers.
  • Contributors.
  • Product managers.
  • Software architects.
  • ML infrastructure engineers.
  • Hardware and accelerator engineers.
  • Plugin authors.
  • Release engineers.
  • Documentation authors.
  • Security reviewers.

2. Product Description

2.1 Product Perspective

AIHW-Bench is a standalone open-source package and CLI that can operate locally, in CI, in research environments, and in hardware labs. It is not tied to one AI runtime or one hardware vendor. The core package defines stable abstractions and lightweight capabilities. Optional extras and plugins add heavy runtime, hardware, simulator, and reporting integrations.

2.2 Product Functions

The product shall provide:

  • Benchmark execution.
  • Profiling.
  • Metric computation.
  • Hardware inspection and abstraction.
  • Model/workload metadata capture.
  • Session management.
  • Comparison.
  • Reporting.
  • Visualization.
  • Export.
  • Configuration resolution.
  • Plugin discovery and validation.
  • CLI workflows.
  • Python API workflows.
  • Release and distribution automation.

2.3 User Classes

  • ML engineers.
  • AI researchers.
  • FPGA engineers.
  • ASIC engineers.
  • RTL designers.
  • Embedded engineers.
  • Compiler engineers.
  • Computer architecture researchers.
  • Students and universities.
  • Open-source contributors.
  • Accelerator vendors.
  • DevOps and release engineers.

2.4 Operating Environment

The core package shall support:

  • Windows, Linux, and macOS.
  • Python 3.12+.
  • Local filesystem session storage.
  • Optional runtime-specific extras.
  • CI execution through GitHub Actions.

The architecture shall allow future execution on remote embedded targets, hardware labs, simulators, and distributed benchmark workers.

2.5 Design Constraints

  • Core installation shall not require PyTorch, ONNX Runtime, TensorFlow Lite, CUDA, ROCm, vendor SDKs, simulator binaries, or FPGA toolchains.
  • Dependencies shall point inward according to the clean architecture model.
  • Public APIs shall be versioned.
  • Benchmark results shall remain serializable without runtime-specific dependencies.
  • Documentation shall be updated with feature changes.

2.6 Assumptions

  • Users can provide valid model/workload inputs for selected backends.
  • Optional hardware runtimes are installed and licensed by users where required.
  • Plugins are trusted code during initial releases.
  • External registries such as PyPI, Docker Hub, and Conda-Forge require maintainer-controlled credentials or trusted publishing configuration.

2.7 Dependencies

Core dependencies include Typer, Rich, Pydantic, Pydantic Settings, PyYAML, NumPy, Pandas, Plotly, and Jinja2. Development and release dependencies include pytest, pytest-cov, Ruff, Black, mypy, MkDocs Material, package build tooling, dependency audit tooling, and SBOM generation tooling.

3. System Requirements

3.1 Benchmark Engine

The system shall execute reproducible benchmark sessions with initialization, preparation, warmup, measured execution, measurement capture, statistical aggregation, persistence, reporting, cleanup, and recovery phases.

3.2 Profiler

The system shall collect execution profile data through composable profilers. Core profiling shall include wall-clock timing and process-level CPU and memory data. Optional profiling shall support GPU, accelerator, simulator, power, and thermal integrations through plugins.

3.3 Metrics Engine

The system shall compute measured and derived metrics with explicit units, sources, assumptions, and comparison semantics.

3.4 Visualization

The system shall produce visualization data and chart artifacts from stored session and comparison data without executing benchmarks.

3.5 Reporting

The system shall generate HTML, Markdown, JSON, CSV, and future PDF reports containing summaries, metrics, charts, configuration, hardware metadata, diagnostics, recommendations, and artifacts.

3.6 Plugin System

The system shall discover and validate plugins that contribute providers for backends, models, metrics, reports, visualizations, exporters, CLI commands, hardware inspectors, profilers, and third-party integrations.

3.7 Configuration

The system shall resolve configuration from defaults, files, environment variables, and CLI options with deterministic precedence and validation.

3.8 CLI

The system shall provide a professional CLI for benchmark, profile, compare, report, dashboard, export, doctor, and version workflows.

3.9 Python API

The system shall provide typed Python APIs for programmatic benchmark, profile, comparison, report, export, configuration, and plugin workflows.

3.10 Hardware Layer

The system shall model host and target hardware separately and shall support extension to CPUs, GPUs, embedded targets, simulators, FPGA platforms, ASIC prototypes, and custom accelerators.

3.11 Model Layer

The system shall model AI workloads independently from runtime-specific loading mechanisms and shall preserve model metadata required for reproducibility.

3.12 Session Management

The system shall persist immutable benchmark sessions, raw observations, metrics, profiles, diagnostics, configuration, hardware metadata, and generated artifacts.

4. Functional Requirements

FR-001 Benchmark Session Execution

  • Description: The system shall execute configurable benchmark sessions supporting warmup, repeated execution, measurement, aggregation, persistence, and cleanup.
  • Priority: P0.
  • Inputs: Resolved configuration, workload, backend, device selector, execution policy.
  • Outputs: Benchmark session, execution results, raw observations, metrics, diagnostics.
  • Dependencies: Configuration, backend provider, session store, metrics engine.
  • Failure Conditions: Invalid configuration, unsupported workload, backend unavailable, timeout, runtime error, storage failure.
  • Acceptance Criteria: A session can complete, fail, or partially complete while preserving status and diagnostics.
  • Traceability: Architecture benchmark flow, benchmark engine design, CLI benchmark command, benchmark tests, user guide.

FR-002 Warmup Handling

  • Description: The system shall execute configured warmup iterations separately from measured iterations.
  • Priority: P0.
  • Inputs: Warmup count, workload, backend.
  • Outputs: Warmup timing and diagnostics.
  • Dependencies: Benchmark engine, backend provider.
  • Failure Conditions: Backend failure during warmup, timeout, invalid warmup count.
  • Acceptance Criteria: Warmup measurements are stored but excluded from primary metrics.
  • Traceability: Benchmark engine, metrics engine, benchmark documentation.

FR-003 Measured Iteration Capture

  • Description: The system shall record per-iteration execution observations for measured runs.
  • Priority: P0.
  • Inputs: Iteration count, timing scope, backend execution result.
  • Outputs: Raw observations with timestamps, duration, status, and metadata.
  • Dependencies: Benchmark engine, clock abstraction, backend provider.
  • Failure Conditions: Clock failure, backend exception, interrupted run.
  • Acceptance Criteria: Each measured iteration has a persisted observation or classified failure.
  • Traceability: Data model, session storage, benchmark validation tests.

FR-004 Statistical Aggregation

  • Description: The metrics engine shall compute statistical summaries from measured observations.
  • Priority: P0.
  • Inputs: Raw observations, metric configuration.
  • Outputs: Min, max, mean, median, standard deviation, percentile, throughput, and FPS metrics where applicable.
  • Dependencies: Metrics engine, data model.
  • Failure Conditions: Missing observations, incompatible units, insufficient samples.
  • Acceptance Criteria: Aggregations are deterministic for identical inputs and preserve units.
  • Traceability: Metrics module, metric tests, report tables.

FR-005 Benchmark Result Persistence

  • Description: The system shall persist benchmark sessions as immutable artifacts.
  • Priority: P0.
  • Inputs: Session metadata, configuration, observations, metrics, hardware info, artifacts.
  • Outputs: Session directory or equivalent store record.
  • Dependencies: Session store, serialization, safe path handling.
  • Failure Conditions: Permission denied, invalid path, disk full, schema mismatch.
  • Acceptance Criteria: Finalized canonical session files are not modified after completion.
  • Traceability: Storage design, data model, storage tests.

FR-006 Profiling

  • Description: The system shall support composable profilers around benchmark scopes.
  • Priority: P1.
  • Inputs: Profiler configuration, scope, backend execution events.
  • Outputs: Profile samples and summaries.
  • Dependencies: Profiler providers, benchmark engine, data model.
  • Failure Conditions: Profiler unavailable, insufficient permissions, unsupported device.
  • Acceptance Criteria: Profiling failures are captured as diagnostics when benchmark execution can continue.
  • Traceability: Module design profiling, profile data model, profiler tests.

FR-007 Metric Provider Extension

  • Description: The system shall allow built-in and plugin-provided metric providers.
  • Priority: P1.
  • Inputs: Session data, profiler samples, provider metadata.
  • Outputs: Metric records.
  • Dependencies: Plugin system, metrics engine.
  • Failure Conditions: Provider validation failure, missing required observations.
  • Acceptance Criteria: Metric providers declare required inputs and produce unit-bearing metrics.
  • Traceability: Plugin architecture, metrics tests, API docs.

FR-008 Hardware Inspection

  • Description: The system shall collect host and target hardware metadata.
  • Priority: P0.
  • Inputs: Hardware inspector configuration, target selector.
  • Outputs: HardwareInfo records.
  • Dependencies: Hardware inspector providers, platform APIs.
  • Failure Conditions: Unsupported platform, permission denied, unavailable target.
  • Acceptance Criteria: Missing hardware fields are explicit rather than fabricated.
  • Traceability: Hardware layer, data model, report metadata.

FR-009 Model Metadata Capture

  • Description: The system shall capture workload and model metadata needed for reproducibility.
  • Priority: P0.
  • Inputs: Model path or identifier, workload configuration, runtime metadata.
  • Outputs: ModelInfo record.
  • Dependencies: Model layer, backend provider.
  • Failure Conditions: Missing model, unsupported format, inaccessible file.
  • Acceptance Criteria: Session includes model identity, format, size when available, precision, and input/output metadata when available.
  • Traceability: Model API, report metadata, data model.

FR-010 Configuration Resolution

  • Description: The system shall resolve configuration using precedence: CLI, environment variables, configuration file, defaults.
  • Priority: P0.
  • Inputs: Defaults, YAML or JSON files, environment variables, CLI overrides.
  • Outputs: Resolved configuration.
  • Dependencies: Configuration loader and schema validators.
  • Failure Conditions: Malformed file, unknown key, invalid type, cyclic profile inheritance.
  • Acceptance Criteria: Resolved configuration can be persisted and explains source precedence.
  • Traceability: Configuration system, CLI, configuration tests.

FR-011 Configuration Profiles

  • Description: The system shall support reusable configuration profiles with inheritance.
  • Priority: P1.
  • Inputs: Profile definitions, selected profile name.
  • Outputs: Resolved profile configuration.
  • Dependencies: Configuration resolver.
  • Failure Conditions: Missing profile, cyclic inheritance, incompatible override.
  • Acceptance Criteria: Profile inheritance is deterministic and traceable.
  • Traceability: Configuration documentation, user guide.

FR-012 CLI Commands

  • Description: The CLI shall expose benchmark, profile, compare, report, dashboard, export, doctor, and version commands.
  • Priority: P0.
  • Inputs: Command arguments, options, environment variables.
  • Outputs: Human-readable output, machine-readable output, exit codes.
  • Dependencies: Presentation layer, application services.
  • Failure Conditions: Invalid usage, configuration error, backend error, unexpected internal error.
  • Acceptance Criteria: Each command has help text, examples, documented exit codes, and tests.
  • Traceability: CLI specification, CLI tests, CLI reference.

FR-013 Python API

  • Description: The package shall expose typed Python APIs for benchmark, profile, compare, report, export, configuration, and plugin workflows.
  • Priority: P0.
  • Inputs: Typed command/configuration objects.
  • Outputs: Typed result/session/report objects.
  • Dependencies: Application and domain layers.
  • Failure Conditions: Invalid command, unavailable provider, execution failure.
  • Acceptance Criteria: Public APIs are documented with examples and version history.
  • Traceability: API specification, API reference, public API tests.

FR-014 Plugin Discovery

  • Description: The system shall discover plugins through the aihw_bench.plugins Python entry point group.
  • Priority: P1.
  • Inputs: Installed package entry points.
  • Outputs: Plugin metadata and registered providers.
  • Dependencies: Packaging metadata APIs, plugin validator.
  • Failure Conditions: Import error, invalid descriptor, incompatible API version.
  • Acceptance Criteria: Failed plugins are reported without breaking unrelated built-in providers.
  • Traceability: Plugin architecture, doctor command, plugin tests.

FR-015 Plugin Provider Registration

  • Description: Plugins shall register providers for hardware, models, metrics, reports, visualizations, exporters, CLI commands, profilers, and third-party integrations.
  • Priority: P1.
  • Inputs: Plugin descriptor, provider factories.
  • Outputs: Provider registry entries.
  • Dependencies: Plugin loader, provider registry.
  • Failure Conditions: Duplicate provider name, invalid capability metadata, incompatible provider contract.
  • Acceptance Criteria: Providers are discoverable by name and capability.
  • Traceability: Plugin API, plugin docs, provider conformance tests.

FR-016 Report Generation

  • Description: The system shall generate report artifacts from session or comparison data.
  • Priority: P0.
  • Inputs: BenchmarkSession, comparison result, report configuration.
  • Outputs: HTML, Markdown, JSON, CSV, and future PDF artifacts.
  • Dependencies: Reporter providers, visualization providers, session store.
  • Failure Conditions: Unsupported format, template error, invalid output path.
  • Acceptance Criteria: Report generation does not mutate canonical session data.
  • Traceability: Reporting system, report tests, user guide.

FR-017 Visualization Generation

  • Description: The system shall generate visualization specifications for timeline, roofline, latency distribution, memory, scaling, comparison, and utilization charts.
  • Priority: P1.
  • Inputs: Session metrics, profiles, observations, chart configuration.
  • Outputs: Chart specifications and rendered artifacts when requested.
  • Dependencies: Visualization providers, report system.
  • Failure Conditions: Missing data, incompatible units, renderer failure.
  • Acceptance Criteria: Missing data is displayed explicitly and chart data is deterministic.
  • Traceability: Visualization system, report docs, visualization tests.

FR-018 Session Comparison

  • Description: The system shall compare sessions by model, backend, device, precision, batch size, configuration, and tags.
  • Priority: P1.
  • Inputs: Two or more benchmark sessions, comparison configuration.
  • Outputs: Comparison result, deltas, threshold outcomes, report artifacts.
  • Dependencies: Session store, metrics engine.
  • Failure Conditions: Missing sessions, incompatible schemas, non-overlapping comparison keys.
  • Acceptance Criteria: Comparison output clearly marks improvements, regressions, and unavailable comparisons.
  • Traceability: Comparison service, reporting, regression tests.

FR-019 Export

  • Description: The system shall export session and comparison data in JSON, CSV, Markdown, HTML, and future PDF formats.
  • Priority: P0.
  • Inputs: Session or comparison object, export format, output destination.
  • Outputs: Export artifact.
  • Dependencies: Exporter providers, safe file handling.
  • Failure Conditions: Unsupported format, serialization failure, invalid output path.
  • Acceptance Criteria: Exports preserve units, source, assumptions, and provenance.
  • Traceability: Export docs, report API, export tests.

FR-020 Doctor Diagnostics

  • Description: The system shall provide diagnostics for installation, optional dependencies, plugins, writable paths, runtime availability, and platform compatibility.
  • Priority: P0.
  • Inputs: Runtime environment, plugin registry, configuration.
  • Outputs: Diagnostic report and exit code.
  • Dependencies: CLI, plugin loader, hardware inspectors.
  • Failure Conditions: Missing dependencies, invalid plugin, inaccessible path.
  • Acceptance Criteria: Diagnostics identify cause and suggested fix.
  • Traceability: CLI doctor command, troubleshooting docs.

FR-021 Release Automation

  • Description: A semantic-version Git tag shall trigger automated quality gates, package builds, SBOM generation, release asset upload, publishing, docs deployment, and install verification.
  • Priority: P0.
  • Inputs: Git tag, repository source, package metadata, changelog.
  • Outputs: Release artifacts, GitHub Release, published packages, documentation site.
  • Dependencies: GitHub Actions, package build tools, registry credentials or trusted publishing.
  • Failure Conditions: Failing tests, version mismatch, publishing failure, docs build failure.
  • Acceptance Criteria: No manual packaging is required for a compliant release.
  • Traceability: Distribution strategy, release workflow, release tests/checks.

FR-022 Documentation Generation

  • Description: The system documentation shall include installation, quick start, tutorials, examples, API reference, CLI reference, architecture, plugin development, FAQ, troubleshooting, performance guide, roadmap, and contribution guide.
  • Priority: P0.
  • Inputs: Markdown documentation, API doc sources, examples.
  • Outputs: MkDocs site and search index.
  • Dependencies: MkDocs, documentation workflow.
  • Failure Conditions: Broken links, missing nav pages, strict build failure.
  • Acceptance Criteria: Documentation builds in strict mode during CI.
  • Traceability: Documentation plan, docs workflow.

FR-023 Package Installation

  • Description: The package shall be installable through Python package managers and prepared for native package ecosystems.
  • Priority: P0.
  • Inputs: Package metadata, distribution artifacts.
  • Outputs: Wheel, source distribution, installable CLI.
  • Dependencies: PEP 517, PEP 518, PEP 621 metadata.
  • Failure Conditions: Invalid metadata, missing files, unsupported Python version.
  • Acceptance Criteria: Built wheel installs and exposes import and CLI checks on Windows, Linux, and macOS.
  • Traceability: Release pipeline, installation docs.

5. Non-Functional Requirements

NFR-001 Performance

Benchmark engine overhead shall be minimized and measured separately from workload execution where practical. Report generation and heavy visualization shall occur outside measured regions.

NFR-002 Scalability

The architecture shall support growth from single local runs to large benchmark campaigns, remote targets, and future distributed execution.

NFR-003 Availability

Local CLI and API workflows shall not require external network services except when users explicitly configure remote integrations.

NFR-004 Reliability

Failed and partial sessions shall remain inspectable. Canonical files shall be written atomically where practical.

NFR-005 Maintainability

Modules shall follow clean architecture boundaries, single responsibility, strict typing, and documented public contracts.

NFR-006 Portability

The core package shall run on Windows, Linux, and macOS with Python 3.12+.

NFR-007 Security

The system shall validate all external inputs, avoid unsafe deserialization, protect against path traversal, and avoid exposing secrets in diagnostics or reports.

NFR-008 Accessibility

CLI and reports shall not rely solely on color. Machine-readable outputs shall be available for automation and accessibility tooling.

NFR-009 Documentation

Every public command, public API, plugin contract, configuration object, and report format shall be documented.

NFR-010 Packaging

The package shall comply with modern Python packaging standards and release automation shall produce wheel and source distribution artifacts.

NFR-011 Developer Experience

Contributors shall be able to run local quality checks, tests, docs builds, and package builds using documented commands.

6. Interface Requirements

6.1 CLI Interface

The CLI shall expose documented commands, options, exit codes, help text, examples, human-readable output, and machine-readable output. CLI errors shall include cause, suggested solution, and documentation reference where practical.

6.2 Python API

The Python API shall expose typed command objects, service interfaces, data models, result objects, and exceptions. Public APIs shall include examples and versioning notes.

6.3 Plugin API

The plugin API shall define descriptor metadata, provider registration, capability declaration, version compatibility, diagnostics, and provider contracts.

6.4 Configuration API

The configuration API shall support loading, merging, validating, resolving, serializing, and explaining configuration sources.

6.5 Report API

The report API shall accept session or comparison data and produce report artifacts without mutating canonical data.

6.6 Visualization API

The visualization API shall accept normalized chart data and produce chart specifications or rendered artifacts.

6.7 Hardware Backend API

The hardware backend API shall support capability discovery, device selection, preparation, execution, diagnostics, and cleanup.

6.8 Model Backend API

The model backend API shall support workload loading, metadata extraction, validation, input preparation, and runtime-specific execution integration.

7. Data Requirements

7.1 BenchmarkSession

Shall include session ID, schema version, timestamps, status, resolved configuration, system information, hardware information, workload metadata, backend metadata, runs, metrics, profiles, artifacts, diagnostics, and plugin metadata.

7.2 BenchmarkResult

Shall include result ID, session ID, status, primary metrics, secondary metrics, statistics, comparison keys, and error classification.

7.3 Metric

Shall include name, display name, value, unit, kind, source, assumptions, precision, tags, and comparison direction.

7.4 Model

Shall include model ID, name, format, source, size, input shapes, output shapes, precision, parameter count, MAC estimate, FLOPS estimate, and metadata.

7.5 Hardware

Shall include host, CPU, memory, GPU, accelerator, embedded target, simulator, driver, firmware, thermal, and power metadata when available.

7.6 Profile

Shall include profile ID, profiler name, scope, sample interval, samples, summary metrics, and diagnostics.

7.7 Execution

Shall include execution ID, phase, iteration, timestamps, duration, status, observations, backend metadata, and errors.

7.8 History

Shall include related session IDs, comparison keys, baseline, candidate, deltas, thresholds, and outcomes.

7.9 Plugin

Shall include name, version, API version, package, description, providers, dependencies, capabilities, status, and diagnostics.

7.10 Configuration

Shall include schema version, sources, selected profile, workload, backend, device, execution, profiling, metrics, reports, storage, plugins, and tags.

8. Error Handling

8.1 Validation

Invalid inputs shall fail before execution when possible. Validation errors shall identify field path, invalid value, expected constraint, and suggested correction.

8.2 Exceptions

Expected exceptions shall be classified as configuration, validation, backend, runtime, profiler, metric, plugin, storage, report, export, security, or internal errors.

8.3 Recovery

The benchmark engine shall preserve partial run data when safe. Cleanup shall execute after recoverable failures. Failed reports shall not erase benchmark results.

8.4 Logging

Logs shall provide structured diagnostics without exposing secrets. CLI output shall remain concise while debug logs preserve deeper context.

8.5 Diagnostics

Diagnostics shall include cause, impact, suggested solution, provider name when relevant, and documentation reference where practical.

9. Security Requirements

SR-001 Input Validation

All external input shall be validated at system boundaries.

SR-002 Dependency Security

Release workflows shall include dependency auditing and SBOM generation.

SR-003 Plugin Security

Plugin descriptors shall be validated. Plugin load failures shall be isolated. Initial plugins are trusted in-process code.

SR-004 File Handling

File paths shall be resolved safely, path traversal shall be rejected, and canonical session files shall not be overwritten after finalization.

SR-005 Credential Handling

Credentials shall not be stored in benchmark sessions, reports, logs, or configuration snapshots unless explicitly marked safe by a future secret-reference design.

SR-006 Supply Chain Security

Release artifacts shall include checksums and SBOMs. Trusted publishing shall be used for supported registries.

10. Performance Requirements

PR-001 Memory Usage

The system shall avoid unnecessary copies of model data, input data, and observation data.

PR-002 Execution Time

Framework overhead shall be measured and minimized for benchmark loops.

PR-003 Startup Time

Core CLI startup should remain under 500 ms on common developer machines once the command set stabilizes.

PR-004 CLI Responsiveness

Long-running commands shall provide progress feedback where practical.

PR-005 Large Model Handling

Large model loading shall be backend-owned and shall avoid eager loading in commands that only inspect configuration or metadata.

PR-006 Large Dataset Handling

Large benchmark campaigns shall support streaming or chunked observation handling in future storage implementations.

PR-007 Concurrent Execution

Future concurrent execution shall isolate devices, output paths, profilers, random seeds, and backend state.

11. Quality Requirements

QR-001 Code Coverage

The project shall target at least 90% test coverage by 1.0.0.

QR-002 Static Analysis

CI shall run static analysis before release.

QR-003 Type Checking

Strict mypy checking shall cover typed source packages.

QR-004 Linting

Ruff shall be used for lint enforcement.

QR-005 Formatting

Black shall be used for formatting checks.

QR-006 Documentation Coverage

Every public API, public command, plugin contract, and configuration schema shall have documentation.

12. Installation Requirements

IR-001 Windows

The core package shall install and pass import and CLI smoke checks on Windows.

IR-002 Linux

The core package shall install and pass import and CLI smoke checks on Linux.

IR-003 macOS

The core package shall install and pass import and CLI smoke checks on macOS.

IR-004 Python Package

The product shall support pip install aihw-bench and uv add aihw-bench.

IR-005 Docker

The product shall publish or prepare container images for reproducible benchmark environments.

IR-006 Conda

The product shall provide metadata suitable for Conda-Forge packaging.

IR-007 Homebrew

The product shall provide Homebrew formula metadata or generated release formula artifacts for CLI installation.

13. Release Requirements

RR-001 GitHub Releases

Every release tag shall create a GitHub Release with release notes and artifacts.

RR-002 PyPI

Release automation shall publish wheel and source distribution artifacts to PyPI through trusted publishing.

RR-003 TestPyPI

Release automation shall publish to TestPyPI before PyPI when configured.

RR-004 GitHub Pages

Release or documentation workflows shall deploy the documentation site to GitHub Pages.

RR-005 Docker Hub or Container Registry

Release automation shall build and publish container images when registry credentials are configured.

RR-006 Conda-Forge

Release artifacts shall support downstream Conda-Forge packaging.

RR-007 Trusted Publishing

Supported package registries shall use trusted publishing when available.

RR-008 Signed Releases

The release design shall allow future signing and provenance attestations.

RR-009 Semantic Versioning

Versions shall follow semantic versioning and release tags shall use vMAJOR.MINOR.PATCH.

14. Traceability Matrix

Requirement Range Architecture Implementation Target Tests Documentation Release
FR-001 to FR-005 Benchmark engine, data model, storage design Application benchmark service, backend ports, session store Unit, integration, regression, CLI Benchmark guide, API reference CI, package build
FR-006 Profiling architecture Profiler providers Unit, integration, optional hardware Profiling guide CI optional matrix
FR-007 Metrics architecture, plugin architecture Metric providers Unit, golden-data, regression Metrics reference CI
FR-008 Hardware layer Hardware inspectors Unit, integration, platform matrix Hardware guide Install verification
FR-009 Model layer Workload loaders Unit, backend integration Model guide CI optional backend matrix
FR-010 to FR-011 Configuration system Configuration resolver Unit, regression, CLI Configuration guide CI
FR-012 Presentation architecture CLI commands CLI tests CLI reference Install verification
FR-013 API specification Public Python API Unit, integration API reference Package verification
FR-014 to FR-015 Plugin architecture Plugin loader and registry Unit, conformance, regression Plugin guide CI
FR-016 to FR-019 Reporting and visualization Reporters, exporters, visualizers Unit, snapshot, integration Report guide Docs build, package build
FR-020 CLI and diagnostics Doctor service CLI, integration Troubleshooting Install verification
FR-021 to FR-023 Distribution strategy GitHub Actions, package metadata Release dry runs Release docs Tag pipeline
NFR/SR/PR/QR/IR/RR Cross-cutting architecture All subsystems Quality gates, security scans, install checks Engineering docs Release pipeline

15. Acceptance Criteria

Benchmark Engine

  • Executes warmup and measured iterations.
  • Stores raw observations and computed metrics.
  • Preserves failed and partial sessions.
  • Cleans up backend and profiler resources.

Profiler

  • Supports composable profiling scopes.
  • Records profiler samples with units and timestamps.
  • Degrades gracefully when optional profilers are unavailable.

Metrics Engine

  • Computes required metrics deterministically.
  • Preserves units, assumptions, and provenance.
  • Supports plugin metric providers.

Visualization

  • Produces deterministic chart data.
  • Handles missing data explicitly.
  • Supports timeline, roofline, latency, memory, scaling, comparison, and utilization chart families.

Reporting

  • Generates HTML, Markdown, JSON, CSV, and future PDF reports.
  • Includes configuration, hardware metadata, metrics, charts, diagnostics, recommendations, and artifact manifest.
  • Does not mutate canonical session data.

Plugin System

  • Discovers entry-point plugins.
  • Validates descriptors and provider contracts.
  • Isolates plugin load failures.

Configuration

  • Supports YAML, JSON, environment variables, CLI options, defaults, and profiles.
  • Enforces precedence deterministically.
  • Persists resolved configuration with sessions.

CLI

  • Provides all required commands.
  • Documents options, examples, and exit codes.
  • Supports human-readable and machine-readable output.

Python API

  • Provides typed command, service, result, exception, and data model interfaces.
  • Documents examples and versioning policy.

Hardware Layer

  • Captures host and target hardware metadata.
  • Supports extensible backend capability discovery.
  • Handles unavailable hardware gracefully.

Model Layer

  • Captures workload metadata.
  • Validates workload/backend compatibility.
  • Keeps runtime-specific logic inside adapters.

Session Management

  • Stores immutable sessions.
  • Supports loading, listing, comparing, and exporting sessions.
  • Protects against path traversal and schema mismatch.

Installation

  • Wheel installs on Windows, Linux, and macOS.
  • CLI and import smoke checks pass after installation.
  • Optional dependencies fail gracefully when absent.

Release

  • A semantic version tag triggers the release pipeline.
  • Release artifacts include wheel, source distribution, checksums, SBOM, changelog, release notes, and documentation.
  • Publishing uses trusted publishing where supported.

Documentation

  • Docs build in strict mode.
  • Public APIs, CLI commands, plugin contracts, configuration schemas, and report formats are documented.
  • Engineering docs remain traceable to this SRS.