Benchmarking for Quantum AI: »Quant²AI« Makes Quantum Computing Applications Comparable

Modular Framework for Research and Application

The framework organizes a pipeline into five core modules:

• Data Loading: This module reads a dataset and divides it into training, validation, and test data.
• Preprocessing: It prepares the data for the respective model and, for example, reduces its complexity.
• Model Core: Here, the framework trains a classical, quantum-based, or hybrid model.
• Post-processing: This module converts the results back into a form that can be interpreted in the context of the original data.
• Evaluation: It calculates metrics and makes the pipeline’s performance measurable.

Researchers can flexibly combine, swap, or supplement these modules with their own components. An internal check detects inappropriate combinations early on. For example, the framework prevents a pipeline designed for image data from accidentally processing tabular data. This modular structure makes it easier to integrate new algorithms, datasets, quantum computers, or evaluation metrics.

From Anomaly Detection to Image Classification

Reference pipelines are already available for several typical machine learning tasks. These include anomaly detection, clustering, regression, and the classification and segmentation of images. The associated datasets range from well-established examples such as »Iris« and »MNIST« to real-world industrial applications.

One example application is the automatic detection of fine cracks on concrete surfaces. For this, we use a hybrid pipeline that combines a classical neural network with a quantum circuit. The classical network identifies relevant features in the images. A quantum circuit then takes over part of the evaluation.

A Direct Comparison of Classical and Quantum-Based AI

For the benchmark, we tested a classical and a hybrid quantum variant of the model on a high-performance computer. In this process, the dataset is split multiple times into training and test data in various ways. This allows for a better assessment of how stable a model is and how well it processes unknown data.

The hybrid quantum model achieved slightly higher values for accuracy and the F1 score than the classical variants examined. The F1 score combines precision and recall into a single metric. At the same time, the simulated quantum circuit required more computational resources. This example therefore demonstrates both the potential of hybrid approaches and the importance of a transparent comparison: good results alone are not enough. Computational time, resource requirements, stability, and the hardware used must also be factored into the evaluation.

Integrating Quantum Hardware from Various Vendors

»QUANT²AI« can run experiments on quantum simulators and real quantum hardware. A dedicated connection module transmits the quantum circuits to the respective system and processes its results. To do this, the framework uses, among other things, OpenQASM, a widely used format for describing quantum circuits.

Currently, interfaces to backends from IBM, IQM, and AQT are available. Additional systems can be integrated. The framework also tracks quantum-specific metrics, such as the number of qubits used or the depth of the quantum circuit. Qubits are the fundamental units of information in a quantum computer. In simple terms, circuit depth indicates how many consecutive computational steps a quantum circuit contains.

Complex benchmarks generate many variants of a pipeline and require a correspondingly high amount of computing power. For this reason, »QUANT²AI« integrates High Performance Computing. A Python interface automatically generates the necessary scripts and submits the computational jobs to the cluster system. Users therefore do not have to manage all the technical processes of the high-performance computer themselves. »QUANT²AI« sets up the required software environment, executes the pipeline, and stores results and log data in a clear and traceable manner.

Interactively Analyze Results

A browser-based dashboard presents the benchmark results in a clear and organized manner. It combines classic machine learning metrics with information from simulators and quantum computers. Users can filter experiments, compare models, and explore relationships between different parameters.

For example, this allows users to analyze how a model’s accuracy changes with the number of qubits. Distributions, outliers, and differences between multiple model variants also become visible. Since the dashboard flexibly retrieves its data from a database, new datasets, models, and metrics can be added without fundamentally redesigning the user interface.

From Research to Industrial Applications

Both researchers and companies benefit from »QUANT²AI«. Scientists can test new algorithms and individual components under standardized conditions. Users gain a solid foundation for selecting suitable methods for specific tasks and realistically assessing the potential benefits of quantum AI.

The framework thus helps distinguish verifiable progress from inflated expectations. At the same time, it facilitates the transfer to industrial applications: new datasets, pipeline steps, AI models, quantum computers, and evaluation metrics can be integrated with minimal effort. »QUANT²AI« thus creates a common foundation on which research and industry can transparently evaluate and systematically advance quantum AI.

Project Funding and Duration

The »QUANT²AI« project is part of the DLR Quantum Computing Initiative and is funded by the Federal Ministry for Economic Affairs and Climate Action (BMWK). The project will run from November 1, 2022, through June 30, 2026.