Application-Motivated, Holistic Benchmarking of a Full Quantum Computing Stack

Quantum computing systems need to be benchmarked in terms of practical tasks they would be expected to do. We propose 3 "application-motivated" circuit classes for benchmarking: deep (relevant for state preparation in the variational quantum eigensolver algorithm), shallow (inspired by IQP-type circuits that might be useful for near-term quantum machine learning), and square (inspired by the quantum volume benchmark). We quantify the performance of a quantum computing system in running circuits from these classes using several figures of merit, all of which require exponential classical computing resources and a polynomial number of classical samples (bitstrings) from the system. We present results of these benchmarks, obtained using systems made available by IBM Quantum. We use these results to study how performance depends on choices made at several levels of the quantum computing stack, including the compilation strategy to use and the device on which the circuits are run. We show that noise-aware compilation strategies may be beneficial, and that device connectivity and noise levels play a crucial role in the performance of the system according to our benchmarks.