Benchmarking Quantum Computers: Best Practices

As quantum computers grow in size and scope the question of how best to benchmark performances has become one of great importance. Here we discuss some of the questions around benchmarking devices at scale, some of the issues that arise and best practices. In particular, we focus on defining benchmarking and diagnostic methods, the use of error-mitigation, possible weaknesses to gaming strategies and the role of applications-oriented benchmarking in the current landscape of quantum computing. Furthermore, we consider one of the available applications-oriented benchmarking suite and show the effects of error-suppression and mitigation strategies, along with gaming strategies, on the benchmark.