Defending smart electrical power grids against cyberattacks with deep Q-learning

A key to ensuring the security of smart electrical power grids is to devise and deploy effective defense strategies against cyberattacks. To achieve this goal, an essential task is to simulate and understand the dynamical interplay between the attacker and defender, for which stochastic game theory and reinforcement learning stand out as a powerful mathematical/computational framework. Existing works were based on conventional Q-learning to find the critical sections of a power grid to choose an effective defense strategy, but the methodology was applicable to small systems only. Additional issues with Q-learning are the difficulty to consider the timings of cascading failures in the reward function and deterministic modeling of the game while the attack success depends on various parameters and typically has a stochastic nature. Our solution to overcoming these difficulties is to develop a deep Q-learning based stochastic zero-sum Nash strategy solution. We demonstrate the workings of our deep-Q learning solution using the benchmark W&W 6-bus and the IEEE 30-bus systems, the latter being a relatively large-scale power-grid system that defies the conventional Q-learning approach. Comparison with alternative reinforcement learning methods provides further support for the general applicability of our deep-Q learning framework in ensuring secure operation of modern power grid systems