Curious exploration in complex environments

Curiosity is widely recognized as a fundamental mode of cognition, driving people and animals towards novel experiences that yield new insights about the world and its underlying processes. Curiosity is therefore linked to information-seeking in novel environments and is an important part of behavior that we perceive as ``intelligent". While simple greedy actors may be optimal in simple and highly determinate environments, it is expected that curiosity is beneficial in more complex, evolving environments. We approach the question of curiosity by comparing reinforcement learning agents using different phenomenological models of curiosity in environments of controlled complexity. We find that indeed, greedy agents are inferior to curious ones in certain complex environments (where high reward strategies may be harder to find). However, when environments are too complex or noisy, curiosity imparts no benefit. These findings imply that curiosity is useful in a window of intermediate environmental complexity.