Divergent Predictive States: The Statistical Complexity Dimension of Stationary, Ergodic Hidden Markov Processes

Even simply-defined, finite-state generators produce stochastic processes that

require tracking an uncountable infinity of probabilistic features for optimal

prediction. For processes generated by hidden Markov chains the consequences are

dramatic. Their predictive models are generically infinite-state. And, until

recently, one could determine neither their intrinsic randomness nor structural

complexity. Recently, methods to accurately calculate the Shannon entropy rate

(randomness) and to constructively determine their minimal (though, infinite) set of

predictive features have been introduced. Leveraging this, we address the

complementary challenge of determining how structured hidden Markov

processes are by calculating their statistical complexity dimension---the

information dimension of the minimal set of predictive features.  This tracks

the divergence rate of the minimal memory resources required to optimally

predict a broad class of truly complex processes.