Probing the Latent Hierarchical Structure of Data via Diffusion Models

High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, allow one to probe the latent structure of data. We predict in simple hierarchical models that the change of latent variables at different level of abstraction can be triggered by considering different noise level. As a consequence, in this process changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition takes place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models.