Neural Estimation of Pairwise Mutual Information in Masked Discrete Sequence Models
New method estimates variable dependencies in masked diffusion models using neural networks.
Researchers propose a neural framework to estimate pairwise conditional mutual information (MI) from hidden states of pretrained masked diffusion models (MDMs). This approach improves interpretability and generation efficiency by capturing inter-variable dependencies not explicitly represented in MDMs. The method uses grou...