Estimating Hallucination via Divergence Functionals and Posterior Risk

Abstract

Hallucination in large language models constitutes a latent statistical phenomenon arising from discrepancies between predictive probability measures and the underlying data-generating process. Existing evaluation methodologies predominantly rely on deterministic factuality scores or benchmark-specific accuracy measures, offering limited inferential interpretation and weak statistical generalizability. This paper introduces a unified inferential framework for estimating hallucination through divergence functionals and posterior risk. Hallucination is formulated as an unobserved random functional defined over sequential predictive distributions, enabling its estimation via discrepancies between posterior predictive measures and externally validated reference distributions. The proposed methodology integrates families of statistical divergence functionals, including Kullback–Leibler, Jensen–Shannon, Rényi, and Wasserstein divergences, with Bayesian posterior risk to construct a consistent estimator of hallucination intensity under heterogeneous uncertainty regimes. Asymptotic properties of the estimator, including consistency, asymptotic normality, and convergence under increasing sample complexity, are established within a semiparametric inference framework. Statistical uncertainty is quantified using influence-function approximations, bootstrap confidence intervals, and posterior credible regions, facilitating principled hypothesis testing of hallucination prevalence across model architectures and prompting strategies. An empirical instantiation on clinical dialogue summarization compares three candidate divergence-style labels and finds that a Wasserstein-2 divergence label yields a substantially more learnable estimation target than a lexical word-overlap baseline (test-set R² of 0.972 versus 0.513), while a reconstruction-based novelty label did not yield a usable signal under the configuration tested. The framework establishes a theoretically grounded statistical paradigm for reproducible hallucination estimation and comparative reliability assessment in generative artificial intelligence.