摘要：
Large language models (LLMs) have transformed how we generate and process information, yet two foundational challenges remain: ensuring the authenticity of their outputs and accurately evaluating their true capabilities. In this talk, I argue that both challenges are, at their core, statistical problems, and that statistical thinking can play an important role in advancing reliable and principled research on large language models. I will present two lines of work that approach these problems from a statistical perspective.
The first part introduces a statistical framework for language watermarks, which embed imperceptible signals into model-generated text for provenance verification. By formulating watermark detection as a hypothesis testing problem, this framework identifies pivotal statistics, provides rigorous Type I error control, and derives optimal detection rules that are both theoretically grounded and computationally efficient. It clarifies the theoretical limits of existing methods, such as the Gumbel-max watermark, and guides the design of more robust and powerful detectors. The second part focuses on language model evaluation, where I study how to quantify the unseen knowledge that models possess but may not reveal through limited queries. To that end, I introduce a statistical pipeline, based on the smoothed Good–Turing estimator, to estimate the total amount of a model’s knowledge beyond what is observed in benchmark datasets. The findings reveal that even advanced LLMs often articulate only a fraction of their internal knowledge, suggesting a new perspective on evaluation and model competence. Together, these projects represent an ongoing effort to develop statistical foundations for trustworthy and reliable language models, with applications ranging from watermark detection to model evaluation.
This talk is based on the following works:
//arxiv.org/abs/2404.01245
//arxiv.org/abs/2506.02058
and will briefly mention follow-up studies:
//arxiv.org/abs/2411.13868
//arxiv.org/abs/2510.22007
论坛简介：该线上论坛是由张志华教授机器学习实验室组织，每两周主办一次（除了公共假期）。论坛每次邀请一位博士生就某个前沿课题做较为系统深入的介绍，主题包括但不限于机器学习、高维统计学、运筹优化和理论计算机科学。
