TOKEN METROLOGY FOR LARGE LANGUAGE MODELS: A SURVEY FROM THE PERSPECTIVE OF MEASUREMENT SCIENCE

Abstract

The token has become the de facto universal unit of measurement in the large language model (LLM) ecosystem: model pricing is quoted per token, context windows are expressed in tokens, inference throughput is measured in tokens per second, and training corpora are reported in trillions of tokens. Yet the token is not a well-defined measurement unit in the metrological sense—it varies with the tokenizer, the vocabulary, text normalization, and the vendor’s counting convention; it is incommensurable across models; and it lacks calibration, traceability, and an assessable uncertainty. This paper examines token metering as an object of measurement science. We first survey the state of token counting, covering the lineage of tokenization techniques, multilingual and domain disparities, divergent billing conventions, and reproducibility issues, substantiated by a first-hand benchmark across 13 tokenizers and 18 multilingual parallel texts. Drawing on the vocabulary of metrology (the International Vocabulary of Metrology and the Guide to the Expression of Uncertainty in Measurement), we then establish a four-dimensional framework—definiteness, commensurability, traceability, and uncertainty—through which we identify four systemic deficiencies of token counting as a measurement system. We argue that these deficiencies undermine billing fairness, cost comparability, and scientific reproducibility. Finally, we review existing mitigation efforts and outline a standardization agenda spanning technical, metrological, and scholarly-reporting levels. We conclude that token metering currently occupies a “pre-SI” stage of measurement, and that establishing a public reference standard and measurement norms is a prerequisite for the LLM economy and for reproducible LLM science.