Large language models (LLMs)

Working Definition

A large language model (LLM) is a deep neural network—almost always built on the Transformer architecture—trained across hundreds of billions or trillions of text tokens on the self-supervised objective of predicting the next token in a sequence, with parameter counts ranging from billions to trillions. Its defining property is not hardcoded domain knowledge, but the fact that scaling a single universal next-token prediction objective across vast corpora induces general-purpose capabilities that were never explicitly programmed: machine translation, summarization, code synthesis, and solving novel tasks specified solely through natural-language instructions in the prompt. In everyday usage, “LLM” refers to a complete conversational product (such as ChatGPT, Claude, or Gemini)—that is, a pretrained base model after instruction tuning, alignment, and tool integration; in its strict technical sense, it denotes the underlying statistical sequence predictor itself. Clear analysis of what LLMs can and cannot do requires keeping these two layers distinct.

Genesis and Global Lineage

The intellectual lineage of the large language model is statistical rather than rule-based. In 1948, Claude Shannon showed how English prose could be approximated by sampling from n-gram letter and word transition probabilities, and statistical n-gram models served for decades as the backbone of speech recognition and machine translation. Neural networks entered language modeling in stages: Yoshua Bengio’s feedforward neural probabilistic language model (2003), recurrent neural network language models (RNN LMs, 2010), and dense word embeddings (word2vec, 2013). The decisive architectural breakthrough arrived with the Transformer (Vaswani et al., 2017), replacing sequential recurrence with parallelizable self-attention. Between 2018 and 2020, OpenAI’s GPT-1, GPT-2, and GPT-3 combined the decoder-only Transformer with a scaling hypothesis: steadily increasing parameters, data, and compute yields predictable power-law gains. Tom B. Brown and colleagues’ May 2020 paper Language Models are Few-Shot Learners (the 175-billion-parameter GPT-3) established the modern concept: a language model large enough to perform new tasks from a few examples inside the context window without weight updates. After November 2022, “LLM” swept into public policy and mass culture as a metonym for all contemporary AI.

Central European Lineage

The transition from classic n-gram tables to modern neural language modeling has a direct, documented root in Central Europe—at the Brno University of Technology (Vysoké učení technické v Brně, VUT) in Czechia. In 2010, Tomáš Mikolov, then a doctoral researcher in Jan Černocký’s Speech@FIT group at VUT Brno, published Recurrent neural network based language model at Interspeech 2010. That four-page paper showed that a recurrent neural network with a continuous hidden state could decisively beat smoothed n-gram baselines in perplexity and speech-recognition word-error rate; Mikolov expanded this work in his 2012 Brno PhD thesis, Statistical Language Models Based on Neural Networks. Moving to Google in California, he led the creation of word2vec (2013). When the GPT-3 authors (Brown et al., 2020) reconstruct their technical lineage in the opening paragraph of their introduction, they trace modern pretrained language models directly back to learned word vectors and cite Mikolov et al. (2013). In addition, the 2017 Transformer architecture on which all modern LLMs run was co-authored by Wrocław-educated computer scientist Łukasz Kaiser and Kharkiv-educated engineer Illia Polosukhin, building on the 2014 neural attention mechanism introduced by Minsk-educated researcher Dzmitry Bahdanau.

Why Now

The concept moved from computational-linguistics conferences to the center of global economics in three steps: GPT-3 (2020) proved scale unlocked few-shot adaptability; Stanford’s 2021 “foundation models” report named the industrial concentration; and ChatGPT (November 2022) turned an instruction-tuned LLM into the fastest-adopted consumer app in history. Since 2023, rapid frontier releases (GPT-4, Claude 3.5, Gemini, Meta’s open-weight Llama series, and Chinese models like Qwen and DeepSeek) have collided with structural conflicts over copyright lawsuits, data-center energy and water footprints, and statutory regulation (the EU AI Act of 2024). Simultaneously, a scientific controversy has erupted over evaluation: because massive web scrapes ingest standard tests (benchmark contamination), researchers struggle to separate genuine generalization from memorized training patterns.

Explanatory Power

Understanding an LLM through its training objective separates what the system literally computes from anthropomorphic illusions. First, it explains why “hallucinations” are an intrinsic property rather than a bug: because the base model maximizes the conditional probability of the next token given preceding tokens, a fluent, genre-appropriate falsehood is mathematically indistinguishable from a fact whenever the training distribution fails to penalize it. Second, it accounts for the jagged capability profile of LLMs—why a model passes bar-exam essays yet fails at counting letters or multiplying multi-digit numbers: performance tracks the density of analogous token patterns in the corpus and single-pass forward computation limits, not human cognitive difficulty. Third, it illuminates AI’s political economy: because capability scales with compute clusters, proprietary data partnerships, and post-training pipelines, capital-intensive labs in the US and China capture upstream infrastructure while smaller countries and firms become downstream API tenants or local fine-tuners.

Axes of Debate

Underappreciated Angle

The least examined structural property of an LLM is that it is a lossy compression of a specific historical and linguistic archive. Well over half of pretraining text in major LLMs consists of English web pages from 2010–2024; Central and Eastern European languages spoken by tens of millions—Polish, Czech, Hungarian, Romanian, Ukrainian—account for tiny single-digit or fractional-percent slices, and their pre-digital print heritage remains largely uningested. Because frontier models speak fluent Polish or Czech at the surface while drawing conceptual associations from an Anglo-American core, they act as invisible cultural translators: answering questions about law, history, or social norms using American institutional defaults dressed in Central European syntax. Moreover, because English-centric tokenizers split morphologically rich Slavic and Finno-Ugric words into more fragments, users in Central Europe literally pay more compute, latency, and API fees per sentence to process their own native languages—introducing a quiet new form of epistemic homogenization.