LLM Tokens: How a Model Understands Your Text
Have you ever wondered how an LLM understands the text you send it? LLM tokens are the small pieces a model cuts your text into, each one turned into a list of numbers. By the end you will know how those numbers capture meaning and how the model reads your whole text.
What really happens when I press send on an LLM request?
Let's follow one request from click to answer in five steps: text, tokens, numbers, reading, and one token at a time. Our example is a support-ticket triage bot, a feature that sorts customer messages by intent, and its whole request is an instruction plus one ticket:
🐍Python12345678910111213141516import json, struct, numpy as np, onnxruntime as ort from tokenizers import Tokenizer tok = Tokenizer.from_file("tokenizer.json") # GPT-2: 50,257 pieces model = ort.InferenceSession("onnx/decoder_model.onnx") f = open("model.safetensors", "rb") n = struct.unpack("<Q", f.read(8))[0] h = json.loads(f.read(n)) a, b = h["wte.weight"]["data_offsets"] f.seek(8 + n + a) table = np.frombuffer(f.read(b - a), dtype=np.float32).reshape(50257, 768) # embeddings def probs(text): ids = tok.encode(text).ids lg = model.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), np.int64)})[0][0, -1] ...
📄Text12341) the request is one string: You are a support triage bot. Reply with JSON only: {"intent": "refund|shipping|account|other"}. Ticket: My order A-1042 has not arrived, please refund it. Reply:
That string is all the model gets. It sees no screen and no "ticket", like a letter whose reader only ever holds the page. Save this snippet and the next four, in order, as behind_the_send.py. It runs GPT-2 small, a 2019 model small enough for a laptop, with the files from openai-community/gpt2 on Hugging Face and pip install tokenizers==0.22.2 onnxruntime==1.23.2 numpy==2.4.2.
Are LLM tokens just words, and who decides the cuts?
A string is only letters and a model only does arithmetic, so the first job is cutting it into LLM tokens: pieces of text from a fixed vocabulary.
🐍Python123ids = tok.encode(TICKET).ids print(f"\n2) {len(TICKET)} characters -> {len(ids)} tokens") print([(tok.decode([i]), i) for i in ids])
📄Text122) 50 characters -> 14 tokens [('My', 3666), (' order', 1502), (' A', 317), ('-', 12), ('10', 940), ('42', 3682), (' has', 468), (' not', 407), (' arrived', 5284), (',', 11), (' please', 3387), (' refund', 12929), (' it', 340), ('.', 13)]
Did you expect 50 characters to become 14 pieces? The model sees only each piece's id, never its letters. The cuts come from GPT-2's 50,257-piece vocabulary, built once, before training, by gluing frequent neighbours together. That method is called byte pair encoding, and the GPT-2 paper by Radford et al. at OpenAI describes it. Like Lego, a common word is one big brick and a rare one is several small ones, so A-1042 became A, -, 10, 42. Even the space counts: refund is one piece, refund without the space is ref + und.
How can a list of numbers carry meaning?
Ids are only catalogue numbers, so where does the meaning come from? Let's look up two pieces in GPT-2's own table and print their nearest neighbours.
🐍Python12345print("3) nearest neighbours in the embedding table") for w in (" refund", " parcel"): v = table[tok.encode(w).ids[0]] s = table @ v / (np.linalg.norm(table, axis=1) * np.linalg.norm(v)) print(f"{w!r}: row of {len(v)} numbers, starts {[round(float(x), 2) for x in v[:3]]} ->", [tok.decode([int(i)]) for i in np.argsort(-s)[1:5]])
📄Text1233) nearest neighbours in the embedding table ' refund': row of 768 numbers, starts [0.13, -0.25, 0.09] -> [' refunds', ' reimb', ' reimbursement', ' reimburse'] ' parcel': row of 768 numbers, starts [0.07, -0.09, 0.05] -> [' parcels', ' courier', ' package', ' postal']
Each id points to a row in the embedding table, 768 numbers per piece, tuned in training so pieces used in similar places get similar numbers. Like shops of one kind on one street, refund lands beside reimburse. Picture each token as a pin on a map. Does the model need a definition of "reimburse"? No, it only needs that pin near the ones it saw in similar sentences.
Does the model read the whole request or only the last word?
A table gives it the same address everywhere, so how does the model learn that this one means the order? Through attention: each piece asks which earlier pieces it needs and takes in their meaning, in 12 rounds, called layers, in GPT-2 small. Same ending, three starts, and the three likeliest next pieces:
🐍Python123print("4) same ending, different start") for start in ("My parcel never arrived.", "I forgot my password.", "I love your product."): print(f"{start} Please send me a ->", top(probs(f"{start} Please send me a")))
📄Text12344) same ending, different start My parcel never arrived. Please send me a -> ' message' 0.43 ' note' 0.10 ' picture' 0.06 I forgot my password. Please send me a -> ' message' 0.40 ' valid' 0.18 ' reset' 0.11 I love your product. Please send me a -> ' message' 0.34 ' note' 0.10 ' picture' 0.07
Only the password start puts valid and reset in the top three, so the earlier words changed what comes next. Here we see the effect, not a printout of attention. It is like scrolling up a group chat to find what "it" means. The it pin listens to the order pin. A piece only looks back, and only within GPT-2's 1024-token context window, the most text it can see at once.
So is it really just predicting the next word?
Every piece knows its neighbours, so how does that become an answer? Like a phone's suggestion strip, tapped again and again, GPT-2 writes the reply in six steps. Each step pins one more token after the last.
🐍Python1234567print("5) the reply, one token at a time") for step in range(1, 7): p = probs(request) best = int(p.argmax()) print(f"step {step}: {top(p)}") request += tok.decode([best]) print("reply so far:", request.split("Reply:")[1])
📄Text123456785) the reply, one token at a time step 1: ' {"' 0.39 '\n' 0.06 ' "' 0.05 step 2: 'intent' 0.16 'ref' 0.04 'status' 0.04 step 3: '":' 0.99 '":"' 0.01 '":{"' 0.00 step 4: ' "' 0.91 ' {"' 0.03 ' {' 0.01 step 5: 'ref' 0.76 'return' 0.01 'Ref' 0.01 step 6: 'und' 1.00 'use' 0.00 'ind' 0.00 reply so far: {"intent": "refund
So yes, it predicts the next word, with two fixes. First, the unit is a token, not a word. Second, what comes out is a probability for each of the 50,257 pieces, then one pick, the most likely or a random draw as in our Temperature post. In "Attention Is All You Need" (2017), Vaswani et al. call the part that writes the reply auto-regressive: it feeds its own output back in. "Reply with JSON only" is earlier text that makes {" likely, and Ouyang et al. show in "Training language models to follow instructions with human feedback" that chat models also learn from instructions followed by answers. GPT-2 is a base model, not tuned to follow instructions, so it only continues the text, and we ran no chat model.
Conclusion
So what really happens when you press send on an LLM request, like our triage bot's ticket? One string is cut into 14 LLM tokens, each a pin on the map, attention lets every pin read the earlier ones, and the reply is written one token at a time: ref, und. Tomorrow, paste your own prompt and a real message into behind_the_send.py and note which piece surprised you. Next time: how a model understands a task.
Sources
- Vaswani et al., "Attention Is All You Need" (2017), section 3: arxiv.org/abs/1706.03762
- Radford et al., "Language Models are Unsupervised Multitask Learners" (GPT-2, 2019): OpenAI PDF
- Ouyang et al., "Training language models to follow instructions with human feedback" (2022): arxiv.org/abs/2203.02155
Enjoyed this post?
Get the next one in your inbox. No spam, unsubscribe any time.
Comments
0 comments

