lots of addition and multiplication (and a few max())
max()
lots and lots of addition and multiplication (and a few max())
source: i.redd.it/t87gswsbmnq41.jpg
(= variable of type float)
float
for our purposes = any textual content
(the linguists are crying right now )
= a group of numbers
[0.51, -0.12, …, 0.49]
...and a point in space
word2vec (projected from ~300 dimensions to 2-D)
Tomáš Mikolov, 2013
source: arxiv.org/abs/1301.3781
Model: Qwen/Qwen3-14B
Character-level: len = 54 tokens, vocab_size = ~27 [T] [r] [a] [n] [s] [f] [o] [r] [m] [e] [r] [s] [ ] [r] [e] [v] [o] [l] [u] [t] [i] [o] [n] [i] [z] [e] [d] [ ] [n] [a] [t] [u] [r] [a] [l] [ ] [l] [a] [n] [g] [u] [a] [g] [e] [ ] [p] [r] [o] [c] [e] [s] [s] [i] [n] [g]
[T] [r] [a] [n] [s] [f] [o] [r] [m] [e] [r] [s] [ ] [r] [e] [v] [o] [l] [u] [t] [i] [o] [n] [i] [z] [e] [d] [ ] [n] [a] [t] [u] [r] [a] [l] [ ] [l] [a] [n] [g] [u] [a] [g] [e] [ ] [p] [r] [o] [c] [e] [s] [s] [i] [n] [g]
Word-level: len = 5 tokens, vocab_size = ~∞ [Transformers] [revolutionized] [natural] [language] [processing]
[Transformers] [revolutionized] [natural] [language] [processing]
Subword-level: len = 8 tokens, vocab_size = ~50000 [Trans][form][ers] [revolution][ized] [natural] [language] [process][ing]
[Trans][form][ers] [revolution][ized] [natural] [language] [process][ing]
Try it: platform.openai.com/tokenizer
source: huggingface.co/blog/royswastik/transformer-tokenization-vocabulary-creation
source: arxiv.org/pdf/2305.13707
animatedllm.github.io/pretraining-simple
source: xkcd.com/2169
11TB of cleaned data (from 200 TB of raw text)
source: arxiv.org/pdf/2402.00159
Input: Explain the moon landing to a 6 year old.
Pre-trained model Explain why the sky is blue to a 6 year old. Explain why planes can fly to a 6 year old. (...)
Instruction-tuned model Many years ago, people wanted to explore the Moon. So they built a huge rocket (...)
source: arxiv.org/abs/2203.02155
Task: Czech → English translation
Pretrained model
src: pes tgt: dog src: Mám hlad. tgt: I am hungry. src: {user_input} tgt:
Instruction-tuned model
You are an expert translator. Your task is to translate the given source sentence from Czech to English. Output nothing else, just the translation. src: {user_input}
Fine-tuned model
{user_input}
= same as pretraining, but on your custom data*
Mámhlad.Iamhungry.
(nope, but this demo is cool)
bbycroft.net/llm
source: en.wikipedia.org/wiki/Bfloat16_floating-point_format
At least 140 GB of GPU RAM for inference (much more for training / finetuning)
source: huggingface.co/meta-llama/Llama-3.3-70B-Instruct
80 GB GPU RAM
~ $25,000
source: en.wikipedia.org/wiki/Matrix_multiplication
source: arxiv.org/pdf/2009.06489
We can run eval() on (specific parts of) the model’s answer.
eval()
source: arxiv.org/abs/2302.04761
source: modelcontextprotocol.io
source: cme295.stanford.edu
source: simonwillison.net/2025/May/22/tools-in-a-loop
source: arxiv.org/abs/2210.03629
https://courses.fit.cvut.cz/NI-NLM/index.html
Zdeněk Kasner
new
Anna Sajdoková
Link to the slides