Možnost č. 1: přímé generování písmena

prompt = f"""{question}
A) {a} B) {b} C) {c} D) {d}
Answer:"""

response = llm.generate(prompt)
# response = "The answer is C"
answer = parse_letter(response)

Možnost č. 2: písmeno s nejvyšší
pravděpodobností

prompt = f"""{question}
A) {a} B) {b} C) {c} D) {d}
Answer:"""

logprobs = llm.logprobs(
  prompt, tokens=["A","B","C","D"]
)
answer = argmax(logprobs)

Možnost č. 3: odpověď s nejvyšší
pravděpodobností

answers = [
f"{question}\nAnswer: {a}",
f"{question}\nAnswer: {b}",
f"{question}\nAnswer: {c}",
f"{question}\nAnswer: {d}",
]
cumulative_logprobs = [llm.cumulative_logprob(p) for p in prompts]
answer = argmax(cumulative_logprobs)

Změna písmen mění i ranking modelů

_{zdroj: Alzahrani et al. (2024)}

LLM-as-a-judge

You are an a fair and impartial evaluator. Rate the following answer on a scale 1–5:

Question: {question}
Reference answer: {reference}
Model answer: {answer}

Respond with JSON:
{ "accuracy": <1-5>, "style": <1-5>, "completeness": <1-5> }

Self-recognition koreluje s self-preferencí

_{zdroj: Panickssery et al. (2024)}

Bias se liší podle modelů a jejich rodin

_{zdroj: Spiliopoulou et al. (2025)}

Jak to dopadne, když skóre přeformulujeme?

_{zdroj: Li et al. (2025)}

Další typy biasů

_{zdroj: Ye et al. (2024)}

A další...

_{zdroj: Ye et al. (2024)}

_{zdroj: Ye et al. (2024)}

Biasy jsou v různé míře napříč modely

_{https://builtin.com/data-science/train-test-split}

_{zdroj: Chollet (2019)}

Zákon nešetří ani ty nejlepší...

_{https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations}

• Finding walkthroughs and answers for cyber CTF challenges online.
• Using DDOS attacks to crash servers instead of exploiting vulnerabilities.
• Cheating on coding benchmarks by looking up more recent code versions, disabling assertions, and adding test-specific logic.

_{zdroj: Miller & Tang (2025)}

As generative AI becomes increasingly embedded in everyday workflows,
it is important to evaluate its performance in ways that reflect real-world usage rather than abstract notions of intelligence

Among the six real-world capabilities we evaluate, Reviewing Work and Data Structuring are not captured at all by the metrics used in current AI benchmarking. Meanwhile, Summarization and Generation, capabilities widely used by non-technical professionals are only weakly supported.