Exploring Benchmarks

Benchmarks are at the heart of how Phaseo measures model quality and progress. They provide objective, repeatable metrics that help you compare model performance across tasks, domains, and providers — from reasoning and coding to language understanding and image recognition.

---

​

What are benchmarks?

A benchmark is a dataset or evaluation designed to test a model’s performance on a specific task. By running multiple models on the same dataset, we can measure relative strengths and weaknesses. Common examples include:

• 🧠 MMLU (Massive Multitask Language Understanding) — tests general knowledge and reasoning.
• 🔢 GSM8K — tests mathematical reasoning and problem-solving.
• 💬 HellaSwag — evaluates common sense and contextual understanding.
• 🧮 ARC-Challenge — measures scientific reasoning and logic.
• 💡 HumanEval — assesses programming and code generation skills.

---

​

How Phaseo handles benchmarks

Phaseo collects benchmark data from multiple public sources and official reports, then standardises them into a consistent format. Each model’s benchmark section includes:

• Score value — typically expressed as a percentage or normalised scale.
• Dataset source — e.g. MMLU, GSM8K, etc.
• Evaluation method — indicates how the score was derived (official report, third-party eval, or community submission).
• Last updated — shows when the benchmark data was last refreshed.

---

​

Benchmark categories

Benchmarks are grouped into categories that represent the skill being tested:

---

​

How to interpret scores

Keep in mind that some benchmarks have different scoring systems. Phaseo normalises these where possible to make comparisons more meaningful.

---

​

Visualising benchmarks on Phaseo

Benchmark results are visualised using:

• 📊 Bar charts for comparing models within the same dataset.
• 📈 Trend charts showing model improvements over time.
• 🧮 Aggregate scores to summarise performance across multiple datasets.

Each benchmark dataset on Phaseo includes filters for:

• Model type (chat, code, embedding, etc.)
• Provider
• Release year
• Dataset category

This lets you quickly identify patterns — like which families dominate reasoning or coding tasks.

---

​

Data sources and methodology

Phaseo aggregates benchmark data from:

• Official research papers and model reports.
• Provider evaluation pages (OpenAI Evals, Anthropic Reports, etc.).
• Open community datasets (e.g. Hugging Face leaderboards).
• Direct submissions from users via GitHub.

All data is standardised and verified before being published. See Research & Methodology → Benchmark Methodology for full details.

---

​

Example use cases

---

​

Contributing benchmark data

Open a GitHub issue with evidence for new or corrected benchmark scores. Each submission is reviewed before inclusion to ensure consistency and accuracy.

---

​

Next steps

Once you understand how benchmark data works, you can explore the API providers that make these models available programmatically.