What it means
Model evaluation is the focused exercise of running the same inputs through different models and scoring the outputs. Sometimes the difference is dramatic (one model handles your domain, another does not). Sometimes it is subtle (both work, but one is 40 percent cheaper).
Good model evaluation goes beyond accuracy. You also measure: latency (how fast does it respond), cost (price per 1,000 tokens), context window (how much can it remember at once), and supported tools (does it do function calling, vision, structured output). The right model is the one that wins on the dimensions that matter for your workflow.
Why it matters
Picking a model on intuition or marketing copy is the easy way and often the wrong way. The same model that is 'best' for general chat may be worse for your specific task. The only honest way to know is to run your eval set against the candidates.
It is also how you keep your stack current. Open-source models ship every few months; frontier models update quarterly. Quarterly model evaluations against your eval set tell you when it is worth swapping.
Example
A wealth advisor evaluates four models on a 180-item eval set of client-memo drafts: GPT-5, Claude Opus, Qwen 3.6 27B, DeepSeek V4 Pro. Claude Opus wins on quality, Qwen 3.6 wins on cost, DeepSeek wins on long-context. The firm picks Claude for client-facing drafts and Qwen for internal summarisation. Same eval set, two different best-fit models.