How to choose a model: sort tasks by difficulty, then test on your own data

Choosing a model takes three steps: place the task in a difficulty tier, pick two to four candidates in that tier, then test them on 50 to 200 of your own real samples and take the cheapest one that meets the bar. Most business tasks do not need the most expensive model. For classification, extraction and rewriting, a small model often costs a few percent of a flagship's price — compare the live prices below.

Sort tasks by difficulty

To read invoices, screenshots or product photos, pick a model with vision. Knowledge-base search also needs an embedding model and a rerank model — see Enterprise knowledge base and RAG.

Light-tier candidates

Mid-tier candidates

Flagship and reasoning candidates

The tiers follow how each vendor positions its models, and models within a tier still differ a lot. Your own evaluation has the final say.

Test on your own data

1. Collect samples: take 50 to 200 real inputs and write the expected result for each, covering common and tricky cases.
2. Fix the prompt: give every candidate the same system prompt and parameters so the comparison is fair.
3. Run and record: note whether each answer is right, how long it took, and the tokens in usage.
4. Do the math: among models that meet your accuracy bar, multiply average tokens by unit prices and pick the lowest cost per request.
5. Keep a runner-up as a fallback for when the main model has problems.

This script sends the same samples to several candidates and reports accuracy, average latency and average tokens:

import os
import time
from openai import OpenAI

client = OpenAI(base_url="https://open.tutujin.com/v1", api_key=os.environ["TUTUJIN_API_KEY"])

candidates = [
    "gpt-5.4-mini",
    "deepseek-v4.1-flash",
    "qwen3.5-plus",
]
samples = [
    {"text": "My parcel has been stuck for three days", "label": "shipping"},
    {"text": "Can I return this jacket?", "label": "returns"},
    # replace with your own 50 to 200 samples
]
system = "Classify the user message as one of: shipping, returns, product, other. Output the label only."

for model in candidates:
    correct, seconds, tokens_in, tokens_out = 0, 0.0, 0, 0
    for s in samples:
        t0 = time.time()
        resp = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": system},
                {"role": "user", "content": s["text"]},
            ],
        )
        seconds += time.time() - t0
        correct += resp.choices[0].message.content.strip() == s["label"]
        tokens_in += resp.usage.prompt_tokens
        tokens_out += resp.usage.completion_tokens
    n = len(samples)
    print(f"{model}: accuracy {correct / n:.0%}, avg {seconds / n:.2f}s, avg tokens {tokens_in / n:.0f} / {tokens_out / n:.0f}")

Same task, different tiers

A classification task with 300 input tokens and 10 output tokens per request, one million requests a month. On a light model:

On a flagship:

If both score about the same in your evaluation, take the first. Reasoning models also add thinking tokens to the output, which widens the gap.

Combine models to save more

• Tiered routing: let a small model judge the type or difficulty of a request, answer the easy ones directly and pass the hard ones to a bigger model.
• Division of labor: in knowledge-base Q&A, use a small model to rewrite and classify questions and a mid-tier model to write the final answer.
• Fallback: when the main model errors or times out, switch to another model in the same tier — see How to choose a group.

Other things to check

For how prices and billing work, see How LLM API pricing works.