Data labeling and classification

• First pass (small model)

  gemini-2.5-flash-liteGoogleinput ¥0.2 / output ¥0.8 per 1M tokens· Standard group

  Handles every item. Picking one label from a fixed list needs no heavy reasoning, so a small model with a low input price and fast responses is enough.

  Alternativesgpt-4o-miniin ¥0.3/1Mqwen-flashin ¥0.196/1Mdoubao-seed-2-0-mini-260215in ¥0.28/1M

• Second opinion on low-confidence items

  qwen3.5-plusQweninput ¥1.12 / output ¥6.72 per 1M tokens· China Models group

  Only sees items the first pass was unsure about. A model from a different vendor reduces the systematic bias of a single model; items where the two disagree go to a person.

  Alternativesdeepseek-v4.1-flashin ¥2.8/1MClaude Haiku 5.5in ¥0.4/1Mgpt-5.4-miniin ¥1.5/1M

Recommended models are picked from the live catalog; prices are each model's cheapest group, updated Nov 12, 2026, 8:17 PM. Click a model to see every group's price.

1. 1

   Write label definitions

   Give each label a one-line definition and two or three examples and counterexamples, and spell out borderline cases such as "late delivery but happy with the product".

2. 2

   First pass with a small model

   Use a fixed system prompt and ask for JSON with the label and a confidence from 0 to 1; temperature 0 and a short output limit.

3. 3

   Validate in code

   Labels outside the list and JSON that fails to parse are marked for review.

4. 4

   Second model

   Items below a confidence threshold, say 0.7, are labeled again by another model; agreement is accepted, disagreement goes to the human queue.

5. 5

   Human spot checks

   Check a random 1% to 2% of each batch and track the agreement rate; when it drops, refine definitions and examples and rerun the affected categories.

Assumptions: 500,000 short texts (reviews, tickets) a month, each with about 400 input tokens (about 300 for definitions and examples, 100 for the text) and 30 output tokens; 5% (25,000) low-confidence items go to the second model.

Calculated from live prices in each model's cheapest group, updated Nov 12, 2026, 8:17 PM.

Model usage only: human spot checks are not included. Longer label definitions mean more input tokens per item, and longer texts cost more. Twice the items means roughly twice the cost.

A small model returns a label and confidence as JSON; low-confidence rows go to a second model, disagreements are flagged for people, and results are written as JSONL.

import json
import os
import random
from concurrent.futures import ThreadPoolExecutor

from openai import OpenAI

client = OpenAI(base_url="https://open.tutujin.com/v1", api_key=os.environ["TUTUJIN_API_KEY"], max_retries=3)

LABELS = ["shipping", "quality", "price", "service", "other"]
SYSTEM = (
    "You label customer reviews. Choose exactly one label from: " + ", ".join(LABELS) + ". "
    'Reply with JSON only, for example {"label": "quality", "confidence": 0.9}.'
)


def label(item, model):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": item["text"]}],
        response_format={"type": "json_object"},
        temperature=0,
        max_tokens=50,
    )
    raw = resp.choices[0].message.content
    try:
        result = json.loads(raw[raw.find("{"): raw.rfind("}") + 1])
    except ValueError:
        result = {}
    if result.get("label") not in LABELS:
        return {"id": item["id"], "label": None, "confidence": 0.0}
    return {"id": item["id"], "label": result["label"], "confidence": float(result.get("confidence", 0))}


items = [{"id": 1, "text": "Arrived two days late, but the mug itself is lovely."}]

with ThreadPoolExecutor(max_workers=8) as pool:
    first = list(pool.map(lambda it: label(it, "gemini-2.5-flash-lite"), items))

# second opinion for low-confidence rows from another vendor's model
by_id = {it["id"]: it for it in items}
for row in first:
    if row["confidence"] < 0.7:
        second = label(by_id[row["id"]], "qwen3.5-plus")
        row["needs_human"] = second["label"] != row["label"]
        row["label"] = second["label"] if row["label"] is None else row["label"]

# a random 2% always goes to people for quality checks
for row in random.sample(first, max(1, len(first) // 50)):
    row["needs_human"] = True

with open("labels.jsonl", "w", encoding="utf-8") as f:
    for row in first:
        f.write(json.dumps(row, ensure_ascii=False) + "\n")