AI customer service
AI customer service usually combines three kinds of models: a cheap small model classifies the intent, an embedding model finds the relevant policy entries, and a chat model writes the reply from them, handing off to a human when it cannot answer. For 20,000 conversations and about 80,000 replies a month, at live prices in each model’s cheapest group, model costs are about ¥311 a month.
Prices updated Nov 12, 2026, 8:17 PM|
01
The business problem
Most support questions repeat: when will it ship, how do returns work, how do I use this coupon, how do I change my account details. Agents spend their day on these, customers queue at peak times, and nobody answers at night.
Old keyword bots only match fixed phrasings, so a reworded question gets a wrong answer. A plain LLM has the opposite problem: it does not know your rules and may invent a refund policy you never had.
The safer setup lets the model answer only from your knowledge base, sends uncertain questions and disputes to a person, and keeps every conversation for review.
02
Recommended models
Intent routing (small model)
gemini-2.5-flash-liteGoogleinput ¥0.2 / output ¥0.8 per 1M tokens· Standard groupEvery message first goes through a cheap model that labels it as order, product, complaint or other. Complaints and sensitive topics go straight to a person, and order-status questions can hit your own order API instead of an LLM.
Knowledge-base search (Embeddings)
text-embedding-3-smallOpenAIinput ¥0.04 / output ¥0.04 per 1M tokens· Standard groupPolicies, FAQs and product details are split into short entries and embedded; each question pulls the most relevant few for the chat model. When a policy changes, re-embed only the entries that changed.
Reply generation (chat model)
qwen3.5-plusQweninput ¥1.12 / output ¥6.72 per 1M tokens· China Models groupWrites the reply from the retrieved entries and the last few turns, streamed so users are not left waiting. Support prompts are long and replies short, so the input price drives most of the cost.
Recommended models are picked from the live catalog; prices are each model's cheapest group, updated Nov 12, 2026, 8:17 PM. Click a model to see every group's price.
03
Reference architecture
- 1
Prepare the knowledge base
Turn return rules, shipping notes, FAQs and product specs into short entries, each with its scope and effective date, then embed them into a vector store.
- 2
Route by intent
Every message goes through the small model first. Complaints, refund disputes and legal or medical questions go to a person; order-status questions call your order system.
- 3
Retrieve entries
For the rest, embed the question with the same model and fetch the 3 to 5 most relevant entries.
- 4
Stream the reply
Send the entries, recent turns and the question to the chat model. The system prompt says to answer only from the entries and to output a handoff marker when unsure; stream the reply to the front end.
- 5
Hand off and review
On a handoff marker, two unhappy replies in a row or an explicit request, pass the full conversation to an agent. Spot-check bot replies regularly and add what is missing to the knowledge base.
04
Cost estimate
Assumptions: 20,000 conversations a month at about 4 turns each, so about 80,000 replies; about 300 input tokens per intent check; about 2,500 input tokens per chat call (800 system prompt, 1,200 knowledge entries, 500 history) and a 150-token reply.
| Item | Model | Usage assumed | Cost |
|---|---|---|---|
| Intent routing | gemini-2.5-flash-lite | 80,000 × (300 input + 20 output tokens) | ¥6.08 |
| Question embedding | text-embedding-3-small | 80,000 × 30 input tokens | ¥0.096 |
| Reply generation | qwen3.5-plus | 80,000 × (2,500 input + 150 output tokens) | ¥304.64 |
| Total (per month) | ¥310.82 | ||
Calculated from live prices in each model's cheapest group, updated Nov 12, 2026, 8:17 PM.
Model usage only: vector store, servers and human agents are not included. Initial knowledge-base embedding is a one-off cost of total entry tokens times the embedding price. Twice the conversations means roughly twice the cost; longer replies raise the output cost.
05
Sample code
A small model routes the intent and complaints go to a person; other questions get a streamed reply grounded in knowledge entries, and a handoff marker in the reply passes the chat to an agent. Replace the retrieval part with your own knowledge base.
import os
from openai import OpenAI
client = OpenAI(base_url="https://open.tutujin.com/v1", api_key=os.environ["TUTUJIN_API_KEY"])
SYSTEM = (
"You are the support assistant of an online store. Answer only from the reference material. "
"If the material does not cover the question, or it is about a refund dispute, legal or medical matters, "
"reply with the single word HANDOFF so a human agent takes over."
)
def classify(message):
"""Cheap intent check before the main model: order, product, complaint or other."""
resp = client.chat.completions.create(
model="gemini-2.5-flash-lite",
messages=[
{"role": "system", "content": "Classify the message as order, product, complaint or other. Reply with the label only."},
{"role": "user", "content": message},
],
max_tokens=5,
temperature=0,
)
return resp.choices[0].message.content.strip().lower()
def reply_stream(message, references, history):
messages = [{"role": "system", "content": SYSTEM + "\n\nReference material:\n" + references}]
messages += history[-6:] # keep only the last few turns
messages.append({"role": "user", "content": message})
stream = client.chat.completions.create(
model="qwen3.5-plus",
messages=messages,
stream=True,
temperature=0.3,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
message = "When will my order ship?"
if classify(message) == "complaint":
print("-> human agent")
else:
# references: top matches from your knowledge base, embedded with text-embedding-3-small
references = "In-stock orders ship within 48 hours of payment. Pre-orders ship on the date shown on the product page."
reply = ""
for piece in reply_stream(message, references, history=[]):
reply += piece
print(piece, end="", flush=True)
if "HANDOFF" in reply:
print("\n-> human agent")FAQ
How much does AI customer service cost per month in model usage?
With 20,000 conversations, about 80,000 replies and about 2,500 tokens of context per reply, model costs in the cheapest groups come to about ¥311 a month. The main drivers are the number of replies and how much knowledge text you send each time.
What if the bot gives a wrong answer?
Tell the model in the system prompt to answer only from the given entries and to output a handoff marker otherwise, keep full transcripts and spot-check them. An LLM cannot be right every time, so commitments involving money, such as refunds or compensation, should always be confirmed by a person.
Do I need vector search?
With only a few dozen entries you can put them all in the system prompt and skip retrieval. Once the knowledge base grows, use vector search, or every request will carry a large number of input tokens.
How do I make replies feel faster?
Turn on streaming so the front end shows text as soon as the first chunk arrives, use a small model with a short output limit for intent routing, and keep only the last few turns of history.
Can web, app and messaging channels share one setup?
Yes. All model calls happen on your server, so each channel just forwards messages to the same service. Use a separate API key per channel or environment to track usage separately.