Skip to navigation

Routing

model supports two ways of calling: send model: "auto" and JoyToken selects the final model and provider for the request (recommended), or send a concrete model ID from the catalog to pin the request to that model.

Common Requests

GoalRequest
Smart routing (recommended)model: "auto"
Select a tiermodel: "auto" + `tier: “economy"
Pin a specific modelmodel: "<modelName from the catalog>"

Smart Routing

Send model: "auto" and JoyToken’s smart routing engine takes over. It combines multi-dimensional request features, semantic rules, and LLM review to identify system settings, tool intent, context, and user needs, determine the task type and complexity, and apply the matching routing strategy.

For complex requests, the system automatically builds a task plan — breaking the request down, matching models and tiers to each task, and aggregating the results. Simple requests go straight to single-model selection.

Within policy constraints, the system schedules intelligently across a continuously expanding pool of online models and capabilities, weighing quality, cost, latency, and real-time success rate, and keeps service quality and continuity through automatic circuit breaking, traffic switching, and cross-tier fallback.

{
"model": "auto",
"messages": [
{ "role": "user", "content": "Summarize this ticket in one sentence." }
]
}
curl -X POST "$JOY_TOKEN_API_BASE_URL/openai/v1/chat/completions" \
-H "Authorization: Bearer $JOY_TOKEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "auto",
"messages": [{ "role": "user", "content": "Summarize this ticket in one sentence." }]
}'

Pin a Specific Model

Pass a modelName from GET /api/v1/models as the request model. JoyToken then skips scoring and routing and sends the request straight to that model.

{
"model": "deepseek-v4-flash",
"messages": [
{ "role": "user", "content": "Summarize this ticket in one sentence." }
]
}
curl -X POST "$JOY_TOKEN_API_BASE_URL/openai/v1/chat/completions" \
-H "Authorization: Bearer $JOY_TOKEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-v4-flash",
"messages": [{ "role": "user", "content": "Summarize this ticket in one sentence." }]
}'

Tiers

tier narrows the candidate pool for an auto request. Omit it to let JoyToken decide.

TierUse
economyCost-first, batch, low-risk tasks
standardBalanced default
premiumQuality-first or critical evaluation tasks

Orchestration (Multi-task Planning)

For complex prompts, JoyToken may handle a single model: "auto" request in orchestration mode: it first builds a task plan, runs each task with a selected sub-model, then aggregates a final answer. You do not opt in explicitly; the gateway decides based on the request.

How to detect it

SignalMeaning
plan (top level)Ordered task list, present only in orchestration mode
choices[].message.contentA JSON-encoded array of per-task outputs (content + title) instead of a single string
metadata (array)One entry per task, including reserved __planner__ and __final__ tasks
streaming planning eventA leading chunk whose orchestration.phase is planning carries the ordered plan, followed by the __planner__ metadata event
streaming task chunkEach task’s content chunk carries an orchestration field (task_id, task_seq, task_status, title)
streaming metadata eventOne standalone metadata event per task (a single object, not an array), emitted right after that task’s content

Task lifecycle

FieldMeaning
task_idTask identifier; __planner__ and __final__ are reserved
task_seqExecution order; 0 is the planner
task_statusTask status, e.g. DONE
tagTask role, e.g. orchestration, search, reasoning, writing, aggregation
titleHuman-readable task title

In orchestration mode each task is billed separately (one metadata/billing entry per task) and may use a different sub-model. Parse choices[].message.content as JSON before rendering, and sum billing.credits_used across all task metadata entries.

Confirm Routing

SignalMeaning
metadata.modelFinal JoyToken model
metadata.tierBilling and routing tier
metadata.billingCredits and token usage

Troubleshooting

SymptomCheck first
tier not allowedAPI key policy allows the tier
no eligible modeltier, blacklist, wallet, and key budget
IP not allowedserver egress IP is allowlisted
auto failspolicy, tier, and balance still leave candidate models
concrete model name rejectedthe name matches the catalog exactly and the API key policy allows that model
pinned model returns model_exhaustedthe name matches the catalog exactly and the model has upstream capacity
pinned model ignores tierexpected: a pinned model always bills at its own tier