Glossary

Tokens

Units of text (≈4 characters). Count toward cost and context.

• Depends on the model, server load, and prompt size
• Low TTFT indicates that the model responds quickly, which is essential for a smooth user experience

Context window

Maximum text the model “remembers” per request.

• Prompt + completion ≤ model limit
• Larger windows keep long chats coherent
• Exceeding the limit returns an error

Temperature

Controls randomness of generation.

• High: more creative, less predictable
• Low: safer and consistent
• Prefer low for deterministic answers

TTFT

Time to first token.

• Affected by model, network, prompt size
• Low TTFT improves chat UX
• Track it with total latency