Rate limits

The API enforces two independent guards:

1. Per-tenant request rate limit — token-bucket on every endpoint.
2. Per-tenant LLM cost cap — rolling-window dollar budget on LLM use.

Both surface as HTTP 429, with different code values so clients can react appropriately.

Request rate limit

A token bucket per tenant_id (or per sub for service tokens that don't carry a tenant). Two settings drive it:

Setting	Default	Meaning
RATE_LIMIT_CAPACITY	120	Bucket size — burst budget.
RATE_LIMIT_REFILL_PER_SECOND	2	Tokens added back per second — sustained rate.

So the default plan allows a burst of 120 requests, then steady-state 2 RPS. Premium tiers raise both numbers.

429 response

{
    "error": {
        "code": "rate_limited",
        "message_key": "errors.common.rate_limited",
        "message": "Too many requests. Retry shortly.",
        "details": { "retry_after_seconds": 5 },
        "request_id": "019e32b5-..."
    }
}

A Retry-After header mirrors details.retry_after_seconds. Clients should honour it; aggressive retries inside the window do not refill the bucket faster.

Recommended client behaviour

• Single-shot work — back off retry_after_seconds, retry once.
• Batch / fan-out — keep a local semaphore matching your steady-state rate. Don't trust 429 as the only signalling channel.
• Background sync — exponential backoff with jitter, cap at 60 s.

LLM cost caps

Per-tenant rolling window over actual provider spend. The router calls ensure_budget(tenant_id) before every LLM call and record(tenant_id, cost_usd) on success.

Setting	Default	Meaning
LLM_COST_CAP_PER_HOUR_USD	unset (guard off)	Dollar cap per window. null disables the guard.
LLM_COST_WINDOW_SECONDS	3600	Window length.

Exceeding the cap produces:

{
    "error": {
        "code": "llm_budget_exceeded",
        "message_key": "errors.llm.budget_exceeded",
        "message": "LLM cost budget exceeded for tenant.",
        "details": {
            "tenant_id": "tenant-a",
            "cap_usd": 5.0,
            "spent_usd": 5.12
        }
    }
}

The 429 is returned before any LLM call is attempted, so a misbehaving tenant cannot trigger fallback storms.

Which endpoints touch the cap?

• POST /v1/interpretations when use_llm_rewrite: true
• Internal eval / training jobs (not exposed via REST)

Endpoints that don't call an LLM are not affected by the cost cap; they remain bounded by the request rate limit only.

Capacity hints

For a single API pod on baseline hardware (1 vCPU, 768 MiB):

Scenario	Comfortable RPS	Notes
Chart create	100	CPU-bound on Swiss Ephemeris.
Talent / interpretation (no LLM)	200	DB-bound.
Interpretation with LLM	5–10	Upstream-bound; subject to provider quotas.
Report enqueue	100	Cheap — just a DB write + dramatiq publish.
Report status poll	500	Cached.

Production runs with HPA targeting 70 % CPU; with 3–10 replicas the sustained ceiling is roughly 10× the per-pod numbers.